andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/util
History
Lei Jin 7e9f28cb23 limit max bytes that can be read/written per pread/write syscall 
Summary:
BlockBasedTable sst file size can grow to a large size when universal
compaction is used. When index block exceeds 2G, pread seems to fail and
return truncated data and causes "trucated block" error. I tried to use
```
  #define _FILE_OFFSET_BITS 64
```
But the problem still persists. Splitting a big write/read into smaller
batches seems to solve the problem.

Test Plan:
successfully compacted a case with resulting sst file at ~90G (2.1G
index block size)

Reviewers: yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22569
2014-08-29 21:21:49 -07:00
..
arena_test.cc
Arena to inline 2KB of data in it.
2014-05-14 11:49:01 -07:00
arena.cc
Fix typo huage => huge
2014-08-14 17:01:20 -07:00
arena.h
In DB::NewIterator(), try to allocate the whole iterator tree in an arena
2014-06-02 17:44:57 -07:00
auto_roll_logger_test.cc
attempt to fix auto_roll_logger_test
2014-08-18 10:23:18 -07:00
auto_roll_logger.cc
Fix a bug causing LOG is not created when max_log_file_size is set.
2014-06-16 10:27:42 -07:00
auto_roll_logger.h
Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL
2014-04-10 15:27:42 -07:00
autovector_test.cc
autovector::resize
2014-05-08 13:50:49 -07:00
autovector.h
Fix autovector iterator increment/decrement comments
2014-08-14 14:56:11 -07:00
benchharness_test.cc
Fix unsigned and signed integer compare
2014-04-29 12:29:22 -07:00
benchharness.cc
Fix more signed/unsigned comparsions
2014-04-29 12:40:18 -07:00
benchharness.h
log_and_apply_bench on a new benchmark framework
2014-05-05 11:11:48 -07:00
blob_store_test.cc
Move the compiler back to 4.8.1 + more small fixes
2013-11-18 11:40:16 -08:00
blob_store.cc
RocksDBLite
2014-04-15 13:39:26 -07:00
blob_store.h
RocksDBLite
2014-04-15 13:39:26 -07:00
bloom_test.cc
Fix more gflag namespace issues
2014-05-09 08:41:02 -07:00
bloom.cc
Changes to support unity build:
2014-08-11 13:22:47 -04:00
build_version.h
Dont build version if compiling for IOS
2014-04-11 10:54:47 -07:00
cache_test.cc
ApplyToAllCacheEntries
2014-05-02 16:24:04 -04:00
cache.cc
ApplyToAllCacheEntries
2014-05-02 16:24:04 -04:00
coding_test.cc
Remove BitStream* tests
2014-08-19 09:52:54 -04:00
coding.cc
Removing BitStream* functions
2014-08-19 06:48:21 -07:00
coding.h
Removing BitStream* functions
2014-08-19 06:48:21 -07:00
comparator.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
crc32c_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
crc32c.cc
Dynamically choose SSE 4.2
2014-04-04 14:03:19 -07:00
crc32c.h
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
db_info_dummper.cc
fix compile error under Mac OS X
2014-08-14 20:01:01 +08:00
dynamic_bloom_test.cc
use arena to allocate memtable's bloomfilter and hashskiplist's buckets_
2014-06-30 15:54:31 -07:00
dynamic_bloom.cc
Changes to support unity build:
2014-08-11 13:22:47 -04:00
dynamic_bloom.h
Adding option to save PlainTable index and bloom filter in SST file.
2014-07-18 16:58:13 -07:00
env_hdfs.cc
cleanup exception text
2014-05-21 07:54:22 -04:00
env_posix.cc
limit max bytes that can be read/written per pread/write syscall
2014-08-29 21:21:49 -07:00
env_test.cc
- hdfs cleanup; fix to NewDirectory to comply with definition in env.h
2014-05-21 07:50:37 -04:00
env.cc
integrate rate limiter into rocksdb
2014-07-08 12:31:49 -07:00
filelock_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
filter_policy.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
hash_cuckoo_rep.cc
Remove MemTableRep::GetIterator(const Slice& slice)
2014-06-25 14:09:29 -07:00
hash_cuckoo_rep.h
Pass logger to memtable rep and TLB page allocation error logged to info logs
2014-05-05 16:43:37 -07:00
hash_linklist_rep.cc
HashLinkList memtable switches a bucket to a skip list to reduce performance outliers
2014-07-01 17:14:15 -07:00
hash_linklist_rep.h
HashLinkList memtable switches a bucket to a skip list to reduce performance outliers
2014-07-01 17:14:15 -07:00
hash_skiplist_rep.cc
use arena to allocate memtable's bloomfilter and hashskiplist's buckets_
2014-06-30 15:54:31 -07:00
hash_skiplist_rep.h
Pass logger to memtable rep and TLB page allocation error logged to info logs
2014-05-05 16:43:37 -07:00
hash.cc
Add a hash-index component for block
2014-03-03 21:11:49 -08:00
hash.h
Changes to support unity build:
2014-08-11 13:22:47 -04:00
histogram_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
histogram.cc
Statistics code cleanup
2014-01-17 12:46:06 -08:00
histogram.h
Statistics code cleanup
2014-01-17 12:46:06 -08:00
iostats_context_imp.h
Fix ios compile
2014-08-28 12:46:05 -04:00
iostats_context.cc
Fix ios compile
2014-08-28 12:46:05 -04:00
ldb_cmd_execute_result.h
Revert "Don't compile ldb tool into static library"
2014-04-15 11:29:02 -07:00
ldb_cmd.cc
move block based table related options BlockBasedTableOptions
2014-08-25 14:22:05 -07:00
ldb_cmd.h
Move include/utilities/*.h to include/rocksdb/utilities/*.h
2014-07-23 10:21:38 -04:00
ldb_tool.cc
[RocksDB] allow LDB tool to have customized key formatter
2014-06-23 15:35:40 -07:00
log_buffer.cc
Fix data corruption by LogBuffer
2014-03-21 15:32:48 -07:00
log_buffer.h
Fix errors while building with clang
2014-05-15 12:34:53 +08:00
log_write_bench.cc
Fix more gflag namespace issues
2014-05-09 08:41:02 -07:00
logging.cc
Clean up compaction logging
2014-05-14 12:13:50 -07:00
logging.h
Clean up compaction logging
2014-05-14 12:13:50 -07:00
manual_compaction_test.cc
Fix CompactRange to apply filter to every key
2014-01-14 16:19:09 -08:00
murmurhash.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
murmurhash.h
Add a hash-index component for block
2014-03-03 21:11:49 -08:00
mutexlock.h
Add separate Read/WriteUnlock methods in MutexRW.
2014-06-16 15:41:46 -07:00
options_builder.cc
Changes to support unity build:
2014-08-11 13:22:47 -04:00
options_test.cc
Add a utility function to guess optimized options based on constraints
2014-07-22 15:24:21 -07:00
options.cc
improve OptimizeForPointLookup()
2014-08-26 14:15:00 -07:00
perf_context_imp.h
macros for perf_context
2014-04-08 10:58:07 -07:00
perf_context.cc
Add a function to return current perf level
2014-07-10 11:35:48 -07:00
posix_logger.h
Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL
2014-04-10 15:27:42 -07:00
random.h
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
rate_limiter_test.cc
expose RateLimiter definition
2014-07-25 15:17:06 -07:00
rate_limiter.cc
expose RateLimiter definition
2014-07-25 15:17:06 -07:00
rate_limiter.h
expose RateLimiter definition
2014-07-25 15:17:06 -07:00
signal_test.cc
Print out stack trace in mac, too
2014-04-23 09:11:35 -04:00
skiplistrep.cc
Remove MemTableRep::GetIterator(const Slice& slice)
2014-06-25 14:09:29 -07:00
slice.cc
FixedPrefixTransform to include prefix length in its name
2014-05-13 16:08:21 -07:00
statistics.cc
Fixing broken Mac build
2014-07-31 20:52:13 -07:00
statistics.h
make statistics forward-able
2014-07-28 12:10:49 -07:00
status.cc
Add timeout_hint_us to WriteOptions and introduce Status::TimeOut.
2014-07-03 15:47:02 -07:00
stl_wrappers.h
Killing Transform Rep
2013-12-03 12:42:15 -08:00
stop_watch.h
Change StopWatch interface
2014-07-28 12:22:37 -07:00
string_util.cc
Turn on -Wmissing-prototypes
2014-04-09 21:17:14 -07:00
string_util.h
Turn on -Wmissing-prototypes
2014-04-09 21:17:14 -07:00
sync_point.cc
Don't compile sync_point if NDEBUG
2014-04-17 10:49:58 -07:00
sync_point.h
Fix ifdef NDEBUG
2014-04-17 14:29:28 -07:00
testharness.cc
Better stack trace in MAC
2014-04-25 09:50:51 -04:00
testharness.h
Print out stack trace in mac, too
2014-04-23 09:11:35 -04:00
testutil.cc
Adding a user comparator for comparing Uint64 slices.
2014-08-27 10:39:31 -07:00
testutil.h
Adding a user comparator for comparing Uint64 slices.
2014-08-27 10:39:31 -07:00
thread_local_test.cc
simplify ThreadLocalPtr a little bit
2014-04-22 21:13:34 -07:00
thread_local.cc
simplify ThreadLocalPtr a little bit
2014-04-22 21:13:34 -07:00
thread_local.h
simplify ThreadLocalPtr a little bit
2014-04-22 21:13:34 -07:00
vectorrep.cc
Remove assert in vector rep
2014-08-27 11:05:41 -07:00
xxhash.cc
xxHash
2014-05-01 14:09:32 -04:00
xxhash.h
xxHash
2014-05-01 14:09:32 -04:00