andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/util
History
Igor Canadi 9ab5adfc59 New BlockBasedTable version -- better compressed block format 
Summary:
This diff adds BlockBasedTable format_version = 2. New format version brings better compressed block format for these compressions:
1) Zlib -- encode decompressed size in compressed block header
2) BZip2 -- encode decompressed size in compressed block header
3) LZ4 and LZ4HC -- instead of doing memcpy of size_t encode size as varint32. memcpy is very bad because the DB is not portable accross big/little endian machines or even platforms where size_t might be 8 or 4 bytes.

It does not affect format for snappy.

If you write a new database with format_version = 2, it will not be readable by RocksDB versions before 3.10. DB::Open() will return corruption in that case.

Test Plan:
Added a new test in db_test.
I will also run db_bench and verify VSIZE when block_cache == 1GB

Reviewers: yhchiang, rven, MarkCallaghan, dhruba, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31461
2015-01-14 16:24:24 -08:00
..
allocator.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
arena_test.cc
Make arena use hugepage if possible
2014-11-21 14:11:40 -08:00
arena.cc
Make arena use hugepage if possible
2014-11-21 14:11:40 -08:00
arena.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
auto_roll_logger_test.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
auto_roll_logger.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
auto_roll_logger.h
Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL
2014-04-10 15:27:42 -07:00
autovector_test.cc
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
autovector.h
Replace exception by assertion in autovector
2014-12-04 11:41:56 -08:00
benchharness_test.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
benchharness.cc
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
benchharness.h
Fix iOS compile with -Wshorten-64-to-32
2014-11-13 14:39:30 -05:00
blob_store_test.cc
Move the compiler back to 4.8.1 + more small fixes
2013-11-18 11:40:16 -08:00
blob_store.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
blob_store.h
Turn on -Wshadow
2014-10-31 11:59:54 -07:00
bloom_test.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
bloom.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
build_version.h
Dont build version if compiling for IOS
2014-04-11 10:54:47 -07:00
cache_bench.cc
Fix -Wshadow for tools
2014-11-07 15:04:30 -08:00
cache_test.cc
Fix Mac compile errors on util/cache_test.cc
2014-12-11 14:15:13 -08:00
cache.cc
Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references
2014-12-10 22:28:53 -08:00
coding_test.cc
Remove BitStream* tests
2014-08-19 09:52:54 -04:00
coding.cc
Removing BitStream* functions
2014-08-19 06:48:21 -07:00
coding.h
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
comparator.cc
Changed name of ReverseBytewiseComparator based on review comment
2014-09-27 10:06:13 +02:00
compression.h
New BlockBasedTable version -- better compressed block format
2015-01-14 16:24:24 -08:00
crc32c_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
crc32c.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
crc32c.h
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
db_info_dumper.cc
Fix iOS compile with -Wshorten-64-to-32
2014-11-13 14:39:30 -05:00
db_info_dumper.h
Fix iOS compile with -Wshorten-64-to-32
2014-11-13 14:39:30 -05:00
dynamic_bloom_test.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
dynamic_bloom.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
dynamic_bloom.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
env_hdfs.cc
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
env_posix.cc
Add structures for exposing thread events and operations.
2014-12-30 10:39:13 -08:00
env_test.cc
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
env.cc
Check InfoLogLevel earlier in Log functions.
2014-10-30 13:36:18 -07:00
file_util.cc
Provide openable snapshots
2014-11-14 11:38:26 -08:00
file_util.h
Provide openable snapshots
2014-11-14 11:38:26 -08:00
filelock_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
filter_policy.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
hash_cuckoo_rep.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash_cuckoo_rep.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash_linklist_rep.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash_linklist_rep.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash_skiplist_rep.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash_skiplist_rep.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
hash.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
hash.h
Introduce GetThreadList API
2014-11-20 10:49:32 -08:00
histogram_test.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
histogram.cc
Fix iOS compile with -Wshorten-64-to-32
2014-11-13 14:39:30 -05:00
histogram.h
Fix iOS compile with -Wshorten-64-to-32
2014-11-13 14:39:30 -05:00
iostats_context_imp.h
Fix ios compile
2014-08-28 12:46:05 -04:00
iostats_context.cc
Fix ios compile
2014-08-28 12:46:05 -04:00
ldb_cmd_execute_result.h
Revert "Fix lint errors and coding style of ldb related codes."
2014-10-31 19:22:49 -07:00
ldb_cmd.cc
Added 'dump_live_files' command to ldb tool.
2014-12-12 17:50:36 -08:00
ldb_cmd.h
Added 'dump_live_files' command to ldb tool.
2014-12-12 17:50:36 -08:00
ldb_tool.cc
Added 'dump_live_files' command to ldb tool.
2014-12-12 17:50:36 -08:00
log_buffer.cc
Enlarge log size cap when printing file summary
2014-09-23 16:56:34 -07:00
log_buffer.h
Enlarge log size cap when printing file summary
2014-09-23 16:56:34 -07:00
log_write_bench.cc
Fix more gflag namespace issues
2014-05-09 08:41:02 -07:00
logging.cc
remove cast, replace %llu with % PRIu64
2014-09-23 01:10:46 +08:00
logging.h
remove unused variable
2014-09-21 22:20:00 +08:00
manual_compaction_test.cc
Fix CompactRange to apply filter to every key
2014-01-14 16:19:09 -08:00
mock_env_test.cc
Add fsync / corrupt simulation to env_mem
2014-10-31 15:16:31 -07:00
mock_env.cc
Provide openable snapshots
2014-11-14 11:38:26 -08:00
mock_env.h
Provide openable snapshots
2014-11-14 11:38:26 -08:00
murmurhash.cc
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
murmurhash.h
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
mutable_cf_options.cc
remove all remaining references to cfd->options()
2014-11-18 10:20:10 -08:00
mutable_cf_options.h
remove all remaining references to cfd->options()
2014-11-18 10:20:10 -08:00
mutexlock.h
Add separate Read/WriteUnlock methods in MutexRW.
2014-06-16 15:41:46 -07:00
options_builder.cc
Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE
2014-12-04 13:56:14 -08:00
options_helper.cc
add support for nested BlockBasedTableOptions in config string
2014-12-22 16:34:21 -08:00
options_helper.h
Missing header in build on CentOS
2014-11-18 22:21:02 +01:00
options_test.cc
add support for nested BlockBasedTableOptions in config string
2014-12-22 16:34:21 -08:00
options.cc
Deprecating skip_log_error_on_recovery
2015-01-05 13:35:56 -08:00
perf_context_imp.h
Refactor PerfStepTimer to stop on destruct
2014-09-02 12:04:22 -07:00
perf_context.cc
Add a function to return current perf level
2014-07-10 11:35:48 -07:00
posix_logger.h
Fix additional -Wshorten-64-to-32 errros
2014-11-11 14:09:14 -08:00
random.h
Add appropriate LICENSE and Copyright message.
2013-10-16 17:48:41 -07:00
rate_limiter_test.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
rate_limiter.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
rate_limiter.h
expose RateLimiter definition
2014-07-25 15:17:06 -07:00
scoped_arena_iterator.h
Remove path with arena==nullptr from NewInternalIterator
2014-09-04 17:40:41 -07:00
signal_test.cc
util/signal_test.cc: suppress intentional null pointer deref
2014-09-30 23:30:32 +02:00
skiplistrep.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
slice.cc
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
sst_dump_test.cc
Dump routine to BlockBasedTableReader (valgrind)
2014-12-23 18:01:29 -08:00
sst_dump_tool_imp.h
Dump routine to BlockBasedTableReader
2014-12-23 13:24:07 -08:00
sst_dump_tool.cc
Dump routine to BlockBasedTableReader
2014-12-23 13:24:07 -08:00
statistics.cc
fix more compile warnings
2014-09-05 14:14:37 +08:00
statistics.h
make statistics forward-able
2014-07-28 12:10:49 -07:00
status.cc
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
stl_wrappers.h
Killing Transform Rep
2013-12-03 12:42:15 -08:00
stop_watch.h
Change StopWatch interface
2014-07-28 12:22:37 -07:00
string_util.cc
Clean up StringSplit
2014-11-21 11:05:28 -05:00
string_util.h
Add rocksdb::ToString() to address cases where std::to_string is not available.
2014-11-24 20:44:49 -08:00
sync_point.cc
Don't compile sync_point if NDEBUG
2014-04-17 10:49:58 -07:00
sync_point.h
Fix ifdef NDEBUG
2014-04-17 14:29:28 -07:00
testharness.cc
introduce TestMemEnv and use it in db_test
2014-10-31 15:08:10 -07:00
testharness.h
add support for nested BlockBasedTableOptions in config string
2014-12-22 16:34:21 -08:00
testutil.cc
Merger test
2014-09-08 22:24:40 -07:00
testutil.h
Add ComparatorDBTest to test non-default comparators
2014-10-29 15:42:36 -07:00
thread_event_info.h
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_list_test.cc
Add structures for exposing thread events and operations.
2014-12-30 10:39:13 -08:00
thread_local_test.cc
Turn on -Wshadow
2014-10-31 11:59:54 -07:00
thread_local.cc
Build with clang 3.5 on Linux.
2015-01-12 09:59:36 -08:00
thread_local.h
Improve the comment of util/thread_local.h
2014-10-21 17:28:31 -07:00
thread_operation.h
Add structures for exposing thread events and operations.
2014-12-30 10:39:13 -08:00
thread_status_updater_debug.cc
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_status_updater.cc
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_status_updater.h
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_status_util_debug.cc
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_status_util.cc
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
thread_status_util.h
Allow GetThreadList() to indicate a thread is doing Compaction.
2015-01-13 00:04:08 -08:00
vectorrep.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
xxhash.cc
xxHash
2014-05-01 14:09:32 -04:00
xxhash.h
xxHash
2014-05-01 14:09:32 -04:00