andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
ea7d0b943a
rocksdb/table
History
Igor Canadi 9ab5adfc59 New BlockBasedTable version -- better compressed block format 
Summary:
This diff adds BlockBasedTable format_version = 2. New format version brings better compressed block format for these compressions:
1) Zlib -- encode decompressed size in compressed block header
2) BZip2 -- encode decompressed size in compressed block header
3) LZ4 and LZ4HC -- instead of doing memcpy of size_t encode size as varint32. memcpy is very bad because the DB is not portable accross big/little endian machines or even platforms where size_t might be 8 or 4 bytes.

It does not affect format for snappy.

If you write a new database with format_version = 2, it will not be readable by RocksDB versions before 3.10. DB::Open() will return corruption in that case.

Test Plan:
Added a new test in db_test.
I will also run db_bench and verify VSIZE when block_cache == 1GB

Reviewers: yhchiang, rven, MarkCallaghan, dhruba, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31461
2015-01-14 16:24:24 -08:00
..
adaptive_table_factory.cc introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
adaptive_table_factory.h Sanitize block-based table index type and check prefix_extractor 2014-10-17 21:18:36 -07:00
block_based_filter_block_test.cc Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block_based_filter_block.cc Dump routine to BlockBasedTableReader 2014-12-23 13:24:07 -08:00
block_based_filter_block.h Dump routine to BlockBasedTableReader 2014-12-23 13:24:07 -08:00
block_based_table_builder.cc New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
block_based_table_builder.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
block_based_table_factory.cc New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
block_based_table_factory.h Added CompatibleOptions for compatibility with LevelDB Options 2014-11-20 19:24:39 -08:00
block_based_table_reader.cc New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
block_based_table_reader.h New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
block_builder.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block_builder.h delete unused Comparator 2014-09-04 09:10:13 +08:00
block_hash_index_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block_hash_index.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block_hash_index.h Turn on -Wshadow 2014-10-31 11:59:54 -07:00
block_prefix_index.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block_prefix_index.h fix a few compile warnings 2014-09-04 23:06:23 +08:00
block_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
block.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
bloom_block.cc table/bloom_block.*: pass func parameter by reference 2014-09-30 23:30:31 +02:00
bloom_block.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
cuckoo_table_builder_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
cuckoo_table_builder.cc Support footer versions bigger than 1 2015-01-13 14:33:04 -08:00
cuckoo_table_builder.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
cuckoo_table_factory.cc option to choose module when calculating CuckooTable hash 2014-09-25 13:53:27 -07:00
cuckoo_table_factory.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
cuckoo_table_reader_test.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
cuckoo_table_reader.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
cuckoo_table_reader.h use GetContext to replace callback function pointer 2014-09-29 11:09:09 -07:00
filter_block.h Dump routine to BlockBasedTableReader 2014-12-23 13:24:07 -08:00
flush_block_policy.cc move block based table related options BlockBasedTableOptions 2014-08-25 14:22:05 -07:00
format.cc New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
format.h New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
full_filter_block_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
full_filter_block.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
full_filter_block.h Replace naked calls to operator new and delete (Fixes #222) 2014-09-17 13:50:07 -07:00
get_context.cc use GetContext to replace callback function pointer 2014-09-29 11:09:09 -07:00
get_context.h use GetContext to replace callback function pointer 2014-09-29 11:09:09 -07:00
iter_heap.h Turn on -Wmissing-prototypes 2014-04-09 21:17:14 -07:00
iterator_wrapper.h Turn -Wshadow back on 2014-11-06 11:14:28 -08:00
iterator.cc In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
merger_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
merger.cc Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
merger.h In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
meta_blocks.cc Support footer versions bigger than 1 2015-01-13 14:33:04 -08:00
meta_blocks.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
mock_table.cc CompactionJobTest 2014-11-14 11:35:48 -08:00
mock_table.h CompactionJobTest 2014-11-14 11:35:48 -08:00
plain_table_builder.cc Support footer versions bigger than 1 2015-01-13 14:33:04 -08:00
plain_table_builder.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
plain_table_factory.cc in_table_factory.cc: use correct format specifier 2014-09-30 23:30:32 +02:00
plain_table_factory.h Sanitize block-based table index type and check prefix_extractor 2014-10-17 21:18:36 -07:00
plain_table_index.cc Block plain_table_index.cc in ROCKSDB_LITE 2014-11-24 20:47:27 -08:00
plain_table_index.h Block plain_table_index.cc in ROCKSDB_LITE 2014-11-24 20:47:27 -08:00
plain_table_key_coding.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
plain_table_key_coding.h typo improvement 2014-09-06 23:21:26 +08:00
plain_table_reader.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
plain_table_reader.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
table_builder.h First phase API clean up 2014-02-03 00:30:43 -08:00
table_properties_internal.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
table_properties.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
table_reader_bench.cc Make db_stress built for ROCKSDB_LITE 2014-11-14 10:20:51 -08:00
table_reader.h Dump routine to BlockBasedTableReader 2014-12-23 13:24:07 -08:00
table_test.cc New BlockBasedTable version -- better compressed block format 2015-01-14 16:24:24 -08:00
two_level_iterator.cc ForwardIterator: reset incomplete iterators on Seek() 2014-08-29 16:21:29 -07:00
two_level_iterator.h Turn on -Wshadow 2014-10-31 11:59:54 -07:00