andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
caf0f53a74
rocksdb/table
History
Maysam Yabandeh caf0f53a74 Index value delta encoding (#3983) 
Summary:
Given that index value is a BlockHandle, which is basically an <offset, size> pair we can apply delta encoding on the values. The first value at each index restart interval encoded the full BlockHandle but the rest encode only the size. Refer to IndexBlockIter::DecodeCurrentValue for the detail of the encoding. This reduces the index size which helps using the  block cache more efficiently. The feature is enabled with using format_version 4.

The feature comes with a bit of cpu overhead which should be paid back by the higher cache hits due to smaller index block size.
Results with sysbench read-only using 4k blocks and using 16 index restart interval:
Format 2:
19585   rocksdb read-only range=100
Format 3:
19569   rocksdb read-only range=100
Format 4:
19352   rocksdb read-only range=100
Pull Request resolved: https://github.com/facebook/rocksdb/pull/3983

Differential Revision: D8361343

Pulled By: maysamyabandeh

fbshipit-source-id: f882ee082322acac32b0072e2bdbb0b5f854e651
2018-08-09 16:58:40 -07:00
..
adaptive_table_factory.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
adaptive_table_factory.h Comment out unused variables 2018-03-05 13:13:41 -08:00
block_based_filter_block_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_filter_block.cc PrefixMayMatch: remove unnecessary check for prefix_extractor_ (#4067) 2018-06-27 20:42:43 -07:00
block_based_filter_block.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_table_builder.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block_based_table_builder.h Write properties metablock last in block-based tables (#4158) 2018-07-20 09:11:59 -07:00
block_based_table_factory.cc Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
block_based_table_factory.h Add DataBlockIndexType option in BlockBasedTableOptions (#4150) 2018-07-27 15:42:27 -07:00
block_based_table_reader.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block_based_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block_builder.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block_builder.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block_fetcher.cc Fix a bug caused by not copying the block trailer. (#4096) 2018-07-06 13:12:39 -07:00
block_fetcher.h Pin mmap files in ReadOnlyDB (#4053) 2018-06-27 17:13:34 -07:00
block_prefix_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_prefix_index.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_test.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
block.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
bloom_block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_block.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cleanable_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
cuckoo_table_builder.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
cuckoo_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_factory.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
cuckoo_table_factory.h comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
cuckoo_table_reader_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
cuckoo_table_reader.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
cuckoo_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
data_block_hash_index_test.cc DataBlockHashIndex: Standalone Implementation with Unit Test (#4139) 2018-07-24 11:43:37 -07:00
data_block_hash_index.cc DataBlockHashIndex: Standalone Implementation with Unit Test (#4139) 2018-07-24 11:43:37 -07:00
data_block_hash_index.h DataBlockHashIndex: Standalone Implementation with Unit Test (#4139) 2018-07-24 11:43:37 -07:00
filter_block.h use user_key and iterate_upper_bound to determine compatibility of bloom filters (#3899) 2018-06-26 15:57:26 -07:00
flush_block_policy.cc Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
format.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
format.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
full_filter_bits_builder.h Skip duplicate bloom keys when whole_key and prefix are mixed 2018-04-24 10:58:16 -07:00
full_filter_block_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
full_filter_block.cc Charging block cache more accurately (#4073) 2018-06-29 08:57:20 -07:00
full_filter_block.h Charging block cache more accurately (#4073) 2018-06-29 08:57:20 -07:00
get_context.cc Avoid unnecessary big for-loop when reporting ticker stats stored in GetContext (#3490) 2018-07-20 16:58:13 -07:00
get_context.h Avoid unnecessary big for-loop when reporting ticker stats stored in GetContext (#3490) 2018-07-20 16:58:13 -07:00
index_builder.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
index_builder.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
internal_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
iter_heap.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
iterator_wrapper.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
iterator.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
merger_test.cc Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
merging_iterator.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
merging_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
meta_blocks.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
meta_blocks.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
mock_table.cc Make BlockBasedTableIterator compaction-aware (#4048) 2018-06-25 13:19:27 -07:00
mock_table.h Make BlockBasedTableIterator compaction-aware (#4048) 2018-06-25 13:19:27 -07:00
partitioned_filter_block_test.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
partitioned_filter_block.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
partitioned_filter_block.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
persistent_cache_helper.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
persistent_cache_helper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_options.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_factory.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_factory.h Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_index.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_key_coding.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_key_coding.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_reader.cc Make BlockBasedTableIterator compaction-aware (#4048) 2018-06-25 13:19:27 -07:00
plain_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
scoped_arena_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_writer_collectors.h Comment out unused variables 2018-03-05 13:13:41 -08:00
sst_file_writer.cc Support range deletion tombstones in IngestExternalFile SSTs (#3778) 2018-07-13 22:43:09 -07:00
table_builder.h Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
table_properties_internal.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_properties.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_reader_bench.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_test.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
two_level_iterator.cc Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
two_level_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00