andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
b810e62b39
rocksdb/table/block_based
History
Peter Dillinger bae6f58696 Basic MultiGet support for partitioned filters (#6757) 
Summary:
In MultiGet, access each applicable filter partition only once
per batch, rather than for each applicable key. Also,

* Fix Bloom stats for MultiGet
* Fix/refactor MultiGetContext::Range::KeysLeft, including
* Add efficient BitsSetToOne implementation
* Assert that MultiGetContext::Range does not go beyond shift range

Performance test: Generate db:

    $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true
    ...

Before (middle performing run of three; note some missing Bloom stats):

    $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
    multireadrandom :      26.403 micros/op 597517 ops/sec; (548427 of 671968 found)
    rocksdb.block.cache.filter.hit COUNT : 83443275
    rocksdb.bloom.filter.useful COUNT : 0
    rocksdb.bloom.filter.full.positive COUNT : 0
    rocksdb.bloom.filter.full.true.positive COUNT : 7931450
    rocksdb.number.multiget.get COUNT : 385984
    rocksdb.number.multiget.keys.read COUNT : 12351488
    rocksdb.number.multiget.bytes.read COUNT : 793145000
    rocksdb.number.multiget.keys.found COUNT : 7931450

After (middle performing run of three):

    $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
    multireadrandom :      21.024 micros/op 752963 ops/sec; (705188 of 863968 found)
    rocksdb.block.cache.filter.hit COUNT : 49856682
    rocksdb.bloom.filter.useful COUNT : 45684579
    rocksdb.bloom.filter.full.positive COUNT : 10395458
    rocksdb.bloom.filter.full.true.positive COUNT : 9908456
    rocksdb.number.multiget.get COUNT : 481984
    rocksdb.number.multiget.keys.read COUNT : 15423488
    rocksdb.number.multiget.bytes.read COUNT : 990845600
    rocksdb.number.multiget.keys.found COUNT : 9908456

So that's about 25% higher throughput even for random keys
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757

Test Plan: unit test included

Reviewed By: anand1976

Differential Revision: D21243256

Pulled By: pdillinger

fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f
2020-04-28 14:49:34 -07:00
..
binary_search_index_reader.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
binary_search_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
block_based_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_based_filter_block.cc Silence false alarms in db_stress fault injection (#6741) 2020-04-24 13:06:12 -07:00
block_based_filter_block.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_based_table_builder.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
block_based_table_builder.h In block based table builder, make variables for estimating file size atomic (#6636) 2020-04-02 16:16:24 -07:00
block_based_table_factory.cc Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) 2020-04-21 17:38:17 -07:00
block_based_table_factory.h Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) 2020-04-21 17:38:17 -07:00
block_based_table_iterator.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
block_based_table_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
block_based_table_reader_impl.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
block_based_table_reader.cc Basic MultiGet support for partitioned filters (#6757) 2020-04-28 14:49:34 -07:00
block_based_table_reader.h Stats for redundant insertions into block cache (#6681) 2020-04-27 13:20:27 -07:00
block_builder.cc Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_builder.h Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_prefetcher.cc De-template block based table iterator (#6531) 2020-03-16 12:20:50 -07:00
block_prefetcher.h De-template block based table iterator (#6531) 2020-03-16 12:20:50 -07:00
block_prefix_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_prefix_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_test.cc Fix range deletion tombstone ingestion with global seqno (#6429) 2020-02-25 15:31:48 -08:00
block_type.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block.cc Fix some typos in code comments (#6733) 2020-04-23 12:28:49 -07:00
block.h fix comparison count for format_version=3 indexes (#6650) 2020-04-13 11:18:37 -07:00
cachable_entry.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index_test.cc Fix range deletion tombstone ingestion with global seqno (#6429) 2020-02-25 15:31:48 -08:00
data_block_hash_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block.h Fix iterator reading filter block despite read_tier == kBlockCacheTier (#6562) 2020-03-26 15:21:26 -07:00
filter_policy_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_policy.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_block_policy.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_block_policy.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block.cc Basic MultiGet support for partitioned filters (#6757) 2020-04-28 14:49:34 -07:00
full_filter_block.h Fix iterator reading filter block despite read_tier == kBlockCacheTier (#6562) 2020-03-26 15:21:26 -07:00
hash_index_reader.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
hash_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
index_builder.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
index_builder.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
index_reader_common.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
index_reader_common.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
mock_block_based_table.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
parsed_full_filter_block.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
parsed_full_filter_block.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
partitioned_filter_block_test.cc Fix range deletion tombstone ingestion with global seqno (#6429) 2020-02-25 15:31:48 -08:00
partitioned_filter_block.cc Basic MultiGet support for partitioned filters (#6757) 2020-04-28 14:49:34 -07:00
partitioned_filter_block.h Basic MultiGet support for partitioned filters (#6757) 2020-04-28 14:49:34 -07:00
partitioned_index_iterator.cc Fix regression bug in partitioned index reseek caused by #6531 (#6551) 2020-03-17 12:33:10 -07:00
partitioned_index_iterator.h Fix compiler warning treated as error (#6547) 2020-03-17 09:59:28 -07:00
partitioned_index_reader.cc Silence false alarms in db_stress fault injection (#6741) 2020-04-24 13:06:12 -07:00
partitioned_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
reader_common.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
reader_common.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
uncompression_dict_reader.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
uncompression_dict_reader.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00