Peter Dillinger
bae6f58696
Basic MultiGet support for partitioned filters (#6757)
Summary:
In MultiGet, access each applicable filter partition only once
per batch, rather than for each applicable key. Also,
* Fix Bloom stats for MultiGet
* Fix/refactor MultiGetContext::Range::KeysLeft, including
* Add efficient BitsSetToOne implementation
* Assert that MultiGetContext::Range does not go beyond shift range
Performance test: Generate db:
$ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true
...
Before (middle performing run of three; note some missing Bloom stats):
$ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found)
rocksdb.block.cache.filter.hit COUNT : 83443275
rocksdb.bloom.filter.useful COUNT : 0
rocksdb.bloom.filter.full.positive COUNT : 0
rocksdb.bloom.filter.full.true.positive COUNT : 7931450
rocksdb.number.multiget.get COUNT : 385984
rocksdb.number.multiget.keys.read COUNT : 12351488
rocksdb.number.multiget.bytes.read COUNT : 793145000
rocksdb.number.multiget.keys.found COUNT : 7931450
After (middle performing run of three):
$ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget'
multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found)
rocksdb.block.cache.filter.hit COUNT : 49856682
rocksdb.bloom.filter.useful COUNT : 45684579
rocksdb.bloom.filter.full.positive COUNT : 10395458
rocksdb.bloom.filter.full.true.positive COUNT : 9908456
rocksdb.number.multiget.get COUNT : 481984
rocksdb.number.multiget.keys.read COUNT : 15423488
rocksdb.number.multiget.bytes.read COUNT : 990845600
rocksdb.number.multiget.keys.found COUNT : 9908456
So that's about 25% higher throughput even for random keys
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757
Test Plan: unit test included
Reviewed By: anand1976
Differential Revision: D21243256
Pulled By: pdillinger
fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f
2020-04-28 14:49:34 -07:00
..
2020-04-23 13:44:29 -07:00
2020-04-15 17:40:44 -07:00
2020-04-27 22:49:56 -07:00
2020-04-15 17:40:44 -07:00
2020-03-06 16:24:27 -08:00
2020-04-15 17:40:44 -07:00
2020-03-27 16:04:43 -07:00
2020-02-18 11:18:25 -08:00
2020-04-07 14:45:39 -07:00
2020-04-21 17:38:17 -07:00
2020-03-20 19:30:48 -07:00
2020-03-20 19:30:48 -07:00
2020-02-20 12:09:57 -08:00
2020-03-02 16:01:00 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-03-23 21:54:21 -07:00
2020-02-20 12:09:57 -08:00
2020-04-27 13:20:27 -07:00
2020-04-27 13:20:27 -07:00
2020-04-28 14:49:34 -07:00
2020-02-20 12:09:57 -08:00
2020-04-20 13:24:25 -07:00
2020-04-20 13:24:25 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-15 17:40:44 -07:00
2020-04-10 09:51:58 -07:00
2020-04-15 17:40:44 -07:00
2020-02-20 12:09:57 -08:00
2020-03-20 15:26:10 -07:00
2020-03-02 16:01:00 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-21 17:38:17 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-10 15:30:38 -07:00
2020-02-20 12:09:57 -08:00
2020-04-01 16:40:18 -07:00
2020-04-25 00:01:03 -07:00
2020-04-27 13:20:27 -07:00
2020-04-20 13:24:25 -07:00
2020-02-20 12:09:57 -08:00
2020-03-05 17:18:16 -08:00
2020-04-27 22:49:56 -07:00
2020-04-10 16:05:49 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-03-06 16:24:27 -08:00
2020-03-06 16:24:27 -08:00
2020-02-20 12:09:57 -08:00
2020-04-03 10:26:49 -07:00
2020-03-28 19:07:25 -07:00
2020-03-27 16:04:43 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-25 15:31:48 -08:00
2020-03-02 16:16:26 -08:00
2020-02-20 12:09:57 -08:00
2020-04-08 21:22:15 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-07 16:00:26 -07:00
2020-03-31 12:11:42 -07:00
2020-03-27 16:04:43 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-15 17:40:44 -07:00
2020-04-15 17:40:44 -07:00
2020-04-16 13:16:06 -07:00
2020-02-20 12:09:57 -08:00
2020-04-16 13:16:06 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-03-12 11:00:56 -07:00
2020-02-20 12:09:57 -08:00
2020-04-20 13:24:25 -07:00
2020-04-20 11:39:31 -07:00
2020-02-20 12:09:57 -08:00
2020-03-27 16:04:43 -07:00
2020-03-27 16:04:43 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-03-04 20:15:25 -08:00
2020-03-27 16:04:43 -07:00
2020-03-27 16:04:43 -07:00
2020-03-27 16:04:43 -07:00
2020-04-20 13:24:25 -07:00
2020-03-02 16:01:00 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-20 13:24:25 -07:00
2020-02-20 12:09:57 -08:00
2020-03-11 12:33:45 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-15 17:40:44 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-21 13:16:31 -07:00
2020-04-21 13:16:31 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-23 13:44:29 -07:00
2020-04-23 13:44:29 -07:00
2020-03-20 19:30:48 -07:00
2020-03-20 19:30:48 -07:00
2020-03-20 19:30:48 -07:00
2020-03-23 10:27:47 -07:00
2020-03-23 10:27:47 -07:00
2020-03-23 10:27:47 -07:00
2020-04-10 16:05:49 -07:00
2020-04-15 17:40:44 -07:00
2020-04-15 17:40:44 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-04-20 13:24:25 -07:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-20 12:09:57 -08:00
2020-02-28 14:14:03 -08:00
2020-02-20 12:09:57 -08:00