andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fix handling of too-small filter partition size (#6905)

Browse Source 
Summary:
Because ARM and some other platforms have a larger cache line
size, they have a larger minimum filter size, which causes recently
added PartitionedMultiGet test in db_bloom_filter_test to fail on those
platforms. The code would actually end up using larger partitions,
because keys_per_partition_ would be 0 and never == number of keys
added.

The code now attempts to get as close as possible to the small target
size, while fully utilizing that filter size, if the target partition
size is smaller than the minimum filter size.

Also updated the test to break more uniformly across platforms
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6905

Test Plan: updated test, tested on ARM

Reviewed By: anand1976

Differential Revision: D21840639

Pulled By: pdillinger

fbshipit-source-id: 11684b6d35f43d2e98b85ddb2c8dcfd59d670817
This commit is contained in:
Peter Dillinger 2020-06-03 10:40:13 -07:00 committed by Facebook GitHub Bot
parent 2adb7e3768
commit 9360776cb9
2 changed files with 30 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
db/db_bloom_filter_test.cc
View File

@ -1063,12 +1063,21 @@ TEST_P(DBBloomFilterTestVaryPrefixAndFormatVer, PartitionedMultiGet) {
bbto.partition_filters = true;
bbto.index_type = BlockBasedTableOptions::IndexType::kTwoLevelIndexSearch;
bbto.whole_key_filtering = !use_prefix_;
bbto.metadata_block_size = 128;
if (use_prefix_) { // (not related to prefix, just alternating between)
// Make sure code appropriately deals with metadata block size setting
// that is "too small" (smaller than minimum size for filter builder)
bbto.metadata_block_size = 63;
} else {
// Make sure the test will work even on platforms with large minimum
// filter size, due to large cache line size.
// (Largest cache line size + 10+% overhead.)
bbto.metadata_block_size = 290;
}
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
DestroyAndReopen(options);
ReadOptions ropts;
constexpr uint32_t N = 10000;
constexpr uint32_t N = 12000;
// Add N/2 evens
for (uint32_t i = 0; i < N; i += 2) {
ASSERT_OK(Put(UKey(i), UKey(i)));

19
table/block_based/partitioned_filter_block.cc
View File

@ -35,6 +35,25 @@ PartitionedFilterBlockBuilder::PartitionedFilterBlockBuilder(
keys_added_to_partition_(0) {
keys_per_partition_ =
filter_bits_builder_->CalculateNumEntry(partition_size);
if (keys_per_partition_ < 1) {
// partition_size (minus buffer, ~10%) might be smaller than minimum
// filter size, sometimes based on cache line size. Try to find that
// minimum size without CalculateSpace (not necessarily available).
uint32_t larger = std::max(partition_size + 4, uint32_t{16});
for (;;) {
keys_per_partition_ = filter_bits_builder_->CalculateNumEntry(larger);
if (keys_per_partition_ >= 1) {
break;
}
larger += larger / 4;
if (larger > 100000) {
// might be a broken implementation. substitute something reasonable:
// 1 key / byte.
keys_per_partition_ = partition_size;
break;
}
}
}
}
PartitionedFilterBlockBuilder::~PartitionedFilterBlockBuilder() {}