andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
138b87eae4
rocksdb/table
History
Mike Kolupaev 138b87eae4 Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… 
Summary:
Fixes the following scenario:
 1. Set prefix extractor. Enable bloom filters, with `whole_key_filtering = false`. Use compaction filter that sometimes returns `kRemoveAndSkipUntil`.
 2. Do a compaction.
 3. Compaction creates an iterator with `total_order_seek = false`, calls `SeekToFirst()` on it, then repeatedly calls `Next()`.
 4. At some point compaction filter returns `kRemoveAndSkipUntil`.
 5. Compaction calls `Seek(skip_until)` on the iterator. The key that it seeks to happens to have prefix that doesn't match the bloom filter. Since `total_order_seek = false`, iterator becomes invalid, and compaction thinks that it has reached the end. The rest of the compaction input is silently discarded.

The fix is to make compaction iterator use `total_order_seek = true`.

The implementation for PlainTable is quite awkward. I've made `kRemoveAndSkipUntil` officially incompatible with PlainTable. If you try to use them together, compaction will fail, and DB will enter read-only mode (`bg_error_`). That's not a very graceful way to communicate a misconfiguration, but the alternatives don't seem worth the implementation time and complexity. To be able to check in advance that `kRemoveAndSkipUntil` is not going to be used with PlainTable, we'd need to extend the interface of either `CompactionFilter` or `InternalIterator`. It seems unlikely that anyone will ever want to use `kRemoveAndSkipUntil` with PlainTable: PlainTable probably has very few users, and `kRemoveAndSkipUntil` has only one user so far: us (logdevice).
Closes https://github.com/facebook/rocksdb/pull/2349

Differential Revision: D5110388

Pulled By: lightmark

fbshipit-source-id: ec29101a99d9dcd97db33923b87f72bce56cc17a
2017-06-02 15:11:38 -07:00
..
adaptive_table_factory.cc solve the problem of table_factory_to_write_=nullptr (#1342) 2016-09-20 10:11:51 -07:00
adaptive_table_factory.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
block_based_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_filter_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_factory.cc Revert "Delete filter before closing the table" 2017-05-02 13:46:39 -07:00
block_based_table_factory.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_reader.cc Remoe unused BlockBasedTable::compaction_optimized_ 2017-05-18 06:41:23 -07:00
block_based_table_reader.h Remoe unused BlockBasedTable::compaction_optimized_ 2017-05-18 06:41:23 -07:00
block_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_prefix_index.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
block_prefix_index.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_test.cc unbiase readamp bitmap 2017-05-10 01:49:52 -07:00
block.cc support merge and delete in file ingestion 2017-05-26 12:11:21 -07:00
block.h unbiase readamp bitmap 2017-05-10 01:49:52 -07:00
bloom_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
bloom_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cleanable_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_builder_test.cc Fixed some spelling mistakes 2017-05-17 23:12:36 -07:00
cuckoo_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_factory.cc Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
cuckoo_table_factory.h Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… 2017-06-02 15:11:38 -07:00
cuckoo_table_reader_test.cc Fixed some spelling mistakes 2017-05-17 23:12:36 -07:00
cuckoo_table_reader.cc Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… 2017-06-02 15:11:38 -07:00
cuckoo_table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
flush_block_policy.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
format.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
format.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
get_context.cc Avoid pinning when row cache is accessed 2017-05-01 18:12:46 -07:00
get_context.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
index_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
index_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
internal_iterator.h fix assertion failure in Prev() 2016-10-13 17:36:48 -07:00
iter_heap.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
iterator_wrapper.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
iterator.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merger_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merging_iterator.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merging_iterator.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
meta_blocks.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
meta_blocks.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
mock_table.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
mock_table.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
partitioned_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
partitioned_filter_block.cc Object lifetime in cache 2017-05-05 23:20:01 -07:00
partitioned_filter_block.h Object lifetime in cache 2017-05-05 23:20:01 -07:00
persistent_cache_helper.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
persistent_cache_helper.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
persistent_cache_options.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_factory.cc store prefix_extractor_name in table 2016-08-26 11:46:32 -07:00
plain_table_factory.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
plain_table_index.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_index.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_key_coding.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_key_coding.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_reader.cc Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… 2017-06-02 15:11:38 -07:00
plain_table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
scoped_arena_iterator.h Compaction Support for Range Deletion 2016-10-18 12:04:56 -07:00
sst_file_writer_collectors.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
sst_file_writer.cc support merge and delete in file ingestion 2017-05-26 12:11:21 -07:00
table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_properties_internal.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_properties.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_reader_bench.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
table_test.cc Pass CF ID to MemTableRepFactory 2017-06-02 12:12:06 -07:00
two_level_iterator.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
two_level_iterator.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00