andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
64aabc9183
rocksdb/db
History
Abhishek Madan 64aabc9183 Properly set smallest key of subcompaction output (#4723) 
Summary:
It is possible to see a situation like the following when
subcompactions are enabled:
1. A subcompaction boundary is set to `[b, e)`.
2. The first output file in a subcompaction has `c@20` as its smallest key
3. The range tombstone `[a, d)30` is encountered.
4. The tombstone is written to the range-del meta block and the new
   smallest key is set to `b@0` (since no keys in this subcompaction's
   output can be smaller than `b`).
5. A key `b@10` in a lower level will now reappear, since it is not
   covered by the truncated start key `b@0`.

In general, unless the smallest data key in a file has a seqnum of 0, it
is not safe to truncate a tombstone at the start key to have a seqnum of
0, since it can expose keys with a seqnum greater than 0 but less than
the tombstone's actual seqnum.

To fix this, when the lower bound of a file is from the subcompaction
boundaries, we now set the seqnum of an artificially extended smallest
key to the tombstone's seqnum. This is safe because subcompactions
operate over disjoint sets of keys, and the subcompactions that can
experience this problem are not the first subcompaction (which is
unbounded on the left).

Furthermore, there is now an assertion to detect the described anomalous
case.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4723

Differential Revision: D13236188

Pulled By: abhimadan

fbshipit-source-id: a6da6a113f2de1e2ff307ca72e055300c8fe5692
2018-12-10 12:38:31 -08:00
..
builder.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
builder.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
c_test.c Revert "BaseDeltaIterator: always check valid() before accessing key(… (#4744) 2018-12-03 23:38:27 -08:00
c.cc Revert "BaseDeltaIterator: always check valid() before accessing key(… (#4744) 2018-12-03 23:38:27 -08:00
column_family_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
column_family.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
column_family.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compact_files_test.cc Check for compression lib support before test exec (#4443) 2018-10-02 10:42:01 -07:00
compacted_db_impl.cc move dump stats to a separate thread (#4382) 2018-10-08 22:54:43 -07:00
compacted_db_impl.h Enable checkpoint of read-only db (#4681) 2018-12-07 17:06:02 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc Drop unnecessary deletion markers during compaction (issue - 3842) (#4289) 2018-08-24 15:17:54 -07:00
compaction_iterator.cc Avoid per-key linear scan over snapshots in compaction (#4495) 2018-10-15 16:21:22 -07:00
compaction_iterator.h option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
compaction_job_stats_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
compaction_job_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
compaction_job.cc Properly set smallest key of subcompaction output (#4723) 2018-12-10 12:38:31 -08:00
compaction_job.h Allow DB resume after background errors (#3997) 2018-06-28 12:34:40 -07:00
compaction_picker_fifo.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker_fifo.h Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker_test.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker_universal.cc Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
compaction_picker_universal.h Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction_picker.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker.h Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction.cc Truncate range tombstones by leveraging InternalKeys (#4432) 2018-10-09 15:19:38 -07:00
compaction.h Truncate range tombstones by leveraging InternalKeys (#4432) 2018-10-09 15:19:38 -07:00
comparator_db_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
convenience.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
corruption_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_db_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
db_basic_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc Make NewBloomFilterPolicy() use full filter by default (#4735) 2018-11-30 13:13:27 -08:00
db_bloom_filter_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_compaction_filter_test.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
db_compaction_test.cc Allow file-ingest-triggered flush to skip waiting for write-stall clear (#4751) 2018-12-05 14:59:29 -08:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_filesnapshot.cc Remove redundant member var and set options (#4631) 2018-11-12 12:24:26 -08:00
db_flush_test.cc Fix a flaky test DBFlushTest.SyncFail (#4633) 2018-11-29 13:39:56 -08:00
db_impl_compaction_flush.cc Move a function to critical section (#4752) 2018-12-05 13:12:09 -08:00
db_impl_debug.cc Enable atomic flush (#4023) 2018-10-26 15:08:43 -07:00
db_impl_experimental.cc Update JobContext. (#3949) 2018-08-03 17:42:34 -07:00
db_impl_files.cc Fix #3840: only SyncClosedLogs for multiple CFs (#4460) 2018-11-13 11:32:16 -08:00
db_impl_open.cc Enable checkpoint of read-only db (#4681) 2018-12-07 17:06:02 -08:00
db_impl_readonly.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
db_impl_readonly.h Enable checkpoint of read-only db (#4681) 2018-12-07 17:06:02 -08:00
db_impl_write.cc Remove redundant member var and set options (#4631) 2018-11-12 12:24:26 -08:00
db_impl.cc Allow file-ingest-triggered flush to skip waiting for write-stall clear (#4751) 2018-12-05 14:59:29 -08:00
db_impl.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
db_info_dumper.cc avoid copying when iterating using range-based for (#4459) 2018-10-09 17:15:51 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_iter_test.cc Fix regression bug of Prev() with upper bound (#3989) 2018-06-12 16:57:36 -07:00
db_iter.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
db_iter.h Introduce RangeDelAggregatorV2 (#4649) 2018-11-21 10:56:45 -08:00
db_iterator_test.cc Fix ticker stat for number files closed (#4703) 2018-11-21 18:31:34 -08:00
db_log_iter_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_memtable_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
db_merge_operator_test.cc WriteUnPrepared Txn: Disable seek to snapshot optimization (#3955) 2018-06-27 12:23:07 -07:00
db_options_test.cc move dump stats to a separate thread (#4382) 2018-10-08 22:54:43 -07:00
db_properties_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_range_del_test.cc Fix range tombstones written to more files than necessary (#4592) 2018-10-29 19:23:27 -07:00
db_sst_test.cc Simplify DBWithMaxSpaceAllowedRandomized (#4235) 2018-08-08 07:27:46 -07:00
db_statistics_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_table_properties_test.cc Allow dynamic modification of window size and deletion trigger (#4403) 2018-09-20 15:15:28 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Add the max trace file size limitation option to Tracing (#4610) 2018-11-27 14:27:05 -08:00
db_test_util.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
db_test_util.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_test.cc Increase wait time in DBTest.SanitizeNumThreads (#4659) 2018-11-14 16:19:36 -08:00
db_universal_compaction_test.cc Adapt three unit tests with newer compiler/libraries (#4562) 2018-10-24 08:17:56 -07:00
db_wal_test.cc Fix #3840: only SyncClosedLogs for multiple CFs (#4460) 2018-11-13 11:32:16 -08:00
db_write_test.cc Suppress tsan lock-order-inversion on FlushWAL 2018-05-14 21:13:35 -07:00
dbformat_test.cc Relax VersionStorageInfo::GetOverlappingInputs check (#4050) 2018-07-13 17:42:38 -07:00
dbformat.cc types: add kEntryBlobIndex for TablePropertiesCollector (#4233) 2018-08-06 18:27:44 -07:00
dbformat.h s/CacheAllocator/MemoryAllocator/g (#4590) 2018-10-26 14:30:30 -07:00
deletefile_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
error_handler_test.cc Fix regression test failures introduced by PR #4164 (#4375) 2018-09-17 13:14:07 -07:00
error_handler.cc Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Auto recovery from out of space errors (#4164) 2018-09-15 13:43:04 -07:00
event_helpers.h Auto recovery from out of space errors (#4164) 2018-09-15 13:43:04 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc Update test to cover a new case in file ingestion (#4614) 2018-11-01 16:23:49 -07:00
external_sst_file_ingestion_job.cc Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
external_sst_file_ingestion_job.h Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
external_sst_file_test.cc Try to fix ExternalSSTFileTest.IngestNonExistingFile flakines (#4625) 2018-11-02 17:26:35 -07:00
fault_injection_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
file_indexer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filename_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_job_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
flush_job.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
flush_job.h Add support to flush multiple CFs atomically (#4262) 2018-10-15 20:01:17 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
forward_iterator.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
forward_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
internal_stats.cc Add DB property for SST files kept from deletion (#4618) 2018-11-05 20:24:40 -08:00
internal_stats.h Add DB property for SST files kept from deletion (#4618) 2018-11-05 20:24:40 -08:00
job_context.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
listener_test.cc Add listener to sample file io (#3933) 2018-10-12 18:36:11 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_reader.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_writer.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
log_writer.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
malloc_stats.cc Fix compile error with jemalloc (#4488) 2018-10-12 11:50:50 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
memtable_list_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
memtable_list.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
memtable_list.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
memtable.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
memtable.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
merge_context.h #3865 fix performance regression introduced by MergeOperator.ShouldMerge (#4266) 2018-08-16 10:58:05 -07:00
merge_helper_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper.cc Range deletion performance improvements + cleanup (#4014) 2018-07-12 14:42:39 -07:00
merge_helper.h Support pragma once in all header files and cleanup some warnings (#4339) 2018-09-05 18:13:31 -07:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
obsolete_files_test.cc Modify verification logic of ObsoleteOptionsFileTest (#4218) 2018-08-03 13:57:40 -07:00
options_file_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
perf_context_test.cc Add PerfContextByLevel to provide per level perf context information (#4226) 2018-10-17 11:19:40 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
pre_release_callback.h Fix pre_release callback argument list. 2018-04-05 11:12:16 -07:00
prefix_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
range_del_aggregator_bench.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_del_aggregator_test.cc Truncate range tombstones by leveraging InternalKeys (#4432) 2018-10-09 15:19:38 -07:00
range_del_aggregator_v2_test.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_del_aggregator_v2.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_del_aggregator_v2.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_del_aggregator.cc Fix two contrun job failures (#4587) 2018-10-24 20:16:45 -07:00
range_del_aggregator.h Lazily initialize RangeDelAggregator stripe map entries (#4497) 2018-10-17 11:47:34 -07:00
range_tombstone_fragmenter_test.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_tombstone_fragmenter.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
range_tombstone_fragmenter.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
read_callback.h WriteUnPrepared Txn: Disable seek to snapshot optimization (#3955) 2018-06-27 12:23:07 -07:00
repair_test.cc Acquire lock on DB LOCK file before starting repair. (#4435) 2018-10-12 10:41:54 -07:00
repair.cc db/repair: reset Repair::db_lock_ in ctor (#4683) 2018-11-29 11:26:41 -08:00
snapshot_checker.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h BlobDB: Fix VisibleToActiveSnapshot() (#4236) 2018-08-06 16:57:42 -07:00
table_cache.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
table_cache.h Introduce RangeDelAggregatorV2 (#4649) 2018-11-21 10:56:45 -08:00
table_properties_collector_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
table_properties_collector.cc Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
table_properties_collector.h Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
transaction_log_impl.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
transaction_log_impl.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
version_builder_test.cc Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
version_builder.cc VersionBuilder: optmize SaveTo() to linear time. (#4366) 2018-09-14 19:43:04 -07:00
version_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
version_edit_test.cc Update recovery code for version edits group commit. (#3945) 2018-08-20 14:58:00 -07:00
version_edit.cc Update recovery code for version edits group commit. (#3945) 2018-08-20 14:58:00 -07:00
version_edit.h Update recovery code for version edits group commit. (#3945) 2018-08-20 14:58:00 -07:00
version_set_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
version_set.cc Revert "Move MemoryAllocator option from Cache to BlockBasedTableOpti… (#4697) 2018-11-21 11:29:57 -08:00
version_set.h Introduce RangeDelAggregatorV2 (#4649) 2018-11-21 10:56:45 -08:00
wal_manager_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
wal_manager.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
wal_manager.h Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h WriteUnPrepared: Add new WAL marker kTypeBeginUnprepareXID (#4069) 2018-06-28 18:58:29 -07:00
write_batch_test.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
write_batch.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
write_callback_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Add more sync point to fix flaky test GroupCommitTest 2018-11-07 14:07:53 -08:00
write_thread.h Handle mixed slowdown/no_slowdown writer properly (#4475) 2018-10-09 22:52:40 -07:00