andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
a8e503e545
rocksdb/db
History
Andrew Kryczka a8e503e545 Fix universal compaction scheduling conflict with CompactFiles (#4055) 
Summary:
Universal size-amp-triggered compaction was pulling the final sorted run into the compaction without checking whether any of its files are already being compacted. When all compactions are automatic, it is safe since it verifies the second-last sorted run is not already being compacted, which implies the last sorted run is also not being compacted (in automatic compaction multiple sorted runs are always compacted together). But with manual compaction, files in the last sorted run can be compacted independently, so the last sorted run also must be checked.

We were seeing the below assertion failure in `db_stress`. Also the test case included in this PR repros the failure.

```
db_universal_compaction_test: db/compaction.cc:312: void rocksdb::Compaction::MarkFilesBeingCompacted(bool): Assertion `mark_as_compacted ? !inputs_[i][j]->being_compacted : inputs_[i][j]->being_compacted' failed.
Aborted (core dumped)
```
Closes https://github.com/facebook/rocksdb/pull/4055

Differential Revision: D8630094

Pulled By: ajkr

fbshipit-source-id: ac3b30a874678b76e113d4f6c42c1260411b08f8
2018-06-26 10:44:56 -07:00
..
builder.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
c_test.c comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
c.cc Pin top-level index on partitioned index/filter blocks (#4037) 2018-06-22 15:27:46 -07:00
column_family_test.cc Extend some tests to format_version=3 (#3942) 2018-06-04 20:13:00 -07:00
column_family.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
column_family.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compact_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compacted_db_impl.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
compacted_db_impl.h Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
compaction_iterator.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
compaction_iterator.h option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
compaction_job_stats_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compaction_job_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_job.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
compaction_job.h MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction_picker_test.cc Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction_picker_universal.cc Fix universal compaction scheduling conflict with CompactFiles (#4055) 2018-06-26 10:44:56 -07:00
compaction_picker_universal.h Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction_picker.cc Check conflict at output level in CompactFiles (#3926) 2018-06-05 14:14:05 -07:00
compaction_picker.h Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction.cc MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction.h MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
comparator_db_test.cc run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
convenience.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
corruption_test.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
cuckoo_table_db_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_basic_test.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc LRUCache midpoint insertion 2018-05-24 15:57:33 -07:00
db_bloom_filter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_compaction_filter_test.cc Remove tests from ROCKSDB_VALGRIND_RUN 2018-05-30 16:15:16 -07:00
db_compaction_test.cc Check conflict at output level in CompactFiles (#3926) 2018-06-05 14:14:05 -07:00
db_dynamic_level_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_encryption_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_filesnapshot.cc fix behavior does not match name for "IsFileDeletionsEnabled" 2018-03-21 22:13:34 -07:00
db_flush_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_compaction_flush.cc Bottommost level-based compactions in bottom-pri pool 2018-05-14 14:57:15 -07:00
db_impl_debug.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
db_impl_experimental.cc Inform caller when rocksdb is stalling writes 2017-10-05 18:11:43 -07:00
db_impl_files.cc DBImpl::FindObsoleteFiles() not to call GetChildren() on the same path 2018-05-31 12:58:33 -07:00
db_impl_open.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
db_impl_readonly.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_impl_readonly.h allowing CompactFiles to return new file names 2018-03-15 11:58:12 -07:00
db_impl_write.cc fix deadlock with enable_pipelined_write=true and max_successive_merges > 0 2018-05-31 11:13:14 -07:00
db_impl.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
db_impl.h Add kOptionsStatistics to GetProperty() (#3966) 2018-06-15 17:28:01 -07:00
db_info_dumper.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Fix LITE unit tests 2017-07-26 21:11:47 -07:00
db_iter_stress_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_iter_test.cc Fix regression bug of Prev() with upper bound (#3989) 2018-06-12 16:57:36 -07:00
db_iter.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_iter.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_iterator_test.cc BlockBasedTableIterator to keep BlockIter after out of upper bound (#4004) 2018-06-19 09:57:11 -07:00
db_log_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_memtable_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
db_merge_operator_test.cc Fix TSAN timeout in MergeOperatorPinningTest.Randomized/x test 2018-03-02 16:27:21 -08:00
db_options_test.cc PersistRocksDBOptions() to use WritableFileWriter 2018-05-21 16:42:22 -07:00
db_properties_test.cc Add kOptionsStatistics to GetProperty() (#3966) 2018-06-15 17:28:01 -07:00
db_range_del_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
db_sst_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_statistics_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_table_properties_test.cc fix deletion-triggered compaction in table builder 2017-09-28 18:17:30 -07:00
db_tailing_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_test2.cc Copy Get() result when file reads use mmap 2018-06-01 16:57:58 -07:00
db_test_util.cc Extend format 3 to partitioned index/filters (#3958) 2018-06-06 16:58:16 -07:00
db_test_util.h Extend some tests to format_version=3 (#3942) 2018-06-04 20:13:00 -07:00
db_test.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
db_universal_compaction_test.cc Fix universal compaction scheduling conflict with CompactFiles (#4055) 2018-06-26 10:44:56 -07:00
db_wal_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_write_test.cc Suppress tsan lock-order-inversion on FlushWAL 2018-05-14 21:13:35 -07:00
dbformat_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
dbformat.cc add kEntryRangeDeletion 2018-04-13 11:27:17 -07:00
dbformat.h Imporve perf of random read and insert compare by suggesting inlining to the compiler 2018-03-23 13:26:55 -07:00
deletefile_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
event_helpers.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
event_helpers.h Change RocksDB License 2017-07-15 16:11:23 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
external_sst_file_ingestion_job.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
external_sst_file_ingestion_job.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
external_sst_file_test.cc Fix ExternalSSTFileTest::OverlappingRanges test on Solaris Sparc (#4012) 2018-06-18 14:57:37 -07:00
fault_injection_test.cc Split FaultInjectionTest.FaultTest to avoid timeout 2018-05-07 12:29:58 -07:00
file_indexer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filename_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_job_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
forward_iterator.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
forward_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
internal_stats.cc Add kOptionsStatistics to GetProperty() (#3966) 2018-06-15 17:28:01 -07:00
internal_stats.h Add kOptionsStatistics to GetProperty() (#3966) 2018-06-15 17:28:01 -07:00
job_context.h Introduce and use the option to disable stall notifications structures 2018-05-09 10:13:53 -07:00
listener_test.cc fix some text in comments. 2018-04-10 15:59:24 -07:00
log_format.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
log_test.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
log_writer.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
log_writer.h Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
malloc_stats.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
managed_iterator.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
managed_iterator.h Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
manual_compaction_test.cc Speedup ManualCompactionTest.Test 2018-05-03 16:13:09 -07:00
memtable_list_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
memtable.h InlineSkiplist: don't decode keys unnecessarily during comparisons 2018-03-23 12:14:30 -07:00
merge_context.h Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper.cc option for timing measurement of non-blocking ops during compaction (#4029) 2018-06-21 21:28:05 -07:00
merge_helper.h WritePrepared Txn: Support merge operator 2018-02-09 14:57:54 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
obsolete_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
options_file_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
perf_context_test.cc Improve write time breakdown stats 2018-04-23 17:58:54 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Should only decode restart points for uncompressed blocks (#3996) 2018-06-15 19:26:58 -07:00
pre_release_callback.h Fix pre_release callback argument list. 2018-04-05 11:12:16 -07:00
prefix_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
range_del_aggregator_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
range_del_aggregator.cc Assert keys/values pinned by range deletion meta-block iterators 2018-05-21 09:57:00 -07:00
range_del_aggregator.h Recommit "Avoid adding tombstones of the same file to RangeDelAggregator multiple times" 2018-05-04 16:45:15 -07:00
read_callback.h write-prepared txn: call IsInSnapshot 2017-09-11 09:14:48 -07:00
repair_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
repair.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
snapshot_checker.h Comment out unused variables 2018-03-05 13:13:41 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h WritePrepared Txn: smallest_prepare optimization 2018-04-02 20:27:41 -07:00
table_cache.cc Make BlockBasedTableIterator compaction-aware (#4048) 2018-06-25 13:19:27 -07:00
table_cache.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
table_properties_collector_test.cc Should only decode restart points for uncompressed blocks (#3996) 2018-06-15 19:26:58 -07:00
table_properties_collector.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
table_properties_collector.h Comment out unused variables 2018-03-05 13:13:41 -08:00
transaction_log_impl.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
transaction_log_impl.h WritePrepared Txn: Refactor conf params 2017-11-10 17:28:12 -08:00
version_builder_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
version_builder.cc Delay verify compaction output table (#3979) 2018-06-15 12:42:53 -07:00
version_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
version_edit_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_edit.cc Specify the underlying type of enums. 2018-05-23 16:12:59 -07:00
version_edit.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_set_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
version_set.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
version_set.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
wal_manager_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
wal_manager.cc Add file name info to SequentialFileReader. (#4026) 2018-06-21 08:42:24 -07:00
wal_manager.h Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Fix some typos in comments and docs. 2018-03-08 10:27:25 -08:00
write_batch_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
write_batch.cc WritePrepared Txn: enable TryAgain for duplicates at the end of the batch 2018-04-20 15:28:19 -07:00
write_callback_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Avoid sleep in DBTest.GroupCommitTest to fix flakiness 2018-05-22 12:16:25 -07:00
write_thread.h WritePrepared Txn: Duplicate Keys, Txn Part 2018-02-05 18:43:24 -08:00