andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
20b244fcca
rocksdb/db
History
Venkatesh Radhakrishnan 20b244fcca Fix CompactFiles by adding all necessary files 
Summary:
The compact files API had a bug where some overlapping files
are not added. These are files which overlap with files which were
added to the compaction input files, but not to the original set of
input files. This happens only when there are more than two levels
involved in the compaction. An example will illustrate this better.

Level 2 has 1 input file 1.sst which spans [20,30].

Level 3 has added file  2.sst which spans [10,25]

Level 4 has file 3.sst which spans [35,40] and
        input file 4.sst which spans [46,50].

The existing code would not add 3.sst to the set of input_files because
it only becomes an overlapping file in level 4 and it wasn't one in
level 3.

When installing the results of the compaction, 3.sst would overlap with
output file from the compact files and result in the assertion in
version_set.cc:1130

 // Must not overlap
   assert(level <= 0 || level_files->empty() ||
            internal_comparator_->Compare(
                (*level_files)[level_files->size() - 1]->largest, f->smallest) <
                0);
This change now adds overlapping files from the current level to the set
of input files also so that we don't hit the assertion above.

Test Plan:
d=/tmp/j; rm -rf $d; seq 1000 | parallel --gnu --eta
'd=/tmp/j/d-{}; mkdir -p $d; TEST_TMPDIR=$d ./db_compaction_test
--gtest_filter=*CompactilesOnLevel* --gtest_also_run_disabled_tests >&
'$d'/log-{}'

Reviewers: igor, yhchiang, sdong

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43437
2015-08-03 15:53:22 -07:00
..
builder.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
builder.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
c_test.c Deprecate CompactionFilterV2 2015-07-17 18:59:11 +02:00
c.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
column_family_test.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
column_family.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
column_family.h Fail DB::Open() when the requested compression is not available 2015-06-18 14:55:05 -07:00
compact_files_test.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
compacted_db_impl.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compacted_db_impl.h Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compaction_job_stats_test.cc Parallelize L0-L1 Compaction: Restructure Compaction Job 2015-08-03 11:32:14 -07:00
compaction_job_test.cc Count number of corrupt keys during compaction 2015-07-28 16:41:40 -07:00
compaction_job.cc Parallelize L0-L1 Compaction: Restructure Compaction Job 2015-08-03 11:32:14 -07:00
compaction_job.h Parallelize L0-L1 Compaction: Restructure Compaction Job 2015-08-03 11:32:14 -07:00
compaction_picker_test.cc Make compaction_picker_test runnable in ROCKSDB_LITE 2015-07-20 10:46:09 -07:00
compaction_picker.cc Fix CompactFiles by adding all necessary files 2015-08-03 15:53:22 -07:00
compaction_picker.h Fix when output level is 0 of universal compaction with trivial move 2015-07-27 14:25:57 -07:00
compaction.cc Fix when output level is 0 of universal compaction with trivial move 2015-07-27 14:25:57 -07:00
compaction.h Parallelize L0-L1 Compaction: Restructure Compaction Job 2015-08-03 11:32:14 -07:00
comparator_db_test.cc Make "make all" work for CYGWIN 2015-06-09 16:36:07 -07:00
convenience.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
corruption_test.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
cuckoo_table_db_test.cc Block cuckoo table tests in ROCKSDB_LITE 2015-07-20 10:50:46 -07:00
db_bench.cc db_bench add an option of --universal_allow_trivial_move 2015-07-20 11:56:12 -07:00
db_compaction_filter_test.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
db_compaction_test.cc Fix CompactFiles by adding all necessary files 2015-08-03 15:53:22 -07:00
db_dynamic_level_test.cc Move DynamicLevel related db-tests to db_dynamic_level_test.cc 2015-07-13 19:00:30 -07:00
db_filesnapshot.cc Add wal files to Checkpoint for multiple column families. 2015-06-19 16:08:31 -07:00
db_impl_debug.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
db_impl_experimental.cc Clean up InstallSuperVersion 2015-06-17 12:37:59 -07:00
db_impl_readonly.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
db_impl_readonly.h Use CompactRangeOptions for CompactRange 2015-06-17 14:36:14 -07:00
db_impl.cc Fix when output level is 0 of universal compaction with trivial move 2015-07-27 14:25:57 -07:00
db_impl.h [wal changes 2/3] write with sync=true syncs previous unsynced wals to prevent illegal data loss 2015-07-22 03:28:08 -07:00
db_inplace_update_test.cc Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc 2015-07-20 16:05:28 -07:00
db_iter_test.cc Revert two diffs related to DBIter::FindPrevUserKey() 2015-07-07 11:36:24 -07:00
db_iter.cc Revert two diffs related to DBIter::FindPrevUserKey() 2015-07-07 11:36:24 -07:00
db_iter.h reduce references to cfd->options() in DBImpl 2014-09-08 15:04:34 -07:00
db_log_iter_test.cc Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc 2015-07-14 16:08:21 -07:00
db_tailing_iter_test.cc Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc 2015-07-14 16:41:08 -07:00
db_test.cc fixed DBTest.GetPropertiesOfAllTablesTest and DBTest.GetUserDefinedTablaProperties flakiness 2015-07-22 12:37:49 -07:00
db_universal_compaction_test.cc Fix when output level is 0 of universal compaction with trivial move 2015-07-27 14:25:57 -07:00
dbformat_test.cc Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
dbformat.cc Replace %llu with format macros in ParsedInternalKey::DebugString()) 2015-06-17 20:44:26 -07:00
dbformat.h Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
deletefile_test.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
event_helpers.cc Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
event_helpers.h Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
experimental.cc Implement DB::PromoteL0 method 2015-04-23 12:10:36 -07:00
fault_injection_test.cc [wal changes 2/3] write with sync=true syncs previous unsynced wals to prevent illegal data loss 2015-07-22 03:28:08 -07:00
file_indexer_test.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.h Fix public API dependency on internal codes and dependency on MAX_INT32 2015-07-11 10:32:11 -07:00
filename_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
filename.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
filename.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
flush_job_test.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
flush_job.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
flush_job.h Add largest sequence to FlushJobInfo 2015-06-11 15:22:22 -07:00
flush_scheduler.cc Don't return (or dereference) dangling pointer 2014-10-02 14:33:16 -07:00
flush_scheduler.h Fix data race #1 2015-01-26 11:48:07 -08:00
forward_iterator.cc fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
forward_iterator.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
internal_stats.cc Report live data size estimate 2015-07-21 21:33:20 -07:00
internal_stats.h Report live data size estimate 2015-07-21 21:33:20 -07:00
job_context.h fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
listener_test.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
log_format.h Some minor refactoring on the code 2014-01-02 16:32:31 -08:00
log_reader.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_reader.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_test.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_writer.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_writer.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
managed_iterator.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
managed_iterator.h Fixed xfunc related compile errors in ROCKSDB_LITE 2015-04-09 21:05:18 -07:00
memtable_allocator.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_list_test.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
memtable_list.cc Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable_list.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable.cc Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtablerep_bench.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
merge_context.h API to fetch from both a WriteBatchWithIndex and the db 2015-05-11 14:51:51 -07:00
merge_helper_test.cc Further cleanup of CompactionJob and MergeHelper 2015-07-28 19:21:55 -07:00
merge_helper.cc Further cleanup of CompactionJob and MergeHelper 2015-07-28 19:21:55 -07:00
merge_helper.h Further cleanup of CompactionJob and MergeHelper 2015-07-28 19:21:55 -07:00
merge_operator.cc Call merge operators with empty values 2015-06-26 11:35:46 -07:00
merge_test.cc Make merge_test runnable in ROCKSDB_LITE 2015-07-20 11:17:52 -07:00
perf_context_test.cc Makefile minor cleanup 2015-03-30 16:05:35 -04:00
plain_table_db_test.cc Block plain_table_db_test in ROCKSDB_LITE 2015-07-20 11:12:02 -07:00
prefix_test.cc Skip unsupported tests in ROCKSDB_LITE 2015-07-20 11:24:54 -07:00
repair.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
skiplist_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
skiplist.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
slice.cc Create an abstract interface for write batches 2015-03-17 19:23:08 -07:00
snapshot.h GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex 2015-06-08 21:57:02 -07:00
table_cache.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
table_cache.h Implement a table-level row cache 2015-06-23 10:25:45 -07:00
table_properties_collector_test.cc Fixing dead code in table_properties_collector_test 2015-07-30 16:59:03 -07:00
table_properties_collector.cc A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge 2015-04-06 10:27:21 -07:00
table_properties_collector.h Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
transaction_log_impl.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
transaction_log_impl.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
version_builder_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_builder.cc Fix deleting obsolete files 2015-02-06 08:44:30 -08:00
version_builder.h Move VersionBuilder logic to a separate .cc file 2014-10-31 16:34:38 -07:00
version_edit_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_edit.cc Added JSON manifest dump option to ldb command 2015-07-17 10:07:40 -07:00
version_edit.h Added JSON manifest dump option to ldb command 2015-07-17 10:07:40 -07:00
version_set_test.cc Report live data size estimate 2015-07-21 21:33:20 -07:00
version_set.cc Parallelize L0-L1 Compaction: Restructure Compaction Job 2015-08-03 11:32:14 -07:00
version_set.h Report live data size estimate 2015-07-21 21:33:20 -07:00
wal_manager_test.cc Skip unsupported tests in ROCKSDB_LITE 2015-07-20 11:24:54 -07:00
wal_manager.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
wal_manager.h Fix -Wnon-virtual-dtor errors 2014-11-10 17:39:38 -05:00
write_batch_base.cc WriteBatch.Merge w/ SliceParts support 2015-05-29 04:30:03 -07:00
write_batch_internal.h WriteBatch Save Points 2015-07-29 16:54:23 -07:00
write_batch_test.cc WriteBatch Save Points 2015-07-29 16:54:23 -07:00
write_batch.cc WriteBatch Save Points 2015-07-29 16:54:23 -07:00
write_callback_test.cc Fix compile for write_callback_test in ROCKSDB_LITE 2015-07-20 10:54:15 -07:00
write_callback.h Optimistic Transactions 2015-05-29 14:36:35 -07:00
write_controller_test.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller.h Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_thread.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
write_thread.h Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
writebuffer.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00