andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Andrew Kryczka 3ae0047278 skip CompactRange flush based on memtable contents 
Summary:
CompactRange has a call to Flush because we guarantee that, at the time it's called, all existing keys in the range will be pushed through the user's compaction filter. However, previously the flush was done blindly, so it'd happen even if the memtable does not contain keys in the range specified by the user. This caused unnecessarily many L0 files to be created, leading to write stalls in some cases. This PR checks the memtable's contents, and decides to flush only if it overlaps with `CompactRange`'s range.

- Move the memtable overlap check logic from `ExternalSstFileIngestionJob` to `ColumnFamilyData::RangesOverlapWithMemtables`
- Reuse the above logic in `CompactRange` and skip flushing if no overlap
Closes https://github.com/facebook/rocksdb/pull/3520

Differential Revision: D7018897

Pulled By: ajkr

fbshipit-source-id: a3c6b1cfae56687b49dd89ccac7c948e53545934
2018-02-27 17:12:44 -08:00
..
builder.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
builder.h
Stream
2017-11-10 09:26:24 -08:00
c_test.c
Added save points for transactions C API
2017-09-14 14:18:59 -07:00
c.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
column_family_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
column_family.cc
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
column_family.h
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
compact_files_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compacted_db_impl.cc
Fix coverity issues column_family, compaction_db/iterator
2017-10-27 11:26:42 -07:00
compacted_db_impl.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_iteration_stats.h
add counter for deletion dropping optimization
2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_iterator.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_iterator.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_job_stats_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_job_test.cc
WritePrepared Txn: PreReleaseCallback
2017-11-30 23:50:45 -08:00
compaction_job.cc
WritePrepared Txn: Support merge operator
2018-02-09 14:57:54 -08:00
compaction_job.h
Make DBOption compaction_readahead_size dynamic
2017-11-16 17:57:25 -08:00
compaction_picker_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_picker_universal.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_picker_universal.h
Make Universal compaction options dynamic
2017-12-11 13:27:06 -08:00
compaction_picker.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction_picker.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
compaction.cc
Improve fallocate size in compaction output
2018-01-22 16:43:46 -08:00
compaction.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
comparator_db_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
convenience.cc
Delete files in multiple ranges at once
2018-01-30 13:56:39 -08:00
corruption_test.cc
fix corruption_test valgrind
2017-08-11 12:29:14 -07:00
cuckoo_table_db_test.cc
Enable MSVC W4 with a few exceptions. Fix warnings and bugs
2017-10-19 10:57:12 -07:00
db_basic_test.cc
Fix the Logger::Close() and DBImpl::Close() design pattern
2018-02-23 13:57:26 -08:00
db_blob_index_test.cc
fix lite build
2017-10-17 08:57:09 -07:00
db_block_cache_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_bloom_filter_test.cc
Revert "comment out unused parameters"
2017-07-21 18:26:26 -07:00
db_compaction_filter_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_compaction_test.cc
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
db_dynamic_level_test.cc
Revert "comment out unused parameters"
2017-07-21 18:26:26 -07:00
db_encryption_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_filesnapshot.cc
log flush reason for better debugging experience
2018-02-09 12:12:43 -08:00
db_flush_test.cc
Handle error return from WriteBuffer()
2018-02-05 13:59:34 -08:00
db_impl_compaction_flush.cc
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
db_impl_debug.cc
log flush reason for better debugging experience
2018-02-09 12:12:43 -08:00
db_impl_experimental.cc
Inform caller when rocksdb is stalling writes
2017-10-05 18:11:43 -07:00
db_impl_files.cc
fix live WALs purged while file deletions disabled
2018-01-17 17:42:04 -08:00
db_impl_open.cc
DB:Open should fail on tmpfs when use_direct_reads=true
2018-02-26 14:58:06 -08:00
db_impl_readonly.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_impl_readonly.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_impl_write.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_impl.cc
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
db_impl.h
Fix the Logger::Close() and DBImpl::Close() design pattern
2018-02-23 13:57:26 -08:00
db_info_dumper.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_info_dumper.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_io_failure_test.cc
Fix LITE unit tests
2017-07-26 21:11:47 -07:00
db_iter_test.cc
fix Seek with lower_bound
2017-11-29 22:56:29 -08:00
db_iter.cc
Add rocksdb.iterator.internal-key property
2018-02-20 19:12:09 -08:00
db_iter.h
WritePrepared Txn: Return NotSupported on iterator refresh
2017-12-18 22:29:30 -08:00
db_iterator_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_log_iter_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_memtable_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_merge_operator_test.cc
WritePrepared Txn: Support merge operator
2018-02-09 14:57:54 -08:00
db_options_test.cc
fix live WALs purged while file deletions disabled
2018-01-17 17:42:04 -08:00
db_properties_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_range_del_test.cc
Fix wrong smallest key of delete range tombstones
2017-08-29 18:41:35 -07:00
db_sst_test.cc
Fix DeleteScheduler::MarkAsTrash() handling existing trash
2017-12-12 18:17:13 -08:00
db_statistics_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_table_properties_test.cc
fix deletion-triggered compaction in table builder
2017-09-28 18:17:30 -07:00
db_tailing_iter_test.cc
Revert "comment out unused parameters"
2017-07-21 18:26:26 -07:00
db_test2.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_test_util.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_test_util.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_universal_compaction_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
db_wal_test.cc
WritePrepared Txn: PreReleaseCallback
2017-11-30 23:50:45 -08:00
db_write_test.cc
BlobDB: Remove the need to get sequence number per write
2017-12-15 13:27:30 -08:00
dbformat_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
dbformat.cc
WritePrepared Txn: Duplicate Keys, Memtable part
2018-01-31 18:57:07 -08:00
dbformat.h
WritePrepared Txn: Duplicate Keys, Memtable part
2018-01-31 18:57:07 -08:00
deletefile_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
event_helpers.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
event_helpers.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
experimental.cc
Replace dynamic_cast<>
2017-07-28 16:27:16 -07:00
external_sst_file_basic_test.cc
optimize file ingestion checks for range deletion overlap
2017-11-28 11:27:02 -08:00
external_sst_file_ingestion_job.cc
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
external_sst_file_ingestion_job.h
skip CompactRange flush based on memtable contents
2018-02-27 17:12:44 -08:00
external_sst_file_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
fault_injection_test.cc
fix UBSAN errors in fault_injection_test
2018-02-13 14:12:40 -08:00
file_indexer_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
file_indexer.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
file_indexer.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
filename_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
flush_job_test.cc
Add a histogram stat for memtable flush
2017-12-15 18:57:00 -08:00
flush_job.cc
log flush reason for better debugging experience
2018-02-09 12:12:43 -08:00
flush_job.h
WritePrepared Txn: Compaction/Flush
2017-10-06 10:41:53 -07:00
flush_scheduler.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
flush_scheduler.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc
fix gflags namespace
2017-12-01 10:42:05 -08:00
forward_iterator.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
forward_iterator.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
internal_stats.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
internal_stats.h
Add DB::Properties::kEstimateOldestKeyTime
2017-10-23 15:27:27 -07:00
job_context.h
Prevent unnecessary calls to PurgeObsoleteFiles
2018-01-12 13:27:08 -08:00
listener_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
log_format.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
log_reader.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
log_reader.h
Suppress lint in old files
2018-01-29 12:56:42 -08:00
log_test.cc
Enable MSVC W4 with a few exceptions. Fix warnings and bugs
2017-10-19 10:57:12 -07:00
log_writer.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
log_writer.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
malloc_stats.cc
Have a different function when ROCKSDB_JEMALLOC=0
2018-02-23 11:42:33 -08:00
malloc_stats.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
managed_iterator.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
managed_iterator.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
manual_compaction_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
memtable_list_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
memtable_list.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
memtable_list.h
Fix Flush() keep waiting after flush finish
2018-01-18 17:45:16 -08:00
memtable.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
memtable.h
WritePrepared Txn: Duplicate Keys, Memtable part
2018-01-31 18:57:07 -08:00
merge_context.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_helper_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_helper.cc
WritePrepared Txn: Support merge operator
2018-02-09 14:57:54 -08:00
merge_helper.h
WritePrepared Txn: Support merge operator
2018-02-09 14:57:54 -08:00
merge_operator.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
options_file_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
perf_context_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
pinned_iterators_manager.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
plain_table_db_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
pre_release_callback.h
WritePrepared Txn: PreReleaseCallback
2017-11-30 23:50:45 -08:00
prefix_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
range_del_aggregator_test.cc
optimize file ingestion checks for range deletion overlap
2017-11-28 11:27:02 -08:00
range_del_aggregator.cc
optimize file ingestion checks for range deletion overlap
2017-11-28 11:27:02 -08:00
range_del_aggregator.h
optimize file ingestion checks for range deletion overlap
2017-11-28 11:27:02 -08:00
read_callback.h
write-prepared txn: call IsInSnapshot
2017-09-11 09:14:48 -07:00
repair_test.cc
fix file numbers after repair
2017-10-10 13:12:37 -07:00
repair.cc
WritePrepared Txn: PreReleaseCallback
2017-11-30 23:50:45 -08:00
snapshot_checker.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
snapshot_impl.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
snapshot_impl.h
Disable options_settable_test in UBSAN and fix UBSAN failure in blob_…
2018-02-07 14:42:26 -08:00
table_cache.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
table_cache.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
table_properties_collector_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
table_properties_collector.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
table_properties_collector.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
transaction_log_impl.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
transaction_log_impl.h
WritePrepared Txn: Refactor conf params
2017-11-10 17:28:12 -08:00
version_builder_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
version_builder.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
version_builder.h
Allow DB reopen with reduced options.num_levels
2017-08-24 16:10:54 -07:00
version_edit_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
version_edit.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
version_edit.h
Fix coverity issues version, write_batch
2017-12-07 11:57:36 -08:00
version_set_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
version_set.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
version_set.h
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
wal_manager_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
wal_manager.cc
Fix memleak when DB::DeleteFile()
2018-01-11 18:57:33 -08:00
wal_manager.h
Fix memleak when DB::DeleteFile()
2018-01-11 18:57:33 -08:00
write_batch_base.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_batch_internal.h
Explictly fail writes if key or value is not smaller than 4GB
2018-02-09 14:57:54 -08:00
write_batch_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
write_batch.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
write_callback_test.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
write_callback.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_thread.cc
Back out "[codemod] - comment out unused parameters"
2018-02-22 12:43:17 -08:00
write_thread.h
WritePrepared Txn: Duplicate Keys, Txn Part
2018-02-05 18:43:24 -08:00