andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
351e25401b
rocksdb/db
History
sdong aa6f7d0995 Support periodic compaction in universal compaction (#5970) 
Summary:
Previously, periodic compaction is not supported in universal compaction. Add the support using following approach: if any file is marked as qualified for periodid compaction, trigger a full compaction. If a full compaction is prevented by files being compacted, try to compact the higher levels than files currently being compacted. If in this way we can only compact the last sorted run and none of the file to be compacted qualifies for periodic compaction, skip the compact. This is to prevent the same single level compaction from being executed again and again.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5970

Test Plan: Add several test cases.

Differential Revision: D18147097

fbshipit-source-id: 8ecc308154d9aca96fb192c51fbceba3947550c1
2019-10-31 11:31:37 -07:00
..
compaction Support periodic compaction in universal compaction (#5970) 2019-10-31 11:31:37 -07:00
db_impl Move pipeline write waiting logic into WaitForPendingWrites() (#5716) 2019-10-29 18:16:36 -07:00
arena_wrapped_db_iter.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
arena_wrapped_db_iter.h Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
blob_index.h Support decoding blob indexes in sst_dump (#5926) 2019-10-17 19:36:54 -07:00
builder.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
builder.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
c_test.c Add C binding for secondary instance (#5505) 2019-06-27 08:58:54 -07:00
c.cc Remove snap_refresh_nanos option (#5826) 2019-09-18 20:26:04 -07:00
column_family_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.cc Make FIFO compaction take default 30 days TTL by default (#5987) 2019-10-31 11:13:12 -07:00
column_family.h Bump up memory order of ref counting of ColumnFamilyData (#5723) 2019-08-20 10:34:33 -07:00
compact_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Do readahead in VerifyChecksum() (#5713) 2019-08-16 16:42:56 -07:00
corruption_test.cc Fix VerifyChecksum readahead with mmap mode (#5945) 2019-10-21 11:38:30 -07:00
cuckoo_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
db_basic_test.cc Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00
db_blob_index_test.cc Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
db_block_cache_test.cc Charge block cache for cache internal usage (#5797) 2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc Auto enable Periodic Compactions if a Compaction Filter is used (#5865) 2019-10-29 15:05:51 -07:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_flush_test.cc Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
db_info_dumper.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Improve readability of DBIter's two seek functions (#5794) 2019-09-16 21:05:07 -07:00
db_iter.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
db_iterator_test.cc Fix data block upper bound checking for iterator reseek case (#5883) 2019-10-03 20:53:29 -07:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_options_test.cc Make FIFO compaction take default 30 days TTL by default (#5987) 2019-10-31 11:13:12 -07:00
db_properties_test.cc Refactor/consolidate legacy Bloom implementation details (#5784) 2019-09-16 16:17:09 -07:00
db_range_del_test.cc Add test showing range tombstones can create excessively large compactions (#5956) 2019-10-24 11:08:44 -07:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test_util.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_test_util.h Support loading custom objects in unit tests (#5676) 2019-08-09 15:12:08 -07:00
db_test.cc Fix for lite build (#5971) 2019-10-25 18:22:24 -07:00
db_test2.cc Update column families' log number altogether after flushing during recovery (#5856) 2019-10-24 18:29:30 -07:00
db_universal_compaction_test.cc Support periodic compaction in universal compaction (#5970) 2019-10-31 11:31:37 -07:00
db_wal_test.cc Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
deletefile_test.cc Refactor deletefile_test.cc (#5822) 2019-09-18 16:58:21 -07:00
error_handler_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
event_helpers.h BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
external_sst_file_ingestion_job.h Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_test.cc Fix a bug in file ingestion (#5760) 2019-08-30 18:29:07 -07:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
flush_job.cc Propagate SST and blob file numbers through the EventListener interface (#5962) 2019-10-24 14:44:15 -07:00
flush_job.h Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
flush_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
import_column_family_job.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
import_column_family_job.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
import_column_family_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.h Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Propagate SST and blob file numbers through the EventListener interface (#5962) 2019-10-24 14:44:15 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_reader.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
memtable_list.cc Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
memtable_list.h Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
memtable.cc Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00
memtable.h Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) 2019-09-18 11:52:17 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
repair.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
table_cache.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
table_properties_collector_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder_test.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
version_builder.cc save a few redundant container lookups (#5875) 2019-10-07 12:28:09 -07:00
version_builder.h Lower the risk for users to run options.force_consistency_checks = true (#5744) 2019-08-29 14:07:37 -07:00
version_edit_test.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
version_edit.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
version_edit.h BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
version_set_test.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
version_set.cc Auto enable Periodic Compactions if a Compaction Filter is used (#5865) 2019-10-29 15:05:51 -07:00
version_set.h Support periodic compaction in universal compaction (#5970) 2019-10-31 11:31:37 -07:00
wal_manager_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
wal_manager.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
wal_manager.h Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00