andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
9f0bd56889
rocksdb/db
History
Zhongyi Xie 3886dddc3b force flushing stats CF to avoid holding old logs (#5509) 
Summary:
WAL records RocksDB writes to all column families. When user flushes a a column family, the old WAL will not accept new writes but cannot be deleted yet because it may still contain live data for other column families. (See https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log#life-cycle-of-a-wal for detailed explanation)
Because of this, if there is a column family that receive very infrequent writes and no manual flush is called for it, it could prevent a lot of WALs from being deleted. PR https://github.com/facebook/rocksdb/pull/5046 introduced persistent stats column family which is a good example of such column families. Depending on the config, it may have long intervals between writes, and user is unaware of it which makes it difficult to call manual flush for it.
This PR addresses the problem for persistent stats column family by forcing a flush for persistent stats column family when 1) another column family is flushed 2) persistent stats column family's log number is the smallest among all column families, this way persistent stats column family will  keep advancing its log number when necessary, allowing RocksDB to delete old WAL files.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5509

Differential Revision: D16045896

Pulled By: miasantreble

fbshipit-source-id: 286837b633e988417f0096ff38384742d3b40ef4
2019-07-01 11:56:43 -07:00
..
compaction Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
db_impl force flushing stats CF to avoid holding old logs (#5509) 2019-07-01 11:56:43 -07:00
builder.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
builder.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
c_test.c Add C binding for secondary instance (#5505) 2019-06-27 08:58:54 -07:00
c.cc Add C binding for secondary instance (#5505) 2019-06-27 08:58:54 -07:00
column_family_test.cc Make format 2019-05-31 15:24:43 -07:00
column_family.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
column_family.h Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
compact_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compacted_db_impl.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compacted_db_impl.h Make format 2019-05-31 15:24:43 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
corruption_test.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
cuckoo_table_db_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
db_basic_test.cc MultiGet parallel IO (#5464) 2019-06-30 20:56:04 -07:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc Move the index readers out of the block cache (#5298) 2019-05-30 11:53:27 -07:00
db_bloom_filter_test.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_compaction_test.cc Combine the read-ahead logic for user reads and compaction reads (#5431) 2019-06-19 14:10:46 -07:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_filesnapshot.cc Add missing check before calling PurgeObsoleteFiles in EnableFileDeletions (#5448) 2019-06-13 14:43:13 -07:00
db_flush_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_info_dumper.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Revert "Reduce iterator key comparison for upper/lower bound check (#5111)" (#5440) 2019-06-11 16:23:41 -07:00
db_iter.h Make format 2019-05-31 15:24:43 -07:00
db_iterator_test.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc Fix tsan complaint in ConcurrentMergeWrite test (#5308) 2019-05-15 11:21:48 -07:00
db_merge_operator_test.cc WriteUnPrepared: less virtual in iterator callback (#5049) 2019-04-02 14:47:16 -07:00
db_options_test.cc fix rocksdb lite and clang contrun test failures (#5477) 2019-06-17 21:16:29 -07:00
db_properties_test.cc Deprecate ttl option from CompactionOptionsFIFO (#4965) 2019-02-15 09:51:41 -08:00
db_range_del_test.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
db_sst_test.cc Increase Trash/DB size ratio in DBSSTTest.RateLimitedWALDelete (#5366) 2019-05-30 11:12:59 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Fix flaky DBTest2.PresetCompressionDict test (#5378) 2019-05-30 16:11:27 -07:00
db_test_util.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_test_util.h simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
db_test.cc sanitize and limit block_size under 4GB (#5492) 2019-06-20 11:45:08 -07:00
db_universal_compaction_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_wal_test.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
dbformat.h Add support for timestamp in Get/Put (#5079) 2019-06-05 23:10:47 -07:00
deletefile_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
error_handler_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Log file_creation_time table property (#5232) 2019-04-22 15:30:07 -07:00
event_helpers.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Fix ingested file and direcotry not being sync (#5435) 2019-06-21 10:15:38 -07:00
external_sst_file_ingestion_job.cc Fix ingested file and direcotry not being sync (#5435) 2019-06-21 10:15:38 -07:00
external_sst_file_ingestion_job.h Fix ingested file and direcotry not being sync (#5435) 2019-06-21 10:15:38 -07:00
external_sst_file_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
flush_job.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
flush_job.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_scheduler.cc Remove global locks from FlushScheduler (#5372) 2019-06-10 16:50:26 -07:00
flush_scheduler.h Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
internal_stats.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
internal_stats.h Collect compaction stats by priority and dump to info LOG (#5050) 2019-03-19 17:28:19 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
log_reader.h secondary instance: add support for WAL tailing on OpenAsSecondary 2019-04-24 12:08:44 -07:00
log_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
log_writer.cc LogWriter to only flush after finish generating whole record (#5328) 2019-05-21 12:33:17 -07:00
log_writer.h Close WAL files before deletion (#5233) 2019-04-25 10:11:41 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
malloc_stats.cc Detect if Jemalloc is linked with the binary (#4844) 2019-01-03 16:30:12 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
memtable_list.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
memtable_list.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
memtable.cc Add support for timestamp in Get/Put (#5079) 2019-06-05 23:10:47 -07:00
memtable.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
range_tombstone_fragmenter.h Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
read_callback.h WritePrepared: fix race condition in reading batch with duplicate keys (#5147) 2019-04-12 14:40:41 -07:00
repair_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
repair.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
table_cache.h Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
table_properties_collector_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Replace Corruption with TryAgain status when new tail is not visible to TransactionLogIterator (#5474) 2019-06-19 08:10:08 -07:00
transaction_log_impl.h Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
version_builder_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
version_builder.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
version_builder.h Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
version_edit_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_edit.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
version_edit.h Make RocksDB secondary instance respect atomic groups in version edits. (#5411) 2019-06-04 10:56:19 -07:00
version_set_test.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
version_set.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
version_set.h Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
wal_manager_test.cc Replace Corruption with TryAgain status when new tail is not visible to TransactionLogIterator (#5474) 2019-06-19 08:10:08 -07:00
wal_manager.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
wal_manager.h improve comment for WalManager (#5350) 2019-05-24 10:40:30 -07:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h WriteUnPrepared: Add new WAL marker kTypeBeginUnprepareXID (#4069) 2018-06-28 18:58:29 -07:00
write_batch_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
write_batch.cc Make format 2019-05-31 15:24:43 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
write_thread.h Fix skip WAL for whole write_group when leader's callback fail (#4838) 2019-01-03 12:40:42 -08:00