andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
d59549298f
rocksdb/db
History
Siying Dong d59549298f Skip deleted WALs during recovery 
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.

Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)

This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765

Differential Revision: D7747618

Pulled By: siying

fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-03 15:43:09 -07:00
..
builder.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
builder.h Stream 2017-11-10 09:26:24 -08:00
c_test.c comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
c.cc expose WAL iterator in the C API 2018-04-27 16:56:59 -07:00
column_family_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
column_family.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
column_family.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compact_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compacted_db_impl.cc Fix coverity issues column_family, compaction_db/iterator 2017-10-27 11:26:42 -07:00
compacted_db_impl.h Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_iterator.cc Blob DB: Improve FIFO eviction 2018-03-06 11:57:42 -08:00
compaction_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_job_stats_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compaction_job_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_job.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_job.h Make DBOption compaction_readahead_size dynamic 2017-11-16 17:57:25 -08:00
compaction_picker_test.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
compaction_picker_universal.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_picker_universal.h Make Universal compaction options dynamic 2017-12-11 13:27:06 -08:00
compaction_picker.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_picker.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
comparator_db_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
convenience.cc Delete files in multiple ranges at once 2018-01-30 13:56:39 -08:00
corruption_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
cuckoo_table_db_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_basic_test.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc avoid double delete on dummy record insertion failure 2018-05-01 16:01:28 -07:00
db_bloom_filter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_compaction_filter_test.cc fix wrong length in snprintf 2018-03-16 13:27:55 -07:00
db_compaction_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_dynamic_level_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_encryption_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_filesnapshot.cc fix behavior does not match name for "IsFileDeletionsEnabled" 2018-03-21 22:13:34 -07:00
db_flush_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_compaction_flush.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_debug.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_experimental.cc Inform caller when rocksdb is stalling writes 2017-10-05 18:11:43 -07:00
db_impl_files.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_open.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_readonly.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_impl_readonly.h allowing CompactFiles to return new file names 2018-03-15 11:58:12 -07:00
db_impl_write.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_info_dumper.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Fix LITE unit tests 2017-07-26 21:11:47 -07:00
db_iter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_iter.cc Add rocksdb.iterator.internal-key property 2018-02-20 19:12:09 -08:00
db_iter.h WritePrepared Txn: Return NotSupported on iterator refresh 2017-12-18 22:29:30 -08:00
db_iterator_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
db_log_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_memtable_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
db_merge_operator_test.cc Fix TSAN timeout in MergeOperatorPinningTest.Randomized/x test 2018-03-02 16:27:21 -08:00
db_options_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_properties_test.cc Add block cache related DB properties 2018-04-18 21:42:25 -07:00
db_range_del_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
db_sst_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_statistics_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_table_properties_test.cc fix deletion-triggered compaction in table builder 2017-09-28 18:17:30 -07:00
db_tailing_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_test2.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_test_util.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
db_test_util.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_universal_compaction_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_wal_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_write_test.cc Rename function for handling WAL write error 2018-03-22 15:58:39 -07:00
dbformat_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
dbformat.cc add kEntryRangeDeletion 2018-04-13 11:27:17 -07:00
dbformat.h Imporve perf of random read and insert compare by suggesting inlining to the compiler 2018-03-23 13:26:55 -07:00
deletefile_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
event_helpers.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
event_helpers.h Change RocksDB License 2017-07-15 16:11:23 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
external_sst_file_ingestion_job.cc Improve accuracy of I/O stats collection of external SST ingestion. 2018-04-13 10:58:42 -07:00
external_sst_file_ingestion_job.h Initialize a boolean member variable of a struct. 2018-04-16 14:28:01 -07:00
external_sst_file_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
fault_injection_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
file_indexer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filename_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_job_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
forward_iterator.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
forward_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
internal_stats.cc Add block cache related DB properties 2018-04-18 21:42:25 -07:00
internal_stats.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
job_context.h comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
listener_test.cc fix some text in comments. 2018-04-10 15:59:24 -07:00
log_format.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
log_test.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
log_writer.cc check return status for Sync() and Append() calls to avoid corruption 2018-04-19 14:13:46 -07:00
log_writer.h Clarification regarding record format 2018-03-08 10:27:25 -08:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
malloc_stats.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
managed_iterator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
managed_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
memtable_list_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable.cc WritePrepared Txn: add stats 2018-04-07 21:56:42 -07:00
memtable.h InlineSkiplist: don't decode keys unnecessarily during comparisons 2018-03-23 12:14:30 -07:00
merge_context.h Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper.cc WritePrepared Txn: Support merge operator 2018-02-09 14:57:54 -08:00
merge_helper.h WritePrepared Txn: Support merge operator 2018-02-09 14:57:54 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
obsolete_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
options_file_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
perf_context_test.cc Improve write time breakdown stats 2018-04-23 17:58:54 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
pre_release_callback.h Fix pre_release callback argument list. 2018-04-05 11:12:16 -07:00
prefix_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
range_del_aggregator_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
range_del_aggregator.cc Revert "Avoid adding tombstones of the same file to RangeDelAggregato… 2018-04-02 19:58:04 -07:00
range_del_aggregator.h Revert "Avoid adding tombstones of the same file to RangeDelAggregato… 2018-04-02 19:58:04 -07:00
read_callback.h write-prepared txn: call IsInSnapshot 2017-09-11 09:14:48 -07:00
repair_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
repair.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
snapshot_checker.h Comment out unused variables 2018-03-05 13:13:41 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h WritePrepared Txn: smallest_prepare optimization 2018-04-02 20:27:41 -07:00
table_cache.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
table_cache.h Change RocksDB License 2017-07-15 16:11:23 -07:00
table_properties_collector_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
table_properties_collector.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
table_properties_collector.h Comment out unused variables 2018-03-05 13:13:41 -08:00
transaction_log_impl.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
transaction_log_impl.h WritePrepared Txn: Refactor conf params 2017-11-10 17:28:12 -08:00
version_builder_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
version_builder.cc Fix some typos in comments and docs. 2018-03-08 10:27:25 -08:00
version_builder.h Allow DB reopen with reduced options.num_levels 2017-08-24 16:10:54 -07:00
version_edit_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_edit.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_edit.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_set_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
version_set.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_set.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
wal_manager_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
wal_manager.cc Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
wal_manager.h Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Fix some typos in comments and docs. 2018-03-08 10:27:25 -08:00
write_batch_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
write_batch.cc WritePrepared Txn: enable TryAgain for duplicates at the end of the batch 2018-04-20 15:28:19 -07:00
write_callback_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Improve write time breakdown stats 2018-04-23 17:58:54 -07:00
write_thread.h WritePrepared Txn: Duplicate Keys, Txn Part 2018-02-05 18:43:24 -08:00