rocksdb/db
Yanqin Jin b9f5900658 Fix WAL replay by skipping old write batches (#5170)
Summary:
1. Fix a bug in WAL replay in which write batches with old sequence numbers are mistakenly inserted into memtables.
2. Add support for benchmarking secondary instance to db_bench_tool.
With changes made in this PR, we can start benchmarking secondary instance
using two processes. It is also possible to vary the frequency at which the
secondary instance tries to catch up with the primary. The info log of the
secondary can be found in a directory whose path can be specified with
'-secondary_path'.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5170

Differential Revision: D15564608

Pulled By: riversand963

fbshipit-source-id: ce97688ed3d33f69d3a0b9266ebbbbf887aa0ec8
2019-05-30 19:33:33 -07:00
..
builder.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
builder.h Periodic Compactions (#5166) 2019-04-10 19:31:18 -07:00
c_test.c add missing rocksdb_flush_cf in c (#5243) 2019-04-25 11:25:43 -07:00
c.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
column_family_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
column_family.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
column_family.h Update comments in column_family.h (#5347) 2019-05-24 12:07:15 -07:00
compact_files_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compacted_db_impl.cc move dump stats to a separate thread (#4382) 2018-10-08 22:54:43 -07:00
compacted_db_impl.h Enable checkpoint of read-only db (#4681) 2018-12-07 17:06:02 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_iterator.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
compaction_iterator.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
compaction_job_stats_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_job_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_job.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_job.h Fix env_options_for_read spelling in CompactionJob 2019-05-30 14:04:53 -07:00
compaction_picker_fifo.cc Restrict L0->L0 compaction according to max_compaction_bytes option (#5329) 2019-05-22 23:40:57 -07:00
compaction_picker_fifo.h Avoid double-compacting data in bottom level in manual compactions (#5138) 2019-04-16 23:32:20 -07:00
compaction_picker_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_picker_universal.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_picker_universal.h Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction_picker.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction_picker.h Add comments in compaction_picker.h 2019-05-28 12:24:38 -07:00
compaction.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
compaction.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
corruption_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
cuckoo_table_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_basic_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc Move the index readers out of the block cache (#5298) 2019-05-30 11:53:27 -07:00
db_bloom_filter_test.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_filesnapshot.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_flush_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_impl_compaction_flush.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_impl_debug.cc Avoid double-compacting data in bottom level in manual compactions (#5138) 2019-04-16 23:32:20 -07:00
db_impl_experimental.cc Update JobContext. (#3949) 2018-08-03 17:42:34 -07:00
db_impl_files.cc Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
db_impl_open.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_impl_readonly.cc Add two more StatsLevel (#5027) 2019-02-28 10:27:59 -08:00
db_impl_readonly.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
db_impl_secondary.cc Fix WAL replay by skipping old write batches (#5170) 2019-05-30 19:33:33 -07:00
db_impl_secondary.h improve comments in db_impl_secondary 2019-05-24 15:32:03 -07:00
db_impl_write.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_impl.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_impl.h Improve comment in db_impl.h (#5338) 2019-05-24 13:09:55 -07:00
db_info_dumper.cc Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iterator_test.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc Fix tsan complaint in ConcurrentMergeWrite test (#5308) 2019-05-15 11:21:48 -07:00
db_merge_operator_test.cc WriteUnPrepared: less virtual in iterator callback (#5049) 2019-04-02 14:47:16 -07:00
db_options_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_properties_test.cc Deprecate ttl option from CompactionOptionsFIFO (#4965) 2019-02-15 09:51:41 -08:00
db_range_del_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_secondary_test.cc Fix WAL replay by skipping old write batches (#5170) 2019-05-30 19:33:33 -07:00
db_sst_test.cc Increase Trash/DB size ratio in DBSSTTest.RateLimitedWALDelete (#5366) 2019-05-30 11:12:59 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Fix flaky DBTest2.PresetCompressionDict test (#5378) 2019-05-30 16:11:27 -07:00
db_test_util.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_test_util.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_universal_compaction_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_wal_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
dbformat.h Add comments in db/dbformat.h 2019-05-23 16:44:20 -07:00
deletefile_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
error_handler_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
error_handler.cc Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Log file_creation_time table property (#5232) 2019-04-22 15:30:07 -07:00
event_helpers.h Auto recovery from out of space errors (#4164) 2018-09-15 13:43:04 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
external_sst_file_ingestion_job.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
external_sst_file_ingestion_job.h Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
external_sst_file_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
fault_injection_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
flush_job_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
flush_job.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
flush_job.h Collect compaction stats by priority and dump to info LOG (#5050) 2019-03-19 17:28:19 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
forward_iterator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
forward_iterator.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
in_memory_stats_history.cc Improve comments for StatsHistoryIterator and InMemoryStatsHistoryIterator 2019-05-24 11:40:05 -07:00
in_memory_stats_history.h Improve comments for StatsHistoryIterator and InMemoryStatsHistoryIterator 2019-05-24 11:40:05 -07:00
internal_stats.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
internal_stats.h Collect compaction stats by priority and dump to info LOG (#5050) 2019-03-19 17:28:19 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
log_reader.h secondary instance: add support for WAL tailing on OpenAsSecondary 2019-04-24 12:08:44 -07:00
log_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
log_writer.cc LogWriter to only flush after finish generating whole record (#5328) 2019-05-21 12:33:17 -07:00
log_writer.h Close WAL files before deletion (#5233) 2019-04-25 10:11:41 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
malloc_stats.cc Detect if Jemalloc is linked with the binary (#4844) 2019-01-03 16:30:12 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list.h Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
memtable.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Add two more StatsLevel (#5027) 2019-02-28 10:27:59 -08:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
obsolete_files_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
options_file_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pre_release_callback.h Mark logs with prepare in PreReleaseCallback (#5121) 2019-04-02 15:17:47 -07:00
prefix_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Make ReadRangeDelAggregator::ShouldDelete() more inline friendly (#5202) 2019-04-18 12:27:25 -07:00
range_del_aggregator.h RangeDelAggregator::StripeRep::Invalidate() to be skipped if empty (#5312) 2019-05-16 15:24:28 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Avoid per-key upper bound check in BlockBasedTableIterator (#5142) 2019-04-16 11:37:47 -07:00
range_tombstone_fragmenter.h Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
read_callback.h WritePrepared: fix race condition in reading batch with duplicate keys (#5147) 2019-04-12 14:40:41 -07:00
repair_test.cc Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
repair.cc Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
table_cache.h Add some comments in table_cache.h 2019-05-24 14:26:43 -07:00
table_properties_collector_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Fix some variable naming in db/transaction_log_impl.* (#5112) 2019-03-27 12:27:54 -07:00
transaction_log_impl.h Move some file related files outside util/ (#5375) 2019-05-29 20:47:06 -07:00
version_builder_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_builder.cc Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
version_builder.h Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
version_edit_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_edit.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_edit.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_set_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_set.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
version_set.h Add class-level comments to version-related classes (#5348) 2019-05-30 16:18:33 -07:00
wal_manager_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
wal_manager.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
wal_manager.h improve comment for WalManager (#5350) 2019-05-24 10:40:30 -07:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h WriteUnPrepared: Add new WAL marker kTypeBeginUnprepareXID (#4069) 2018-06-28 18:58:29 -07:00
write_batch_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
write_batch.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
write_callback_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
write_thread.h Fix skip WAL for whole write_group when leader's callback fail (#4838) 2019-01-03 12:40:42 -08:00