andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
sdong 2a6d0cde72 Ignore stale logs while restarting DBs 
Summary:
Stale log files can be deleted out of order. This can happen for various reasons. One of the reason is that no data is ever inserted to a column family and we have an optimization to update its log number, but not all the old log files are cleaned up (the case shown in the unit tests added). It can also happen when we simply delete multiple log files out of order.

This causes data corruption because we simply increase seqID after processing the next row and we may end up with writing data with smaller seqID than what is already flushed to memtables.

In DB recovery, for the oldest files we are replaying, if there it contains no data for any column family, we ignore the sequence IDs in the file.

Test Plan: Add two unit tests that fail without the fix.

Reviewers: IslamAbdelRahman, igor, yiwu

Reviewed By: yiwu

Subscribers: hermanlee4, yoshinorim, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60891
2016-07-25 11:47:31 -07:00
..
auto_roll_logger_test.cc
Cleanup auto-roll logger flush-while-rolling test
2016-07-07 11:35:40 -07:00
auto_roll_logger.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
auto_roll_logger.h
Redo SyncPoints for flush while rolling test
2016-02-22 21:32:19 -08:00
builder.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
builder.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
c_test.c
Shared dictionary compression using reference block
2016-04-27 17:36:03 -07:00
c.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
column_family_test.cc
Ignore stale logs while restarting DBs
2016-07-25 11:47:31 -07:00
column_family.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
column_family.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
compact_files_test.cc
Fixed CompactFiles() spuriously failing or corrupting DB
2016-02-22 13:54:58 -08:00
compacted_db_impl.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
compacted_db_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
compaction_iterator.h
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
compaction_job_stats_test.cc
Introduce XPRESS compresssion on Windows. (#1081)
2016-04-19 22:54:24 -07:00
compaction_job_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
compaction_job.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_job.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_picker_test.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
compaction_picker.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
compaction_picker.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
compaction.h
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
comparator_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
convenience.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
corruption_test.cc
Change several option defaults
2016-04-28 17:50:58 -07:00
cuckoo_table_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_block_cache_test.cc
add option to not flush memtable on open()
2016-06-13 11:34:16 -07:00
db_bloom_filter_test.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
db_compaction_filter_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_compaction_test.cc
Add a new feature to enforce a sync point only active on a thread
2016-07-07 11:29:14 -07:00
db_dynamic_level_test.cc
Fix flaky DBTestDynamicLevel.DynamicLevelMaxBytesBase2
2016-05-26 10:13:24 -07:00
db_filesnapshot.cc
Backup Options
2016-06-09 19:03:10 -07:00
db_flush_test.cc
Fix unit test which breaks lite build
2016-07-21 14:32:12 -07:00
db_impl_add_file.cc
update DB::AddFile to ingest list of sst files
2016-07-11 10:43:12 -07:00
db_impl_debug.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
db_impl_experimental.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
db_impl_readonly.cc
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
db_impl_readonly.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl.cc
Ignore stale logs while restarting DBs
2016-07-25 11:47:31 -07:00
db_impl.h
Need to make sure log file synced before flushing memtable of one column family
2016-07-21 16:29:06 -07:00
db_info_dumper.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_info_dumper.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_io_failure_test.cc
Move IO failure test to separate file
2016-05-18 17:09:20 -07:00
db_iter_test.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
db_iter.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
db_iter.h
Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
2016-04-26 12:41:07 -07:00
db_iterator_test.cc
Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224)
2016-07-20 15:28:04 -07:00
db_log_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_options_test.cc
Fix deadlock when trying update options when write stalls
2016-07-12 15:30:38 -07:00
db_properties_test.cc
Miscellaneous performance improvements
2016-07-12 14:15:32 -07:00
db_sst_test.cc
Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224)
2016-07-20 15:28:04 -07:00
db_table_properties_test.cc
DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky
2016-07-19 15:46:20 -07:00
db_tailing_iter_test.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
db_test2.cc
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
db_test_util.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
db_test_util.h
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
db_test.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
db_universal_compaction_test.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
db_wal_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
dbformat_test.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
dbformat.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat.h
Miscellaneous performance improvements
2016-07-12 14:15:32 -07:00
deletefile_test.cc
Fix a bug that accesses invalid address in iterator cleanup function
2016-07-05 11:57:14 -07:00
event_helpers.cc
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
event_helpers.h
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
experimental.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
fault_injection_test.cc
[rocksdb] Recovery path sequence miscount fix
2016-05-10 14:06:07 -07:00
file_indexer_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job_test.cc
Need to make sure log file synced before flushing memtable of one column family
2016-07-21 16:29:06 -07:00
flush_job.cc
Need to make sure log file synced before flushing memtable of one column family
2016-07-21 16:29:06 -07:00
flush_job.h
Need to make sure log file synced before flushing memtable of one column family
2016-07-21 16:29:06 -07:00
flush_scheduler.cc
Fix clang analyzer errors
2016-07-08 17:50:51 -07:00
flush_scheduler.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
forward_iterator.h
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
inlineskiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
inlineskiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
internal_stats.cc
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
internal_stats.h
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
job_context.h
Add More Logging to track total_log_size
2016-07-06 14:29:18 -07:00
listener_test.cc
Add a callback for when memtable is moved to immutable (#1137)
2016-06-02 11:57:31 -07:00
log_format.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_reader.cc
Fix clang analyzer errors
2016-07-08 17:50:51 -07:00
log_reader.h
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_test.cc
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_writer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_writer.h
Need to make sure log file synced before flushing memtable of one column family
2016-07-21 16:29:06 -07:00
managed_iterator.cc
Fixes warnings and ensure correct int behavior on 32-bit platforms.
2016-03-16 22:57:57 +01:00
managed_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
manual_compaction_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.cc
Avoid updating memtable allocated bytes if write_buffer_size is not set
2016-07-13 19:33:57 -07:00
memtable_allocator.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
memtable_list_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
memtable_list.cc
Fix flush not being commit while writing manifest
2016-07-21 10:10:41 -07:00
memtable_list.h
[rocksdb] Memtable Log Referencing and Prepared Batch Recovery
2016-05-10 14:06:07 -07:00
memtable.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
memtable.h
MemTable::PostProcess() can skip updating num_deletes if the delta is 0
2016-07-19 18:10:18 -07:00
memtablerep_bench.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
merge_context.h
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
merge_helper_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
merge_helper.h
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
merge_operator.cc
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
merge_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_file_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_context_test.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h
Introduce FullMergeV2 (eliminate memcpy from merge operators)
2016-07-20 09:49:03 -07:00
plain_table_db_test.cc
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
prefix_test.cc
memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
2016-06-10 12:12:10 -07:00
repair_test.cc
ColumnFamilyOptions API [CF + RepairDB part 3/3]
2016-06-24 16:29:43 -07:00
repair.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
skiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
skiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_cache.cc
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
table_cache.h
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
table_properties_collector_test.cc
New Statistics to track Compression/Decompression (#1197)
2016-07-19 09:44:03 -07:00
table_properties_collector.cc
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
table_properties_collector.h
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
transaction_log_impl.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
transaction_log_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder.cc
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
version_builder.h
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
version_edit_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.cc
Miscellaneous performance improvements
2016-07-12 14:15:32 -07:00
version_edit.h
Avoid FileMetaData copy
2016-07-13 15:36:22 -07:00
version_set_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set.cc
Only cache level 0 indexes and filter when opening table reader
2016-07-20 11:23:31 -07:00
version_set.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
wal_manager_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
wal_manager.cc
In all the places where log records are read, there was a check that
2016-03-30 23:05:22 +05:30
wal_manager.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_base.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_internal.h
Ignore stale logs while restarting DBs
2016-07-25 11:47:31 -07:00
write_batch_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
write_batch.cc
Ignore stale logs while restarting DBs
2016-07-25 11:47:31 -07:00
write_callback_test.cc
write_callback_test: clean test directory before running tests
2016-04-18 10:18:41 -07:00
write_callback.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.h
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
xfunc_test_points.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00