andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6c0c8dee7b
rocksdb/db
History
sdong 6c0c8dee7b Fix data loss after DB recovery by not allowing flush/compaction to be scheduled until DB opened 
Summary:
Previous run may leave some SST files with higher file numbers than manifest indicates.
Compaction or flush may start to run while DB::Open() is still going on. SST file garbage collection may happen interleaving with compaction or flush, and overwrite files generated by compaction of flushes after they are generated. This might cause data loss. This possibility of interleaving is recently introduced.
Fix it by not allowing compaction or flush to be scheduled before DB::Open() finishes.

Test Plan: Add a unit test. This verification will have a chance to fail without the fix but doesn't fix without the fix.

Reviewers: kradhakrishnan, anthony, yhchiang, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42399
2015-07-16 10:57:41 -07:00
..
builder.cc Deprecate purge_redundant_kvs_while_flush 2015-07-14 13:07:02 +02:00
builder.h Add more table properties to EventLogger 2015-05-12 15:53:55 -07:00
c_test.c Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
c.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
column_family_test.cc Merge the latest changes from github/master 2015-07-02 17:23:41 -07:00
column_family.cc fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
column_family.h Fail DB::Open() when the requested compression is not available 2015-06-18 14:55:05 -07:00
compact_files_test.cc Make "make all" work for CYGWIN 2015-06-09 16:36:07 -07:00
compacted_db_impl.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compacted_db_impl.h Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compaction_job_stats_test.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
compaction_job_test.cc Test for compaction of corrupted keys 2015-07-16 09:18:35 -07:00
compaction_job.cc Refactoring of writing key/value pairs 2015-07-15 09:55:45 -07:00
compaction_job.h Refactoring of writing key/value pairs 2015-07-15 09:55:45 -07:00
compaction_picker_test.cc Enabling trivial move in universal compaction 2015-07-07 14:18:55 -07:00
compaction_picker.cc Build fix. 2015-07-15 11:25:10 -07:00
compaction_picker.h Enabling trivial move in universal compaction 2015-07-07 14:18:55 -07:00
compaction.cc "make format" against last 10 commits 2015-07-13 13:50:18 -07:00
compaction.h "make format" against last 10 commits 2015-07-13 13:50:18 -07:00
comparator_db_test.cc Make "make all" work for CYGWIN 2015-06-09 16:36:07 -07:00
convenience.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
corruption_test.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
cuckoo_table_db_test.cc Allowing L0 -> L1 trivial move on sorted data 2015-06-04 16:51:25 -07:00
db_bench.cc Update --help message in db_bench. 2015-07-15 10:21:09 -07:00
db_compaction_filter_test.cc Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc 2015-07-14 16:03:47 -07:00
db_dynamic_level_test.cc Move DynamicLevel related db-tests to db_dynamic_level_test.cc 2015-07-13 19:00:30 -07:00
db_filesnapshot.cc Add wal files to Checkpoint for multiple column families. 2015-06-19 16:08:31 -07:00
db_impl_debug.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
db_impl_experimental.cc Clean up InstallSuperVersion 2015-06-17 12:37:59 -07:00
db_impl_readonly.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
db_impl_readonly.h Use CompactRangeOptions for CompactRange 2015-06-17 14:36:14 -07:00
db_impl.cc Fix data loss after DB recovery by not allowing flush/compaction to be scheduled until DB opened 2015-07-16 10:57:41 -07:00
db_impl.h Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
db_iter_test.cc Revert two diffs related to DBIter::FindPrevUserKey() 2015-07-07 11:36:24 -07:00
db_iter.cc Revert two diffs related to DBIter::FindPrevUserKey() 2015-07-07 11:36:24 -07:00
db_iter.h reduce references to cfd->options() in DBImpl 2014-09-08 15:04:34 -07:00
db_log_iter_test.cc Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc 2015-07-14 16:08:21 -07:00
db_tailing_iter_test.cc Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc 2015-07-14 16:41:08 -07:00
db_test.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
db_universal_compaction_test.cc Move UniversalCompaction related db-tests to db_universal_compaction_test.cc 2015-07-14 18:24:45 -07:00
dbformat_test.cc Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
dbformat.cc Replace %llu with format macros in ParsedInternalKey::DebugString()) 2015-06-17 20:44:26 -07:00
dbformat.h Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
deletefile_test.cc Use CompactRangeOptions for CompactRange 2015-06-17 14:36:14 -07:00
event_helpers.cc Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
event_helpers.h Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
experimental.cc Implement DB::PromoteL0 method 2015-04-23 12:10:36 -07:00
fault_injection_test.cc Fix data loss after DB recovery by not allowing flush/compaction to be scheduled until DB opened 2015-07-16 10:57:41 -07:00
file_indexer_test.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.h Fix public API dependency on internal codes and dependency on MAX_INT32 2015-07-11 10:32:11 -07:00
filename_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
filename.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
filename.h Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
flush_job_test.cc Optimistic Transactions 2015-05-29 14:36:35 -07:00
flush_job.cc Add largest sequence to FlushJobInfo 2015-06-11 15:22:22 -07:00
flush_job.h Add largest sequence to FlushJobInfo 2015-06-11 15:22:22 -07:00
flush_scheduler.cc Don't return (or dereference) dangling pointer 2014-10-02 14:33:16 -07:00
flush_scheduler.h Fix data race #1 2015-01-26 11:48:07 -08:00
forward_iterator.cc fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
forward_iterator.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
internal_stats.cc Fixed a bug of CompactionStats in multi-level universal compaction case 2015-06-17 23:40:34 -07:00
internal_stats.h Fixed a bug of CompactionStats in multi-level universal compaction case 2015-06-17 23:40:34 -07:00
job_context.h fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
listener_test.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
log_format.h Some minor refactoring on the code 2014-01-02 16:32:31 -08:00
log_reader.cc Introduce WAL recovery consistency levels 2015-06-22 15:28:12 -07:00
log_reader.h Introduce WAL recovery consistency levels 2015-06-22 15:28:12 -07:00
log_test.cc Introduce WAL recovery consistency levels 2015-06-22 15:28:12 -07:00
log_writer.cc Fix comparison between signed and usigned integers 2015-05-19 10:59:30 -07:00
log_writer.h Log writer record format doc. 2015-04-07 16:25:56 -07:00
managed_iterator.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
managed_iterator.h Fixed xfunc related compile errors in ROCKSDB_LITE 2015-04-09 21:05:18 -07:00
memtable_allocator.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_list_test.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
memtable_list.cc Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable_list.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable.cc Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtablerep_bench.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
merge_context.h API to fetch from both a WriteBatchWithIndex and the db 2015-05-11 14:51:51 -07:00
merge_helper.cc Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
merge_helper.h Helper function to time Merges 2015-04-27 20:23:50 -07:00
merge_operator.cc Call merge operators with empty values 2015-06-26 11:35:46 -07:00
merge_test.cc Call merge operators with empty values 2015-06-26 11:35:46 -07:00
perf_context_test.cc Makefile minor cleanup 2015-03-30 16:05:35 -04:00
plain_table_db_test.cc rocksdb: Remove #include "util/string_util.h" from util/testharness.h 2015-03-19 17:29:37 -07:00
prefix_test.cc rocksdb: Remove #include "util/string_util.h" from util/testharness.h 2015-03-19 17:29:37 -07:00
repair.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
skiplist_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
skiplist.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
slice.cc Create an abstract interface for write batches 2015-03-17 19:23:08 -07:00
snapshot.h GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex 2015-06-08 21:57:02 -07:00
table_cache.cc perf_context: report time spent on reading index and bloom blocks 2015-07-10 14:45:42 -07:00
table_cache.h Implement a table-level row cache 2015-06-23 10:25:45 -07:00
table_properties_collector_test.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
table_properties_collector.cc A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge 2015-04-06 10:27:21 -07:00
table_properties_collector.h Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
transaction_log_impl.cc Turn -Wshadow back on 2014-11-06 11:14:28 -08:00
transaction_log_impl.h "make format" against last 10 commits 2015-07-13 13:50:18 -07:00
version_builder_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_builder.cc Fix deleting obsolete files 2015-02-06 08:44:30 -08:00
version_builder.h Move VersionBuilder logic to a separate .cc file 2014-10-31 16:34:38 -07:00
version_edit_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_edit.cc Turn on -Wshadow 2014-10-31 11:59:54 -07:00
version_edit.h Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_set_test.cc Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true 2015-04-03 09:04:35 -07:00
version_set.cc Merge the latest changes from github/master 2015-07-02 17:23:41 -07:00
version_set.h [wal changes 1/3] fixed unbounded wal growth in some workloads 2015-07-02 14:27:00 -07:00
wal_manager_test.cc Fix flakiness of WalManagerTest 2015-04-13 16:15:05 -07:00
wal_manager.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
wal_manager.h Fix -Wnon-virtual-dtor errors 2014-11-10 17:39:38 -05:00
write_batch_base.cc WriteBatch.Merge w/ SliceParts support 2015-05-29 04:30:03 -07:00
write_batch_internal.h WriteBatch.Merge w/ SliceParts support 2015-05-29 04:30:03 -07:00
write_batch_test.cc Allow write_batch_test to run with ROCKSDB_LITE 2015-07-14 10:35:35 -07:00
write_batch.cc WriteBatch.Merge w/ SliceParts support 2015-05-29 04:30:03 -07:00
write_callback_test.cc Optimistic Transactions 2015-05-29 14:36:35 -07:00
write_callback.h Optimistic Transactions 2015-05-29 14:36:35 -07:00
write_controller_test.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller.h Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_thread.cc Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
write_thread.h Deprecate WriteOptions::timeout_hint_us 2015-07-14 09:35:48 +02:00
writebuffer.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00