andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
af42561165
rocksdb/db
History
Mike Kolupaev af42561165 Fixed CompactFiles() spuriously failing or corrupting DB 
Summary:
We started getting two kinds of crashes since we started using `DB::CompactFiles()`:
(1) `CompactFiles()` fails saying something like "/data/logdevice/4440/shard12/012302.sst: No such file or directory", and presumably makes DB read-only,
(2) DB fails to open saying "Corruption: Can't access /267000.sst: IO error: /data/logdevice/4440/shard1/267000.sst: No such file or directory".

AFAICT, both can be explained by background thread deleting compaction output as "obsolete" while it's being written, before it's committed to manifest. If it ends up committed to the manifest, we get (2); if compaction notices the disappearance and fails, we get (1). The internal tasks t10068021 and t10134177 have some details about the investigation that led to this.

Test Plan: `make -j check`; the new test fails to reopen the DB without the fix

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, sdong

Differential Revision: https://reviews.facebook.net/D54561
2016-02-23 09:45:42 -08:00
..
auto_roll_logger_test.cc [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
auto_roll_logger.cc RollLogFile tries to find non conflicting file until there is no conflict. 2016-02-02 10:33:49 +01:00
auto_roll_logger.h [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
builder.cc Use SST files for Transaction conflict detection 2015-12-11 12:34:11 -08:00
builder.h Use SST files for Transaction conflict detection 2015-12-11 12:34:11 -08:00
c_test.c Deprecate CompactionFilterV2 2015-07-17 18:59:11 +02:00
c.cc Merge pull request #916 from warrenfalk/capi_huge_page_option 2016-01-04 09:24:30 -08:00
column_family_test.cc Fixed the asan error on column_family_test 2016-02-01 12:45:45 -08:00
column_family.cc Explictly fail when memtable doesn't support concurrent insert 2016-02-05 15:44:21 -08:00
column_family.h Add ColumnFamilyHandle::GetDescriptor() 2016-01-06 18:14:01 -08:00
compact_files_test.cc Fixed CompactFiles() spuriously failing or corrupting DB 2016-02-23 09:45:42 -08:00
compacted_db_impl.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compacted_db_impl.h Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compaction_iterator_test.cc Support marking snapshots for write-conflict checking - Take 2 2015-12-08 16:47:31 -08:00
compaction_iterator.cc compaction assertion triggering test fix for sequence zeroing assertion trip 2015-12-18 16:08:31 -08:00
compaction_iterator.h Change SingleDelete to support conflict checking 2015-12-10 11:35:38 -08:00
compaction_job_stats_test.cc Fix rebase issues and new code warnings. 2015-12-11 16:56:24 -08:00
compaction_job_test.cc compaction assertion triggering test fix for sequence zeroing assertion trip 2015-12-18 16:08:31 -08:00
compaction_job.cc Add SstFileManager (component tracking all SST file in DBs and control the deletion rate) 2016-01-28 18:35:01 -08:00
compaction_job.h fix typos in comments 2015-12-11 01:54:48 +09:00
compaction_picker_test.cc Not scheduling more L1->L2 compaction if L0->L1 is pending with higher priority 2016-01-08 13:56:57 -08:00
compaction_picker.cc Not scheduling more L1->L2 compaction if L0->L1 is pending with higher priority 2016-01-08 13:56:57 -08:00
compaction_picker.h Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
compaction.cc Merge pull request #846 from yuslepukhin/enble_c4244_lossofdata 2015-12-23 22:59:42 -08:00
compaction.h Report compaction reason in CompactionListener 2015-12-22 11:37:19 -08:00
comparator_db_test.cc Moving memtable related files from util to a new directory memtable 2015-10-16 14:10:33 -07:00
convenience.cc Delete files in given key range 2015-12-29 13:22:13 -08:00
corruption_test.cc Merge pull request #846 from yuslepukhin/enble_c4244_lossofdata 2015-12-23 22:59:42 -08:00
cuckoo_table_db_test.cc Block cuckoo table tests in ROCKSDB_LITE 2015-07-20 10:50:46 -07:00
db_bench.cc db_bench: explicitly clear buffer in compress benchmark 2016-01-19 18:11:46 -08:00
db_compaction_filter_test.cc compaction assertion triggering test fix for sequence zeroing assertion trip 2015-12-18 16:08:31 -08:00
db_compaction_test.cc Add options.base_background_compactions as a number of compaction threads for low compaction debt 2016-01-29 16:15:53 -08:00
db_dynamic_level_test.cc No need to #ifdef test only code on windows 2015-10-22 15:15:37 -07:00
db_filesnapshot.cc Add wal files to Checkpoint for multiple column families. 2015-06-19 16:08:31 -07:00
db_impl_debug.cc Running manual compactions in parallel with other automatic or manual compactions in restricted cases 2015-12-14 11:20:34 -08:00
db_impl_experimental.cc Clean up InstallSuperVersion 2015-06-17 12:37:59 -07:00
db_impl_readonly.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
db_impl_readonly.h Override DBImplReadOnly::SyncWAL() to return NotSupported. Previously, calling it caused program abort. 2015-09-25 21:25:30 -07:00
db_impl.cc Fixed CompactFiles() spuriously failing or corrupting DB 2016-02-23 09:45:42 -08:00
db_impl.h Allows Get and MultiGet to read directly from SST files. 2016-02-09 11:25:33 -08:00
db_info_dumper.cc [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
db_info_dumper.h [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
db_inplace_update_test.cc Clean up dependency: Move db_test_util.* to db directory 2015-10-12 13:05:42 -07:00
db_iter_test.cc No need to #ifdef test only code on windows 2015-10-22 15:15:37 -07:00
db_iter.cc Optimize DBIter::Prev() by reducing stack overhead 2016-01-07 07:59:14 -08:00
db_iter.h Introduce ReadOptions::pin_data (support zero copy for keys) 2015-12-16 12:08:30 -08:00
db_log_iter_test.cc Fix clang warnings regarding unnecessary std::move 2015-12-24 04:10:00 +08:00
db_properties_test.cc Eliminate duplicated property constants 2016-02-02 19:14:56 -08:00
db_table_properties_test.cc Avoid empty ranges vector with subsequent zero element access 2015-12-02 14:50:33 -08:00
db_tailing_iter_test.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
db_test_util.cc Add BlockBasedTableOptions::index_block_restart_interval 2016-02-05 15:44:21 -08:00
db_test_util.h Add BlockBasedTableOptions::index_block_restart_interval 2016-02-05 15:44:21 -08:00
db_test.cc Allows Get and MultiGet to read directly from SST files. 2016-02-09 11:25:33 -08:00
db_universal_compaction_test.cc Skip filters for last L0 file if hit-optimized 2016-02-01 14:58:46 -08:00
db_wal_test.cc No need to #ifdef test only code on windows 2015-10-22 15:15:37 -07:00
dbformat_test.cc Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
dbformat.cc Support for SingleDelete() 2015-09-17 11:42:56 -07:00
dbformat.h Introduce ReadOptions::pin_data (support zero copy for keys) 2015-12-16 12:08:30 -08:00
deletefile_test.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
event_helpers.cc Enable MS compiler warning c4244. 2015-12-11 16:47:34 -08:00
event_helpers.h Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
experimental.cc Implement DB::PromoteL0 method 2015-04-23 12:10:36 -07:00
fault_injection_test.cc fix sporadic failure in fault_injection_test 2016-01-06 11:51:53 -08:00
file_indexer_test.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.h fix typos in comments 2015-12-11 01:54:48 +09:00
filename_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
filename.cc Enable RocksDB to persist Options file. 2015-11-10 22:58:01 -08:00
filename.h Enable RocksDB to persist Options file. 2015-11-10 22:58:01 -08:00
flush_job_test.cc Use SST files for Transaction conflict detection 2015-12-11 12:34:11 -08:00
flush_job.cc [directory includes cleanup] Remove util->db dependency for ThreadStatusUtil 2016-01-26 10:49:16 -08:00
flush_job.h Use SST files for Transaction conflict detection 2015-12-11 12:34:11 -08:00
flush_scheduler.cc support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
flush_scheduler.h support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
forward_iterator_bench.cc Block forward_iterator_bench under MAC and Windows 2015-11-17 11:51:37 -08:00
forward_iterator.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
forward_iterator.h Reuse file iterators in tailing iterator when memtable is flushed 2015-11-13 15:50:59 -08:00
inlineskiplist_test.cc support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
inlineskiplist.h always invalidate sequential-insertion cache for concurrent skiplist adds 2016-02-03 11:08:16 -08:00
internal_stats.cc Eliminate duplicated property constants 2016-02-02 19:14:56 -08:00
internal_stats.h Eliminate duplicated property constants 2016-02-02 19:14:56 -08:00
job_context.h fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
listener_test.cc Report compaction reason in CompactionListener 2015-12-22 11:37:19 -08:00
log_format.h log_{reader,write}: recyclable record format 2015-10-19 17:24:05 -04:00
log_reader.cc log_{reader,write}: recyclable record format 2015-10-19 17:24:05 -04:00
log_reader.h log_{reader,write}: recyclable record format 2015-10-19 17:24:05 -04:00
log_test.cc log_{reader,write}: recyclable record format 2015-10-19 17:24:05 -04:00
log_writer.cc Enable C4267 warning 2015-11-24 16:33:09 +03:00
log_writer.h Enable C4267 warning 2015-11-24 16:33:09 +03:00
managed_iterator.cc [directory includes cleanup] Move cross-function test points 2016-01-26 10:49:05 -08:00
managed_iterator.h Fixed xfunc related compile errors in ROCKSDB_LITE 2015-04-09 21:05:18 -07:00
manual_compaction_test.cc Move manual_compaction_test.cc from util to db 2015-10-14 11:06:27 -07:00
memtable_allocator.cc support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
memtable_allocator.h support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
memtable_list_test.cc Removing duplicate code 2015-08-05 07:33:27 -07:00
memtable_list.cc Seperate InternalIterator from Iterator 2015-10-13 15:32:13 -07:00
memtable_list.h Seperate InternalIterator from Iterator 2015-10-13 15:32:13 -07:00
memtable.cc Optimize GetLatestSequenceForKey 2016-01-06 13:43:22 -08:00
memtable.h support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
memtablerep_bench.cc Merge pull request #811 from OverlordQ/unused-variable-warning 2015-11-02 12:44:27 -08:00
merge_context.h API to fetch from both a WriteBatchWithIndex and the db 2015-05-11 14:51:51 -07:00
merge_helper_test.cc Compaction filter on merge operands 2015-10-07 09:30:03 -07:00
merge_helper.cc Seperate InternalIterator from Iterator 2015-10-13 15:32:13 -07:00
merge_helper.h Seperate InternalIterator from Iterator 2015-10-13 15:32:13 -07:00
merge_operator.cc Call merge operators with empty values 2015-06-26 11:35:46 -07:00
merge_test.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
options_file_test.cc Fixed build failure of RocksDBLite test on options_file_test.cc 2015-11-10 23:23:36 -08:00
perf_context_test.cc PerfContext::ToString() add option to exclude zero counters 2016-02-01 13:41:13 -08:00
plain_table_db_test.cc Fix compile error. 2016-01-11 16:10:48 -08:00
prefix_test.cc Prefix-based iterating only shows keys in prefix 2015-11-05 13:24:05 -08:00
repair.cc Disable Visual Studio Warning C4351 2015-12-28 15:06:34 -08:00
skiplist_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
skiplist.h Switch to thread-local random for skiplist 2015-11-09 19:25:22 -08:00
slice.cc Create an abstract interface for write batches 2015-03-17 19:23:08 -07:00
snapshot_impl.cc Support marking snapshots for write-conflict checking - Take 2 2015-12-08 16:47:31 -08:00
snapshot_impl.h Use SST files for Transaction conflict detection 2015-12-11 12:34:11 -08:00
table_cache.cc Allows Get and MultiGet to read directly from SST files. 2016-02-09 11:25:33 -08:00
table_cache.h Skip bottom-level filter block caching when hit-optimized 2015-12-23 10:15:07 -08:00
table_properties_collector_test.cc Pass column family ID to table property collector 2015-10-09 14:36:51 -07:00
table_properties_collector.cc Support for SingleDelete() 2015-09-17 11:42:56 -07:00
table_properties_collector.h Pass column family ID to table property collector 2015-10-09 14:36:51 -07:00
transaction_log_impl.cc log_reader: pass log_number and optional info_log to ctor 2015-10-18 21:24:32 -04:00
transaction_log_impl.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
version_builder_test.cc Add a mode to always pick the oldest file to compact for each level 2015-09-21 17:21:59 -07:00
version_builder.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
version_builder.h Log more information for the add file with overlapping range failure 2015-10-19 17:31:13 -07:00
version_edit_test.cc New Manifest format to allow customized fields in NewFile. 2015-10-08 15:51:45 -07:00
version_edit.cc New Manifest format to allow customized fields in NewFile. 2015-10-08 15:51:45 -07:00
version_edit.h Enable MS compiler warning c4244. 2015-12-11 16:47:34 -08:00
version_set_test.cc Fix minor bugs in delete operator, snprintf, and size_t usage 2015-12-15 15:26:20 -08:00
version_set.cc Skip filters for last L0 file if hit-optimized 2016-02-01 14:58:46 -08:00
version_set.h Skip filters for last L0 file if hit-optimized 2016-02-01 14:58:46 -08:00
wal_manager_test.cc Fix clang warnings regarding unnecessary std::move 2015-12-24 04:10:00 +08:00
wal_manager.cc Fix clang warnings regarding unnecessary std::move 2015-12-24 04:10:00 +08:00
wal_manager.h Fix -Wnon-virtual-dtor errors 2014-11-10 17:39:38 -05:00
write_batch_base.cc Support for SingleDelete() 2015-09-17 11:42:56 -07:00
write_batch_internal.h support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
write_batch_test.cc Fix WriteBatchTest.ManyUpdates, WriteBatchTest.LargeKeyValue under clang 2016-02-01 16:07:53 -08:00
write_batch.cc support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
write_callback_test.cc Fix compile for write_callback_test in ROCKSDB_LITE 2015-07-20 10:54:15 -07:00
write_callback.h Optimistic Transactions 2015-05-29 14:36:35 -07:00
write_controller_test.cc When slowdown is triggered, reduce the write rate 2015-12-23 11:33:15 -08:00
write_controller.cc Add options.base_background_compactions as a number of compaction threads for low compaction debt 2016-01-29 16:15:53 -08:00
write_controller.h Add options.base_background_compactions as a number of compaction threads for low compaction debt 2016-01-29 16:15:53 -08:00
write_thread.cc use -Werror=missing-field-initializers, to closer match MyRocks build 2015-12-30 14:56:18 -08:00
write_thread.h use -Werror=missing-field-initializers, to closer match MyRocks build 2015-12-30 14:56:18 -08:00
writebuffer.h support for concurrent adds to memtable 2015-12-25 11:03:40 -08:00
xfunc_test_points.cc [directory includes cleanup] Move cross-function test points 2016-01-26 10:49:05 -08:00
xfunc_test_points.h [directory includes cleanup] Move cross-function test points 2016-01-26 10:49:05 -08:00