andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Mike Kolupaev eef63ef807 Fixed CompactFiles() spuriously failing or corrupting DB 
Summary:
We started getting two kinds of crashes since we started using `DB::CompactFiles()`:
(1) `CompactFiles()` fails saying something like "/data/logdevice/4440/shard12/012302.sst: No such file or directory", and presumably makes DB read-only,
(2) DB fails to open saying "Corruption: Can't access /267000.sst: IO error: /data/logdevice/4440/shard1/267000.sst: No such file or directory".

AFAICT, both can be explained by background thread deleting compaction output as "obsolete" while it's being written, before it's committed to manifest. If it ends up committed to the manifest, we get (2); if compaction notices the disappearance and fails, we get (1). The internal tasks t10068021 and t10134177 have some details about the investigation that led to this.

Test Plan: `make -j check`; the new test fails to reopen the DB without the fix

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, sdong

Differential Revision: https://reviews.facebook.net/D54561
2016-02-22 13:54:58 -08:00
..
auto_roll_logger_test.cc
Use condition variable in log roller test
2016-02-18 18:03:53 -08:00
auto_roll_logger.cc
Fix race conditions in auto-rolling logger
2016-02-17 12:06:45 -08:00
auto_roll_logger.h
This addresses build issues on Windows
2016-02-19 12:29:54 -08:00
builder.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
builder.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
c_test.c
Deprecate CompactionFilterV2
2015-07-17 18:59:11 +02:00
c.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
column_family_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
column_family.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
column_family.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compact_files_test.cc
Fixed CompactFiles() spuriously failing or corrupting DB
2016-02-22 13:54:58 -08:00
compacted_db_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compacted_db_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_job_stats_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_job_test.cc
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
compaction_job.cc
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
compaction_job.h
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
compaction_picker_test.cc
Add a new compaction priority that picks file whose overlapping ratio is smallest
2016-02-11 15:59:19 -08:00
compaction_picker.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_picker.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
comparator_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
convenience.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
corruption_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
cuckoo_table_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_compaction_filter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_compaction_test.cc
Add a new compaction priority that picks file whose overlapping ratio is smallest
2016-02-11 15:59:19 -08:00
db_dynamic_level_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_filesnapshot.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl_debug.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl_experimental.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl_readonly.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl_readonly.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl.cc
Fixed CompactFiles() spuriously failing or corrupting DB
2016-02-22 13:54:58 -08:00
db_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_info_dumper.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_info_dumper.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_iter.cc
Use user_comparator when comparing against iterate_upper_bound.
2016-02-11 08:47:16 -05:00
db_iter.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_log_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_properties_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_table_properties_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_tailing_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_test_util.cc
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
db_test_util.h
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
db_test.cc
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
db_universal_compaction_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_wal_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
deletefile_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
event_helpers.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
event_helpers.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
experimental.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
fault_injection_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_scheduler.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_scheduler.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
inlineskiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
inlineskiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
internal_stats.cc
Eliminate duplicated property constants
2016-02-02 19:14:56 -08:00
internal_stats.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
job_context.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
listener_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_format.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_reader.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_reader.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_writer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_writer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
managed_iterator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
managed_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
manual_compaction_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_list_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_list.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_list.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtablerep_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_context.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_operator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_file_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_context_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
plain_table_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
prefix_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
repair.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
skiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
skiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
slice.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_cache.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_cache.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_properties_collector_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_properties_collector.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_properties_collector.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
transaction_log_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
transaction_log_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set.cc
Add a new compaction priority that picks file whose overlapping ratio is smallest
2016-02-11 15:59:19 -08:00
version_set.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
wal_manager_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
wal_manager.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
wal_manager.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_base.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_internal.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_callback_test.cc
Improve write_with_callback_test to sync WAL
2016-02-16 14:04:14 -08:00
write_callback.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
writebuffer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00