andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
67789419fa
rocksdb/db
History
Andrew Kryczka 69c471bd9b Handle concurrent manifest update and backup creation 
Summary:
Fixed two related race conditions in backup creation.

(1) CreateNewBackup() uses DB::DisableFileDeletions() to prevent table files
from being deleted while it is copying; however, the MANIFEST file could still
rotate during this time. The fix is to stop deleting the old manifest in the
rotation logic. It will be deleted safely later when PurgeObsoleteFiles() runs
(can only happen when file deletions are enabled).

(2) CreateNewBackup() did not account for the CURRENT file being mutable.
This is significant because the files returned by GetLiveFiles() contain a
particular manifest filename, but the manifest to which CURRENT refers can
change at any time. This causes problems when CURRENT changes between the call
to GetLiveFiles() and when it's copied to the backup directory. To workaround this, I
manually forge a CURRENT file referring to the manifest filename returned in
GetLiveFiles().

(2) also applies to the checkpointing code, so let me know if this approach is
good and I'll make the same change there.

Test Plan:
new test for roll manifest during backup creation.

running the test before this change:

  $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
  ...
  IO error: /tmp/rocksdbtest-9383/backupable_db/MANIFEST-000001: No such file or directory

running the test after this change:

  $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation
  ...
  [ RUN      ] BackupableDBTest.ChangeManifestDuringBackupCreation
  [       OK ] BackupableDBTest.ChangeManifestDuringBackupCreation (2836 ms)

Reviewers: IslamAbdelRahman, anthony, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54711
2016-02-29 12:56:55 -08:00
..
auto_roll_logger_test.cc Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
auto_roll_logger.cc Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
auto_roll_logger.h Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
builder.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
builder.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
c_test.c Deprecate CompactionFilterV2 2015-07-17 18:59:11 +02:00
c.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
column_family_test.cc Add test to make sure DropColumnFamily doesn't impact existing iterators 2016-02-24 10:25:38 -08:00
column_family.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
column_family.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compact_files_test.cc Fixed CompactFiles() spuriously failing or corrupting DB 2016-02-22 13:54:58 -08:00
compacted_db_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compacted_db_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_job_stats_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_job_test.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_job.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_job.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_picker_test.cc Add a new compaction priority that picks file whose overlapping ratio is smallest 2016-02-11 15:59:19 -08:00
compaction_picker.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_picker.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
comparator_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cuckoo_table_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_compaction_filter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_compaction_test.cc Add a new compaction priority that picks file whose overlapping ratio is smallest 2016-02-11 15:59:19 -08:00
db_dynamic_level_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_filesnapshot.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_readonly.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_readonly.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl.cc Fix assert failure when DBImpl::SyncWAL() conflicts with log rolling 2016-02-23 11:42:15 -08:00
db_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_info_dumper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_info_dumper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_iter.cc Use user_comparator when comparing against iterate_upper_bound. 2016-02-11 08:47:16 -05:00
db_iter.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_properties_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_table_properties_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_tailing_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_test_util.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
db_test_util.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
db_test.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
db_universal_compaction_test.cc Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more robust 2016-02-26 11:59:31 -08:00
db_wal_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dbformat_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dbformat.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dbformat.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
deletefile_test.cc Handle concurrent manifest update and backup creation 2016-02-29 12:56:55 -08:00
event_helpers.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_helpers.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_job_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_job.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_job.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
inlineskiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
inlineskiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
internal_stats.cc Eliminate duplicated property constants 2016-02-02 19:14:56 -08:00
internal_stats.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
job_context.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
listener_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_format.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
managed_iterator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
managed_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtablerep_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_context.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_operator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
prefix_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
repair.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
skiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
skiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_properties_collector_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_properties_collector.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_properties_collector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
transaction_log_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
transaction_log_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set.cc Handle concurrent manifest update and backup creation 2016-02-29 12:56:55 -08:00
version_set.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
wal_manager_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
wal_manager.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
wal_manager.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_base.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_internal.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_callback_test.cc Improve write_with_callback_test to sync WAL 2016-02-16 14:04:14 -08:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
writebuffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00