andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Make backup restore atomic, with sync option (#8568)

Browse Source 
Summary:
Guarantees that if a restore is interrupted, DB::Open will fail. This works by
restoring CURRENT first to CURRENT.tmp then as a final step renaming to CURRENT.

Also makes restore respect BackupEngineOptions::sync (default true). When set,
the restore is guaranteed persisted by the time it returns OK. Also makes the above
atomicity guarantee work in case the interruption is power loss or OS crash (not just
process interruption or crash).

Fixes https://github.com/facebook/rocksdb/issues/8500

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8568

Test Plan:
added to backup mini-stress unit test. Passes with
gtest_repeat=100 (whereas fails 7 times without the CURRENT.tmp)

Reviewed By: akankshamahajan15

Differential Revision: D29812605

Pulled By: pdillinger

fbshipit-source-id: 24e9a993b305b1835ca95558fa7a7152e54cda8e
This commit is contained in:
Peter Dillinger 2021-08-06 09:48:53 -07:00 committed by Facebook GitHub Bot
parent 8ca081780b
commit a7fd1d0881
4 changed files with 169 additions and 69 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
HISTORY.md
View File

@ -2,6 +2,7 @@
## Unreleased
### Bug Fixes
* If the primary's CURRENT file is missing or inaccessible, the secondary instance should not hang repeatedly trying to switch to a new MANIFEST. It should instead return the error code encountered while accessing the file.
* Restoring backups with BackupEngine is now a logically atomic operation, so that if a restore operation is interrupted, DB::Open on it will fail. Using BackupEngineOptions::sync (default) ensures atomicity even in case of power loss or OS crash.
* Fixed a race related to the destruction of `ColumnFamilyData` objects. The earlier logic unlocked the DB mutex before destroying the thread-local `SuperVersion` pointers, which could result in a process crash if another thread managed to get a reference to the `ColumnFamilyData` object.
* Removed a call to `RenameFile()` on a non-existent info log file ("LOG") when opening a new DB. Such a call was guaranteed to fail though did not impact applications since we swallowed the error. Now we also stopped swallowing errors in renaming "LOG" file.
* Fixed an issue where `OnFlushCompleted` was not called for atomic flush.
@ -17,6 +18,7 @@
### Behavior Changes
* `StringAppendOperator` additionally accepts a string as the delimiter.
* BackupEngineOptions::sync (default true) now applies to restoring backups in addition to creating backups. This could slow down restores, but ensures they are fully persisted before returning OK. (Consider increasing max_background_operations to improve performance.)
## 6.23.0 (2021-07-16)
### Behavior Changes

9
include/rocksdb/utilities/backup_engine.h
View File

@ -56,10 +56,11 @@ struct BackupEngineOptions {
// Default: nullptr
Logger* info_log;
// If sync == true, we can guarantee you'll get consistent backup even
// on a machine crash/reboot. Backup process is slower with sync enabled.
// If sync == false, we don't guarantee anything on machine reboot. However,
// chances are some of the backups are consistent.
// If sync == true, we can guarantee you'll get consistent backup and
// restore even on a machine crash/reboot. Backup and restore processes are
// slower with sync enabled. If sync == false, we can only guarantee that
// other previously synced backups and restores are not modified while
// creating a new one.
// Default: true
bool sync;

57
utilities/backupable/backupable_db.cc
View File

@ -1707,6 +1707,11 @@ Status BackupEngineImpl::RestoreDBFromBackup(const RestoreOptions& options,
}
Status s;
std::vector<RestoreAfterCopyOrCreateWorkItem> restore_items_to_finish;
std::string temporary_current_file;
std::string final_current_file;
std::unique_ptr<Directory> db_dir_for_fsync;
std::unique_ptr<Directory> wal_dir_for_fsync;
for (const auto& file_info : backup->GetFiles()) {
const std::string& file = file_info->filename;
// 1. get DB filename
@ -1722,13 +1727,36 @@ Status BackupEngineImpl::RestoreDBFromBackup(const RestoreOptions& options,
}
// 3. Construct the final path
// kWalFile lives in wal_dir and all the rest live in db_dir
dst = ((type == kWalFile) ? wal_dir : db_dir) + "/" + dst;
if (type == kWalFile) {
dst = wal_dir + "/" + dst;
if (options_.sync && !wal_dir_for_fsync) {
s = db_env_->NewDirectory(wal_dir, &wal_dir_for_fsync);
if (!s.ok()) {
return s;
}
}
} else {
dst = db_dir + "/" + dst;
if (options_.sync && !db_dir_for_fsync) {
s = db_env_->NewDirectory(db_dir, &db_dir_for_fsync);
if (!s.ok()) {
return s;
}
}
}
// For atomicity, initially restore CURRENT file to a temporary name.
// This is useful even without options_.sync e.g. in case the restore
// process is interrupted.
if (type == kCurrentFile) {
final_current_file = dst;
dst = temporary_current_file = dst + ".tmp";
}
ROCKS_LOG_INFO(options_.info_log, "Restoring %s to %s\n", file.c_str(),
dst.c_str());
CopyOrCreateWorkItem copy_or_create_work_item(
GetAbsolutePath(file), dst, "" /* contents */, backup_env_, db_env_,
EnvOptions() /* src_env_options */, false, rate_limiter,
EnvOptions() /* src_env_options */, options_.sync, rate_limiter,
0 /* size_limit */);
RestoreAfterCopyOrCreateWorkItem after_copy_or_create_work_item(
copy_or_create_work_item.result.get_future(), file, dst,
@ -1757,6 +1785,31 @@ Status BackupEngineImpl::RestoreDBFromBackup(const RestoreOptions& options,
}
}
// When enabled, the first Fsync is to ensure all files are fully persisted
// before renaming CURRENT.tmp
if (s.ok() && db_dir_for_fsync) {
ROCKS_LOG_INFO(options_.info_log, "Restore: fsync\n");
s = db_dir_for_fsync->Fsync();
}
if (s.ok() && wal_dir_for_fsync) {
s = wal_dir_for_fsync->Fsync();
}
if (s.ok() && !temporary_current_file.empty()) {
ROCKS_LOG_INFO(options_.info_log, "Restore: atomic rename CURRENT.tmp\n");
assert(!final_current_file.empty());
s = db_env_->RenameFile(temporary_current_file, final_current_file);
}
if (s.ok() && db_dir_for_fsync && !temporary_current_file.empty()) {
// Second Fsync is to ensure the final atomic rename of DB restore is
// fully persisted even if power goes out right after restore operation
// returns success
assert(db_dir_for_fsync);
s = db_dir_for_fsync->Fsync();
}
ROCKS_LOG_INFO(options_.info_log, "Restoring done -- %s\n",
s.ToString().c_str());
return s;

64
utilities/backupable/backupable_db_test.cc
View File

@ -642,9 +642,22 @@ class BackupEngineTest : public testing::Test {
CreateLoggerFromOptions(dbname_, logger_options, &logger_)
.PermitUncheckedError();
// The sync option is not easily testable in unit tests, but should be
// smoke tested across all the other backup tests. However, it is
// certainly not worth doubling the runtime of backup tests for it.
// Thus, we can enable sync for one of our alternate testing
// configurations.
constexpr bool kUseSync =
#ifdef ROCKSDB_MODIFY_NPHASH
true;
#else
false;
#endif // ROCKSDB_MODIFY_NPHASH
// set up backup db options
backupable_options_.reset(new BackupableDBOptions(
backupdir_, test_backup_env_.get(), true, logger_.get(), true));
backupdir_, test_backup_env_.get(), /*share_table_files*/ true,
logger_.get(), kUseSync));
// most tests will use multi-threaded backups
backupable_options_->max_background_operations = 7;
@ -3122,15 +3135,18 @@ TEST_F(BackupEngineTest, Concurrency) {
Options db_opts = options_;
db_opts.wal_dir = "";
db_opts.create_if_missing = false;
BackupableDBOptions be_opts = *backupable_options_;
be_opts.destroy_old_data = false;
std::mt19937 rng{std::random_device()()};
std::array<std::thread, 4> read_threads;
std::array<std::thread, 4> restore_verify_threads;
for (uint32_t i = 0; i < read_threads.size(); ++i) {
uint32_t sleep_micros = rng() % 100000;
read_threads[i] = std::thread([this, i, sleep_micros, &db_opts, &be_opts] {
read_threads[i] = std::thread(
[this, i, sleep_micros, &db_opts, &be_opts, &restore_verify_threads] {
test_db_env_->SleepForMicroseconds(sleep_micros);
// Whether to also re-open the BackupEngine, potentially seeing
@ -3163,28 +3179,51 @@ TEST_F(BackupEngineTest, Concurrency) {
my_be->GetCorruptedBackups(&ids);
ASSERT_EQ(ids.size(), 0U);
// Restore one of the backups, or "latest"
// (Eventually, see below) Restore one of the backups, or "latest"
std::string restore_db_dir = dbname_ + "/restore" + ToString(i);
DestroyDir(test_db_env_.get(), restore_db_dir).PermitUncheckedError();
BackupID to_restore;
if (latest) {
to_restore = count;
ASSERT_OK(
my_be->RestoreDBFromLatestBackup(restore_db_dir, restore_db_dir));
} else {
to_restore = i + 1;
ASSERT_OK(my_be->VerifyBackup(to_restore, true));
ASSERT_OK(my_be->RestoreDBFromBackup(to_restore, restore_db_dir,
restore_db_dir));
}
// Open restored DB to verify its contents
// Open restored DB to verify its contents, but test atomic restore
// by doing it async and ensuring we either get OK or InvalidArgument
restore_verify_threads[i] =
std::thread([this, &db_opts, restore_db_dir, to_restore] {
DB* restored;
ASSERT_OK(DB::Open(db_opts, restore_db_dir, &restored));
Status s;
for (;;) {
s = DB::Open(db_opts, restore_db_dir, &restored);
if (s.IsInvalidArgument()) {
// Restore hasn't finished
test_db_env_->SleepForMicroseconds(1000);
continue;
} else {
// We should only get InvalidArgument if restore is
// incomplete, or OK if complete
ASSERT_OK(s);
break;
}
}
int factor = std::min(static_cast<int>(to_restore), max_factor);
AssertExists(restored, 0, factor * keys_iteration);
AssertEmpty(restored, factor * keys_iteration,
(factor + 1) * keys_iteration);
delete restored;
});
// (Ok now) Restore one of the backups, or "latest"
if (latest) {
ASSERT_OK(my_be->RestoreDBFromLatestBackup(restore_db_dir,
restore_db_dir));
} else {
ASSERT_OK(my_be->VerifyBackup(to_restore, true));
ASSERT_OK(my_be->RestoreDBFromBackup(to_restore, restore_db_dir,
restore_db_dir));
}
// Re-verify metadata (we don't receive updates from concurrently
// creating a new backup)
@ -3229,6 +3268,11 @@ TEST_F(BackupEngineTest, Concurrency) {
}
delete alt_be;
for (auto& t : restore_verify_threads) {
t.join();
}
CloseDBAndBackupEngine();
}