andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Make backups openable as read-only DBs (#8142)

Browse Source 
Summary:
A current limitation of backups is that you don't know the
exact database state of when the backup was taken. With this new
feature, you can at least inspect the backup's DB state without
restoring it by opening it as a read-only DB.

Rather than add something like OpenAsReadOnlyDB to the BackupEngine API,
which would inhibit opening stackable DB implementations read-only
(if/when their APIs support it), we instead provide a DB name and Env
that can be used to open as a read-only DB.

Possible follow-up work:

* Add a version of GetBackupInfo for a single backup.
* Let CreateNewBackup return the BackupID of the newly-created backup.

Implementation details:

Refactored ChrootFileSystem to split off new base class RemapFileSystem,
which allows more general remapping of files. We use this base class to
implement BackupEngineImpl::RemapSharedFileSystem.

To minimize API impact, I decided to just add these fields `name_for_open`
and `env_for_open` to those set by GetBackupInfo when
include_file_details=true. Creating the RemapSharedFileSystem adds a bit
to the memory consumption, perhaps unnecessarily in some cases, but this
has been mitigated by (a) only initialize the RemapSharedFileSystem
lazily when GetBackupInfo with include_file_details=true is called, and
(b) using the existing `shared_ptr<FileInfo>` objects to hold most of the
mapping data.

To enhance API safety, RemapSharedFileSystem is wrapped by new
ReadOnlyFileSystem which rejects any attempts to write. This uncovered a
couple of places in which DB::OpenForReadOnly would write to the
filesystem, so I fixed these. Added a release note because this affects
logging.

Additional minor refactoring in backupable_db.cc to support the new
functionality.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8142

Test Plan:
new test (run with ASAN and UBSAN), added to stress test and
ran it for a while with amplified backup_one_in

Reviewed By: ajkr

Differential Revision: D27535408

Pulled By: pdillinger

fbshipit-source-id: 04666d310aa0261ef6b2385c43ca793ce1dfd148
This commit is contained in:
Peter Dillinger 2021-04-06 14:36:45 -07:00 committed by Facebook GitHub Bot
parent 09528f9fa1
commit 879357fdb0
19 changed files with 908 additions and 360 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CMakeLists.txt
View File

@ -651,6 +651,7 @@ set(SOURCES
env/env_hdfs.cc env/env_hdfs.cc
env/file_system.cc env/file_system.cc
env/file_system_tracer.cc env/file_system_tracer.cc
env/fs_remap.cc
env/mock_env.cc env/mock_env.cc
file/delete_scheduler.cc file/delete_scheduler.cc
file/file_prefetch_buffer.cc file/file_prefetch_buffer.cc

4
HISTORY.md
View File

@ -3,6 +3,7 @@
### Behavior Changes ### Behavior Changes
* `ColumnFamilyOptions::sample_for_compression` now takes effect for creation of all block-based tables. Previously it only took effect for block-based tables created by flush. * `ColumnFamilyOptions::sample_for_compression` now takes effect for creation of all block-based tables. Previously it only took effect for block-based tables created by flush.
* `CompactFiles()` can no longer compact files from lower level to up level, which has the risk to corrupt DB (details: #8063). The validation is also added to all compactions. * `CompactFiles()` can no longer compact files from lower level to up level, which has the risk to corrupt DB (details: #8063). The validation is also added to all compactions.
* Fixed some cases in which DB::OpenForReadOnly() could write to the filesystem. If you want a Logger with a read-only DB, you must now set DBOptions::info_log yourself, such as using CreateLoggerFromOptions().
### Bug Fixes ### Bug Fixes
* Use thread-safe `strerror_r()` to get error messages. * Use thread-safe `strerror_r()` to get error messages.
@ -18,6 +19,9 @@
* Added `TableProperties::slow_compression_estimated_data_size` and `TableProperties::fast_compression_estimated_data_size`. When `ColumnFamilyOptions::sample_for_compression > 0`, they estimate what `TableProperties::data_size` would have been if the "fast" or "slow" (see `ColumnFamilyOptions::sample_for_compression` API doc for definitions) compression had been used instead. * Added `TableProperties::slow_compression_estimated_data_size` and `TableProperties::fast_compression_estimated_data_size`. When `ColumnFamilyOptions::sample_for_compression > 0`, they estimate what `TableProperties::data_size` would have been if the "fast" or "slow" (see `ColumnFamilyOptions::sample_for_compression` API doc for definitions) compression had been used instead.
* Update DB::StartIOTrace and remove Env object from the arguments as its redundant and DB already has Env object that is passed down to IOTracer::StartIOTrace * Update DB::StartIOTrace and remove Env object from the arguments as its redundant and DB already has Env object that is passed down to IOTracer::StartIOTrace
### New Features
* Added the ability to open BackupEngine backups as read-only DBs, using BackupInfo::name_for_open and env_for_open provided by BackupEngine::GetBackupInfo() with include_file_details=true.
## 6.19.0 (03/21/2021) ## 6.19.0 (03/21/2021)
### Bug Fixes ### Bug Fixes
* Fixed the truncation error found in APIs/tools when dumping block-based SST files in a human-readable format. After fix, the block-based table can be fully dumped as a readable file. * Fixed the truncation error found in APIs/tools when dumping block-based SST files in a human-readable format. After fix, the block-based table can be fully dumped as a readable file.

2
TARGETS
View File

@ -218,6 +218,7 @@ cpp_library(
"env/file_system.cc", "env/file_system.cc",
"env/file_system_tracer.cc", "env/file_system_tracer.cc",
"env/fs_posix.cc", "env/fs_posix.cc",
"env/fs_remap.cc",
"env/io_posix.cc", "env/io_posix.cc",
"env/mock_env.cc", "env/mock_env.cc",
"file/delete_scheduler.cc", "file/delete_scheduler.cc",
@ -526,6 +527,7 @@ cpp_library(
"env/file_system.cc", "env/file_system.cc",
"env/file_system_tracer.cc", "env/file_system_tracer.cc",
"env/fs_posix.cc", "env/fs_posix.cc",
"env/fs_remap.cc",
"env/io_posix.cc", "env/io_posix.cc",
"env/mock_env.cc", "env/mock_env.cc",
"file/delete_scheduler.cc", "file/delete_scheduler.cc",

12
db/db_impl/compacted_db_impl.cc
View File

@ -17,11 +17,13 @@ extern void MarkKeyMayExist(void* arg);
extern bool SaveValue(void* arg, const ParsedInternalKey& parsed_key, extern bool SaveValue(void* arg, const ParsedInternalKey& parsed_key,
const Slice& v, bool hit_and_return); const Slice& v, bool hit_and_return);
CompactedDBImpl::CompactedDBImpl( CompactedDBImpl::CompactedDBImpl(const DBOptions& options,
const DBOptions& options, const std::string& dbname) const std::string& dbname)
: DBImpl(options, dbname), cfd_(nullptr), version_(nullptr), : DBImpl(options, dbname, /*seq_per_batch*/ false, +/*batch_per_txn*/ true,
user_comparator_(nullptr) { /*read_only*/ true),
} cfd_(nullptr),
version_(nullptr),
user_comparator_(nullptr) {}
CompactedDBImpl::~CompactedDBImpl() { CompactedDBImpl::~CompactedDBImpl() {
} }

5
db/db_impl/db_impl.cc
View File

@ -146,10 +146,11 @@ void DumpSupportInfo(Logger* logger) {
} // namespace } // namespace
DBImpl::DBImpl(const DBOptions& options, const std::string& dbname, DBImpl::DBImpl(const DBOptions& options, const std::string& dbname,
const bool seq_per_batch, const bool batch_per_txn) const bool seq_per_batch, const bool batch_per_txn,
bool read_only)
: dbname_(dbname), : dbname_(dbname),
own_info_log_(options.info_log == nullptr), own_info_log_(options.info_log == nullptr),
initial_db_options_(SanitizeOptions(dbname, options)), initial_db_options_(SanitizeOptions(dbname, options, read_only)),
env_(initial_db_options_.env), env_(initial_db_options_.env),
io_tracer_(std::make_shared<IOTracer>()), io_tracer_(std::make_shared<IOTracer>()),
immutable_db_options_(initial_db_options_), immutable_db_options_(initial_db_options_),

11
db/db_impl/db_impl.h
View File

@ -129,7 +129,8 @@ class Directories {
class DBImpl : public DB { class DBImpl : public DB {
public: public:
DBImpl(const DBOptions& options, const std::string& dbname, DBImpl(const DBOptions& options, const std::string& dbname,
const bool seq_per_batch = false, const bool batch_per_txn = true); const bool seq_per_batch = false, const bool batch_per_txn = true,
bool read_only = false);
// No copying allowed // No copying allowed
DBImpl(const DBImpl&) = delete; DBImpl(const DBImpl&) = delete;
void operator=(const DBImpl&) = delete; void operator=(const DBImpl&) = delete;
@ -1236,7 +1237,7 @@ class DBImpl : public DB {
virtual bool OwnTablesAndLogs() const { return true; } virtual bool OwnTablesAndLogs() const { return true; }
// Set DB identity file, and write DB ID to manifest if necessary. // Set DB identity file, and write DB ID to manifest if necessary.
Status SetDBId(); Status SetDBId(bool read_only);
// REQUIRES: db mutex held when calling this function, but the db mutex can // REQUIRES: db mutex held when calling this function, but the db mutex can
// be released and re-acquired. Db mutex will be held when the function // be released and re-acquired. Db mutex will be held when the function
@ -2231,9 +2232,11 @@ class DBImpl : public DB {
BlobFileCompletionCallback blob_callback_; BlobFileCompletionCallback blob_callback_;
}; };
extern Options SanitizeOptions(const std::string& db, const Options& src); extern Options SanitizeOptions(const std::string& db, const Options& src,
bool read_only = false);
extern DBOptions SanitizeOptions(const std::string& db, const DBOptions& src); extern DBOptions SanitizeOptions(const std::string& db, const DBOptions& src,
bool read_only = false);
extern CompressionType GetCompressionFlush( extern CompressionType GetCompressionFlush(
const ImmutableCFOptions& ioptions, const ImmutableCFOptions& ioptions,

14
db/db_impl/db_impl_files.cc
View File

@ -854,7 +854,7 @@ uint64_t PrecomputeMinLogNumberToKeep2PC(
return min_log_number_to_keep; return min_log_number_to_keep;
} }
Status DBImpl::SetDBId() { Status DBImpl::SetDBId(bool read_only) {
Status s; Status s;
// Happens when immutable_db_options_.write_dbid_to_manifest is set to true // Happens when immutable_db_options_.write_dbid_to_manifest is set to true
// the very first time. // the very first time.
@ -865,9 +865,15 @@ Status DBImpl::SetDBId() {
// it is no longer available then at this point DB ID is not in Identity // it is no longer available then at this point DB ID is not in Identity
// file or Manifest. // file or Manifest.
if (s.IsNotFound()) { if (s.IsNotFound()) {
s = SetIdentityFile(env_, dbname_); // Create a new DB ID, saving to file only if allowed
if (!s.ok()) { if (read_only) {
return s; db_id_ = env_->GenerateUniqueId();
return Status::OK();
} else {
s = SetIdentityFile(env_, dbname_);
if (!s.ok()) {
return s;
}
} }
} else if (!s.ok()) { } else if (!s.ok()) {
assert(s.IsIOError()); assert(s.IsIOError());

12
db/db_impl/db_impl_open.cc
View File

@ -24,15 +24,17 @@
#include "util/rate_limiter.h" #include "util/rate_limiter.h"
namespace ROCKSDB_NAMESPACE { namespace ROCKSDB_NAMESPACE {
Options SanitizeOptions(const std::string& dbname, const Options& src) { Options SanitizeOptions(const std::string& dbname, const Options& src,
auto db_options = SanitizeOptions(dbname, DBOptions(src)); bool read_only) {
auto db_options = SanitizeOptions(dbname, DBOptions(src), read_only);
ImmutableDBOptions immutable_db_options(db_options); ImmutableDBOptions immutable_db_options(db_options);
auto cf_options = auto cf_options =
SanitizeOptions(immutable_db_options, ColumnFamilyOptions(src)); SanitizeOptions(immutable_db_options, ColumnFamilyOptions(src));
return Options(db_options, cf_options); return Options(db_options, cf_options);
} }
DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) { DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src,
bool read_only) {
DBOptions result(src); DBOptions result(src);
if (result.env == nullptr) { if (result.env == nullptr) {
@ -50,7 +52,7 @@ DBOptions SanitizeOptions(const std::string& dbname, const DBOptions& src) {
&result.max_open_files); &result.max_open_files);
} }
if (result.info_log == nullptr) { if (result.info_log == nullptr && !read_only) {
Status s = CreateLoggerFromOptions(dbname, result, &result.info_log); Status s = CreateLoggerFromOptions(dbname, result, &result.info_log);
if (!s.ok()) { if (!s.ok()) {
// No place suitable for logging // No place suitable for logging
@ -488,7 +490,7 @@ Status DBImpl::Recover(
if (!s.ok()) { if (!s.ok()) {
return s; return s;
} }
s = SetDBId(); s = SetDBId(read_only);
if (s.ok() && !read_only) { if (s.ok() && !read_only) {
s = DeleteUnreferencedSstFiles(); s = DeleteUnreferencedSstFiles();
} }

11
db/db_impl/db_impl_readonly.cc
View File

@ -19,7 +19,8 @@ namespace ROCKSDB_NAMESPACE {
DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options, DBImplReadOnly::DBImplReadOnly(const DBOptions& db_options,
const std::string& dbname) const std::string& dbname)
: DBImpl(db_options, dbname) { : DBImpl(db_options, dbname, /*seq_per_batch*/ false,
/*batch_per_txn*/ true, /*read_only*/ true) {
ROCKS_LOG_INFO(immutable_db_options_.info_log, ROCKS_LOG_INFO(immutable_db_options_.info_log,
"Opening the db in read only mode"); "Opening the db in read only mode");
LogFlush(immutable_db_options_.info_log); LogFlush(immutable_db_options_.info_log);
@ -131,8 +132,8 @@ Status DBImplReadOnly::NewIterators(
} }
namespace { namespace {
// Return OK if dbname exists in the file system // Return OK if dbname exists in the file system or create it if
// or create_if_missing is false // create_if_missing
Status OpenForReadOnlyCheckExistence(const DBOptions& db_options, Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
const std::string& dbname) { const std::string& dbname) {
Status s; Status s;
@ -143,6 +144,9 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
uint64_t manifest_file_number; uint64_t manifest_file_number;
s = VersionSet::GetCurrentManifestPath(dbname, fs.get(), &manifest_path, s = VersionSet::GetCurrentManifestPath(dbname, fs.get(), &manifest_path,
&manifest_file_number); &manifest_file_number);
} else {
// Historic behavior that doesn't necessarily make sense
s = db_options.env->CreateDirIfMissing(dbname);
} }
return s; return s;
} }
@ -150,7 +154,6 @@ Status OpenForReadOnlyCheckExistence(const DBOptions& db_options,
Status DB::OpenForReadOnly(const Options& options, const std::string& dbname, Status DB::OpenForReadOnly(const Options& options, const std::string& dbname,
DB** dbptr, bool /*error_if_wal_file_exists*/) { DB** dbptr, bool /*error_if_wal_file_exists*/) {
// If dbname does not exist in the file system, should not do anything
Status s = OpenForReadOnlyCheckExistence(options, dbname); Status s = OpenForReadOnlyCheckExistence(options, dbname);
if (!s.ok()) { if (!s.ok()) {
return s; return s;

54
db_stress_tool/db_stress_test_base.cc
View File

@ -1368,8 +1368,13 @@ Status StressTest::TestBackupRestore(
} }
} }
std::vector<BackupInfo> backup_info; std::vector<BackupInfo> backup_info;
// If inplace_not_restore, we verify the backup by opening it as a
// read-only DB. If !inplace_not_restore, we restore it to a temporary
// directory for verification.
bool inplace_not_restore = thread->rand.OneIn(3);
if (s.ok()) { if (s.ok()) {
backup_engine->GetBackupInfo(&backup_info); backup_engine->GetBackupInfo(&backup_info,
/*include_file_details*/ inplace_not_restore);
if (backup_info.empty()) { if (backup_info.empty()) {
s = Status::NotFound("no backups found"); s = Status::NotFound("no backups found");
from = "BackupEngine::GetBackupInfo"; from = "BackupEngine::GetBackupInfo";
@ -1385,8 +1390,8 @@ Status StressTest::TestBackupRestore(
} }
const bool allow_persistent = thread->tid == 0; // not too many const bool allow_persistent = thread->tid == 0; // not too many
bool from_latest = false; bool from_latest = false;
if (s.ok()) { int count = static_cast<int>(backup_info.size());
int count = static_cast<int>(backup_info.size()); if (s.ok() && !inplace_not_restore) {
if (count > 1) { if (count > 1) {
s = backup_engine->RestoreDBFromBackup( s = backup_engine->RestoreDBFromBackup(
RestoreOptions(), backup_info[thread->rand.Uniform(count)].backup_id, RestoreOptions(), backup_info[thread->rand.Uniform(count)].backup_id,
@ -1404,7 +1409,9 @@ Status StressTest::TestBackupRestore(
} }
} }
} }
if (s.ok()) { if (s.ok() && !inplace_not_restore) {
// Purge early if restoring, to ensure the restored directory doesn't
// have some secret dependency on the backup directory.
uint32_t to_keep = 0; uint32_t to_keep = 0;
if (allow_persistent) { if (allow_persistent) {
// allow one thread to keep up to 2 backups // allow one thread to keep up to 2 backups
@ -1432,10 +1439,21 @@ Status StressTest::TestBackupRestore(
for (auto name : column_family_names_) { for (auto name : column_family_names_) {
cf_descriptors.emplace_back(name, ColumnFamilyOptions(restore_options)); cf_descriptors.emplace_back(name, ColumnFamilyOptions(restore_options));
} }
s = DB::Open(DBOptions(restore_options), restore_dir, cf_descriptors, if (inplace_not_restore) {
&restored_cf_handles, &restored_db); BackupInfo& info = backup_info[thread->rand.Uniform(count)];
if (!s.ok()) { restore_options.env = info.env_for_open.get();
from = "DB::Open in backup/restore"; s = DB::OpenForReadOnly(DBOptions(restore_options), info.name_for_open,
cf_descriptors, &restored_cf_handles,
&restored_db);
if (!s.ok()) {
from = "DB::OpenForReadOnly in backup/restore";
}
} else {
s = DB::Open(DBOptions(restore_options), restore_dir, cf_descriptors,
&restored_cf_handles, &restored_db);
if (!s.ok()) {
from = "DB::Open in backup/restore";
}
} }
} }
// Note the column families chosen by `rand_column_families` cannot be // Note the column families chosen by `rand_column_families` cannot be
@ -1476,10 +1494,6 @@ Status StressTest::TestBackupRestore(
} }
} }
} }
if (backup_engine != nullptr) {
delete backup_engine;
backup_engine = nullptr;
}
if (restored_db != nullptr) { if (restored_db != nullptr) {
for (auto* cf_handle : restored_cf_handles) { for (auto* cf_handle : restored_cf_handles) {
restored_db->DestroyColumnFamilyHandle(cf_handle); restored_db->DestroyColumnFamilyHandle(cf_handle);
@ -1487,6 +1501,22 @@ Status StressTest::TestBackupRestore(
delete restored_db; delete restored_db;
restored_db = nullptr; restored_db = nullptr;
} }
if (s.ok() && inplace_not_restore) {
// Purge late if inplace open read-only
uint32_t to_keep = 0;
if (allow_persistent) {
// allow one thread to keep up to 2 backups
to_keep = thread->rand.Uniform(3);
}
s = backup_engine->PurgeOldBackups(to_keep);
if (!s.ok()) {
from = "BackupEngine::PurgeOldBackups";
}
}
if (backup_engine != nullptr) {
delete backup_engine;
backup_engine = nullptr;
}
if (s.ok()) { if (s.ok()) {
// Preserve directories on failure, or allowed persistent backup // Preserve directories on failure, or allowed persistent backup
if (!allow_persistent) { if (!allow_persistent) {

289
env/env_chroot.cc vendored
View File

@ -7,27 +7,21 @@
#include "env/env_chroot.h" #include "env/env_chroot.h"
#include <errno.h> #include <errno.h> // errno
#include <stdlib.h> #include <stdlib.h> // realpath, free
#include <string.h> #include <unistd.h> // geteuid
#include <unistd.h>
#include <string>
#include <utility>
#include <vector>
#include "env/composite_env_wrapper.h" #include "env/composite_env_wrapper.h"
#include "rocksdb/file_system.h" #include "env/fs_remap.h"
#include "rocksdb/status.h" #include "util/string_util.h" // errnoStr
#include "util/string_util.h"
namespace ROCKSDB_NAMESPACE { namespace ROCKSDB_NAMESPACE {
namespace { namespace {
class ChrootFileSystem : public FileSystemWrapper { class ChrootFileSystem : public RemapFileSystem {
public: public:
ChrootFileSystem(const std::shared_ptr<FileSystem>& base, ChrootFileSystem(const std::shared_ptr<FileSystem>& base,
const std::string& chroot_dir) const std::string& chroot_dir)
: FileSystemWrapper(base) { : RemapFileSystem(base) {
#if defined(OS_AIX) #if defined(OS_AIX)
char resolvedName[PATH_MAX]; char resolvedName[PATH_MAX];
char* real_chroot_dir = realpath(chroot_dir.c_str(), resolvedName); char* real_chroot_dir = realpath(chroot_dir.c_str(), resolvedName);
@ -43,245 +37,6 @@ class ChrootFileSystem : public FileSystemWrapper {
} }
const char* Name() const override { return "ChrootFS"; } const char* Name() const override { return "ChrootFS"; }
Status RegisterDbPaths(const std::vector<std::string>& paths) override {
std::vector<std::string> encoded_paths;
encoded_paths.reserve(paths.size());
for (auto& path : paths) {
auto status_and_enc_path = EncodePathWithNewBasename(path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
encoded_paths.emplace_back(status_and_enc_path.second);
}
return FileSystemWrapper::RegisterDbPaths(encoded_paths);
}
Status UnregisterDbPaths(const std::vector<std::string>& paths) override {
std::vector<std::string> encoded_paths;
encoded_paths.reserve(paths.size());
for (auto& path : paths) {
auto status_and_enc_path = EncodePathWithNewBasename(path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
encoded_paths.emplace_back(status_and_enc_path.second);
}
return FileSystemWrapper::UnregisterDbPaths(encoded_paths);
}
IOStatus NewSequentialFile(const std::string& fname,
const FileOptions& options,
std::unique_ptr<FSSequentialFile>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewSequentialFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& options,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewRandomAccessFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus NewWritableFile(const std::string& fname, const FileOptions& options,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewWritableFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus ReuseWritableFile(const std::string& fname,
const std::string& old_fname,
const FileOptions& options,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
auto status_and_old_enc_path = EncodePath(old_fname);
if (!status_and_old_enc_path.first.ok()) {
return status_and_old_enc_path.first;
}
return FileSystemWrapper::ReuseWritableFile(status_and_old_enc_path.second,
status_and_old_enc_path.second,
options, result, dbg);
}
IOStatus NewRandomRWFile(const std::string& fname, const FileOptions& options,
std::unique_ptr<FSRandomRWFile>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewRandomRWFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus NewDirectory(const std::string& dir, const IOOptions& options,
std::unique_ptr<FSDirectory>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewDirectory(status_and_enc_path.second, options,
result, dbg);
}
IOStatus FileExists(const std::string& fname, const IOOptions& options,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::FileExists(status_and_enc_path.second, options,
dbg);
}
IOStatus GetChildren(const std::string& dir, const IOOptions& options,
std::vector<std::string>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetChildren(status_and_enc_path.second, options,
result, dbg);
}
IOStatus GetChildrenFileAttributes(const std::string& dir,
const IOOptions& options,
std::vector<FileAttributes>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetChildrenFileAttributes(
status_and_enc_path.second, options, result, dbg);
}
IOStatus DeleteFile(const std::string& fname, const IOOptions& options,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::DeleteFile(status_and_enc_path.second, options,
dbg);
}
IOStatus CreateDir(const std::string& dirname, const IOOptions& options,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::CreateDir(status_and_enc_path.second, options,
dbg);
}
IOStatus CreateDirIfMissing(const std::string& dirname,
const IOOptions& options,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::CreateDirIfMissing(status_and_enc_path.second,
options, dbg);
}
IOStatus DeleteDir(const std::string& dirname, const IOOptions& options,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::DeleteDir(status_and_enc_path.second, options,
dbg);
}
IOStatus GetFileSize(const std::string& fname, const IOOptions& options,
uint64_t* file_size, IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetFileSize(status_and_enc_path.second, options,
file_size, dbg);
}
IOStatus GetFileModificationTime(const std::string& fname,
const IOOptions& options,
uint64_t* file_mtime,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetFileModificationTime(
status_and_enc_path.second, options, file_mtime, dbg);
}
IOStatus RenameFile(const std::string& src, const std::string& dest,
const IOOptions& options, IODebugContext* dbg) override {
auto status_and_src_enc_path = EncodePath(src);
if (!status_and_src_enc_path.first.ok()) {
return status_and_src_enc_path.first;
}
auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
if (!status_and_dest_enc_path.first.ok()) {
return status_and_dest_enc_path.first;
}
return FileSystemWrapper::RenameFile(status_and_src_enc_path.second,
status_and_dest_enc_path.second,
options, dbg);
}
IOStatus LinkFile(const std::string& src, const std::string& dest,
const IOOptions& options, IODebugContext* dbg) override {
auto status_and_src_enc_path = EncodePath(src);
if (!status_and_src_enc_path.first.ok()) {
return status_and_src_enc_path.first;
}
auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
if (!status_and_dest_enc_path.first.ok()) {
return status_and_dest_enc_path.first;
}
return FileSystemWrapper::LinkFile(status_and_src_enc_path.second,
status_and_dest_enc_path.second, options,
dbg);
}
IOStatus LockFile(const std::string& fname, const IOOptions& options,
FileLock** lock, IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
// FileLock subclasses may store path (e.g., PosixFileLock stores it). We
// can skip stripping the chroot directory from this path because callers
// shouldn't use it.
return FileSystemWrapper::LockFile(status_and_enc_path.second, options,
lock, dbg);
}
IOStatus GetTestDirectory(const IOOptions& options, std::string* path, IOStatus GetTestDirectory(const IOOptions& options, std::string* path,
IODebugContext* dbg) override { IODebugContext* dbg) override {
@ -295,33 +50,12 @@ class ChrootFileSystem : public FileSystemWrapper {
return CreateDirIfMissing(*path, options, dbg); return CreateDirIfMissing(*path, options, dbg);
} }
IOStatus NewLogger(const std::string& fname, const IOOptions& options, protected:
std::shared_ptr<Logger>* result,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewLogger(status_and_enc_path.second, options,
result, dbg);
}
IOStatus GetAbsolutePath(const std::string& db_path, const IOOptions& options,
std::string* output_path,
IODebugContext* dbg) override {
auto status_and_enc_path = EncodePath(db_path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetAbsolutePath(status_and_enc_path.second,
options, output_path, dbg);
}
private:
// Returns status and expanded absolute path including the chroot directory. // Returns status and expanded absolute path including the chroot directory.
// Checks whether the provided path breaks out of the chroot. If it returns // Checks whether the provided path breaks out of the chroot. If it returns
// non-OK status, the returned path should not be used. // non-OK status, the returned path should not be used.
std::pair<IOStatus, std::string> EncodePath(const std::string& path) { std::pair<IOStatus, std::string> EncodePath(
const std::string& path) override {
if (path.empty() || path[0] != '/') { if (path.empty() || path[0] != '/') {
return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""}; return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""};
} }
@ -352,7 +86,7 @@ class ChrootFileSystem : public FileSystemWrapper {
// Similar to EncodePath() except assumes the basename in the path hasn't been // Similar to EncodePath() except assumes the basename in the path hasn't been
// created yet. // created yet.
std::pair<IOStatus, std::string> EncodePathWithNewBasename( std::pair<IOStatus, std::string> EncodePathWithNewBasename(
const std::string& path) { const std::string& path) override {
if (path.empty() || path[0] != '/') { if (path.empty() || path[0] != '/') {
return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""}; return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""};
} }
@ -371,6 +105,7 @@ class ChrootFileSystem : public FileSystemWrapper {
return status_and_enc_path; return status_and_enc_path;
} }
private:
std::string chroot_dir_; std::string chroot_dir_;
}; };
} // namespace } // namespace

3
env/env_chroot.h vendored
View File

@ -15,6 +15,9 @@ namespace ROCKSDB_NAMESPACE {
// Returns an Env that translates paths such that the root directory appears to // Returns an Env that translates paths such that the root directory appears to
// be chroot_dir. chroot_dir should refer to an existing directory. // be chroot_dir. chroot_dir should refer to an existing directory.
//
// This class has not been fully analyzed for providing strong security
// guarantees.
Env* NewChrootEnv(Env* base_env, const std::string& chroot_dir); Env* NewChrootEnv(Env* base_env, const std::string& chroot_dir);
} // namespace ROCKSDB_NAMESPACE } // namespace ROCKSDB_NAMESPACE

97
env/fs_readonly.h vendored Normal file
View File

@ -0,0 +1,97 @@
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#ifndef ROCKSDB_LITE
#include "rocksdb/file_system.h"
namespace ROCKSDB_NAMESPACE {
// A FileSystem wrapper that only allows read-only operation.
//
// This class has not been fully analyzed for providing strong security
// guarantees.
class ReadOnlyFileSystem : public FileSystemWrapper {
static inline IOStatus FailReadOnly() {
IOStatus s = IOStatus::IOError("Attempted write to ReadOnlyFileSystem");
assert(s.GetRetryable() == false);
return s;
}
public:
ReadOnlyFileSystem(const std::shared_ptr<FileSystem>& base)
: FileSystemWrapper(base) {}
IOStatus NewWritableFile(const std::string& /*fname*/,
const FileOptions& /*options*/,
std::unique_ptr<FSWritableFile>* /*result*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus ReuseWritableFile(const std::string& /*fname*/,
const std::string& /*old_fname*/,
const FileOptions& /*options*/,
std::unique_ptr<FSWritableFile>* /*result*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus NewRandomRWFile(const std::string& /*fname*/,
const FileOptions& /*options*/,
std::unique_ptr<FSRandomRWFile>* /*result*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus NewDirectory(const std::string& /*dir*/,
const IOOptions& /*options*/,
std::unique_ptr<FSDirectory>* /*result*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus DeleteFile(const std::string& /*fname*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus CreateDir(const std::string& /*dirname*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus CreateDirIfMissing(const std::string& /*dirname*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus DeleteDir(const std::string& /*dirname*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus RenameFile(const std::string& /*src*/, const std::string& /*dest*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus LinkFile(const std::string& /*src*/, const std::string& /*dest*/,
const IOOptions& /*options*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus LockFile(const std::string& /*fname*/, const IOOptions& /*options*/,
FileLock** /*lock*/, IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
IOStatus NewLogger(const std::string& /*fname*/, const IOOptions& /*options*/,
std::shared_ptr<Logger>* /*result*/,
IODebugContext* /*dbg*/) override {
return FailReadOnly();
}
};
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE

306
env/fs_remap.cc vendored Normal file
View File

@ -0,0 +1,306 @@
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#ifndef ROCKSDB_LITE
#include "env/fs_remap.h"
namespace ROCKSDB_NAMESPACE {
RemapFileSystem::RemapFileSystem(const std::shared_ptr<FileSystem>& base)
: FileSystemWrapper(base) {}
std::pair<IOStatus, std::string> RemapFileSystem::EncodePathWithNewBasename(
const std::string& path) {
// No difference by default
return EncodePath(path);
}
Status RemapFileSystem::RegisterDbPaths(const std::vector<std::string>& paths) {
std::vector<std::string> encoded_paths;
encoded_paths.reserve(paths.size());
for (auto& path : paths) {
auto status_and_enc_path = EncodePathWithNewBasename(path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
encoded_paths.emplace_back(status_and_enc_path.second);
}
return FileSystemWrapper::RegisterDbPaths(encoded_paths);
}
Status RemapFileSystem::UnregisterDbPaths(
const std::vector<std::string>& paths) {
std::vector<std::string> encoded_paths;
encoded_paths.reserve(paths.size());
for (auto& path : paths) {
auto status_and_enc_path = EncodePathWithNewBasename(path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
encoded_paths.emplace_back(status_and_enc_path.second);
}
return FileSystemWrapper::UnregisterDbPaths(encoded_paths);
}
IOStatus RemapFileSystem::NewSequentialFile(
const std::string& fname, const FileOptions& options,
std::unique_ptr<FSSequentialFile>* result, IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewSequentialFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus RemapFileSystem::NewRandomAccessFile(
const std::string& fname, const FileOptions& options,
std::unique_ptr<FSRandomAccessFile>* result, IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewRandomAccessFile(status_and_enc_path.second,
options, result, dbg);
}
IOStatus RemapFileSystem::NewWritableFile(
const std::string& fname, const FileOptions& options,
std::unique_ptr<FSWritableFile>* result, IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewWritableFile(status_and_enc_path.second, options,
result, dbg);
}
IOStatus RemapFileSystem::ReuseWritableFile(
const std::string& fname, const std::string& old_fname,
const FileOptions& options, std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
auto status_and_old_enc_path = EncodePath(old_fname);
if (!status_and_old_enc_path.first.ok()) {
return status_and_old_enc_path.first;
}
return FileSystemWrapper::ReuseWritableFile(status_and_old_enc_path.second,
status_and_old_enc_path.second,
options, result, dbg);
}
IOStatus RemapFileSystem::NewRandomRWFile(
const std::string& fname, const FileOptions& options,
std::unique_ptr<FSRandomRWFile>* result, IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewRandomRWFile(status_and_enc_path.second, options,
result, dbg);
}
IOStatus RemapFileSystem::NewDirectory(const std::string& dir,
const IOOptions& options,
std::unique_ptr<FSDirectory>* result,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewDirectory(status_and_enc_path.second, options,
result, dbg);
}
IOStatus RemapFileSystem::FileExists(const std::string& fname,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::FileExists(status_and_enc_path.second, options,
dbg);
}
IOStatus RemapFileSystem::GetChildren(const std::string& dir,
const IOOptions& options,
std::vector<std::string>* result,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetChildren(status_and_enc_path.second, options,
result, dbg);
}
IOStatus RemapFileSystem::GetChildrenFileAttributes(
const std::string& dir, const IOOptions& options,
std::vector<FileAttributes>* result, IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(dir);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetChildrenFileAttributes(
status_and_enc_path.second, options, result, dbg);
}
IOStatus RemapFileSystem::DeleteFile(const std::string& fname,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::DeleteFile(status_and_enc_path.second, options,
dbg);
}
IOStatus RemapFileSystem::CreateDir(const std::string& dirname,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::CreateDir(status_and_enc_path.second, options, dbg);
}
IOStatus RemapFileSystem::CreateDirIfMissing(const std::string& dirname,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::CreateDirIfMissing(status_and_enc_path.second,
options, dbg);
}
IOStatus RemapFileSystem::DeleteDir(const std::string& dirname,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(dirname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::DeleteDir(status_and_enc_path.second, options, dbg);
}
IOStatus RemapFileSystem::GetFileSize(const std::string& fname,
const IOOptions& options,
uint64_t* file_size,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetFileSize(status_and_enc_path.second, options,
file_size, dbg);
}
IOStatus RemapFileSystem::GetFileModificationTime(const std::string& fname,
const IOOptions& options,
uint64_t* file_mtime,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetFileModificationTime(status_and_enc_path.second,
options, file_mtime, dbg);
}
IOStatus RemapFileSystem::IsDirectory(const std::string& path,
const IOOptions& options, bool* is_dir,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::IsDirectory(status_and_enc_path.second, options,
is_dir, dbg);
}
IOStatus RemapFileSystem::RenameFile(const std::string& src,
const std::string& dest,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_src_enc_path = EncodePath(src);
if (!status_and_src_enc_path.first.ok()) {
return status_and_src_enc_path.first;
}
auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
if (!status_and_dest_enc_path.first.ok()) {
return status_and_dest_enc_path.first;
}
return FileSystemWrapper::RenameFile(status_and_src_enc_path.second,
status_and_dest_enc_path.second, options,
dbg);
}
IOStatus RemapFileSystem::LinkFile(const std::string& src,
const std::string& dest,
const IOOptions& options,
IODebugContext* dbg) {
auto status_and_src_enc_path = EncodePath(src);
if (!status_and_src_enc_path.first.ok()) {
return status_and_src_enc_path.first;
}
auto status_and_dest_enc_path = EncodePathWithNewBasename(dest);
if (!status_and_dest_enc_path.first.ok()) {
return status_and_dest_enc_path.first;
}
return FileSystemWrapper::LinkFile(status_and_src_enc_path.second,
status_and_dest_enc_path.second, options,
dbg);
}
IOStatus RemapFileSystem::LockFile(const std::string& fname,
const IOOptions& options, FileLock** lock,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
// FileLock subclasses may store path (e.g., PosixFileLock stores it). We
// can skip stripping the chroot directory from this path because callers
// shouldn't use it.
return FileSystemWrapper::LockFile(status_and_enc_path.second, options, lock,
dbg);
}
IOStatus RemapFileSystem::NewLogger(const std::string& fname,
const IOOptions& options,
std::shared_ptr<Logger>* result,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePathWithNewBasename(fname);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::NewLogger(status_and_enc_path.second, options,
result, dbg);
}
IOStatus RemapFileSystem::GetAbsolutePath(const std::string& db_path,
const IOOptions& options,
std::string* output_path,
IODebugContext* dbg) {
auto status_and_enc_path = EncodePath(db_path);
if (!status_and_enc_path.first.ok()) {
return status_and_enc_path.first;
}
return FileSystemWrapper::GetAbsolutePath(status_and_enc_path.second, options,
output_path, dbg);
}
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE

131
env/fs_remap.h vendored Normal file
View File

@ -0,0 +1,131 @@
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#ifndef ROCKSDB_LITE
#include <utility>
#include "rocksdb/file_system.h"
namespace ROCKSDB_NAMESPACE {
// An abstract FileSystem wrapper that creates a view of an existing
// FileSystem by remapping names in some way.
//
// This class has not been fully analyzed for providing strong security
// guarantees.
class RemapFileSystem : public FileSystemWrapper {
public:
RemapFileSystem(const std::shared_ptr<FileSystem>& base);
protected:
// Returns status and mapped-to path in the wrapped filesystem.
// If it returns non-OK status, the returned path should not be used.
virtual std::pair<IOStatus, std::string> EncodePath(
const std::string& path) = 0;
// Similar to EncodePath() except used in cases in which it is OK for
// no file or directory on 'path' to already exist, such as if the
// operation would create one. However, the parent of 'path' is expected
// to exist for the operation to succeed.
// Default implementation: call EncodePath
virtual std::pair<IOStatus, std::string> EncodePathWithNewBasename(
const std::string& path);
public:
// Left abstract:
// const char* Name() const override { ... }
Status RegisterDbPaths(const std::vector<std::string>& paths) override;
Status UnregisterDbPaths(const std::vector<std::string>& paths) override;
IOStatus NewSequentialFile(const std::string& fname,
const FileOptions& options,
std::unique_ptr<FSSequentialFile>* result,
IODebugContext* dbg) override;
IOStatus NewRandomAccessFile(const std::string& fname,
const FileOptions& options,
std::unique_ptr<FSRandomAccessFile>* result,
IODebugContext* dbg) override;
IOStatus NewWritableFile(const std::string& fname, const FileOptions& options,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override;
IOStatus ReuseWritableFile(const std::string& fname,
const std::string& old_fname,
const FileOptions& options,
std::unique_ptr<FSWritableFile>* result,
IODebugContext* dbg) override;
IOStatus NewRandomRWFile(const std::string& fname, const FileOptions& options,
std::unique_ptr<FSRandomRWFile>* result,
IODebugContext* dbg) override;
IOStatus NewDirectory(const std::string& dir, const IOOptions& options,
std::unique_ptr<FSDirectory>* result,
IODebugContext* dbg) override;
IOStatus FileExists(const std::string& fname, const IOOptions& options,
IODebugContext* dbg) override;
IOStatus GetChildren(const std::string& dir, const IOOptions& options,
std::vector<std::string>* result,
IODebugContext* dbg) override;
IOStatus GetChildrenFileAttributes(const std::string& dir,
const IOOptions& options,
std::vector<FileAttributes>* result,
IODebugContext* dbg) override;
IOStatus DeleteFile(const std::string& fname, const IOOptions& options,
IODebugContext* dbg) override;
IOStatus CreateDir(const std::string& dirname, const IOOptions& options,
IODebugContext* dbg) override;
IOStatus CreateDirIfMissing(const std::string& dirname,
const IOOptions& options,
IODebugContext* dbg) override;
IOStatus DeleteDir(const std::string& dirname, const IOOptions& options,
IODebugContext* dbg) override;
IOStatus GetFileSize(const std::string& fname, const IOOptions& options,
uint64_t* file_size, IODebugContext* dbg) override;
IOStatus GetFileModificationTime(const std::string& fname,
const IOOptions& options,
uint64_t* file_mtime,
IODebugContext* dbg) override;
IOStatus IsDirectory(const std::string& path, const IOOptions& options,
bool* is_dir, IODebugContext* dbg) override;
IOStatus RenameFile(const std::string& src, const std::string& dest,
const IOOptions& options, IODebugContext* dbg) override;
IOStatus LinkFile(const std::string& src, const std::string& dest,
const IOOptions& options, IODebugContext* dbg) override;
IOStatus LockFile(const std::string& fname, const IOOptions& options,
FileLock** lock, IODebugContext* dbg) override;
IOStatus NewLogger(const std::string& fname, const IOOptions& options,
std::shared_ptr<Logger>* result,
IODebugContext* dbg) override;
IOStatus GetAbsolutePath(const std::string& db_path, const IOOptions& options,
std::string* output_path,
IODebugContext* dbg) override;
};
} // namespace ROCKSDB_NAMESPACE
#endif // ROCKSDB_LITE

33
include/rocksdb/utilities/backupable_db.h
View File

@ -280,24 +280,43 @@ struct BackupFileInfo {
typedef uint32_t BackupID; typedef uint32_t BackupID;
struct BackupInfo { struct BackupInfo {
BackupID backup_id; BackupID backup_id = 0U;
// Creation time, according to GetCurrentTime // Creation time, according to GetCurrentTime
int64_t timestamp; int64_t timestamp = 0;
// Total size in bytes (based on file payloads, not including filesystem // Total size in bytes (based on file payloads, not including filesystem
// overheads or backup meta file) // overheads or backup meta file)
uint64_t size; uint64_t size = 0U;
// Number of backed up files, some of which might be shared with other // Number of backed up files, some of which might be shared with other
// backups. Does not include backup meta file. // backups. Does not include backup meta file.
uint32_t number_files; uint32_t number_files = 0U;
// Backup API user metadata // Backup API user metadata
std::string app_metadata; std::string app_metadata;
// Backup file details, if requested // Backup file details, if requested with include_file_details=true
std::vector<BackupFileInfo> file_details; std::vector<BackupFileInfo> file_details;
// DB "name" (a directory in the backup_env) for opening this backup as a
// read-only DB. This should also be used as the DBOptions::wal_dir, such
// as by default setting wal_dir="". See also env_for_open.
// This field is only set if include_file_details=true
std::string name_for_open;
// An Env(+FileSystem) for opening this backup as a read-only DB, with
// DB::OpenForReadOnly or similar. This field is only set if
// include_file_details=true. (The FileSystem in this Env takes care
// of making shared backup files openable from the `name_for_open` DB
// directory.) See also name_for_open.
//
// This Env might or might not be shared with other backups. To work
// around DBOptions::env being a raw pointer, this is a shared_ptr so
// that keeping either this BackupInfo, the BackupEngine, or a copy of
// this shared_ptr alive is sufficient to keep the Env alive for use by
// a read-only DB.
std::shared_ptr<Env> env_for_open;
BackupInfo() {} BackupInfo() {}
BackupInfo(BackupID _backup_id, int64_t _timestamp, uint64_t _size, BackupInfo(BackupID _backup_id, int64_t _timestamp, uint64_t _size,
@ -344,10 +363,8 @@ class BackupEngineReadOnlyBase {
virtual ~BackupEngineReadOnlyBase() {} virtual ~BackupEngineReadOnlyBase() {}
// Returns info about backups in backup_info // Returns info about backups in backup_info
// You can GetBackupInfo safely, even with other BackupEngine performing
// backups on the same directory.
// Setting include_file_details=true provides information about each // Setting include_file_details=true provides information about each
// backed-up file in BackupInfo::file_details. // backed-up file in BackupInfo::file_details and more.
virtual void GetBackupInfo(std::vector<BackupInfo>* backup_info, virtual void GetBackupInfo(std::vector<BackupInfo>* backup_info,
bool include_file_details = false) const = 0; bool include_file_details = false) const = 0;

1
src.mk
View File

@ -86,6 +86,7 @@ LIB_SOURCES = \
env/env_posix.cc \ env/env_posix.cc \
env/file_system.cc \ env/file_system.cc \
env/fs_posix.cc \ env/fs_posix.cc \
env/fs_remap.cc \
env/file_system_tracer.cc \ env/file_system_tracer.cc \
env/io_posix.cc \ env/io_posix.cc \
env/mock_env.cc \ env/mock_env.cc \

223
utilities/backupable/backupable_db.cc
View File

@ -27,6 +27,8 @@
#include <vector> #include <vector>
#include "env/composite_env_wrapper.h" #include "env/composite_env_wrapper.h"
#include "env/fs_readonly.h"
#include "env/fs_remap.h"
#include "file/filename.h" #include "file/filename.h"
#include "file/line_file_reader.h" #include "file/line_file_reader.h"
#include "file/sequence_file_reader.h" #include "file/sequence_file_reader.h"
@ -65,6 +67,16 @@ inline std::string ChecksumInt32ToHex(const uint32_t& checksum_value) {
PutFixed32(&checksum_str, EndianSwapValue(checksum_value)); PutFixed32(&checksum_str, EndianSwapValue(checksum_value));
return ChecksumStrToHex(checksum_str); return ChecksumStrToHex(checksum_str);
} }
const std::string kPrivateDirName = "private";
const std::string kMetaDirName = "meta";
const std::string kSharedDirName = "shared";
const std::string kSharedChecksumDirName = "shared_checksum";
const std::string kPrivateDirSlash = kPrivateDirName + "/";
const std::string kMetaDirSlash = kMetaDirName + "/";
const std::string kSharedDirSlash = kSharedDirName + "/";
const std::string kSharedChecksumDirSlash = kSharedChecksumDirName + "/";
} // namespace } // namespace
void BackupStatistics::IncrementNumberSuccessBackup() { void BackupStatistics::IncrementNumberSuccessBackup() {
@ -195,6 +207,133 @@ class BackupEngineImpl {
// db_session_id appears in the backup SST filename if the table naming // db_session_id appears in the backup SST filename if the table naming
// option is kUseDbSessionId // option is kUseDbSessionId
const std::string db_session_id; const std::string db_session_id;
std::string GetDbFileName() {
std::string rv;
// extract the filename part
size_t slash = filename.find_last_of('/');
// file will either be shared/<file>, shared_checksum/<file_crc32c_size>,
// shared_checksum/<file_session>, shared_checksum/<file_crc32c_session>,
// or private/<number>/<file>
assert(slash != std::string::npos);
rv = filename.substr(slash + 1);
// if the file was in shared_checksum, extract the real file name
// in this case the file is <number>_<checksum>_<size>.<type>,
// <number>_<session>.<type>, or <number>_<checksum>_<session>.<type>
if (filename.substr(0, slash) == kSharedChecksumDirName) {
rv = GetFileFromChecksumFile(rv);
}
return rv;
}
};
static inline std::string WithoutTrailingSlash(const std::string& path) {
if (path.empty() || path.back() != '/') {
return path;
} else {
return path.substr(path.size() - 1);
}
}
static inline std::string WithTrailingSlash(const std::string& path) {
if (path.empty() || path.back() != '/') {
return path + '/';
} else {
return path;
}
}
// A filesystem wrapper that makes shared backup files appear to be in the
// private backup directory (dst_dir), so that the private backup dir can
// be opened as a read-only DB.
class RemapSharedFileSystem : public RemapFileSystem {
public:
RemapSharedFileSystem(const std::shared_ptr<FileSystem>& base,
const std::string& dst_dir,
const std::string& src_base_dir,
const std::vector<std::shared_ptr<FileInfo>>& files)
: RemapFileSystem(base),
dst_dir_(WithoutTrailingSlash(dst_dir)),
dst_dir_slash_(WithTrailingSlash(dst_dir)),
src_base_dir_(WithTrailingSlash(src_base_dir)) {
for (auto& info : files) {
if (!StartsWith(info->filename, kPrivateDirSlash)) {
assert(StartsWith(info->filename, kSharedDirSlash) ||
StartsWith(info->filename, kSharedChecksumDirSlash));
remaps_[info->GetDbFileName()] = info;
}
}
}
const char* Name() const override {
return "BackupEngineImpl::RemapSharedFileSystem";
}
// Sometimes a directory listing is required in opening a DB
IOStatus GetChildren(const std::string& dir, const IOOptions& options,
std::vector<std::string>* result,
IODebugContext* dbg) override {
IOStatus s = RemapFileSystem::GetChildren(dir, options, result, dbg);
if (s.ok() && (dir == dst_dir_ || dir == dst_dir_slash_)) {
// Assume remapped files exist
for (auto& r : remaps_) {
result->push_back(r.first);
}
}
return s;
}
// Sometimes a directory listing is required in opening a DB
IOStatus GetChildrenFileAttributes(const std::string& dir,
const IOOptions& options,
std::vector<FileAttributes>* result,
IODebugContext* dbg) override {
IOStatus s =
RemapFileSystem::GetChildrenFileAttributes(dir, options, result, dbg);
if (s.ok() && (dir == dst_dir_ || dir == dst_dir_slash_)) {
// Assume remapped files exist with recorded size
for (auto& r : remaps_) {
result->emplace_back(); // clean up with C++20
FileAttributes& attr = result->back();
attr.name = r.first;
attr.size_bytes = r.second->size;
}
}
return s;
}
protected:
// When a file in dst_dir is requested, see if we need to remap to shared
// file path.
std::pair<IOStatus, std::string> EncodePath(
const std::string& path) override {
if (path.empty() || path[0] != '/') {
return {IOStatus::InvalidArgument(path, "Not an absolute path"), ""};
}
std::pair<IOStatus, std::string> rv{IOStatus(), path};
if (StartsWith(path, dst_dir_slash_)) {
std::string relative = path.substr(dst_dir_slash_.size());
auto it = remaps_.find(relative);
if (it != remaps_.end()) {
rv.second = src_base_dir_ + it->second->filename;
}
}
return rv;
}
private:
// Absolute path to a directory that some extra files will be mapped into.
const std::string dst_dir_;
// Includes a trailing slash.
const std::string dst_dir_slash_;
// Absolute path to a directory containing some files to be mapped into
// dst_dir_. Includes a trailing slash.
const std::string src_base_dir_;
// If remaps_[x] exists, attempt to read dst_dir_ / x should instead read
// src_base_dir_ / remaps_[x]->filename. FileInfo is used to maximize
// sharing with other backup data in memory.
std::unordered_map<std::string, std::shared_ptr<FileInfo>> remaps_;
}; };
class BackupMeta { class BackupMeta {
@ -284,6 +423,27 @@ class BackupEngineImpl {
return ss.str(); return ss.str();
} }
const std::shared_ptr<Env>& GetEnvForOpen() const {
if (!env_for_open_) {
// Lazy initialize
// Find directories
std::string dst_dir = meta_filename_;
auto i = dst_dir.rfind(kMetaDirSlash);
assert(i != std::string::npos);
std::string src_base_dir = dst_dir.substr(0, i);
dst_dir.replace(i, kMetaDirSlash.size(), kPrivateDirSlash);
// Make the RemapSharedFileSystem
std::shared_ptr<FileSystem> remap_fs =
std::make_shared<RemapSharedFileSystem>(
env_->GetFileSystem(), dst_dir, src_base_dir, files_);
// Make it read-only for safety
remap_fs = std::make_shared<ReadOnlyFileSystem>(remap_fs);
// Make an Env wrapper
env_for_open_ = std::make_shared<CompositeEnvWrapper>(env_, remap_fs);
}
return env_for_open_;
}
private: private:
int64_t timestamp_; int64_t timestamp_;
// sequence number is only approximate, should not be used // sequence number is only approximate, should not be used
@ -297,6 +457,7 @@ class BackupEngineImpl {
std::vector<std::shared_ptr<FileInfo>> files_; std::vector<std::shared_ptr<FileInfo>> files_;
std::unordered_map<std::string, std::shared_ptr<FileInfo>>* file_infos_; std::unordered_map<std::string, std::shared_ptr<FileInfo>>* file_infos_;
Env* env_; Env* env_;
mutable std::shared_ptr<Env> env_for_open_;
}; // BackupMeta }; // BackupMeta
inline std::string GetAbsolutePath( inline std::string GetAbsolutePath(
@ -304,30 +465,23 @@ class BackupEngineImpl {
assert(relative_path.size() == 0 || relative_path[0] != '/'); assert(relative_path.size() == 0 || relative_path[0] != '/');
return options_.backup_dir + "/" + relative_path; return options_.backup_dir + "/" + relative_path;
} }
inline std::string GetPrivateDirRel() const {
return "private";
}
inline std::string GetSharedDirRel() const { return "shared"; }
inline std::string GetSharedChecksumDirRel() const {
return "shared_checksum";
}
inline std::string GetPrivateFileRel(BackupID backup_id, inline std::string GetPrivateFileRel(BackupID backup_id,
bool tmp = false, bool tmp = false,
const std::string& file = "") const { const std::string& file = "") const {
assert(file.size() == 0 || file[0] != '/'); assert(file.size() == 0 || file[0] != '/');
return GetPrivateDirRel() + "/" + ROCKSDB_NAMESPACE::ToString(backup_id) + return kPrivateDirSlash + ROCKSDB_NAMESPACE::ToString(backup_id) +
(tmp ? ".tmp" : "") + "/" + file; (tmp ? ".tmp" : "") + "/" + file;
} }
inline std::string GetSharedFileRel(const std::string& file = "", inline std::string GetSharedFileRel(const std::string& file = "",
bool tmp = false) const { bool tmp = false) const {
assert(file.size() == 0 || file[0] != '/'); assert(file.size() == 0 || file[0] != '/');
return GetSharedDirRel() + "/" + (tmp ? "." : "") + file + return kSharedDirSlash + std::string(tmp ? "." : "") + file +
(tmp ? ".tmp" : ""); (tmp ? ".tmp" : "");
} }
inline std::string GetSharedFileWithChecksumRel(const std::string& file = "", inline std::string GetSharedFileWithChecksumRel(const std::string& file = "",
bool tmp = false) const { bool tmp = false) const {
assert(file.size() == 0 || file[0] != '/'); assert(file.size() == 0 || file[0] != '/');
return GetSharedChecksumDirRel() + "/" + (tmp ? "." : "") + file + return kSharedChecksumDirSlash + std::string(tmp ? "." : "") + file +
(tmp ? ".tmp" : ""); (tmp ? ".tmp" : "");
} }
inline bool UseLegacyNaming(const std::string& sid) const { inline bool UseLegacyNaming(const std::string& sid) const {
@ -354,18 +508,15 @@ class BackupEngineImpl {
} }
return file_copy; return file_copy;
} }
inline std::string GetFileFromChecksumFile(const std::string& file) const { static inline std::string GetFileFromChecksumFile(const std::string& file) {
assert(file.size() == 0 || file[0] != '/'); assert(file.size() == 0 || file[0] != '/');
std::string file_copy = file; std::string file_copy = file;
size_t first_underscore = file_copy.find_first_of('_'); size_t first_underscore = file_copy.find_first_of('_');
return file_copy.erase(first_underscore, return file_copy.erase(first_underscore,
file_copy.find_last_of('.') - first_underscore); file_copy.find_last_of('.') - first_underscore);
} }
inline std::string GetBackupMetaDir() const {
return GetAbsolutePath("meta");
}
inline std::string GetBackupMetaFile(BackupID backup_id, bool tmp) const { inline std::string GetBackupMetaFile(BackupID backup_id, bool tmp) const {
return GetBackupMetaDir() + "/" + (tmp ? "." : "") + return GetAbsolutePath(kMetaDirName) + "/" + (tmp ? "." : "") +
ROCKSDB_NAMESPACE::ToString(backup_id) + (tmp ? ".tmp" : ""); ROCKSDB_NAMESPACE::ToString(backup_id) + (tmp ? ".tmp" : "");
} }
@ -784,6 +935,8 @@ Status BackupEngineImpl::Initialize() {
} }
options_.Dump(options_.info_log); options_.Dump(options_.info_log);
auto meta_path = GetAbsolutePath(kMetaDirName);
if (!read_only_) { if (!read_only_) {
// we might need to clean up from previous crash or I/O errors // we might need to clean up from previous crash or I/O errors
might_need_garbage_collect_ = true; might_need_garbage_collect_ = true;
@ -810,9 +963,9 @@ Status BackupEngineImpl::Initialize() {
&shared_directory_); &shared_directory_);
} }
} }
directories.emplace_back(GetAbsolutePath(GetPrivateDirRel()), directories.emplace_back(GetAbsolutePath(kPrivateDirName),
&private_directory_); &private_directory_);
directories.emplace_back(GetBackupMetaDir(), &meta_directory_); directories.emplace_back(meta_path, &meta_directory_);
// create all the dirs we need // create all the dirs we need
for (const auto& d : directories) { for (const auto& d : directories) {
auto s = backup_env_->CreateDirIfMissing(d.first); auto s = backup_env_->CreateDirIfMissing(d.first);
@ -827,9 +980,9 @@ Status BackupEngineImpl::Initialize() {
std::vector<std::string> backup_meta_files; std::vector<std::string> backup_meta_files;
{ {
auto s = backup_env_->GetChildren(GetBackupMetaDir(), &backup_meta_files); auto s = backup_env_->GetChildren(meta_path, &backup_meta_files);
if (s.IsNotFound()) { if (s.IsNotFound()) {
return Status::NotFound(GetBackupMetaDir() + " is missing"); return Status::NotFound(meta_path + " is missing");
} else if (!s.ok()) { } else if (!s.ok()) {
return s; return s;
} }
@ -1396,14 +1549,19 @@ void BackupEngineImpl::GetBackupInfo(std::vector<BackupInfo>* backup_info,
backup_info->push_back(BackupInfo(backup.first, meta.GetTimestamp(), backup_info->push_back(BackupInfo(backup.first, meta.GetTimestamp(),
meta.GetSize(), meta.GetNumberFiles(), meta.GetSize(), meta.GetNumberFiles(),
meta.GetAppMetadata())); meta.GetAppMetadata()));
BackupInfo& binfo = backup_info->back();
if (include_file_details) { if (include_file_details) {
auto& file_details = backup_info->back().file_details; auto& file_details = binfo.file_details;
file_details.reserve(meta.GetFiles().size()); file_details.reserve(meta.GetFiles().size());
for (auto& file_ptr : meta.GetFiles()) { for (auto& file_ptr : meta.GetFiles()) {
BackupFileInfo& info = *file_details.emplace(file_details.end()); BackupFileInfo& finfo = *file_details.emplace(file_details.end());
info.relative_filename = file_ptr->filename; finfo.relative_filename = file_ptr->filename;
info.size = file_ptr->size; finfo.size = file_ptr->size;
} }
binfo.name_for_open =
GetAbsolutePath(GetPrivateFileRel(binfo.backup_id));
binfo.name_for_open.pop_back(); // remove trailing '/'
binfo.env_for_open = meta.GetEnvForOpen();
} }
} }
} }
@ -1488,21 +1646,8 @@ Status BackupEngineImpl::RestoreDBFromBackup(const RestoreOptions& options,
std::vector<RestoreAfterCopyOrCreateWorkItem> restore_items_to_finish; std::vector<RestoreAfterCopyOrCreateWorkItem> restore_items_to_finish;
for (const auto& file_info : backup->GetFiles()) { for (const auto& file_info : backup->GetFiles()) {
const std::string& file = file_info->filename; const std::string& file = file_info->filename;
std::string dst; // 1. get DB filename
// 1. extract the filename std::string dst = file_info->GetDbFileName();
size_t slash = file.find_last_of('/');
// file will either be shared/<file>, shared_checksum/<file_crc32c_size>,
// shared_checksum/<file_session>, shared_checksum/<file_crc32c_session>,
// or private/<number>/<file>
assert(slash != std::string::npos);
dst = file.substr(slash + 1);
// if the file was in shared_checksum, extract the real file name
// in this case the file is <number>_<checksum>_<size>.<type>,
// <number>_<session>.<type>, or <number>_<checksum>_<session>.<type>
if (file.substr(0, slash) == GetSharedChecksumDirRel()) {
dst = GetFileFromChecksumFile(dst);
}
// 2. find the filetype // 2. find the filetype
uint64_t number; uint64_t number;
@ -2135,7 +2280,7 @@ Status BackupEngineImpl::GarbageCollect() {
// delete obsolete private files // delete obsolete private files
std::vector<std::string> private_children; std::vector<std::string> private_children;
{ {
auto s = backup_env_->GetChildren(GetAbsolutePath(GetPrivateDirRel()), auto s = backup_env_->GetChildren(GetAbsolutePath(kPrivateDirName),
&private_children); &private_children);
if (!s.ok()) { if (!s.ok()) {
overall_status = s; overall_status = s;

59
utilities/backupable/backupable_db_test.cc
View File

@ -2614,6 +2614,65 @@ TEST_F(BackupableDBTest, ReadOnlyBackupEngine) {
delete db; delete db;
} }
TEST_F(BackupableDBTest, OpenBackupAsReadOnlyDB) {
DestroyDB(dbname_, options_);
OpenDBAndBackupEngine(true);
FillDB(db_.get(), 0, 100);
ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), true));
FillDB(db_.get(), 100, 200);
ASSERT_OK(backup_engine_->CreateNewBackup(db_.get(), true));
db_.reset(); // CloseDB
DestroyDB(dbname_, options_);
std::vector<BackupInfo> backup_info;
// First, check that we get empty fields without include_file_details
backup_engine_->GetBackupInfo(&backup_info, /*with file details*/ false);
ASSERT_EQ(backup_info.size(), 2);
ASSERT_EQ(backup_info[0].name_for_open, "");
ASSERT_FALSE(backup_info[0].env_for_open);
// Now for the real test
backup_info.clear();
backup_engine_->GetBackupInfo(&backup_info, /*with file details*/ true);
ASSERT_EQ(backup_info.size(), 2);
// Caution: DBOptions only holds a raw pointer to Env, so something else
// must keep it alive.
// Case 1: Keeping BackupEngine open suffices to keep Env alive
DB* db = nullptr;
Options opts = options_;
// Ensure some key defaults are set
opts.wal_dir = "";
opts.create_if_missing = false;
opts.info_log.reset();
opts.env = backup_info[0].env_for_open.get();
std::string name = backup_info[0].name_for_open;
backup_info.clear();
ASSERT_OK(DB::OpenForReadOnly(opts, name, &db));
AssertExists(db, 0, 100);
AssertEmpty(db, 100, 200);
delete db;
db = nullptr;
// Case 2: Keeping BackupInfo alive rather than BackupEngine also suffices
backup_engine_->GetBackupInfo(&backup_info, /*with file details*/ true);
ASSERT_EQ(backup_info.size(), 2);
CloseBackupEngine();
opts.env = backup_info[1].env_for_open.get();
name = backup_info[1].name_for_open;
// Note: keeping backup_info[1] alive
ASSERT_OK(DB::OpenForReadOnly(opts, name, &db));
AssertExists(db, 0, 200);
delete db;
db = nullptr;
// Now try opening read-write and make sure it fails, for safety.
ASSERT_TRUE(DB::Open(opts, name, &db).IsIOError());
}
TEST_F(BackupableDBTest, ProgressCallbackDuringBackup) { TEST_F(BackupableDBTest, ProgressCallbackDuringBackup) {
DestroyDB(dbname_, options_); DestroyDB(dbname_, options_);
OpenDBAndBackupEngine(true); OpenDBAndBackupEngine(true);