2019-12-09 08:49:32 +01:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
|
|
|
|
#ifdef GFLAGS
|
|
|
|
#pragma once
|
|
|
|
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
#include <mutex>
|
|
|
|
#include <unordered_set>
|
|
|
|
|
|
|
|
#include "file/filename.h"
|
2021-09-10 18:59:05 +02:00
|
|
|
#include "rocksdb/db.h"
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
#include "rocksdb/file_system.h"
|
2019-12-09 08:49:32 +01:00
|
|
|
#include "rocksdb/listener.h"
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
#include "rocksdb/table_properties.h"
|
|
|
|
#include "rocksdb/unique_id.h"
|
2019-12-09 08:49:32 +01:00
|
|
|
#include "util/gflags_compat.h"
|
2021-09-10 18:59:05 +02:00
|
|
|
#include "util/random.h"
|
2019-12-09 08:49:32 +01:00
|
|
|
|
|
|
|
DECLARE_int32(compact_files_one_in);
|
|
|
|
|
2020-02-20 21:07:53 +01:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
|
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
// Verify across process executions that all seen IDs are unique
|
|
|
|
class UniqueIdVerifier {
|
|
|
|
public:
|
|
|
|
explicit UniqueIdVerifier(const std::string& db_name);
|
|
|
|
~UniqueIdVerifier();
|
|
|
|
|
|
|
|
void Verify(const std::string& id);
|
|
|
|
|
|
|
|
private:
|
|
|
|
void VerifyNoWrite(const std::string& id);
|
|
|
|
|
|
|
|
private:
|
|
|
|
std::mutex mutex_;
|
|
|
|
// IDs persisted to a hidden file inside DB dir
|
|
|
|
std::string path_;
|
|
|
|
std::unique_ptr<FSWritableFile> data_file_writer_;
|
|
|
|
// Starting byte for which 8 bytes to check in memory within 24 byte ID
|
|
|
|
size_t offset_;
|
|
|
|
// Working copy of the set of 8 byte pieces
|
|
|
|
std::unordered_set<uint64_t> id_set_;
|
|
|
|
};
|
|
|
|
|
2019-12-09 08:49:32 +01:00
|
|
|
class DbStressListener : public EventListener {
|
|
|
|
public:
|
|
|
|
DbStressListener(const std::string& db_name,
|
|
|
|
const std::vector<DbPath>& db_paths,
|
|
|
|
const std::vector<ColumnFamilyDescriptor>& column_families)
|
|
|
|
: db_name_(db_name),
|
|
|
|
db_paths_(db_paths),
|
|
|
|
column_families_(column_families),
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
num_pending_file_creations_(0),
|
|
|
|
unique_ids_(db_name) {}
|
2021-07-27 16:46:09 +02:00
|
|
|
|
|
|
|
const char* Name() const override { return kClassName(); }
|
|
|
|
static const char* kClassName() { return "DBStressListener"; }
|
|
|
|
|
2019-12-21 06:45:01 +01:00
|
|
|
~DbStressListener() override { assert(num_pending_file_creations_ == 0); }
|
|
|
|
void OnFlushCompleted(DB* /*db*/, const FlushJobInfo& info) override {
|
2019-12-09 08:49:32 +01:00
|
|
|
assert(IsValidColumnFamilyName(info.cf_name));
|
|
|
|
VerifyFilePath(info.file_path);
|
|
|
|
// pretending doing some work here
|
2019-12-21 06:45:01 +01:00
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnFlushBegin(DB* /*db*/,
|
|
|
|
const FlushJobInfo& /*flush_job_info*/) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnTableFileDeleted(const TableFileDeletionInfo& /*info*/) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnCompactionBegin(DB* /*db*/, const CompactionJobInfo& /*ci*/) override {
|
|
|
|
RandomSleep();
|
2019-12-09 08:49:32 +01:00
|
|
|
}
|
|
|
|
|
2019-12-21 06:45:01 +01:00
|
|
|
void OnCompactionCompleted(DB* /*db*/, const CompactionJobInfo& ci) override {
|
2019-12-09 08:49:32 +01:00
|
|
|
assert(IsValidColumnFamilyName(ci.cf_name));
|
|
|
|
assert(ci.input_files.size() + ci.output_files.size() > 0U);
|
|
|
|
for (const auto& file_path : ci.input_files) {
|
|
|
|
VerifyFilePath(file_path);
|
|
|
|
}
|
|
|
|
for (const auto& file_path : ci.output_files) {
|
|
|
|
VerifyFilePath(file_path);
|
|
|
|
}
|
|
|
|
// pretending doing some work here
|
2019-12-21 06:45:01 +01:00
|
|
|
RandomSleep();
|
2019-12-09 08:49:32 +01:00
|
|
|
}
|
|
|
|
|
2019-12-21 06:45:01 +01:00
|
|
|
void OnTableFileCreationStarted(
|
2019-12-09 08:49:32 +01:00
|
|
|
const TableFileCreationBriefInfo& /*info*/) override {
|
|
|
|
++num_pending_file_creations_;
|
|
|
|
}
|
2019-12-21 06:45:01 +01:00
|
|
|
|
|
|
|
void OnTableFileCreated(const TableFileCreationInfo& info) override {
|
2019-12-09 08:49:32 +01:00
|
|
|
assert(info.db_name == db_name_);
|
|
|
|
assert(IsValidColumnFamilyName(info.cf_name));
|
|
|
|
if (info.file_size) {
|
|
|
|
VerifyFilePath(info.file_path);
|
|
|
|
}
|
|
|
|
assert(info.job_id > 0 || FLAGS_compact_files_one_in > 0);
|
|
|
|
if (info.status.ok() && info.file_size > 0) {
|
|
|
|
assert(info.table_properties.data_size > 0 ||
|
|
|
|
info.table_properties.num_range_deletions > 0);
|
|
|
|
assert(info.table_properties.raw_key_size > 0);
|
|
|
|
assert(info.table_properties.num_entries > 0);
|
|
|
|
}
|
|
|
|
--num_pending_file_creations_;
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
|
|
|
|
VerifyTableFileUniqueId(info.table_properties);
|
2019-12-09 08:49:32 +01:00
|
|
|
}
|
|
|
|
|
2019-12-21 06:45:01 +01:00
|
|
|
void OnMemTableSealed(const MemTableInfo& /*info*/) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnColumnFamilyHandleDeletionStarted(
|
|
|
|
ColumnFamilyHandle* /*handle*/) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
void OnExternalFileIngested(DB* /*db*/,
|
|
|
|
const ExternalFileIngestionInfo& info) override {
|
2019-12-21 06:45:01 +01:00
|
|
|
RandomSleep();
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
// Here we assume that each generated external file is ingested
|
|
|
|
// exactly once (or thrown away in case of crash)
|
|
|
|
VerifyTableFileUniqueId(info.table_properties);
|
2019-12-21 06:45:01 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
void OnBackgroundError(BackgroundErrorReason /* reason */,
|
|
|
|
Status* /* bg_error */) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnStallConditionsChanged(const WriteStallInfo& /*info*/) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnFileReadFinish(const FileOperationInfo& info) override {
|
|
|
|
// Even empty callback is valuable because sometimes some locks are
|
|
|
|
// released in order to make the callback.
|
|
|
|
|
|
|
|
// Sleep carefully here as it is a frequent operation and we don't want
|
|
|
|
// to slow down the tests. We always sleep when the read is large.
|
|
|
|
// When read is small, sleep in a small chance.
|
|
|
|
size_t length_read = info.length;
|
|
|
|
if (length_read >= 1000000 || Random::GetTLSInstance()->OneIn(1000)) {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnFileWriteFinish(const FileOperationInfo& info) override {
|
|
|
|
// Even empty callback is valuable because sometimes some locks are
|
|
|
|
// released in order to make the callback.
|
|
|
|
|
|
|
|
// Sleep carefully here as it is a frequent operation and we don't want
|
|
|
|
// to slow down the tests. When the write is large, always sleep.
|
|
|
|
// Otherwise, sleep in a relatively small chance.
|
|
|
|
size_t length_write = info.length;
|
|
|
|
if (length_write >= 1000000 || Random::GetTLSInstance()->OneIn(64)) {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
bool ShouldBeNotifiedOnFileIO() override {
|
|
|
|
RandomSleep();
|
|
|
|
return static_cast<bool>(Random::GetTLSInstance()->OneIn(1));
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnErrorRecoveryBegin(BackgroundErrorReason /* reason */,
|
|
|
|
Status /* bg_error */,
|
|
|
|
bool* /* auto_recovery */) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
|
|
|
void OnErrorRecoveryCompleted(Status /* old_bg_error */) override {
|
|
|
|
RandomSleep();
|
|
|
|
}
|
|
|
|
|
2019-12-09 08:49:32 +01:00
|
|
|
protected:
|
|
|
|
bool IsValidColumnFamilyName(const std::string& cf_name) const {
|
|
|
|
if (cf_name == kDefaultColumnFamilyName) {
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
// The column family names in the stress tests are numbers.
|
|
|
|
for (size_t i = 0; i < cf_name.size(); ++i) {
|
|
|
|
if (cf_name[i] < '0' || cf_name[i] > '9') {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
void VerifyFileDir(const std::string& file_dir) {
|
|
|
|
#ifndef NDEBUG
|
|
|
|
if (db_name_ == file_dir) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
for (const auto& db_path : db_paths_) {
|
|
|
|
if (db_path.path == file_dir) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
for (auto& cf : column_families_) {
|
|
|
|
for (const auto& cf_path : cf.options.cf_paths) {
|
|
|
|
if (cf_path.path == file_dir) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
assert(false);
|
|
|
|
#else
|
|
|
|
(void)file_dir;
|
|
|
|
#endif // !NDEBUG
|
|
|
|
}
|
|
|
|
|
|
|
|
void VerifyFileName(const std::string& file_name) {
|
|
|
|
#ifndef NDEBUG
|
|
|
|
uint64_t file_number;
|
|
|
|
FileType file_type;
|
|
|
|
bool result = ParseFileName(file_name, &file_number, &file_type);
|
|
|
|
assert(result);
|
|
|
|
assert(file_type == kTableFile);
|
|
|
|
#else
|
|
|
|
(void)file_name;
|
|
|
|
#endif // !NDEBUG
|
|
|
|
}
|
|
|
|
|
|
|
|
void VerifyFilePath(const std::string& file_path) {
|
|
|
|
#ifndef NDEBUG
|
|
|
|
size_t pos = file_path.find_last_of("/");
|
|
|
|
if (pos == std::string::npos) {
|
|
|
|
VerifyFileName(file_path);
|
|
|
|
} else {
|
|
|
|
if (pos > 0) {
|
|
|
|
VerifyFileDir(file_path.substr(0, pos));
|
|
|
|
}
|
|
|
|
VerifyFileName(file_path.substr(pos));
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
(void)file_path;
|
|
|
|
#endif // !NDEBUG
|
|
|
|
}
|
2019-12-21 06:45:01 +01:00
|
|
|
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
void VerifyTableFileUniqueId(const TableProperties& new_file_properties);
|
|
|
|
|
2019-12-21 06:45:01 +01:00
|
|
|
void RandomSleep() {
|
|
|
|
std::this_thread::sleep_for(
|
|
|
|
std::chrono::microseconds(Random::GetTLSInstance()->Uniform(5000)));
|
|
|
|
}
|
2019-12-09 08:49:32 +01:00
|
|
|
|
|
|
|
private:
|
|
|
|
std::string db_name_;
|
|
|
|
std::vector<DbPath> db_paths_;
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families_;
|
|
|
|
std::atomic<int> num_pending_file_creations_;
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
UniqueIdVerifier unique_ids_;
|
2019-12-09 08:49:32 +01:00
|
|
|
};
|
Experimental support for SST unique IDs (#8990)
Summary:
* New public header unique_id.h and function GetUniqueIdFromTableProperties
which computes a universally unique identifier based on table properties
of table files from recent RocksDB versions.
* Generation of DB session IDs is refactored so that they are
guaranteed unique in the lifetime of a process running RocksDB.
(SemiStructuredUniqueIdGen, new test included.) Along with file numbers,
this enables SST unique IDs to be guaranteed unique among SSTs generated
in a single process, and "better than random" between processes.
See https://github.com/pdillinger/unique_id
* In addition to public API producing 'external' unique IDs, there is a function
for producing 'internal' unique IDs, with functions for converting between the
two. In short, the external ID is "safe" for things people might do with it, and
the internal ID enables more "power user" features for the future. Specifically,
the external ID goes through a hashing layer so that any subset of bits in the
external ID can be used as a hash of the full ID, while also preserving
uniqueness guarantees in the first 128 bits (bijective both on first 128 bits
and on full 192 bits).
Intended follow-up:
* Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into
the third 64-bit value of the unique ID.)
* Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990
Test Plan:
Unit tests added, and checking of unique ids in stress test.
NOTE in stress test we do not generate nearly enough files to thoroughly
stress uniqueness, but the test trims off pieces of the ID to check for
uniqueness so that we can infer (with some assumptions) stronger
properties in the aggregate.
Reviewed By: zhichao-cao, mrambacher
Differential Revision: D31582865
Pulled By: pdillinger
fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243
2021-10-19 08:28:28 +02:00
|
|
|
#endif // !ROCKSDB_LITE
|
2020-02-20 21:07:53 +01:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|
2019-12-09 08:49:32 +01:00
|
|
|
#endif // GFLAGS
|