2016-02-10 00:12:00 +01:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
2017-07-16 01:03:42 +02:00
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
2014-10-28 19:54:33 +01:00
|
|
|
//
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
#include <atomic>
|
|
|
|
#include <deque>
|
|
|
|
#include <limits>
|
2019-10-16 19:39:00 +02:00
|
|
|
#include <list>
|
2014-10-28 19:54:33 +01:00
|
|
|
#include <set>
|
2019-10-16 19:39:00 +02:00
|
|
|
#include <string>
|
2014-10-28 19:54:33 +01:00
|
|
|
#include <utility>
|
|
|
|
#include <vector>
|
|
|
|
|
2021-03-18 04:43:22 +01:00
|
|
|
#include "db/blob/blob_file_completion_callback.h"
|
2014-10-28 19:54:33 +01:00
|
|
|
#include "db/column_family.h"
|
2016-08-04 02:42:06 +02:00
|
|
|
#include "db/dbformat.h"
|
|
|
|
#include "db/flush_scheduler.h"
|
|
|
|
#include "db/internal_stats.h"
|
|
|
|
#include "db/job_context.h"
|
2015-08-07 02:59:05 +02:00
|
|
|
#include "db/log_writer.h"
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
#include "db/logs_with_prep_tracker.h"
|
2014-10-28 19:54:33 +01:00
|
|
|
#include "db/memtable_list.h"
|
2015-08-07 02:59:05 +02:00
|
|
|
#include "db/snapshot_impl.h"
|
|
|
|
#include "db/version_edit.h"
|
2016-08-04 02:42:06 +02:00
|
|
|
#include "db/write_controller.h"
|
|
|
|
#include "db/write_thread.h"
|
2019-06-01 02:19:43 +02:00
|
|
|
#include "logging/event_logger.h"
|
2017-04-06 04:02:00 +02:00
|
|
|
#include "monitoring/instrumented_mutex.h"
|
|
|
|
#include "options/db_options.h"
|
2014-10-28 19:54:33 +01:00
|
|
|
#include "port/port.h"
|
|
|
|
#include "rocksdb/db.h"
|
|
|
|
#include "rocksdb/env.h"
|
2019-10-16 19:39:00 +02:00
|
|
|
#include "rocksdb/listener.h"
|
2014-10-28 19:54:33 +01:00
|
|
|
#include "rocksdb/memtablerep.h"
|
|
|
|
#include "rocksdb/transaction_log.h"
|
2015-10-13 00:06:38 +02:00
|
|
|
#include "table/scoped_arena_iterator.h"
|
2014-10-28 19:54:33 +01:00
|
|
|
#include "util/autovector.h"
|
|
|
|
#include "util/stop_watch.h"
|
|
|
|
#include "util/thread_local.h"
|
|
|
|
|
2020-02-20 21:07:53 +01:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
2014-10-28 19:54:33 +01:00
|
|
|
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
class DBImpl;
|
2014-10-28 19:54:33 +01:00
|
|
|
class MemTable;
|
2017-10-06 19:26:38 +02:00
|
|
|
class SnapshotChecker;
|
2014-10-28 19:54:33 +01:00
|
|
|
class TableCache;
|
|
|
|
class Version;
|
|
|
|
class VersionEdit;
|
|
|
|
class VersionSet;
|
|
|
|
class Arena;
|
|
|
|
|
|
|
|
class FlushJob {
|
|
|
|
public:
|
|
|
|
// TODO(icanadi) make effort to reduce number of parameters here
|
|
|
|
// IMPORTANT: mutable_cf_options needs to be alive while FlushJob is alive
|
|
|
|
FlushJob(const std::string& dbname, ColumnFamilyData* cfd,
|
2016-09-24 01:34:04 +02:00
|
|
|
const ImmutableDBOptions& db_options,
|
2020-12-02 18:29:50 +01:00
|
|
|
const MutableCFOptions& mutable_cf_options, uint64_t max_memtable_id,
|
|
|
|
const FileOptions& file_options, VersionSet* versions,
|
|
|
|
InstrumentedMutex* db_mutex, std::atomic<bool>* shutting_down,
|
2015-08-24 20:11:12 +02:00
|
|
|
std::vector<SequenceNumber> existing_snapshots,
|
Use SST files for Transaction conflict detection
Summary:
Currently, transactions can fail even if there is no actual write conflict. This is due to relying on only the memtables to check for write-conflicts. Users have to tune memtable settings to try to avoid this, but it's hard to figure out exactly how to tune these settings.
With this diff, TransactionDB will use both memtables and SST files to determine if there are any write conflicts. This relies on the fact that BlockBasedTable stores sequence numbers for all writes that happen after any open snapshot. Also, D50295 is needed to prevent SingleDelete from disappearing writes (the TODOs in this test code will be fixed once the other diff is approved and merged).
Note that Optimistic transactions will still rely on tuning memtable settings as we do not want to read from SST while on the write thread. Also, memtable settings can still be used to reduce how often TransactionDB needs to read SST files.
Test Plan: unit tests, db bench
Reviewers: rven, yhchiang, kradhakrishnan, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb, yoshinorim
Differential Revision: https://reviews.facebook.net/D50475
2015-10-16 01:37:15 +02:00
|
|
|
SequenceNumber earliest_write_conflict_snapshot,
|
2017-10-06 19:26:38 +02:00
|
|
|
SnapshotChecker* snapshot_checker, JobContext* job_context,
|
2020-03-03 01:14:00 +01:00
|
|
|
LogBuffer* log_buffer, FSDirectory* db_directory,
|
|
|
|
FSDirectory* output_file_directory,
|
|
|
|
CompressionType output_compression, Statistics* stats,
|
|
|
|
EventLogger* event_logger, bool measure_io_stats,
|
2019-03-20 01:24:09 +01:00
|
|
|
const bool sync_output_directory, const bool write_manifest,
|
2020-09-08 19:49:01 +02:00
|
|
|
Env::Priority thread_pri, const std::shared_ptr<IOTracer>& io_tracer,
|
2020-11-13 03:43:30 +01:00
|
|
|
const std::string& db_id = "", const std::string& db_session_id = "",
|
2021-03-18 04:43:22 +01:00
|
|
|
std::string full_history_ts_low = "",
|
|
|
|
BlobFileCompletionCallback* blob_callback = nullptr);
|
2015-03-13 18:45:40 +01:00
|
|
|
|
|
|
|
~FlushJob();
|
2014-10-28 19:54:33 +01:00
|
|
|
|
2017-01-20 08:03:45 +01:00
|
|
|
// Require db_mutex held.
|
2017-06-28 02:02:20 +02:00
|
|
|
// Once PickMemTable() is called, either Run() or Cancel() has to be called.
|
2016-07-20 00:12:46 +02:00
|
|
|
void PickMemTable();
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
Status Run(LogsWithPrepTracker* prep_tracker = nullptr,
|
|
|
|
FileMetaData* file_meta = nullptr);
|
2017-01-20 08:03:45 +01:00
|
|
|
void Cancel();
|
2018-10-16 04:59:20 +02:00
|
|
|
const autovector<MemTable*>& GetMemTables() const { return mems_; }
|
2014-10-28 19:54:33 +01:00
|
|
|
|
2019-10-16 19:39:00 +02:00
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
std::list<std::unique_ptr<FlushJobInfo>>* GetCommittedFlushJobsInfo() {
|
|
|
|
return &committed_flush_jobs_info_;
|
|
|
|
}
|
|
|
|
#endif // !ROCKSDB_LITE
|
|
|
|
|
Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487)
Summary:
In the current code base, we use Status to get and store the returned status from the call. Specifically, for IO related functions, the current Status cannot reflect the IO Error details such as error scope, error retryable attribute, and others. With the implementation of https://github.com/facebook/rocksdb/issues/5761, we have the new Wrapper for IO, which returns IOStatus instead of Status. However, the IOStatus is purged at the lower level of write path and transferred to Status.
The first job of this PR is to pass the IOStatus to the write path (flush, WAL write, and Compaction). The second job is to identify the Retryable IO Error as HardError, and set the bg_error_ as HardError. In this case, the DB Instance becomes read only. User is informed of the Status and need to take actions to deal with it (e.g., call db->Resume()).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6487
Test Plan: Added the testing case to error_handler_fs_test. Pass make asan_check
Reviewed By: anand1976
Differential Revision: D20685017
Pulled By: zhichao-cao
fbshipit-source-id: ff85f042896243abcd6ef37877834e26f36b6eb0
2020-03-28 00:03:05 +01:00
|
|
|
// Return the IO status
|
|
|
|
IOStatus io_status() const { return io_status_; }
|
|
|
|
|
2014-10-28 19:54:33 +01:00
|
|
|
private:
|
2015-05-16 08:22:22 +02:00
|
|
|
void ReportStartedFlush();
|
|
|
|
void ReportFlushInputSize(const autovector<MemTable*>& mems);
|
|
|
|
void RecordFlushIOStats();
|
2016-07-20 00:12:46 +02:00
|
|
|
Status WriteLevel0Table();
|
Make mempurge a background process (equivalent to in-memory compaction). (#8505)
Summary:
In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype.
In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...).
Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge.
MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`.
The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505
Reviewed By: pdillinger
Differential Revision: D29619283
Pulled By: bjlemaire
fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95
2021-07-10 02:16:00 +02:00
|
|
|
|
|
|
|
// Memtable Garbage Collection algorithm: a MemPurge takes the list
|
|
|
|
// of immutable memtables and filters out (or "purge") the outdated bytes
|
|
|
|
// out of it. The output (the filtered bytes, or "useful payload") is
|
|
|
|
// then transfered into a new memtable. If this memtable is filled, then
|
|
|
|
// the mempurge is aborted and rerouted to a regular flush process. Else,
|
|
|
|
// depending on the heuristics, placed onto the immutable memtable list.
|
|
|
|
// The addition to the imm list will not trigger a flush operation. The
|
|
|
|
// flush of the imm list will instead be triggered once the mutable memtable
|
|
|
|
// is added to the imm list.
|
|
|
|
// This process is typically intended for workloads with heavy overwrites
|
|
|
|
// when we want to avoid SSD writes (and reads) as much as possible.
|
|
|
|
// "MemPurge" is an experimental feature still at a very early stage
|
|
|
|
// of development. At the moment it is only compatible with the Get, Put,
|
|
|
|
// Delete operations as well as Iterators and CompactionFilters.
|
|
|
|
// For this early version, "MemPurge" is called by setting the
|
2021-08-11 03:07:48 +02:00
|
|
|
// options.experimental_mempurge_threshold value as >0.0. When this is
|
Make mempurge a background process (equivalent to in-memory compaction). (#8505)
Summary:
In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype.
In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...).
Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge.
MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`.
The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505
Reviewed By: pdillinger
Differential Revision: D29619283
Pulled By: bjlemaire
fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95
2021-07-10 02:16:00 +02:00
|
|
|
// the case, ALL automatic flush operations (kWRiteBufferManagerFull) will
|
2021-08-11 03:07:48 +02:00
|
|
|
// first go through the MemPurge process. Therefore, we strongly
|
Make mempurge a background process (equivalent to in-memory compaction). (#8505)
Summary:
In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype.
In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...).
Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge.
MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`.
The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505
Reviewed By: pdillinger
Differential Revision: D29619283
Pulled By: bjlemaire
fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95
2021-07-10 02:16:00 +02:00
|
|
|
// recommend all users not to set this flag as true given that the MemPurge
|
|
|
|
// process has not matured yet.
|
|
|
|
Status MemPurge();
|
Add simple heuristics for experimental mempurge. (#8583)
Summary:
Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE.
`ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput).
`ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios.
This PR also includes support for `experimental_mempurge_policy` in `db_bench`.
Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8583
Reviewed By: pdillinger
Differential Revision: D29888050
Pulled By: bjlemaire
fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
2021-07-26 20:55:27 +02:00
|
|
|
bool MemPurgeDecider();
|
2019-10-16 19:39:00 +02:00
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
std::unique_ptr<FlushJobInfo> GetFlushJobInfo() const;
|
|
|
|
#endif // !ROCKSDB_LITE
|
2018-10-16 04:59:20 +02:00
|
|
|
|
2014-10-28 19:54:33 +01:00
|
|
|
const std::string& dbname_;
|
2020-06-17 19:55:42 +02:00
|
|
|
const std::string db_id_;
|
|
|
|
const std::string db_session_id_;
|
2014-10-28 19:54:33 +01:00
|
|
|
ColumnFamilyData* cfd_;
|
2016-09-24 01:34:04 +02:00
|
|
|
const ImmutableDBOptions& db_options_;
|
2014-10-28 19:54:33 +01:00
|
|
|
const MutableCFOptions& mutable_cf_options_;
|
2020-12-02 18:29:50 +01:00
|
|
|
// A variable storing the largest memtable id to flush in this
|
2018-10-16 04:59:20 +02:00
|
|
|
// flush job. RocksDB uses this variable to select the memtables to flush in
|
|
|
|
// this job. All memtables in this column family with an ID smaller than or
|
2020-12-02 18:29:50 +01:00
|
|
|
// equal to max_memtable_id_ will be selected for flush.
|
|
|
|
uint64_t max_memtable_id_;
|
Introduce a new storage specific Env API (#5761)
Summary:
The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc.
This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO.
The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before.
This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection.
The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761
Differential Revision: D18868376
Pulled By: anand1976
fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f
2019-12-13 23:47:08 +01:00
|
|
|
const FileOptions file_options_;
|
2014-10-28 19:54:33 +01:00
|
|
|
VersionSet* versions_;
|
2015-02-05 06:39:45 +01:00
|
|
|
InstrumentedMutex* db_mutex_;
|
2014-10-28 19:54:33 +01:00
|
|
|
std::atomic<bool>* shutting_down_;
|
2015-08-24 20:11:12 +02:00
|
|
|
std::vector<SequenceNumber> existing_snapshots_;
|
Use SST files for Transaction conflict detection
Summary:
Currently, transactions can fail even if there is no actual write conflict. This is due to relying on only the memtables to check for write-conflicts. Users have to tune memtable settings to try to avoid this, but it's hard to figure out exactly how to tune these settings.
With this diff, TransactionDB will use both memtables and SST files to determine if there are any write conflicts. This relies on the fact that BlockBasedTable stores sequence numbers for all writes that happen after any open snapshot. Also, D50295 is needed to prevent SingleDelete from disappearing writes (the TODOs in this test code will be fixed once the other diff is approved and merged).
Note that Optimistic transactions will still rely on tuning memtable settings as we do not want to read from SST while on the write thread. Also, memtable settings can still be used to reduce how often TransactionDB needs to read SST files.
Test Plan: unit tests, db bench
Reviewers: rven, yhchiang, kradhakrishnan, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb, yoshinorim
Differential Revision: https://reviews.facebook.net/D50475
2015-10-16 01:37:15 +02:00
|
|
|
SequenceNumber earliest_write_conflict_snapshot_;
|
2017-10-06 19:26:38 +02:00
|
|
|
SnapshotChecker* snapshot_checker_;
|
2014-10-28 19:54:33 +01:00
|
|
|
JobContext* job_context_;
|
|
|
|
LogBuffer* log_buffer_;
|
2020-03-03 01:14:00 +01:00
|
|
|
FSDirectory* db_directory_;
|
|
|
|
FSDirectory* output_file_directory_;
|
2014-10-28 19:54:33 +01:00
|
|
|
CompressionType output_compression_;
|
|
|
|
Statistics* stats_;
|
EventLogger
Summary:
Here's my proposal for making our LOGs easier to read by machines.
The idea is to dump all events as JSON objects. JSON is easy to read by humans, but more importantly, it's easy to read by machines. That way, we can parse this, load into SQLite/mongo and then query or visualize.
I started with table_create and table_delete events, but if everybody agrees, I'll continue by adding more events (flush/compaction/etc etc)
Test Plan:
Ran db_bench. Observed:
2015/01/15-14:13:25.788019 1105ef000 EVENT_LOG_v1 {"time_micros": 1421360005788015, "event": "table_file_creation", "file_number": 12, "file_size": 1909699}
2015/01/15-14:13:25.956500 110740000 EVENT_LOG_v1 {"time_micros": 1421360005956498, "event": "table_file_deletion", "file_number": 12}
Reviewers: yhchiang, rven, dhruba, MarkCallaghan, lgalanis, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D31647
2015-03-13 18:15:54 +01:00
|
|
|
EventLogger* event_logger_;
|
2015-09-15 18:03:08 +02:00
|
|
|
TableProperties table_properties_;
|
2016-04-14 22:56:29 +02:00
|
|
|
bool measure_io_stats_;
|
2018-10-16 04:59:20 +02:00
|
|
|
// True if this flush job should call fsync on the output directory. False
|
|
|
|
// otherwise.
|
|
|
|
// Usually sync_output_directory_ is true. A flush job needs to call sync on
|
|
|
|
// the output directory before committing to the MANIFEST.
|
|
|
|
// However, an individual flush job does not have to call sync on the output
|
|
|
|
// directory if it is part of an atomic flush. After all flush jobs in the
|
|
|
|
// atomic flush succeed, call sync once on each distinct output directory.
|
|
|
|
const bool sync_output_directory_;
|
|
|
|
// True if this flush job should write to MANIFEST after successfully
|
|
|
|
// flushing memtables. False otherwise.
|
|
|
|
// Usually write_manifest_ is true. A flush job commits to the MANIFEST after
|
|
|
|
// flushing the memtables.
|
|
|
|
// However, an individual flush job cannot rashly write to the MANIFEST
|
|
|
|
// immediately after it finishes the flush if it is part of an atomic flush.
|
|
|
|
// In this case, only after all flush jobs succeed in flush can RocksDB
|
|
|
|
// commit to the MANIFEST.
|
|
|
|
const bool write_manifest_;
|
2019-10-16 19:39:00 +02:00
|
|
|
// The current flush job can commit flush result of a concurrent flush job.
|
|
|
|
// We collect FlushJobInfo of all jobs committed by current job and fire
|
|
|
|
// OnFlushCompleted for them.
|
|
|
|
std::list<std::unique_ptr<FlushJobInfo>> committed_flush_jobs_info_;
|
2016-07-20 00:12:46 +02:00
|
|
|
|
|
|
|
// Variables below are set by PickMemTable():
|
|
|
|
FileMetaData meta_;
|
|
|
|
autovector<MemTable*> mems_;
|
|
|
|
VersionEdit* edit_;
|
|
|
|
Version* base_;
|
|
|
|
bool pick_memtable_called;
|
2019-03-20 01:24:09 +01:00
|
|
|
Env::Priority thread_pri_;
|
Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487)
Summary:
In the current code base, we use Status to get and store the returned status from the call. Specifically, for IO related functions, the current Status cannot reflect the IO Error details such as error scope, error retryable attribute, and others. With the implementation of https://github.com/facebook/rocksdb/issues/5761, we have the new Wrapper for IO, which returns IOStatus instead of Status. However, the IOStatus is purged at the lower level of write path and transferred to Status.
The first job of this PR is to pass the IOStatus to the write path (flush, WAL write, and Compaction). The second job is to identify the Retryable IO Error as HardError, and set the bg_error_ as HardError. In this case, the DB Instance becomes read only. User is informed of the Status and need to take actions to deal with it (e.g., call db->Resume()).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6487
Test Plan: Added the testing case to error_handler_fs_test. Pass make asan_check
Reviewed By: anand1976
Differential Revision: D20685017
Pulled By: zhichao-cao
fbshipit-source-id: ff85f042896243abcd6ef37877834e26f36b6eb0
2020-03-28 00:03:05 +01:00
|
|
|
IOStatus io_status_;
|
2020-09-08 19:49:01 +02:00
|
|
|
|
|
|
|
const std::shared_ptr<IOTracer> io_tracer_;
|
2021-03-15 12:32:24 +01:00
|
|
|
SystemClock* clock_;
|
2020-11-13 03:43:30 +01:00
|
|
|
|
|
|
|
const std::string full_history_ts_low_;
|
2021-03-18 04:43:22 +01:00
|
|
|
BlobFileCompletionCallback* blob_callback_;
|
Add simple heuristics for experimental mempurge. (#8583)
Summary:
Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE.
`ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput).
`ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios.
This PR also includes support for `experimental_mempurge_policy` in `db_bench`.
Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8583
Reviewed By: pdillinger
Differential Revision: D29888050
Pulled By: bjlemaire
fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
2021-07-26 20:55:27 +02:00
|
|
|
|
2021-08-11 03:07:48 +02:00
|
|
|
// Used when experimental_mempurge_threshold > 0.0.
|
Add simple heuristics for experimental mempurge. (#8583)
Summary:
Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE.
`ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput).
`ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios.
This PR also includes support for `experimental_mempurge_policy` in `db_bench`.
Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8583
Reviewed By: pdillinger
Differential Revision: D29888050
Pulled By: bjlemaire
fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
2021-07-26 20:55:27 +02:00
|
|
|
bool contains_mempurge_outcome_;
|
2014-10-28 19:54:33 +01:00
|
|
|
};
|
|
|
|
|
2020-02-20 21:07:53 +01:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|