Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
2017-07-16 01:03:42 +02:00
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
//
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
|
2019-10-16 19:39:00 +02:00
|
|
|
#include <atomic>
|
|
|
|
|
|
|
|
#include "db/db_impl/db_impl.h"
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
#include "db/db_test_util.h"
|
2019-10-16 19:39:00 +02:00
|
|
|
#include "port/port.h"
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
#include "port/stack_trace.h"
|
2019-05-30 20:21:38 +02:00
|
|
|
#include "test_util/fault_injection_test_env.h"
|
|
|
|
#include "test_util/sync_point.h"
|
2019-10-16 19:39:00 +02:00
|
|
|
#include "util/cast_util.h"
|
|
|
|
#include "util/mutexlock.h"
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
|
2020-02-20 21:07:53 +01:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
|
|
|
|
class DBFlushTest : public DBTestBase {
|
|
|
|
public:
|
2020-07-02 19:21:12 +02:00
|
|
|
DBFlushTest() : DBTestBase("/db_flush_test") {}
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
};
|
|
|
|
|
2017-04-13 22:07:33 +02:00
|
|
|
class DBFlushDirectIOTest : public DBFlushTest,
|
|
|
|
public ::testing::WithParamInterface<bool> {
|
|
|
|
public:
|
|
|
|
DBFlushDirectIOTest() : DBFlushTest() {}
|
|
|
|
};
|
|
|
|
|
2018-10-27 00:06:44 +02:00
|
|
|
class DBAtomicFlushTest : public DBFlushTest,
|
|
|
|
public ::testing::WithParamInterface<bool> {
|
|
|
|
public:
|
|
|
|
DBAtomicFlushTest() : DBFlushTest() {}
|
|
|
|
};
|
|
|
|
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
// We had issue when two background threads trying to flush at the same time,
|
|
|
|
// only one of them get committed. The test verifies the issue is fixed.
|
|
|
|
TEST_F(DBFlushTest, FlushWhileWritingManifest) {
|
|
|
|
Options options;
|
|
|
|
options.disable_auto_compactions = true;
|
|
|
|
options.max_background_flushes = 2;
|
2017-03-13 17:41:30 +01:00
|
|
|
options.env = env_;
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
Reopen(options);
|
|
|
|
FlushOptions no_wait;
|
|
|
|
no_wait.wait = false;
|
2018-08-29 20:58:13 +02:00
|
|
|
no_wait.allow_write_stall=true;
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"VersionSet::LogAndApply:WriteManifest",
|
|
|
|
"DBFlushTest::FlushWhileWritingManifest:1"},
|
2018-10-06 00:37:45 +02:00
|
|
|
{"MemTableList::TryInstallMemtableFlushResults:InProgress",
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
"VersionSet::LogAndApply:WriteManifestDone"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_OK(Put("foo", "v"));
|
|
|
|
ASSERT_OK(dbfull()->Flush(no_wait));
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::FlushWhileWritingManifest:1");
|
|
|
|
ASSERT_OK(Put("bar", "v"));
|
|
|
|
ASSERT_OK(dbfull()->Flush(no_wait));
|
|
|
|
// If the issue is hit we will wait here forever.
|
|
|
|
dbfull()->TEST_WaitForFlushMemTable();
|
2016-07-21 23:32:12 +02:00
|
|
|
#ifndef ROCKSDB_LITE
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
ASSERT_EQ(2, TotalTableFiles());
|
2016-07-21 23:32:12 +02:00
|
|
|
#endif // ROCKSDB_LITE
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
}
|
|
|
|
|
2018-07-19 03:41:26 +02:00
|
|
|
// Disable this test temporarily on Travis as it fails intermittently.
|
|
|
|
// Github issue: #4151
|
2017-01-20 08:03:45 +01:00
|
|
|
TEST_F(DBFlushTest, SyncFail) {
|
|
|
|
std::unique_ptr<FaultInjectionTestEnv> fault_injection_env(
|
2017-03-13 17:41:30 +01:00
|
|
|
new FaultInjectionTestEnv(env_));
|
2017-01-20 08:03:45 +01:00
|
|
|
Options options;
|
|
|
|
options.disable_auto_compactions = true;
|
|
|
|
options.env = fault_injection_env.get();
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
2018-11-29 22:38:03 +01:00
|
|
|
{{"DBFlushTest::SyncFail:GetVersionRefCount:1",
|
|
|
|
"DBImpl::FlushMemTableToOutputFile:BeforePickMemtables"},
|
|
|
|
{"DBImpl::FlushMemTableToOutputFile:AfterPickMemtables",
|
|
|
|
"DBFlushTest::SyncFail:GetVersionRefCount:2"},
|
|
|
|
{"DBFlushTest::SyncFail:1", "DBImpl::SyncClosedLogs:Start"},
|
2017-01-20 08:03:45 +01:00
|
|
|
{"DBImpl::SyncClosedLogs:Failed", "DBFlushTest::SyncFail:2"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
2018-11-13 20:27:32 +01:00
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
2017-01-20 08:03:45 +01:00
|
|
|
Put("key", "value");
|
|
|
|
auto* cfd =
|
2020-07-03 04:24:25 +02:00
|
|
|
static_cast_with_check<ColumnFamilyHandleImpl>(db_->DefaultColumnFamily())
|
2017-01-20 08:03:45 +01:00
|
|
|
->cfd();
|
|
|
|
FlushOptions flush_options;
|
|
|
|
flush_options.wait = false;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_options));
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
// Flush installs a new super-version. Get the ref count after that.
|
|
|
|
auto current_before = cfd->current();
|
|
|
|
int refs_before = cfd->current()->TEST_refs();
|
2018-11-29 22:38:03 +01:00
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncFail:GetVersionRefCount:1");
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncFail:GetVersionRefCount:2");
|
|
|
|
int refs_after_picking_memtables = cfd->current()->TEST_refs();
|
|
|
|
ASSERT_EQ(refs_before + 1, refs_after_picking_memtables);
|
2017-01-20 08:03:45 +01:00
|
|
|
fault_injection_env->SetFilesystemActive(false);
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncFail:1");
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncFail:2");
|
|
|
|
fault_injection_env->SetFilesystemActive(true);
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
// Now the background job will do the flush; wait for it.
|
2017-01-20 08:03:45 +01:00
|
|
|
dbfull()->TEST_WaitForFlushMemTable();
|
2017-02-16 19:25:06 +01:00
|
|
|
#ifndef ROCKSDB_LITE
|
2017-01-20 08:03:45 +01:00
|
|
|
ASSERT_EQ("", FilesPerLevel()); // flush failed.
|
2017-02-16 19:25:06 +01:00
|
|
|
#endif // ROCKSDB_LITE
|
Skip deleted WALs during recovery
Summary:
This patch record min log number to keep to the manifest while flushing SST files to ignore them and any WAL older than them during recovery. This is to avoid scenarios when we have a gap between the WAL files are fed to the recovery procedure. The gap could happen by for example out-of-order WAL deletion. Such gap could cause problems in 2PC recovery where the prepared and commit entry are placed into two separate WAL and gap in the WALs could result into not processing the WAL with the commit entry and hence breaking the 2PC recovery logic.
Before the commit, for 2PC case, we determined which log number to keep in FindObsoleteFiles(). We looked at the earliest logs with outstanding prepare entries, or prepare entries whose respective commit or abort are in memtable. With the commit, the same calculation is done while we apply the SST flush. Just before installing the flush file, we precompute the earliest log file to keep after the flush finishes using the same logic (but skipping the memtables just flushed), record this information to the manifest entry for this new flushed SST file. This pre-computed value is also remembered in memory, and will later be used to determine whether a log file can be deleted. This value is unlikely to change until next flush because the commit entry will stay in memtable. (In WritePrepared, we could have removed the older log files as soon as all prepared entries are committed. It's not yet done anyway. Even if we do it, the only thing we loss with this new approach is earlier log deletion between two flushes, which does not guarantee to happen anyway because the obsolete file clean-up function is only executed after flush or compaction)
This min log number to keep is stored in the manifest using the safely-ignore customized field of AddFile entry, in order to guarantee that the DB generated using newer release can be opened by previous releases no older than 4.2.
Closes https://github.com/facebook/rocksdb/pull/3765
Differential Revision: D7747618
Pulled By: siying
fbshipit-source-id: d00c92105b4f83852e9754a1b70d6b64cb590729
2018-05-04 00:35:11 +02:00
|
|
|
// Backgroun flush job should release ref count to current version.
|
|
|
|
ASSERT_EQ(current_before, cfd->current());
|
2017-01-20 08:03:45 +01:00
|
|
|
ASSERT_EQ(refs_before, cfd->current()->TEST_refs());
|
|
|
|
Destroy(options);
|
|
|
|
}
|
2018-11-13 20:27:32 +01:00
|
|
|
|
|
|
|
TEST_F(DBFlushTest, SyncSkip) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBFlushTest::SyncSkip:1", "DBImpl::SyncClosedLogs:Skip"},
|
|
|
|
{"DBImpl::SyncClosedLogs:Skip", "DBFlushTest::SyncSkip:2"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
Reopen(options);
|
|
|
|
Put("key", "value");
|
|
|
|
|
|
|
|
FlushOptions flush_options;
|
|
|
|
flush_options.wait = false;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_options));
|
|
|
|
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncSkip:1");
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::SyncSkip:2");
|
|
|
|
|
|
|
|
// Now the background job will do the flush; wait for it.
|
|
|
|
dbfull()->TEST_WaitForFlushMemTable();
|
|
|
|
|
|
|
|
Destroy(options);
|
|
|
|
}
|
2017-01-20 08:03:45 +01:00
|
|
|
|
2017-05-23 20:04:25 +02:00
|
|
|
TEST_F(DBFlushTest, FlushInLowPriThreadPool) {
|
|
|
|
// Verify setting an empty high-pri (flush) thread pool causes flushes to be
|
|
|
|
// scheduled in the low-pri (compaction) thread pool.
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.level0_file_num_compaction_trigger = 4;
|
|
|
|
options.memtable_factory.reset(new SpecialSkipListFactory(1));
|
|
|
|
Reopen(options);
|
|
|
|
env_->SetBackgroundThreads(0, Env::HIGH);
|
|
|
|
|
|
|
|
std::thread::id tid;
|
|
|
|
int num_flushes = 0, num_compactions = 0;
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
2018-04-13 02:55:14 +02:00
|
|
|
"DBImpl::BGWorkFlush", [&](void* /*arg*/) {
|
2017-05-23 20:04:25 +02:00
|
|
|
if (tid == std::thread::id()) {
|
|
|
|
tid = std::this_thread::get_id();
|
|
|
|
} else {
|
|
|
|
ASSERT_EQ(tid, std::this_thread::get_id());
|
|
|
|
}
|
|
|
|
++num_flushes;
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
2018-04-13 02:55:14 +02:00
|
|
|
"DBImpl::BGWorkCompaction", [&](void* /*arg*/) {
|
2017-05-23 20:04:25 +02:00
|
|
|
ASSERT_EQ(tid, std::this_thread::get_id());
|
|
|
|
++num_compactions;
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_OK(Put("key", "val"));
|
|
|
|
for (int i = 0; i < 4; ++i) {
|
|
|
|
ASSERT_OK(Put("key", "val"));
|
|
|
|
dbfull()->TEST_WaitForFlushMemTable();
|
|
|
|
}
|
|
|
|
dbfull()->TEST_WaitForCompact();
|
|
|
|
ASSERT_EQ(4, num_flushes);
|
|
|
|
ASSERT_EQ(1, num_compactions);
|
|
|
|
}
|
|
|
|
|
2018-01-19 02:32:50 +01:00
|
|
|
TEST_F(DBFlushTest, ManualFlushWithMinWriteBufferNumberToMerge) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.write_buffer_size = 100;
|
|
|
|
options.max_write_buffer_number = 4;
|
|
|
|
options.min_write_buffer_number_to_merge = 3;
|
|
|
|
Reopen(options);
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::BGWorkFlush",
|
|
|
|
"DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:1"},
|
|
|
|
{"DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:2",
|
2018-01-30 03:44:01 +01:00
|
|
|
"FlushJob::WriteLevel0Table"}});
|
2018-01-19 02:32:50 +01:00
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_OK(Put("key1", "value1"));
|
|
|
|
|
|
|
|
port::Thread t([&]() {
|
|
|
|
// The call wait for flush to finish, i.e. with flush_options.wait = true.
|
|
|
|
ASSERT_OK(Flush());
|
|
|
|
});
|
|
|
|
|
|
|
|
// Wait for flush start.
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:1");
|
|
|
|
// Insert a second memtable before the manual flush finish.
|
|
|
|
// At the end of the manual flush job, it will check if further flush
|
|
|
|
// is needed, but it will not trigger flush of the second memtable because
|
|
|
|
// min_write_buffer_number_to_merge is not reached.
|
|
|
|
ASSERT_OK(Put("key2", "value2"));
|
|
|
|
ASSERT_OK(dbfull()->TEST_SwitchMemtable());
|
|
|
|
TEST_SYNC_POINT("DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:2");
|
|
|
|
|
|
|
|
// Manual flush should return, without waiting for flush indefinitely.
|
|
|
|
t.join();
|
|
|
|
}
|
|
|
|
|
2019-11-27 23:46:38 +01:00
|
|
|
TEST_F(DBFlushTest, ScheduleOnlyOneBgThread) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
Reopen(options);
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
int called = 0;
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
|
|
|
"DBImpl::MaybeScheduleFlushOrCompaction:AfterSchedule:0", [&](void* arg) {
|
|
|
|
ASSERT_NE(nullptr, arg);
|
|
|
|
auto unscheduled_flushes = *reinterpret_cast<int*>(arg);
|
|
|
|
ASSERT_EQ(0, unscheduled_flushes);
|
|
|
|
++called;
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_OK(Put("a", "foo"));
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_opts));
|
|
|
|
ASSERT_EQ(1, called);
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
}
|
|
|
|
|
2017-04-13 22:07:33 +02:00
|
|
|
TEST_P(DBFlushDirectIOTest, DirectIO) {
|
|
|
|
Options options;
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.disable_auto_compactions = true;
|
|
|
|
options.max_background_flushes = 2;
|
|
|
|
options.use_direct_io_for_flush_and_compaction = GetParam();
|
|
|
|
options.env = new MockEnv(Env::Default());
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
|
|
|
"BuildTable:create_file", [&](void* arg) {
|
|
|
|
bool* use_direct_writes = static_cast<bool*>(arg);
|
|
|
|
ASSERT_EQ(*use_direct_writes,
|
|
|
|
options.use_direct_io_for_flush_and_compaction);
|
|
|
|
});
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
Reopen(options);
|
|
|
|
ASSERT_OK(Put("foo", "v"));
|
|
|
|
FlushOptions flush_options;
|
|
|
|
flush_options.wait = true;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_options));
|
|
|
|
Destroy(options);
|
|
|
|
delete options.env;
|
|
|
|
}
|
|
|
|
|
2018-02-05 22:48:25 +01:00
|
|
|
TEST_F(DBFlushTest, FlushError) {
|
|
|
|
Options options;
|
|
|
|
std::unique_ptr<FaultInjectionTestEnv> fault_injection_env(
|
|
|
|
new FaultInjectionTestEnv(env_));
|
|
|
|
options.write_buffer_size = 100;
|
|
|
|
options.max_write_buffer_number = 4;
|
|
|
|
options.min_write_buffer_number_to_merge = 3;
|
|
|
|
options.disable_auto_compactions = true;
|
|
|
|
options.env = fault_injection_env.get();
|
|
|
|
Reopen(options);
|
|
|
|
|
|
|
|
ASSERT_OK(Put("key1", "value1"));
|
|
|
|
ASSERT_OK(Put("key2", "value2"));
|
|
|
|
fault_injection_env->SetFilesystemActive(false);
|
|
|
|
Status s = dbfull()->TEST_SwitchMemtable();
|
|
|
|
fault_injection_env->SetFilesystemActive(true);
|
|
|
|
Destroy(options);
|
|
|
|
ASSERT_NE(s, Status::OK());
|
|
|
|
}
|
|
|
|
|
2018-11-01 23:23:20 +01:00
|
|
|
TEST_F(DBFlushTest, ManualFlushFailsInReadOnlyMode) {
|
|
|
|
// Regression test for bug where manual flush hangs forever when the DB
|
|
|
|
// is in read-only mode. Verify it now at least returns, despite failing.
|
|
|
|
Options options;
|
|
|
|
std::unique_ptr<FaultInjectionTestEnv> fault_injection_env(
|
|
|
|
new FaultInjectionTestEnv(env_));
|
|
|
|
options.env = fault_injection_env.get();
|
|
|
|
options.max_write_buffer_number = 2;
|
|
|
|
Reopen(options);
|
|
|
|
|
|
|
|
// Trigger a first flush but don't let it run
|
|
|
|
ASSERT_OK(db_->PauseBackgroundWork());
|
|
|
|
ASSERT_OK(Put("key1", "value1"));
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.wait = false;
|
|
|
|
ASSERT_OK(db_->Flush(flush_opts));
|
|
|
|
|
|
|
|
// Write a key to the second memtable so we have something to flush later
|
|
|
|
// after the DB is in read-only mode.
|
|
|
|
ASSERT_OK(Put("key2", "value2"));
|
|
|
|
|
|
|
|
// Let the first flush continue, hit an error, and put the DB in read-only
|
|
|
|
// mode.
|
|
|
|
fault_injection_env->SetFilesystemActive(false);
|
|
|
|
ASSERT_OK(db_->ContinueBackgroundWork());
|
|
|
|
dbfull()->TEST_WaitForFlushMemTable();
|
2018-11-02 19:25:06 +01:00
|
|
|
#ifndef ROCKSDB_LITE
|
2018-11-01 23:23:20 +01:00
|
|
|
uint64_t num_bg_errors;
|
|
|
|
ASSERT_TRUE(db_->GetIntProperty(DB::Properties::kBackgroundErrors,
|
|
|
|
&num_bg_errors));
|
|
|
|
ASSERT_GT(num_bg_errors, 0);
|
2018-11-02 19:25:06 +01:00
|
|
|
#endif // ROCKSDB_LITE
|
2018-11-01 23:23:20 +01:00
|
|
|
|
|
|
|
// In the bug scenario, triggering another flush would cause the second flush
|
|
|
|
// to hang forever. After the fix we expect it to return an error.
|
|
|
|
ASSERT_NOK(db_->Flush(FlushOptions()));
|
|
|
|
|
|
|
|
Close();
|
|
|
|
}
|
|
|
|
|
2019-07-01 23:04:10 +02:00
|
|
|
TEST_F(DBFlushTest, CFDropRaceWithWaitForFlushMemTables) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::FlushMemTable:AfterScheduleFlush",
|
|
|
|
"DBFlushTest::CFDropRaceWithWaitForFlushMemTables:BeforeDrop"},
|
|
|
|
{"DBFlushTest::CFDropRaceWithWaitForFlushMemTables:AfterFree",
|
|
|
|
"DBImpl::BackgroundCallFlush:start"},
|
|
|
|
{"DBImpl::BackgroundCallFlush:start",
|
|
|
|
"DBImpl::FlushMemTable:BeforeWaitForBgFlush"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
ASSERT_EQ(2, handles_.size());
|
|
|
|
ASSERT_OK(Put(1, "key", "value"));
|
|
|
|
auto* cfd = static_cast<ColumnFamilyHandleImpl*>(handles_[1])->cfd();
|
|
|
|
port::Thread drop_cf_thr([&]() {
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBFlushTest::CFDropRaceWithWaitForFlushMemTables:BeforeDrop");
|
|
|
|
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
|
|
|
|
ASSERT_OK(dbfull()->DestroyColumnFamilyHandle(handles_[1]));
|
|
|
|
handles_.resize(1);
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBFlushTest::CFDropRaceWithWaitForFlushMemTables:AfterFree");
|
|
|
|
});
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.allow_write_stall = true;
|
|
|
|
ASSERT_NOK(dbfull()->TEST_FlushMemTable(cfd, flush_opts));
|
|
|
|
drop_cf_thr.join();
|
|
|
|
Close();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
|
2019-10-16 19:39:00 +02:00
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
TEST_F(DBFlushTest, FireOnFlushCompletedAfterCommittedResult) {
|
|
|
|
class TestListener : public EventListener {
|
|
|
|
public:
|
|
|
|
void OnFlushCompleted(DB* db, const FlushJobInfo& info) override {
|
|
|
|
// There's only one key in each flush.
|
|
|
|
ASSERT_EQ(info.smallest_seqno, info.largest_seqno);
|
|
|
|
ASSERT_NE(0, info.smallest_seqno);
|
|
|
|
if (info.smallest_seqno == seq1) {
|
|
|
|
// First flush completed
|
|
|
|
ASSERT_FALSE(completed1);
|
|
|
|
completed1 = true;
|
|
|
|
CheckFlushResultCommitted(db, seq1);
|
|
|
|
} else {
|
|
|
|
// Second flush completed
|
|
|
|
ASSERT_FALSE(completed2);
|
|
|
|
completed2 = true;
|
|
|
|
ASSERT_EQ(info.smallest_seqno, seq2);
|
|
|
|
CheckFlushResultCommitted(db, seq2);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void CheckFlushResultCommitted(DB* db, SequenceNumber seq) {
|
|
|
|
DBImpl* db_impl = static_cast_with_check<DBImpl>(db);
|
|
|
|
InstrumentedMutex* mutex = db_impl->mutex();
|
|
|
|
mutex->Lock();
|
2020-07-03 04:24:25 +02:00
|
|
|
auto* cfd = static_cast_with_check<ColumnFamilyHandleImpl>(
|
|
|
|
db->DefaultColumnFamily())
|
|
|
|
->cfd();
|
2019-10-16 19:39:00 +02:00
|
|
|
ASSERT_LT(seq, cfd->imm()->current()->GetEarliestSequenceNumber());
|
|
|
|
mutex->Unlock();
|
|
|
|
}
|
|
|
|
|
|
|
|
std::atomic<SequenceNumber> seq1{0};
|
|
|
|
std::atomic<SequenceNumber> seq2{0};
|
|
|
|
std::atomic<bool> completed1{false};
|
|
|
|
std::atomic<bool> completed2{false};
|
|
|
|
};
|
|
|
|
std::shared_ptr<TestListener> listener = std::make_shared<TestListener>();
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
2019-11-08 22:45:31 +01:00
|
|
|
{{"DBImpl::BackgroundCallFlush:start",
|
2019-10-16 19:39:00 +02:00
|
|
|
"DBFlushTest::FireOnFlushCompletedAfterCommittedResult:WaitFirst"},
|
|
|
|
{"DBImpl::FlushMemTableToOutputFile:Finish",
|
|
|
|
"DBFlushTest::FireOnFlushCompletedAfterCommittedResult:WaitSecond"}});
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
|
|
|
"FlushJob::WriteLevel0Table", [&listener](void* arg) {
|
|
|
|
// Wait for the second flush finished, out of mutex.
|
|
|
|
auto* mems = reinterpret_cast<autovector<MemTable*>*>(arg);
|
|
|
|
if (mems->front()->GetEarliestSequenceNumber() == listener->seq1 - 1) {
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBFlushTest::FireOnFlushCompletedAfterCommittedResult:"
|
|
|
|
"WaitSecond");
|
|
|
|
}
|
|
|
|
});
|
|
|
|
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.listeners.push_back(listener);
|
|
|
|
// Setting max_flush_jobs = max_background_jobs / 4 = 2.
|
|
|
|
options.max_background_jobs = 8;
|
|
|
|
// Allow 2 immutable memtables.
|
|
|
|
options.max_write_buffer_number = 3;
|
|
|
|
Reopen(options);
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
ASSERT_OK(Put("foo", "v"));
|
|
|
|
listener->seq1 = db_->GetLatestSequenceNumber();
|
|
|
|
// t1 will wait for the second flush complete before committing flush result.
|
|
|
|
auto t1 = port::Thread([&]() {
|
|
|
|
// flush_opts.wait = true
|
|
|
|
ASSERT_OK(db_->Flush(FlushOptions()));
|
|
|
|
});
|
2019-11-08 22:45:31 +01:00
|
|
|
// Wait for first flush started.
|
2019-10-16 19:39:00 +02:00
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBFlushTest::FireOnFlushCompletedAfterCommittedResult:WaitFirst");
|
|
|
|
// The second flush will exit early without commit its result. The work
|
|
|
|
// is delegated to the first flush.
|
|
|
|
ASSERT_OK(Put("bar", "v"));
|
|
|
|
listener->seq2 = db_->GetLatestSequenceNumber();
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.wait = false;
|
|
|
|
ASSERT_OK(db_->Flush(flush_opts));
|
|
|
|
t1.join();
|
|
|
|
ASSERT_TRUE(listener->completed1);
|
|
|
|
ASSERT_TRUE(listener->completed2);
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
}
|
|
|
|
#endif // !ROCKSDB_LITE
|
|
|
|
|
2018-10-27 00:06:44 +02:00
|
|
|
TEST_P(DBAtomicFlushTest, ManualAtomicFlush) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = GetParam();
|
|
|
|
options.write_buffer_size = (static_cast<size_t>(64) << 20);
|
|
|
|
|
|
|
|
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
|
|
|
|
size_t num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(3, num_cfs);
|
|
|
|
WriteOptions wopts;
|
|
|
|
wopts.disableWAL = true;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
ASSERT_OK(Put(static_cast<int>(i) /*cf*/, "key", "value", wopts));
|
|
|
|
}
|
|
|
|
std::vector<int> cf_ids;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
cf_ids.emplace_back(static_cast<int>(i));
|
|
|
|
}
|
|
|
|
ASSERT_OK(Flush(cf_ids));
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
|
|
|
|
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
|
|
|
|
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
TEST_P(DBAtomicFlushTest, AtomicFlushTriggeredByMemTableFull) {
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = GetParam();
|
|
|
|
// 4KB so that we can easily trigger auto flush.
|
|
|
|
options.write_buffer_size = 4096;
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::BackgroundCallFlush:FlushFinish:0",
|
|
|
|
"DBAtomicFlushTest::AtomicFlushTriggeredByMemTableFull:BeforeCheck"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
|
|
|
|
size_t num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(3, num_cfs);
|
|
|
|
WriteOptions wopts;
|
|
|
|
wopts.disableWAL = true;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
ASSERT_OK(Put(static_cast<int>(i) /*cf*/, "key", "value", wopts));
|
|
|
|
}
|
|
|
|
// Keep writing to one of them column families to trigger auto flush.
|
|
|
|
for (int i = 0; i != 4000; ++i) {
|
|
|
|
ASSERT_OK(Put(static_cast<int>(num_cfs) - 1 /*cf*/,
|
|
|
|
"key" + std::to_string(i), "value" + std::to_string(i),
|
|
|
|
wopts));
|
|
|
|
}
|
|
|
|
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBAtomicFlushTest::AtomicFlushTriggeredByMemTableFull:BeforeCheck");
|
|
|
|
if (options.atomic_flush) {
|
2020-06-03 00:02:44 +02:00
|
|
|
for (size_t i = 0; i + 1 != num_cfs; ++i) {
|
2018-10-27 00:06:44 +02:00
|
|
|
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
|
|
|
|
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
|
|
|
|
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
|
|
|
|
}
|
|
|
|
} else {
|
2020-06-03 00:02:44 +02:00
|
|
|
for (size_t i = 0; i + 1 != num_cfs; ++i) {
|
2018-10-27 00:06:44 +02:00
|
|
|
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
|
|
|
|
ASSERT_EQ(0, cfh->cfd()->imm()->NumNotFlushed());
|
|
|
|
ASSERT_FALSE(cfh->cfd()->mem()->IsEmpty());
|
|
|
|
}
|
|
|
|
}
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
|
2018-11-15 05:52:21 +01:00
|
|
|
TEST_P(DBAtomicFlushTest, AtomicFlushRollbackSomeJobs) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
std::unique_ptr<FaultInjectionTestEnv> fault_injection_env(
|
|
|
|
new FaultInjectionTestEnv(env_));
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
options.env = fault_injection_env.get();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::AtomicFlushMemTablesToOutputFiles:SomeFlushJobsComplete:1",
|
|
|
|
"DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:1"},
|
|
|
|
{"DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:2",
|
|
|
|
"DBImpl::AtomicFlushMemTablesToOutputFiles:SomeFlushJobsComplete:2"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
|
|
|
|
size_t num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(3, num_cfs);
|
|
|
|
WriteOptions wopts;
|
|
|
|
wopts.disableWAL = true;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
int cf_id = static_cast<int>(i);
|
|
|
|
ASSERT_OK(Put(cf_id, "key", "value", wopts));
|
|
|
|
}
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.wait = false;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_opts, handles_));
|
|
|
|
TEST_SYNC_POINT("DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:1");
|
|
|
|
fault_injection_env->SetFilesystemActive(false);
|
|
|
|
TEST_SYNC_POINT("DBAtomicFlushTest::AtomicFlushRollbackSomeJobs:2");
|
|
|
|
for (auto* cfh : handles_) {
|
|
|
|
dbfull()->TEST_WaitForFlushMemTable(cfh);
|
|
|
|
}
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
auto cfh = static_cast<ColumnFamilyHandleImpl*>(handles_[i]);
|
|
|
|
ASSERT_EQ(1, cfh->cfd()->imm()->NumNotFlushed());
|
|
|
|
ASSERT_TRUE(cfh->cfd()->mem()->IsEmpty());
|
|
|
|
}
|
|
|
|
fault_injection_env->SetFilesystemActive(true);
|
|
|
|
Destroy(options);
|
|
|
|
}
|
|
|
|
|
2018-12-14 00:10:16 +01:00
|
|
|
TEST_P(DBAtomicFlushTest, FlushMultipleCFs_DropSomeBeforeRequestFlush) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
|
|
|
|
size_t num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(3, num_cfs);
|
|
|
|
WriteOptions wopts;
|
|
|
|
wopts.disableWAL = true;
|
|
|
|
std::vector<int> cf_ids;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
int cf_id = static_cast<int>(i);
|
|
|
|
ASSERT_OK(Put(cf_id, "key", "value", wopts));
|
|
|
|
cf_ids.push_back(cf_id);
|
|
|
|
}
|
|
|
|
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
|
2019-05-20 19:37:37 +02:00
|
|
|
ASSERT_TRUE(Flush(cf_ids).IsColumnFamilyDropped());
|
2018-12-14 00:10:16 +01:00
|
|
|
Destroy(options);
|
|
|
|
}
|
|
|
|
|
|
|
|
TEST_P(DBAtomicFlushTest,
|
|
|
|
FlushMultipleCFs_DropSomeAfterScheduleFlushBeforeFlushJobRun) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
|
|
|
|
CreateAndReopenWithCF({"pikachu", "eevee"}, options);
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::AtomicFlushMemTables:AfterScheduleFlush",
|
|
|
|
"DBAtomicFlushTest::BeforeDropCF"},
|
|
|
|
{"DBAtomicFlushTest::AfterDropCF",
|
|
|
|
"DBImpl::BackgroundCallFlush:start"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
size_t num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(3, num_cfs);
|
|
|
|
WriteOptions wopts;
|
|
|
|
wopts.disableWAL = true;
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
int cf_id = static_cast<int>(i);
|
|
|
|
ASSERT_OK(Put(cf_id, "key", "value", wopts));
|
|
|
|
}
|
|
|
|
port::Thread user_thread([&]() {
|
|
|
|
TEST_SYNC_POINT("DBAtomicFlushTest::BeforeDropCF");
|
|
|
|
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
|
|
|
|
TEST_SYNC_POINT("DBAtomicFlushTest::AfterDropCF");
|
|
|
|
});
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.wait = true;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_opts, handles_));
|
|
|
|
user_thread.join();
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
int cf_id = static_cast<int>(i);
|
|
|
|
ASSERT_EQ("value", Get(cf_id, "key"));
|
|
|
|
}
|
|
|
|
|
|
|
|
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "eevee"}, options);
|
|
|
|
num_cfs = handles_.size();
|
|
|
|
ASSERT_EQ(2, num_cfs);
|
|
|
|
for (size_t i = 0; i != num_cfs; ++i) {
|
|
|
|
int cf_id = static_cast<int>(i);
|
|
|
|
ASSERT_EQ("value", Get(cf_id, "key"));
|
|
|
|
}
|
|
|
|
Destroy(options);
|
|
|
|
}
|
|
|
|
|
2019-04-29 21:29:57 +02:00
|
|
|
TEST_P(DBAtomicFlushTest, TriggerFlushAndClose) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
const int kNumKeysTriggerFlush = 4;
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
options.memtable_factory.reset(
|
|
|
|
new SpecialSkipListFactory(kNumKeysTriggerFlush));
|
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
|
|
|
|
|
|
|
for (int i = 0; i != kNumKeysTriggerFlush; ++i) {
|
|
|
|
ASSERT_OK(Put(0, "key" + std::to_string(i), "value" + std::to_string(i)));
|
|
|
|
}
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
ASSERT_OK(Put(0, "key", "value"));
|
|
|
|
Close();
|
|
|
|
|
|
|
|
ReopenWithColumnFamilies({kDefaultColumnFamilyName, "pikachu"}, options);
|
|
|
|
ASSERT_EQ("value", Get(0, "key"));
|
|
|
|
}
|
|
|
|
|
2019-05-11 02:53:41 +02:00
|
|
|
TEST_P(DBAtomicFlushTest, PickMemtablesRaceWithBackgroundFlush) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
options.max_write_buffer_number = 4;
|
|
|
|
// Set min_write_buffer_number_to_merge to be greater than 1, so that
|
|
|
|
// a column family with one memtable in the imm will not cause IsFlushPending
|
|
|
|
// to return true when flush_requested_ is false.
|
|
|
|
options.min_write_buffer_number_to_merge = 2;
|
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
|
|
|
ASSERT_EQ(2, handles_.size());
|
|
|
|
ASSERT_OK(dbfull()->PauseBackgroundWork());
|
|
|
|
ASSERT_OK(Put(0, "key00", "value00"));
|
|
|
|
ASSERT_OK(Put(1, "key10", "value10"));
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.wait = false;
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_opts, handles_));
|
|
|
|
ASSERT_OK(Put(0, "key01", "value01"));
|
|
|
|
// Since max_write_buffer_number is 4, the following flush won't cause write
|
|
|
|
// stall.
|
|
|
|
ASSERT_OK(dbfull()->Flush(flush_opts));
|
|
|
|
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
|
|
|
|
ASSERT_OK(dbfull()->DestroyColumnFamilyHandle(handles_[1]));
|
|
|
|
handles_[1] = nullptr;
|
|
|
|
ASSERT_OK(dbfull()->ContinueBackgroundWork());
|
|
|
|
ASSERT_OK(dbfull()->TEST_WaitForFlushMemTable(handles_[0]));
|
|
|
|
delete handles_[0];
|
|
|
|
handles_.clear();
|
|
|
|
}
|
|
|
|
|
2019-07-01 23:04:10 +02:00
|
|
|
TEST_P(DBAtomicFlushTest, CFDropRaceWithWaitForFlushMemTables) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->LoadDependency(
|
|
|
|
{{"DBImpl::AtomicFlushMemTables:AfterScheduleFlush",
|
|
|
|
"DBAtomicFlushTest::CFDropRaceWithWaitForFlushMemTables:BeforeDrop"},
|
|
|
|
{"DBAtomicFlushTest::CFDropRaceWithWaitForFlushMemTables:AfterFree",
|
|
|
|
"DBImpl::BackgroundCallFlush:start"},
|
|
|
|
{"DBImpl::BackgroundCallFlush:start",
|
|
|
|
"DBImpl::AtomicFlushMemTables:BeforeWaitForBgFlush"}});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
ASSERT_EQ(2, handles_.size());
|
|
|
|
ASSERT_OK(Put(0, "key", "value"));
|
|
|
|
ASSERT_OK(Put(1, "key", "value"));
|
|
|
|
auto* cfd_default =
|
|
|
|
static_cast<ColumnFamilyHandleImpl*>(dbfull()->DefaultColumnFamily())
|
|
|
|
->cfd();
|
|
|
|
auto* cfd_pikachu = static_cast<ColumnFamilyHandleImpl*>(handles_[1])->cfd();
|
|
|
|
port::Thread drop_cf_thr([&]() {
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBAtomicFlushTest::CFDropRaceWithWaitForFlushMemTables:BeforeDrop");
|
|
|
|
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
|
|
|
|
delete handles_[1];
|
|
|
|
handles_.resize(1);
|
|
|
|
TEST_SYNC_POINT(
|
|
|
|
"DBAtomicFlushTest::CFDropRaceWithWaitForFlushMemTables:AfterFree");
|
|
|
|
});
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
flush_opts.allow_write_stall = true;
|
|
|
|
ASSERT_OK(dbfull()->TEST_AtomicFlushMemTables({cfd_default, cfd_pikachu},
|
|
|
|
flush_opts));
|
|
|
|
drop_cf_thr.join();
|
|
|
|
Close();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
|
2020-02-07 19:50:17 +01:00
|
|
|
TEST_P(DBAtomicFlushTest, RollbackAfterFailToInstallResults) {
|
|
|
|
bool atomic_flush = GetParam();
|
|
|
|
if (!atomic_flush) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
auto fault_injection_env = std::make_shared<FaultInjectionTestEnv>(env_);
|
|
|
|
Options options = CurrentOptions();
|
|
|
|
options.env = fault_injection_env.get();
|
|
|
|
options.create_if_missing = true;
|
|
|
|
options.atomic_flush = atomic_flush;
|
|
|
|
CreateAndReopenWithCF({"pikachu"}, options);
|
|
|
|
ASSERT_EQ(2, handles_.size());
|
|
|
|
for (size_t cf = 0; cf < handles_.size(); ++cf) {
|
|
|
|
ASSERT_OK(Put(static_cast<int>(cf), "a", "value"));
|
|
|
|
}
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
|
|
|
"VersionSet::ProcessManifestWrites:BeforeWriteLastVersionEdit:0",
|
|
|
|
[&](void* /*arg*/) { fault_injection_env->SetFilesystemActive(false); });
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
FlushOptions flush_opts;
|
|
|
|
Status s = db_->Flush(flush_opts, handles_);
|
|
|
|
ASSERT_NOK(s);
|
|
|
|
fault_injection_env->SetFilesystemActive(true);
|
|
|
|
Close();
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
}
|
|
|
|
|
2020-06-04 00:53:09 +02:00
|
|
|
INSTANTIATE_TEST_CASE_P(DBFlushDirectIOTest, DBFlushDirectIOTest,
|
|
|
|
testing::Bool());
|
2017-04-13 22:07:33 +02:00
|
|
|
|
2020-06-04 00:53:09 +02:00
|
|
|
INSTANTIATE_TEST_CASE_P(DBAtomicFlushTest, DBAtomicFlushTest, testing::Bool());
|
2018-10-27 00:06:44 +02:00
|
|
|
|
2020-02-20 21:07:53 +01:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
|
|
|
|
int main(int argc, char** argv) {
|
2020-02-20 21:07:53 +01:00
|
|
|
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
|
Fix flush not being commit while writing manifest
Summary:
Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075.
The issue:
# Options.max_background_flushes > 1
# Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.)
# Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit.
# After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted.
Test Plan: run the test. Also verify the new test hit deadlock without the fix.
Reviewers: sdong, igor, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, omegaga, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D60969
2016-07-21 19:10:41 +02:00
|
|
|
::testing::InitGoogleTest(&argc, argv);
|
|
|
|
return RUN_ALL_TESTS();
|
|
|
|
}
|