CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
// Copyright (c) 2013, Facebook, Inc. All rights reserved.
|
|
|
|
// This source code is licensed under the BSD-style license found in the
|
|
|
|
// LICENSE file in the root directory of this source tree. An additional grant
|
|
|
|
// of patent rights can be found in the PATENTS file in the same directory.
|
|
|
|
//
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
|
|
|
|
#pragma once
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
#include <vector>
|
|
|
|
#include <memory>
|
|
|
|
#include <set>
|
|
|
|
#include <unordered_set>
|
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
#include "db/version_set.h"
|
|
|
|
#include "db/compaction.h"
|
|
|
|
#include "rocksdb/status.h"
|
|
|
|
#include "rocksdb/options.h"
|
2014-02-05 01:31:18 +01:00
|
|
|
#include "rocksdb/env.h"
|
2014-10-02 01:19:16 +02:00
|
|
|
#include "util/mutable_cf_options.h"
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
#include <vector>
|
|
|
|
#include <memory>
|
|
|
|
#include <set>
|
2014-10-27 23:49:46 +01:00
|
|
|
#include <string>
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
namespace rocksdb {
|
|
|
|
|
Buffer info logs when picking compactions and write them out after releasing the mutex
Summary: Now while the background thread is picking compactions, it writes out multiple info_logs, especially for universal compaction, which introduces a chance of waiting log writing in mutex, which is bad. To remove this risk, write all those info logs to a buffer and flush it after releasing the mutex.
Test Plan:
make all check
check the log lines while running some tests that trigger compactions.
Reviewers: haobo, igor, dhruba
Reviewed By: dhruba
CC: i.am.jin.lei, dhruba, yhchiang, leveldb, nkg-
Differential Revision: https://reviews.facebook.net/D16515
2014-03-04 23:32:55 +01:00
|
|
|
class LogBuffer;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
class Compaction;
|
2014-10-27 23:49:46 +01:00
|
|
|
class VersionStorageInfo;
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
struct CompactionInputFiles;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
class CompactionPicker {
|
|
|
|
public:
|
2014-10-02 01:19:16 +02:00
|
|
|
CompactionPicker(const ImmutableCFOptions& ioptions,
|
|
|
|
const InternalKeyComparator* icmp);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
virtual ~CompactionPicker();
|
|
|
|
|
|
|
|
// Pick level and inputs for a new compaction.
|
|
|
|
// Returns nullptr if there is no compaction to be done.
|
|
|
|
// Otherwise returns a pointer to a heap-allocated object that
|
|
|
|
// describes the compaction. Caller should delete the result.
|
2014-10-27 23:49:46 +01:00
|
|
|
virtual Compaction* PickCompaction(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage,
|
|
|
|
LogBuffer* log_buffer) = 0;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
// Return a compaction object for compacting the range [begin,end] in
|
|
|
|
// the specified level. Returns nullptr if there is nothing in that
|
|
|
|
// level that overlaps the specified range. Caller should delete
|
|
|
|
// the result.
|
|
|
|
//
|
|
|
|
// The returned Compaction might not include the whole requested range.
|
|
|
|
// In that case, compaction_end will be set to the next key that needs
|
|
|
|
// compacting. In case the compaction will compact the whole range,
|
|
|
|
// compaction_end will be set to nullptr.
|
|
|
|
// Client is responsible for compaction_end storage -- when called,
|
|
|
|
// *compaction_end should point to valid InternalKey!
|
2014-10-02 01:19:16 +02:00
|
|
|
virtual Compaction* CompactRange(
|
2014-10-27 23:49:46 +01:00
|
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, int input_level, int output_level,
|
|
|
|
uint32_t output_path_id, const InternalKey* begin, const InternalKey* end,
|
2014-10-02 01:19:16 +02:00
|
|
|
InternalKey** compaction_end);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
2014-07-18 03:01:04 +02:00
|
|
|
// Given the current number of levels, returns the lowest allowed level
|
|
|
|
// for compaction input.
|
|
|
|
virtual int MaxInputLevel(int current_num_levels) const = 0;
|
|
|
|
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
// The maximum allowed output level. Default value is NumberLevels() - 1.
|
|
|
|
virtual int MaxOutputLevel() const {
|
|
|
|
return NumberLevels() - 1;
|
|
|
|
}
|
|
|
|
|
2014-11-14 00:21:04 +01:00
|
|
|
virtual bool NeedsCompaction(const VersionStorageInfo* vstorage) const = 0;
|
2014-11-13 22:41:43 +01:00
|
|
|
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
// Sanitize the input set of compaction input files.
|
|
|
|
// When the input parameters do not describe a valid compaction, the
|
|
|
|
// function will try to fix the input_files by adding necessary
|
|
|
|
// files. If it's not possible to conver an invalid input_files
|
|
|
|
// into a valid one by adding more files, the function will return a
|
|
|
|
// non-ok status with specific reason.
|
2014-11-13 22:45:33 +01:00
|
|
|
#ifndef ROCKSDB_LITE
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
Status SanitizeCompactionInputFiles(
|
|
|
|
std::unordered_set<uint64_t>* input_files,
|
|
|
|
const ColumnFamilyMetaData& cf_meta,
|
|
|
|
const int output_level) const;
|
2014-11-13 22:45:33 +01:00
|
|
|
#endif // ROCKSDB_LITE
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
// Free up the files that participated in a compaction
|
|
|
|
void ReleaseCompactionFiles(Compaction* c, Status status);
|
|
|
|
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
// Returns true if any one of the specified files are being compacted
|
|
|
|
bool FilesInCompaction(const std::vector<FileMetaData*>& files);
|
|
|
|
|
|
|
|
// Takes a list of CompactionInputFiles and returns a Compaction object.
|
|
|
|
Compaction* FormCompaction(
|
|
|
|
const CompactionOptions& compact_options,
|
|
|
|
const autovector<CompactionInputFiles>& input_files,
|
|
|
|
int output_level, VersionStorageInfo* vstorage,
|
|
|
|
const MutableCFOptions& mutable_cf_options) const;
|
|
|
|
|
|
|
|
// Converts a set of compaction input file numbers into
|
|
|
|
// a list of CompactionInputFiles.
|
|
|
|
Status GetCompactionInputsFromFileNumbers(
|
|
|
|
autovector<CompactionInputFiles>* input_files,
|
|
|
|
std::unordered_set<uint64_t>* input_set,
|
|
|
|
const VersionStorageInfo* vstorage,
|
|
|
|
const CompactionOptions& compact_options) const;
|
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
protected:
|
2014-10-02 01:19:16 +02:00
|
|
|
int NumberLevels() const { return ioptions_.num_levels; }
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
// Stores the minimal range that covers all entries in inputs in
|
|
|
|
// *smallest, *largest.
|
|
|
|
// REQUIRES: inputs is not empty
|
|
|
|
void GetRange(const std::vector<FileMetaData*>& inputs, InternalKey* smallest,
|
|
|
|
InternalKey* largest);
|
|
|
|
|
|
|
|
// Stores the minimal range that covers all entries in inputs1 and inputs2
|
|
|
|
// in *smallest, *largest.
|
|
|
|
// REQUIRES: inputs is not empty
|
|
|
|
void GetRange(const std::vector<FileMetaData*>& inputs1,
|
|
|
|
const std::vector<FileMetaData*>& inputs2,
|
|
|
|
InternalKey* smallest, InternalKey* largest);
|
|
|
|
|
2014-01-17 21:02:03 +01:00
|
|
|
// Add more files to the inputs on "level" to make sure that
|
|
|
|
// no newer version of a key is compacted to "level+1" while leaving an older
|
|
|
|
// version in a "level". Otherwise, any Get() will search "level" first,
|
|
|
|
// and will likely return an old/stale value for the key, since it always
|
|
|
|
// searches in increasing order of level to find the value. This could
|
|
|
|
// also scramble the order of merge operands. This function should be
|
|
|
|
// called any time a new Compaction is created, and its inputs_[0] are
|
|
|
|
// populated.
|
|
|
|
//
|
|
|
|
// Will return false if it is impossible to apply this compaction.
|
2014-10-27 23:49:46 +01:00
|
|
|
bool ExpandWhileOverlapping(const std::string& cf_name,
|
|
|
|
VersionStorageInfo* vstorage, Compaction* c);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
// Returns true if any one of the parent files are being compacted
|
options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically.
Summary:
When having fixed max_bytes_for_level_base, the ratio of size of largest level and the second one can range from 0 to the multiplier. This makes LSM tree frequently irregular and unpredictable. It can also cause poor space amplification in some cases.
In this improvement (proposed by Igor Kabiljo), we introduce a parameter option.level_compaction_use_dynamic_max_bytes. When turning it on, RocksDB is free to pick a level base in the range of (options.max_bytes_for_level_base/options.max_bytes_for_level_multiplier, options.max_bytes_for_level_base] so that real level ratios are close to options.max_bytes_for_level_multiplier.
Test Plan: New unit tests and pass tests suites including valgrind.
Reviewers: MarkCallaghan, rven, yhchiang, igor, ikabiljo
Reviewed By: ikabiljo
Subscribers: yoshinorim, ikabiljo, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D31437
2015-02-05 20:44:17 +01:00
|
|
|
bool RangeInCompaction(VersionStorageInfo* vstorage,
|
|
|
|
const InternalKey* smallest,
|
|
|
|
const InternalKey* largest, int level, int* index);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
2014-10-27 23:49:46 +01:00
|
|
|
void SetupOtherInputs(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, Compaction* c);
|
2014-10-02 01:19:16 +02:00
|
|
|
|
|
|
|
const ImmutableCFOptions& ioptions_;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
// A helper function to SanitizeCompactionInputFiles() that
|
|
|
|
// sanitizes "input_files" by adding necessary files.
|
2014-11-13 22:45:33 +01:00
|
|
|
#ifndef ROCKSDB_LITE
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
virtual Status SanitizeCompactionInputFilesForAllLevels(
|
|
|
|
std::unordered_set<uint64_t>* input_files,
|
|
|
|
const ColumnFamilyMetaData& cf_meta,
|
|
|
|
const int output_level) const;
|
2014-11-13 22:45:33 +01:00
|
|
|
#endif // ROCKSDB_LITE
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
|
2015-03-19 02:25:15 +01:00
|
|
|
// Keeps track of all compactions that are running on Level0.
|
|
|
|
// It is protected by DB mutex
|
|
|
|
std::set<Compaction*> level0_compactions_in_progress_;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
const InternalKeyComparator* const icmp_;
|
|
|
|
};
|
|
|
|
|
2014-11-27 00:45:11 +01:00
|
|
|
class LevelCompactionPicker : public CompactionPicker {
|
|
|
|
public:
|
|
|
|
LevelCompactionPicker(const ImmutableCFOptions& ioptions,
|
|
|
|
const InternalKeyComparator* icmp)
|
|
|
|
: CompactionPicker(ioptions, icmp) {}
|
|
|
|
virtual Compaction* PickCompaction(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage,
|
|
|
|
LogBuffer* log_buffer) override;
|
|
|
|
|
|
|
|
// Returns current_num_levels - 2, meaning the last level cannot be
|
|
|
|
// compaction input level.
|
|
|
|
virtual int MaxInputLevel(int current_num_levels) const override {
|
|
|
|
return current_num_levels - 2;
|
|
|
|
}
|
|
|
|
|
|
|
|
virtual bool NeedsCompaction(const VersionStorageInfo* vstorage) const
|
|
|
|
override;
|
|
|
|
|
2014-12-16 06:48:16 +01:00
|
|
|
// Pick a path ID to place a newly generated file, with its level
|
|
|
|
static uint32_t GetPathId(const ImmutableCFOptions& ioptions,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
int level);
|
|
|
|
|
2014-11-27 00:45:11 +01:00
|
|
|
private:
|
|
|
|
// For the specfied level, pick a compaction.
|
|
|
|
// Returns nullptr if there is no compaction to be done.
|
|
|
|
// If level is 0 and there is already a compaction on that level, this
|
|
|
|
// function will return nullptr.
|
|
|
|
Compaction* PickCompactionBySize(const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, int level,
|
|
|
|
double score);
|
|
|
|
};
|
|
|
|
|
|
|
|
#ifndef ROCKSDB_LITE
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
class UniversalCompactionPicker : public CompactionPicker {
|
|
|
|
public:
|
2014-10-02 01:19:16 +02:00
|
|
|
UniversalCompactionPicker(const ImmutableCFOptions& ioptions,
|
2014-03-11 22:52:17 +01:00
|
|
|
const InternalKeyComparator* icmp)
|
2014-10-02 01:19:16 +02:00
|
|
|
: CompactionPicker(ioptions, icmp) {}
|
2014-10-27 23:49:46 +01:00
|
|
|
virtual Compaction* PickCompaction(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage,
|
|
|
|
LogBuffer* log_buffer) override;
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
2014-07-18 03:01:04 +02:00
|
|
|
virtual int MaxInputLevel(int current_num_levels) const override {
|
2015-03-30 23:04:21 +02:00
|
|
|
return NumberLevels() - 2;
|
2014-07-18 03:01:04 +02:00
|
|
|
}
|
|
|
|
|
2015-03-30 23:04:21 +02:00
|
|
|
virtual int MaxOutputLevel() const override { return NumberLevels() - 1; }
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
|
2014-11-14 00:21:04 +01:00
|
|
|
virtual bool NeedsCompaction(const VersionStorageInfo* vstorage) const
|
|
|
|
override;
|
2014-11-13 22:41:43 +01:00
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
private:
|
2015-03-30 23:04:21 +02:00
|
|
|
struct SortedRun {
|
|
|
|
SortedRun(int _level, FileMetaData* _file, uint64_t _size,
|
|
|
|
uint64_t _compensated_file_size, bool _being_compacted)
|
|
|
|
: level(_level),
|
|
|
|
file(_file),
|
|
|
|
size(_size),
|
|
|
|
compensated_file_size(_compensated_file_size),
|
|
|
|
being_compacted(_being_compacted) {
|
|
|
|
assert(compensated_file_size > 0);
|
|
|
|
assert(level != 0 || file != nullptr);
|
|
|
|
}
|
|
|
|
|
|
|
|
void Dump(char* out_buf, size_t out_buf_size,
|
|
|
|
bool print_path = false) const;
|
|
|
|
|
|
|
|
// sorted_run_count is added into the string to print
|
|
|
|
void DumpSizeInfo(char* out_buf, size_t out_buf_size,
|
|
|
|
int sorted_run_count) const;
|
|
|
|
|
|
|
|
int level;
|
|
|
|
// `file` Will be null for level > 0. For level = 0, the sorted run is
|
|
|
|
// for this file.
|
|
|
|
FileMetaData* file;
|
|
|
|
// For level > 0, `size` and `compensated_file_size` are sum of sizes all
|
|
|
|
// files in the level. `being_compacted` should be the same for all files
|
|
|
|
// in a non-zero level. Use the value here.
|
|
|
|
uint64_t size;
|
|
|
|
uint64_t compensated_file_size;
|
|
|
|
bool being_compacted;
|
|
|
|
};
|
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
// Pick Universal compaction to limit read amplification
|
2014-10-02 01:19:16 +02:00
|
|
|
Compaction* PickCompactionUniversalReadAmp(
|
2014-10-27 23:49:46 +01:00
|
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, double score, unsigned int ratio,
|
2015-03-30 23:04:21 +02:00
|
|
|
unsigned int num_files, const std::vector<SortedRun>& sorted_runs,
|
|
|
|
LogBuffer* log_buffer);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
|
|
|
|
// Pick Universal compaction to limit space amplification.
|
2014-10-02 01:19:16 +02:00
|
|
|
Compaction* PickCompactionUniversalSizeAmp(
|
2014-10-27 23:49:46 +01:00
|
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
2015-03-30 23:04:21 +02:00
|
|
|
VersionStorageInfo* vstorage, double score,
|
|
|
|
const std::vector<SortedRun>& sorted_runs, LogBuffer* log_buffer);
|
|
|
|
|
|
|
|
static std::vector<SortedRun> CalculateSortedRuns(
|
|
|
|
const VersionStorageInfo& vstorage);
|
2014-07-15 00:34:30 +02:00
|
|
|
|
|
|
|
// Pick a path ID to place a newly generated file, with its estimated file
|
|
|
|
// size.
|
2014-10-02 01:19:16 +02:00
|
|
|
static uint32_t GetPathId(const ImmutableCFOptions& ioptions,
|
|
|
|
uint64_t file_size);
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
};
|
|
|
|
|
2014-05-21 20:43:35 +02:00
|
|
|
class FIFOCompactionPicker : public CompactionPicker {
|
|
|
|
public:
|
2014-10-02 01:19:16 +02:00
|
|
|
FIFOCompactionPicker(const ImmutableCFOptions& ioptions,
|
2014-05-21 20:43:35 +02:00
|
|
|
const InternalKeyComparator* icmp)
|
2014-10-02 01:19:16 +02:00
|
|
|
: CompactionPicker(ioptions, icmp) {}
|
2014-05-21 20:43:35 +02:00
|
|
|
|
2014-10-27 23:49:46 +01:00
|
|
|
virtual Compaction* PickCompaction(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* version,
|
|
|
|
LogBuffer* log_buffer) override;
|
2014-05-21 20:43:35 +02:00
|
|
|
|
2014-10-02 01:19:16 +02:00
|
|
|
virtual Compaction* CompactRange(
|
2014-10-27 23:49:46 +01:00
|
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, int input_level, int output_level,
|
|
|
|
uint32_t output_path_id, const InternalKey* begin, const InternalKey* end,
|
2014-10-02 01:19:16 +02:00
|
|
|
InternalKey** compaction_end) override;
|
2014-07-18 03:01:04 +02:00
|
|
|
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
// The maxinum allowed input level. Always returns 0.
|
2014-07-18 03:01:04 +02:00
|
|
|
virtual int MaxInputLevel(int current_num_levels) const override {
|
|
|
|
return 0;
|
|
|
|
}
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
|
|
|
|
// The maximum allowed output level. Always returns 0.
|
|
|
|
virtual int MaxOutputLevel() const override {
|
|
|
|
return 0;
|
|
|
|
}
|
2014-11-13 22:41:43 +01:00
|
|
|
|
2014-11-14 00:21:04 +01:00
|
|
|
virtual bool NeedsCompaction(const VersionStorageInfo* vstorage) const
|
|
|
|
override;
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
};
|
|
|
|
|
|
|
|
class NullCompactionPicker : public CompactionPicker {
|
|
|
|
public:
|
|
|
|
NullCompactionPicker(const ImmutableCFOptions& ioptions,
|
|
|
|
const InternalKeyComparator* icmp) :
|
|
|
|
CompactionPicker(ioptions, icmp) {}
|
|
|
|
virtual ~NullCompactionPicker() {}
|
|
|
|
|
|
|
|
// Always return "nullptr"
|
|
|
|
Compaction* PickCompaction(const std::string& cf_name,
|
|
|
|
const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage,
|
|
|
|
LogBuffer* log_buffer) override {
|
|
|
|
return nullptr;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Always return "nullptr"
|
|
|
|
Compaction* CompactRange(
|
|
|
|
const std::string& cf_name, const MutableCFOptions& mutable_cf_options,
|
|
|
|
VersionStorageInfo* vstorage, int input_level, int output_level,
|
|
|
|
uint32_t output_path_id, const InternalKey* begin, const InternalKey* end,
|
|
|
|
InternalKey** compaction_end) override {
|
|
|
|
return nullptr;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Given the current number of levels, returns the highest allowed level
|
|
|
|
// for compaction input.
|
2015-02-26 20:28:41 +01:00
|
|
|
virtual int MaxInputLevel(int current_num_levels) const override {
|
CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.
= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h
= EventListener =
* A virtual class that allows users to implement a set of
call-back functions which will be called when specific
events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners
= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
will try to compact those files into the specified level.
= Example =
* Example code can be found in example/compact_files_example.cc, which implements
a simple external compactor using EventListener, GetColumnFamilyMetaData, and
CompactFiles API.
Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test
Reviewers: ljin, igor, rven, sdong
Reviewed By: sdong
Subscribers: MarkCallaghan, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
|
|
|
return current_num_levels - 2;
|
|
|
|
}
|
2014-11-13 22:41:43 +01:00
|
|
|
|
|
|
|
// Always returns false.
|
2014-11-14 00:21:04 +01:00
|
|
|
virtual bool NeedsCompaction(const VersionStorageInfo* vstorage) const
|
|
|
|
override {
|
2014-11-13 22:41:43 +01:00
|
|
|
return false;
|
|
|
|
}
|
2014-05-21 20:43:35 +02:00
|
|
|
};
|
2014-11-27 00:45:11 +01:00
|
|
|
#endif // !ROCKSDB_LITE
|
2014-05-21 20:43:35 +02:00
|
|
|
|
2015-04-02 20:06:30 +02:00
|
|
|
CompressionType GetCompressionType(const ImmutableCFOptions& ioptions,
|
|
|
|
int level, int base_level,
|
|
|
|
const bool enable_compression = true);
|
|
|
|
|
CompactionPicker
Summary:
This is a big one. This diff moves all the code related to picking compactions from VersionSet to new class CompactionPicker. Column families' compactions will be completely separate processes, so we need to have multiple CompactionPickers.
To make this easier to review, most of the code change is just copy/paste. There is also a small change not to use VersionSet::current_, but rather to take `Version* version` as a parameter. Most of the other code is exactly the same.
In future diffs, I will also make some improvements to CompactionPickers. I think the most important part will be encapsulating it better. Currently Version, VersionSet, Compaction and CompactionPicker are all friend classes, which makes it harder to change the implementation.
This diff depends on D15171, D15183, D15189 and D15201
Test Plan: `make check`
Reviewers: kailiu, sdong, dhruba, haobo
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D15207
2014-01-16 22:03:52 +01:00
|
|
|
} // namespace rocksdb
|