rocksdb/include/rocksdb/metadata.h
Venki Pallipadi 22ce462450 Export Import sst files (#5495)
Summary:
Refresh of the earlier change here - https://github.com/facebook/rocksdb/issues/5135

This is a review request for code change needed for - https://github.com/facebook/rocksdb/issues/3469
"Add support for taking snapshot of a column family and creating column family from a given CF snapshot"

We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality.

(1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below.
// Exports all live SST files of a specified Column Family onto export_dir,
// returning SST files information in metadata.
// - SST files will be created as hard links when the directory specified
//   is in the same partition as the db directory, copied otherwise.
// - export_dir should not already exist and will be created by this API.
// - Always triggers a flush.
virtual Status ExportColumnFamily(ColumnFamilyHandle* handle,
                                  const std::string& export_dir,
                                  ExportImportFilesMetaData** metadata);

Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through
metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by
returning the list of file metadata.

(2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below.
// CreateColumnFamilyWithImport() will create a new column family with
// column_family_name and import external SST files specified in metadata into
// this column family.
// (1) External SST files can be created using SstFileWriter.
// (2) External SST files can be exported from a particular column family in
//     an existing DB.
// Option in import_options specifies whether the external files are copied or
// moved (default is copy). When option specifies copy, managing files at
// external_file_path is caller's responsibility. When option specifies a
// move, the call ensures that the specified files at external_file_path are
// deleted on successful return and files are not modified on any error
// return.
// On error return, column family handle returned will be nullptr.
// ColumnFamily will be present on successful return and will not be present
// on error return. ColumnFamily may be present on any crash during this call.
virtual Status CreateColumnFamilyWithImport(
    const ColumnFamilyOptions& options, const std::string& column_family_name,
    const ImportColumnFamilyOptions& import_options,
    const ExportImportFilesMetaData& metadata,
    ColumnFamilyHandle** handle);

Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported.

If incoming sequence number is higher than current local sequence number, local sequence
number is updated to reflect this.

Note, as the sst files is are being moved across Column Families, Column Family name in sst file
will no longer match the actual column family on destination DB. The API does not modify Column
Family name or id in the sst files being imported.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5495

Differential Revision: D16018881

fbshipit-source-id: 9ae2251025d5916d35a9fc4ea4d6707f6be16ff9
2019-07-17 12:27:14 -07:00

119 lines
3.9 KiB
C++

// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#include <stdint.h>
#include <limits>
#include <string>
#include <vector>
#include "rocksdb/types.h"
namespace rocksdb {
struct ColumnFamilyMetaData;
struct LevelMetaData;
struct SstFileMetaData;
// The metadata that describes a column family.
struct ColumnFamilyMetaData {
ColumnFamilyMetaData() : size(0), file_count(0), name("") {}
ColumnFamilyMetaData(const std::string& _name, uint64_t _size,
const std::vector<LevelMetaData>&& _levels)
: size(_size), name(_name), levels(_levels) {}
// The size of this column family in bytes, which is equal to the sum of
// the file size of its "levels".
uint64_t size;
// The number of files in this column family.
size_t file_count;
// The name of the column family.
std::string name;
// The metadata of all levels in this column family.
std::vector<LevelMetaData> levels;
};
// The metadata that describes a level.
struct LevelMetaData {
LevelMetaData(int _level, uint64_t _size,
const std::vector<SstFileMetaData>&& _files)
: level(_level), size(_size), files(_files) {}
// The level which this meta data describes.
const int level;
// The size of this level in bytes, which is equal to the sum of
// the file size of its "files".
const uint64_t size;
// The metadata of all sst files in this level.
const std::vector<SstFileMetaData> files;
};
// The metadata that describes a SST file.
struct SstFileMetaData {
SstFileMetaData()
: size(0),
name(""),
db_path(""),
smallest_seqno(0),
largest_seqno(0),
smallestkey(""),
largestkey(""),
num_reads_sampled(0),
being_compacted(false),
num_entries(0),
num_deletions(0) {}
SstFileMetaData(const std::string& _file_name, const std::string& _path,
size_t _size, SequenceNumber _smallest_seqno,
SequenceNumber _largest_seqno,
const std::string& _smallestkey,
const std::string& _largestkey, uint64_t _num_reads_sampled,
bool _being_compacted)
: size(_size),
name(_file_name),
db_path(_path),
smallest_seqno(_smallest_seqno),
largest_seqno(_largest_seqno),
smallestkey(_smallestkey),
largestkey(_largestkey),
num_reads_sampled(_num_reads_sampled),
being_compacted(_being_compacted),
num_entries(0),
num_deletions(0) {}
// File size in bytes.
size_t size;
// The name of the file.
std::string name;
// The full path where the file locates.
std::string db_path;
SequenceNumber smallest_seqno; // Smallest sequence number in file.
SequenceNumber largest_seqno; // Largest sequence number in file.
std::string smallestkey; // Smallest user defined key in the file.
std::string largestkey; // Largest user defined key in the file.
uint64_t num_reads_sampled; // How many times the file is read.
bool being_compacted; // true if the file is currently being compacted.
uint64_t num_entries;
uint64_t num_deletions;
};
// The full set of metadata associated with each SST file.
struct LiveFileMetaData : SstFileMetaData {
std::string column_family_name; // Name of the column family
int level; // Level at which this file resides.
LiveFileMetaData() : column_family_name(), level(0) {}
};
// Metadata returned as output from ExportColumnFamily() and used as input to
// CreateColumnFamiliesWithImport().
struct ExportImportFilesMetaData {
std::string db_comparator_name; // Used to safety check at import.
std::vector<LiveFileMetaData> files; // Vector of file metadata.
};
} // namespace rocksdb