e45673dece
Summary: Context: Index type `kBinarySearchWithFirstKey` added the ability for sst file iterator to sometimes report a key from index without reading the corresponding data block. This is useful when sst blocks are cut at some meaningful boundaries (e.g. one block per key prefix), and many seeks land between blocks (e.g. for each prefix, the ranges of keys in different sst files are nearly disjoint, so a typical seek needs to read a data block from only one file even if all files have the prefix). But this added a new error condition, which rocksdb code was really not equipped to deal with: `InternalIterator::value()` may fail with an IO error or Status::Incomplete, but it's just a method returning a Slice, with no way to report error instead. Before this PR, this type of error wasn't handled at all (an empty slice was returned), and kBinarySearchWithFirstKey implementation was considered a prototype. Now that we (LogDevice) have experimented with kBinarySearchWithFirstKey for a while and confirmed that it's really useful, this PR is adding the missing error handling. It's a pretty inconvenient situation implementation-wise. The error needs to be reported from InternalIterator when trying to access value. But there are ~700 call sites of `InternalIterator::value()`, most of which either can't hit the error condition (because the iterator is reading from memtable or from index or something) or wouldn't benefit from the deferred loading of the value (e.g. compaction iterator that reads all values anyway). Adding error handling to all these call sites would needlessly bloat the code. So instead I made the deferred value loading optional: only the call sites that may use deferred loading have to call the new method `PrepareValue()` before calling `value()`. The feature is enabled with a new bool argument `allow_unprepared_value` to a bunch of methods that create iterators (it wouldn't make sense to put it in ReadOptions because it's completely internal to iterators, with virtually no user-visible effect). Lmk if you have better ideas. Note that the deferred value loading only happens for *internal* iterators. The user-visible iterator (DBIter) always prepares the value before returning from Seek/Next/etc. We could go further and add an API to defer that value loading too, but that's most likely not useful for LogDevice, so it doesn't seem worth the complexity for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6621 Test Plan: make -j5 check . Will also deploy to some logdevice test clusters and look at stats. Reviewed By: siying Differential Revision: D20786930 Pulled By: al13n321 fbshipit-source-id: 6da77d918bad3780522e918f17f4d5513d3e99ee
139 lines
5.9 KiB
C++
139 lines
5.9 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#pragma once
|
|
#include <memory>
|
|
#include "db/range_tombstone_fragmenter.h"
|
|
#include "rocksdb/slice_transform.h"
|
|
#include "table/get_context.h"
|
|
#include "table/internal_iterator.h"
|
|
#include "table/multiget_context.h"
|
|
#include "table/table_reader_caller.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
class Iterator;
|
|
struct ParsedInternalKey;
|
|
class Slice;
|
|
class Arena;
|
|
struct ReadOptions;
|
|
struct TableProperties;
|
|
class GetContext;
|
|
class MultiGetContext;
|
|
|
|
// A Table (also referred to as SST) is a sorted map from strings to strings.
|
|
// Tables are immutable and persistent. A Table may be safely accessed from
|
|
// multiple threads without external synchronization. Table readers are used
|
|
// for reading various types of table formats supported by rocksdb including
|
|
// BlockBasedTable, PlainTable and CuckooTable format.
|
|
class TableReader {
|
|
public:
|
|
virtual ~TableReader() {}
|
|
|
|
// Returns a new iterator over the table contents.
|
|
// The result of NewIterator() is initially invalid (caller must
|
|
// call one of the Seek methods on the iterator before using it).
|
|
// arena: If not null, the arena needs to be used to allocate the Iterator.
|
|
// When destroying the iterator, the caller will not call "delete"
|
|
// but Iterator::~Iterator() directly. The destructor needs to destroy
|
|
// all the states but those allocated in arena.
|
|
// skip_filters: disables checking the bloom filters even if they exist. This
|
|
// option is effective only for block-based table format.
|
|
// compaction_readahead_size: its value will only be used if caller =
|
|
// kCompaction
|
|
virtual InternalIterator* NewIterator(
|
|
const ReadOptions&, const SliceTransform* prefix_extractor, Arena* arena,
|
|
bool skip_filters, TableReaderCaller caller,
|
|
size_t compaction_readahead_size = 0,
|
|
bool allow_unprepared_value = false) = 0;
|
|
|
|
virtual FragmentedRangeTombstoneIterator* NewRangeTombstoneIterator(
|
|
const ReadOptions& /*read_options*/) {
|
|
return nullptr;
|
|
}
|
|
|
|
// Given a key, return an approximate byte offset in the file where
|
|
// the data for that key begins (or would begin if the key were
|
|
// present in the file). The returned value is in terms of file
|
|
// bytes, and so includes effects like compression of the underlying data.
|
|
// E.g., the approximate offset of the last key in the table will
|
|
// be close to the file length.
|
|
virtual uint64_t ApproximateOffsetOf(const Slice& key,
|
|
TableReaderCaller caller) = 0;
|
|
|
|
// Given start and end keys, return the approximate data size in the file
|
|
// between the keys. The returned value is in terms of file bytes, and so
|
|
// includes effects like compression of the underlying data.
|
|
virtual uint64_t ApproximateSize(const Slice& start, const Slice& end,
|
|
TableReaderCaller caller) = 0;
|
|
|
|
// Set up the table for Compaction. Might change some parameters with
|
|
// posix_fadvise
|
|
virtual void SetupForCompaction() = 0;
|
|
|
|
virtual std::shared_ptr<const TableProperties> GetTableProperties() const = 0;
|
|
|
|
// Prepare work that can be done before the real Get()
|
|
virtual void Prepare(const Slice& /*target*/) {}
|
|
|
|
// Report an approximation of how much memory has been used.
|
|
virtual size_t ApproximateMemoryUsage() const = 0;
|
|
|
|
// Calls get_context->SaveValue() repeatedly, starting with
|
|
// the entry found after a call to Seek(key), until it returns false.
|
|
// May not make such a call if filter policy says that key is not present.
|
|
//
|
|
// get_context->MarkKeyMayExist needs to be called when it is configured to be
|
|
// memory only and the key is not found in the block cache.
|
|
//
|
|
// readOptions is the options for the read
|
|
// key is the key to search for
|
|
// skip_filters: disables checking the bloom filters even if they exist. This
|
|
// option is effective only for block-based table format.
|
|
virtual Status Get(const ReadOptions& readOptions, const Slice& key,
|
|
GetContext* get_context,
|
|
const SliceTransform* prefix_extractor,
|
|
bool skip_filters = false) = 0;
|
|
|
|
virtual void MultiGet(const ReadOptions& readOptions,
|
|
const MultiGetContext::Range* mget_range,
|
|
const SliceTransform* prefix_extractor,
|
|
bool skip_filters = false) {
|
|
for (auto iter = mget_range->begin(); iter != mget_range->end(); ++iter) {
|
|
*iter->s = Get(readOptions, iter->ikey, iter->get_context,
|
|
prefix_extractor, skip_filters);
|
|
}
|
|
}
|
|
|
|
// Prefetch data corresponding to a give range of keys
|
|
// Typically this functionality is required for table implementations that
|
|
// persists the data on a non volatile storage medium like disk/SSD
|
|
virtual Status Prefetch(const Slice* begin = nullptr,
|
|
const Slice* end = nullptr) {
|
|
(void) begin;
|
|
(void) end;
|
|
// Default implementation is NOOP.
|
|
// The child class should implement functionality when applicable
|
|
return Status::OK();
|
|
}
|
|
|
|
// convert db file to a human readable form
|
|
virtual Status DumpTable(WritableFile* /*out_file*/) {
|
|
return Status::NotSupported("DumpTable() not supported");
|
|
}
|
|
|
|
// check whether there is corruption in this db file
|
|
virtual Status VerifyChecksum(const ReadOptions& /*read_options*/,
|
|
TableReaderCaller /*caller*/) {
|
|
return Status::NotSupported("VerifyChecksum() not supported");
|
|
}
|
|
};
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|