8bf555f487
Summary:
Before this PR, Iterator/InternalIterator may simultaneously have non-ok status() and Valid() = true. That state means that the last operation failed, but the iterator is nevertheless positioned on some unspecified record. Likely intended uses of that are:
* If some sst files are corrupted, a normal iterator can be used to read the data from files that are not corrupted.
* When using read_tier = kBlockCacheTier, read the data that's in block cache, skipping over the data that is not.
However, this behavior wasn't documented well (and until recently the wiki on github had misleading incorrect information). In the code there's a lot of confusion about the relationship between status() and Valid(), and about whether Seek()/SeekToLast()/etc reset the status or not. There were a number of bugs caused by this confusion, both inside rocksdb and in the code that uses rocksdb (including ours).
This PR changes the convention to:
* If status() is not ok, Valid() always returns false.
* Any seek operation resets status. (Before the PR, it depended on iterator type and on particular error.)
This does sacrifice the two use cases listed above, but siying said it's ok.
Overview of the changes:
* A commit that adds missing status checks in MergingIterator. This fixes a bug that actually affects us, and we need it fixed. `DBIteratorTest.NonBlockingIterationBugRepro` explains the scenario.
* Changes to lots of iterator types to make all of them conform to the new convention. Some bug fixes along the way. By far the biggest changes are in DBIter, which is a big messy piece of code; I tried to make it less big and messy but mostly failed.
* A stress-test for DBIter, to gain some confidence that I didn't break it. It does a few million random operations on the iterator, while occasionally modifying the underlying data (like ForwardIterator does) and occasionally returning non-ok status from internal iterator.
To find the iterator types that needed changes I searched for "public .*Iterator" in the code. Here's an overview of all 27 iterator types:
Iterators that didn't need changes:
* status() is always ok(), or Valid() is always false: MemTableIterator, ModelIter, TestIterator, KVIter (2 classes with this name anonymous namespaces), LoggingForwardVectorIterator, VectorIterator, MockTableIterator, EmptyIterator, EmptyInternalIterator.
* Thin wrappers that always pass through Valid() and status(): ArenaWrappedDBIter, TtlIterator, InternalIteratorFromIterator.
Iterators with changes (see inline comments for details):
* DBIter - an overhaul:
- It used to silently skip corrupted keys (`FindParseableKey()`), which seems dangerous. This PR makes it just stop immediately after encountering a corrupted key, just like it would for other kinds of corruption. Let me know if there was actually some deeper meaning in this behavior and I should put it back.
- It had a few code paths silently discarding subiterator's status. The stress test caught a few.
- The backwards iteration code path was expecting the internal iterator's set of keys to be immutable. It's probably always true in practice at the moment, since ForwardIterator doesn't support backwards iteration, but this PR fixes it anyway. See added DBIteratorTest.ReverseToForwardBug for an example.
- Some parts of backwards iteration code path even did things like `assert(iter_->Valid())` after a seek, which is never a safe assumption.
- It used to not reset status on seek for some types of errors.
- Some simplifications and better comments.
- Some things got more complicated from the added error handling. I'm open to ideas for how to make it nicer.
* MergingIterator - check status after every operation on every subiterator, and in some places assert that valid subiterators have ok status.
* ForwardIterator - changed to the new convention, also slightly simplified.
* ForwardLevelIterator - fixed some bugs and simplified.
* LevelIterator - simplified.
* TwoLevelIterator - changed to the new convention. Also fixed a bug that would make SeekForPrev() sometimes silently ignore errors from first_level_iter_.
* BlockBasedTableIterator - minor changes.
* BlockIter - replaced `SetStatus()` with `Invalidate()` to make sure non-ok BlockIter is always invalid.
* PlainTableIterator - some seeks used to not reset status.
* CuckooTableIterator - tiny code cleanup.
* ManagedIterator - fixed some bugs.
* BaseDeltaIterator - changed to the new convention and fixed a bug.
* BlobDBIterator - seeks used to not reset status.
* KeyConvertingIterator - some small change.
Closes https://github.com/facebook/rocksdb/pull/3810
Differential Revision: D7888019
Pulled By: al13n321
fbshipit-source-id: 4aaf6d3421c545d16722a815b2fa2e7912bc851d
86 lines
2.4 KiB
C++
86 lines
2.4 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
#pragma once
|
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
#include <mutex>
|
|
#include <queue>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
#include "db/column_family.h"
|
|
#include "rocksdb/db.h"
|
|
#include "rocksdb/iterator.h"
|
|
#include "rocksdb/options.h"
|
|
#include "util/arena.h"
|
|
|
|
namespace rocksdb {
|
|
|
|
class DBImpl;
|
|
struct SuperVersion;
|
|
class ColumnFamilyData;
|
|
|
|
/**
|
|
* ManagedIterator is a special type of iterator that supports freeing the
|
|
* underlying iterator and still being able to access the current key/value
|
|
* pair. This is done by copying the key/value pair so that clients can
|
|
* continue to access the data without getting a SIGSEGV.
|
|
* The underlying iterator can be freed manually through the call to
|
|
* ReleaseIter or automatically (as needed on space pressure or age.)
|
|
* The iterator is recreated using the saved original arguments.
|
|
*/
|
|
class ManagedIterator : public Iterator {
|
|
public:
|
|
ManagedIterator(DBImpl* db, const ReadOptions& read_options,
|
|
ColumnFamilyData* cfd);
|
|
virtual ~ManagedIterator();
|
|
|
|
virtual void SeekToLast() override;
|
|
virtual void Prev() override;
|
|
virtual bool Valid() const override;
|
|
void SeekToFirst() override;
|
|
virtual void Seek(const Slice& target) override;
|
|
virtual void SeekForPrev(const Slice& target) override;
|
|
virtual void Next() override;
|
|
virtual Slice key() const override;
|
|
virtual Slice value() const override;
|
|
virtual Status status() const override;
|
|
void ReleaseIter(bool only_old);
|
|
void SetDropOld(bool only_old) {
|
|
only_drop_old_ = read_options_.tailing || only_old;
|
|
}
|
|
|
|
private:
|
|
void RebuildIterator(bool reseek = false);
|
|
void UpdateCurrent();
|
|
void SeekInternal(const Slice& user_key, bool seek_to_first);
|
|
bool NeedToRebuild();
|
|
void Lock();
|
|
bool TryLock();
|
|
void UnLock();
|
|
DBImpl* const db_;
|
|
ReadOptions read_options_;
|
|
ColumnFamilyData* const cfd_;
|
|
ColumnFamilyHandleInternal cfh_;
|
|
|
|
uint64_t svnum_;
|
|
std::unique_ptr<Iterator> mutable_iter_;
|
|
// internal iterator status
|
|
Status status_;
|
|
bool valid_;
|
|
|
|
IterKey cached_key_;
|
|
IterKey cached_value_;
|
|
|
|
bool only_drop_old_ = true;
|
|
bool snapshot_created_;
|
|
bool release_supported_;
|
|
std::mutex in_use_; // is managed iterator in use
|
|
};
|
|
|
|
} // namespace rocksdb
|
|
#endif // !ROCKSDB_LITE
|