rocksdb/db
Mike Kolupaev 8bf555f487 Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs
Summary:
Before this PR, Iterator/InternalIterator may simultaneously have non-ok status() and Valid() = true. That state means that the last operation failed, but the iterator is nevertheless positioned on some unspecified record. Likely intended uses of that are:
 * If some sst files are corrupted, a normal iterator can be used to read the data from files that are not corrupted.
 * When using read_tier = kBlockCacheTier, read the data that's in block cache, skipping over the data that is not.

However, this behavior wasn't documented well (and until recently the wiki on github had misleading incorrect information). In the code there's a lot of confusion about the relationship between status() and Valid(), and about whether Seek()/SeekToLast()/etc reset the status or not. There were a number of bugs caused by this confusion, both inside rocksdb and in the code that uses rocksdb (including ours).

This PR changes the convention to:
 * If status() is not ok, Valid() always returns false.
 * Any seek operation resets status. (Before the PR, it depended on iterator type and on particular error.)

This does sacrifice the two use cases listed above, but siying said it's ok.

Overview of the changes:
 * A commit that adds missing status checks in MergingIterator. This fixes a bug that actually affects us, and we need it fixed. `DBIteratorTest.NonBlockingIterationBugRepro` explains the scenario.
 * Changes to lots of iterator types to make all of them conform to the new convention. Some bug fixes along the way. By far the biggest changes are in DBIter, which is a big messy piece of code; I tried to make it less big and messy but mostly failed.
 * A stress-test for DBIter, to gain some confidence that I didn't break it. It does a few million random operations on the iterator, while occasionally modifying the underlying data (like ForwardIterator does) and occasionally returning non-ok status from internal iterator.

To find the iterator types that needed changes I searched for "public .*Iterator" in the code. Here's an overview of all 27 iterator types:

Iterators that didn't need changes:
 * status() is always ok(), or Valid() is always false: MemTableIterator, ModelIter, TestIterator, KVIter (2 classes with this name anonymous namespaces), LoggingForwardVectorIterator, VectorIterator, MockTableIterator, EmptyIterator, EmptyInternalIterator.
 * Thin wrappers that always pass through Valid() and status(): ArenaWrappedDBIter, TtlIterator, InternalIteratorFromIterator.

Iterators with changes (see inline comments for details):
 * DBIter - an overhaul:
    - It used to silently skip corrupted keys (`FindParseableKey()`), which seems dangerous. This PR makes it just stop immediately after encountering a corrupted key, just like it would for other kinds of corruption. Let me know if there was actually some deeper meaning in this behavior and I should put it back.
    - It had a few code paths silently discarding subiterator's status. The stress test caught a few.
    - The backwards iteration code path was expecting the internal iterator's set of keys to be immutable. It's probably always true in practice at the moment, since ForwardIterator doesn't support backwards iteration, but this PR fixes it anyway. See added DBIteratorTest.ReverseToForwardBug for an example.
    - Some parts of backwards iteration code path even did things like `assert(iter_->Valid())` after a seek, which is never a safe assumption.
    - It used to not reset status on seek for some types of errors.
    - Some simplifications and better comments.
    - Some things got more complicated from the added error handling. I'm open to ideas for how to make it nicer.
 * MergingIterator - check status after every operation on every subiterator, and in some places assert that valid subiterators have ok status.
 * ForwardIterator - changed to the new convention, also slightly simplified.
 * ForwardLevelIterator - fixed some bugs and simplified.
 * LevelIterator - simplified.
 * TwoLevelIterator - changed to the new convention. Also fixed a bug that would make SeekForPrev() sometimes silently ignore errors from first_level_iter_.
 * BlockBasedTableIterator - minor changes.
 * BlockIter - replaced `SetStatus()` with `Invalidate()` to make sure non-ok BlockIter is always invalid.
 * PlainTableIterator - some seeks used to not reset status.
 * CuckooTableIterator - tiny code cleanup.
 * ManagedIterator - fixed some bugs.
 * BaseDeltaIterator - changed to the new convention and fixed a bug.
 * BlobDBIterator - seeks used to not reset status.
 * KeyConvertingIterator - some small change.
Closes https://github.com/facebook/rocksdb/pull/3810

Differential Revision: D7888019

Pulled By: al13n321

fbshipit-source-id: 4aaf6d3421c545d16722a815b2fa2e7912bc851d
2018-05-17 02:56:56 -07:00
..
builder.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
builder.h Stream 2017-11-10 09:26:24 -08:00
c_test.c comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
c.cc expose WAL iterator in the C API 2018-04-27 16:56:59 -07:00
column_family_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
column_family.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
column_family.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compact_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compacted_db_impl.cc Fix coverity issues column_family, compaction_db/iterator 2017-10-27 11:26:42 -07:00
compacted_db_impl.h Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_iterator.cc Blob DB: Improve FIFO eviction 2018-03-06 11:57:42 -08:00
compaction_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
compaction_job_stats_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
compaction_job_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction_job.cc MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction_job.h MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction_picker_test.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
compaction_picker_universal.cc MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction_picker_universal.h Make Universal compaction options dynamic 2017-12-11 13:27:06 -08:00
compaction_picker.cc Fix formatting in log message 2018-05-11 11:28:54 -07:00
compaction_picker.h Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
compaction.cc MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
compaction.h MaxFileSizeForLevel: adjust max_file_size for dynamic level compaction 2018-05-03 16:42:13 -07:00
comparator_db_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
convenience.cc Delete files in multiple ranges at once 2018-01-30 13:56:39 -08:00
corruption_test.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
cuckoo_table_db_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_basic_test.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc avoid double delete on dummy record insertion failure 2018-05-01 16:01:28 -07:00
db_bloom_filter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_compaction_filter_test.cc fix wrong length in snprintf 2018-03-16 13:27:55 -07:00
db_compaction_test.cc Bottommost level-based compactions in bottom-pri pool 2018-05-14 14:57:15 -07:00
db_dynamic_level_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_encryption_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_filesnapshot.cc fix behavior does not match name for "IsFileDeletionsEnabled" 2018-03-21 22:13:34 -07:00
db_flush_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_compaction_flush.cc Bottommost level-based compactions in bottom-pri pool 2018-05-14 14:57:15 -07:00
db_impl_debug.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
db_impl_experimental.cc Inform caller when rocksdb is stalling writes 2017-10-05 18:11:43 -07:00
db_impl_files.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl_open.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
db_impl_readonly.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_impl_readonly.h allowing CompactFiles to return new file names 2018-03-15 11:58:12 -07:00
db_impl_write.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_impl.cc Fix race condition between log_.erase and log_.back 2018-05-16 13:01:33 -07:00
db_impl.h Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
db_info_dumper.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Fix LITE unit tests 2017-07-26 21:11:47 -07:00
db_iter_stress_test.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
db_iter_test.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
db_iter.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
db_iter.h WritePrepared Txn: Return NotSupported on iterator refresh 2017-12-18 22:29:30 -08:00
db_iterator_test.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
db_log_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_memtable_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
db_merge_operator_test.cc Fix TSAN timeout in MergeOperatorPinningTest.Randomized/x test 2018-03-02 16:27:21 -08:00
db_options_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_properties_test.cc Add block cache related DB properties 2018-04-18 21:42:25 -07:00
db_range_del_test.cc Add max_subcompactions as a compaction option 2018-04-27 11:57:39 -07:00
db_sst_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_statistics_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_table_properties_test.cc fix deletion-triggered compaction in table builder 2017-09-28 18:17:30 -07:00
db_tailing_iter_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
db_test2.cc Apply use_direct_io_for_flush_and_compaction to writes only 2018-05-09 19:42:58 -07:00
db_test_util.cc Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
db_test_util.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_test.cc Introduce and use the option to disable stall notifications structures 2018-05-09 10:13:53 -07:00
db_universal_compaction_test.cc Bottommost level-based compactions in bottom-pri pool 2018-05-14 14:57:15 -07:00
db_wal_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
db_write_test.cc Suppress tsan lock-order-inversion on FlushWAL 2018-05-14 21:13:35 -07:00
dbformat_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
dbformat.cc add kEntryRangeDeletion 2018-04-13 11:27:17 -07:00
dbformat.h Imporve perf of random read and insert compare by suggesting inlining to the compiler 2018-03-23 13:26:55 -07:00
deletefile_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
event_helpers.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
event_helpers.h Change RocksDB License 2017-07-15 16:11:23 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
external_sst_file_ingestion_job.cc Improve accuracy of I/O stats collection of external SST ingestion. 2018-04-13 10:58:42 -07:00
external_sst_file_ingestion_job.h Initialize a boolean member variable of a struct. 2018-04-16 14:28:01 -07:00
external_sst_file_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
fault_injection_test.cc Split FaultInjectionTest.FaultTest to avoid timeout 2018-05-07 12:29:58 -07:00
file_indexer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filename_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_job_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_job.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
forward_iterator.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
forward_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
internal_stats.cc Rename vars to satisfy unity built 2018-05-04 15:28:06 -07:00
internal_stats.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
job_context.h Introduce and use the option to disable stall notifications structures 2018-05-09 10:13:53 -07:00
listener_test.cc fix some text in comments. 2018-04-10 15:59:24 -07:00
log_format.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_reader.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
log_test.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
log_writer.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
log_writer.h Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
malloc_stats.cc Disallow compactions if there isn't enough free space 2018-03-06 16:27:54 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
managed_iterator.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
managed_iterator.h Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
manual_compaction_test.cc Speedup ManualCompactionTest.Test 2018-05-03 16:13:09 -07:00
memtable_list_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable_list.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
memtable.cc WritePrepared Txn: add stats 2018-04-07 21:56:42 -07:00
memtable.h InlineSkiplist: don't decode keys unnecessarily during comparisons 2018-03-23 12:14:30 -07:00
merge_context.h Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_helper.cc WritePrepared Txn: Support merge operator 2018-02-09 14:57:54 -08:00
merge_helper.h WritePrepared Txn: Support merge operator 2018-02-09 14:57:54 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
obsolete_files_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
options_file_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
perf_context_test.cc Improve write time breakdown stats 2018-04-23 17:58:54 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
pre_release_callback.h Fix pre_release callback argument list. 2018-04-05 11:12:16 -07:00
prefix_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
range_del_aggregator_test.cc optimize file ingestion checks for range deletion overlap 2017-11-28 11:27:02 -08:00
range_del_aggregator.cc Recommit "Avoid adding tombstones of the same file to RangeDelAggregator multiple times" 2018-05-04 16:45:15 -07:00
range_del_aggregator.h Recommit "Avoid adding tombstones of the same file to RangeDelAggregator multiple times" 2018-05-04 16:45:15 -07:00
read_callback.h write-prepared txn: call IsInSnapshot 2017-09-11 09:14:48 -07:00
repair_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
repair.cc Fix formatting in log message 2018-05-11 11:28:54 -07:00
snapshot_checker.h Comment out unused variables 2018-03-05 13:13:41 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h WritePrepared Txn: smallest_prepare optimization 2018-04-02 20:27:41 -07:00
table_cache.cc Disable readahead when using mmap for reads 2018-05-08 12:13:18 -07:00
table_cache.h Change RocksDB License 2017-07-15 16:11:23 -07:00
table_properties_collector_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
table_properties_collector.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
table_properties_collector.h Comment out unused variables 2018-03-05 13:13:41 -08:00
transaction_log_impl.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
transaction_log_impl.h WritePrepared Txn: Refactor conf params 2017-11-10 17:28:12 -08:00
version_builder_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
version_builder.cc Fix some typos in comments and docs. 2018-03-08 10:27:25 -08:00
version_builder.h Allow DB reopen with reduced options.num_levels 2017-08-24 16:10:54 -07:00
version_edit_test.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_edit.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_edit.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
version_set_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
version_set.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
version_set.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
wal_manager_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
wal_manager.cc Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
wal_manager.h Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Fix some typos in comments and docs. 2018-03-08 10:27:25 -08:00
write_batch_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
write_batch.cc WritePrepared Txn: enable TryAgain for duplicates at the end of the batch 2018-04-20 15:28:19 -07:00
write_callback_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Improve write time breakdown stats 2018-04-23 17:58:54 -07:00
write_thread.h WritePrepared Txn: Duplicate Keys, Txn Part 2018-02-05 18:43:24 -08:00