rocksdb/table/block_based
Mike Kolupaev b4d7209428 Add an option to put first key of each sst block in the index (#5289)
Summary:
The first key is used to defer reading the data block until this file gets to the top of merging iterator's heap. For short range scans, most files never make it to the top of the heap, so this change can reduce read amplification by a lot sometimes.

Consider the following workload. There are a few data streams (we'll be calling them "logs"), each stream consisting of a sequence of blobs (we'll be calling them "records"). Each record is identified by log ID and a sequence number within the log. RocksDB key is concatenation of log ID and sequence number (big endian). Reads are mostly relatively short range scans, each within a single log. Writes are mostly sequential for each log, but writes to different logs are randomly interleaved. Compactions are disabled; instead, when we accumulate a few tens of sst files, we create a new column family and start writing to it.

So, a typical sst file consists of a few ranges of blocks, each range corresponding to one log ID (we use FlushBlockPolicy to cut blocks at log boundaries). A typical read would go like this. First, iterator Seek() reads one block from each sst file. Then a series of Next()s move through one sst file (since writes to each log are mostly sequential) until the subiterator reaches the end of this log in this sst file; then Next() switches to the next sst file and reads sequentially from that, and so on. Often a range scan will only return records from a small number of blocks in small number of sst files; in this case, the cost of initial Seek() reading one block from each file may be bigger than the cost of reading the actually useful blocks.

Neither iterate_upper_bound nor bloom filters can prevent reading one block from each file in Seek(). But this PR can: if the index contains first key from each block, we don't have to read the block until this block actually makes it to the top of merging iterator's heap, so for short range scans we won't read any blocks from most of the sst files.

This PR does the deferred block loading inside value() call. This is not ideal: there's no good way to report an IO error from inside value(). As discussed with siying offline, it would probably be better to change InternalIterator's interface to explicitly fetch deferred value and get status. I'll do it in a separate PR.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5289

Differential Revision: D15256423

Pulled By: al13n321

fbshipit-source-id: 750e4c39ce88e8d41662f701cf6275d9388ba46a
2019-06-24 20:54:04 -07:00
..
block_based_filter_block_test.cc Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
block_based_filter_block.cc Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
block_based_filter_block.h Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
block_based_table_builder.cc Add support for timestamp in Get/Put (#5079) 2019-06-05 23:10:47 -07:00
block_based_table_builder.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
block_based_table_factory.cc sanitize and limit block_size under 4GB (#5492) 2019-06-20 11:45:08 -07:00
block_based_table_factory.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_based_table_reader.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
block_based_table_reader.h Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
block_builder.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_builder.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_prefix_index.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
block_prefix_index.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_test.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
block_type.h Make the 'block read count' performance counters consistent (#5484) 2019-06-18 19:03:24 -07:00
block.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
block.h Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
cachable_entry.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_footer.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_footer.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_hash_index_test.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
data_block_hash_index.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_hash_index.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
filter_block.h Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
flush_block_policy.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
flush_block_policy.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
full_filter_block_test.cc Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
full_filter_block.cc Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
full_filter_block.h Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
index_builder.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
index_builder.h Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
partitioned_filter_block_test.cc Integrate block cache tracer into db_impl (#5433) 2019-06-13 15:43:10 -07:00
partitioned_filter_block.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
partitioned_filter_block.h Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00