rocksdb/table/block_based
Zhichao Cao cddd637997 Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089)
Summary:
In the current MultiGet, if the KV-pairs do not belong to the data blocks in the block cache, multiple blocks are read from a SST. It will trigger one block read for each block request and read them in parallel. In some cases, if some data blocks are adjacent in the SST, the reads for these blocks can be combined to a single large read, which can reduce the system calls and reduce the read latency if possible.

Considering to fill the block cache, if multiple data blocks are in the same memory buffer, we need to copy them to the heap separately. Therefore, only in the case that 1) data block compression is enabled, and 2) compressed block cache is null, we can do combined read. Otherwise, extra memory copy is needed, which may cause extra overhead. In the current case, data blocks will be uncompressed to a new memory space.

Also, in the case that 1) data block compression is enabled, and 2) compressed block cache is null, it is possible the data block is actually not compressed. In the current logic, these data blocks will not be added to the uncompressed_cache. So if memory buffer is shared and the data block is not compressed, the data block are copied to the head and fill the cache.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6089

Test Plan: Added test case to ParallelIO.MultiGet. Pass make asan_check

Differential Revision: D18734668

Pulled By: zhichao-cao

fbshipit-source-id: 67c5615ed373e51e42635fd74b36f8f3a66d5da4
2019-12-16 16:26:03 -08:00
..
block_based_filter_block_test.cc Prepare filter tests for more implementations (#5967) 2019-10-31 14:12:33 -07:00
block_based_filter_block.cc Fix regression affecting partitioned indexes/filters when cache_index_and_filter_blocks is false (#5705) 2019-08-14 18:16:06 -07:00
block_based_filter_block.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
block_based_table_builder.cc Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) 2019-12-16 16:26:03 -08:00
block_based_table_builder.h Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
block_based_table_factory.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
block_based_table_factory.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_based_table_reader.cc Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) 2019-12-16 16:26:03 -08:00
block_based_table_reader.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
block_builder.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_builder.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_prefix_index.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
block_prefix_index.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
block_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
block_type.h Make the 'block read count' performance counters consistent (#5484) 2019-06-18 19:03:24 -07:00
block.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
block.h Add class comment for Block 2019-09-24 11:02:11 -07:00
cachable_entry.h Move the filter readers out of the block cache (#5504) 2019-07-16 13:14:58 -07:00
data_block_footer.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_footer.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_hash_index_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
data_block_hash_index.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
data_block_hash_index.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
filter_block_reader_common.cc Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
filter_block_reader_common.h Fix regression affecting partitioned indexes/filters when cache_index_and_filter_blocks is false (#5705) 2019-08-14 18:16:06 -07:00
filter_block.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
filter_policy_internal.h Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
filter_policy.cc Optimize memory and CPU for building new Bloom filter (#6175) 2019-12-15 21:31:08 -08:00
flush_block_policy.cc Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
flush_block_policy.h Organizing rocksdb/table directory by format 2019-05-30 14:51:11 -07:00
full_filter_block_test.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
full_filter_block.cc Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
full_filter_block.h Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
index_builder.cc Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
index_builder.h Add an option to put first key of each sst block in the index (#5289) 2019-06-24 20:54:04 -07:00
mock_block_based_table.h Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
parsed_full_filter_block.cc Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
parsed_full_filter_block.h New Bloom filter implementation for full and partitioned filters (#6007) 2019-11-13 16:44:01 -08:00
partitioned_filter_block_test.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
partitioned_filter_block.cc Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
partitioned_filter_block.h Store the filter bits reader alongside the filter block contents (#5936) 2019-10-18 19:32:59 -07:00
uncompression_dict_reader.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
uncompression_dict_reader.h Revert to storing UncompressionDicts in the cache (#5645) 2019-08-23 08:27:30 -07:00