andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
cddd637997
rocksdb/db
History
Zhichao Cao cddd637997 Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) 
Summary:
In the current MultiGet, if the KV-pairs do not belong to the data blocks in the block cache, multiple blocks are read from a SST. It will trigger one block read for each block request and read them in parallel. In some cases, if some data blocks are adjacent in the SST, the reads for these blocks can be combined to a single large read, which can reduce the system calls and reduce the read latency if possible.

Considering to fill the block cache, if multiple data blocks are in the same memory buffer, we need to copy them to the heap separately. Therefore, only in the case that 1) data block compression is enabled, and 2) compressed block cache is null, we can do combined read. Otherwise, extra memory copy is needed, which may cause extra overhead. In the current case, data blocks will be uncompressed to a new memory space.

Also, in the case that 1) data block compression is enabled, and 2) compressed block cache is null, it is possible the data block is actually not compressed. In the current logic, these data blocks will not be added to the uncompressed_cache. So if memory buffer is shared and the data block is not compressed, the data block are copied to the head and fill the cache.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6089

Test Plan: Added test case to ParallelIO.MultiGet. Pass make asan_check

Differential Revision: D18734668

Pulled By: zhichao-cao

fbshipit-source-id: 67c5615ed373e51e42635fd74b36f8f3a66d5da4
2019-12-16 16:26:03 -08:00
..
compaction Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
db_impl Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
arena_wrapped_db_iter.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
arena_wrapped_db_iter.h Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
blob_index.h Mark BlobIndex::DebugString const 2019-12-11 17:19:43 -08:00
builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
c_test.c Fix c_test:filter for various CACHE_LINE_SIZEs (#6153) 2019-12-11 15:17:08 -08:00
c.cc Apply formatter to some recent commits (#6138) 2019-12-09 15:49:49 -08:00
column_family_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
column_family.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
compact_files_test.cc Use aggregate initialization for FlushJobInfo/CompactionJobInfo (#5997) 2019-11-01 11:46:19 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
corruption_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
cuckoo_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
db_basic_test.cc Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) 2019-12-16 16:26:03 -08:00
db_blob_index_test.cc Disable blob iterator test with max_sequential_skip_in_iterations==0 in LITE mode (#6052) 2019-11-19 15:02:41 -08:00
db_block_cache_test.cc Charge block cache for cache internal usage (#5797) 2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
db_flush_test.cc Fix a potential bug scheduling unnecessary threads (#6104) 2019-11-27 14:48:49 -08:00
db_info_dumper.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Fix blob context when db_iter uses seek (#6051) 2019-11-19 11:39:02 -08:00
db_iter.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
db_iterator_test.cc Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154) 2019-12-13 14:30:14 -08:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_options_test.cc Make default value of options.ttl to be 30 days when it is supported. (#6073) 2019-11-26 10:00:32 -08:00
db_properties_test.cc Refactor/consolidate legacy Bloom implementation details (#5784) 2019-09-16 16:17:09 -07:00
db_range_del_test.cc Fix RangeDeletion bug (#6062) 2019-12-12 15:18:02 -08:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Fix a regression bug on total order seek with prefix enabled and range delete (#6028) 2019-11-13 10:11:34 -08:00
db_test_util.cc Batched MultiGet API for multiple column families (#5816) 2019-11-12 13:52:55 -08:00
db_test_util.h Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
db_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
db_universal_compaction_test.cc Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125) 2019-12-09 09:23:50 -08:00
db_wal_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
deletefile_test.cc Refactor deletefile_test.cc (#5822) 2019-09-18 16:58:21 -07:00
error_handler_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
event_helpers.h BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_ingestion_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
import_column_family_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.h Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Propagate SST and blob file numbers through the EventListener interface (#5962) 2019-10-24 14:44:15 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_reader.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
log_writer.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
memtable_list.cc Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
memtable_list.h Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
memtable.cc Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00
memtable.h Do not schedule memtable trimming if there is no history (#6177) 2019-12-13 19:11:19 -08:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) 2019-09-18 11:52:17 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
repair.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_cache.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_properties_collector_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_edit_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.h Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_set_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_set.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_set.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch.cc Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00