rocksdb/table
Zhichao Cao 699f45049d Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912)
Summary:
Background: Cache warming up will cause potential read performance degradation due to reading blocks from storage to the block cache. Since in production, the workload and access pattern to a certain DB is stable, it is a potential solution to dump out the blocks belonging to a certain DB to persist storage (e.g., to a file) and bulk-load the blocks to Secondary cache before the DB is relaunched. For example, when migrating a DB form host A to host B, it will take a short period of time, the access pattern to blocks in the block cache will not change much. It is efficient to dump out the blocks of certain DB, migrate to the destination host and insert them to the Secondary cache before we relaunch the DB.

Design: we introduce the interface of CacheDumpWriter and CacheDumpRead for user to store the blocks dumped out from block cache. RocksDB will encode all the information and send the string to the writer. User can implement their own writer it they want. CacheDumper and CacheLoad are introduced to save the blocks and load the blocks respectively.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8912

Test Plan: add new tests to lru_cache_test and pass make check.

Reviewed By: pdillinger

Differential Revision: D31452871

Pulled By: zhichao-cao

fbshipit-source-id: 11ab4f5d03e383f476947116361d54188d36ec48
2021-10-07 11:42:31 -07:00
..
adaptive Refactor: use TableBuilderOptions to reduce parameter lists (#8240) 2021-04-29 07:00:50 -07:00
block_based Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
cuckoo Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
plain Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
block_fetcher_test.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
block_fetcher.cc Rename variables in ImmutableCFOptions to avoid conflicts with ImmutableDBOptions (#8227) 2021-04-26 12:43:45 -07:00
block_fetcher.h Add db_test2 to to ASSERT_STATUS_CHECKED (#8640) 2021-08-16 08:10:32 -07:00
cleanable_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
format.cc More clear error message on uncompressing block (#8934) 2021-09-27 10:38:16 -07:00
format.h Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
get_context.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
get_context.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
internal_iterator.h Clean up InternalIterator upper bound logic a little bit (#7200) 2020-08-05 10:44:57 -07:00
iter_heap.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
iterator_wrapper.h Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
iterator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merger_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
merging_iterator.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
merging_iterator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
meta_blocks.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
meta_blocks.h Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
mock_table.cc Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912) 2021-10-07 11:42:31 -07:00
mock_table.h Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
multiget_context.h Batch blob read IO for MultiGet (#8699) 2021-09-17 19:23:13 -07:00
persistent_cache_helper.cc Add more tests to ASSERT_STATUS_CHECKED (#7367) 2020-09-16 15:48:07 -07:00
persistent_cache_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
persistent_cache_options.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
scoped_arena_iterator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
sst_file_dumper.cc Make it possible to apply only a subrange of table property collectors (#8298) 2021-05-17 18:28:39 -07:00
sst_file_dumper.h Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
sst_file_reader_test.cc Support timestamps in SstFileWriter (#8899) 2021-09-09 18:58:01 -07:00
sst_file_reader.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
sst_file_writer_collectors.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
sst_file_writer.cc Support timestamps in SstFileWriter (#8899) 2021-09-09 18:58:01 -07:00
table_builder.h Partially revert the "apply subrange of table property collectors" change (#8465) 2021-07-06 10:14:32 -07:00
table_factory.cc Add customizable_util.h to the public API (#8301) 2021-06-29 09:08:57 -07:00
table_properties_internal.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
table_properties.cc Embed original file number in SST table properties (#8686) 2021-08-20 20:40:48 -07:00
table_reader_bench.cc Make it possible to apply only a subrange of table property collectors (#8298) 2021-05-17 18:28:39 -07:00
table_reader_caller.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
table_reader.h dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
table_test.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
two_level_iterator.cc Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
two_level_iterator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00