rocksdb/utilities
Zhichao Cao 699f45049d Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912)
Summary:
Background: Cache warming up will cause potential read performance degradation due to reading blocks from storage to the block cache. Since in production, the workload and access pattern to a certain DB is stable, it is a potential solution to dump out the blocks belonging to a certain DB to persist storage (e.g., to a file) and bulk-load the blocks to Secondary cache before the DB is relaunched. For example, when migrating a DB form host A to host B, it will take a short period of time, the access pattern to blocks in the block cache will not change much. It is efficient to dump out the blocks of certain DB, migrate to the destination host and insert them to the Secondary cache before we relaunch the DB.

Design: we introduce the interface of CacheDumpWriter and CacheDumpRead for user to store the blocks dumped out from block cache. RocksDB will encode all the information and send the string to the writer. User can implement their own writer it they want. CacheDumper and CacheLoad are introduced to save the blocks and load the blocks respectively.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8912

Test Plan: add new tests to lru_cache_test and pass make check.

Reviewed By: pdillinger

Differential Revision: D31452871

Pulled By: zhichao-cao

fbshipit-source-id: 11ab4f5d03e383f476947116361d54188d36ec48
2021-10-07 11:42:31 -07:00
..
backupable Finish BackupEngine migration to IOStatus (#8940) 2021-09-21 11:13:17 -07:00
blob_db Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
cassandra Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
checkpoint Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
compaction_filters Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
convenience Add a SystemClock class to capture the time functions of an Env (#7858) 2021-01-25 22:09:11 -08:00
leveldb_options Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
memory Make types of Immutable/Mutable Options fields match that of the underlying Option (#8176) 2021-04-22 20:43:54 -07:00
merge_operators Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
option_change_migration Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
options Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
persistent_cache Improve support for using regexes (#8740) 2021-09-07 13:05:23 -07:00
simulator_cache Make TraceRecord and Replayer public (#8611) 2021-08-11 19:32:46 -07:00
table_properties_collectors Make WalFilter, SstPartitionerFactory, FileChecksumGenFactory, and TableProperties Customizable (#8638) 2021-09-28 05:32:02 -07:00
trace Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
transactions Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
ttl Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
write_batch_with_index Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
cache_dump_load_impl.cc Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912) 2021-10-07 11:42:31 -07:00
cache_dump_load_impl.h Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912) 2021-10-07 11:42:31 -07:00
cache_dump_load.cc Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912) 2021-10-07 11:42:31 -07:00
compaction_filters.cc Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
debug.cc In ParseInternalKey(), include corrupt key info in Status (#7515) 2020-10-28 10:12:58 -07:00
env_librados_test.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
env_librados.cc Built-in support for generating unique IDs, bug fix (#8708) 2021-08-30 15:20:41 -07:00
env_librados.md Update branch name to main in env_librados.md (#8738) 2021-09-01 14:28:58 -07:00
env_mirror_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_mirror.cc Add new Append API with DataVerificationInfo to Env WritableFile (#8071) 2021-03-19 11:44:13 -07:00
env_timed_test.cc Make env*_test work with ASSERT_STATUS_CHECKED (#7176) 2020-07-28 22:59:48 -07:00
env_timed.cc Make ChRootEnv, EncryptedEnv, and TimedEnv into FileSystems (#7968) 2021-03-15 19:50:11 -07:00
fault_injection_env.cc No elide constructors (#7798) 2020-12-23 16:55:53 -08:00
fault_injection_env.h Fix incorrect Status::NoSpace() status check (#8504) 2021-07-20 18:09:51 -07:00
fault_injection_fs.cc FaultInjectionTestFS::InjectThreadSpecificReadError() should not corrupt mmaped bytes (#8952) 2021-09-23 12:00:47 -07:00
fault_injection_fs.h Improve fault injection to MultiRead (#8937) 2021-09-21 14:48:15 -07:00
merge_operators.cc Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
merge_operators.h Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
object_registry_test.cc Add support to the ObjectRegistry for ManagedObjects (#8658) 2021-09-10 05:21:04 -07:00
object_registry.cc Add support to the ObjectRegistry for ManagedObjects (#8658) 2021-09-10 05:21:04 -07:00
util_merge_operators_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
wal_filter.cc Make WalFilter, SstPartitionerFactory, FileChecksumGenFactory, and TableProperties Customizable (#8638) 2021-09-28 05:32:02 -07:00