andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/util
History
Hui Xiao 49623f9c8e Account memory of big memory users in BlockBasedTable in global memory limit (#9748) 
Summary:
**Context:**
Through heap profiling, we discovered that `BlockBasedTableReader` objects can accumulate and lead to high memory usage (e.g, `max_open_file = -1`). These memories are currently not saved, not tracked, not constrained and not cache evict-able. As a first step to improve this, similar to https://github.com/facebook/rocksdb/pull/8428,  this PR is to track an estimate of `BlockBasedTableReader` object's memory in block cache and fail future creation if the memory usage exceeds the available space of cache at the time of creation.

**Summary:**
- Approximate big memory users  (`BlockBasedTable::Rep` and `TableProperties` )' memory usage in addition to the existing estimated ones (filter block/index block/un-compression dictionary)
- Charge all of these memory usages to block cache on `BlockBasedTable::Open()` and release them on `~BlockBasedTable()` as there is no memory usage fluctuation of concern in between
- Refactor on CacheReservationManager (and its call-sites) to add concurrent support for BlockBasedTable  used in this PR.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9748

Test Plan:
- New unit tests
- db bench: `OpenDb` : **-0.52% in ms**
  - Setup `./db_bench -benchmarks=fillseq -db=/dev/shm/testdb -disable_auto_compactions=1 -write_buffer_size=1048576`
  - Repeated run with pre-change w/o feature and post-change with feature, benchmark `OpenDb`:  `./db_bench -benchmarks=readrandom -use_existing_db=1 -db=/dev/shm/testdb -reserve_table_reader_memory=true (remove this when running w/o feature) -file_opening_threads=3 -open_files=-1 -report_open_timing=true| egrep 'OpenDb:'`

#-run | (feature-off) avg milliseconds | std milliseconds | (feature-on) avg milliseconds | std milliseconds | change (%)
-- | -- | -- | -- | -- | --
10 | 11.4018 | 5.95173 | 9.47788 | 1.57538 | -16.87382694
20 | 9.23746 | 0.841053 | 9.32377 | 1.14074 | 0.9343477536
40 | 9.0876 | 0.671129 | 9.35053 | 1.11713 | 2.893283155
80 | 9.72514 | 2.28459 | 9.52013 | 1.0894 | -2.108041632
160 | 9.74677 | 0.991234 | 9.84743 | 1.73396 | 1.032752389
320 | 10.7297 | 5.11555 | 10.547 | 1.97692 | **-1.70275031**
640 | 11.7092 | 2.36565 | 11.7869 | 2.69377 | **0.6635807741**

-  db bench on write with cost to cache in WriteBufferManager (just in case this PR's CRM refactoring accidentally slows down anything in WBM) : `fillseq` : **+0.54% in micros/op**
`./db_bench -benchmarks=fillseq -db=/dev/shm/testdb -disable_auto_compactions=1 -cost_write_buffer_to_cache=true -write_buffer_size=10000000000 | egrep 'fillseq'`

#-run | (pre-PR) avg micros/op | std micros/op | (post-PR)  avg micros/op | std micros/op | change (%)
-- | -- | -- | -- | -- | --
10 | 6.15 | 0.260187 | 6.289 | 0.371192 | 2.260162602
20 | 7.28025 | 0.465402 | 7.37255 | 0.451256 | 1.267813605
40 | 7.06312 | 0.490654 | 7.13803 | 0.478676 | **1.060579461**
80 | 7.14035 | 0.972831 | 7.14196 | 0.92971 | **0.02254791432**

-  filter bench: `bloom filter`: **-0.78% in ms/key**
    - ` ./filter_bench -impl=2 -quick -reserve_table_builder_memory=true | grep 'Build avg'`

#-run | (pre-PR) avg ns/key | std ns/key | (post-PR)  ns/key | std ns/key | change (%)
-- | -- | -- | -- | -- | --
10 | 26.4369 | 0.442182 | 26.3273 | 0.422919 | **-0.4145720565**
20 | 26.4451 | 0.592787 | 26.1419 | 0.62451 | **-1.1465262**

- Crash test `python3 tools/db_crashtest.py blackbox --reserve_table_reader_memory=1 --cache_size=1` killed as normal

Reviewed By: ajkr

Differential Revision: D35136549

Pulled By: hx235

fbshipit-source-id: 146978858d0f900f43f4eb09bfd3e83195e3be28
2022-04-06 10:33:00 -07:00
..
aligned_buffer.h
Fix wrong comments about function TruncateToPageBoundary. (#6975)
2020-10-07 12:34:34 -07:00
autovector_test.cc
Replace most typedef with using= (#8751)
2021-09-07 11:31:59 -07:00
autovector.h
Add manifest fix-up utility for file temperatures (#9683)
2022-03-18 16:35:51 -07:00
bloom_impl.h
FilterPolicy API changes for 7.0 (#9501)
2022-02-08 13:56:46 -08:00
bloom_test.cc
Account memory of big memory users in BlockBasedTable in global memory limit (#9748)
2022-04-06 10:33:00 -07:00
build_version.cc.in
Make builds reproducible (#7866)
2021-01-28 17:42:16 -08:00
cast_util.h
More refactoring ahead of footer & meta changes (#9240)
2021-12-10 08:13:26 -08:00
channel.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
coding_lean.h
New stable, fixed-length cache keys (#9126)
2021-12-16 17:15:13 -08:00
coding_test.cc
Fix potential overflow of unsigned type in for loop (#6902)
2020-06-02 15:05:07 -07:00
coding.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
coding.h
More refactoring ahead of footer & meta changes (#9240)
2021-12-10 08:13:26 -08:00
compaction_job_stats_impl.cc
Update compaction statistics to include the amount of data read from blob files (#8022)
2021-03-04 00:43:48 -08:00
comparator.cc
Return different Status based on ObjectRegistry::NewObject calls (#9333)
2022-02-11 05:11:24 -08:00
compression_context_cache.cc
Remove using namespace (#9369)
2022-01-12 09:31:12 -08:00
compression_context_cache.h
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
compression.cc
Integrate WAL compression into log reader/writer. (#9642)
2022-03-09 15:49:53 -08:00
compression.h
Minor fix for Windows build with zlib (#9699)
2022-03-16 21:16:12 -07:00
concurrent_task_limiter_impl.cc
Remove TaskLimiterToken::ReleaseOnce for fix (#8567)
2021-07-21 17:37:53 -07:00
concurrent_task_limiter_impl.h
Remove TaskLimiterToken::ReleaseOnce for fix (#8567)
2021-07-21 17:37:53 -07:00
core_local.h
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
crc32c_arm64.cc
Mac M1 crc32 intrinsics ARM64 check support proposal (#7893)
2021-03-10 09:05:56 -08:00
crc32c_arm64.h
Fix compilation on Apple Silicon (#7714)
2020-12-04 15:22:33 -08:00
crc32c_ppc_asm.S
Fix Compilation on ppc64le using Clang 11 (#7713)
2020-12-01 11:21:44 -08:00
crc32c_ppc_constants.h
Remove PATENTS text from a few straggler files (#5326)
2019-05-21 16:22:35 -07:00
crc32c_ppc.c
Fix Compilation on ppc64le using Clang 11 (#7713)
2020-12-01 11:21:44 -08:00
crc32c_ppc.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
crc32c_test.cc
Implementation of Crc32c combine function (#8305)
2021-06-16 18:30:34 -07:00
crc32c.cc
Replace most typedef with using= (#8751)
2021-09-07 11:31:59 -07:00
crc32c.h
Implementation of Crc32c combine function (#8305)
2021-06-16 18:30:34 -07:00
defer_test.cc
Fix insecure internal API for GetImpl (#8590)
2021-07-29 17:23:01 -07:00
defer.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
duplicate_detector.h
Cleanup includes in dbformat.h (#8930)
2021-09-29 04:04:40 -07:00
dynamic_bloom_test.cc
Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)
2021-03-15 04:34:11 -07:00
dynamic_bloom.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
dynamic_bloom.h
Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453)
2022-01-27 14:55:04 -08:00
fastrange.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
file_checksum_helper.cc
Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)
2022-01-11 06:33:48 -08:00
file_checksum_helper.h
New stable, fixed-length cache keys (#9126)
2021-12-16 17:15:13 -08:00
file_reader_writer_test.cc
Disallow a combination of options (#9348)
2022-01-27 19:30:24 -08:00
filelock_test.cc
Fix MSVC-related build issues (#7439)
2020-10-01 09:23:04 -07:00
filter_bench.cc
Refactor FilterPolicies toward Customizable (#9567)
2022-02-16 08:30:03 -08:00
gflags_compat.h
Require C++17 (#9481)
2022-02-04 17:13:10 -08:00
hash128.h
Upgrade xxhash, add Hash128 (#8634)
2021-08-20 18:41:51 -07:00
hash_map.h
Change HashMap::Insert()'s value to a const reference (#6567)
2020-03-20 14:59:54 -07:00
hash_test.cc
New stable, fixed-length cache keys (#9126)
2021-12-16 17:15:13 -08:00
hash.cc
Experimental support for SST unique IDs (#8990)
2021-10-18 23:32:01 -07:00
hash.h
Experimental support for SST unique IDs (#8990)
2021-10-18 23:32:01 -07:00
heap_test.cc
Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923)
2020-06-03 15:55:03 -07:00
heap.h
Avoid self-move-assign in pop operation of binary heap. (#7942)
2021-02-19 13:47:25 -08:00
kv_map.h
Replace most typedef with using= (#8751)
2021-09-07 11:31:59 -07:00
log_write_bench.cc
Add a SystemClock class to capture the time functions of an Env (#7858)
2021-01-25 22:09:11 -08:00
math128.h
New stable, fixed-length cache keys (#9126)
2021-12-16 17:15:13 -08:00
math.h
Avoid popcnt on Windows when unavailable and in portable builds (#9680)
2022-03-09 21:07:31 -08:00
murmurhash.cc
C++20 compatibility (#6697)
2020-04-20 13:24:25 -07:00
murmurhash.h
Replace most typedef with using= (#8751)
2021-09-07 11:31:59 -07:00
mutexlock.h
Prevents Table Cache to open same files more times (#6707)
2020-04-21 13:16:31 -07:00
ppc-opcode.h
Remove PATENTS text from a few straggler files (#5326)
2019-05-21 16:22:35 -07:00
random_test.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
random.cc
Add De/Serialization for CompactionInput/Result (#8247)
2021-05-12 12:36:43 -07:00
random.h
Experimental support for SST unique IDs (#8990)
2021-10-18 23:32:01 -07:00
rate_limiter_test.cc
Make RateLimiter Customizable (#9141)
2021-12-01 06:57:02 -08:00
rate_limiter.cc
remove unused instance variable in GenericRateLimiter (#9484)
2022-02-01 14:04:12 -08:00
rate_limiter.h
remove unused instance variable in GenericRateLimiter (#9484)
2022-02-01 14:04:12 -08:00
repeatable_thread_test.cc
Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)
2021-03-15 04:34:11 -07:00
repeatable_thread.h
Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)
2021-03-15 04:34:11 -07:00
ribbon_alg.h
Refine Ribbon configuration, improve testing, add Homogeneous (#7879)
2021-02-26 08:50:42 -08:00
ribbon_config.cc
Refine Ribbon configuration, improve testing, add Homogeneous (#7879)
2021-02-26 08:50:42 -08:00
ribbon_config.h
Refine Ribbon configuration, improve testing, add Homogeneous (#7879)
2021-02-26 08:50:42 -08:00
ribbon_impl.h
Account Bloom/Ribbon filter construction memory in global memory limit (#9073)
2021-11-18 09:42:20 -08:00
ribbon_test.cc
Upgrade xxhash, add Hash128 (#8634)
2021-08-20 18:41:51 -07:00
set_comparator.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
slice_test.cc
Require C++17 (#9481)
2022-02-04 17:13:10 -08:00
slice_transform_test.cc
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
slice.cc
Fix LITE build for SliceTransform::AsString (#9460)
2022-01-27 16:58:22 -08:00
status.cc
Combine data members of IOStatus with Status (#9549)
2022-02-22 11:23:01 -08:00
stderr_logger.h
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
stop_watch.h
Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033)
2021-03-15 04:34:11 -07:00
string_util.cc
Remove some unneeded code (#8736)
2021-09-01 14:28:58 -07:00
string_util.h
Experimental support for SST unique IDs (#8990)
2021-10-18 23:32:01 -07:00
thread_guard.h
Introduce a ThreadGuard class and use it in ExternalSSTFileTest.PickedLevelBug (#8112)
2021-03-25 22:08:58 -07:00
thread_list_test.cc
fix thread status synchronization in thread_list_test (#7825)
2021-01-04 10:46:24 -08:00
thread_local_test.cc
Add StartThread type checking wrapper (#8303)
2021-05-19 16:51:13 -07:00
thread_local.cc
Fix typo in ThreadData comment (#7131)
2020-07-15 09:23:23 -07:00
thread_local.h
Replace most typedef with using= (#8751)
2021-09-07 11:31:59 -07:00
thread_operation.h
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
threadpool_imp.cc
Remove incremental ID from background thread pool names (#9165)
2021-11-16 18:26:12 -08:00
threadpool_imp.h
Make it able to lower cpu priority to specific level in threadpool (#6969)
2020-06-13 13:25:20 -07:00
timer_queue_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
timer_queue.h
Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)
2020-02-20 12:09:57 -08:00
timer_test.cc
Fix a timer crash caused by invalid memory management (#9656)
2022-03-12 11:45:56 -08:00
timer.h
Fix a timer crash caused by invalid memory management (#9656)
2022-03-12 11:45:56 -08:00
user_comparator_wrapper.h
Enable backward iterator for keys with user-defined timestamp (#8035)
2021-03-10 11:15:46 -08:00
vector_iterator.h
Cleanup multiple implementations of VectorIterator (#8901)
2021-10-06 07:48:31 -07:00
work_queue_test.cc
Add pipelined & parallel compression optimization (#6262)
2020-04-01 16:40:18 -07:00
work_queue.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00
xxhash.cc
Upgrade xxhash, add Hash128 (#8634)
2021-08-20 18:41:51 -07:00
xxhash.h
Upgrade xxhash, add Hash128 (#8634)
2021-08-20 18:41:51 -07:00
xxph3.h
Fix and detect headers with missing dependencies (#8893)
2021-09-10 10:00:26 -07:00