Hui Xiao 49623f9c8e Account memory of big memory users in BlockBasedTable in global memory limit (#9748) ... Summary: **Context:** Through heap profiling, we discovered that `BlockBasedTableReader` objects can accumulate and lead to high memory usage (e.g, `max_open_file = -1`). These memories are currently not saved, not tracked, not constrained and not cache evict-able. As a first step to improve this, similar to https://github.com/facebook/rocksdb/pull/8428, this PR is to track an estimate of `BlockBasedTableReader` object's memory in block cache and fail future creation if the memory usage exceeds the available space of cache at the time of creation. **Summary:** - Approximate big memory users (`BlockBasedTable::Rep` and `TableProperties` )' memory usage in addition to the existing estimated ones (filter block/index block/un-compression dictionary) - Charge all of these memory usages to block cache on `BlockBasedTable::Open()` and release them on `~BlockBasedTable()` as there is no memory usage fluctuation of concern in between - Refactor on CacheReservationManager (and its call-sites) to add concurrent support for BlockBasedTable used in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9748 Test Plan: - New unit tests - db bench: `OpenDb` : **-0.52% in ms** - Setup `./db_bench -benchmarks=fillseq -db=/dev/shm/testdb -disable_auto_compactions=1 -write_buffer_size=1048576` - Repeated run with pre-change w/o feature and post-change with feature, benchmark `OpenDb`: `./db_bench -benchmarks=readrandom -use_existing_db=1 -db=/dev/shm/testdb -reserve_table_reader_memory=true (remove this when running w/o feature) -file_opening_threads=3 -open_files=-1 -report_open_timing=true| egrep 'OpenDb:'` #-run | (feature-off) avg milliseconds | std milliseconds | (feature-on) avg milliseconds | std milliseconds | change (%) -- | -- | -- | -- | -- | -- 10 | 11.4018 | 5.95173 | 9.47788 | 1.57538 | -16.87382694 20 | 9.23746 | 0.841053 | 9.32377 | 1.14074 | 0.9343477536 40 | 9.0876 | 0.671129 | 9.35053 | 1.11713 | 2.893283155 80 | 9.72514 | 2.28459 | 9.52013 | 1.0894 | -2.108041632 160 | 9.74677 | 0.991234 | 9.84743 | 1.73396 | 1.032752389 320 | 10.7297 | 5.11555 | 10.547 | 1.97692 | **-1.70275031** 640 | 11.7092 | 2.36565 | 11.7869 | 2.69377 | **0.6635807741** - db bench on write with cost to cache in WriteBufferManager (just in case this PR's CRM refactoring accidentally slows down anything in WBM) : `fillseq` : **+0.54% in micros/op** `./db_bench -benchmarks=fillseq -db=/dev/shm/testdb -disable_auto_compactions=1 -cost_write_buffer_to_cache=true -write_buffer_size=10000000000 | egrep 'fillseq'` #-run | (pre-PR) avg micros/op | std micros/op | (post-PR) avg micros/op | std micros/op | change (%) -- | -- | -- | -- | -- | -- 10 | 6.15 | 0.260187 | 6.289 | 0.371192 | 2.260162602 20 | 7.28025 | 0.465402 | 7.37255 | 0.451256 | 1.267813605 40 | 7.06312 | 0.490654 | 7.13803 | 0.478676 | **1.060579461** 80 | 7.14035 | 0.972831 | 7.14196 | 0.92971 | **0.02254791432** - filter bench: `bloom filter`: **-0.78% in ms/key** - ` ./filter_bench -impl=2 -quick -reserve_table_builder_memory=true | grep 'Build avg'` #-run | (pre-PR) avg ns/key | std ns/key | (post-PR) ns/key | std ns/key | change (%) -- | -- | -- | -- | -- | -- 10 | 26.4369 | 0.442182 | 26.3273 | 0.422919 | **-0.4145720565** 20 | 26.4451 | 0.592787 | 26.1419 | 0.62451 | **-1.1465262** - Crash test `python3 tools/db_crashtest.py blackbox --reserve_table_reader_memory=1 --cache_size=1` killed as normal Reviewed By: ajkr Differential Revision: D35136549 Pulled By: hx235 fbshipit-source-id: 146978858d0f900f43f4eb09bfd3e83195e3be28		2022-04-06 10:33:00 -07:00
..
batched_ops_stress.cc	Revise APIs related to user-defined timestamp (#8946)	2022-02-01 22:19:01 -08:00
cf_consistency_stress.cc	Add rate limiter priority to ReadOptions (#9424)	2022-02-16 23:18:14 -08:00
CMakeLists.txt	Stress test for RocksDB transactions (#8936)	2021-12-14 13:34:43 -08:00
db_stress_common.cc	db_stress: db_stress fails on custom filesystems. (#9352)	2022-01-25 16:22:58 -08:00
db_stress_common.h	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	2022-04-06 10:33:00 -07:00
db_stress_compaction_filter.h	Prevent deadlock in db_stress with DbStressCompactionFilter (#8956)	2021-09-24 16:54:02 -07:00
db_stress_driver.cc	Fix a copy-paste bug related to background threads in db_stress (#9485)	2022-02-01 15:56:17 -08:00
db_stress_driver.h	Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433)	2020-02-20 12:09:57 -08:00
db_stress_env_wrapper.h	Make the Env class Customizable (#9293)	2022-01-04 16:45:49 -08:00
db_stress_gflags.cc	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	2022-04-06 10:33:00 -07:00
db_stress_listener.cc	Avoid usage of ReopenWritableFile in db_stress (#9649)	2022-03-04 10:30:10 -08:00
db_stress_listener.h	Avoid usage of ReopenWritableFile in db_stress (#9649)	2022-03-04 10:30:10 -08:00
db_stress_shared_state.cc	Silence false alarms in db_stress fault injection (#6741)	2020-04-24 13:06:12 -07:00
db_stress_shared_state.h	Optimize db_stress setup phase (#9475)	2022-02-01 11:47:28 -08:00
db_stress_stat.cc	Fix Statistics in db_stress (#9260)	2021-12-07 16:24:22 -08:00
db_stress_stat.h	Fix Statistics in db_stress (#9260)	2021-12-07 16:24:22 -08:00
db_stress_table_properties_collector.h	Fix and detect headers with missing dependencies (#8893)	2021-09-10 10:00:26 -07:00
db_stress_test_base.cc	Account memory of big memory users in BlockBasedTable in global memory limit (#9748)	2022-04-06 10:33:00 -07:00
db_stress_test_base.h	db_stress begin tracking expected state after verification (#9470)	2022-01-31 13:35:32 -08:00
db_stress_tool.cc	db_stress: db_stress fails on custom filesystems. (#9352)	2022-01-25 16:22:58 -08:00
db_stress.cc	Add (& fix) some simple source code checks (#8821)	2021-09-07 21:19:27 -07:00
expected_state.cc	Add Temperature info in NewSequentialFile() (#9499)	2022-02-18 18:23:07 -08:00
expected_state.h	db_stress verify with lost unsynced operations (#8966)	2021-12-15 12:54:44 -08:00
multi_ops_txns_stress.cc	Improve stress test for transactions (#9568)	2022-03-16 19:00:04 -07:00
multi_ops_txns_stress.h	Improve stress test for transactions (#9568)	2022-03-16 19:00:04 -07:00
no_batched_ops_stress.cc	Added GetMergeOperands() to stress test (#9804)	2022-04-05 14:56:28 -07:00