andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
16bff5370d
rocksdb/table/block_based
History
Andrew Kryczka 75d3b6fdf0 Redesign block cache pinning API (#7520) 
Summary:
The old flag-based APIs (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`) were insufficient for our needs. For example, it was impossible to pin only unpartitioned meta-blocks, which could prevent block cache contention when turning on dictionary compression or during a migration to partitioned indexes/filters. It was also impossible to pin all meta-blocks in memory while having predictable memory usage via block cache. If we had continued adding flags to address these scenarios, they would have had significant overlap causing confusion. Instead, this PR deprecates the flags and starts a new API with non-overlapping options.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7520

Test Plan:
- new unit test
- added new options to stress/crash test and ran for a while: `$ python tools/db_crashtest.py blackbox --simple --max_key=1000000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 --interval=10 -value_size_mult=33 -column_families=1 -reopen=0`

Reviewed By: pdillinger

Differential Revision: D24200034

Pulled By: ajkr

fbshipit-source-id: 3fa7cfc71e7960f7a867511dd6ae5834dd73b13e
2020-10-11 14:58:24 -07:00
..
binary_search_index_reader.cc Separate internal and user key comparators in BlockIter (#6944) 2020-07-07 17:26:16 -07:00
binary_search_index_reader.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
block_based_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_based_filter_block.cc Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
block_based_filter_block.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
block_based_table_builder.cc Avoid to suppress status code in ~BlockBasedTableBuilder (#7507) 2020-10-05 14:58:49 -07:00
block_based_table_builder.h Store DB identity and DB session ID in SST files (#6983) 2020-06-17 10:57:40 -07:00
block_based_table_factory.cc Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
block_based_table_factory.h Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
block_based_table_iterator.cc Clean up InternalIterator upper bound logic a little bit (#7200) 2020-08-05 10:44:57 -07:00
block_based_table_iterator.h Clean up InternalIterator upper bound logic a little bit (#7200) 2020-08-05 10:44:57 -07:00
block_based_table_reader_impl.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
block_based_table_reader_test.cc Fix misspelling of PartitionedIndexIterator (#7450) 2020-09-29 16:28:13 -07:00
block_based_table_reader.cc Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
block_based_table_reader.h Add sst_file_dumper status check (#7315) 2020-09-04 19:26:42 -07:00
block_builder.cc Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_builder.h Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_prefetcher.cc Add buffer prefetch support for non directIO usecase (#7312) 2020-08-27 18:16:53 -07:00
block_prefetcher.h De-template block based table iterator (#6531) 2020-03-16 12:20:50 -07:00
block_prefix_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_prefix_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
block_type.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block.cc Fix typo: rename "bounary" to "boundary" in block.cc (#7328) 2020-09-02 20:47:18 -07:00
block.h Add EnvTestWithParam::OptionsTest to the ASSERT_STATUS_CHECKED passes (#7283) 2020-08-20 19:18:35 -07:00
cachable_entry.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
data_block_hash_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block.h Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) 2020-10-03 18:38:04 -07:00
filter_policy_internal.h Minimize memory internal fragmentation for Bloom filters (#6427) 2020-06-22 13:32:07 -07:00
filter_policy.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
flush_block_policy.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_block_policy.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block.cc Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
full_filter_block.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
hash_index_reader.cc Separate internal and user key comparators in BlockIter (#6944) 2020-07-07 17:26:16 -07:00
hash_index_reader.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
index_builder.cc Move break into block (#7468) 2020-09-30 20:24:23 -07:00
index_builder.h Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
index_reader_common.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
index_reader_common.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
mock_block_based_table.h For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) 2020-06-02 12:30:23 -07:00
parsed_full_filter_block.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
parsed_full_filter_block.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
partitioned_filter_block_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
partitioned_filter_block.cc Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) 2020-10-03 18:38:04 -07:00
partitioned_filter_block.h Revert "Return error if Get() fails in Prefetching Filter blocks (#7463)" (#7505) 2020-10-03 18:38:04 -07:00
partitioned_index_iterator.cc Fix misspelling of PartitionedIndexIterator (#7450) 2020-09-29 16:28:13 -07:00
partitioned_index_iterator.h Fix misspelling of PartitionedIndexIterator (#7450) 2020-09-29 16:28:13 -07:00
partitioned_index_reader.cc Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
partitioned_index_reader.h Get() to fail with underlying failures in PartitionIndexReader::CacheDependencies() (#7297) 2020-08-25 19:01:05 -07:00
reader_common.cc Fix block checksum for >=4GB, refactor (#6978) 2020-06-19 16:18:24 -07:00
reader_common.h Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
uncompression_dict_reader.cc Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
uncompression_dict_reader.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00