andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
a5063c8931
rocksdb/table/block_based
History
Hui Xiao a5063c8931 Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869) 
Summary:
**Context:**
Unit test for https://github.com/facebook/rocksdb/pull/9748 keeps opening new files to see whether the new feature is able to correctly constrain the opening based on block cache capacity.

However, the unit test has two places written inefficiently that can lead to opening too many new files relative to underlying operating system/file system constraint, even before hitting the block cache capacity:
(1) [opened_table_reader_num < 2 * max_table_reader_num](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR438), which can leads to 1200 + open files because of (2) below
(2) NewLRUCache(6 * CacheReservationManagerImpl<CacheEntryRole::kBlockBasedTableReader>::GetDummyEntrySize()) in [here](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR364)

Therefore we see CI failures like this on machine with a strict open file limit ~1000 (see the "table_1021" naming in following error msg)
https://app.circleci.com/pipelines/github/facebook/rocksdb/12886/workflows/75524682-3fa4-41ee-9a61-81827b51d99b/jobs/345270
```
fs_->NewWritableFile(path, foptions, &file, nullptr)
IO error: While open a file for appending: /dev/shm/rocksdb.Jedwt/run-block_based_table_reader_test-CapMemoryUsageUnderCacheCapacity-BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity-0/block_based_table_reader_test_1668910_829492452552920927/**table_1021**: Too many open files
```

**Summary:**
- Revised the test more efficiently on the above 2 places,  including using 1.1 instead 2 in the threshold and lowering down the block cache capacity a bit
- Renamed some variables for clarity

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9869

Test Plan:
- Manual inspection of max opened table reader in all test case, which is around ~389
- Circle CI to see if error is gone

Reviewed By: ajkr

Differential Revision: D35752655

Pulled By: hx235

fbshipit-source-id: 8a0953d39d561babfa4257b8ed8550bb21b04839
2022-04-19 19:02:00 -07:00
..
binary_search_index_reader.cc Separate internal and user key comparators in BlockIter (#6944) 2020-07-07 17:26:16 -07:00
binary_search_index_reader.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
block_based_filter_block_test.cc FilterPolicy API changes for 7.0 (#9501) 2022-02-08 13:56:46 -08:00
block_based_filter_block.cc Refactor FilterPolicies toward Customizable (#9567) 2022-02-16 08:30:03 -08:00
block_based_filter_block.h FilterPolicy API changes for 7.0 (#9501) 2022-02-08 13:56:46 -08:00
block_based_table_builder.cc Account memory of big memory users in BlockBasedTable in global memory limit (#9748) 2022-04-06 10:33:00 -07:00
block_based_table_builder.h Fix segmentation fault in table_options.prepopulate_block_cache when used with partition_filters (#9263) 2021-12-08 12:44:38 -08:00
block_based_table_factory.cc Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
block_based_table_factory.h Account memory of big memory users in BlockBasedTable in global memory limit (#9748) 2022-04-06 10:33:00 -07:00
block_based_table_iterator.cc Provide implementation to prefetch data asynchronously in FilePrefetchBuffer (#9674) 2022-03-21 07:12:43 -07:00
block_based_table_iterator.h Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
block_based_table_reader_impl.h Fb 9718 verify checksums is ignored (#9767) 2022-03-29 11:54:54 -07:00
block_based_table_reader_test.cc Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869) 2022-04-19 19:02:00 -07:00
block_based_table_reader.cc Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
block_based_table_reader.h Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
block_builder.cc Improve data block construction performance (#9040) 2021-10-19 12:36:21 -07:00
block_builder.h Improve data block construction performance (#9040) 2021-10-19 12:36:21 -07:00
block_like_traits.h Change type of cache buffer passed to Cache::CreateCallback() to const void* (#9595) 2022-02-17 21:09:56 -08:00
block_prefetcher.cc Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
block_prefetcher.h Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
block_prefix_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_prefix_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
block_type.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
block.cc Add NewMetaDataIterator method (#8692) 2021-12-21 11:32:49 -08:00
block.h Add NewMetaDataIterator method (#8692) 2021-12-21 11:32:49 -08:00
cachable_entry.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
data_block_footer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index_test.cc Fast path for detecting unchanged prefix_extractor (#9407) 2022-01-21 11:37:46 -08:00
data_block_hash_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.cc Parallelize secondary cache lookup in MultiGet (#8405) 2021-06-18 09:35:59 -07:00
filter_block_reader_common.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block.h Detect (new) Bloom/Ribbon Filter construction corruption (#9342) 2022-02-01 17:42:35 -08:00
filter_policy_internal.h Fix a major performance bug in 7.0 re: filter compatibility (#9736) 2022-03-23 10:00:54 -07:00
filter_policy.cc Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
flush_block_policy.cc Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362) 2022-01-11 06:33:48 -08:00
flush_block_policy.h Make FlushBlockPolicyFactory into a Customizable class (#8432) 2021-07-12 09:04:59 -07:00
full_filter_block_test.cc Fix a major performance bug in 7.0 re: filter compatibility (#9736) 2022-03-23 10:00:54 -07:00
full_filter_block.cc Detect (new) Bloom/Ribbon Filter construction corruption (#9342) 2022-02-01 17:42:35 -08:00
full_filter_block.h Hide FilterBits{Builder,Reader} from public API (#9592) 2022-02-17 16:34:46 -08:00
hash_index_reader.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
hash_index_reader.h Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
index_builder.cc Add (& fix) some simple source code checks (#8821) 2021-09-07 21:19:27 -07:00
index_builder.h Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
index_reader_common.cc Parallelize secondary cache lookup in MultiGet (#8405) 2021-06-18 09:35:59 -07:00
index_reader_common.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
mock_block_based_table.h Refactor FilterPolicies toward Customizable (#9567) 2022-02-16 08:30:03 -08:00
parsed_full_filter_block.cc Hide FilterBits{Builder,Reader} from public API (#9592) 2022-02-17 16:34:46 -08:00
parsed_full_filter_block.h Use new Insert and Lookup APIs in table reader to support secondary cache (#8315) 2021-05-21 18:29:12 -07:00
partitioned_filter_block_test.cc More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
partitioned_filter_block.cc Enable READ_BLOCK_COMPACTION_MICROS to track stats (#9722) 2022-03-24 15:06:24 -07:00
partitioned_filter_block.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
partitioned_index_iterator.cc Provide implementation to prefetch data asynchronously in FilePrefetchBuffer (#9674) 2022-03-21 07:12:43 -07:00
partitioned_index_iterator.h Make initial auto readahead_size configurable (#9836) 2022-04-15 17:28:09 -07:00
partitioned_index_reader.cc Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
partitioned_index_reader.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
reader_common.cc Update Cache::Release param from force_erase to erase_if_last_ref (#9728) 2022-03-22 10:22:18 -07:00
reader_common.h Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
uncompression_dict_reader.cc Fb 9718 verify checksums is ignored (#9767) 2022-03-29 11:54:54 -07:00
uncompression_dict_reader.h Fb 9718 verify checksums is ignored (#9767) 2022-03-29 11:54:54 -07:00