andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
d13825e586
rocksdb/table
History
Hui Xiao a5063c8931 Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869) 
Summary:
**Context:**
Unit test for https://github.com/facebook/rocksdb/pull/9748 keeps opening new files to see whether the new feature is able to correctly constrain the opening based on block cache capacity.

However, the unit test has two places written inefficiently that can lead to opening too many new files relative to underlying operating system/file system constraint, even before hitting the block cache capacity:
(1) [opened_table_reader_num < 2 * max_table_reader_num](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR438), which can leads to 1200 + open files because of (2) below
(2) NewLRUCache(6 * CacheReservationManagerImpl<CacheEntryRole::kBlockBasedTableReader>::GetDummyEntrySize()) in [here](https://github.com/facebook/rocksdb/pull/9748/files?show-viewed-files=true&file-filters%5B%5D=#diff-ec9f5353e317df71093094734ba29193b94a998f0f9c9af924e4c99692195eeaR364)

Therefore we see CI failures like this on machine with a strict open file limit ~1000 (see the "table_1021" naming in following error msg)
https://app.circleci.com/pipelines/github/facebook/rocksdb/12886/workflows/75524682-3fa4-41ee-9a61-81827b51d99b/jobs/345270
```
fs_->NewWritableFile(path, foptions, &file, nullptr)
IO error: While open a file for appending: /dev/shm/rocksdb.Jedwt/run-block_based_table_reader_test-CapMemoryUsageUnderCacheCapacity-BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity-0/block_based_table_reader_test_1668910_829492452552920927/**table_1021**: Too many open files
```

**Summary:**
- Revised the test more efficiently on the above 2 places,  including using 1.1 instead 2 in the threshold and lowering down the block cache capacity a bit
- Renamed some variables for clarity

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9869

Test Plan:
- Manual inspection of max opened table reader in all test case, which is around ~389
- Circle CI to see if error is gone

Reviewed By: ajkr

Differential Revision: D35752655

Pulled By: hx235

fbshipit-source-id: 8a0953d39d561babfa4257b8ed8550bb21b04839
2022-04-19 19:02:00 -07:00
..
adaptive More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
block_based Fix issue of opening too many files in BlockBasedTableReaderCapMemoryTest.CapMemoryUsageUnderCacheCapacity (#9869) 2022-04-19 19:02:00 -07:00
cuckoo Add rate limiter priority to ReadOptions (#9424) 2022-02-16 23:18:14 -08:00
plain Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
block_fetcher_test.cc Make MemoryAllocator into a Customizable class (#8980) 2021-12-17 04:20:47 -08:00
block_fetcher.cc Fix segfault in FilePrefetchBuffer with async_io enabled (#9777) 2022-04-04 15:35:43 -07:00
block_fetcher.h More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
cleanable_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
format.cc Add rate limiter priority to ReadOptions (#9424) 2022-02-16 23:18:14 -08:00
format.h Optimize & clean up footer code (#9280) 2021-12-13 17:43:07 -08:00
get_context.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
get_context.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
internal_iterator.h Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056) 2021-11-10 16:20:04 -08:00
iter_heap.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
iterator_wrapper.h Reuse internal auto readhead_size at each Level (expect L0) for Iterations (#9056) 2021-11-10 16:20:04 -08:00
iterator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merger_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
merging_iterator.cc MergingIterator: rearrange fields to reduce paddings (#9024) 2021-10-14 12:01:56 -07:00
merging_iterator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
meta_blocks.cc Tests for filter compatibility (#9773) 2022-04-06 15:54:40 -07:00
meta_blocks.h Tests for filter compatibility (#9773) 2022-04-06 15:54:40 -07:00
mock_table.cc Add rate limiter priority to ReadOptions (#9424) 2022-02-16 23:18:14 -08:00
mock_table.h Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
multiget_context.h Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453) 2022-01-27 14:55:04 -08:00
persistent_cache_helper.cc New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
persistent_cache_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
persistent_cache_options.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
scoped_arena_iterator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
sst_file_dumper.cc New backup meta schema, with file temperatures (#9660) 2022-03-18 11:06:17 -07:00
sst_file_dumper.h New backup meta schema, with file temperatures (#9660) 2022-03-18 11:06:17 -07:00
sst_file_reader_test.cc Use the comparator from the sst file table properties in sst_dump_tool (#9491) 2022-02-08 12:15:35 -08:00
sst_file_reader.cc Fast path for detecting unchanged prefix_extractor (#9407) 2022-01-21 11:37:46 -08:00
sst_file_writer_collectors.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
sst_file_writer.cc compression_per_level should be used for flush and changeable (#9658) 2022-03-07 18:06:19 -08:00
table_builder.h Fast path for detecting unchanged prefix_extractor (#9407) 2022-01-21 11:37:46 -08:00
table_factory.cc Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362) 2022-01-11 06:33:48 -08:00
table_properties_internal.h Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
table_properties.cc Fix a bug of TEST_SetRandomTableProperties due to non-zero padding between fields in TableProperties struct (#9812) 2022-04-07 12:25:43 -07:00
table_reader_bench.cc Fast path for detecting unchanged prefix_extractor (#9407) 2022-01-21 11:37:46 -08:00
table_reader_caller.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
table_reader.h dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
table_test.cc Remove BlockBasedTableOptions.hash_index_allow_collision (#9454) 2022-03-01 13:58:02 -08:00
two_level_iterator.cc Clarify caching behavior for index and filter partitions (#9068) 2021-10-27 17:23:04 -07:00
two_level_iterator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
unique_id_impl.h Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00
unique_id.cc Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00