rocksdb/file
Mike Kolupaev 1ed7d9b1b5 Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363)
Summary:
Before this PR it calls GetFileSize() once for each sst file in the DB. This can take a long time if there are be tens of thousands of sst files (e.g. in thousands of column families), and even longer if Env is talking to some remote service rather than local filesystem. This PR makes DB::Open() use sst file sizes that are already known from manifest (typically almost all files in the DB) and only call GetFileSize() for non-sst or obsolete files. Note that GetFileSize() is also called and checked against manifest in CheckConsistency(), so the calls in SstFileManagerImpl were completely redundant.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6363

Test Plan: deployed to a test cluster, looked at a dump of Env calls (from a custom instrumented Env) - no more thousands of GetFileSize()s.

Differential Revision: D19702509

Pulled By: al13n321

fbshipit-source-id: 99f8110620cb2e9d0c092dfcdbb11f3af4ff8b73
2020-02-04 13:41:53 -08:00
..
delete_scheduler_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
delete_scheduler.cc Fix use-after-free and double-deleting files in BackgroundCallPurge() (#6193) 2019-12-17 20:08:56 -08:00
delete_scheduler.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
file_prefetch_buffer.cc Prevent file prefetch when mmap is enabled. (#6206) 2019-12-18 11:01:29 -08:00
file_prefetch_buffer.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
file_util.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
file_util.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
filename.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
filename.h Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
random_access_file_reader.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
random_access_file_reader.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
read_write_util.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
read_write_util.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
readahead_raf.cc Dedup IsFileSectorAligned() to fix unity build. (#5812) 2019-09-16 20:42:07 -07:00
readahead_raf.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
sequence_file_reader.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
sequence_file_reader.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
sst_file_manager_impl.cc Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363) 2020-02-04 13:41:53 -08:00
sst_file_manager_impl.h Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363) 2020-02-04 13:41:53 -08:00
writable_file_writer.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
writable_file_writer.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00