Vijay Nadimpalli 22028aa9ab Compaction Reads should read no more than compaction_readahead_size bytes, when set! (#5498) ... Summary: As a result of https://github.com/facebook/rocksdb/issues/5431 the compaction_readahead_size given by a user was not used exactly, the reason being the code behind readahead for user-read and compaction-read was unified in the above PR and the behavior for user-read is to read readahead_size+n bytes (see FilePrefetchBuffer::TryReadFromCache method). Before the unification the ReadaheadRandomAccessFileReader used compaction_readahead_size as it is. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5498 Test Plan: Ran strace command : strace -e pread64 -f -T -t ./db_compaction_test --gtest_filter=DBCompactionTest.PartialManualCompaction In the test the compaction_readahead_size was configured to 2MB and verified the pread syscall did indeed request 2MB. Before the change it was requesting more than 2MB. Strace Output: strace: Process 3798982 attached Note: Google Test filter = DBCompactionTest.PartialManualCompaction [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from DBCompactionTest [ RUN ] DBCompactionTest.PartialManualCompaction strace: Process 3798983 attached strace: Process 3798984 attached strace: Process 3798985 attached strace: Process 3798986 attached strace: Process 3798987 attached strace: Process 3798992 attached [pid 3798987] 12:07:05 +++ exited with 0 +++ strace: Process 3798993 attached [pid 3798993] 12:07:05 +++ exited with 0 +++ strace: Process 3798994 attached strace: Process 3799008 attached strace: Process 3799009 attached [pid 3799008] 12:07:05 +++ exited with 0 +++ strace: Process 3799010 attached [pid 3799009] 12:07:05 +++ exited with 0 +++ strace: Process 3799011 attached [pid 3799010] 12:07:05 +++ exited with 0 +++ [pid 3799011] 12:07:05 +++ exited with 0 +++ strace: Process 3799012 attached [pid 3799012] 12:07:05 +++ exited with 0 +++ strace: Process 3799013 attached strace: Process 3799014 attached [pid 3799013] 12:07:05 +++ exited with 0 +++ strace: Process 3799015 attached [pid 3799014] 12:07:05 +++ exited with 0 +++ [pid 3799015] 12:07:05 +++ exited with 0 +++ strace: Process 3799016 attached [pid 3799016] 12:07:05 +++ exited with 0 +++ strace: Process 3799017 attached [pid 3799017] 12:07:05 +++ exited with 0 +++ strace: Process 3799019 attached [pid 3799019] 12:07:05 +++ exited with 0 +++ strace: Process 3799020 attached strace: Process 3799021 attached [pid 3799020] 12:07:05 +++ exited with 0 +++ [pid 3799021] 12:07:05 +++ exited with 0 +++ strace: Process 3799022 attached [pid 3799022] 12:07:05 +++ exited with 0 +++ strace: Process 3799023 attached [pid 3799023] 12:07:05 +++ exited with 0 +++ strace: Process 3799047 attached strace: Process 3799048 attached [pid 3799047] 12:07:06 +++ exited with 0 +++ [pid 3799048] 12:07:06 +++ exited with 0 +++ [pid 3798994] 12:07:06 +++ exited with 0 +++ strace: Process 3799052 attached [pid 3799052] 12:07:06 +++ exited with 0 +++ strace: Process 3799054 attached strace: Process 3799069 attached strace: Process 3799070 attached [pid 3799069] 12:07:06 +++ exited with 0 +++ strace: Process 3799071 attached [pid 3799070] 12:07:06 +++ exited with 0 +++ [pid 3799071] 12:07:06 +++ exited with 0 +++ strace: Process 3799072 attached strace: Process 3799073 attached [pid 3799072] 12:07:06 +++ exited with 0 +++ [pid 3799073] 12:07:06 +++ exited with 0 +++ strace: Process 3799074 attached [pid 3799074] 12:07:06 +++ exited with 0 +++ strace: Process 3799075 attached [pid 3799075] 12:07:06 +++ exited with 0 +++ strace: Process 3799076 attached [pid 3799076] 12:07:06 +++ exited with 0 +++ strace: Process 3799077 attached [pid 3799077] 12:07:06 +++ exited with 0 +++ strace: Process 3799078 attached [pid 3799078] 12:07:06 +++ exited with 0 +++ strace: Process 3799079 attached [pid 3799079] 12:07:06 +++ exited with 0 +++ strace: Process 3799080 attached [pid 3799080] 12:07:06 +++ exited with 0 +++ strace: Process 3799081 attached [pid 3799081] 12:07:06 +++ exited with 0 +++ strace: Process 3799082 attached [pid 3799082] 12:07:06 +++ exited with 0 +++ strace: Process 3799083 attached [pid 3799083] 12:07:06 +++ exited with 0 +++ strace: Process 3799086 attached strace: Process 3799087 attached [pid 3798984] 12:07:06 pread64(9, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000121> [pid 3798984] 12:07:06 pread64(9, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000106> [pid 3798984] 12:07:06 pread64(9, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000081> [pid 3798984] 12:07:06 pread64(9, "\0\v\3foo\2\7\0\0\0\0\0\0\0\270 \0\v\4foo\2\3\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000138> [pid 3798984] 12:07:06 pread64(11, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000097> [pid 3798984] 12:07:06 pread64(11, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000086> [pid 3798984] 12:07:06 pread64(11, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000064> [pid 3798984] 12:07:06 pread64(11, "\0\v\3foo\2\21\0\0\0\0\0\0\0\270 \0\v\4foo\2\r\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000064> [pid 3798984] 12:07:06 pread64(12, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000080> [pid 3798984] 12:07:06 pread64(12, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000090> [pid 3798984] 12:07:06 pread64(12, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000059> [pid 3798984] 12:07:06 pread64(12, "\0\v\3foo\2\33\0\0\0\0\0\0\0\270 \0\v\4foo\2\27\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000065> [pid 3798984] 12:07:06 pread64(13, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000070> [pid 3798984] 12:07:06 pread64(13, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000059> [pid 3798984] 12:07:06 pread64(13, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000061> [pid 3798984] 12:07:06 pread64(13, "\0\v\3foo\2%\0\0\0\0\0\0\0\270 \0\v\4foo\2!\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000065> [pid 3798984] 12:07:06 pread64(14, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000118> [pid 3798984] 12:07:06 pread64(14, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000093> [pid 3798984] 12:07:06 pread64(14, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000050> [pid 3798984] 12:07:06 pread64(14, "\0\v\3foo\2/\0\0\0\0\0\0\0\270 \0\v\4foo\2+\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000082> [pid 3798984] 12:07:06 pread64(15, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000080> [pid 3798984] 12:07:06 pread64(15, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000086> [pid 3798984] 12:07:06 pread64(15, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000091> [pid 3798984] 12:07:06 pread64(15, "\0\v\3foo\0029\0\0\0\0\0\0\0\270 \0\v\4foo\0025\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000174> [pid 3798984] 12:07:06 pread64(16, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000080> [pid 3798984] 12:07:06 pread64(16, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000093> [pid 3798984] 12:07:06 pread64(16, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000194> [pid 3798984] 12:07:06 pread64(16, "\0\v\3foo\2C\0\0\0\0\0\0\0\270 \0\v\4foo\2?\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000086> [pid 3798984] 12:07:06 pread64(17, "\1\203W!\241QE\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 11177) = 53 <0.000079> [pid 3798984] 12:07:06 pread64(17, "\0\22\4rocksdb.properties\353Q\223\5\0\0\0\0\1\0\0"..., 38, 11139) = 38 <0.000047> [pid 3798984] 12:07:06 pread64(17, "\0$\4rocksdb.block.based.table.ind"..., 664, 10475) = 664 <0.000045> [pid 3798984] 12:07:06 pread64(17, "\0\v\3foo\2M\0\0\0\0\0\0\0\270 \0\v\4foo\2I\0\0\0\0\0\0\275"..., 74, 10401) = 74 <0.000107> [pid 3798983] 12:07:06 pread64(17, "\0\v\200\10foo\2P\0\0\0\0\0\0)U?MSg_)j(roFn($e"..., 2097152, 0) = 11230 <0.000091> [pid 3798983] 12:07:06 pread64(17, "", 2085922, 11230) = 0 <0.000073> [pid 3798983] 12:07:06 pread64(16, "\0\v\200\10foo\2F\0\0\0\0\0\0k[h3%.OPH_^:\\S7T&"..., 2097152, 0) = 11230 <0.000083> [pid 3798983] 12:07:06 pread64(16, "", 2085922, 11230) = 0 <0.000078> [pid 3798983] 12:07:06 pread64(15, "\0\v\200\10foo\2<\0\0\0\0\0\0+qToi_c{*S+4:N(:"..., 2097152, 0) = 11230 <0.000095> [pid 3798983] 12:07:06 pread64(15, "", 2085922, 11230) = 0 <0.000067> [pid 3798983] 12:07:06 pread64(14, "\0\v\200\10foo\0022\0\0\0\0\0\0%hw%OMa\"}9I609Q!B"..., 2097152, 0) = 11230 <0.000111> [pid 3798983] 12:07:06 pread64(14, "", 2085922, 11230) = 0 <0.000093> [pid 3798983] 12:07:06 pread64(13, "\0\v\200\10foo\2(\0\0\0\0\0\0p}Y&mu^DcaSGb2&nP"..., 2097152, 0) = 11230 <0.000128> [pid 3798983] 12:07:06 pread64(13, "", 2085922, 11230) = 0 <0.000076> [pid 3798983] 12:07:06 pread64(12, "\0\v\200\10foo\2\36\0\0\0\0\0\0YIyW#]oSs^6VHfB<`"..., 2097152, 0) = 11230 <0.000092> [pid 3798983] 12:07:06 pread64(12, "", 2085922, 11230) = 0 <0.000073> [pid 3798983] 12:07:06 pread64(11, "\0\v\200\10foo\2\24\0\0\0\0\0\0mfF8Jel/*Zf :-#s("..., 2097152, 0) = 11230 <0.000088> [pid 3798983] 12:07:06 pread64(11, "", 2085922, 11230) = 0 <0.000067> [pid 3798983] 12:07:06 pread64(9, "\0\v\200\10foo\2\n\0\0\0\0\0\0\\X'cjiHX)D,RSj1X!"..., 2097152, 0) = 11230 <0.000115> [pid 3798983] 12:07:06 pread64(9, "", 2085922, 11230) = 0 <0.000073> [pid 3798983] 12:07:06 pread64(8, "\1\315\5 \36\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 53, 754) = 53 <0.000098> [pid 3798983] 12:07:06 pread64(8, "\0\22\3rocksdb.properties;\215\5\0\0\0\0\1\0\0\0"..., 37, 717) = 37 <0.000064> [pid 3798983] 12:07:06 pread64(8, "\0$\4rocksdb.block.based.table.ind"..., 658, 59) = 658 <0.000074> [pid 3798983] 12:07:06 pread64(8, "\0\v\2foo\1\0\0\0\0\0\0\0\0\31\0\0\0\0\1\0\0\0\0\212\216\222P", 29, 30) = 29 <0.000064> [pid 3799086] 12:07:06 +++ exited with 0 +++ [pid 3799087] 12:07:06 +++ exited with 0 +++ [pid 3799054] 12:07:06 +++ exited with 0 +++ strace: Process 3799104 attached [pid 3799104] 12:07:06 +++ exited with 0 +++ [ OK ] DBCompactionTest.PartialManualCompaction (757 ms) [----------] 1 test from DBCompactionTest (758 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (759 ms total) [ PASSED ] 1 test. [pid 3798983] 12:07:06 +++ exited with 0 +++ [pid 3798984] 12:07:06 +++ exited with 0 +++ [pid 3798992] 12:07:06 +++ exited with 0 +++ [pid 3798986] 12:07:06 +++ exited with 0 +++ [pid 3798982] 12:07:06 +++ exited with 0 +++ [pid 3798985] 12:07:06 +++ exited with 0 +++ 12:07:06 +++ exited with 0 +++ Differential Revision: D15948422 Pulled By: vjnadimpalli fbshipit-source-id: 9b189d1e8675d290c7784e4b33e5d3b5761d2ac8		2019-06-21 21:31:49 -07:00
..
aligned_buffer.h	Document AlignedBuffer (#5345)	2019-05-24 10:05:40 -07:00
autovector_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
autovector.h	Use placement new and delete in autovector (#5080)	2019-03-20 10:42:04 -07:00
bloom_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
bloom.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
build_version.cc.in	Add copyright headers per FB open-source checkup tool. (#5199)	2019-04-18 10:55:01 -07:00
build_version.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
cast_util.h	Add a missing "once" in .h	2017-07-31 12:12:03 -07:00
channel.h	Support pragma once in all header files and cleanup some warnings (#4339)	2018-09-05 18:13:31 -07:00
coding_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
coding.cc	Enable MSVC W4 with a few exceptions. Fix warnings and bugs	2017-10-19 10:57:12 -07:00
coding.h	Add path to WritableFileWriter. (#4039)	2018-08-23 10:12:58 -07:00
compaction_job_stats_impl.cc	Refresh snapshot list during long compactions (2nd attempt) (#5278)	2019-05-03 17:30:22 -07:00
comparator.cc	Add support for timestamp in Get/Put (#5079)	2019-06-05 23:10:47 -07:00
compression_context_cache.cc	run make format for PR 3838 (#3954)	2018-06-05 12:58:02 -07:00
compression_context_cache.h	run make format for PR 3838 (#3954)	2018-06-05 12:58:02 -07:00
compression.h	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
concurrent_task_limiter_impl.cc	Compaction limiter miscs (#4795)	2018-12-26 13:59:35 -08:00
concurrent_task_limiter_impl.h	Concurrent task limiter for compaction thread control (#4332)	2018-12-13 13:18:28 -08:00
core_local.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
crc32c_arm64.cc	RocksDB CRC32c optimization with ARMv8 Intrinsic (#5221)	2019-04-30 10:59:05 -07:00
crc32c_arm64.h	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
crc32c_ppc_asm.S	Remove PATENTS text from a few straggler files (#5326)	2019-05-21 16:22:35 -07:00
crc32c_ppc_constants.h	Remove PATENTS text from a few straggler files (#5326)	2019-05-21 16:22:35 -07:00
crc32c_ppc.c	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
crc32c_ppc.h	Remove PATENTS text from a few straggler files (#5326)	2019-05-21 16:22:35 -07:00
crc32c_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
crc32c.cc	RocksDB CRC32c optimization with ARMv8 Intrinsic (#5221)	2019-04-30 10:59:05 -07:00
crc32c.h	Updated CRC32 Power Optimization Changes	2017-08-31 14:16:30 -07:00
duplicate_detector.h	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
dynamic_bloom_test.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
dynamic_bloom.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
dynamic_bloom.h	Disallow customized hash function in DynamicBloom (#4915)	2019-01-24 10:34:30 -08:00
file_reader_writer_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
file_reader_writer.cc	Compaction Reads should read no more than compaction_readahead_size bytes, when set! (#5498)	2019-06-21 21:31:49 -07:00
file_reader_writer.h	Combine the read-ahead logic for user reads and compaction reads (#5431)	2019-06-19 14:10:46 -07:00
filelock_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
filter_policy.cc	Change RocksDB License	2017-07-15 16:11:23 -07:00
gflags_compat.h	fix gflags namespace	2017-12-01 10:42:05 -08:00
hash_map.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
hash_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
hash.cc	Add GCC 8 to Travis (#3433)	2018-07-13 10:58:06 -07:00
hash.h	Consolidate hash function used for non-persistent data in a new function (#5155)	2019-04-08 13:32:06 -07:00
heap_test.cc	fix gflags namespace	2017-12-01 10:42:05 -08:00
heap.h	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
kv_map.h	Consolidate hash function used for non-persistent data in a new function (#5155)	2019-04-08 13:32:06 -07:00
log_write_bench.cc	util: fix log_write_bench (#5335)	2019-05-31 17:17:57 -07:00
murmurhash.cc	Add GCC 8 to Travis (#3433)	2018-07-13 10:58:06 -07:00
murmurhash.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
mutexlock.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
ppc-opcode.h	Remove PATENTS text from a few straggler files (#5326)	2019-05-21 16:22:35 -07:00
random.cc	Change RocksDB License	2017-07-15 16:11:23 -07:00
random.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
rate_limiter_test.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
rate_limiter.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
rate_limiter.h	rate limit auto-tuning	2017-10-04 19:15:01 -07:00
repeatable_thread_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
repeatable_thread.h	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
set_comparator.h	WritePrepared Txn: Move DuplicateDetector to util	2018-03-05 23:57:12 -08:00
slice_transform_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
slice.cc	Apply modernize-use-override (2nd iteration)	2019-02-14 14:41:36 -08:00
status.cc	Use separate status code for column family drop and db shutdown in progress (#5275)	2019-05-20 10:47:32 -07:00
stderr_logger.h	Change RocksDB License	2017-07-15 16:11:23 -07:00
stop_watch.h	Make statistics's stats_level change thread-safe (#5030)	2019-03-01 10:42:09 -08:00
string_util.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
string_util.h	add OptionType kInt32T and kInt64T	2019-03-12 13:49:52 -07:00
thread_list_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
thread_local_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
thread_local.cc	Enable building of ARM32 (#4349)	2018-10-09 16:58:25 -07:00
thread_local.h	Provide a way to override windows memory allocator with jemalloc for ZSTD	2018-06-04 12:12:48 -07:00
thread_operation.h	Add inline comments to flush job (#4464)	2018-10-05 15:41:17 -07:00
threadpool_imp.cc	refactor SavePoints (#5192)	2019-04-19 20:33:04 -07:00
threadpool_imp.h	Support lowering CPU priority of background threads	2018-04-24 08:41:51 -07:00
timer_queue_test.cc	Change RocksDB License	2017-07-15 16:11:23 -07:00
timer_queue.h	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
user_comparator_wrapper.h	Fix perf_context.user_key_comparison_count for range scan (#5098)	2019-03-27 10:34:27 -07:00
util.h	Add GCC 8 to Travis (#3433)	2018-07-13 10:58:06 -07:00
vector_iterator.h	Add copyright headers per FB open-source checkup tool. (#5199)	2019-04-18 10:55:01 -07:00
xxhash.cc	Add copyright headers per FB open-source checkup tool. (#5199)	2019-04-18 10:55:01 -07:00
xxhash.h	Add copyright headers per FB open-source checkup tool. (#5199)	2019-04-18 10:55:01 -07:00