rocksdb/db/compaction
haoyuhuang 5efa0d6b0d Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421)
Summary:
BlockCacheLookupContext only contains the caller for now.
We will trace block accesses at five places:
1. BlockBasedTable::GetFilter.
2. BlockBasedTable::GetUncompressedDict.
3. BlockBasedTable::MaybeReadAndLoadToCache. (To trace access on data, index, and range deletion block.)
4. BlockBasedTable::Get. (To trace the referenced key and whether the referenced key exists in a fetched data block.)
5. BlockBasedTable::MultiGet. (To trace the referenced key and whether the referenced key exists in a fetched data block.)

We create the context at:
1. BlockBasedTable::Get. (kUserGet)
2. BlockBasedTable::MultiGet. (kUserMGet)
3. BlockBasedTable::NewIterator. (either kUserIterator, kCompaction, or external SST ingestion calls this function.)
4. BlockBasedTable::Open. (kPrefetch)
5. Index/Filter::CacheDependencies. (kPrefetch)
6. BlockBasedTable::ApproximateOffsetOf. (kCompaction or kUserApproximateSize).

I loaded 1 million key-value pairs into the database and ran the readrandom benchmark with a single thread. I gave the block cache 10 GB to make sure all reads hit the block cache after warmup. The throughput is comparable.
Throughput of this PR: 231334 ops/s.
Throughput of the master branch: 238428 ops/s.

Experiment setup:
RocksDB:    version 6.2
Date:       Mon Jun 10 10:42:51 2019
CPU:        24 * Intel Core Processor (Skylake)
CPUCache:   16384 KB
Keys:       20 bytes each
Values:     100 bytes each (100 bytes after compression)
Entries:    1000000
Prefix:    20 bytes
Keys per prefix:    0
RawSize:    114.4 MB (estimated)
FileSize:   114.4 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: NoCompression
Compression sampling rate: 0
Memtablerep: skip_list
Perf Level: 1

Load command: ./db_bench --benchmarks="fillseq" --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --statistics --cache_index_and_filter_blocks --cache_size=10737418240 --disable_auto_compactions=1 --disable_wal=1 --compression_type=none --min_level_to_compress=-1 --compression_ratio=1 --num=1000000

Run command: ./db_bench --benchmarks="readrandom,stats" --use_existing_db --threads=1 --duration=120 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --statistics --cache_index_and_filter_blocks --cache_size=10737418240 --disable_auto_compactions=1 --disable_wal=1 --compression_type=none --min_level_to_compress=-1 --compression_ratio=1 --num=1000000 --duration=120

TODOs:
1. Create a caller for external SST file ingestion and differentiate the callers for iterator.
2. Integrate tracer to trace block cache accesses.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5421

Differential Revision: D15704258

Pulled By: HaoyuHuang

fbshipit-source-id: 4aa8a55f8cb1576ffb367bfa3186a91d8f06d93a
2019-06-10 15:33:27 -07:00
..
compaction_iteration_stats.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_iterator_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_iterator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_iterator.h Make format 2019-05-31 15:24:43 -07:00
compaction_job_stats_test.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_job_test.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_job.cc Create a BlockCacheLookupContext to enable fine-grained block cache tracing. (#5421) 2019-06-10 15:33:27 -07:00
compaction_job.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
compaction_picker_fifo.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_picker_fifo.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_picker_level.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_picker_level.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_picker_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
compaction_picker_universal.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_picker_universal.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction_picker.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction_picker.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compaction.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
compaction.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00