andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
23d4cf483c
rocksdb/util
History
Aaron Gao 5d660258e7 add simulator Cache as class SimCache/SimLRUCache(with test) 
Summary: add class SimCache(base class with instrumentation api) and SimLRUCache(derived class with detailed implementation) which is used as an instrumented block cache that can predict hit rate for different cache size

Test Plan:
Add a test case in `db_block_cache_test.cc` called `SimCacheTest` to test basic logic of SimCache.
Also add option `-simcache_size` in db_bench. if set with a value other than -1, then the benchmark will use this value as the size of the simulator cache and finally output the simulation result.
```
[gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 1000000
RocksDB:    version 4.8
Date:       Tue May 17 16:56:16 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :       6.809 micros/op 146874 ops/sec;   16.2 MB/s
DB path: [/tmp/rocksdbtest-112628/dbbench]
readrandom   :       6.343 micros/op 157665 ops/sec;   17.4 MB/s (1000000 of 1000000 found)

SIMULATOR CACHE STATISTICS:
SimCache LOOKUPs: 986559
SimCache HITs:    264760
SimCache HITRATE: 26.84%

[gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 10000000
RocksDB:    version 4.8
Date:       Tue May 17 16:57:10 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :       5.066 micros/op 197394 ops/sec;   21.8 MB/s
DB path: [/tmp/rocksdbtest-112628/dbbench]
readrandom   :       6.457 micros/op 154870 ops/sec;   17.1 MB/s (1000000 of 1000000 found)

SIMULATOR CACHE STATISTICS:
SimCache LOOKUPs: 1059764
SimCache HITs:    374501
SimCache HITRATE: 35.34%

[gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 100000000
RocksDB:    version 4.8
Date:       Tue May 17 16:57:32 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :       5.632 micros/op 177572 ops/sec;   19.6 MB/s
DB path: [/tmp/rocksdbtest-112628/dbbench]
readrandom   :       6.892 micros/op 145094 ops/sec;   16.1 MB/s (1000000 of 1000000 found)

SIMULATOR CACHE STATISTICS:
SimCache LOOKUPs: 1150767
SimCache HITs:    1034535
SimCache HITRATE: 89.90%
```

Reviewers: IslamAbdelRahman, andrewkr, sdong

Reviewed By: sdong

Subscribers: MarkCallaghan, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57999
2016-05-23 23:35:23 -07:00
..
aligned_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
autovector_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
autovector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
build_version.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cache_bench.cc Cache to have an option to fail Cache::Insert() when full 2016-03-10 17:35:19 -08:00
cache_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
cache.cc add simulator Cache as class SimCache/SimLRUCache(with test) 2016-05-23 23:35:23 -07:00
channel.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.h Fixed compile warnings in posix_logger.h and coding.h 2016-03-31 16:01:47 -07:00
compaction_job_stats_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
comparator.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
compression.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
concurrent_arena.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
concurrent_arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
env_chroot.cc Follow symlinks in chroot directory 2016-05-10 09:53:52 -07:00
env_chroot.h Introduce chroot Env 2016-05-06 17:42:50 -07:00
env_hdfs.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
env_posix.cc Direct IO fix for Mac 2016-05-23 16:38:25 -07:00
env_test.cc Direct IO fix for Mac 2016-05-23 16:38:25 -07:00
env.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test_env.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
fault_injection_test_env.h [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
file_reader_writer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_util.cc Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
file_util.h Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
filelock_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filter_policy.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
heap_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
heap.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
histogram_test.cc Fix Build Error 2016-03-11 22:56:25 -08:00
histogram_windowing.cc Fix in HistogramWindowingImpl 2016-03-17 14:28:41 -07:00
histogram_windowing.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
histogram.cc Fix FB internal CI build failure 2016-03-15 11:38:15 -07:00
histogram.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
instrumented_mutex.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
instrumented_mutex.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
io_posix.cc Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
io_posix.h Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
iostats_context_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
iostats_context_test.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
iostats_context.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
kv_map.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_write_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
lru_cache_handle.h add simulator Cache as class SimCache/SimLRUCache(with test) 2016-05-23 23:35:23 -07:00
memenv_test.cc [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
memenv.cc Refactor to support file_reader_writer on Windows. 2015-09-11 09:57:02 -07:00
mock_env_test.cc [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
mock_env.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mock_env.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mutable_cf_options.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mutable_cf_options.h Clean up the ComputeCompactionScore() API 2016-05-23 15:55:29 -07:00
mutexlock.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_builder.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_helper.cc Add GetStringFromCompressionType to include/rocksdb/convenience.h 2016-05-18 15:03:21 -07:00
options_helper.h [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
options_parser.cc Add parsing of missing DB options 2016-03-02 10:34:14 -08:00
options_parser.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_settable_test.cc Added PersistentCache abstraction 2016-05-15 22:17:18 -07:00
options_test.cc Add GetStringFromCompressionType to include/rocksdb/convenience.h 2016-05-18 15:03:21 -07:00
options.cc [rocksdb] make more options dynamic 2016-05-17 13:11:56 -07:00
perf_context_imp.h fix ios build error 2016-02-17 20:22:40 +08:00
perf_context.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_step_timer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
posix_logger.h Fixed compile warnings in posix_logger.h and coding.h 2016-03-31 16:01:47 -07:00
random.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
random.h Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
rate_limiter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
rate_limiter.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
rate_limiter.h Add a minimum value for the refill bytes per period value 2016-04-13 09:01:42 -07:00
slice_transform_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc [ldb] Export ldb_cmd*.h 2016-05-06 16:09:09 -07:00
sst_file_manager_impl.cc Fix SstFileManager uninitialized data member 2016-02-18 11:25:19 -08:00
sst_file_manager_impl.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
statistics.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
statistics.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status_message.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
stderr_logger.h Stderr info logger 2016-04-01 11:06:06 -07:00
stop_watch.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
sync_point.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
sync_point.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testharness.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testharness.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testutil.cc Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. 2016-04-15 10:22:18 -07:00
testutil.h db/log_test: add recycle log test 2016-05-22 22:00:15 -07:00
thread_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_local_test.cc Fix LITE build thread_local_test 2016-02-19 13:57:18 -08:00
thread_local.cc Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_local.h Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_operation.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater.cc Use pure if-then check instead of assert in EraseColumnFamilyInfo 2016-03-04 16:03:31 -08:00
thread_status_updater.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
threadpool.cc Use generic threadpool for Windows environment (#1120) 2016-05-12 18:34:04 -07:00
threadpool.h Use generic threadpool for Windows environment (#1120) 2016-05-12 18:34:04 -07:00
transaction_test_util.cc Add multithreaded transaction test 2016-03-11 15:16:52 -08:00
transaction_test_util.h Fix AppVeyor build error 2016-03-15 10:57:33 -07:00
xfunc.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xxhash.cc Enable MS Warning C4804 : unsafe use of type 'bool' in operation 2015-11-18 16:23:19 -08:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00