rocksdb/tools
Zhichao Cao 999d955e4f RocksDB Trace Analyzer (#4091)
Summary:
A framework of trace analyzing for RocksDB

After collecting the trace by using the tool of [PR #3837](https://github.com/facebook/rocksdb/pull/3837). User can use the Trace Analyzer to interpret, analyze, and characterize the collected workload.
**Input:**
1. trace file
2. Whole keys space file

**Statistics:**
1. Access count of each operation (Get, Put, Delete, SingleDelete, DeleteRange, Merge) in each column family.
2. Key hotness (access count) of each one
3. Key space separation based on given prefix
4. Key size distribution
5. Value size distribution if appliable
6. Top K accessed keys
7. QPS statistics including the average QPS and peak QPS
8. Top K accessed prefix
9. The query correlation analyzing, output the number of X after Y and the corresponding average time
    intervals

**Output:**
1. key access heat map (either in the accessed key space or whole key space)
2. trace sequence file (interpret the raw trace file to line base text file for future use)
3. Time serial (The key space ID and its access time)
4. Key access count distritbution
5. Key size distribution
6. Value size distribution (in each intervals)
7. whole key space separation by the prefix
8. Accessed key space separation by the prefix
9. QPS of each operation and each column family
10. Top K QPS and their accessed prefix range

**Test:**
1. Added the unit test of analyzing Get, Put, Delete, SingleDelete, DeleteRange, Merge
2. Generated the trace and analyze the trace

**Implemented but not tested (due to the limitation of trace_replay):**
1. Analyzing Iterator, supporting Seek() and SeekForPrev() analyzing
2. Analyzing the number of Key found by Get

**Future Work:**
1.  Support execution time analyzing of each requests
2.  Support cache hit situation and block read situation of Get
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4091

Differential Revision: D9256157

Pulled By: zhichao-cao

fbshipit-source-id: f0ceacb7eedbc43a3eee6e85b76087d7832a8fe6
2018-08-13 11:44:02 -07:00
..
advisor Rules Advisor: some fixes to support fetching stats from ODS (#4223) 2018-08-02 15:42:42 -07:00
dump fix gflags namespace 2017-12-01 10:42:05 -08:00
rdb Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
auto_sanity_test.sh Suppress lint in old files 2018-01-29 12:56:42 -08:00
benchmark_leveldb.sh Suppress lint in old files 2018-01-29 12:56:42 -08:00
benchmark.sh benchmark.sh to use --max_background_job 2018-03-20 18:57:55 -07:00
blob_dump.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
check_format_compatible.sh tools/check_format_compatible.sh to cover forward option reading too (#3994) 2018-06-15 11:12:29 -07:00
CMakeLists.txt cmake support for linux and osx (#1358) 2016-09-28 11:53:15 -07:00
db_bench_tool_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
db_bench_tool.cc Fix db_bench default compression level (#4248) 2018-08-09 10:28:14 -07:00
db_bench.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_crashtest.py Support dictionary compression in stress/crash tests (#4234) 2018-08-06 15:27:29 -07:00
db_repl_stress.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
db_sanity_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_stress.cc Remove an assersion about file size (#4268) 2018-08-13 11:12:50 -07:00
dbench_monitor Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
Dockerfile adding docker build script and dockerfile 2015-05-22 16:03:39 -07:00
generate_random_db.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
ldb_cmd_impl.h Add SST ingestion to ldb (#4205) 2018-08-09 14:29:11 -07:00
ldb_cmd_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
ldb_cmd.cc Add SST ingestion to ldb (#4205) 2018-08-09 14:29:11 -07:00
ldb_test.py Add SST ingestion to ldb (#4205) 2018-08-09 14:29:11 -07:00
ldb_tool.cc Add SST ingestion to ldb (#4205) 2018-08-09 14:29:11 -07:00
ldb.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
pflag Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
reduce_levels_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
regression_test.sh Suppress lint in old files 2018-01-29 12:56:42 -08:00
report_lite_binary_size.sh Legocastle job to report lite build binary size to scuba 2018-02-15 17:27:24 -08:00
rocksdb_dump_test.sh Suppress lint in old files 2018-01-29 12:56:42 -08:00
run_flash_bench.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
run_leveldb.sh Fix /bin/bash shebangs 2017-08-03 15:56:46 -07:00
sample-dump.dmp First version of rocksdb_dump and rocksdb_undump. 2015-06-19 16:24:36 -07:00
sst_dump_test.cc Fix a TSAN failure (#4250) 2018-08-09 19:42:32 -07:00
sst_dump_tool_imp.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
sst_dump_tool.cc Pin mmap files in ReadOnlyDB (#4053) 2018-06-27 17:13:34 -07:00
sst_dump.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
trace_analyzer_test.cc RocksDB Trace Analyzer (#4091) 2018-08-13 11:44:02 -07:00
trace_analyzer_tool.cc RocksDB Trace Analyzer (#4091) 2018-08-13 11:44:02 -07:00
trace_analyzer_tool.h RocksDB Trace Analyzer (#4091) 2018-08-13 11:44:02 -07:00
trace_analyzer.cc RocksDB Trace Analyzer (#4091) 2018-08-13 11:44:02 -07:00
verify_random_db.sh tools/check_format_compatible.sh to cover forward option reading too (#3994) 2018-06-15 11:12:29 -07:00
write_stress_runner.py Suppress lint in old files 2018-01-29 12:56:42 -08:00
write_stress.cc Compilation fixes for powerpc build, -Wparentheses-equality error and missing header guards 2018-02-09 14:12:43 -08:00