andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Igor Canadi d59d90bb1f db_bench periodically writes QPS to CSV file 
Summary:
This is part of an effort to better understand and optimize RocksDB stalls under high load. I added a feature to db_bench to periodically write QPS to CSV files. That way we can nicely see how our QPS changes in time (especially when DB is stalled) and can do a better job of evaluating our stall system (i.e. we want the QPS to be as constant as possible, as opposed to having bunch of stalls)

Cool part of CSV files is that we can easily graph them -- there are a bunch of tools available.

Test Plan:
Ran ./db_bench --report_interval_seconds=10 --benchmarks=fillrandom --num=10000000
and observed this in report.csv:

secs_elapsed,interval_qps
10,2725860
20,1980480
30,1863456
40,1454359
50,1460389

Reviewers: sdong, MarkCallaghan, rven, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40047
2015-06-12 14:31:53 -07:00
..
builder.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
builder.h
Add more table properties to EventLogger
2015-05-12 15:53:55 -07:00
c_test.c
Fix ASAN errors in c_test
2015-06-08 11:28:40 -07:00
c.cc
C: add WriteBatch.PutLogData support
2015-06-10 00:12:33 -07:00
column_family_test.cc
Make "make all" work for CYGWIN
2015-06-09 16:36:07 -07:00
column_family.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
column_family.h
Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
2015-06-02 17:07:16 -07:00
compact_files_test.cc
Make "make all" work for CYGWIN
2015-06-09 16:36:07 -07:00
compaction_job_stats_test.cc
Fixed false alarm of size comparison in compaction_job_stats_test
2015-06-12 10:44:07 -07:00
compaction_job_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
compaction_job.cc
Print info message about files need compaction for debuging purpose
2015-06-09 11:23:29 -07:00
compaction_job.h
Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
2015-06-02 17:07:16 -07:00
compaction_picker_test.cc
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
compaction_picker.cc
Don't let two L0->L1 compactions run in parallel
2015-06-11 15:42:16 -07:00
compaction_picker.h
Optimize GetRange Function
2015-05-05 09:57:47 -07:00
compaction.cc
Use nullptr for default compaction_filter_factory
2015-06-08 16:34:26 -07:00
compaction.h
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
comparator_db_test.cc
Make "make all" work for CYGWIN
2015-06-09 16:36:07 -07:00
corruption_test.cc
fix typos
2015-04-25 18:14:27 +09:00
cuckoo_table_db_test.cc
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
db_bench.cc
db_bench periodically writes QPS to CSV file
2015-06-12 14:31:53 -07:00
db_filesnapshot.cc
Don't delete files when column family is dropped
2015-03-19 17:04:29 -07:00
db_impl_debug.cc
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
db_impl_experimental.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
db_impl_readonly.cc
Move GetThreadList() feature under Env.
2014-12-22 12:20:17 -08:00
db_impl_readonly.h
Block ReadOnlyDB in ROCKSDB_LITE
2014-11-26 11:37:59 -08:00
db_impl.cc
Revert skip bottommost compaction
2015-06-12 10:43:33 -07:00
db_impl.h
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
db_iter_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
db_iter.cc
fix typos
2015-04-25 18:14:27 +09:00
db_iter.h
reduce references to cfd->options() in DBImpl
2014-09-08 15:04:34 -07:00
db_test.cc
Revert skip bottommost compaction
2015-06-12 10:43:33 -07:00
dbformat_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
dbformat.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
dbformat.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
deletefile_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
event_helpers.cc
Add EventListener::OnTableFileDeletion()
2015-06-03 19:57:01 -07:00
event_helpers.h
Add EventListener::OnTableFileDeletion()
2015-06-03 19:57:01 -07:00
experimental.cc
Implement DB::PromoteL0 method
2015-04-23 12:10:36 -07:00
fault_injection_test.cc
fault_injection_test: add a test case to cover log syncing after a log roll
2015-04-09 16:15:42 -07:00
file_indexer_test.cc
Fix possible SIGSEGV in CompactRange (github issue #596)
2015-04-29 10:52:31 -07:00
file_indexer.cc
Fix possible SIGSEGV in CompactRange (github issue #596)
2015-04-29 10:52:31 -07:00
file_indexer.h
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
filename_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
filename.cc
Sync manifest file when initializing it
2015-01-22 14:32:03 -08:00
filename.h
Sync manifest file when initializing it
2015-01-22 14:32:03 -08:00
flush_job_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
flush_job.cc
Add largest sequence to FlushJobInfo
2015-06-11 15:22:22 -07:00
flush_job.h
Add largest sequence to FlushJobInfo
2015-06-11 15:22:22 -07:00
flush_scheduler.cc
Don't return (or dereference) dangling pointer
2014-10-02 14:33:16 -07:00
flush_scheduler.h
Fix data race #1
2015-01-26 11:48:07 -08:00
forward_iterator.cc
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
forward_iterator.h
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
internal_stats.cc
Support saving history in memtable_list
2015-05-28 16:34:24 -07:00
internal_stats.h
Support saving history in memtable_list
2015-05-28 16:34:24 -07:00
job_context.h
Clean up old log files in background threads
2015-03-30 15:04:10 -04:00
listener_test.cc
Add Env::GetThreadID(), which returns the ID of the current thread.
2015-06-11 14:18:02 -07:00
log_format.h
Some minor refactoring on the code
2014-01-02 16:32:31 -08:00
log_reader.cc
rocksdb: Fixed 'Dead assignment' and 'Dead initialization' scan-build warnings
2015-02-23 14:10:09 -08:00
log_reader.h
Log writer record format doc.
2015-04-07 16:25:56 -07:00
log_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
log_writer.cc
Fix comparison between signed and usigned integers
2015-05-19 10:59:30 -07:00
log_writer.h
Log writer record format doc.
2015-04-07 16:25:56 -07:00
managed_iterator.cc
Fix compile error on MacOS.
2015-02-24 16:24:53 -08:00
managed_iterator.h
Fixed xfunc related compile errors in ROCKSDB_LITE
2015-04-09 21:05:18 -07:00
memtable_allocator.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
memtable_allocator.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
memtable_list_test.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
memtable_list.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
memtable_list.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
memtable.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
memtable.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
memtablerep_bench.cc
build: avoid unused-variable warning
2015-05-02 13:19:10 -07:00
merge_context.h
API to fetch from both a WriteBatchWithIndex and the db
2015-05-11 14:51:51 -07:00
merge_helper.cc
Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
2015-06-02 17:07:16 -07:00
merge_helper.h
Helper function to time Merges
2015-04-27 20:23:50 -07:00
merge_operator.cc
Some small cleaning up to make some compiling environment happy
2014-03-26 18:11:41 -07:00
merge_test.cc
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
perf_context_test.cc
Makefile minor cleanup
2015-03-30 16:05:35 -04:00
plain_table_db_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
prefix_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
repair.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
skiplist_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
skiplist.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
slice.cc
Create an abstract interface for write batches
2015-03-17 19:23:08 -07:00
snapshot.h
GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex
2015-06-08 21:57:02 -07:00
table_cache.cc
TableMock + framework for mock classes
2014-10-28 17:52:32 -07:00
table_cache.h
use GetContext to replace callback function pointer
2014-09-29 11:09:09 -07:00
table_properties_collector_test.cc
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
2015-04-06 10:27:21 -07:00
table_properties_collector.cc
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
2015-04-06 10:27:21 -07:00
table_properties_collector.h
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
transaction_log_impl.cc
Turn -Wshadow back on
2014-11-06 11:14:28 -08:00
transaction_log_impl.h
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
version_builder_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_builder.cc
Fix deleting obsolete files
2015-02-06 08:44:30 -08:00
version_builder.h
Move VersionBuilder logic to a separate .cc file
2014-10-31 16:34:38 -07:00
version_edit_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_edit.cc
Turn on -Wshadow
2014-10-31 11:59:54 -07:00
version_edit.h
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_set_test.cc
Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true
2015-04-03 09:04:35 -07:00
version_set.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
version_set.h
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
wal_manager_test.cc
Fix flakiness of WalManagerTest
2015-04-13 16:15:05 -07:00
wal_manager.cc
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
wal_manager.h
Fix -Wnon-virtual-dtor errors
2014-11-10 17:39:38 -05:00
write_batch_base.cc
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_batch_internal.h
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_batch_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_batch.cc
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_callback_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_callback.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_controller_test.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_controller.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_controller.h
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_thread.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_thread.h
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
writebuffer.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00