rocksdb/db
Mark Callaghan e9e6b00d29 Add monitoring for universal compaction and add counters for compaction IO
Summary:
Adds these counters
{ WAL_FILE_SYNCED, "rocksdb.wal.synced" }
  number of writes that request a WAL sync
{ WAL_FILE_BYTES, "rocksdb.wal.bytes" },
  number of bytes written to the WAL
{ WRITE_DONE_BY_SELF, "rocksdb.write.self" },
  number of writes processed by the calling thread
{ WRITE_DONE_BY_OTHER, "rocksdb.write.other" },
  number of writes not processed by the calling thread. Instead these were
  processed by the current holder of the write lock
{ WRITE_WITH_WAL, "rocksdb.write.wal" },
  number of writes that request WAL logging
{ COMPACT_READ_BYTES, "rocksdb.compact.read.bytes" },
  number of bytes read during compaction
{ COMPACT_WRITE_BYTES, "rocksdb.compact.write.bytes" },
  number of bytes written during compaction

Per-interval stats output was updated with WAL stats and correct stats for universal compaction
including a correct value for write-amplification. It now looks like:
                               Compactions
Level  Files Size(MB) Score Time(sec)  Read(MB) Write(MB)    Rn(MB)  Rnp1(MB)  Wnew(MB) RW-Amplify Read(MB/s) Write(MB/s)      Rn     Rnp1     Wnp1     NewW    Count  Ln-stall Stall-cnt
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  0        7      464  46.4       281      3411      3875      3411         0      3875        2.1      12.1        13.8      621        0      240      240      628       0.0         0
Uptime(secs): 310.8 total, 2.0 interval
Writes cumulative: 9999999 total, 9999999 batches, 1.0 per batch, 1.22 ingest GB
WAL cumulative: 9999999 WAL writes, 9999999 WAL syncs, 1.00 writes per sync, 1.22 GB written
Compaction IO cumulative (GB): 1.22 new, 3.33 read, 3.78 write, 7.12 read+write
Compaction IO cumulative (MB/sec): 4.0 new, 11.0 read, 12.5 write, 23.4 read+write
Amplification cumulative: 4.1 write, 6.8 compaction
Writes interval: 100000 total, 100000 batches, 1.0 per batch, 12.5 ingest MB
WAL interval: 100000 WAL writes, 100000 WAL syncs, 1.00 writes per sync, 0.01 MB written
Compaction IO interval (MB): 12.49 new, 14.98 read, 21.50 write, 36.48 read+write
Compaction IO interval (MB/sec): 6.4 new, 7.6 read, 11.0 write, 18.6 read+write
Amplification interval: 101.7 write, 102.9 compaction
Stalls(secs): 142.924 level0_slowdown, 0.000 level0_numfiles, 0.805 memtable_compaction, 0.000 leveln_slowdown
Stalls(count): 132461 level0_slowdown, 0 level0_numfiles, 3 memtable_compaction, 0 leveln_slowdown

Task ID: #3329644, #3301695

Blame Rev:

Test Plan:
Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D14583
2013-12-12 13:27:43 -08:00
..
builder.cc [RocksDB] Use raw pointer instead of shared pointer when passing Statistics object internally 2013-11-25 10:38:15 -08:00
builder.h Fixing the warning messages captured under mac os # Consider using git commit -m 'One line title' && arc diff. # You will save time by running lint and unit in the background. 2013-11-12 20:05:28 -08:00
c_test.c Rename leveldb to rocksdb in C api 2013-12-10 10:48:35 +02:00
c.cc Rename leveldb to rocksdb in C api 2013-12-10 10:48:35 +02:00
corruption_test.cc Make "Table" pluggable 2013-10-28 17:54:09 -07:00
db_bench.cc Add monitoring for universal compaction and add counters for compaction IO 2013-12-12 13:27:43 -08:00
db_filesnapshot.cc Speed up FindObsoleteFiles 2013-11-08 15:23:46 -08:00
db_impl_readonly.cc Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
db_impl_readonly.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_impl.cc Add monitoring for universal compaction and add counters for compaction IO 2013-12-12 13:27:43 -08:00
db_impl.h Add monitoring for universal compaction and add counters for compaction IO 2013-12-12 13:27:43 -08:00
db_iter.cc Get rid of some shared_ptrs 2013-12-03 11:17:58 -08:00
db_iter.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_statistics.cc Fix #26 by putting the implementation of CreateDBStatistics() to a cc file 2013-12-05 22:29:03 -08:00
db_statistics.h Fix #26 by putting the implementation of CreateDBStatistics() to a cc file 2013-12-05 22:29:03 -08:00
db_stats_logger.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_test.cc [RocksDB] BackupableDB 2013-12-09 14:06:52 -08:00
dbformat_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
dbformat.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
dbformat.h Make "Table" pluggable 2013-10-28 17:54:09 -07:00
deletefile_test.cc Start DeleteFileTest with clean plate 2013-11-15 16:30:23 -08:00
filename_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
filename.cc Unify DeleteFile and DeleteWalFiles 2013-10-25 08:32:14 -07:00
filename.h Unify DeleteFile and DeleteWalFiles 2013-10-25 08:32:14 -07:00
log_format.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
log_reader.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
log_reader.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
log_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
log_writer.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
log_writer.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
memtable.cc Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
memtable.h Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
memtablelist.cc Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
memtablelist.h Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
merge_context.h Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
merge_helper.cc [RocksDB] Use raw pointer instead of shared pointer when passing Statistics object internally 2013-11-25 10:38:15 -08:00
merge_helper.h [RocksDB] Use raw pointer instead of shared pointer when passing Statistics object internally 2013-11-25 10:38:15 -08:00
merge_operator.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
merge_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
perf_context_test.cc Killing Transform Rep 2013-12-03 12:42:15 -08:00
prefix_filter_iterator.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
prefix_test.cc Killing Transform Rep 2013-12-03 12:42:15 -08:00
repair.cc Get rid of some shared_ptrs 2013-12-03 11:17:58 -08:00
simple_table_db_test.cc Improve the "table stats" 2013-11-19 16:29:42 -08:00
skiplist_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
skiplist.h Implementing DynamicIterator for TransformRepNoLock 2013-11-08 00:31:09 -08:00
snapshot.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
table_cache.cc [RocksDB] Use raw pointer instead of shared pointer when passing Statistics object internally 2013-11-25 10:38:15 -08:00
table_cache.h Follow-up Cleaning-up After D13521 2013-10-30 10:52:33 -07:00
table_properties_collector_test.cc Move flush_block_policy from Options to TableFactory 2013-11-19 22:00:48 -08:00
table_properties_collector.cc Improve the readability of the TableProperties::ToString() 2013-11-21 17:54:23 -08:00
table_properties_collector.h Improve the "table stats" 2013-11-19 16:29:42 -08:00
transaction_log_impl.cc Fix the string format issue 2013-11-12 21:05:39 -08:00
transaction_log_impl.h Making the transaction log iterator more robust 2013-11-04 20:49:03 -08:00
version_edit_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
version_edit.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
version_edit.h Fix the log number bug when updating MANIFEST file 2013-10-24 21:05:33 -07:00
version_set_reduce_num_levels.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
version_set_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
version_set.cc Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
version_set.h Introduce MergeContext to Lazily Initialize merge operand list 2013-12-11 11:37:28 -08:00
write_batch_internal.h In-place updates for equal keys and similar sized values 2013-10-31 11:27:12 -07:00
write_batch_test.cc Get rid of some shared_ptrs 2013-12-03 11:17:58 -08:00
write_batch.cc [RocksDB] Use raw pointer instead of shared pointer when passing Statistics object internally 2013-11-25 10:38:15 -08:00