andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
sdong 907f24d0e1 Concurrent memtable inserter to update counters and flush state after all inserts 
Summary: In concurrent memtable insert case, updating counters in MemTable::Add() can count for 5% CPU usage. By batch all the counters and update in the end of the write batch, the CPU overheads are overhead in the use cases where more than one key is updated in one write batch.

Test Plan:
Write throughput increases 12% with this benchmark setting:

TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom -disable_auto_compactions -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -num=10000000 --writes=1000000 -max_background_flushes=16 -max_write_buffer_number=16 --threads=64 --batch_size=128   -allow_concurrent_memtable_write -enable_write_thread_adaptive_yield

Reviewers: andrewkr, IslamAbdelRahman, ngbronson, igor

Reviewed By: ngbronson

Subscribers: ngbronson, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60495
2016-07-08 10:19:55 -07:00
..
auto_roll_logger_test.cc
Cleanup auto-roll logger flush-while-rolling test
2016-07-07 11:35:40 -07:00
auto_roll_logger.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
auto_roll_logger.h
Redo SyncPoints for flush while rolling test
2016-02-22 21:32:19 -08:00
builder.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
builder.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
c_test.c
Shared dictionary compression using reference block
2016-04-27 17:36:03 -07:00
c.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
column_family_test.cc
Add More Logging to track total_log_size
2016-07-06 14:29:18 -07:00
column_family.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
column_family.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
compact_files_test.cc
Fixed CompactFiles() spuriously failing or corrupting DB
2016-02-22 13:54:58 -08:00
compacted_db_impl.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
compacted_db_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator.cc
Fixing snapshot 0 assertion
2016-04-16 01:47:15 -07:00
compaction_iterator.h
Fix data race issue when sub-compaction is used in CompactionJob
2016-03-24 19:36:39 -07:00
compaction_job_stats_test.cc
Introduce XPRESS compresssion on Windows. (#1081)
2016-04-19 22:54:24 -07:00
compaction_job_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
compaction_job.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_job.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_picker_test.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
compaction_picker.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
compaction_picker.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
compaction.h
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
comparator_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
convenience.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
corruption_test.cc
Change several option defaults
2016-04-28 17:50:58 -07:00
cuckoo_table_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_block_cache_test.cc
add option to not flush memtable on open()
2016-06-13 11:34:16 -07:00
db_bloom_filter_test.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
db_compaction_filter_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_compaction_test.cc
Add a new feature to enforce a sync point only active on a thread
2016-07-07 11:29:14 -07:00
db_dynamic_level_test.cc
Fix flaky DBTestDynamicLevel.DynamicLevelMaxBytesBase2
2016-05-26 10:13:24 -07:00
db_filesnapshot.cc
Backup Options
2016-06-09 19:03:10 -07:00
db_impl_add_file.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_impl_debug.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
db_impl_experimental.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
db_impl_readonly.cc
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
db_impl_readonly.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl.cc
Add More Logging to track total_log_size
2016-07-06 14:29:18 -07:00
db_impl.h
Add More Logging to track total_log_size
2016-07-06 14:29:18 -07:00
db_info_dumper.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_info_dumper.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_io_failure_test.cc
Move IO failure test to separate file
2016-05-18 17:09:20 -07:00
db_iter_test.cc
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
db_iter.cc
Reuse TimedFullMerge instead of FullMerge + instrumentation
2016-06-13 16:17:26 -07:00
db_iter.h
Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
2016-04-26 12:41:07 -07:00
db_iterator_test.cc
Fix win build
2016-05-09 11:52:28 -07:00
db_log_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_properties_test.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
db_sst_test.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_table_properties_test.cc
Embed column family name in SST file
2016-04-06 23:10:32 -07:00
db_tailing_iter_test.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
db_test2.cc
Add a new feature to enforce a sync point only active on a thread
2016-07-07 11:29:14 -07:00
db_test_util.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_test_util.h
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
db_universal_compaction_test.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
db_wal_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
dbformat_test.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
dbformat.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat.h
Modification of WriteBatch to support two phase commit
2016-05-10 14:06:07 -07:00
deletefile_test.cc
Fix a bug that accesses invalid address in iterator cleanup function
2016-07-05 11:57:14 -07:00
event_helpers.cc
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
event_helpers.h
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
experimental.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
fault_injection_test.cc
[rocksdb] Recovery path sequence miscount fix
2016-05-10 14:06:07 -07:00
file_indexer_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
flush_job.cc
Small tweaks to logging to track the number of immutable memtables
2016-06-01 11:11:33 -07:00
flush_job.h
Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
2016-04-15 10:22:18 -07:00
flush_scheduler.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_scheduler.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
forward_iterator.h
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
inlineskiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
inlineskiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
internal_stats.cc
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
internal_stats.h
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
job_context.h
Add More Logging to track total_log_size
2016-07-06 14:29:18 -07:00
listener_test.cc
Add a callback for when memtable is moved to immutable (#1137)
2016-06-02 11:57:31 -07:00
log_format.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_reader.cc
db/log_reader: combine kBadRecord{Len,Checksum} for readability
2016-05-22 22:00:15 -07:00
log_reader.h
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_test.cc
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_writer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_writer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
managed_iterator.cc
Fixes warnings and ensure correct int behavior on 32-bit platforms.
2016-03-16 22:57:57 +01:00
managed_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
manual_compaction_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
memtable_allocator.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
memtable_list_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
memtable_list.cc
group multiple batch of flush into one manifest file (one call to LogAndApply)
2016-07-05 18:09:59 -07:00
memtable_list.h
[rocksdb] Memtable Log Referencing and Prepared Batch Recovery
2016-05-10 14:06:07 -07:00
memtable.cc
Concurrent memtable inserter to update counters and flush state after all inserts
2016-07-08 10:19:55 -07:00
memtable.h
Concurrent memtable inserter to update counters and flush state after all inserts
2016-07-08 10:19:55 -07:00
memtablerep_bench.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
merge_context.h
Eliminate std::deque initialization while iterating over merge operands
2016-04-01 15:48:55 -07:00
merge_helper_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper.cc
Remove wasteful instrumentation in FullMerge (stacked on D59577)
2016-06-13 16:22:14 -07:00
merge_helper.h
Reuse TimedFullMerge instead of FullMerge + instrumentation
2016-06-13 16:17:26 -07:00
merge_operator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_file_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_context_test.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h
Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
2016-04-26 12:41:07 -07:00
plain_table_db_test.cc
Embed column family name in SST file
2016-04-06 23:10:32 -07:00
prefix_test.cc
memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
2016-06-10 12:12:10 -07:00
repair_test.cc
ColumnFamilyOptions API [CF + RepairDB part 3/3]
2016-06-24 16:29:43 -07:00
repair.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
skiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
skiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_cache.cc
Enable configurable readahead for iterators
2016-05-04 15:25:58 -07:00
table_cache.h
Enable configurable readahead for iterators
2016-05-04 15:25:58 -07:00
table_properties_collector_test.cc
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
table_properties_collector.cc
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
table_properties_collector.h
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
transaction_log_impl.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
transaction_log_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder.cc
Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
2016-04-01 10:42:39 -07:00
version_builder.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set.cc
Fix release build for MyRocks by using debug-only code only in debug builds
2016-07-06 16:07:53 -07:00
version_set.h
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
wal_manager_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
wal_manager.cc
In all the places where log records are read, there was a check that
2016-03-30 23:05:22 +05:30
wal_manager.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_base.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_internal.h
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
write_batch_test.cc
Add options.write_buffer_manager: control total memtable size across DB instances
2016-07-05 18:11:25 -07:00
write_batch.cc
Concurrent memtable inserter to update counters and flush state after all inserts
2016-07-08 10:19:55 -07:00
write_callback_test.cc
write_callback_test: clean test directory before running tests
2016-04-18 10:18:41 -07:00
write_callback.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.h
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
xfunc_test_points.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00