andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Aaron Gao 5aaef91d4a group multiple batch of flush into one manifest file (one call to LogAndApply) 
Summary: Currently, if several flush outputs are committed together, we issue each manifest write per batch (1 batch = 1 flush = 1 sst file = 1+ continuous memtables). Each manifest write requires one fsync and one fsync to parent directory. In some cases, it becomes the bottleneck of write. We should batch them and write in one manifest write when possible.

Test Plan:
` ./db_bench -benchmarks="fillseq" -max_write_buffer_number=16 -max_background_flushes=16 -disable_auto_compactions=true -min_write_buffer_number_to_merge=1 -write_buffer_size=65536 -level0_stop_writes_trigger=10000 -level0_slowdown_writes_trigger=10000`
**Before**
```
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 4.9
Date:       Fri Jul  1 15:38:17 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :     166.277 micros/op 6014 ops/sec;    0.7 MB/s
```
**After**
```
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 4.9
Date:       Fri Jul  1 15:35:05 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :      52.328 micros/op 19110 ops/sec;    2.1 MB/s
```

Reviewers: andrewkr, IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: igor, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D60075
2016-07-05 18:09:59 -07:00
..
auto_roll_logger_test.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
auto_roll_logger.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
auto_roll_logger.h
Redo SyncPoints for flush while rolling test
2016-02-22 21:32:19 -08:00
builder.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
builder.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
c_test.c
Shared dictionary compression using reference block
2016-04-27 17:36:03 -07:00
c.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
column_family_test.cc
Adding test for contiguous WAL detection
2016-06-07 18:04:15 -07:00
column_family.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
column_family.h
Fix calling GetCurrentMutableCFOptions in CompactionJob::ProcessKeyValueCompaction()
2016-04-29 17:00:50 -07:00
compact_files_test.cc
Fixed CompactFiles() spuriously failing or corrupting DB
2016-02-22 13:54:58 -08:00
compacted_db_impl.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
compacted_db_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
compaction_iterator.cc
Fixing snapshot 0 assertion
2016-04-16 01:47:15 -07:00
compaction_iterator.h
Fix data race issue when sub-compaction is used in CompactionJob
2016-03-24 19:36:39 -07:00
compaction_job_stats_test.cc
Introduce XPRESS compresssion on Windows. (#1081)
2016-04-19 22:54:24 -07:00
compaction_job_test.cc
Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
2016-02-17 15:20:23 -08:00
compaction_job.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_job.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction_picker_test.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
compaction_picker.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
compaction_picker.h
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
compaction.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
compaction.h
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
comparator_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
convenience.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
corruption_test.cc
Change several option defaults
2016-04-28 17:50:58 -07:00
cuckoo_table_db_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_block_cache_test.cc
add option to not flush memtable on open()
2016-06-13 11:34:16 -07:00
db_bloom_filter_test.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
db_compaction_filter_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_compaction_test.cc
Fix Flaky DBCompactionTest.SkipStatsUpdateTest
2016-06-15 12:00:51 -07:00
db_dynamic_level_test.cc
Fix flaky DBTestDynamicLevel.DynamicLevelMaxBytesBase2
2016-05-26 10:13:24 -07:00
db_filesnapshot.cc
Backup Options
2016-06-09 19:03:10 -07:00
db_impl_add_file.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_impl_debug.cc
[rocksdb] make more options dynamic
2016-05-17 13:11:56 -07:00
db_impl_experimental.cc
Clean up the ComputeCompactionScore() API
2016-05-23 15:55:29 -07:00
db_impl_readonly.cc
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
db_impl_readonly.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_impl.cc
Fix a bug that accesses invalid address in iterator cleanup function
2016-07-05 11:57:14 -07:00
db_impl.h
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
db_info_dumper.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_info_dumper.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc
Change some RocksDB default options
2016-03-31 17:12:18 -07:00
db_io_failure_test.cc
Move IO failure test to separate file
2016-05-18 17:09:20 -07:00
db_iter_test.cc
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
db_iter.cc
Reuse TimedFullMerge instead of FullMerge + instrumentation
2016-06-13 16:17:26 -07:00
db_iter.h
Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
2016-04-26 12:41:07 -07:00
db_iterator_test.cc
Fix win build
2016-05-09 11:52:28 -07:00
db_log_iter_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
db_properties_test.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
db_sst_test.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_table_properties_test.cc
Embed column family name in SST file
2016-04-06 23:10:32 -07:00
db_tailing_iter_test.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
db_test2.cc
Fix mutex unlock issue between scheduled compaction and ReleaseCompactionFiles()
2016-05-18 14:56:30 -07:00
db_test_util.cc
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_test_util.h
Update DB::AddFile() to ingest the file to the lowest possible level
2016-06-21 17:57:59 -07:00
db_test.cc
Backup Options
2016-06-09 19:03:10 -07:00
db_universal_compaction_test.cc
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
2016-06-27 15:19:27 -07:00
db_wal_test.cc
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
dbformat_test.cc
Improve BytewiseComparatorImpl::FindShortestSeparator
2016-04-25 23:02:14 -07:00
dbformat.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
dbformat.h
Modification of WriteBatch to support two phase commit
2016-05-10 14:06:07 -07:00
deletefile_test.cc
Fix a bug that accesses invalid address in iterator cleanup function
2016-07-05 11:57:14 -07:00
event_helpers.cc
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
event_helpers.h
Added EventListener::OnTableFileCreationStarted() callback
2016-04-29 11:35:00 -07:00
experimental.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
fault_injection_test.cc
[rocksdb] Recovery path sequence miscount fix
2016-05-10 14:06:07 -07:00
file_indexer_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
file_indexer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
filename.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_job_test.cc
Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
2016-04-15 10:22:18 -07:00
flush_job.cc
Small tweaks to logging to track the number of immutable memtables
2016-06-01 11:11:33 -07:00
flush_job.h
Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
2016-04-15 10:22:18 -07:00
flush_scheduler.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
flush_scheduler.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
forward_iterator.cc
Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
2016-03-04 18:15:29 -08:00
forward_iterator.h
Add Iterator Property rocksdb.iterator.version_number
2016-03-02 16:23:59 -08:00
inlineskiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
inlineskiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
internal_stats.cc
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
internal_stats.h
Add InternalStats and logging for AddFile()
2016-06-16 16:21:41 -07:00
job_context.h
Cleanup stale manifests outside of full purge
2016-03-10 18:16:21 -08:00
listener_test.cc
Add a callback for when memtable is moved to immutable (#1137)
2016-06-02 11:57:31 -07:00
log_format.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_reader.cc
db/log_reader: combine kBadRecord{Len,Checksum} for readability
2016-05-22 22:00:15 -07:00
log_reader.h
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_test.cc
db/log_reader: treat bad record length or checksum as EOF
2016-05-22 22:00:15 -07:00
log_writer.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
log_writer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
managed_iterator.cc
Fixes warnings and ensure correct int behavior on 32-bit platforms.
2016-03-16 22:57:57 +01:00
managed_iterator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
manual_compaction_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_allocator.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_list_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
memtable_list.cc
group multiple batch of flush into one manifest file (one call to LogAndApply)
2016-07-05 18:09:59 -07:00
memtable_list.h
[rocksdb] Memtable Log Referencing and Prepared Batch Recovery
2016-05-10 14:06:07 -07:00
memtable.cc
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
memtable.h
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
memtablerep_bench.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_context.h
Eliminate std::deque initialization while iterating over merge operands
2016-04-01 15:48:55 -07:00
merge_helper_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_helper.cc
Remove wasteful instrumentation in FullMerge (stacked on D59577)
2016-06-13 16:22:14 -07:00
merge_helper.h
Reuse TimedFullMerge instead of FullMerge + instrumentation
2016-06-13 16:17:26 -07:00
merge_operator.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
merge_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
options_file_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
perf_context_test.cc
CompactedDB should not be used if there is outstanding WAL files
2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h
Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
2016-04-26 12:41:07 -07:00
plain_table_db_test.cc
Embed column family name in SST file
2016-04-06 23:10:32 -07:00
prefix_test.cc
memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
2016-06-10 12:12:10 -07:00
repair_test.cc
ColumnFamilyOptions API [CF + RepairDB part 3/3]
2016-06-24 16:29:43 -07:00
repair.cc
Fix UBSan build break caused by variable not initialized
2016-06-29 10:49:25 -07:00
skiplist_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
skiplist.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
snapshot_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
table_cache.cc
Enable configurable readahead for iterators
2016-05-04 15:25:58 -07:00
table_cache.h
Enable configurable readahead for iterators
2016-05-04 15:25:58 -07:00
table_properties_collector_test.cc
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
table_properties_collector.cc
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
table_properties_collector.h
Added "number of merge operands" to statistics in ssts.
2016-05-19 14:24:48 -07:00
transaction_log_impl.cc
Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
2016-05-20 07:42:18 -07:00
transaction_log_impl.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_builder.cc
Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
2016-04-01 10:42:39 -07:00
version_builder.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_edit.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
version_set.cc
group multiple batch of flush into one manifest file (one call to LogAndApply)
2016-07-05 18:09:59 -07:00
version_set.h
group multiple batch of flush into one manifest file (one call to LogAndApply)
2016-07-05 18:09:59 -07:00
wal_manager_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
wal_manager.cc
In all the places where log records are read, there was a check that
2016-03-30 23:05:22 +05:30
wal_manager.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_base.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_batch_internal.h
Deprectate filter_deletes
2016-06-17 10:30:47 -07:00
write_batch_test.cc
Modification of WriteBatch to support two phase commit
2016-05-10 14:06:07 -07:00
write_batch.cc
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
write_callback_test.cc
write_callback_test: clean test directory before running tests
2016-04-18 10:18:41 -07:00
write_callback.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller_test.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_controller.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
write_thread.h
fix simple typos (#1183)
2016-06-25 08:29:40 +01:00
writebuffer.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.cc
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00
xfunc_test_points.h
Updated all copyright headers to the new format.
2016-02-09 15:12:00 -08:00