andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
68a8e6b8fa
rocksdb/util
History
Islam AbdelRahman 68a8e6b8fa Introduce FullMergeV2 (eliminate memcpy from merge operators) 
Summary:
This diff update the code to pin the merge operator operands while the merge operation is done, so that we can eliminate the memcpy cost, to do that we need a new public API for FullMerge that replace the std::deque<std::string> with std::vector<Slice>

This diff is stacked on top of D56493 and D56511

In this diff we
- Update FullMergeV2 arguments to be encapsulated in MergeOperationInput and MergeOperationOutput which will make it easier to add new arguments in the future
- Replace std::deque<std::string> with std::vector<Slice> to pass operands
- Replace MergeContext std::deque with std::vector (based on a simple benchmark I ran https://gist.github.com/IslamAbdelRahman/78fc86c9ab9f52b1df791e58943fb187)
- Allow FullMergeV2 output to be an existing operand

```
[Everything in Memtable | 10K operands | 10 KB each | 1 operand per key]

DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=10000 --num=10000 --disable_auto_compactions --value_size=10240 --write_buffer_size=1000000000

[FullMergeV2]
readseq      :       0.607 micros/op 1648235 ops/sec; 16121.2 MB/s
readseq      :       0.478 micros/op 2091546 ops/sec; 20457.2 MB/s
readseq      :       0.252 micros/op 3972081 ops/sec; 38850.5 MB/s
readseq      :       0.237 micros/op 4218328 ops/sec; 41259.0 MB/s
readseq      :       0.247 micros/op 4043927 ops/sec; 39553.2 MB/s

[master]
readseq      :       3.935 micros/op 254140 ops/sec; 2485.7 MB/s
readseq      :       3.722 micros/op 268657 ops/sec; 2627.7 MB/s
readseq      :       3.149 micros/op 317605 ops/sec; 3106.5 MB/s
readseq      :       3.125 micros/op 320024 ops/sec; 3130.1 MB/s
readseq      :       4.075 micros/op 245374 ops/sec; 2400.0 MB/s
```

```
[Everything in Memtable | 10K operands | 10 KB each | 10 operand per key]

DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=1000 --num=10000 --disable_auto_compactions --value_size=10240 --write_buffer_size=1000000000

[FullMergeV2]
readseq      :       3.472 micros/op 288018 ops/sec; 2817.1 MB/s
readseq      :       2.304 micros/op 434027 ops/sec; 4245.2 MB/s
readseq      :       1.163 micros/op 859845 ops/sec; 8410.0 MB/s
readseq      :       1.192 micros/op 838926 ops/sec; 8205.4 MB/s
readseq      :       1.250 micros/op 800000 ops/sec; 7824.7 MB/s

[master]
readseq      :      24.025 micros/op 41623 ops/sec;  407.1 MB/s
readseq      :      18.489 micros/op 54086 ops/sec;  529.0 MB/s
readseq      :      18.693 micros/op 53495 ops/sec;  523.2 MB/s
readseq      :      23.621 micros/op 42335 ops/sec;  414.1 MB/s
readseq      :      18.775 micros/op 53262 ops/sec;  521.0 MB/s

```

```
[Everything in Block cache | 10K operands | 10 KB each | 1 operand per key]

[FullMergeV2]
$ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --num=100000 --db="/dev/shm/merge-random-10K-10KB" --cache_size=1000000000 --use_existing_db --disable_auto_compactions
readseq      :      14.741 micros/op 67837 ops/sec;  663.5 MB/s
readseq      :       1.029 micros/op 971446 ops/sec; 9501.6 MB/s
readseq      :       0.974 micros/op 1026229 ops/sec; 10037.4 MB/s
readseq      :       0.965 micros/op 1036080 ops/sec; 10133.8 MB/s
readseq      :       0.943 micros/op 1060657 ops/sec; 10374.2 MB/s

[master]
readseq      :      16.735 micros/op 59755 ops/sec;  584.5 MB/s
readseq      :       3.029 micros/op 330151 ops/sec; 3229.2 MB/s
readseq      :       3.136 micros/op 318883 ops/sec; 3119.0 MB/s
readseq      :       3.065 micros/op 326245 ops/sec; 3191.0 MB/s
readseq      :       3.014 micros/op 331813 ops/sec; 3245.4 MB/s
```

```
[Everything in Block cache | 10K operands | 10 KB each | 10 operand per key]

DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --num=100000 --db="/dev/shm/merge-random-10-operands-10K-10KB" --cache_size=1000000000 --use_existing_db --disable_auto_compactions

[FullMergeV2]
readseq      :      24.325 micros/op 41109 ops/sec;  402.1 MB/s
readseq      :       1.470 micros/op 680272 ops/sec; 6653.7 MB/s
readseq      :       1.231 micros/op 812347 ops/sec; 7945.5 MB/s
readseq      :       1.091 micros/op 916590 ops/sec; 8965.1 MB/s
readseq      :       1.109 micros/op 901713 ops/sec; 8819.6 MB/s

[master]
readseq      :      27.257 micros/op 36687 ops/sec;  358.8 MB/s
readseq      :       4.443 micros/op 225073 ops/sec; 2201.4 MB/s
readseq      :       5.830 micros/op 171526 ops/sec; 1677.7 MB/s
readseq      :       4.173 micros/op 239635 ops/sec; 2343.8 MB/s
readseq      :       4.150 micros/op 240963 ops/sec; 2356.8 MB/s
```

Test Plan: COMPILE_WITH_ASAN=1 make check -j64

Reviewers: yhchiang, andrewkr, sdong

Reviewed By: sdong

Subscribers: lovro, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57075
2016-07-20 09:49:03 -07:00
..
aligned_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
autovector_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
autovector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
build_version.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cache_bench.cc Cache to have an option to fail Cache::Insert() when full 2016-03-10 17:35:19 -08:00
cache_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
channel.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.h New Statistics to track Compression/Decompression (#1197) 2016-07-19 09:44:03 -07:00
compaction_job_stats_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
comparator.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
compression.h Fixed output size and removed unneeded loop 2016-07-13 09:10:06 -07:00
concurrent_arena.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
concurrent_arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler_test.cc Fix name conflict in delete_shceduler_test and db_sst_test 2016-06-10 15:39:17 -07:00
delete_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom_test.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
dynamic_bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
env_basic_test.cc CLI option & Rename() allow overwrite 2016-07-07 17:51:36 -07:00
env_chroot.cc env_chroot.cc: FreeBSD likes stdlib.h for realpaht() and friends (#1154) 2016-06-05 17:22:55 -07:00
env_chroot.h Introduce chroot Env 2016-05-06 17:42:50 -07:00
env_hdfs.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
env_posix.cc Direct IO fix for Mac 2016-05-23 16:38:25 -07:00
env_test.cc Fix Windows build break 2016-06-01 18:07:59 -07:00
env.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test_env.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
fault_injection_test_env.h [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
file_reader_writer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_util.cc Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
file_util.h Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
filelock_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filter_policy.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.h Refactor cache.cc 2016-07-15 10:41:36 -07:00
heap_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
heap.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
histogram_test.cc Fix Build Error 2016-03-11 22:56:25 -08:00
histogram_windowing.cc Fix in HistogramWindowingImpl 2016-03-17 14:28:41 -07:00
histogram_windowing.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
histogram.cc Fix FB internal CI build failure 2016-03-15 11:38:15 -07:00
histogram.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
instrumented_mutex.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
instrumented_mutex.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
io_posix.cc Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
io_posix.h Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
iostats_context_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
iostats_context_test.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
iostats_context.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
kv_map.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_write_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
lru_cache.cc Refactor cache.cc 2016-07-15 10:41:36 -07:00
memenv.cc In-memory environment read beyond EOF 2016-05-27 12:10:26 -07:00
mock_env_test.cc Create env_basic_test [pluggable Env part 2] 2016-06-03 15:13:03 -07:00
mock_env.cc In-memory environment read beyond EOF 2016-05-27 12:10:26 -07:00
mock_env.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mutable_cf_options.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
mutable_cf_options.h Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
mutexlock.h Persistent Read Cache (6) Persistent cache tier implentation - File layout 2016-07-19 12:01:46 -07:00
options_helper.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options_helper.h Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options_parser.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
options_parser.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_settable_test.cc Add options.write_buffer_manager: control total memtable size across DB instances 2016-07-05 18:11:25 -07:00
options_test.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options.cc Add options.write_buffer_manager: control total memtable size across DB instances 2016-07-05 18:11:25 -07:00
perf_context_imp.h fix ios build error 2016-02-17 20:22:40 +08:00
perf_context.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level.cc Assert boundary checks for SetPerfLevel() 2016-06-01 09:07:09 -07:00
perf_step_timer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
posix_logger.h Cleanup auto-roll logger flush-while-rolling test 2016-07-07 11:35:40 -07:00
random.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
random.h Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
rate_limiter_test.cc fix rate limiter to avoid starvation 2016-07-01 00:16:29 -07:00
rate_limiter.cc fix rate limiter to avoid starvation 2016-07-01 00:16:29 -07:00
rate_limiter.h Handle overflow case of rate limiter's paramters 2016-05-27 16:15:28 -07:00
sharded_cache.cc Refactor cache.cc 2016-07-15 10:41:36 -07:00
sharded_cache.h Refactor cache.cc 2016-07-15 10:41:36 -07:00
slice_transform_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc Merge db/slice.cc into util/slice.cc 2016-06-10 16:37:36 -07:00
sst_file_manager_impl.cc Fix SstFileManager uninitialized data member 2016-02-18 11:25:19 -08:00
sst_file_manager_impl.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
statistics.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
statistics.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status_message.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
stderr_logger.h Stderr info logger 2016-04-01 11:06:06 -07:00
stop_watch.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.h FreeBSD does not have std::to_string (#1190) 2016-06-29 07:35:17 -07:00
sync_point.cc Add a new feature to enforce a sync point only active on a thread 2016-07-07 11:29:14 -07:00
sync_point.h Add a new feature to enforce a sync point only active on a thread 2016-07-07 11:29:14 -07:00
testharness.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testharness.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testutil.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
testutil.h Introduce FullMergeV2 (eliminate memcpy from merge operators) 2016-07-20 09:49:03 -07:00
thread_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_local_test.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
thread_local.cc Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_local.h Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_operation.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater.cc Use pure if-then check instead of assert in EraseColumnFamilyInfo 2016-03-04 16:03:31 -08:00
thread_status_updater.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
threadpool.cc threadpool.cc: abort() lives in stdlib.h on FreeBSD (#1155) 2016-06-05 17:23:38 -07:00
threadpool.h Use generic threadpool for Windows environment (#1120) 2016-05-12 18:34:04 -07:00
transaction_test_util.cc Add multithreaded transaction test 2016-03-11 15:16:52 -08:00
transaction_test_util.h Fix AppVeyor build error 2016-03-15 10:57:33 -07:00
xfunc.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xxhash.cc Enable MS Warning C4804 : unsafe use of type 'bool' in operation 2015-11-18 16:23:19 -08:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00