rocksdb/table
Sagar Vemuri 93c2b91740 Introduce conditional merge-operator invocation in point lookups
Summary:
For every merge operand encountered for a key in the read path we now have the ability to decide whether to look further (to retrieve more merge operands for the key) or stop and invoke the merge operator to return the value. The user needs to override `ShouldMerge()` method with a condition to terminate search when true to avail this facility.

This has a couple of advantages:
1. It helps in limiting the number of merge operands that are looked at to compute a value as part of a user Get operation.
2. It allows to peek at a merge key-value to see if further merge operands need to look at.

Example: Limiting the number of merge operands that are looked at: Lets say you have 10 merge operands for a key spread over various levels. If you only want RocksDB to look at the latest two merge operands instead of all 10 to compute the value, it is now possible with this PR. You can set the condition in `ShouldMerge()` to return true when the size of the operand list is 2. Look at the example implementation in the unit test. Without this PR, a Get might look at all the 10 merge operands in different levels before invoking the merge-operator.

Added a new unit test.
Made sure that there is no perf regression by running benchmarks.

Command line to Load data:
```
TEST_TMPDIR=/dev/shm ./db_bench --benchmarks="mergerandom" --merge_operator="uint64add" --num=10000000
...
mergerandom  :      12.861 micros/op 77757 ops/sec;    8.6 MB/s ( updates:10000000)
```

**ReadRandomMergeRandom bechmark results:**
Command line:
```
TEST_TMPDIR=/dev/shm ./db_bench --benchmarks="readrandommergerandom" --merge_operator="uint64add" --num=10000000
```

Base -- Without this code change (on commit fc7476b):
```
readrandommergerandom :      38.586 micros/op 25916 ops/sec; (reads:3001599 merges:6998401 total:10000000 hits:842235 maxlength:8)
```

With this code change:
```
readrandommergerandom :      38.653 micros/op 25870 ops/sec; (reads:3001599 merges:6998401 total:10000000 hits:842235 maxlength:8)
```
Closes https://github.com/facebook/rocksdb/pull/2923

Differential Revision: D5898239

Pulled By: sagar0

fbshipit-source-id: daefa325019f77968639a75c851d46352c2303ef
2017-09-28 15:58:49 -07:00
..
adaptive_table_factory.cc Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
adaptive_table_factory.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
block_based_filter_block_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_based_filter_block.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
block_based_filter_block.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_based_table_builder.cc support disabling checksum in block-based table 2017-08-23 19:40:47 -07:00
block_based_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_based_table_factory.cc Replace dynamic_cast<> 2017-07-28 16:27:16 -07:00
block_based_table_factory.h Replace dynamic_cast<> 2017-07-28 16:27:16 -07:00
block_based_table_reader.cc prevent nullptr dereference in table reader error case 2017-09-28 00:12:34 -07:00
block_based_table_reader.h Extend pin_l0 to filter partitions 2017-08-23 07:56:08 -07:00
block_builder.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_prefix_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_prefix_index.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_test.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block.h table/block.h: change memset 2017-07-17 10:41:56 -07:00
bloom_block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_block.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cleanable_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder_test.cc allow nullptr Slice only as sentinel 2017-08-23 10:56:06 -07:00
cuckoo_table_builder.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_factory.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
cuckoo_table_factory.h Replace dynamic_cast<> 2017-07-28 16:27:16 -07:00
cuckoo_table_reader_test.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
cuckoo_table_reader.cc remove unnecessary internal_comparator param in newIterator 2017-07-27 14:30:42 -07:00
cuckoo_table_reader.h remove unnecessary internal_comparator param in newIterator 2017-07-27 14:30:42 -07:00
filter_block.h Extend pin_l0 to filter partitions 2017-08-23 07:56:08 -07:00
flush_block_policy.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
format.cc support disabling checksum in block-based table 2017-08-23 19:40:47 -07:00
format.h Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
full_filter_bits_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
full_filter_block_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
full_filter_block.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
full_filter_block.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
get_context.cc Introduce conditional merge-operator invocation in point lookups 2017-09-28 15:58:49 -07:00
get_context.h write-prepared txn: call IsInSnapshot 2017-09-11 09:14:48 -07:00
index_builder.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
index_builder.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
internal_iterator.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
iter_heap.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
iterator_wrapper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
iterator.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
merger_test.cc Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
merging_iterator.cc Three code-level optimization to Iterator::Next() 2017-09-14 17:57:31 -07:00
merging_iterator.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
meta_blocks.cc Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
meta_blocks.h Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
mock_table.cc remove unnecessary internal_comparator param in newIterator 2017-07-27 14:30:42 -07:00
mock_table.h remove unnecessary internal_comparator param in newIterator 2017-07-27 14:30:42 -07:00
partitioned_filter_block_test.cc Extend pin_l0 to filter partitions 2017-08-23 07:56:08 -07:00
partitioned_filter_block.cc Extend pin_l0 to filter partitions 2017-08-23 07:56:08 -07:00
partitioned_filter_block.h Extend pin_l0 to filter partitions 2017-08-23 07:56:08 -07:00
persistent_cache_helper.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_helper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_options.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_factory.cc Replace dynamic_cast<> 2017-07-28 16:27:16 -07:00
plain_table_factory.h Replace dynamic_cast<> 2017-07-28 16:27:16 -07:00
plain_table_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_index.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_key_coding.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
plain_table_key_coding.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_reader.cc Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
plain_table_reader.h remove unnecessary internal_comparator param in newIterator 2017-07-27 14:30:42 -07:00
scoped_arena_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_writer_collectors.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
sst_file_writer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
table_properties_internal.h Change RocksDB License 2017-07-15 16:11:23 -07:00
table_properties.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
table_reader_bench.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
table_reader.h add VerifyChecksum() to db.h 2017-08-09 15:58:13 -07:00
table_test.cc Preload l0 index partitions 2017-08-18 10:56:20 -07:00
two_level_iterator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
two_level_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00