rocksdb/table
Lei Jin 51af7c326c CuckooTable: add one option to allow identity function for the first hash function
Summary:
MurmurHash becomes expensive when we do millions Get() a second in one
thread. Add this option to allow the first hash function to use identity
function as hash function. It results in QPS increase from 3.7M/s to
~4.3M/s. I did not observe improvement for end to end RocksDB
performance. This may be caused by other bottlenecks that I will address
in a separate diff.

Test Plan:
```
[ljin@dev1964 rocksdb] ./cuckoo_table_reader_test --enable_perf --file_dir=/dev/shm --write --identity_as_first_hash=0
==== Test CuckooReaderTest.WhenKeyExists
==== Test CuckooReaderTest.WhenKeyExistsWithUint64Comparator
==== Test CuckooReaderTest.CheckIterator
==== Test CuckooReaderTest.CheckIteratorUint64
==== Test CuckooReaderTest.WhenKeyNotFound
==== Test CuckooReaderTest.TestReadPerformance
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.272us (3.7 Mqps) with batch size of 0, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.138us (7.2 Mqps) with batch size of 10, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.142us (7.1 Mqps) with batch size of 25, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.142us (7.0 Mqps) with batch size of 50, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.144us (6.9 Mqps) with batch size of 100, # of found keys 125829120

With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.201us (5.0 Mqps) with batch size of 0, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.121us (8.3 Mqps) with batch size of 10, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.123us (8.1 Mqps) with batch size of 25, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.121us (8.3 Mqps) with batch size of 50, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.112us (8.9 Mqps) with batch size of 100, # of found keys 104857600

With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.251us (4.0 Mqps) with batch size of 0, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.107us (9.4 Mqps) with batch size of 10, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.099us (10.1 Mqps) with batch size of 25, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.100us (10.0 Mqps) with batch size of 50, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.116us (8.6 Mqps) with batch size of 100, # of found keys 83886080

With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.189us (5.3 Mqps) with batch size of 0, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.095us (10.5 Mqps) with batch size of 10, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.096us (10.4 Mqps) with batch size of 25, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.098us (10.2 Mqps) with batch size of 50, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.105us (9.5 Mqps) with batch size of 100, # of found keys 73400320

[ljin@dev1964 rocksdb] ./cuckoo_table_reader_test --enable_perf --file_dir=/dev/shm --write --identity_as_first_hash=1
==== Test CuckooReaderTest.WhenKeyExists
==== Test CuckooReaderTest.WhenKeyExistsWithUint64Comparator
==== Test CuckooReaderTest.CheckIterator
==== Test CuckooReaderTest.CheckIteratorUint64
==== Test CuckooReaderTest.WhenKeyNotFound
==== Test CuckooReaderTest.TestReadPerformance
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.230us (4.3 Mqps) with batch size of 0, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.086us (11.7 Mqps) with batch size of 10, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.088us (11.3 Mqps) with batch size of 25, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.083us (12.1 Mqps) with batch size of 50, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.083us (12.1 Mqps) with batch size of 100, # of found keys 125829120

With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.159us (6.3 Mqps) with batch size of 0, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 10, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.080us (12.6 Mqps) with batch size of 25, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.080us (12.5 Mqps) with batch size of 50, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.082us (12.2 Mqps) with batch size of 100, # of found keys 104857600

With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.154us (6.5 Mqps) with batch size of 0, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.077us (13.0 Mqps) with batch size of 10, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.077us (12.9 Mqps) with batch size of 25, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 50, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.079us (12.6 Mqps) with batch size of 100, # of found keys 83886080

With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.218us (4.6 Mqps) with batch size of 0, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.083us (12.0 Mqps) with batch size of 10, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.085us (11.7 Mqps) with batch size of 25, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.086us (11.6 Mqps) with batch size of 50, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 100, # of found keys 73400320
```

Reviewers: sdong, igor, yhchiang

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23451
2014-09-18 11:00:48 -07:00
..
adaptive_table_factory.cc introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
adaptive_table_factory.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
block_based_filter_block_test.cc Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block_based_filter_block.cc Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block_based_filter_block.h Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block_based_table_builder.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
block_based_table_builder.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
block_based_table_factory.cc introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
block_based_table_factory.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
block_based_table_reader.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
block_based_table_reader.h Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
block_builder.cc Fix compile 2014-09-02 11:49:38 -07:00
block_builder.h delete unused Comparator 2014-09-04 09:10:13 +08:00
block_hash_index_test.cc Materialize the hash index 2014-05-15 14:09:03 -07:00
block_hash_index.cc Materialize the hash index 2014-05-15 14:09:03 -07:00
block_hash_index.h Materialize the hash index 2014-05-15 14:09:03 -07:00
block_prefix_index.cc fix a few compile warnings 2014-09-04 23:06:23 +08:00
block_prefix_index.h fix a few compile warnings 2014-09-04 23:06:23 +08:00
block_test.cc Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block.cc Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
block.h Remove some unnecessary constructors 2014-09-17 16:45:58 -07:00
bloom_block.cc Adding option to save PlainTable index and bloom filter in SST file. 2014-07-18 16:58:13 -07:00
bloom_block.h Adding option to save PlainTable index and bloom filter in SST file. 2014-07-18 16:58:13 -07:00
cuckoo_table_builder_test.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_builder.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_builder.h CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_factory.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_factory.h CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_reader_test.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_reader.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
cuckoo_table_reader.h CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
filter_block.h Replace naked calls to operator new and delete (Fixes #222) 2014-09-17 13:50:07 -07:00
flush_block_policy.cc move block based table related options BlockBasedTableOptions 2014-08-25 14:22:05 -07:00
format.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
format.h Run make format on PR #249 2014-09-17 15:08:50 -07:00
full_filter_block_test.cc Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
full_filter_block.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
full_filter_block.h Replace naked calls to operator new and delete (Fixes #222) 2014-09-17 13:50:07 -07:00
iter_heap.h Turn on -Wmissing-prototypes 2014-04-09 21:17:14 -07:00
iterator_wrapper.h In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
iterator.cc In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
merger_test.cc Merger test 2014-09-08 22:24:40 -07:00
merger.cc Refactor PerfStepTimer to stop on destruct 2014-09-02 12:04:22 -07:00
merger.h In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
meta_blocks.cc Run make format on PR #249 2014-09-17 15:08:50 -07:00
meta_blocks.h Adding option to save PlainTable index and bloom filter in SST file. 2014-07-18 16:58:13 -07:00
plain_table_builder.cc Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
plain_table_builder.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
plain_table_factory.cc introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
plain_table_factory.h fix comments 2014-09-08 16:34:04 +08:00
plain_table_index.cc introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
plain_table_index.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
plain_table_key_coding.cc replace hard-coded number with named variable 2014-09-08 11:10:17 +08:00
plain_table_key_coding.h typo improvement 2014-09-06 23:21:26 +08:00
plain_table_reader.cc Replace naked calls to operator new and delete (Fixes #222) 2014-09-17 13:50:07 -07:00
plain_table_reader.h introduce ImmutableOptions 2014-09-04 16:18:36 -07:00
table_builder.h First phase API clean up 2014-02-03 00:30:43 -08:00
table_properties.cc Fix SIGFAULT when running sst_dump on v2.6 db 2014-04-21 17:49:47 -07:00
table_reader_bench.cc CuckooTable: add one option to allow identity function for the first hash function 2014-09-18 11:00:48 -07:00
table_reader.h Add DB property "rocksdb.estimate-table-readers-mem" 2014-08-06 11:39:46 -07:00
table_test.cc Merge pull request #249 from tdfischer/decompression-refactoring 2014-09-17 15:01:40 -07:00
two_level_iterator.cc ForwardIterator: reset incomplete iterators on Seek() 2014-08-29 16:21:29 -07:00
two_level_iterator.h In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00