andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
7b3222e10a
rocksdb/util
History
Peter Dillinger 18f57f5ef8 Add new persistent 64-bit hash (#5984) 
Summary:
For upcoming new SST filter implementations, we will use a new
64-bit hash function (XXH3 preview, slightly modified). This change
updates hash.{h,cc} for that change, adds unit tests, and out-of-lines
the implementations to keep hash.h as clean/small as possible.

In developing the unit tests, I discovered that the XXH3 preview always
returns zero for the empty string. Zero is problematic for some
algorithms (including an upcoming SST filter implementation) if it
occurs more often than at the "natural" rate, so it should not be
returned from trivial values using trivial seeds. I modified our fork
of XXH3 to return a modest hash of the seed for the empty string.

With hash function details out-of-lines in hash.h, it makes sense to
enable XXH_INLINE_ALL, so that direct calls to XXH64/XXH32/XXH3p
are inlined. To fix array-bounds warnings on some inline calls, I
injected some casts to uintptr_t in xxhash.cc. (Issue reported to Yann.)
Revised: Reverted using XXH_INLINE_ALL for now.  Some Facebook
checks are unhappy about #include on xxhash.cc file. I would
fix that by rename to xxhash_cc.h, but to best preserve history I want
to do that in a separate commit (PR) from the uintptr casts.

Also updated filter_bench for this change, improving the performance
predictability of dry run hashing and adding support for 64-bit hash
(for upcoming new SST filter implementations, minor dead code in the
tool for now).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5984

Differential Revision: D18246567

Pulled By: pdillinger

fbshipit-source-id: 6162fbf6381d63c8cc611dd7ec70e1ddc883fbb8
2019-10-31 16:36:35 -07:00
..
aligned_buffer.h Document AlignedBuffer (#5345) 2019-05-24 10:05:40 -07:00
autovector_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
autovector.h Use placement new and delete in autovector (#5080) 2019-03-20 10:42:04 -07:00
bloom_impl.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
bloom_test.cc Prepare filter tests for more implementations (#5967) 2019-10-31 14:12:33 -07:00
build_version.cc.in Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
build_version.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cast_util.h Add a missing "once" in .h 2017-07-31 12:12:03 -07:00
channel.h Support pragma once in all header files and cleanup some warnings (#4339) 2018-09-05 18:13:31 -07:00
coding_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
coding.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
coding.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
compaction_job_stats_impl.cc Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
comparator.cc Add support for timestamp in Get/Put (#5079) 2019-06-05 23:10:47 -07:00
compression_context_cache.cc run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
compression_context_cache.h run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
compression.h Revert to storing UncompressionDicts in the cache (#5645) 2019-08-23 08:27:30 -07:00
concurrent_task_limiter_impl.cc Compaction limiter miscs (#4795) 2018-12-26 13:59:35 -08:00
concurrent_task_limiter_impl.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
core_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c_arm64.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
crc32c_arm64.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
crc32c_ppc_asm.S Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_ppc_constants.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_ppc.c C file should not include <cinttypes>, it is a C++ header. (#5499) 2019-06-24 16:12:39 -07:00
crc32c_ppc.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
crc32c.cc Cleanup the Arm64 CRC32 unused warning (#5565) 2019-07-15 11:20:26 -07:00
crc32c.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
duplicate_detector.h simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
dynamic_bloom_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dynamic_bloom.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dynamic_bloom.h MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
file_reader_writer_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
filelock_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filter_bench.cc Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
gflags_compat.h filter_bench - a prelim tool for SST filter benchmarking (#5825) 2019-10-07 20:10:53 -07:00
hash_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_test.cc Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
hash.cc Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
hash.h Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
heap_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
heap.h Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
kv_map.h Consolidate hash function used for non-persistent data in a new function (#5155) 2019-04-08 13:32:06 -07:00
log_write_bench.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
murmurhash.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
murmurhash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mutexlock.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
ppc-opcode.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
random.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
random.h Fix some implicit conversions in filter_bench (#5894) 2019-10-08 19:22:07 -07:00
rate_limiter_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
rate_limiter.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
rate_limiter.h rate limit auto-tuning 2017-10-04 19:15:01 -07:00
repeatable_thread_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
repeatable_thread.h Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
set_comparator.h WritePrepared Txn: Move DuplicateDetector to util 2018-03-05 23:57:12 -08:00
slice_transform_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
slice.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
status.cc Allow users to stop manual compactions (#3971) 2019-09-16 21:01:47 -07:00
stderr_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
stop_watch.h Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
string_util.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
string_util.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
thread_list_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
thread_local_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
thread_local.cc Enable building of ARM32 (#4349) 2018-10-09 16:58:25 -07:00
thread_local.h Provide a way to override windows memory allocator with jemalloc for ZSTD 2018-06-04 12:12:48 -07:00
thread_operation.h Add inline comments to flush job (#4464) 2018-10-05 15:41:17 -07:00
threadpool_imp.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
threadpool_imp.h Support lowering CPU priority of background threads 2018-04-24 08:41:51 -07:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_queue.h Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
user_comparator_wrapper.h Fix perf_context.user_key_comparison_count for range scan (#5098) 2019-03-27 10:34:27 -07:00
util.h Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
vector_iterator.h Make clang-analyzer happy (#5821) 2019-09-18 15:25:48 -07:00
xxh3p.h Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
xxhash.cc Add new persistent 64-bit hash (#5984) 2019-10-31 16:36:35 -07:00
xxhash.h Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00