andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
4824014e3b
rocksdb/util
History
Lei Jin 4824014e3b speed up db_bench filluniquerandom mode 
Summary:
filluniquerandom is painfully slow due to the naive bitmap check to find
out if a key has been seen before. Majority of time is spent on searching
the last few keys. Split a giant BitSet to smaller ones so that we can
quickly check if a BitSet is full and thus can skip quickly.

It used to take over one hour to filluniquerandom for 100M keys, now it
takes about 10 mins.

Test Plan:
unit test
also verified correctness in db_bench and make sure all keys are
generated

Reviewers: igor, haobo, yhchiang

Reviewed By: igor

CC: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D17607
2014-04-09 11:25:21 -07:00
..
arena_test.cc Clean up arena API 2014-01-30 22:10:10 -08:00
arena.cc A heuristic way to check if a memtable is full 2014-03-12 16:40:14 -07:00
arena.h Fix HashSkipList and HashLinkedList SIGSEGV 2014-03-14 10:02:04 -07:00
auto_roll_logger_test.cc Merge branch 'master' into columnfamilies 2014-03-03 18:29:03 -08:00
auto_roll_logger.cc Merge branch 'master' into columnfamilies 2014-04-02 10:48:05 -07:00
auto_roll_logger.h Merge branch 'master' into columnfamilies 2014-03-05 16:59:22 -08:00
autovector_test.cc Fix issue #57 2014-01-06 11:11:19 -08:00
autovector.h Replace vector with autovector 2014-01-02 16:43:35 -08:00
blob_store_test.cc Move the compiler back to 4.8.1 + more small fixes 2013-11-18 11:40:16 -08:00
blob_store.cc IOError cleanup 2014-02-12 11:42:54 -08:00
blob_store.h Implement max_size in BlobStore 2013-10-23 14:38:52 -07:00
bloom_test.cc [RocksDB] [Performance Branch] Added dynamic bloom, to be used for memable non-existing key filtering 2013-12-11 00:15:14 -08:00
bloom.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
build_version.h Fixed typos 2013-11-16 11:21:34 +00:00
cache_test.cc [RocksDB] [Performance Branch] Trivia build fix 2013-12-13 14:21:59 -08:00
cache.cc Merge branch 'master' into performance 2014-01-28 10:35:55 -08:00
coding_test.cc Misc cleanup on performance branch 2014-01-17 14:26:29 -08:00
coding.cc Merge branch 'master' into performance 2014-01-28 10:35:55 -08:00
coding.h Merge branch 'master' into performance 2014-01-28 10:35:55 -08:00
comparator.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
crc32c_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
crc32c.cc Dynamically choose SSE 4.2 2014-04-04 14:03:19 -07:00
crc32c.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
dynamic_bloom_test.cc Fix some other signed & unsigned comparisons 2014-04-03 20:51:27 -07:00
dynamic_bloom.cc dynamicbloom fix: don't offset address when it is already aligned 2014-03-28 17:30:20 -07:00
dynamic_bloom.h cache friendly blocked bloomfilter 2014-03-28 09:21:20 -07:00
env_hdfs.cc Enable log info with different levels. 2014-02-26 14:41:28 -08:00
env_posix.cc Fixed a compile error which tries to check whether a size_t < 0 in env_posix.cc 2014-04-01 11:09:06 -07:00
env_test.cc Some small cleaning up to make some compiling environment happy 2014-03-26 18:11:41 -07:00
env.cc Merge branch 'master' into columnfamilies 2014-03-18 13:24:27 -07:00
filelock_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
filter_policy.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
hash_linklist_rep.cc make hash_link_list Node's key space consecutively followed at the end 2014-04-04 15:37:28 -07:00
hash_linklist_rep.h Consolidate SliceTransform object ownership 2014-03-10 12:56:46 -07:00
hash_skiplist_rep.cc make hash_link_list Node's key space consecutively followed at the end 2014-04-04 15:37:28 -07:00
hash_skiplist_rep.h Consolidate SliceTransform object ownership 2014-03-10 12:56:46 -07:00
hash.cc Add a hash-index component for block 2014-03-03 21:11:49 -08:00
hash.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
histogram_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
histogram.cc Statistics code cleanup 2014-01-17 12:46:06 -08:00
histogram.h Statistics code cleanup 2014-01-17 12:46:06 -08:00
ldb_cmd_execute_result.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
ldb_cmd.cc Column family support for DB::OpenForReadOnly() 2014-04-09 09:56:17 -07:00
ldb_cmd.h Merge branch 'master' into columnfamilies 2014-03-20 14:41:37 -07:00
ldb_tool.cc Merge branch 'master' into columnfamilies 2014-03-20 14:41:37 -07:00
log_buffer.cc Fix data corruption by LogBuffer 2014-03-21 15:32:48 -07:00
log_buffer.h Fix data race against logging data structure because of LogBuffer 2014-03-11 16:09:53 -07:00
log_write_bench.cc A simple benchmark to measure WAL append latency 2014-02-24 14:39:32 -08:00
logging.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
logging.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
manual_compaction_test.cc Fix CompactRange to apply filter to every key 2014-01-14 16:19:09 -08:00
murmurhash.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
murmurhash.h Add a hash-index component for block 2014-03-03 21:11:49 -08:00
mutexlock.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
options.cc Initialize Options::bloom_locality 2014-04-07 16:33:03 -07:00
perf_context_imp.h macros for perf_context 2014-04-08 10:58:07 -07:00
perf_context.cc macros for perf_context 2014-04-08 10:58:07 -07:00
posix_logger.h Enable log info with different levels. 2014-02-26 14:41:28 -08:00
random.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
signal_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
skiplistrep.cc make hash_link_list Node's key space consecutively followed at the end 2014-04-04 15:37:28 -07:00
slice.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
stack_trace.h Print stack trace on assertion failure 2013-12-06 17:11:09 -08:00
statistics.cc convert Tickers back to array with padding and alignment 2014-01-29 15:08:41 -08:00
statistics.h cache SuperVersion in thread local storage to avoid mutex lock 2014-02-27 11:38:55 -08:00
stats_logger.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
status.cc IOError cleanup 2014-02-12 11:42:54 -08:00
stl_wrappers.h Killing Transform Rep 2013-12-03 12:42:15 -08:00
stop_watch.h Statistics code cleanup 2014-01-17 12:46:06 -08:00
string_util.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
string_util.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
sync_point.cc [RocksDB] Fix a race condition in GetSortedWalFiles 2014-04-02 22:12:29 -07:00
sync_point.h [RocksDB] Fix a race condition in GetSortedWalFiles 2014-04-02 22:12:29 -07:00
testharness.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
testharness.h Print stack trace on assertion failure 2013-12-06 17:11:09 -08:00
testutil.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
testutil.h [Performance Branch] PlainTable to encode rows with seqID 0, value type using 1 internal byte. 2014-02-03 12:19:30 -08:00
thread_local_test.cc reduce thread count in ThreadLocalTest.ConcurrentReadWriteTest 2014-04-01 16:26:07 -07:00
thread_local.cc use CAS when returning SuperVersion to ThreadLocal 2014-03-07 14:43:22 -08:00
thread_local.h Merge branch 'master' into columnfamilies 2014-03-07 16:59:47 -08:00
vectorrep.cc make hash_link_list Node's key space consecutively followed at the end 2014-04-04 15:37:28 -07:00