andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
0af157f9bf
rocksdb/util
History
Feng Zhu 0af157f9bf Implement full filter for block based table. 
Summary:
1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file.

2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter.

3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type.

4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h.

5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc

Benchmark: base commit 1d23b5c470
Command:
db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1
Read QPS increase for about 30% from 2230002 to 2991411.

Test Plan:
make all check
valgrind db_test
db_stress --use_block_based_filter = 0
./auto_sanity_test.sh

Reviewers: igor, yhchiang, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D20979
2014-09-08 10:37:05 -07:00
..
arena_test.cc Arena to inline 2KB of data in it. 2014-05-14 11:49:01 -07:00
arena.cc Fix typo huage => huge 2014-08-14 17:01:20 -07:00
arena.h In DB::NewIterator(), try to allocate the whole iterator tree in an arena 2014-06-02 17:44:57 -07:00
auto_roll_logger_test.cc attempt to fix auto_roll_logger_test 2014-08-18 10:23:18 -07:00
auto_roll_logger.cc Fix a bug causing LOG is not created when max_log_file_size is set. 2014-06-16 10:27:42 -07:00
auto_roll_logger.h Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL 2014-04-10 15:27:42 -07:00
autovector_test.cc autovector::resize 2014-05-08 13:50:49 -07:00
autovector.h Fix autovector iterator increment/decrement comments 2014-08-14 14:56:11 -07:00
benchharness_test.cc Fix unsigned and signed integer compare 2014-04-29 12:29:22 -07:00
benchharness.cc Fix more signed/unsigned comparsions 2014-04-29 12:40:18 -07:00
benchharness.h log_and_apply_bench on a new benchmark framework 2014-05-05 11:11:48 -07:00
blob_store_test.cc Move the compiler back to 4.8.1 + more small fixes 2013-11-18 11:40:16 -08:00
blob_store.cc RocksDBLite 2014-04-15 13:39:26 -07:00
blob_store.h RocksDBLite 2014-04-15 13:39:26 -07:00
bloom_test.cc Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
bloom.cc Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
build_version.h Dont build version if compiling for IOS 2014-04-11 10:54:47 -07:00
cache_bench.cc add cache bench 2014-09-05 15:55:43 -07:00
cache_test.cc ApplyToAllCacheEntries 2014-05-02 16:24:04 -04:00
cache.cc ApplyToAllCacheEntries 2014-05-02 16:24:04 -04:00
coding_test.cc Remove BitStream* tests 2014-08-19 09:52:54 -04:00
coding.cc Removing BitStream* functions 2014-08-19 06:48:21 -07:00
coding.h Removing BitStream* functions 2014-08-19 06:48:21 -07:00
comparator.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
crc32c_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
crc32c.cc Dynamically choose SSE 4.2 2014-04-04 14:03:19 -07:00
crc32c.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_info_dummper.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
dynamic_bloom_test.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
dynamic_bloom.cc Changes to support unity build: 2014-08-11 13:22:47 -04:00
dynamic_bloom.h Adding option to save PlainTable index and bloom filter in SST file. 2014-07-18 16:58:13 -07:00
env_hdfs.cc cleanup exception text 2014-05-21 07:54:22 -04:00
env_posix.cc limit max bytes that can be read/written per pread/write syscall 2014-08-29 21:21:49 -07:00
env_test.cc Fix travis builds 2014-09-04 10:23:45 -07:00
env.cc integrate rate limiter into rocksdb 2014-07-08 12:31:49 -07:00
filelock_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
filter_policy.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
hash_cuckoo_rep.cc sizeof(int*) where sizeof(int) was intended 2014-09-05 20:50:29 -07:00
hash_cuckoo_rep.h Pass logger to memtable rep and TLB page allocation error logged to info logs 2014-05-05 16:43:37 -07:00
hash_linklist_rep.cc HashLinkList memtable switches a bucket to a skip list to reduce performance outliers 2014-07-01 17:14:15 -07:00
hash_linklist_rep.h HashLinkList memtable switches a bucket to a skip list to reduce performance outliers 2014-07-01 17:14:15 -07:00
hash_skiplist_rep.cc use arena to allocate memtable's bloomfilter and hashskiplist's buckets_ 2014-06-30 15:54:31 -07:00
hash_skiplist_rep.h Pass logger to memtable rep and TLB page allocation error logged to info logs 2014-05-05 16:43:37 -07:00
hash.cc Add a hash-index component for block 2014-03-03 21:11:49 -08:00
hash.h Changes to support unity build: 2014-08-11 13:22:47 -04:00
histogram_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
histogram.cc fix a few compile warnings 2014-09-04 23:06:23 +08:00
histogram.h fix a few compile warnings 2014-09-04 23:06:23 +08:00
iostats_context_imp.h Fix ios compile 2014-08-28 12:46:05 -04:00
iostats_context.cc Fix ios compile 2014-08-28 12:46:05 -04:00
ldb_cmd_execute_result.h Revert "Don't compile ldb tool into static library" 2014-04-15 11:29:02 -07:00
ldb_cmd.cc Don't leak data returned by opendir 2014-09-05 20:50:29 -07:00
ldb_cmd.h Move include/utilities/*.h to include/rocksdb/utilities/*.h 2014-07-23 10:21:38 -04:00
ldb_tool.cc [RocksDB] allow LDB tool to have customized key formatter 2014-06-23 15:35:40 -07:00
log_buffer.cc Fix data corruption by LogBuffer 2014-03-21 15:32:48 -07:00
log_buffer.h Fix errors while building with clang 2014-05-15 12:34:53 +08:00
log_write_bench.cc Fix more gflag namespace issues 2014-05-09 08:41:02 -07:00
logging.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
logging.h Clean up compaction logging 2014-05-14 12:13:50 -07:00
manual_compaction_test.cc Fix CompactRange to apply filter to every key 2014-01-14 16:19:09 -08:00
murmurhash.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
murmurhash.h Add a hash-index component for block 2014-03-03 21:11:49 -08:00
mutexlock.h Add separate Read/WriteUnlock methods in MutexRW. 2014-06-16 15:41:46 -07:00
options_builder.cc Changes to support unity build: 2014-08-11 13:22:47 -04:00
options_test.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
options.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
perf_context_imp.h Refactor PerfStepTimer to stop on destruct 2014-09-02 12:04:22 -07:00
perf_context.cc Add a function to return current perf level 2014-07-10 11:35:48 -07:00
posix_logger.h Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL 2014-04-10 15:27:42 -07:00
random.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
rate_limiter_test.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
rate_limiter.cc expose RateLimiter definition 2014-07-25 15:17:06 -07:00
rate_limiter.h expose RateLimiter definition 2014-07-25 15:17:06 -07:00
scoped_arena_iterator.h Remove path with arena==nullptr from NewInternalIterator 2014-09-04 17:40:41 -07:00
signal_test.cc Print out stack trace in mac, too 2014-04-23 09:11:35 -04:00
skiplistrep.cc Remove MemTableRep::GetIterator(const Slice& slice) 2014-06-25 14:09:29 -07:00
slice.cc FixedPrefixTransform to include prefix length in its name 2014-05-13 16:08:21 -07:00
statistics.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
statistics.h make statistics forward-able 2014-07-28 12:10:49 -07:00
status.cc Add timeout_hint_us to WriteOptions and introduce Status::TimeOut. 2014-07-03 15:47:02 -07:00
stl_wrappers.h Killing Transform Rep 2013-12-03 12:42:15 -08:00
stop_watch.h Change StopWatch interface 2014-07-28 12:22:37 -07:00
string_util.cc Turn on -Wmissing-prototypes 2014-04-09 21:17:14 -07:00
string_util.h Turn on -Wmissing-prototypes 2014-04-09 21:17:14 -07:00
sync_point.cc Don't compile sync_point if NDEBUG 2014-04-17 10:49:58 -07:00
sync_point.h Fix ifdef NDEBUG 2014-04-17 14:29:28 -07:00
testharness.cc Better stack trace in MAC 2014-04-25 09:50:51 -04:00
testharness.h Print out stack trace in mac, too 2014-04-23 09:11:35 -04:00
testutil.cc Adding a user comparator for comparing Uint64 slices. 2014-08-27 10:39:31 -07:00
testutil.h Adding a user comparator for comparing Uint64 slices. 2014-08-27 10:39:31 -07:00
thread_local_test.cc simplify ThreadLocalPtr a little bit 2014-04-22 21:13:34 -07:00
thread_local.cc simplify ThreadLocalPtr a little bit 2014-04-22 21:13:34 -07:00
thread_local.h simplify ThreadLocalPtr a little bit 2014-04-22 21:13:34 -07:00
vectorrep.cc Remove assert in vector rep 2014-08-27 11:05:41 -07:00
xxhash.cc xxHash 2014-05-01 14:09:32 -04:00
xxhash.h xxHash 2014-05-01 14:09:32 -04:00