andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
0f559abdb7
rocksdb/table
History
Aaron Gao 259a00eaca unbiase readamp bitmap 
Summary:
Consider BlockReadAmpBitmap with bytes_per_bit = 32. Suppose bytes [a, b) were used, while bytes [a-32, a)
 and [b+1, b+33) weren't used; more formally, the union of ranges passed to BlockReadAmpBitmap::Mark() contains [a, b) and doesn't intersect with [a-32, a) and [b+1, b+33). Then bits [floor(a/32), ceil(b/32)] will be set, and so the number of useful bytes will be estimated as (ceil(b/32) - floor(a/32)) * 32, which is on average equal to b-a+31.

An extreme example: if we use 1 byte from each block, it'll be counted as 32 bytes from each block.

It's easy to remove this bias by slightly changing the semantics of the bitmap. Currently each bit represents a byte range [i*32, (i+1)*32).

This diff makes each bit represent a single byte: i*32 + X, where X is a random number in [0, 31] generated when bitmap is created. So, e.g., if you read a single byte at random, with probability 31/32 it won't be counted at all, and with probability 1/32 it will be counted as 32 bytes; so, on average it's counted as 1 byte.

*But there is one exception: the last bit will always set with the old way.*

(*) - assuming read_amp_bytes_per_bit = 32.
Closes https://github.com/facebook/rocksdb/pull/2259

Differential Revision: D5035652

Pulled By: lightmark

fbshipit-source-id: bd98b1b9b49fbe61f9e3781d07f624e3cbd92356
2017-05-10 01:49:52 -07:00
..
adaptive_table_factory.cc solve the problem of table_factory_to_write_=nullptr (#1342) 2016-09-20 10:11:51 -07:00
adaptive_table_factory.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
block_based_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_filter_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_factory.cc Revert "Delete filter before closing the table" 2017-05-02 13:46:39 -07:00
block_based_table_factory.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_based_table_reader.cc dont skip IO for filter blocks 2017-05-09 09:52:02 -07:00
block_based_table_reader.h Object lifetime in cache 2017-05-05 23:20:01 -07:00
block_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block_prefix_index.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
block_prefix_index.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_test.cc unbiase readamp bitmap 2017-05-10 01:49:52 -07:00
block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
block.h unbiase readamp bitmap 2017-05-10 01:49:52 -07:00
bloom_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
bloom_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cleanable_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_builder_test.cc AIX and Solaris Sparc Support 2017-04-21 20:48:04 -07:00
cuckoo_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
cuckoo_table_factory.cc Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
cuckoo_table_factory.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
cuckoo_table_reader_test.cc AIX and Solaris Sparc Support 2017-04-21 20:48:04 -07:00
cuckoo_table_reader.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
cuckoo_table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
flush_block_policy.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
format.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
format.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
full_filter_block.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
get_context.cc Avoid pinning when row cache is accessed 2017-05-01 18:12:46 -07:00
get_context.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
index_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
index_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
internal_iterator.h fix assertion failure in Prev() 2016-10-13 17:36:48 -07:00
iter_heap.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
iterator_wrapper.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
iterator.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merger_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merging_iterator.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
merging_iterator.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
meta_blocks.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
meta_blocks.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
mock_table.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
mock_table.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
partitioned_filter_block_test.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
partitioned_filter_block.cc Object lifetime in cache 2017-05-05 23:20:01 -07:00
partitioned_filter_block.h Object lifetime in cache 2017-05-05 23:20:01 -07:00
persistent_cache_helper.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
persistent_cache_helper.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
persistent_cache_options.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_builder.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_factory.cc store prefix_extractor_name in table 2016-08-26 11:46:32 -07:00
plain_table_factory.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
plain_table_index.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_index.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_key_coding.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_key_coding.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
plain_table_reader.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
plain_table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
scoped_arena_iterator.h Compaction Support for Range Deletion 2016-10-18 12:04:56 -07:00
sst_file_writer_collectors.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
sst_file_writer.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_builder.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_properties_internal.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_properties.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_reader_bench.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
table_reader.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
table_test.cc Object lifetime in cache 2017-05-05 23:20:01 -07:00
two_level_iterator.cc do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00
two_level_iterator.h do not read next datablock if upperbound is reached 2017-05-05 23:20:01 -07:00