andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
7e62c5d67a
rocksdb/db
History
Aaron Gao 7e62c5d67a unbiase readamp bitmap 
Summary:
Consider BlockReadAmpBitmap with bytes_per_bit = 32. Suppose bytes [a, b) were used, while bytes [a-32, a)
 and [b+1, b+33) weren't used; more formally, the union of ranges passed to BlockReadAmpBitmap::Mark() contains [a, b) and doesn't intersect with [a-32, a) and [b+1, b+33). Then bits [floor(a/32), ceil(b/32)] will be set, and so the number of useful bytes will be estimated as (ceil(b/32) - floor(a/32)) * 32, which is on average equal to b-a+31.

An extreme example: if we use 1 byte from each block, it'll be counted as 32 bytes from each block.

It's easy to remove this bias by slightly changing the semantics of the bitmap. Currently each bit represents a byte range [i*32, (i+1)*32).

This diff makes each bit represent a single byte: i*32 + X, where X is a random number in [0, 31] generated when bitmap is created. So, e.g., if you read a single byte at random, with probability 31/32 it won't be counted at all, and with probability 1/32 it will be counted as 32 bytes; so, on average it's counted as 1 byte.

*But there is one exception: the last bit will always set with the old way.*

(*) - assuming read_amp_bytes_per_bit = 32.
Closes https://github.com/facebook/rocksdb/pull/2259

Differential Revision: D5035652

Pulled By: lightmark

fbshipit-source-id: bd98b1b9b49fbe61f9e3781d07f624e3cbd92356
2017-05-10 14:06:54 -07:00
..
builder.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
builder.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
c_test.c Add C API functions (and tests) for WriteBatchWithIndex 2017-03-23 15:54:13 -07:00
c.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
column_family_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
column_family.cc Refactor compaction picker code 2017-04-06 20:09:34 -07:00
column_family.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
compact_files_test.cc Windows thread 2017-02-06 14:54:18 -08:00
compacted_db_impl.cc Add macros to include file name and line number during Logging 2017-03-15 19:39:12 -07:00
compacted_db_impl.h Pinnableslice (2nd attempt) 2017-03-13 11:54:10 -07:00
compaction_iteration_stats.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_iterator_test.cc Blob storage helper methods 2017-04-18 12:44:15 -07:00
compaction_iterator.cc Hide event listeners from lite build 2017-04-18 18:02:42 -07:00
compaction_iterator.h Blob storage helper methods 2017-04-18 12:44:15 -07:00
compaction_job_stats_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
compaction_job_test.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
compaction_job.cc Hide event listeners from lite build 2017-04-18 18:02:42 -07:00
compaction_job.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
compaction_picker_test.cc Refactor compaction picker code 2017-04-06 20:09:34 -07:00
compaction_picker_universal.cc Refactor compaction picker code 2017-04-06 20:09:34 -07:00
compaction_picker_universal.h Refactor compaction picker code 2017-04-06 20:09:34 -07:00
compaction_picker.cc Fix compilation for GCC-5 2017-04-07 10:54:12 -07:00
compaction_picker.h Refactor compaction picker code 2017-04-06 20:09:34 -07:00
compaction.cc Avoid calling fallocate with UINT64_MAX 2017-05-08 17:32:32 +00:00
compaction.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
comparator_db_test.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Move various string utility functions into string_util 2017-04-06 14:54:12 -07:00
cuckoo_table_db_test.cc Move various string utility functions into string_util 2017-04-06 14:54:12 -07:00
db_basic_test.cc Separate small subset tests in DBTest 2017-02-27 12:24:11 -08:00
db_block_cache_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
db_bloom_filter_test.cc Configure index partition size 2017-03-28 12:09:12 -07:00
db_compaction_filter_test.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
db_compaction_test.cc Roundup read bytes in ReadaheadRandomAccessFile 2017-05-09 15:18:49 -07:00
db_dynamic_level_test.cc Fix some bugs in MockEnv 2017-03-13 09:54:11 -07:00
db_filesnapshot.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
db_flush_test.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
db_impl_compaction_flush.cc Blob storage helper methods 2017-04-18 12:44:15 -07:00
db_impl_debug.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
db_impl_experimental.cc Add macros to include file name and line number during Logging 2017-03-15 19:39:12 -07:00
db_impl_files.cc Blob storage helper methods 2017-04-18 12:44:15 -07:00
db_impl_open.cc Roundup read bytes in ReadaheadRandomAccessFile 2017-05-09 15:18:49 -07:00
db_impl_readonly.cc Reduce the number of params needed to construct DBIter 2017-04-10 11:14:14 -07:00
db_impl_readonly.h Pinnableslice (2nd attempt) 2017-03-13 11:54:10 -07:00
db_impl_write.cc Adding comments to the write path 2017-04-10 12:43:34 -07:00
db_impl.cc Add DB:ResetStats() 2017-04-18 17:22:35 -07:00
db_impl.h Add DB:ResetStats() 2017-04-18 17:22:35 -07:00
db_info_dumper.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
db_info_dumper.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
db_inplace_update_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_io_failure_test.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_iter_test.cc Reduce the number of params needed to construct DBIter 2017-04-10 11:14:14 -07:00
db_iter.cc Reduce the number of params needed to construct DBIter 2017-04-10 11:14:14 -07:00
db_iter.h Reduce the number of params needed to construct DBIter 2017-04-10 11:14:14 -07:00
db_iterator_test.cc do not read next datablock if upperbound is reached 2017-05-10 14:06:33 -07:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_memtable_test.cc Fix some bugs in MockEnv 2017-03-13 09:54:11 -07:00
db_merge_operator_test.cc Move MergeOperatorPinning tests to be with other merge operator tests 2017-04-11 16:15:06 -07:00
db_options_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
db_properties_test.cc Add DB:ResetStats() 2017-04-18 17:22:35 -07:00
db_range_del_test.cc Level-based L0->L0 compaction 2017-04-04 18:09:11 -07:00
db_sst_test.cc Level-based L0->L0 compaction 2017-04-04 18:09:11 -07:00
db_table_properties_test.cc Fix some bugs in MockEnv 2017-03-13 09:54:11 -07:00
db_tailing_iter_test.cc Fix some bugs in MockEnv 2017-03-13 09:54:11 -07:00
db_test2.cc unbiase readamp bitmap 2017-05-10 14:06:54 -07:00
db_test_util.cc Configure index partition size 2017-03-28 12:09:12 -07:00
db_test_util.h Avoid calling fallocate with UINT64_MAX 2017-05-08 17:32:32 +00:00
db_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
db_universal_compaction_test.cc Fix flaky DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 2017-02-23 18:39:13 -08:00
db_wal_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
dbformat_test.cc update IterKey that can get user key and internal key explicitly 2017-04-04 14:24:20 -07:00
dbformat.cc do not read next datablock if upperbound is reached 2017-05-10 14:06:33 -07:00
dbformat.h update IterKey that can get user key and internal key explicitly 2017-04-04 14:24:20 -07:00
deletefile_test.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
event_helpers.cc Performance: Iterate vector by reference 2017-01-11 10:54:37 -08:00
event_helpers.h Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
external_sst_file_basic_test.cc Add a new SstFileWriter constructor without explicit comparator 2017-03-13 11:39:13 -07:00
external_sst_file_ingestion_job.cc Add macros to include file name and line number during Logging 2017-03-15 19:39:12 -07:00
external_sst_file_ingestion_job.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
external_sst_file_test.cc Add a new SstFileWriter constructor without explicit comparator 2017-03-13 11:39:13 -07:00
fault_injection_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc gcc-7 requires include <functional> for std::function 2016-12-16 11:24:18 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
flush_job_test.cc Make range deletion inclusive-exclusive 2016-11-14 17:39:13 -08:00
flush_job.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
flush_job.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
flush_scheduler.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
forward_iterator.cc do not read next datablock if upperbound is reached 2017-05-10 14:06:33 -07:00
forward_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
internal_stats.cc File level histogram should be printed per CF, not per DB 2017-04-11 08:42:03 -07:00
internal_stats.h Add DB:ResetStats() 2017-04-18 17:22:35 -07:00
job_context.h fixup commit 2016-10-13 08:48:40 -07:00
listener_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
log_format.h Blob storage helper methods 2017-04-18 12:44:15 -07:00
log_reader.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
log_reader.h db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_test.cc db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Need to make sure log file synced before flushing memtable of one column family 2016-07-21 16:29:06 -07:00
managed_iterator.cc update IterKey that can get user key and internal key explicitly 2017-04-04 14:24:20 -07:00
managed_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list_test.cc DeleteRange Get support 2016-11-03 18:54:20 -07:00
memtable_list.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
memtable_list.h Move memtable related files into memtable directory 2017-04-06 14:09:13 -07:00
memtable.cc Blob storage helper methods 2017-04-18 12:44:15 -07:00
memtable.h Move memtable related files into memtable directory 2017-04-06 14:09:13 -07:00
merge_context.h Fix MergeContext::copied_operands_ strings moving 2016-07-25 15:31:41 -07:00
merge_helper_test.cc [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp… 2017-02-23 15:09:12 -08:00
merge_helper.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
merge_helper.h [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp… 2017-02-23 15:09:12 -08:00
merge_operator.cc Introduce FullMergeV2 (eliminate memcpy from merge operators) 2016-07-20 09:49:03 -07:00
merge_test.cc [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp… 2017-02-23 15:09:12 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
pinned_iterators_manager.h Delegate Cleanables 2016-12-29 15:54:19 -08:00
plain_table_db_test.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
prefix_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
range_del_aggregator_test.cc Maintain position in range deletions map 2017-01-05 10:39:12 -08:00
range_del_aggregator.cc Fix DeleteRange including sentinels in output files 2017-01-25 11:09:12 -08:00
range_del_aggregator.h Maintain position in range deletions map 2017-01-05 10:39:12 -08:00
repair_test.cc avoid direct io in rocksdb_lite 2017-02-16 10:39:13 -08:00
repair.cc change use_direct_writes to use_direct_io_for_flush_and_compaction 2017-04-14 16:19:53 -07:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc do not read next datablock if upperbound is reached 2017-05-10 14:06:33 -07:00
table_cache.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
table_properties_collector_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
table_properties_collector.cc Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
table_properties_collector.h Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
transaction_log_impl.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
transaction_log_impl.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
version_builder_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_builder.cc Windows thread 2017-02-06 14:54:18 -08:00
version_builder.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Move various string utility functions into string_util 2017-04-06 14:54:12 -07:00
version_edit.h Avoid FileMetaData copy 2016-07-13 15:36:22 -07:00
version_set_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_set.cc do not read next datablock if upperbound is reached 2017-05-10 14:06:33 -07:00
version_set.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
wal_manager_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
wal_manager.cc Move auto_roll_logger and filename out of db/ 2017-04-03 18:39:14 -07:00
wal_manager.h Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
write_batch_base.cc Limit maximum memory used in the WriteBatch representation 2017-04-10 15:42:26 -07:00
write_batch_internal.h Limit maximum memory used in the WriteBatch representation 2017-04-10 15:42:26 -07:00
write_batch_test.cc Limit maximum memory used in the WriteBatch representation 2017-04-10 15:42:26 -07:00
write_batch.cc Limit maximum memory used in the WriteBatch representation 2017-04-10 15:42:26 -07:00
write_callback_test.cc Move memtable related files into memtable directory 2017-04-06 14:09:13 -07:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Use monotonic time points in write_controller.cc and rate_limiter.cc 2017-02-14 18:24:24 -08:00
write_controller.cc Break stalls when no bg work is happening 2017-03-16 18:24:17 -07:00
write_controller.h Expose the stalling information through DB::GetProperty() 2017-03-29 11:54:20 -07:00
write_thread.cc Adding comments to the write path 2017-04-10 12:43:34 -07:00
write_thread.h Adding comments to the write path 2017-04-10 12:43:34 -07:00