andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
57f3032285
rocksdb/java/rocksjni
History
Peter Dillinger 57f3032285 Allow fractional bits/key in BloomFilterPolicy (#6092) 
Summary:
There's no technological impediment to allowing the Bloom
filter bits/key to be non-integer (fractional/decimal) values, and it
provides finer control over the memory vs. accuracy trade-off. This is
especially handy in using the format_version=5 Bloom filter in place
of the old one, because bits_per_key=9.55 provides the same accuracy as
the old bits_per_key=10.

This change not only requires refining the logic for choosing the best
num_probes for a given bits/key setting, it revealed a flaw in that logic.
As bits/key gets higher, the best num_probes for a cache-local Bloom
filter is closer to bpk / 2 than to bpk * 0.69, the best choice for a
standard Bloom filter. For example, at 16 bits per key, the best
num_probes is 9 (FP rate = 0.0843%) not 11 (FP rate = 0.0884%).
This change fixes and refines that logic (for the format_version=5
Bloom filter only, just in case) based on empirical tests to find
accuracy inflection points between each num_probes.

Although bits_per_key is now specified as a double, the new Bloom
filter converts/rounds this to "millibits / key" for predictable/precise
internal computations. Just in case of unforeseen compatibility
issues, we round to the nearest whole number bits / key for the
legacy Bloom filter, so as not to unlock new behaviors for it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6092

Test Plan: unit tests included

Differential Revision: D18711313

Pulled By: pdillinger

fbshipit-source-id: 1aa73295f152a995328cb846ef9157ae8a05522a
2019-11-26 15:59:34 -08:00
..
backupablejni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
backupenginejni.cc Support range deletion tombstones in IngestExternalFile SSTs (#3778) 2018-07-13 22:43:09 -07:00
cassandra_compactionfilterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
cassandra_value_operator.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
checkpoint.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
clock_cache.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
columnfamilyhandle.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
compact_range_options.cc Add CompactRangeOptions for Java (#4220) 2018-08-17 10:57:25 -07:00
compaction_filter_factory_jnicallback.cc Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
compaction_filter_factory_jnicallback.h Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
compaction_filter_factory.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_filter.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
compaction_job_info.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_job_stats.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options_fifo.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options_universal.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
comparator.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
comparatorjnicallback.cc Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
comparatorjnicallback.h Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
compression_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
env_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
env.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
filter.cc Allow fractional bits/key in BloomFilterPolicy (#6092) 2019-11-26 15:59:34 -08:00
ingest_external_file_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
iterator.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
jnicallback.cc Suppress lint in old files 2018-01-29 12:56:42 -08:00
jnicallback.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
loggerjnicallback.cc Remove invalid comparison of va_list and nullptr (#5836) 2019-09-23 12:15:01 -07:00
loggerjnicallback.h Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
lru_cache.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
memory_util.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
memtablejni.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
merge_operator.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
native_comparator_wrapper_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
optimistic_transaction_db.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
optimistic_transaction_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
options_util.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
options.cc Add unordered write option rocksjava (#5839) 2019-09-27 10:19:40 -07:00
persistent_cache.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
portal.h WritePrepared: fix Get without snapshot (#5664) 2019-08-05 13:41:21 -07:00
ratelimiterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
remove_emptyvalue_compactionfilterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
restorejni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocks_callback_object.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocksdb_exception_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocksjni.cc Adding support for deleteFilesInRanges in JNI (#4031) 2019-09-18 09:45:11 -07:00
slice.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
snapshot.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
sst_file_manager.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
sst_file_reader_iterator.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
sst_file_readerjni.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
sst_file_writerjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
statistics.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
statisticsjni.cc Get CompactionJobInfo from CompactFiles 2018-12-13 14:21:24 -08:00
statisticsjni.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
table_filter_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table_filter_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table_filter.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table.cc JNI: Do not create 8M block cache for negative blockCacheSize values (#5465) 2019-06-24 11:37:04 -07:00
thread_status.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
transaction_db_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_db.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
transaction_log.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_notifier_jnicallback.cc Add TransactionDB and OptimisticTransactionDB to the Java API 2018-03-02 10:34:13 -08:00
transaction_notifier_jnicallback.h Add TransactionDB and OptimisticTransactionDB to the Java API 2018-03-02 10:34:13 -08:00
transaction_notifier.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction.cc Extend Transaction::GetForUpdate with do_validate (#4680) 2018-12-06 17:49:00 -08:00
ttl.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch_with_index.cc Revert "BaseDeltaIterator: always check valid() before accessing key(… (#4744) 2018-12-03 23:38:27 -08:00
write_batch.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
write_buffer_manager.cc Plumb WriteBufferManager through JNI (#4492) 2018-10-17 11:49:57 -07:00
writebatchhandlerjnicallback.cc Remove warnings caused by unused variables in jni (#4345) 2018-09-05 13:42:34 -07:00
writebatchhandlerjnicallback.h WriteUnPrepared: Add support for recovering WriteUnprepared transactions (#4078) 2018-07-06 17:59:13 -07:00