rocksdb/options
Peter Dillinger 2a383f21f4 Add Bloom/Ribbon hybrid API support (#8679)
Summary:
This is essentially resurrection and fixing of the part of
https://github.com/facebook/rocksdb/issues/8198 that was reverted in https://github.com/facebook/rocksdb/issues/8212, using data added in https://github.com/facebook/rocksdb/issues/8246. Basically,
when configuring Ribbon filter, you can specify an LSM level before which
Bloom will be used instead of Ribbon. But Bloom is only considered for
Leveled and Universal compaction styles and file going into a known LSM
level. This way, SST file writer, FIFO compaction, etc. use Ribbon filter as
you would expect with NewRibbonFilterPolicy.

So that this can be controlled with a single int value and so that flushes
can be distinguished from intra-L0, we consider flush to go to level -1 for
the purposes of this option. (Explained in API comment.)

I also expect the most common and recommended Ribbon configuration to
use Bloom during flush, to minimize slowing down writes and because according
to my estimates, Ribbon only pays off if the structure lives in memory for
more than an hour. Thus, I have changed the default for NewRibbonFilterPolicy
to be this mild hybrid configuration. I don't really want to add something like
NewHybridFilterPolicy because at least the mild hybrid configuration (Bloom for
flush, Ribbon otherwise) should be considered a natural choice.

C APIs also updated, but because they don't support overloading,
rocksdb_filterpolicy_create_ribbon is kept pure ribbon for clarity and
rocksdb_filterpolicy_create_ribbon_hybrid must be called for a hybrid
configuration. While touching C API, I changed bits per key options from
int to double.

BuiltinFilterPolicy is needed so that LevelThresholdFilterPolicy doesn't inherit
unused fields from BloomFilterPolicy.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8679

Test Plan: new + updated tests, including crash test

Reviewed By: jay-zhuang

Differential Revision: D30445797

Pulled By: pdillinger

fbshipit-source-id: 6f5aeddfd6d79f7e55493b563c2d1d2d568892e1
2021-08-20 18:00:16 -07:00
..
cf_options.cc Move old files to warm tier in FIFO compactions (#8310) 2021-08-09 12:51:14 -07:00
cf_options.h Added static methods for simple types to OptionTypeInfo (#8249) 2021-05-11 16:15:47 -07:00
configurable_helper.h Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
configurable_test.cc Move RegisterOptions into the Configurable API (#8223) 2021-04-26 03:13:24 -07:00
configurable_test.h Move RegisterOptions into the Configurable API (#8223) 2021-04-26 03:13:24 -07:00
configurable.cc Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
customizable_test.cc Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
customizable.cc Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
db_options.cc Re-add retired mempurge flag definitions for legacy-options-file temporary support. (#8650) 2021-08-11 16:07:30 -07:00
db_options.h Memtable sampling for mempurge heuristic. (#8628) 2021-08-10 18:09:03 -07:00
options_helper.cc Fix some minor issues in the Customizable infrastructure (#8566) 2021-08-19 10:10:47 -07:00
options_helper.h Added static methods for simple types to OptionTypeInfo (#8249) 2021-05-11 16:15:47 -07:00
options_parser.cc Refactor: add LineFileReader and Status::MustCheck (#8026) 2021-03-09 20:12:38 -08:00
options_parser.h Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
options_settable_test.cc Move old files to warm tier in FIFO compactions (#8310) 2021-08-09 12:51:14 -07:00
options_test.cc Add Bloom/Ribbon hybrid API support (#8679) 2021-08-20 18:00:16 -07:00
options.cc Limit buffering for collecting samples for compression dictionary (#7970) 2021-02-19 14:09:54 -08:00