rocksdb/java
Andrew Kryczka d904233d2f Limit buffering for collecting samples for compression dictionary (#7970)
Summary:
For dictionary compression, we need to collect some representative samples of the data to be compressed, which we use to either generate or train (when `CompressionOptions::zstd_max_train_bytes > 0`) a dictionary. Previously, the strategy was to buffer all the data blocks during flush, and up to the target file size during compaction. That strategy allowed us to randomly pick samples from as wide a range as possible that'd be guaranteed to land in a single output file.

However, some users try to make huge files in memory-constrained environments, where this strategy can cause OOM. This PR introduces an option, `CompressionOptions::max_dict_buffer_bytes`, that limits how much data blocks are buffered before we switch to unbuffered mode (which means creating the per-SST dictionary, writing out the buffered data, and compressing/writing new blocks as soon as they are built). It is not strict as we currently buffer more than just data blocks -- also keys are buffered. But it does make a step towards giving users predictable memory usage.

Related changes include:

- Changed sampling for dictionary compression to select unique data blocks when there is limited availability of data blocks
- Made use of `BlockBuilder::SwapAndReset()` to save an allocation+memcpy when buffering data blocks for building a dictionary
- Changed `ParseBoolean()` to accept an input containing characters after the boolean. This is necessary since, with this PR, a value for `CompressionOptions::enabled` is no longer necessarily the final component in the `CompressionOptions` string.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7970

Test Plan:
- updated `CompressionOptions` unit tests to verify limit is respected (to the extent expected in the current implementation) in various scenarios of flush/compaction to bottommost/non-bottommost level
- looked at jemalloc heap profiles right before and after switching to unbuffered mode during flush/compaction. Verified memory usage in buffering is proportional to the limit set.

Reviewed By: pdillinger

Differential Revision: D26467994

Pulled By: ajkr

fbshipit-source-id: 3da4ef9fba59974e4ef40e40c01611002c861465
2021-02-19 14:09:54 -08:00
..
benchmark/src/main/java/org/rocksdb/benchmark Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
crossbuild Automatically number the Maven artifacts (#7219) 2020-08-06 14:13:30 -07:00
jmh Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
rocksjni Limit buffering for collecting samples for compression dictionary (#7970) 2021-02-19 14:09:54 -08:00
samples/src/main/java fix java sample typo and replace deprecated code with latest (#7906) 2021-02-01 14:45:34 -08:00
src rocksdbjni: Possible NPE in RocksDB.setOptions #7869 (#7909) 2021-02-18 15:48:39 -08:00
CMakeLists.txt Fixing Windows build using CMake (#7854) 2021-01-15 17:53:16 -08:00
HISTORY-JAVA.md Update JAVA-HISTORY.md for v3.13 2015-08-04 18:12:58 -07:00
jdb_bench.sh Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
Makefile Update the versions of the test dependencies used for RocksJava (#7805) 2021-01-13 16:01:38 -08:00
pom.xml.template Update the versions of the test dependencies used for RocksJava (#7805) 2021-01-13 16:01:38 -08:00
RELEASE.md Add shared library for musl-libc (#3143) 2019-11-26 18:24:09 -08:00