rocksdb/db
Andrew Kryczka c8c8104d7e Dictionary compression for files written by SstFileWriter (#4978)
Summary:
If `CompressionOptions::max_dict_bytes` and/or `CompressionOptions::zstd_max_train_bytes` are set, `SstFileWriter` will now generate files respecting those options.

I refactored the logic a bit for deciding when to use dictionary compression. Previously we plumbed `is_bottommost_level` down to the table builder and used that. However it was kind of confusing in `SstFileWriter`'s context since we don't know what level the file will be ingested to. Instead, now the higher-level callers (e.g., flush, compaction, file writer) are responsible for building the right `CompressionOptions` to give the table builder.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4978

Differential Revision: D14060763

Pulled By: ajkr

fbshipit-source-id: dc802c327896df2b319dc162d6acc82b9cdb452a
2019-02-14 11:23:55 -08:00
..
builder.cc Dictionary compression for files written by SstFileWriter (#4978) 2019-02-14 11:23:55 -08:00
builder.h Dictionary compression for files written by SstFileWriter (#4978) 2019-02-14 11:23:55 -08:00
c_test.c Get CompactionJobInfo from CompactFiles 2018-12-13 14:21:24 -08:00
c.cc Deprecate CompactionFilter::IgnoreSnapshots() = false (#4954) 2019-02-07 16:57:33 -08:00
column_family_test.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
column_family.cc Digest ZSTD compression dictionary once when writing SST file (#4849) 2019-01-18 19:12:57 -08:00
column_family.h Lock free MultiGet (#4754) 2019-01-02 11:42:54 -08:00
compact_files_test.cc Get CompactionJobInfo from CompactFiles 2018-12-13 14:21:24 -08:00
compacted_db_impl.cc move dump stats to a separate thread (#4382) 2018-10-08 22:54:43 -07:00
compacted_db_impl.h Enable checkpoint of read-only db (#4681) 2018-12-07 17:06:02 -08:00
compaction_iteration_stats.h add counter for deletion dropping optimization 2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
compaction_iterator.cc WritePrepared: relax assert in compaction iterator (#4969) 2019-02-11 15:01:46 -08:00
compaction_iterator.h Deprecate CompactionFilter::IgnoreSnapshots() = false (#4954) 2019-02-07 16:57:33 -08:00
compaction_job_stats_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
compaction_job_test.cc Zero seqnum of final key / drop final tombstone when compacting to bottommost level 2019-02-01 09:21:57 -08:00
compaction_job.cc Dictionary compression for files written by SstFileWriter (#4978) 2019-02-14 11:23:55 -08:00
compaction_job.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
compaction_picker_fifo.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker_fifo.h Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker_test.cc Make compaction_pri = kMinOverlappingRatio to be default (#4911) 2019-01-23 16:47:38 -08:00
compaction_picker_universal.cc Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
compaction_picker_universal.h Delete triggered compaction for universal style 2018-05-29 15:44:34 -07:00
compaction_picker.cc Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction_picker.h Move FIFOCompactionPicker to a separate file (#4724) 2018-11-29 16:04:52 -08:00
compaction.cc Dictionary compression for files written by SstFileWriter (#4978) 2019-02-14 11:23:55 -08:00
compaction.h Truncate range tombstones by leveraging InternalKeys (#4432) 2018-10-09 15:19:38 -07:00
comparator_db_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
convenience.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
corruption_test.cc fix DeleteRange memory leak for mmap and block cache (#4810) 2018-12-20 21:59:49 -08:00
cuckoo_table_db_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
db_basic_test.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_blob_index_test.cc fix lite build 2017-10-17 08:57:09 -07:00
db_block_cache_test.cc Reduce scope of compression dictionary to single SST (#4952) 2019-02-11 19:47:32 -08:00
db_bloom_filter_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_compaction_filter_test.cc Deprecate CompactionFilter::IgnoreSnapshots() = false (#4954) 2019-02-07 16:57:33 -08:00
db_compaction_test.cc Preload some files even if options.max_open_files (#3340) 2018-12-28 18:02:28 -08:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_filesnapshot.cc Remove redundant member var and set options (#4631) 2018-11-12 12:24:26 -08:00
db_flush_test.cc Improve flushing multiple column families (#4708) 2018-12-13 15:12:40 -08:00
db_impl_compaction_flush.cc Avoid fsync on the same directory in atomic flush (#4817) 2019-02-12 12:28:36 -08:00
db_impl_debug.cc Enable atomic flush (#4023) 2018-10-26 15:08:43 -07:00
db_impl_experimental.cc Update JobContext. (#3949) 2018-08-03 17:42:34 -07:00
db_impl_files.cc Fix #3840: only SyncClosedLogs for multiple CFs (#4460) 2018-11-13 11:32:16 -08:00
db_impl_open.cc Add a new CPU time counter to compaction report (#4889) 2019-01-29 17:24:00 -08:00
db_impl_readonly.cc Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
db_impl_readonly.h Get CompactionJobInfo from CompactFiles 2018-12-13 14:21:24 -08:00
db_impl_write.cc Add a new CPU time counter to compaction report (#4889) 2019-01-29 17:24:00 -08:00
db_impl.cc Avoid using kInAtomicGroup tag for single-cf op (#4981) 2019-02-13 18:33:42 -08:00
db_impl.h Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
db_info_dumper.cc avoid copying when iterating using range-based for (#4459) 2018-10-09 17:15:51 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
db_iter_test.cc Fix regression bug of Prev() with upper bound (#3989) 2018-06-12 16:57:36 -07:00
db_iter.cc Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
db_iter.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
db_iterator_test.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_log_iter_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
db_memtable_test.cc Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
db_merge_operator_test.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_options_test.cc move dump stats to a separate thread (#4382) 2018-10-08 22:54:43 -07:00
db_properties_test.cc fix accounting for range tombstones in TableProperties (#4841) 2019-01-02 15:08:53 -08:00
db_range_del_test.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_sst_test.cc Try to fix DBSSTTest.RateLimitedDelete flakiness (#4840) 2019-01-03 11:05:19 -08:00
db_statistics_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_table_properties_test.cc Allow dynamic modification of window size and deletion trigger (#4403) 2018-09-20 15:15:28 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Reduce scope of compression dictionary to single SST (#4952) 2019-02-11 19:47:32 -08:00
db_test_util.cc Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_test_util.h Remove cuckoo hash memtable (#4953) 2019-02-07 16:15:27 -08:00
db_test.cc Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
db_universal_compaction_test.cc Adapt three unit tests with newer compiler/libraries (#4562) 2018-10-24 08:17:56 -07:00
db_wal_test.cc Preload some files even if options.max_open_files (#3340) 2018-12-28 18:02:28 -08:00
db_write_test.cc Suppress tsan lock-order-inversion on FlushWAL 2018-05-14 21:13:35 -07:00
dbformat_test.cc Relax VersionStorageInfo::GetOverlappingInputs check (#4050) 2018-07-13 17:42:38 -07:00
dbformat.cc types: add kEntryBlobIndex for TablePropertiesCollector (#4233) 2018-08-06 18:27:44 -07:00
dbformat.h s/CacheAllocator/MemoryAllocator/g (#4590) 2018-10-26 14:30:30 -07:00
deletefile_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
error_handler_test.cc Fix regression test failures introduced by PR #4164 (#4375) 2018-09-17 13:14:07 -07:00
error_handler.cc Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Auto recovery from out of space errors (#4164) 2018-09-15 13:43:04 -07:00
event_helpers.h Auto recovery from out of space errors (#4164) 2018-09-15 13:43:04 -07:00
experimental.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc Checksum properties block for block-based table (#4956) 2019-02-11 11:50:01 -08:00
external_sst_file_ingestion_job.cc Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
external_sst_file_ingestion_job.h Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
external_sst_file_test.cc Dictionary compression for files written by SstFileWriter (#4978) 2019-02-14 11:23:55 -08:00
fault_injection_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
file_indexer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filename_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_job_test.cc Use correct FileMeta for atomic flush result install (#4932) 2019-01-31 14:49:51 -08:00
flush_job.cc Add a new CPU time counter to compaction report (#4889) 2019-01-29 17:24:00 -08:00
flush_job.h Add support to flush multiple CFs atomically (#4262) 2018-10-15 20:01:17 -07:00
flush_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
flush_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
forward_iterator.cc Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
forward_iterator.h Comment out unused variables 2018-03-05 13:13:41 -08:00
internal_stats.cc Add a new CPU time counter to compaction report (#4889) 2019-01-29 17:24:00 -08:00
internal_stats.h Add a new CPU time counter to compaction report (#4889) 2019-01-29 17:24:00 -08:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Use chrono::time_point instead of time_t (#4868) 2019-01-16 09:51:05 -08:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_reader.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
log_writer.cc Pass manual_wal_flush also to the first wal file 2018-05-14 10:57:56 -07:00
log_writer.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
malloc_stats.cc Detect if Jemalloc is linked with the binary (#4844) 2019-01-03 16:30:12 -08:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
memtable_list_test.cc Use correct FileMeta for atomic flush result install (#4932) 2019-01-31 14:49:51 -08:00
memtable_list.cc Avoid using kInAtomicGroup tag for single-cf op (#4981) 2019-02-13 18:33:42 -08:00
memtable_list.h Use correct FileMeta for atomic flush result install (#4932) 2019-01-31 14:49:51 -08:00
memtable.cc Disallow customized hash function in DynamicBloom (#4915) 2019-01-24 10:34:30 -08:00
memtable.h Refactor atomic flush result installation to MANIFEST (#4791) 2019-01-03 20:56:24 -08:00
merge_context.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_helper_test.cc Allow full merge when root of history for a key is reached (#4909) 2019-01-23 21:46:10 -08:00
merge_helper.cc Fix analyze error on possible un-initialized value (#4937) 2019-02-01 09:41:27 -08:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
obsolete_files_test.cc Modify verification logic of ObsoleteOptionsFileTest (#4218) 2018-08-03 13:57:40 -07:00
options_file_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
perf_context_test.cc Allow copy for PerfContext objects (#4919) 2019-02-05 14:29:08 -08:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Remove PlainTable's feature store_index_in_file (#4914) 2019-01-28 12:50:22 -08:00
pre_release_callback.h Fix pre_release callback argument list. 2018-04-05 11:12:16 -07:00
prefix_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
range_del_aggregator_bench.cc Fix Windows broken build error due to non-const override (#4798) 2018-12-19 13:29:51 -08:00
range_del_aggregator_test.cc Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
range_del_aggregator.cc Remove stale TODO (#4800) 2018-12-19 15:45:37 -08:00
range_del_aggregator.h Fix unused member compile error 2018-12-18 14:28:42 -08:00
range_tombstone_fragmenter_test.cc Prepare FragmentedRangeTombstoneIterator for use in compaction (#4740) 2018-12-11 12:10:48 -08:00
range_tombstone_fragmenter.cc Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
range_tombstone_fragmenter.h Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
read_callback.h WriteUnPrepared Txn: Disable seek to snapshot optimization (#3955) 2018-06-27 12:23:07 -07:00
repair_test.cc Acquire lock on DB LOCK file before starting repair. (#4435) 2018-10-12 10:41:54 -07:00
repair.cc Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Remove duplicates from SnapshotList::GetAll (#4860) 2019-01-09 16:25:42 -08:00
table_cache.cc Introduce a CPU time counter in perf_context (#4741) 2018-12-20 12:03:44 -08:00
table_cache.h Preload some files even if options.max_open_files (#3340) 2018-12-28 18:02:28 -08:00
table_properties_collector_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
table_properties_collector.cc Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
table_properties_collector.h Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
transaction_log_impl.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
transaction_log_impl.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
version_builder_test.cc Remove random writes from SST file ingestion (#4172) 2018-07-27 16:12:23 -07:00
version_builder.cc Preload some files even if options.max_open_files (#3340) 2018-12-28 18:02:28 -08:00
version_builder.h Preload some files even if options.max_open_files (#3340) 2018-12-28 18:02:28 -08:00
version_edit_test.cc Add a unit test to Ignorable manfiest record (#4964) 2019-02-11 11:20:24 -08:00
version_edit.cc Add a placeholder in manifest indicating ignorable record (#4960) 2019-02-08 11:33:11 -08:00
version_edit.h Add a unit test to Ignorable manfiest record (#4964) 2019-02-11 11:20:24 -08:00
version_set_test.cc Refactor atomic flush result installation to MANIFEST (#4791) 2019-01-03 20:56:24 -08:00
version_set.cc Atomic ingest (#4895) 2019-02-12 19:16:17 -08:00
version_set.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
wal_manager_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
wal_manager.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
wal_manager.h Fix memleak when DB::DeleteFile() 2018-01-11 18:57:33 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h WriteUnPrepared: Add new WAL marker kTypeBeginUnprepareXID (#4069) 2018-06-28 18:58:29 -07:00
write_batch_test.cc Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
write_batch.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
write_callback_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Add more sync point to fix flaky test GroupCommitTest 2018-11-07 14:07:53 -08:00
write_thread.h Fix skip WAL for whole write_group when leader's callback fail (#4838) 2019-01-03 12:40:42 -08:00