andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
843d2e3137
rocksdb/db
History
Andrew Kryczka 843d2e3137 Shared dictionary compression using reference block 
Summary:
This adds a new metablock containing a shared dictionary that is used
to compress all data blocks in the SST file. The size of the shared dictionary
is configurable in CompressionOptions and defaults to 0. It's currently only
used for zlib/lz4/lz4hc, but the block will be stored in the SST regardless of
the compression type if the user chooses a nonzero dictionary size.

During compaction, computes the dictionary by randomly sampling the first
output file in each subcompaction. It pre-computes the intervals to sample
by assuming the output file will have the maximum allowable length. In case
the file is smaller, some of the pre-computed sampling intervals can be beyond
end-of-file, in which case we skip over those samples and the dictionary will
be a bit smaller. After the dictionary is generated using the first file in a
subcompaction, it is loaded into the compression library before writing each
block in each subsequent file of that subcompaction.

On the read path, gets the dictionary from the metablock, if it exists. Then,
loads that dictionary into the compression library before reading each block.

Test Plan: new unit test

Reviewers: yhchiang, IslamAbdelRahman, cyan, sdong

Reviewed By: sdong

Subscribers: andrewkr, yoshinorim, kradhakrishnan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52287
2016-04-27 17:36:03 -07:00
..
auto_roll_logger_test.cc Fix unit tests issues on Windows (#1078) 2016-04-14 17:33:53 -07:00
auto_roll_logger.cc Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
auto_roll_logger.h Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
builder.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
builder.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
c_test.c Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
c.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
column_family_test.cc Fix unit tests issues on Windows (#1078) 2016-04-14 17:33:53 -07:00
column_family.cc ColumnFamilyOptions SanitizeOptions is buggy on 32-bit platforms. 2016-03-14 16:21:54 -07:00
column_family.h Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
compact_files_test.cc Fixed CompactFiles() spuriously failing or corrupting DB 2016-02-22 13:54:58 -08:00
compacted_db_impl.cc CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
compacted_db_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator.cc Fixing snapshot 0 assertion 2016-04-16 01:47:15 -07:00
compaction_iterator.h Fix data race issue when sub-compaction is used in CompactionJob 2016-03-24 19:36:39 -07:00
compaction_job_stats_test.cc Introduce XPRESS compresssion on Windows. (#1081) 2016-04-19 22:54:24 -07:00
compaction_job_test.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_job.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
compaction_job.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_picker_test.cc Change some RocksDB default options 2016-03-31 17:12:18 -07:00
compaction_picker.cc Compaction always needs to be removed from level0_compactions_in_progress_ for universal compaction 2016-03-02 21:23:28 -08:00
compaction_picker.h Improve comment in compaction.h and compaction_picker.h 2016-03-08 16:46:41 -08:00
compaction.cc Fix data race issue when sub-compaction is used in CompactionJob 2016-03-24 19:36:39 -07:00
compaction.h Fix data race issue when sub-compaction is used in CompactionJob 2016-03-24 19:36:39 -07:00
comparator_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cuckoo_table_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_block_cache_test.cc Fix db_block_cache_test in lite build 2016-04-18 11:34:11 -07:00
db_bloom_filter_test.cc Split db_test.cc 2016-04-18 09:42:50 -07:00
db_compaction_filter_test.cc Change some RocksDB default options 2016-03-31 17:12:18 -07:00
db_compaction_test.cc Split db_test.cc 2016-04-18 09:42:50 -07:00
db_dynamic_level_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_filesnapshot.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_readonly.cc Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
db_impl_readonly.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl.cc Print memory allocation counters 2016-04-27 16:23:33 -07:00
db_impl.h CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
db_info_dumper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_info_dumper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc Change some RocksDB default options 2016-03-31 17:12:18 -07:00
db_iter_test.cc Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
db_iter.cc Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
db_iter.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
db_iterator_test.cc Split db_test.cc 2016-04-18 09:42:50 -07:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_properties_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
db_sst_test.cc Fix DBTest.RateLimitedDelete flakiness 2016-04-19 14:05:48 -07:00
db_table_properties_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
db_tailing_iter_test.cc Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number" 2016-03-04 18:15:29 -08:00
db_test2.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
db_test_util.cc Add per-level compression ratio property 2016-04-20 18:46:54 -07:00
db_test_util.h Add per-level compression ratio property 2016-04-20 18:46:54 -07:00
db_test.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
db_universal_compaction_test.cc Alpine Linux Build (#990) 2016-04-22 16:49:12 -07:00
db_wal_test.cc Introduce XPRESS compresssion on Windows. (#1081) 2016-04-19 22:54:24 -07:00
dbformat_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
dbformat.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dbformat.h WriteBatchWithIndex micro optimization 2016-04-01 15:23:46 -07:00
deletefile_test.cc Handle concurrent manifest update and backup creation 2016-02-29 12:56:55 -08:00
event_helpers.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_helpers.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_job_test.cc Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. 2016-04-15 10:22:18 -07:00
flush_job.cc Correct Statistics FLUSH_WRITE_BYTES 2016-04-25 12:01:01 -07:00
flush_job.h Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. 2016-04-15 10:22:18 -07:00
flush_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator.cc Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number" 2016-03-04 18:15:29 -08:00
forward_iterator.h Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
inlineskiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
inlineskiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
internal_stats.cc Shouldn't report default column family's compaction stats as DB compaction stats 2016-04-25 13:56:59 -07:00
internal_stats.h Shouldn't report default column family's compaction stats as DB compaction stats 2016-04-25 13:56:59 -07:00
job_context.h Cleanup stale manifests outside of full purge 2016-03-10 18:16:21 -08:00
listener_test.cc Make EventListenerTest.CompactionReasonLevel more deterministic 2016-04-25 18:18:35 -07:00
log_format.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
managed_iterator.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
managed_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
memtable_list.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable.cc Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
memtable.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtablerep_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_context.h Eliminate std::deque initialization while iterating over merge operands 2016-04-01 15:48:55 -07:00
merge_helper_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_operator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
plain_table_db_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
prefix_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
repair_test.cc Add unit tests for RepairDB 2016-03-18 15:18:42 -07:00
repair.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
skiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
skiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
table_cache.h Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
table_properties_collector_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
table_properties_collector.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_properties_collector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
transaction_log_impl.cc In all the places where log records are read, there was a check that 2016-03-30 23:05:22 +05:30
transaction_log_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder.cc Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
version_builder.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set.cc Add per-level compression ratio property 2016-04-20 18:46:54 -07:00
version_set.h Add per-level compression ratio property 2016-04-20 18:46:54 -07:00
wal_manager_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
wal_manager.cc In all the places where log records are read, there was a check that 2016-03-30 23:05:22 +05:30
wal_manager.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_base.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_internal.h In all the places where log records are read, there was a check that 2016-03-30 23:05:22 +05:30
write_batch_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch.cc WriteBatchWithIndex micro optimization 2016-04-01 15:23:46 -07:00
write_callback_test.cc write_callback_test: clean test directory before running tests 2016-04-18 10:18:41 -07:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
writebuffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00