andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
54de13abac
rocksdb/table
History
Andrew Kryczka 843d2e3137 Shared dictionary compression using reference block 
Summary:
This adds a new metablock containing a shared dictionary that is used
to compress all data blocks in the SST file. The size of the shared dictionary
is configurable in CompressionOptions and defaults to 0. It's currently only
used for zlib/lz4/lz4hc, but the block will be stored in the SST regardless of
the compression type if the user chooses a nonzero dictionary size.

During compaction, computes the dictionary by randomly sampling the first
output file in each subcompaction. It pre-computes the intervals to sample
by assuming the output file will have the maximum allowable length. In case
the file is smaller, some of the pre-computed sampling intervals can be beyond
end-of-file, in which case we skip over those samples and the dictionary will
be a bit smaller. After the dictionary is generated using the first file in a
subcompaction, it is loaded into the compression library before writing each
block in each subsequent file of that subcompaction.

On the read path, gets the dictionary from the metablock, if it exists. Then,
loads that dictionary into the compression library before reading each block.

Test Plan: new unit test

Reviewers: yhchiang, IslamAbdelRahman, cyan, sdong

Reviewed By: sdong

Subscribers: andrewkr, yoshinorim, kradhakrishnan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52287
2016-04-27 17:36:03 -07:00
..
adaptive_table_factory.cc Fix null-pointer-dereference detected by Infer (https://github.com/facebook/infer) 2016-04-25 20:09:36 +01:00
adaptive_table_factory.h Pass column family ID to table property collector 2015-10-09 14:36:51 -07:00
block_based_filter_block_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_based_filter_block.cc Revert "Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set."" 2016-02-22 16:33:26 -08:00
block_based_filter_block.h Revert "Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set."" 2016-02-22 16:33:26 -08:00
block_based_table_builder.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
block_based_table_builder.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
block_based_table_factory.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
block_based_table_factory.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_based_table_reader.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
block_based_table_reader.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
block_builder.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_builder.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_hash_index_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_hash_index.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_hash_index.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_prefix_index.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_prefix_index.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
block.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
bloom_block.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom_block.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cuckoo_table_builder_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
cuckoo_table_builder.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
cuckoo_table_builder.h Embed column family name in SST file 2016-04-06 23:10:32 -07:00
cuckoo_table_factory.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
cuckoo_table_factory.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cuckoo_table_reader_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
cuckoo_table_reader.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cuckoo_table_reader.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filter_block.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_block_policy.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
format.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
format.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
full_filter_block_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
full_filter_block.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
full_filter_block.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
get_context.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
get_context.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
internal_iterator.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
iter_heap.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
iterator_wrapper.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
iterator.cc Rename iterator property "rocksdb.iterator.is.key.pinned" => "rocksdb.iterator.is-key-pinned" 2016-03-01 13:47:12 -08:00
merger_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merger.cc Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
merger.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
meta_blocks.cc Add comparator, merge operator, property collectors to SST file properties (again) 2016-04-21 10:16:28 -07:00
meta_blocks.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mock_table.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mock_table.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_builder.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
plain_table_builder.h Embed column family name in SST file 2016-04-06 23:10:32 -07:00
plain_table_factory.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
plain_table_factory.h Fix build broken by previous commit of "option helper refactor" 2015-11-17 16:52:54 -08:00
plain_table_index.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_index.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_key_coding.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_key_coding.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
plain_table_reader.cc Skip bottom-level filter block caching when hit-optimized 2015-12-23 10:15:07 -08:00
plain_table_reader.h Skip bottom-level filter block caching when hit-optimized 2015-12-23 10:15:07 -08:00
scoped_arena_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
sst_file_writer.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
table_builder.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
table_properties_internal.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
table_properties.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
table_reader_bench.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
table_reader.h Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
table_test.cc Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
two_level_iterator.cc Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
two_level_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00