andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
247d0979aa
rocksdb/db
History
Mike Kolupaev 247d0979aa Support for range skips in compaction filter 
Summary:
This adds the ability for compaction filter to say "drop this key-value, and also drop everything up to key x". This will cause the compaction to seek input iterator to x, without reading the data. This can make compaction much faster when large consecutive chunks of data are filtered out. See the changes in include/rocksdb/compaction_filter.h for the new API.

Along the way this diff also adds ability for compaction filter changing merge operands, similar to how it can change values; we're not going to use this feature, it just seemed easier and cleaner to implement it than to document that it's not implemented :)

The diff is not as big as it may seem, about half of the lines are a test.
Closes https://github.com/facebook/rocksdb/pull/1599

Differential Revision: D4252092

Pulled By: al13n321

fbshipit-source-id: 41e1e48
2016-12-01 07:09:15 -08:00
..
auto_roll_logger_test.cc Cleanup auto-roll logger flush-while-rolling test 2016-07-07 11:35:40 -07:00
auto_roll_logger.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
auto_roll_logger.h Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
builder.cc Range deletion microoptimizations 2016-11-21 12:24:13 -08:00
builder.h Remove Arena in RangeDelAggregator 2016-11-19 14:24:12 -08:00
c_test.c Add C API to set base_backgroud_compactions 2016-11-30 11:09:13 -08:00
c.cc c api: expose option for dynamic level size target 2016-11-30 11:24:14 -08:00
column_family_test.cc Improve Write Stalling System 2016-11-23 09:24:15 -08:00
column_family.cc Fix implicit conversion between int64_t to int 2016-11-29 10:54:15 -08:00
column_family.h Improve Write Stalling System 2016-11-23 09:24:15 -08:00
compact_files_test.cc fix data race in compact_files_test 2016-10-19 13:37:51 -07:00
compacted_db_impl.cc DeleteRange Get support 2016-11-03 18:54:20 -07:00
compacted_db_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iteration_stats.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_iterator_test.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_iterator.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_iterator.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_job_stats_test.cc UniversalCompaction should ignore sorted runs being compacted (when compacting for file num) 2016-08-10 12:37:43 -07:00
compaction_job_test.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
compaction_job.cc Fix mis-reporting of compaction read bytes to the base level 2016-11-29 11:09:22 -08:00
compaction_job.h DeleteRange compaction statistics 2016-11-28 11:54:12 -08:00
compaction_picker_test.cc Fix a bug that may cause a deleted row to appear again 2016-10-24 09:49:07 -07:00
compaction_picker.cc Change max_bytes_for_level_multiplier to double 2016-11-01 21:09:23 -07:00
compaction_picker.h Fix compaction conflict with running compaction 2016-10-13 10:49:06 -07:00
compaction.cc Refactor MutableCFOptions 2016-09-13 21:11:59 -07:00
compaction.h Fix compaction conflict with running compaction 2016-10-13 10:49:06 -07:00
comparator_db_test.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Change several option defaults 2016-04-28 17:50:58 -07:00
cuckoo_table_db_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_block_cache_test.cc Not insert into block cache if cache is full and not holding handle 2016-08-23 13:53:49 -07:00
db_bloom_filter_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_compaction_filter_test.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
db_compaction_test.cc Eliminate redundant cache lookup with range deletion 2016-11-21 21:24:11 -08:00
db_dynamic_level_test.cc Merge options source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes 2016-09-01 14:33:24 -07:00
db_filesnapshot.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_flush_test.cc Fix unit test which breaks lite build 2016-07-21 14:32:12 -07:00
db_impl_debug.cc Make max_background_compactions and base_background_compactions dynamic changeable 2016-10-14 12:25:39 -07:00
db_impl_experimental.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_impl_readonly.cc Lazily initialize RangeDelAggregator's map and pinning manager 2016-11-18 17:09:11 -08:00
db_impl_readonly.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_impl.h Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_info_dumper.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_info_dumper.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_inplace_update_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_io_failure_test.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_iter_test.cc fix db_stress assertion failure 2016-10-18 16:07:10 -07:00
db_iter.cc Remove unused assignment in db/db_iter.cc 2016-11-29 09:09:14 -08:00
db_iter.h DeleteRange user iterator support 2016-11-04 12:09:22 -07:00
db_iterator_test.cc Less linear search in DBIter::Seek() when keys are overwritten a lot 2016-11-28 10:24:11 -08:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_memtable_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_options_test.cc Dynamic max_total_wal_size option 2016-11-14 22:54:17 -08:00
db_properties_test.cc Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
db_range_del_test.cc deleterange end-to-end test improvements for lite/robustness 2016-11-29 12:24:13 -08:00
db_sst_test.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_table_properties_test.cc DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky 2016-07-19 15:46:20 -07:00
db_tailing_iter_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_test2.cc Persistent Cache: Expose stats to user via public API 2016-11-21 17:39:13 -08:00
db_test_util.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_test_util.h Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_test.cc Improve Write Stalling System 2016-11-23 09:24:15 -08:00
db_universal_compaction_test.cc Restrict running condition of UniversalCompactionTrivialMoveTest2 2016-10-20 15:43:00 -07:00
db_wal_test.cc Fix 2PC Recovery SeqId Miscount 2016-11-10 11:09:22 -08:00
dbformat_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
dbformat.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
dbformat.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
deletefile_test.cc Fix a bug that accesses invalid address in iterator cleanup function 2016-07-05 11:57:14 -07:00
event_helpers.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
event_helpers.h Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
external_sst_file_ingestion_job.cc DeleteRange user iterator support 2016-11-04 12:09:22 -07:00
external_sst_file_ingestion_job.h Support IngestExternalFile (remove AddFile restrictions) 2016-10-20 17:05:32 -07:00
external_sst_file_test.cc Fix SstFileWriter destructor 2016-11-12 20:11:19 -08:00
fault_injection_test.cc disable UBSAN for functions with intentional -ve shift / overflow 2016-11-28 17:54:12 -08:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
filename.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
flush_job_test.cc Make range deletion inclusive-exclusive 2016-11-14 17:39:13 -08:00
flush_job.cc Range deletion microoptimizations 2016-11-21 12:24:13 -08:00
flush_job.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
flush_scheduler.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator.cc Decouple data iterator and range deletion iterator in TableCache 2016-11-15 17:24:28 -08:00
forward_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
inlineskiplist_test.cc Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
inlineskiplist.h Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
internal_stats.cc Exporting compaction stats in the form of a map 2016-11-11 20:54:14 -08:00
internal_stats.h Exporting compaction stats in the form of a map 2016-11-11 20:54:14 -08:00
job_context.h fixup commit 2016-10-13 08:48:40 -07:00
listener_test.cc add cfh deletion started listener 2016-09-22 11:56:18 -07:00
log_format.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
log_reader.h db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_test.cc db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Need to make sure log file synced before flushing memtable of one column family 2016-07-21 16:29:06 -07:00
managed_iterator.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
managed_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.cc Avoid updating memtable allocated bytes if write_buffer_size is not set 2016-07-13 19:33:57 -07:00
memtable_allocator.h Add options.write_buffer_manager: control total memtable size across DB instances 2016-07-05 18:11:25 -07:00
memtable_list_test.cc DeleteRange Get support 2016-11-03 18:54:20 -07:00
memtable_list.cc Remove Arena in RangeDelAggregator 2016-11-19 14:24:12 -08:00
memtable_list.h DeleteRange user iterator support 2016-11-04 12:09:22 -07:00
memtable.cc Less linear search in DBIter::Seek() when keys are overwritten a lot 2016-11-28 10:24:11 -08:00
memtable.h Range deletion microoptimizations 2016-11-21 12:24:13 -08:00
memtablerep_bench.cc Add options.write_buffer_manager: control total memtable size across DB instances 2016-07-05 18:11:25 -07:00
merge_context.h Fix MergeContext::copied_operands_ strings moving 2016-07-25 15:31:41 -07:00
merge_helper_test.cc Compaction Support for Range Deletion 2016-10-18 12:04:56 -07:00
merge_helper.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
merge_helper.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
merge_operator.cc Introduce FullMergeV2 (eliminate memcpy from merge operators) 2016-07-20 09:49:03 -07:00
merge_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h Minor PinnedIteratorsManager Refactoring 2016-08-11 11:54:17 -07:00
plain_table_db_test.cc Allow plain table to store index on file with bloom filter disabled 2016-11-17 11:09:13 -08:00
prefix_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
range_del_aggregator.cc DeleteRange compaction statistics 2016-11-28 11:54:12 -08:00
range_del_aggregator.h DeleteRange compaction statistics 2016-11-28 11:54:12 -08:00
repair_test.cc ColumnFamilyOptions API [CF + RepairDB part 3/3] 2016-06-24 16:29:43 -07:00
repair.cc Remove Arena in RangeDelAggregator 2016-11-19 14:24:12 -08:00
skiplist_test.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
skiplist.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc Fix range deletion covering key in same SST file 2016-11-28 22:54:13 -08:00
table_cache.h Eliminate redundant cache lookup with range deletion 2016-11-21 21:24:11 -08:00
table_properties_collector_test.cc Change ioptions to store user_comparator, fix bug 2016-10-21 11:31:42 -07:00
table_properties_collector.cc Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
table_properties_collector.h Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
transaction_log_impl.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
transaction_log_impl.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
version_builder_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_builder.cc Support IngestExternalFile (remove AddFile restrictions) 2016-10-20 17:05:32 -07:00
version_builder.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Miscellaneous performance improvements 2016-07-12 14:15:32 -07:00
version_edit.h Avoid FileMetaData copy 2016-07-13 15:36:22 -07:00
version_set_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_set.cc Decouple data iterator and range deletion iterator in TableCache 2016-11-15 17:24:28 -08:00
version_set.h DeleteRange user iterator support 2016-11-04 12:09:22 -07:00
wal_manager_test.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
wal_manager.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
wal_manager.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
write_batch_base.cc WriteBatch support for range deletion 2016-08-16 08:16:04 -07:00
write_batch_internal.h Add facility to write only a portion of WriteBatch to WAL 2016-10-07 11:32:10 -07:00
write_batch_test.cc Range deletion microoptimizations 2016-11-21 12:24:13 -08:00
write_batch.cc DeleteRange unsupported in non-block-based tables 2016-11-15 15:24:16 -08:00
write_callback_test.cc write_callback_test: clean test directory before running tests 2016-04-18 10:18:41 -07:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Improve Write Stalling System 2016-11-23 09:24:15 -08:00
write_controller.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.h Improve Write Stalling System 2016-11-23 09:24:15 -08:00
write_thread.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
write_thread.h Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
xfunc_test_points.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00