rocksdb/include/leveldb
Dhruba Borthakur 47c4191fe8 Reduce write amplification by merging files in L0 back into L0
Summary:
There is a new option called hybrid_mode which, when switched on,
causes HBase style compactions.  Files from L0 are
compacted back into L0. This meat of this compaction algorithm
is in PickCompactionHybrid().

All files reside in L0. That means all files have overlapping
keys. Each file has a time-bound, i.e. each file contains a
range of keys that were inserted around the same time. The
start-seqno and the end-seqno refers to the timeframe when
these keys were inserted.  Files that have contiguous seqno
are compacted together into a larger file. All files are
ordered from most recent to the oldest.

The current compaction algorithm starts to look for
candidate files starting from the most recent file. It continues to
add more files to the same compaction run as long as the
sum of the files chosen till now is smaller than the next
candidate file size. This logic needs to be debated
and validated.

The above logic should reduce write amplification to a
large extent... will publish numbers shortly.

Test Plan: dbstress runs for 6 hours with no data corruption (tested so far).

Differential Revision: https://reviews.facebook.net/D11289
2013-06-30 20:07:04 -07:00
..
c.h Fix poor error on num_levels mismatch and few other minor improvements 2013-01-25 15:37:26 -08:00
cache.h Codemod NULL to nullptr 2013-02-28 18:04:58 -08:00
compaction_filter.h [RocksDB] Cleanup compaction filter to use a class interface, instead of function pointer and additional context pointer. 2013-05-13 14:06:10 -07:00
comparator.h A number of fixes: 2011-10-31 17:22:06 +00:00
db.h Very basic Multiget and simple test cases. 2013-06-05 11:22:38 -07:00
env.h [RocksDB] Option for incremental sync 2013-06-18 15:00:32 -07:00
filter_policy.h Added bloom filter support. 2012-04-17 08:36:46 -07:00
iterator.h A number of fixes: 2011-10-31 17:22:06 +00:00
ldb_tool.h [RocksDB] Expose LDB functioanality as a library call - clients can build their own LDB binary with additional options 2013-04-11 20:21:49 -07:00
merge_operator.h [Rocksdb] Support Merge operation in rocksdb 2013-05-03 16:59:02 -07:00
options.h Reduce write amplification by merging files in L0 back into L0 2013-06-30 20:07:04 -07:00
slice.h manifest_dump: Add --hex=1 option 2012-12-16 08:58:28 -08:00
statistics.h Reduce write amplification by merging files in L0 back into L0 2013-06-30 20:07:04 -07:00
status.h [Rocksdb] Support Merge operation in rocksdb 2013-05-03 16:59:02 -07:00
table_builder.h Fix all the lint errors. 2012-11-28 17:18:41 -08:00
transaction_log_iterator.h Do not allow Transaction Log Iterator to fall ahead when writer is writing the same file 2013-03-06 14:05:53 -08:00
types.h GetUpdatesSince API to enable replication. 2012-12-07 11:42:13 -08:00
write_batch.h [RocksDB] Expose count for WriteBatch 2013-06-26 15:13:21 -07:00