andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
db03739340
rocksdb/db
History
Igor Canadi db03739340 options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 
Summary:
When having fixed max_bytes_for_level_base, the ratio of size of largest level and the second one can range from 0 to the multiplier. This makes LSM tree frequently irregular and unpredictable. It can also cause poor space amplification in some cases.

In this improvement (proposed by Igor Kabiljo), we introduce a parameter option.level_compaction_use_dynamic_max_bytes. When turning it on, RocksDB is free to pick a level base in the range of (options.max_bytes_for_level_base/options.max_bytes_for_level_multiplier, options.max_bytes_for_level_base] so that real level ratios are close to options.max_bytes_for_level_multiplier.

Test Plan: New unit tests and pass tests suites including valgrind.

Reviewers: MarkCallaghan, rven, yhchiang, igor, ikabiljo

Reviewed By: ikabiljo

Subscribers: yoshinorim, ikabiljo, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31437
2015-03-02 22:40:41 -08:00
..
builder.cc Add columnfamily option optimize_filters_for_hits to optimize for key hits only 2015-02-26 16:25:56 -08:00
builder.h Add columnfamily option optimize_filters_for_hits to optimize for key hits only 2015-02-26 16:25:56 -08:00
c_test.c rocksdb: Fixed 'Dead assignment' and 'Dead initialization' scan-build warnings 2015-02-23 14:10:09 -08:00
c.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
column_family_test.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
column_family.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
column_family.h options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction_job_test.cc Introduce job_id for flush and compaction 2015-02-12 09:54:48 -08:00
compaction_job.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction_job.h Introduce job_id for flush and compaction 2015-02-12 09:54:48 -08:00
compaction_picker_test.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction_picker.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction_picker.h options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
compaction.h options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
comparator_db_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
corruption_test.cc Fail DB::Open() on WAL corruption 2015-01-05 10:26:34 -08:00
cuckoo_table_db_test.cc remove unreliable test in db/cuckoo_table_db_test.cc 2014-11-24 15:18:09 -08:00
db_bench.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
db_filesnapshot.cc Introduce job_id for flush and compaction 2015-02-12 09:54:48 -08:00
db_impl_debug.cc Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
db_impl_readonly.cc Move GetThreadList() feature under Env. 2014-12-22 12:20:17 -08:00
db_impl_readonly.h Block ReadOnlyDB in ROCKSDB_LITE 2014-11-26 11:37:59 -08:00
db_impl.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
db_impl.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
db_iter_test.cc Divide test DBIteratorTest.DBIterator to smaller tests 2015-02-03 09:48:03 -08:00
db_iter.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
db_iter.h reduce references to cfd->options() in DBImpl 2014-09-08 15:04:34 -07:00
db_test.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
dbformat_test.cc Use IterKey instead of string in Block::Iter to reduce malloc 2014-07-23 12:31:11 -07:00
dbformat.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
dbformat.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
deletefile_test.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
fault_injection_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
file_indexer_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
file_indexer.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
file_indexer.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
filename_test.cc Support purging logs from separate log directory 2014-08-14 13:22:50 -07:00
filename.cc Sync manifest file when initializing it 2015-01-22 14:32:03 -08:00
filename.h Sync manifest file when initializing it 2015-01-22 14:32:03 -08:00
flush_job_test.cc Introduce job_id for flush and compaction 2015-02-12 09:54:48 -08:00
flush_job.cc Allow GetThreadList to reflect flush activity. 2015-02-17 10:13:52 -08:00
flush_job.h Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
flush_scheduler.cc Don't return (or dereference) dangling pointer 2014-10-02 14:33:16 -07:00
flush_scheduler.h Fix data race #1 2015-01-26 11:48:07 -08:00
forward_iterator.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
forward_iterator.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
internal_stats.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
internal_stats.h options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
job_context.h Introduce job_id for flush and compaction 2015-02-12 09:54:48 -08:00
listener_test.cc Fixed a compile warning in clang in db/listener_test.cc 2015-01-27 15:01:04 -08:00
log_and_apply_bench.cc Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
log_format.h Some minor refactoring on the code 2014-01-02 16:32:31 -08:00
log_reader.cc rocksdb: Fixed 'Dead assignment' and 'Dead initialization' scan-build warnings 2015-02-23 14:10:09 -08:00
log_reader.h Fix UnmarkEOF for partial blocks 2014-01-27 14:49:10 -08:00
log_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
log_writer.cc fix erroneous assert: cast kBlockSize (of type unsigned int) to "int" 2015-02-20 11:07:17 -08:00
log_writer.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
managed_iterator.cc Fix compile error on MacOS. 2015-02-24 16:24:53 -08:00
managed_iterator.h Managed iterator 2015-02-18 11:49:31 -08:00
memtable_allocator.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_list.cc Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
memtable_list.h Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
memtable.cc Instrument memtable seeks 2015-02-27 17:06:06 -08:00
memtable.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
memtablerep_bench.cc Fix clang build 2015-01-13 12:27:28 -08:00
merge_context.h Enhance partial merge to support multiple arguments 2014-03-24 17:57:13 -07:00
merge_helper.cc Turn -Wshadow back on 2014-11-06 11:14:28 -08:00
merge_helper.h Fixed the crash when merge_operator is not properly set after reopen. 2014-07-30 17:24:36 -07:00
merge_operator.cc Some small cleaning up to make some compiling environment happy 2014-03-26 18:11:41 -07:00
merge_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
perf_context_test.cc Perf Context to report DB mutex waiting time 2015-02-09 17:55:12 -08:00
plain_table_db_test.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
prefix_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
repair.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
skiplist_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
skiplist.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
snapshot.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
table_cache.cc TableMock + framework for mock classes 2014-10-28 17:52:32 -07:00
table_cache.h use GetContext to replace callback function pointer 2014-09-29 11:09:09 -07:00
table_properties_collector_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
table_properties_collector.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
table_properties_collector.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
transaction_log_impl.cc Turn -Wshadow back on 2014-11-06 11:14:28 -08:00
transaction_log_impl.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
version_builder_test.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
version_builder.cc Fix deleting obsolete files 2015-02-06 08:44:30 -08:00
version_builder.h Move VersionBuilder logic to a separate .cc file 2014-10-31 16:34:38 -07:00
version_edit_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
version_edit.cc Turn on -Wshadow 2014-10-31 11:59:54 -07:00
version_edit.h Fix deleting obsolete files 2015-02-06 08:44:30 -08:00
version_set_test.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
version_set.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
version_set.h options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
wal_manager_test.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
wal_manager.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
wal_manager.h Fix -Wnon-virtual-dtor errors 2014-11-10 17:39:38 -05:00
write_batch_internal.h remove all remaining references to cfd->options() 2014-11-18 10:20:10 -08:00
write_batch_test.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
write_batch.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
write_controller_test.cc Push- instead of pull-model for managing Write stalls 2014-09-08 11:20:25 -07:00
write_controller.cc Push- instead of pull-model for managing Write stalls 2014-09-08 11:20:25 -07:00
write_controller.h Fix #284 2014-09-13 14:14:10 -07:00
write_thread.cc WriteThread 2014-09-12 16:23:58 -07:00
write_thread.h Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
writebuffer.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00