andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
f383641a1d
rocksdb/table
History
Maysam Yabandeh f383641a1d Unordered Writes (#5218) 
Summary:
Performing unordered writes in rocksdb when unordered_write option is set to true. When enabled the writes to memtable are done without joining any write thread. This offers much higher write throughput since the upcoming writes would not have to wait for the slowest memtable write to finish. The tradeoff is that the writes visible to a snapshot might change over time. If the application cannot tolerate that, it should implement its own mechanisms to work around that. Using TransactionDB with WRITE_PREPARED write policy is one way to achieve that. Doing so increases the max throughput by 2.2x without however compromising the snapshot guarantees.
The patch is prepared based on an original by siying
Existing unit tests are extended to include unordered_write option.

Benchmark Results:
```
TEST_TMPDIR=/dev/shm/ ./db_bench_unordered --benchmarks=fillrandom --threads=32 --num=10000000 -max_write_buffer_number=16 --max_background_jobs=64 --batch_size=8 --writes=3000000 -level0_file_num_compaction_trigger=99999 --level0_slowdown_writes_trigger=99999 --level0_stop_writes_trigger=99999 -enable_pipelined_write=false -disable_auto_compactions  --unordered_write=1
```
With WAL
- Vanilla RocksDB: 78.6 MB/s
- WRITER_PREPARED with unordered_write: 177.8 MB/s (2.2x)
- unordered_write: 368.9 MB/s (4.7x with relaxed snapshot guarantees)

Without WAL
- Vanilla RocksDB: 111.3 MB/s
- WRITER_PREPARED with unordered_write: 259.3 MB/s MB/s (2.3x)
- unordered_write: 645.6 MB/s (5.8x with relaxed snapshot guarantees)

- WRITER_PREPARED with unordered_write disable concurrency control: 185.3 MB/s MB/s (2.35x)

Limitations:
- The feature is not yet extended to `max_successive_merges` > 0. The feature is also incompatible with `enable_pipelined_write` = true as well as with `allow_concurrent_memtable_write` = false.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5218

Differential Revision: D15219029

Pulled By: maysamyabandeh

fbshipit-source-id: 38f2abc4af8780148c6128acdba2b3227bc81759
2019-05-13 17:47:21 -07:00
..
adaptive_table_factory.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
adaptive_table_factory.h Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
block_based_filter_block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_table_builder.cc Fix compilation errors for 32bits/LITE/ios build. (#5220) 2019-04-22 16:02:16 -07:00
block_based_table_builder.h Periodic Compactions (#5166) 2019-04-10 19:31:18 -07:00
block_based_table_factory.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
block_based_table_factory.h Add BlockBasedTableOptions::index_shortening (#5174) 2019-04-22 08:20:35 -07:00
block_based_table_reader.cc Fix crash in BlockBasedTableIterator::Seek() (#5291) 2019-05-10 12:40:57 -07:00
block_based_table_reader.h Turn CachableEntry into a proper resource handle (#5252) 2019-05-10 11:57:49 -07:00
block_builder.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_builder.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.h Cache dictionary used for decompressing data blocks (#4881) 2019-01-23 18:15:47 -08:00
block_prefix_index.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_prefix_index.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block.cc Reduce binary search when reseek into the same data block (#5256) 2019-05-01 14:26:30 -07:00
block.h Introduce InternalIteratorBase::NextAndGetResult() (#5197) 2019-04-18 11:12:39 -07:00
bloom_block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_block.h Disallow customized hash function in DynamicBloom (#4915) 2019-01-24 10:34:30 -08:00
cachable_entry.h Turn CachableEntry into a proper resource handle (#5252) 2019-05-10 11:57:49 -07:00
cleanable_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_builder.cc Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
cuckoo_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_factory.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_factory.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_reader_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_reader.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
cuckoo_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
data_block_footer.cc Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_footer.h Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_hash_index_test.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
data_block_hash_index.cc DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
data_block_hash_index.h DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
filter_block.h Remove a couple of non-public includes from public header file (#5219) 2019-04-19 11:10:33 -07:00
flush_block_policy.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
flush_block_policy.h Avoid per-key upper bound check in BlockBasedTableIterator (#5142) 2019-04-16 11:37:47 -07:00
format.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
format.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
full_filter_bits_builder.h Skip duplicate bloom keys when whole_key and prefix are mixed 2018-04-24 10:58:16 -07:00
full_filter_block_test.cc Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
full_filter_block.cc Fix bugs detected by clang analyzer (#5185) 2019-04-12 10:45:56 -07:00
full_filter_block.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
get_context.cc PlainTable should avoid copying Get() results from immortal source. (#4924) 2019-01-25 17:12:19 -08:00
get_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
index_builder.cc Add BlockBasedTableOptions::index_shortening (#5174) 2019-04-22 08:20:35 -07:00
index_builder.h Add BlockBasedTableOptions::index_shortening (#5174) 2019-04-22 08:20:35 -07:00
internal_iterator.h Merging iterator to avoid child iterator reseek for some cases (#5286) 2019-05-09 14:20:04 -07:00
iter_heap.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
iterator_wrapper.h Merging iterator to avoid child iterator reseek for some cases (#5286) 2019-05-09 14:20:04 -07:00
iterator.cc Add back NewEmptyIterator (#5203) 2019-04-16 20:28:05 -07:00
merger_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
merging_iterator.cc Merging iterator to avoid child iterator reseek for some cases (#5286) 2019-05-09 14:20:04 -07:00
merging_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
meta_blocks.cc Fix compilation errors for 32bits/LITE/ios build. (#5220) 2019-04-22 16:02:16 -07:00
meta_blocks.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
mock_table.cc Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
mock_table.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
multiget_context.h multiget: fix memory issues due to vector auto resizing (#5279) 2019-05-03 15:58:43 -07:00
partitioned_filter_block_test.cc Turn CachableEntry into a proper resource handle (#5252) 2019-05-10 11:57:49 -07:00
partitioned_filter_block.cc Turn CachableEntry into a proper resource handle (#5252) 2019-05-10 11:57:49 -07:00
partitioned_filter_block.h Turn CachableEntry into a proper resource handle (#5252) 2019-05-10 11:57:49 -07:00
persistent_cache_helper.cc Remove two variables from BlockContents class and don't use class Block for compressed block (#4650) 2018-11-13 17:02:55 -08:00
persistent_cache_helper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_options.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.cc Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_builder.h Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_factory.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
plain_table_factory.h Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
plain_table_index.cc Fix many bugs in log statement arguments (#5089) 2019-04-04 12:12:11 -07:00
plain_table_index.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_key_coding.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_key_coding.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
plain_table_reader.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
plain_table_reader.h Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
scoped_arena_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_reader_test.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_reader.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_writer_collectors.h Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_writer.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_builder.h Periodic Compactions (#5166) 2019-04-10 19:31:18 -07:00
table_properties_internal.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_properties.cc Periodic Compactions (#5166) 2019-04-10 19:31:18 -07:00
table_reader_bench.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_reader.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
table_test.cc Fix crash in BlockBasedTableIterator::Seek() (#5291) 2019-05-10 12:40:57 -07:00
two_level_iterator.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
two_level_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00