andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
18eeb7b90e
rocksdb/db
History
Mike Kolupaev 18eeb7b90e Fix interference between max_total_wal_size and db_write_buffer_size checks 
Summary:
This is a trivial fix for OOMs we've seen a few days ago in logdevice.

RocksDB get into the following state:
(1) Write throughput is too high for flushes to keep up. Compactions are out of the picture - automatic compactions are disabled, and for manual compactions we don't care that much if they fall behind. We write to many CFs, with only a few L0 sst files in each, so compactions are not needed most of the time.
(2) total_log_size_ is consistently greater than GetMaxTotalWalSize(). It doesn't get smaller since flushes are falling ever further behind.
(3) Total size of memtables is way above db_write_buffer_size and keeps growing. But the write_buffer_manager_->ShouldFlush() is not checked because (2) prevents it (for no good reason, afaict; this is what this commit fixes).
(4) Every call to WriteImpl() hits the MaybeFlushColumnFamilies() path. This keeps flushing the memtables one by one in order of increasing log file number.
(5) No write stalling trigger is hit. We rely on max_write_buffer_number
Closes https://github.com/facebook/rocksdb/pull/1893

Differential Revision: D4593590

Pulled By: yiwu-arbug

fbshipit-source-id: af79c5f
2017-02-21 16:09:10 -08:00
..
auto_roll_logger_test.cc Windows thread 2017-02-06 14:54:18 -08:00
auto_roll_logger.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
auto_roll_logger.h Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
builder.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
builder.h Remove Arena in RangeDelAggregator 2016-11-19 14:24:12 -08:00
c_test.c c: allow set savepoint to writebatch 2017-01-20 13:24:13 -08:00
c.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
column_family_test.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
column_family.cc Fix for 2PC causing WAL to grow too large 2017-01-19 15:39:12 -08:00
column_family.h Fix for 2PC causing WAL to grow too large 2017-01-19 15:39:12 -08:00
compact_files_test.cc Windows thread 2017-02-06 14:54:18 -08:00
compacted_db_impl.cc Revert "PinnableSlice" 2017-01-08 14:24:12 -08:00
compacted_db_impl.h Disable IngestExternalFile in ReadOnly mode 2017-01-19 15:54:19 -08:00
compaction_iteration_stats.h Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
compaction_iterator_test.cc Windows thread 2017-02-06 14:54:18 -08:00
compaction_iterator.cc Abort compactions more reliably when closing DB 2017-01-11 15:09:21 -08:00
compaction_iterator.h Abort compactions more reliably when closing DB 2017-01-11 15:09:21 -08:00
compaction_job_stats_test.cc UniversalCompaction should ignore sorted runs being compacted (when compacting for file num) 2016-08-10 12:37:43 -07:00
compaction_job_test.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
compaction_job.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
compaction_job.h Abort compactions more reliably when closing DB 2017-01-11 15:09:21 -08:00
compaction_picker_test.cc level compaction expansion 2017-02-21 10:24:17 -08:00
compaction_picker.cc level compaction expansion 2017-02-21 10:24:17 -08:00
compaction_picker.h Compaction::IsTrivialMove relaxing 2016-12-07 11:54:11 -08:00
compaction.cc Compaction::IsTrivialMove relaxing 2016-12-07 11:54:11 -08:00
compaction.h Compaction::IsTrivialMove relaxing 2016-12-07 11:54:11 -08:00
comparator_db_test.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Change corruption_test to use 4 bits. 2017-01-27 11:24:16 -08:00
cuckoo_table_db_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_block_cache_test.cc Not insert into block cache if cache is full and not holding handle 2016-08-23 13:53:49 -07:00
db_bloom_filter_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_compaction_filter_test.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
db_compaction_test.cc Windows thread 2017-02-06 14:54:18 -08:00
db_dynamic_level_test.cc Windows thread 2017-02-06 14:54:18 -08:00
db_filesnapshot.cc Remove unnecessary header include 2016-12-09 11:54:21 -08:00
db_flush_test.cc avoid direct io in rocksdb_lite 2017-02-16 10:39:13 -08:00
db_impl_debug.cc Fix for 2PC causing WAL to grow too large 2017-01-19 15:39:12 -08:00
db_impl_experimental.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_impl_readonly.cc Revert "PinnableSlice" 2017-01-08 14:24:12 -08:00
db_impl_readonly.h Disable IngestExternalFile in ReadOnly mode 2017-01-19 15:54:19 -08:00
db_impl.cc Fix interference between max_total_wal_size and db_write_buffer_size checks 2017-02-21 16:09:10 -08:00
db_impl.h Make DBImpl::has_unpersisted_data_ atomic 2017-02-13 18:54:13 -08:00
db_info_dumper.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_info_dumper.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
db_inplace_update_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_io_failure_test.cc Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
db_iter_test.cc Rename merger.h -> merging_iterator.h 2017-02-02 16:54:19 -08:00
db_iter.cc Maintain position in range deletions map 2017-01-05 10:39:12 -08:00
db_iter.h DeleteRange user iterator support 2016-11-04 12:09:22 -07:00
db_iterator_test.cc Windows thread 2017-02-06 14:54:18 -08:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_memtable_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_merge_operator_test.cc Iterator should be in corrupted status if merge operator return false 2016-12-16 11:09:16 -08:00
db_options_test.cc Made delete_obsolete_files_period_micros option dynamic 2016-12-05 14:24:16 -08:00
db_properties_test.cc Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
db_range_del_test.cc Fix DeleteRange including sentinels in output files 2017-01-25 11:09:12 -08:00
db_sst_test.cc fixed typo 2017-01-23 12:54:13 -08:00
db_table_properties_test.cc DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky 2016-07-19 15:46:20 -07:00
db_tailing_iter_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
db_test2.cc Windows thread 2017-02-06 14:54:18 -08:00
db_test_util.cc Two-level Indexes 2017-02-06 16:39:12 -08:00
db_test_util.h Two-level Indexes 2017-02-06 16:39:12 -08:00
db_test.cc Remove timeout_hint_us from WriteOptions 2017-02-17 15:24:17 -08:00
db_universal_compaction_test.cc Restrict running condition of UniversalCompactionTrivialMoveTest2 2016-10-20 15:43:00 -07:00
db_wal_test.cc Windows thread 2017-02-06 14:54:18 -08:00
dbformat_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
dbformat.cc Support for range skips in compaction filter 2016-12-01 07:09:15 -08:00
dbformat.h Collapse range deletions 2016-12-19 16:54:12 -08:00
deletefile_test.cc Fix a bug that accesses invalid address in iterator cleanup function 2016-07-05 11:57:14 -07:00
event_helpers.cc Performance: Iterate vector by reference 2017-01-11 10:54:37 -08:00
event_helpers.h Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
external_sst_file_ingestion_job.cc Allow adding external v1 sst file with no global seqno support 2017-02-16 17:09:12 -08:00
external_sst_file_ingestion_job.h Add EventListener::OnExternalFileIngested() event 2016-12-06 14:09:17 -08:00
external_sst_file_test.cc Windows thread 2017-02-06 14:54:18 -08:00
fault_injection_test.cc Fix travis (compile for clang < 3.9) 2016-12-01 10:09:22 -08:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc gcc-7 requires include <functional> for std::function 2016-12-16 11:24:18 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
filename.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
flush_job_test.cc Make range deletion inclusive-exclusive 2016-11-14 17:39:13 -08:00
flush_job.cc Remove disableDataSync option 2017-02-13 11:09:13 -08:00
flush_job.h Flush job should release reference current version if sync log failed 2017-01-19 23:09:15 -08:00
flush_scheduler.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc Windows thread 2017-02-06 14:54:18 -08:00
forward_iterator.cc Rename merger.h -> merging_iterator.h 2017-02-02 16:54:19 -08:00
forward_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
inlineskiplist_test.cc Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
inlineskiplist.h Unified InlineSkipList::Insert algorithm with hinting 2016-11-22 14:09:13 -08:00
internal_stats.cc Print the missed last layer in cfstats 2017-02-08 10:39:15 -08:00
internal_stats.h Print the missed last layer in cfstats 2017-02-08 10:39:15 -08:00
job_context.h fixup commit 2016-10-13 08:48:40 -07:00
listener_test.cc Gcc 7 error expansion to defined 2016-12-13 18:39:14 -08:00
log_format.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.cc Fix clang analyzer errors 2016-07-08 17:50:51 -07:00
log_reader.h db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_test.cc db/log_reader: treat bad record length or checksum as EOF 2016-05-22 22:00:15 -07:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Need to make sure log file synced before flushing memtable of one column family 2016-07-21 16:29:06 -07:00
managed_iterator.cc Rename merger.h -> merging_iterator.h 2017-02-02 16:54:19 -08:00
managed_iterator.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.cc Avoid updating memtable allocated bytes if write_buffer_size is not set 2016-07-13 19:33:57 -07:00
memtable_allocator.h Add options.write_buffer_manager: control total memtable size across DB instances 2016-07-05 18:11:25 -07:00
memtable_list_test.cc DeleteRange Get support 2016-11-03 18:54:20 -07:00
memtable_list.cc Adding GetApproximateMemTableStats method 2017-02-06 14:54:16 -08:00
memtable_list.h Adding GetApproximateMemTableStats method 2017-02-06 14:54:16 -08:00
memtable.cc Adding GetApproximateMemTableStats method 2017-02-06 14:54:16 -08:00
memtable.h Adding GetApproximateMemTableStats method 2017-02-06 14:54:16 -08:00
memtablerep_bench.cc Windows thread 2017-02-06 14:54:18 -08:00
merge_context.h Fix MergeContext::copied_operands_ strings moving 2016-07-25 15:31:41 -07:00
merge_helper_test.cc Compaction Support for Range Deletion 2016-10-18 12:04:56 -07:00
merge_helper.cc Abort compactions more reliably when closing DB 2017-01-11 15:09:21 -08:00
merge_helper.h Abort compactions more reliably when closing DB 2017-01-11 15:09:21 -08:00
merge_operator.cc Introduce FullMergeV2 (eliminate memcpy from merge operators) 2016-07-20 09:49:03 -07:00
merge_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc Windows thread 2017-02-06 14:54:18 -08:00
pinned_iterators_manager.h Delegate Cleanables 2016-12-29 15:54:19 -08:00
plain_table_db_test.cc Allow plain table to store index on file with bloom filter disabled 2016-11-17 11:09:13 -08:00
prefix_test.cc Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default 2016-11-16 09:39:09 -08:00
range_del_aggregator_test.cc Maintain position in range deletions map 2017-01-05 10:39:12 -08:00
range_del_aggregator.cc Fix DeleteRange including sentinels in output files 2017-01-25 11:09:12 -08:00
range_del_aggregator.h Maintain position in range deletions map 2017-01-05 10:39:12 -08:00
repair_test.cc avoid direct io in rocksdb_lite 2017-02-16 10:39:13 -08:00
repair.cc Fix repair issues 2017-02-10 10:54:12 -08:00
skiplist_test.cc Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
skiplist.h Add SeekForPrev() to Iterator 2016-09-27 18:20:57 -07:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc Fix range deletion covering key in same SST file 2016-11-28 22:54:13 -08:00
table_cache.h Eliminate redundant cache lookup with range deletion 2016-11-21 21:24:11 -08:00
table_properties_collector_test.cc Change ioptions to store user_comparator, fix bug 2016-10-21 11:31:42 -07:00
table_properties_collector.cc Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
table_properties_collector.h Added "number of merge operands" to statistics in ssts. 2016-05-19 14:24:48 -07:00
transaction_log_impl.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
transaction_log_impl.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
version_builder_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_builder.cc Windows thread 2017-02-06 14:54:18 -08:00
version_builder.h Only cache level 0 indexes and filter when opening table reader 2016-07-20 11:23:31 -07:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Miscellaneous performance improvements 2016-07-12 14:15:32 -07:00
version_edit.h Avoid FileMetaData copy 2016-07-13 15:36:22 -07:00
version_set_test.cc Support running consistency checks in release mode 2016-10-07 17:21:45 -07:00
version_set.cc level compaction expansion 2017-02-21 10:24:17 -08:00
version_set.h level compaction expansion 2017-02-21 10:24:17 -08:00
wal_manager_test.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
wal_manager.cc Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
wal_manager.h Split DBOptions into ImmutableDBOptions and MutableDBOptions 2016-09-23 16:34:04 -07:00
write_batch_base.cc WriteBatch support for range deletion 2016-08-16 08:16:04 -07:00
write_batch_internal.h Add facility to write only a portion of WriteBatch to WAL 2016-10-07 11:32:10 -07:00
write_batch_test.cc Range deletion microoptimizations 2016-11-21 12:24:13 -08:00
write_batch.cc DeleteRange unsupported in non-block-based tables 2016-11-15 15:24:16 -08:00
write_callback_test.cc Windows thread 2017-02-06 14:54:18 -08:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Use monotonic time points in write_controller.cc and rate_limiter.cc 2017-02-14 18:24:24 -08:00
write_controller.cc Use monotonic time points in write_controller.cc and rate_limiter.cc 2017-02-14 18:24:24 -08:00
write_controller.h Use monotonic time points in write_controller.cc and rate_limiter.cc 2017-02-14 18:24:24 -08:00
write_thread.cc Fix 2PC with concurrent memtable insert 2017-01-17 11:24:28 -08:00
write_thread.h Add WriteOptions.no_slowdown 2016-11-21 18:09:13 -08:00
xfunc_test_points.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00