rocksdb/db
Zhichao Cao a02495cf80 Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310)
Summary:
In the current implementation, any retryable IO error happens during Flush is mapped to a hard error. In this case, DB is stopped and write is stalled unless the background error is cleaned. In this PR, if WAL is DISABLED, the retryable IO error during FLush is mapped to a soft error. Such that, the memtable can continue receive the writes. At the same time, if auto resume is triggered, SwtichMemtable will not be called during Flush when resuming the DB to avoid to many small memtables. Testing cases are added.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7310

Test Plan: adding new unit test, pass make check.

Reviewed By: anand1976

Differential Revision: D23710892

Pulled By: zhichao-cao

fbshipit-source-id: bc4ca50d11c6b23b60d2c0cb171d86d542b038e9
2020-09-21 07:59:19 -07:00
..
blob Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
compaction Cancel tombstone skipping during bottommost compaction (#7356) 2020-09-11 17:45:43 -07:00
db_impl Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-21 07:59:19 -07:00
arena_wrapped_db_iter.cc dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
arena_wrapped_db_iter.h dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
builder.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
builder.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
c_test.c Add getters to the C API for flush, write, cache and compact options (#7321) 2020-09-09 11:45:27 -07:00
c.cc Add getters to the C API for flush, write, cache and compact options (#7321) 2020-09-09 11:45:27 -07:00
column_family_test.cc Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-21 07:59:19 -07:00
column_family.cc Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
column_family.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
compact_files_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
compacted_db_impl.cc Introduce a global StatsDumpScheduler for stats dumping (#7223) 2020-08-14 20:12:44 -07:00
compacted_db_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
comparator_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
convenience.cc sst_dump to reduce number of file reads (#6836) 2020-05-12 18:23:33 -07:00
corruption_test.cc Upgrade tool chain (#7251) 2020-08-12 19:30:00 -07:00
cuckoo_table_db_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
db_basic_test.cc Parameterize DBBasicTest.CompactBetweenSnapshots (#7301) 2020-08-25 15:42:11 -07:00
db_block_cache_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_bloom_filter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_compaction_filter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_compaction_test.cc Eliminates a no-op compaction upon snapshot release when disabling auto compactions (#7267) 2020-08-24 22:06:45 -07:00
db_dynamic_level_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_encryption_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_filesnapshot.cc Fix checkpoint file deletion race with avoid_unnecessary_blocking_io (#7369) 2020-09-10 22:35:25 -07:00
db_flush_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_info_dumper.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
db_info_dumper.h Add a DB Session ID (#6959) 2020-06-15 10:47:02 -07:00
db_inplace_update_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_io_failure_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_iter_stress_test.cc Test CircleCI with CLANG-10 (#7025) 2020-06-24 16:22:49 -07:00
db_iter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_iter.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
db_iter.h ReadOptions.iter_start_ts should support tombstones (#7178) 2020-08-04 18:52:08 -07:00
db_iterator_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_log_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_logical_block_size_cache_test.cc Get block size only in direct IO mode (#6522) 2020-03-20 15:26:10 -07:00
db_memtable_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operand_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operator_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_options_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_properties_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_range_del_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_sst_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_statistics_test.cc Add a new stats level to exclude tickers (#7329) 2020-09-04 23:25:03 -07:00
db_table_properties_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_tailing_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_test2.cc Get() to fail with underlying failures in PartitionIndexReader::CacheDependencies() (#7297) 2020-08-25 19:01:05 -07:00
db_test_util.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
db_test_util.h Get() to fail with underlying failures in PartitionIndexReader::CacheDependencies() (#7297) 2020-08-25 19:01:05 -07:00
db_test.cc Add unit test for max_write_buffer_size_to_maintain (#7311) 2020-08-28 17:38:05 -07:00
db_universal_compaction_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_wal_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_with_timestamp_basic_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_with_timestamp_compaction_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_write_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
dbformat_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
dbformat.cc Add a new EntryType for deletion with timestamp (#7195) 2020-08-17 16:26:06 -07:00
dbformat.h Separate internal and user key comparators in BlockIter (#6944) 2020-07-07 17:26:16 -07:00
deletefile_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
error_handler_fs_test.cc Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-21 07:59:19 -07:00
error_handler.cc Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-21 07:59:19 -07:00
error_handler.h Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-21 07:59:19 -07:00
event_helpers.cc Pass SST file checksum information through OnTableFileCreated (#7108) 2020-08-25 10:46:11 -07:00
event_helpers.h Pass SST file checksum information through OnTableFileCreated (#7108) 2020-08-25 10:46:11 -07:00
experimental.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_basic_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
external_sst_file_ingestion_job.cc Use FSRandomRWFilePtr Object to call underlying file system. (#7198) 2020-09-08 12:21:58 -07:00
external_sst_file_ingestion_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
external_sst_file_test.cc Disable fsync in some ExternalSSTFileTest tests (#7303) 2020-08-24 11:26:09 -07:00
fault_injection_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
file_indexer_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filename_test.cc Add more tests to ASSERT_STATUS_CHECKED (#7211) 2020-08-06 17:19:41 -07:00
flush_job_test.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
flush_job.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
flush_job.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
flush_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator.cc make L0 index/filter pinned memory usage predictable (#6911) 2020-06-09 16:51:23 -07:00
forward_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
import_column_family_job.cc Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
import_column_family_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
import_column_family_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
internal_stats.cc First step towards handling MANIFEST write error (#6949) 2020-06-24 19:07:08 -07:00
internal_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
job_context.h Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
listener_test.cc Pass SST file checksum information through OnTableFileCreated (#7108) 2020-08-25 10:46:11 -07:00
log_format.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_reader.cc Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_reader.h Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
log_writer.cc Fail recovery when MANIFEST record checksum mismatch (#6996) 2020-06-18 10:09:12 -07:00
log_writer.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
logs_with_prep_tracker.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
logs_with_prep_tracker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
lookup_key.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
manual_compaction_test.cc Skip high levels with no key falling in the range in CompactRange (#6482) 2020-03-04 20:15:25 -08:00
memtable_list_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
memtable_list.cc Bug Fix for memtables not trimmed down. (#7296) 2020-08-21 13:29:05 -07:00
memtable_list.h Bug Fix for memtables not trimmed down. (#7296) 2020-08-21 13:29:05 -07:00
memtable.cc Clean up InternalIterator upper bound logic a little bit (#7200) 2020-08-05 10:44:57 -07:00
memtable.h return timestamp from get (#6409) 2020-03-02 16:01:00 -08:00
merge_context.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
obsolete_files_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
options_file_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
perf_context_test.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
pinned_iterators_manager.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
plain_table_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
pre_release_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
prefix_test.cc Fix cmake build on MacOS (#7205) 2020-08-10 10:48:05 -07:00
range_del_aggregator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
read_callback.h Get() with timestamp should respect snapshot (#7227) 2020-08-14 19:20:58 -07:00
repair_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
repair.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
snapshot_checker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_cache.cc Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
table_cache.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
table_properties_collector_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
table_properties_collector.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_properties_collector.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
transaction_log_impl.cc Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
transaction_log_impl.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
trim_history_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
version_builder_test.cc Fix the handling of the case when a blob file with a lower number gets added in VersionBuilder (#7349) 2020-09-09 10:25:12 -07:00
version_builder.cc Fix the handling of the case when a blob file with a lower number gets added in VersionBuilder (#7349) 2020-09-09 10:25:12 -07:00
version_builder.h make L0 index/filter pinned memory usage predictable (#6911) 2020-06-09 16:51:23 -07:00
version_edit_handler.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_handler.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_test.cc Define WAL related classes to be used in VersionEdit and VersionSet (#7164) 2020-08-05 16:34:38 -07:00
version_edit.cc Build blob file reader/writer classes in LITE mode as well (#7272) 2020-08-17 15:19:05 -07:00
version_edit.h BackupEngine supports custom file checksums (#7085) 2020-08-12 13:31:09 -07:00
version_set_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
version_set.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_set.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
wal_edit_test.cc Track WAL in MANIFEST: minor udpates (#7282) 2020-08-20 15:12:00 -07:00
wal_edit.cc Track WAL in MANIFEST: minor udpates (#7282) 2020-08-20 15:12:00 -07:00
wal_edit.h Define WAL related classes to be used in VersionEdit and VersionSet (#7164) 2020-08-05 16:34:38 -07:00
wal_manager_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
wal_manager.cc Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
wal_manager.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
write_batch_base.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
write_callback_test.cc Divide WriteCallbackTest.WriteWithCallbackTest (#7037) 2020-06-30 12:31:30 -07:00
write_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_thread.cc Assert valid linked list for write group (#7375) 2020-09-11 07:58:31 -07:00
write_thread.h Add EnvTestWithParam::OptionsTest to the ASSERT_STATUS_CHECKED passes (#7283) 2020-08-20 19:18:35 -07:00