andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
afa2420c2b
rocksdb/db
History
anand76 afa2420c2b Introduce a new storage specific Env API (#5761) 
Summary:
The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc.

This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO.

The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before.

This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection.

The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761

Differential Revision: D18868376

Pulled By: anand1976

fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f
2019-12-13 14:48:41 -08:00
..
compaction Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
db_impl Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
arena_wrapped_db_iter.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
arena_wrapped_db_iter.h Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
blob_index.h Mark BlobIndex::DebugString const 2019-12-11 17:19:43 -08:00
builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
c_test.c Fix c_test:filter for various CACHE_LINE_SIZEs (#6153) 2019-12-11 15:17:08 -08:00
c.cc Apply formatter to some recent commits (#6138) 2019-12-09 15:49:49 -08:00
column_family_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
column_family.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
compact_files_test.cc Use aggregate initialization for FlushJobInfo/CompactionJobInfo (#5997) 2019-11-01 11:46:19 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
corruption_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
cuckoo_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
db_basic_test.cc Apply formatter to some recent commits (#6138) 2019-12-09 15:49:49 -08:00
db_blob_index_test.cc Disable blob iterator test with max_sequential_skip_in_iterations==0 in LITE mode (#6052) 2019-11-19 15:02:41 -08:00
db_block_cache_test.cc Charge block cache for cache internal usage (#5797) 2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
db_flush_test.cc Fix a potential bug scheduling unnecessary threads (#6104) 2019-11-27 14:48:49 -08:00
db_info_dumper.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Fix blob context when db_iter uses seek (#6051) 2019-11-19 11:39:02 -08:00
db_iter.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
db_iterator_test.cc Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154) 2019-12-13 14:30:14 -08:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_options_test.cc Make default value of options.ttl to be 30 days when it is supported. (#6073) 2019-11-26 10:00:32 -08:00
db_properties_test.cc Refactor/consolidate legacy Bloom implementation details (#5784) 2019-09-16 16:17:09 -07:00
db_range_del_test.cc Fix RangeDeletion bug (#6062) 2019-12-12 15:18:02 -08:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Fix a regression bug on total order seek with prefix enabled and range delete (#6028) 2019-11-13 10:11:34 -08:00
db_test_util.cc Batched MultiGet API for multiple column families (#5816) 2019-11-12 13:52:55 -08:00
db_test_util.h Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
db_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
db_universal_compaction_test.cc Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125) 2019-12-09 09:23:50 -08:00
db_wal_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
deletefile_test.cc Refactor deletefile_test.cc (#5822) 2019-09-18 16:58:21 -07:00
error_handler_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
event_helpers.h BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_ingestion_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
import_column_family_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.h Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Propagate SST and blob file numbers through the EventListener interface (#5962) 2019-10-24 14:44:15 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_reader.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
log_writer.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
memtable_list.cc Fix RangeDeletion bug (#6062) 2019-12-12 15:18:02 -08:00
memtable_list.h Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
memtable.cc Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00
memtable.h Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) 2019-10-16 10:40:23 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) 2019-09-18 11:52:17 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
repair.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_cache.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_properties_collector_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_edit_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.h Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_set_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_set.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_set.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00