andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
b00761eea6
rocksdb/db
History
Andrew Kryczka b00761eea6 Fix block cache ID uniqueness for Windows builds (#5844) 
Summary:
Since we do not evict a file's blocks from block cache before that file
is deleted, we require a file's cache ID prefix is both unique and
non-reusable. However, the Windows functionality we were relying on only
guaranteed uniqueness. That meant a newly created file could be assigned
the same cache ID prefix as a deleted file. If the newly created file
had block offsets matching the deleted file, full cache keys could be
exactly the same, resulting in obsolete data blocks returned from cache
when trying to read from the new file.

We noticed this when running on FAT32 where compaction was writing out
of order keys due to reading obsolete blocks from its input files. The
functionality is documented as behaving the same on NTFS, although I
wasn't able to repro it there.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5844

Test Plan:
we had a reliable repro of out-of-order keys on FAT32 that
was fixed by this change

Differential Revision: D17752442

fbshipit-source-id: 95d983f9196cf415f269e19293b97341edbf7e00
2019-10-11 18:19:31 -07:00
..
compaction Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_impl MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
arena_wrapped_db_iter.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
arena_wrapped_db_iter.h Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
builder.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
builder.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
c_test.c Add C binding for secondary instance (#5505) 2019-06-27 08:58:54 -07:00
c.cc Remove snap_refresh_nanos option (#5826) 2019-09-18 20:26:04 -07:00
column_family_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
column_family.h Bump up memory order of ref counting of ColumnFamilyData (#5723) 2019-08-20 10:34:33 -07:00
compact_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Do readahead in VerifyChecksum() (#5713) 2019-08-16 16:42:56 -07:00
corruption_test.cc Do readahead in VerifyChecksum() (#5713) 2019-08-16 16:42:56 -07:00
cuckoo_table_db_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
db_basic_test.cc MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
db_blob_index_test.cc Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
db_block_cache_test.cc Charge block cache for cache internal usage (#5797) 2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_flush_test.cc Ref and unref cfd before and after calling WaitForFlushMemTables (#5513) 2019-07-01 14:12:02 -07:00
db_info_dumper.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Improve readability of DBIter's two seek functions (#5794) 2019-09-16 21:05:07 -07:00
db_iter.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
db_iterator_test.cc Fix data block upper bound checking for iterator reseek case (#5883) 2019-10-03 20:53:29 -07:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_options_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_properties_test.cc Refactor/consolidate legacy Bloom implementation details (#5784) 2019-09-16 16:17:09 -07:00
db_range_del_test.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Fix block cache ID uniqueness for Windows builds (#5844) 2019-10-11 18:19:31 -07:00
db_test_util.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_test_util.h Support loading custom objects in unit tests (#5676) 2019-08-09 15:12:08 -07:00
db_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_universal_compaction_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_wal_test.cc Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
deletefile_test.cc Refactor deletefile_test.cc (#5822) 2019-09-18 16:58:21 -07:00
error_handler_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Fix wrong info log printing for num_range_deletions (#5617) 2019-07-23 19:38:16 -07:00
event_helpers.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
external_sst_file_ingestion_job.h Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_test.cc Fix a bug in file ingestion (#5760) 2019-08-30 18:29:07 -07:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
flush_job.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
flush_job.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
import_column_family_job.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
import_column_family_job.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
import_column_family_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.h Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_reader.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
memtable_list.cc MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
memtable_list.h MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
memtable.cc MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
memtable.h MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) 2019-09-18 11:52:17 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
repair.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
table_cache.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
table_properties_collector_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
version_builder.cc save a few redundant container lookups (#5875) 2019-10-07 12:28:09 -07:00
version_builder.h Lower the risk for users to run options.force_consistency_checks = true (#5744) 2019-08-29 14:07:37 -07:00
version_edit_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
version_edit.cc Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
version_edit.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
version_set_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
version_set.cc Revert "Merging iterator to avoid child iterator reseek for some cases (#5286)" (#5871) 2019-10-01 11:22:41 -07:00
version_set.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
wal_manager_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
wal_manager.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
wal_manager.h Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00