andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6.25.fb
rocksdb/db
History
Andrew Kryczka fa4e0558bf Fix sequence number bump logic in multi-CF SST ingestion (#9005) 
Summary:
The code in `IngestExternalFiles()` that bumps the DB's sequence number
depending on what seqnos were assigned to the files has 3 bugs:

1) There is an assertion that the sequence number is increased in all the
affected column families, but this is unnecessary, it is fine if some files can
stick to a lower sequence number. It is very easy to hit the assertion: it is
sufficient to insert 2 files in 2 CFs, one which overlaps the CF and one that
doesn't (for example the CF is empty). The line added in the
`IngestFilesIntoMultipleColumnFamilies_Success` test makes the assertion fail.

2) SetLastSequence() is called with the sum of all the bumps across CFs, but we
should take the maximum instead, as all CFs start with the current seqno and bump
it independently.

3) The code above is accidentally under a `#ifndef NDEBUG`, so it doesn't run in
optimized builds, so some files may be assigned seqnos from the future.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9005

Test Plan:
Added line in `IngestFilesIntoMultipleColumnFamilies_Success` that
triggers the assertion, verified that the test (and all the others) pass after
the fix.

Reviewed By: ajkr

Differential Revision: D31597892

Pulled By: ot

fbshipit-source-id: c2d3237f90290df1178736ace8653a9623f5a770
2021-10-14 10:45:19 -07:00
..
blob Sort per-file blob read requests by offset (#8953) 2021-09-28 13:24:17 -07:00
compaction RemoteCompaction support Fallback to local compaction (#8709) 2021-09-18 00:25:04 -07:00
db_impl Fix sequence number bump logic in multi-CF SST ingestion (#9005) 2021-10-14 10:45:19 -07:00
arena_wrapped_db_iter.cc Rename ImmutableOptions variables (#8409) 2021-06-16 16:51:38 -07:00
arena_wrapped_db_iter.h Always iniitalize ArenaWrappedDBIter::db_iter_ to nullptr (#8889) 2021-09-14 14:33:15 -07:00
builder.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
builder.h Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
c_test.c Add Bloom/Ribbon hybrid API support (#8679) 2021-08-20 18:00:16 -07:00
c.cc Add Bloom/Ribbon hybrid API support (#8679) 2021-08-20 18:00:16 -07:00
column_family_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
column_family.cc Add DB properties for BlobDB (#8734) 2021-09-08 12:22:04 -07:00
column_family.h Add DB properties for BlobDB (#8734) 2021-09-08 12:22:04 -07:00
compact_files_test.cc Compaction should not move data to up level (#8116) 2021-03-29 17:10:42 -07:00
comparator_db_test.cc Remove unused includes (#7604) 2020-10-28 23:22:27 -07:00
convenience.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
corruption_test.cc Add CreateFrom methods to Env/FileSystem (#8174) 2021-06-15 03:43:48 -07:00
cuckoo_table_db_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
db_basic_test.cc Change the File System File Wrappers to std::unique_ptr (#8618) 2021-09-13 08:46:19 -07:00
db_block_cache_test.cc Fix PrepopulateBlockCache::kFlushOnly (#8750) 2021-09-15 15:33:20 -07:00
db_bloom_filter_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_compaction_filter_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_compaction_test.cc Cancel manual compactions waiting on automatic compactions to drain (#8991) 2021-10-11 16:36:54 -07:00
db_dynamic_level_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_encryption_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_filesnapshot.cc DB::GetSortedWalFiles() to ensure file deletion is disabled (#8591) 2021-07-29 11:51:08 -07:00
db_flush_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_info_dumper.cc Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
db_info_dumper.h Add a DB Session ID (#6959) 2020-06-15 10:47:02 -07:00
db_inplace_update_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_io_failure_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_iter_stress_test.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
db_iter_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
db_iter.cc Rename ImmutableOptions variables (#8409) 2021-06-16 16:51:38 -07:00
db_iter.h Rename ImmutableOptions variables (#8409) 2021-06-16 16:51:38 -07:00
db_iterator_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_kv_checksum_test.cc Allow WriteBatch to have keys with different timestamp sizes (#8725) 2021-09-12 15:34:26 -07:00
db_log_iter_test.cc Attempt to deflake DBTestXactLogIterator.TransactionLogIteratorCorruptedLog (#8627) 2021-08-10 11:10:07 -07:00
db_logical_block_size_cache_test.cc Add further tests to ASSERT_STATUS_CHECKED (2) (#7698) 2020-12-09 21:21:16 -08:00
db_memtable_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_merge_operand_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_merge_operator_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_options_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_properties_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_range_del_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_secondary_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_sst_test.cc Fix flaky WALTrashCleanupOnOpen (#8917) 2021-09-15 21:31:20 -07:00
db_statistics_test.cc Bytes read stat for VerifyChecksum() and VerifyFileChecksums() APIs (#8741) 2021-09-07 13:28:29 -07:00
db_table_properties_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_tailing_iter_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_test2.cc Make Statistics a Customizable Class (#8637) 2021-09-10 09:47:39 -07:00
db_test_util.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
db_test_util.h Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_test.cc Make RateLimiter::GetTotalPendingRequest() non pure virtual for backward compability (#8938) 2021-09-21 21:48:55 -07:00
db_universal_compaction_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_wal_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_with_timestamp_basic_test.cc Allow WriteBatch to have keys with different timestamp sizes (#8725) 2021-09-12 15:34:26 -07:00
db_with_timestamp_compaction_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_write_buffer_manager_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_write_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
dbformat_test.cc Remove unused includes (#7604) 2020-10-28 23:22:27 -07:00
dbformat.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
dbformat.h Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
deletefile_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
error_handler_fs_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
error_handler.cc DB::GetSortedWalFiles() to ensure file deletion is disabled (#8591) 2021-07-29 11:51:08 -07:00
error_handler.h Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
event_helpers.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
event_helpers.h Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
experimental.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_basic_test.cc Protect existing files in FaultInjectionTest{Env,FS}::ReopenWritableFile() (#8995) 2021-10-11 16:39:36 -07:00
external_sst_file_ingestion_job.cc Support custom Env in db_sst_test and external_sst_file_basic_test (#8888) 2021-09-08 21:21:49 -07:00
external_sst_file_ingestion_job.h Fix sequence number bump logic in multi-CF SST ingestion (#9005) 2021-10-14 10:45:19 -07:00
external_sst_file_test.cc Fix sequence number bump logic in multi-CF SST ingestion (#9005) 2021-10-14 10:45:19 -07:00
fault_injection_test.cc Add more tests for assert status checked (#7524) 2020-12-22 23:45:58 -08:00
file_indexer_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filename_test.cc Remove unused includes (#7604) 2020-10-28 23:22:27 -07:00
flush_job_test.cc Fix NotifyOnFlushCompleted() for atomic flush (#8585) 2021-08-03 13:31:10 -07:00
flush_job.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
flush_job.h Add condition on NotifyOnFlushComplete that FlushJob was not mempurge. Add event listeners to mempurge tests. (#8672) 2021-08-18 17:40:01 -07:00
flush_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_scheduler.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
forward_iterator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator.cc Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
forward_iterator.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
import_column_family_job.cc Add a SystemClock class to capture the time functions of an Env (#7858) 2021-01-25 22:09:11 -08:00
import_column_family_job.h Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
import_column_family_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
internal_stats.cc Use GetBlobFileSize instead of GetTotalBlobBytes in DB properties (#8902) 2021-09-13 10:47:16 -07:00
internal_stats.h Add DB properties for BlobDB (#8734) 2021-09-08 12:22:04 -07:00
job_context.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
kv_checksum.h Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
listener_test.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
log_format.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_reader.cc Fix kPointInTimeRecovery handling of truncated WAL (#7701) 2020-11-30 18:11:38 -08:00
log_reader.h Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_test.cc Make StringEnv, StringSink, StringSource use FS classes (#7786) 2021-01-04 16:01:01 -08:00
log_writer.cc Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412) 2021-06-25 00:47:17 -07:00
log_writer.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
logs_with_prep_tracker.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
logs_with_prep_tracker.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
lookup_key.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
malloc_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
manual_compaction_test.cc Add more tests for assert status checked (#7524) 2020-12-22 23:45:58 -08:00
memtable_list_test.cc Fix NotifyOnFlushCompleted() for atomic flush (#8585) 2021-08-03 13:31:10 -07:00
memtable_list.cc Improve MemPurge sampling (#8656) 2021-08-13 14:35:41 -07:00
memtable_list.h Improve MemPurge sampling (#8656) 2021-08-13 14:35:41 -07:00
memtable.cc Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
memtable.h Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
merge_context.h Add Merge Operator support to WriteBatchWithIndex (#8135) 2021-05-10 12:50:25 -07:00
merge_helper_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.cc Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445) 2021-06-24 18:11:30 -07:00
merge_helper.h Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445) 2021-06-24 18:11:30 -07:00
merge_operator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_test.cc Built-in support for generating unique IDs, bug fix (#8708) 2021-08-30 15:20:41 -07:00
obsolete_files_test.cc Attempt to deflake ObsoleteFilesTest.DeleteObsoleteOptionsFile (#8624) 2021-08-05 18:36:16 -07:00
options_file_test.cc No elide constructors (#7798) 2020-12-23 16:55:53 -08:00
output_validator.cc Use NPHash64 in more places (#7632) 2020-11-10 23:42:13 -08:00
output_validator.h Add remote compaction public API (#8300) 2021-05-19 21:41:31 -07:00
perf_context_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
periodic_work_scheduler_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
periodic_work_scheduler.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
periodic_work_scheduler.h Add a SystemClock class to capture the time functions of an Env (#7858) 2021-01-25 22:09:11 -08:00
pinned_iterators_manager.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
plain_table_db_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
pre_release_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
prefix_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
range_del_aggregator_bench.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
range_del_aggregator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator.cc In ParseInternalKey(), include corrupt key info in Status (#7515) 2020-10-28 10:12:58 -07:00
range_del_aggregator.h Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
range_tombstone_fragmenter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.cc Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
range_tombstone_fragmenter.h Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
read_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
repair_test.cc Some fixes and enhancements to ldb repair (#8544) 2021-07-28 16:44:14 -07:00
repair.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
snapshot_checker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
table_cache.cc Fix use-after-free on implicit temporary FileOptions (#8571) 2021-07-27 21:49:14 -07:00
table_cache.h Fix use-after-free on implicit temporary FileOptions (#8571) 2021-07-27 21:49:14 -07:00
table_properties_collector_test.cc Make it possible to apply only a subrange of table property collectors (#8298) 2021-05-17 18:28:39 -07:00
table_properties_collector.cc Apply sample_for_compression to all block-based tables (#8105) 2021-03-25 15:00:45 -07:00
table_properties_collector.h Partially revert the "apply subrange of table property collectors" change (#8465) 2021-07-06 10:14:32 -07:00
transaction_log_impl.cc Add more tests for assert status checked (#7524) 2020-12-22 23:45:58 -08:00
transaction_log_impl.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
trim_history_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
version_builder_test.cc Print blob file checksums as hex (#8437) 2021-06-22 09:49:44 -07:00
version_builder.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
version_builder.h Handle blob files when options.best_efforts_recovery is true (#8180) 2021-04-19 11:56:14 -07:00
version_edit_handler.cc Fix a bug of secondary instance sequence going backward (#8653) 2021-08-24 18:18:36 -07:00
version_edit_handler.h Fixed manifest_dump issues when printing keys and values containing null characters (#8378) 2021-06-10 12:55:20 -07:00
version_edit_test.cc Make it able to ignore WAL related VersionEdits in older versions (#7873) 2021-01-19 19:27:53 -08:00
version_edit.cc Write file temperature information to manifest (#8284) 2021-05-17 15:15:23 -07:00
version_edit.h Write file temperature information to manifest (#8284) 2021-05-17 15:15:23 -07:00
version_set_test.cc Print blob file checksums as hex (#8437) 2021-06-22 09:49:44 -07:00
version_set.cc Sort per-file blob read requests by offset (#8953) 2021-09-28 13:24:17 -07:00
version_set.h Sort per-file blob read requests by offset (#8953) 2021-09-28 13:24:17 -07:00
wal_edit_test.cc Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_edit.cc Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_edit.h Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_manager_test.cc Use DbSessionId as cache key prefix when secondary cache is enabled (#8360) 2021-06-10 11:02:43 -07:00
wal_manager.cc Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
wal_manager.h Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
write_batch_base.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_internal.h Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
write_batch_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
write_batch.cc Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
write_callback_test.cc Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475) 2021-07-07 11:14:05 -07:00
write_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller_test.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_controller.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_controller.h Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_thread.cc Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898) 2021-04-21 13:54:02 -07:00
write_thread.h typo: fix typo in db/write_thread's state (#8423) 2021-06-18 17:14:51 -07:00