andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
a657ee9a9c
rocksdb/db
History
Reid Horuff a657ee9a9c [rocksdb] Recovery path sequence miscount fix 
Summary:
Consider the following WAL with 4 batch entries prefixed with their sequence at time of memtable insert.
[1: BEGIN_PREPARE, PUT, PUT, PUT, PUT, END_PREPARE(a)]
[1: BEGIN_PREPARE, PUT, PUT, PUT, PUT, END_PREPARE(b)]
[4: COMMIT(a)]
[7: COMMIT(b)]

The first two batches do not consume any sequence numbers so are both prefixed with seq=1.
For 2pc commit, memtable insertion takes place before COMMIT batch is written to WAL.
We can see that sequence number consumption takes place between WAL entries giving us the seemingly sparse sequence prefix for WAL entries.
This is a valid WAL.

Because with 2PC markers one WriteBatch points to another batch containing its inserts a writebatch can consume more or less sequence numbers than the number of sequence consuming entries that it contains.

We can see that, given the entries in the WAL, 6 sequence ids were consumed. Yet on recovery the maximum sequence consumed would be 7 + 3 (the number of sequence numbers consumed by COMMIT(b))

So, now upon recovery we must track the actual consumption of sequence numbers.
In the provided scenario there will be no sequence gaps, but it is possible to produce a sequence gap. This should not be a problem though. correct?

Test Plan: provided test.

Reviewers: sdong

Subscribers: andrewkr, leveldb, dhruba, hermanlee4

Differential Revision: https://reviews.facebook.net/D57645
2016-05-10 14:06:07 -07:00
..
auto_roll_logger_test.cc Fix unit tests issues on Windows (#1078) 2016-04-14 17:33:53 -07:00
auto_roll_logger.cc Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
auto_roll_logger.h Redo SyncPoints for flush while rolling test 2016-02-22 21:32:19 -08:00
builder.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
builder.h Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
c_test.c Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
c.cc Fix compression dictionary clang errors 2016-04-27 18:30:04 -07:00
column_family_test.cc Change several option defaults 2016-04-28 17:50:58 -07:00
column_family.cc Fix #1110, 32-bit build failure on Mac OSX (#1112) 2016-05-02 10:04:37 -07:00
column_family.h Fix calling GetCurrentMutableCFOptions in CompactionJob::ProcessKeyValueCompaction() 2016-04-29 17:00:50 -07:00
compact_files_test.cc Fixed CompactFiles() spuriously failing or corrupting DB 2016-02-22 13:54:58 -08:00
compacted_db_impl.cc CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
compacted_db_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
compaction_iterator.cc Fixing snapshot 0 assertion 2016-04-16 01:47:15 -07:00
compaction_iterator.h Fix data race issue when sub-compaction is used in CompactionJob 2016-03-24 19:36:39 -07:00
compaction_job_stats_test.cc Introduce XPRESS compresssion on Windows. (#1081) 2016-04-19 22:54:24 -07:00
compaction_job_test.cc Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_job.cc Fix calling GetCurrentMutableCFOptions in CompactionJob::ProcessKeyValueCompaction() 2016-04-29 17:00:50 -07:00
compaction_job.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
compaction_picker_test.cc Estimate pending compaction bytes more accurately 2016-05-09 15:30:02 -07:00
compaction_picker.cc Add bottommost_compression option 2016-05-09 15:57:19 -07:00
compaction_picker.h Add bottommost_compression option 2016-05-09 15:57:19 -07:00
compaction.cc Add bottommost_compression option 2016-05-09 15:57:19 -07:00
compaction.h Merge pull request #1101 from flyd1005/wip-fix-typo 2016-04-28 02:30:44 -07:00
comparator_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
convenience.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
corruption_test.cc Change several option defaults 2016-04-28 17:50:58 -07:00
cuckoo_table_db_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_block_cache_test.cc Fix db_block_cache_test in lite build 2016-04-18 11:34:11 -07:00
db_bloom_filter_test.cc BlockBasedTable::Get() not to use prefix bloom if read_options.total_order_seek = true 2016-05-06 10:16:11 -07:00
db_compaction_filter_test.cc Change some RocksDB default options 2016-03-31 17:12:18 -07:00
db_compaction_test.cc Change several option defaults 2016-04-28 17:50:58 -07:00
db_dynamic_level_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_filesnapshot.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_debug.cc [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
db_impl_experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl_readonly.cc Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
db_impl_readonly.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_impl.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
db_impl.h [rocksdb] Two Phase Transaction 2016-05-10 14:06:07 -07:00
db_info_dumper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_info_dumper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc Change some RocksDB default options 2016-03-31 17:12:18 -07:00
db_iter_test.cc Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
db_iter.cc Fix Iterator::Prev memory pinning bug 2016-05-03 16:50:01 -07:00
db_iter.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
db_iterator_test.cc Fix win build 2016-05-09 11:52:28 -07:00
db_log_iter_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
db_properties_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
db_sst_test.cc Fix DBTest.RateLimitedDelete flakiness 2016-04-19 14:05:48 -07:00
db_table_properties_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
db_tailing_iter_test.cc Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number" 2016-03-04 18:15:29 -08:00
db_test2.cc Fix lite build 2016-05-09 16:08:30 -07:00
db_test_util.cc Change several option defaults 2016-04-28 17:50:58 -07:00
db_test_util.h Fix win build 2016-05-09 11:52:28 -07:00
db_test.cc Estimate pending compaction bytes more accurately 2016-05-09 15:30:02 -07:00
db_universal_compaction_test.cc Alpine Linux Build (#990) 2016-04-22 16:49:12 -07:00
db_wal_test.cc Introduce XPRESS compresssion on Windows. (#1081) 2016-04-19 22:54:24 -07:00
dbformat_test.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
dbformat.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dbformat.h Modification of WriteBatch to support two phase commit 2016-05-10 14:06:07 -07:00
deletefile_test.cc Handle concurrent manifest update and backup creation 2016-02-29 12:56:55 -08:00
event_helpers.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
event_helpers.h Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
experimental.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
file_indexer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_indexer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filename.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_job_test.cc Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. 2016-04-15 10:22:18 -07:00
flush_job.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
flush_job.h Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. 2016-04-15 10:22:18 -07:00
flush_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
flush_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
forward_iterator.cc Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number" 2016-03-04 18:15:29 -08:00
forward_iterator.h Add Iterator Property rocksdb.iterator.version_number 2016-03-02 16:23:59 -08:00
inlineskiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
inlineskiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
internal_stats.cc Shouldn't report default column family's compaction stats as DB compaction stats 2016-04-25 13:56:59 -07:00
internal_stats.h Shouldn't report default column family's compaction stats as DB compaction stats 2016-04-25 13:56:59 -07:00
job_context.h Cleanup stale manifests outside of full purge 2016-03-10 18:16:21 -08:00
listener_test.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
log_format.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_reader.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
managed_iterator.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
managed_iterator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
manual_compaction_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
memtable_list.cc [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
memtable_list.h [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
memtable.cc [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
memtable.h [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
memtablerep_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_context.h Eliminate std::deque initialization while iterating over merge operands 2016-04-01 15:48:55 -07:00
merge_helper_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_helper.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_operator.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
merge_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_file_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_context_test.cc CompactedDB should not be used if there is outstanding WAL files 2016-04-26 14:22:07 -07:00
pinned_iterators_manager.h Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) 2016-04-26 12:41:07 -07:00
plain_table_db_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
prefix_test.cc Use prefix_same_as_start to avoid iteration in FindNextUserEntryInternal. (#1102) 2016-04-28 16:48:03 -07:00
repair_test.cc Add unit tests for RepairDB 2016-03-18 15:18:42 -07:00
repair.cc Added EventListener::OnTableFileCreationStarted() callback 2016-04-29 11:35:00 -07:00
skiplist_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
skiplist.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
snapshot_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_cache.cc Enable configurable readahead for iterators 2016-05-04 15:25:58 -07:00
table_cache.h Enable configurable readahead for iterators 2016-05-04 15:25:58 -07:00
table_properties_collector_test.cc Embed column family name in SST file 2016-04-06 23:10:32 -07:00
table_properties_collector.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
table_properties_collector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
transaction_log_impl.cc In all the places where log records are read, there was a check that 2016-03-30 23:05:22 +05:30
transaction_log_impl.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_builder.cc Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. 2016-04-01 10:42:39 -07:00
version_builder.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_edit.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
version_set.cc Estimate pending compaction bytes more accurately 2016-05-09 15:30:02 -07:00
version_set.h Add per-level compression ratio property 2016-04-20 18:46:54 -07:00
wal_manager_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
wal_manager.cc In all the places where log records are read, there was a check that 2016-03-30 23:05:22 +05:30
wal_manager.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_base.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_batch_internal.h [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
write_batch_test.cc Modification of WriteBatch to support two phase commit 2016-05-10 14:06:07 -07:00
write_batch.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
write_callback_test.cc write_callback_test: clean test directory before running tests 2016-04-18 10:18:41 -07:00
write_callback.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_controller.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
write_thread.h [rocksdb] Memtable Log Referencing and Prepared Batch Recovery 2016-05-10 14:06:07 -07:00
writebuffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc_test_points.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00