rocksdb/db
anand76 94265234de Fix race due to delete triggered compaction in Universal compaction mode (#6799)
Summary:
Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions.
1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys.
2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run.

Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799

Reviewed By: pdillinger

Differential Revision: D21431286

Pulled By: anand1976

fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a
2020-05-07 17:32:17 -07:00
..
blob Make sure (Shared)BlobFileMetaData are owned by shared_ptrs (#6749) 2020-04-23 13:44:29 -07:00
compaction Fix race due to delete triggered compaction in Universal compaction mode (#6799) 2020-05-07 17:32:17 -07:00
db_impl Find/purge obsolete blob files (#6807) 2020-05-07 09:32:51 -07:00
arena_wrapped_db_iter.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
arena_wrapped_db_iter.h Iterator with timestamp (#6255) 2020-03-06 16:24:27 -08:00
builder.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
builder.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
c_test.c Return NotSupported from WriteBatchWithIndex::DeleteRange (#5393) 2020-02-18 11:18:25 -08:00
c.cc added new functions to c-api (#5630) 2020-04-07 14:45:39 -07:00
column_family_test.cc Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) 2020-04-21 17:38:17 -07:00
column_family.cc Attempt to recover from db with missing table files (#6334) 2020-03-20 19:30:48 -07:00
column_family.h Attempt to recover from db with missing table files (#6334) 2020-03-20 19:30:48 -07:00
compact_files_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
compacted_db_impl.cc return timestamp from get (#6409) 2020-03-02 16:01:00 -08:00
compacted_db_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
comparator_db_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
convenience.cc Fix FilterBench when RTTI=0 (#6732) 2020-04-29 13:09:23 -07:00
corruption_test.cc Simplify migration to FileSystem API (#6552) 2020-03-23 21:54:21 -07:00
cuckoo_table_db_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_basic_test.cc Fix false NotFound from batched MultiGet with kHashSearch (#6821) 2020-05-07 15:41:37 -07:00
db_block_cache_test.cc Stats for redundant insertions into block cache (#6681) 2020-04-27 13:20:27 -07:00
db_bloom_filter_test.cc Fix LITE build (#6770) 2020-04-28 21:37:20 -07:00
db_compaction_filter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_compaction_test.cc Slightly expand converage to file consistency check failure (#6800) 2020-05-05 18:31:53 -07:00
db_dynamic_level_test.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
db_encryption_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_filesnapshot.cc Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) 2020-05-04 17:45:49 -07:00
db_flush_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_info_dumper.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_info_dumper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_inplace_update_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_io_failure_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_iter_stress_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_iter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_iter.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
db_iter.h make iterator return versions between timestamp bounds (#6544) 2020-04-10 09:51:58 -07:00
db_iterator_test.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
db_log_iter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_logical_block_size_cache_test.cc Get block size only in direct IO mode (#6522) 2020-03-20 15:26:10 -07:00
db_memtable_test.cc return timestamp from get (#6409) 2020-03-02 16:01:00 -08:00
db_merge_operand_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_merge_operator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_options_test.cc Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) 2020-04-30 17:01:17 -07:00
db_properties_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_range_del_test.cc validate range tombstone covers positive range (#6788) 2020-05-07 11:55:30 -07:00
db_sst_test.cc Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) 2020-05-05 13:11:29 -07:00
db_statistics_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_table_properties_test.cc Report kFilesMarkedForCompaction for delete triggered compactions (#6680) 2020-04-10 15:30:38 -07:00
db_tailing_iter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_test2.cc Avoid Swallowing Some File Consistency Checking Bugs (#6793) 2020-05-04 14:18:11 -07:00
db_test_util.cc Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) 2020-05-05 13:11:29 -07:00
db_test_util.h Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) 2020-05-05 13:11:29 -07:00
db_test.cc Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
db_universal_compaction_test.cc Fix race due to delete triggered compaction in Universal compaction mode (#6799) 2020-05-07 17:32:17 -07:00
db_wal_test.cc Add OptionTypeInfo::Enum and related methods (#6423) 2020-05-05 15:04:04 -07:00
db_with_timestamp_basic_test.cc Fix timestamp support for MultiGet (#6748) 2020-04-27 22:49:56 -07:00
db_with_timestamp_compaction_test.cc Compaction with timestamp: input boundaries (#6645) 2020-04-10 16:05:49 -07:00
db_write_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
dbformat_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
dbformat.cc Iterator with timestamp (#6255) 2020-03-06 16:24:27 -08:00
dbformat.h Iterator with timestamp (#6255) 2020-03-06 16:24:27 -08:00
deletefile_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
error_handler_fs_test.cc Revamp cache_bench to resemble a real workload (#6629) 2020-04-03 10:26:49 -07:00
error_handler.cc Be able to decrease background thread's CPU priority when creating database backup (#6602) 2020-03-28 19:07:25 -07:00
error_handler.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
event_helpers.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
event_helpers.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
experimental.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_basic_test.cc Fix range deletion tombstone ingestion with global seqno (#6429) 2020-02-25 15:31:48 -08:00
external_sst_file_ingestion_job.cc Replace Directory with FSDirectory in DB (#6468) 2020-03-02 16:16:26 -08:00
external_sst_file_ingestion_job.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_test.cc Fix wrong key being read on ingested file with global seqno and delta encoding (#6669) 2020-04-08 21:22:15 -07:00
fault_injection_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filename_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_job_test.cc Fix initializer syntax for old Xcode compiler (#6662) 2020-04-07 16:00:26 -07:00
flush_job.cc Make options.bottommost_compression, compression_opts and bottommost_compression_opts dynamically changeable. (#6615) 2020-03-31 12:11:42 -07:00
flush_job.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
flush_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
forward_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
import_column_family_job.cc Fix potential size_t overflow in import_column_family (#6762) 2020-04-30 08:40:42 -07:00
import_column_family_job.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
import_column_family_test.cc Fix CF import with overlapping SST files (#6663) 2020-04-16 13:16:06 -07:00
internal_stats.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
internal_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
job_context.h Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
listener_test.cc Move BlobDB related files under db/ to db/blob/ (#6519) 2020-03-12 11:00:56 -07:00
log_format.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_reader.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
log_reader.h Fix tabs and lint-ignores (#6734) 2020-04-20 11:39:31 -07:00
log_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_writer.cc Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
log_writer.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
logs_with_prep_tracker.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
logs_with_prep_tracker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
lookup_key.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
manual_compaction_test.cc Skip high levels with no key falling in the range in CompactRange (#6482) 2020-03-04 20:15:25 -08:00
memtable_list_test.cc Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
memtable_list.cc Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
memtable_list.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
memtable.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
memtable.h return timestamp from get (#6409) 2020-03-02 16:01:00 -08:00
merge_context.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
obsolete_files_test.cc Find/purge obsolete blob files (#6807) 2020-05-07 09:32:51 -07:00
options_file_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
perf_context_test.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
pinned_iterators_manager.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
plain_table_db_test.cc Add ppc64le builds to Travis (#6144) 2020-03-11 12:33:45 -07:00
pre_release_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
prefix_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
read_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
repair_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
repair.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
snapshot_checker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_cache.cc Fix FilterBench when RTTI=0 (#6732) 2020-04-29 13:09:23 -07:00
table_cache.h Prevents Table Cache to open same files more times (#6707) 2020-04-21 13:16:31 -07:00
table_properties_collector_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_properties_collector.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_properties_collector.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
transaction_log_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
transaction_log_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
version_builder_test.cc fix swallowed error for file deletion consistency check (#6809) 2020-05-05 14:54:21 -07:00
version_builder.cc Slightly expand converage to file consistency check failure (#6800) 2020-05-05 18:31:53 -07:00
version_builder.h Keep track of obsolete blob files in VersionSet (#6755) 2020-04-30 11:25:51 -07:00
version_edit_handler.cc Do not swallow error returned from SaveTo() (#6801) 2020-05-05 10:46:20 -07:00
version_edit_handler.h Attempt to recover from db with missing table files (#6334) 2020-03-20 19:30:48 -07:00
version_edit_test.cc Revert "Added the safe-to-ignore tag to version_edit (#6530)" (#6569) 2020-03-23 10:27:47 -07:00
version_edit.cc Revert "Added the safe-to-ignore tag to version_edit (#6530)" (#6569) 2020-03-23 10:27:47 -07:00
version_edit.h Revert "Added the safe-to-ignore tag to version_edit (#6530)" (#6569) 2020-03-23 10:27:47 -07:00
version_set_test.cc Do not swallow error returned from SaveTo() (#6801) 2020-05-05 10:46:20 -07:00
version_set.cc Fix race due to delete triggered compaction in Universal compaction mode (#6799) 2020-05-07 17:32:17 -07:00
version_set.h Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
wal_manager_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
wal_manager.cc Fix FilterBench when RTTI=0 (#6732) 2020-04-29 13:09:23 -07:00
wal_manager.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_base.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch.cc validate range tombstone covers positive range (#6788) 2020-05-07 11:55:30 -07:00
write_callback_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_thread.cc fix some spelling typos (#6464) 2020-02-28 14:14:03 -08:00
write_thread.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00