rocksdb/utilities
Zitan Chen 373d5ac485 BackupEngine verifies table file checksums on creating new backups (#7015)
Summary:
When table file checksums are enabled and stored in the DB manifest by using the RocksDB default crc32c checksum function, BackupEngine will calculate the crc32c checksum of the file to be copied and compare the calculated result with the one stored in the DB manifest before copying the file to the backup directory.

After copying to the backup directory, BackupEngine will verify the checksum of the copied file with the one calculated before copying. This helps detect some rare corruption events such as bit-flips during the copying process.

No verification with checksums in DB manifest will be performed if the table file checksum function is not the RocksDB default crc32c checksum function.

In addition, If `share_table_files` and `share_files_with_checksum` are true, BackupEngine will compare the checksums computed before and after copying of the table files.

Corresponding tests are added.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7015

Test Plan: Passed make check

Reviewed By: pdillinger

Differential Revision: D22165732

Pulled By: gg814

fbshipit-source-id: ee0e8cc397c455eba64545c29380b9d9853588ec
2020-07-02 18:15:12 -07:00
..
backupable BackupEngine verifies table file checksums on creating new backups (#7015) 2020-07-02 18:15:12 -07:00
blob_db Compaction filter support for BlobDB (#6850) 2020-06-29 17:32:14 -07:00
cassandra Remove gtest dependency in non-test code under utilities/cassandra (#6908) 2020-06-02 13:56:29 -07:00
checkpoint BackupEngine verifies table file checksums on creating new backups (#7015) 2020-07-02 18:15:12 -07:00
compaction_filters Compaction filter support for BlobDB (#6850) 2020-06-29 17:32:14 -07:00
convenience Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
leveldb_options Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
memory Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operators Make StringAppendOperatorTest a parameterized test (#6930) 2020-06-04 14:17:11 -07:00
option_change_migration Revert "Whole DBTest to skip fsync (#7049)" (#7070) 2020-07-02 10:22:43 -07:00
options Extend Get/MultiGet deadline support to table open (#6982) 2020-06-29 14:53:17 -07:00
persistent_cache Revert "Whole DBTest to skip fsync (#7049)" (#7070) 2020-07-02 10:22:43 -07:00
simulator_cache Revert "Whole DBTest to skip fsync (#7049)" (#7070) 2020-07-02 10:22:43 -07:00
table_properties_collectors Trigger compaction in CompactOnDeletionCollector based on deletion ratio (#6806) 2020-05-18 08:42:05 -07:00
trace Pass a timeout to FileSystem for random reads (#6751) 2020-04-30 14:50:39 -07:00
transactions Fix some defects reported by Coverity Scan (#6933) 2020-06-04 15:46:27 -07:00
ttl Compaction filter support for BlobDB (#6850) 2020-06-29 17:32:14 -07:00
write_batch_with_index Fix FilterBench when RTTI=0 (#6732) 2020-04-29 13:09:23 -07:00
debug.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_librados_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_librados.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_librados.md Add EnvLibrados - RocksDB Env of RADOS (#1222) 2016-07-21 11:16:34 -07:00
env_mirror_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_mirror.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_timed_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env_timed.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operators.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
object_registry_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
object_registry.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
util_merge_operators_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00