andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
096beb787e
rocksdb/table/block_based
History
Peter Dillinger 25a0d0ca30 Fix block checksum for >=4GB, refactor (#6978) 
Summary:
Although RocksDB falls over in various other ways with KVs
around 4GB or more, this change fixes how XXH32 and XXH64 were being
called by the block checksum code to support >= 4GB in case that should
ever happen, or the code copied for other uses.

This change is not a schema compatibility issue because the checksum
verification code would checksum the first (block_size + 1) mod 2^32
bytes while the checksum construction code would checksum the first
block_size mod 2^32 plus the compression type byte, meaning the
XXH32/64 checksums for >=4GB block would not match about 255/256 times.

While touching this code, I refactored to consolidate redundant
implementations, improving diagnostics and performance tracking in some
cases. Also used less confusing language in those diagnostics.

Makes https://github.com/facebook/rocksdb/issues/6875 obsolete.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6978

Test Plan:
I was able to write a test for this using an SST file writer
and VerifyChecksum in a reader. The test fails before the fix, though
I'm leaving the test disabled because I don't think it's worth the
expense of running regularly.

Reviewed By: gg814

Differential Revision: D22143260

Pulled By: pdillinger

fbshipit-source-id: 982993d16134e8c50bea2269047f901c1783726e
2020-06-19 16:18:24 -07:00
..
binary_search_index_reader.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
binary_search_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
block_based_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_based_filter_block.cc Reduce dependency on gtest dependency in release code (#6907) 2020-06-02 12:11:24 -07:00
block_based_filter_block.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_based_table_builder.cc Fix block checksum for >=4GB, refactor (#6978) 2020-06-19 16:18:24 -07:00
block_based_table_builder.h Store DB identity and DB session ID in SST files (#6983) 2020-06-17 10:57:40 -07:00
block_based_table_factory.cc Store DB identity and DB session ID in SST files (#6983) 2020-06-17 10:57:40 -07:00
block_based_table_factory.h sst_dump to reduce number of file reads (#6836) 2020-05-12 18:23:33 -07:00
block_based_table_iterator.cc Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
block_based_table_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
block_based_table_reader_impl.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
block_based_table_reader_test.cc Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) 2020-06-05 11:08:25 -07:00
block_based_table_reader.cc Fix block checksum for >=4GB, refactor (#6978) 2020-06-19 16:18:24 -07:00
block_based_table_reader.h Fix persistent cache on windows (#6932) 2020-06-13 13:28:31 -07:00
block_builder.cc Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_builder.h Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
block_prefetcher.cc De-template block based table iterator (#6531) 2020-03-16 12:20:50 -07:00
block_prefetcher.h De-template block based table iterator (#6531) 2020-03-16 12:20:50 -07:00
block_prefix_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_prefix_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
block_type.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
block.cc save a key comparison in block seeks (#6646) 2020-06-10 13:58:39 -07:00
block.h save a key comparison in block seeks (#6646) 2020-06-10 13:58:39 -07:00
cachable_entry.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_footer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index_test.cc Fix range deletion tombstone ingestion with global seqno (#6429) 2020-02-25 15:31:48 -08:00
data_block_hash_index.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
data_block_hash_index.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block_reader_common.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_block.h Fix iterator reading filter block despite read_tier == kBlockCacheTier (#6562) 2020-03-26 15:21:26 -07:00
filter_policy_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filter_policy.cc Add Functions to OptionTypeInfo (#6422) 2020-04-28 18:04:26 -07:00
flush_block_policy.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_block_policy.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
full_filter_block.cc Reduce dependency on gtest dependency in release code (#6907) 2020-06-02 12:11:24 -07:00
full_filter_block.h Fix iterator reading filter block despite read_tier == kBlockCacheTier (#6562) 2020-03-26 15:21:26 -07:00
hash_index_reader.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
hash_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
index_builder.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
index_builder.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
index_reader_common.cc Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
index_reader_common.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
mock_block_based_table.h For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) 2020-06-02 12:30:23 -07:00
parsed_full_filter_block.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
parsed_full_filter_block.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
partitioned_filter_block_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
partitioned_filter_block.cc Fix handling of too-small filter partition size (#6905) 2020-06-03 10:43:01 -07:00
partitioned_filter_block.h Basic MultiGet support for partitioned filters (#6757) 2020-04-28 14:49:34 -07:00
partitioned_index_iterator.cc Fix regression bug in partitioned index reseek caused by #6531 (#6551) 2020-03-17 12:33:10 -07:00
partitioned_index_iterator.h Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) 2020-06-05 11:08:25 -07:00
partitioned_index_reader.cc Reduce dependency on gtest dependency in release code (#6907) 2020-06-02 12:11:24 -07:00
partitioned_index_reader.h Divide block_based_table_reader.cc (#6527) 2020-03-12 21:41:50 -07:00
reader_common.cc Fix block checksum for >=4GB, refactor (#6978) 2020-06-19 16:18:24 -07:00
reader_common.h Fix block checksum for >=4GB, refactor (#6978) 2020-06-19 16:18:24 -07:00
uncompression_dict_reader.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
uncompression_dict_reader.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00