efd013d6d8
Summary: I was investigating performance issues in the SstFileWriter and found all of the following: - The SstFileWriter::Add() function created a local InternalKey every time it was called generating a allocation and free each time. Changed to have an InternalKey member variable that can be reset with the new InternalKey::Set() function. - In SstFileWriter::Add() the smallest_key and largest_key values were assigned the result of a ToString() call, but it is simpler to just assign them directly from the user's key. - The Slice class had no move constructor so each time one was returned from a function a new one had to be allocated, the old data copied to the new, and the old one was freed. I added the move constructor which also required a copy constructor and assignment operator. - The BlockBuilder::CurrentSizeEstimate() function calculates the current estimate size, but was being called 2 or 3 times for each key added. I changed the class to maintain a running estimate (equal to the original calculation) so that the function can return an already calculated value. - The code in BlockBuilder::Add() that calculated the shared bytes between the last key and the new key duplicated what Slice::difference_offset does, so I replaced it with the standard function. - BlockBuilder::Add() had code to copy just the changed portion into the last key value (and asserted that it now matched the new key). It is more efficient just to copy the whole new key over. - Moved this same code up into the 'if (use_delta_encoding_)' since the last key value is only needed when delta encoding is on. - FlushBlockBySizePolicy::BlockAlmostFull calculated a standard deviation value each time it was called, but this information would only change if block_size of block_size_deviation changed, so I created a member variable to hold the value to avoid the calculation each time. - Each PutVarint??() function has a buffer and calls std::string::append(). Two or three calls in a row could share a buffer and a single call to std::string::append(). Some of these will be helpful outside of the SstFileWriter. I'm not 100% the addition of the move constructor is appropriate as I wonder why this wasn't done before - maybe because of compiler compatibility? I tried it on gcc 4.8 and 4.9. Test Plan: The changes should not affect the results so the existing tests should all still work and no new tests were added. The value of the changes was seen by manually testing the SstFileWriter class through MyRocks and adding timing code to identify problem areas. Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59607
75 lines
2.7 KiB
C++
75 lines
2.7 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under the BSD-style license found in the
|
|
// LICENSE file in the root directory of this source tree. An additional grant
|
|
// of patent rights can be found in the PATENTS file in the same directory.
|
|
|
|
#include "rocksdb/options.h"
|
|
#include "rocksdb/flush_block_policy.h"
|
|
#include "rocksdb/slice.h"
|
|
#include "table/block_builder.h"
|
|
|
|
#include <cassert>
|
|
|
|
namespace rocksdb {
|
|
|
|
// Flush block by size
|
|
class FlushBlockBySizePolicy : public FlushBlockPolicy {
|
|
public:
|
|
// @params block_size: Approximate size of user data packed per
|
|
// block.
|
|
// @params block_size_deviation: This is used to close a block before it
|
|
// reaches the configured
|
|
FlushBlockBySizePolicy(const uint64_t block_size,
|
|
const uint64_t block_size_deviation,
|
|
const BlockBuilder& data_block_builder)
|
|
: block_size_(block_size),
|
|
block_size_deviation_limit_(
|
|
((block_size * (100 - block_size_deviation)) + 99) / 100),
|
|
data_block_builder_(data_block_builder) {}
|
|
|
|
virtual bool Update(const Slice& key,
|
|
const Slice& value) override {
|
|
// it makes no sense to flush when the data block is empty
|
|
if (data_block_builder_.empty()) {
|
|
return false;
|
|
}
|
|
|
|
auto curr_size = data_block_builder_.CurrentSizeEstimate();
|
|
|
|
// Do flush if one of the below two conditions is true:
|
|
// 1) if the current estimated size already exceeds the block size,
|
|
// 2) block_size_deviation is set and the estimated size after appending
|
|
// the kv will exceed the block size and the current size is under the
|
|
// the deviation.
|
|
return curr_size >= block_size_ || BlockAlmostFull(key, value);
|
|
}
|
|
|
|
private:
|
|
bool BlockAlmostFull(const Slice& key, const Slice& value) const {
|
|
if (block_size_deviation_limit_ == 0) {
|
|
return false;
|
|
}
|
|
|
|
const auto curr_size = data_block_builder_.CurrentSizeEstimate();
|
|
const auto estimated_size_after =
|
|
data_block_builder_.EstimateSizeAfterKV(key, value);
|
|
|
|
return estimated_size_after > block_size_ &&
|
|
curr_size > block_size_deviation_limit_;
|
|
}
|
|
|
|
const uint64_t block_size_;
|
|
const uint64_t block_size_deviation_limit_;
|
|
const BlockBuilder& data_block_builder_;
|
|
};
|
|
|
|
FlushBlockPolicy* FlushBlockBySizePolicyFactory::NewFlushBlockPolicy(
|
|
const BlockBasedTableOptions& table_options,
|
|
const BlockBuilder& data_block_builder) const {
|
|
return new FlushBlockBySizePolicy(
|
|
table_options.block_size, table_options.block_size_deviation,
|
|
data_block_builder);
|
|
}
|
|
|
|
} // namespace rocksdb
|