e532877940
Summary: With `table_options.cache_index_and_filter_blocks = true`, index and filter blocks are stored in block cache. Then people are curious how much of the block cache total size is used by indexes and bloom filters. It will be nice we have a way to report that. It can help people tune performance and plan for optimized hardware setting. We add several enum values for db Statistics. BLOCK_CACHE_INDEX/FILTER_BYTES_INSERT - BLOCK_CACHE_INDEX/FILTER_BYTES_ERASE = current INDEX/FILTER total block size in bytes. Test Plan: write a test case called `DBBlockCacheTest.IndexAndFilterBlocksStats`. The result is: ``` [gzh@dev9927.prn1 ~/local/rocksdb] make db_block_cache_test -j64 && ./db_block_cache_test --gtest_filter=DBBlockCacheTest.IndexAndFilterBlocksStats Makefile:101: Warning: Compiling in debug mode. Don't use the resulting binary in production GEN util/build_version.cc make: `db_block_cache_test' is up to date. Note: Google Test filter = DBBlockCacheTest.IndexAndFilterBlocksStats [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from DBBlockCacheTest [ RUN ] DBBlockCacheTest.IndexAndFilterBlocksStats [ OK ] DBBlockCacheTest.IndexAndFilterBlocksStats (689 ms) [----------] 1 test from DBBlockCacheTest (689 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (689 ms total) [ PASSED ] 1 test. ``` Reviewers: IslamAbdelRahman, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58677
97 lines
3.4 KiB
C++
97 lines
3.4 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under the BSD-style license found in the
|
|
// LICENSE file in the root directory of this source tree. An additional grant
|
|
// of patent rights can be found in the PATENTS file in the same directory.
|
|
//
|
|
// Copyright (c) 2012 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
//
|
|
// A filter block is stored near the end of a Table file. It contains
|
|
// filters (e.g., bloom filters) for all data blocks in the table combined
|
|
// into a single filter block.
|
|
//
|
|
// It is a base class for BlockBasedFilter and FullFilter.
|
|
// These two are both used in BlockBasedTable. The first one contain filter
|
|
// For a part of keys in sst file, the second contain filter for all keys
|
|
// in sst file.
|
|
|
|
#pragma once
|
|
|
|
#include <memory>
|
|
#include <stddef.h>
|
|
#include <stdint.h>
|
|
#include <string>
|
|
#include <vector>
|
|
#include "rocksdb/options.h"
|
|
#include "rocksdb/slice.h"
|
|
#include "rocksdb/slice_transform.h"
|
|
#include "rocksdb/table.h"
|
|
#include "util/hash.h"
|
|
#include "format.h"
|
|
|
|
namespace rocksdb {
|
|
|
|
const uint64_t kNotValid = ULLONG_MAX;
|
|
class FilterPolicy;
|
|
|
|
// A FilterBlockBuilder is used to construct all of the filters for a
|
|
// particular Table. It generates a single string which is stored as
|
|
// a special block in the Table.
|
|
//
|
|
// The sequence of calls to FilterBlockBuilder must match the regexp:
|
|
// (StartBlock Add*)* Finish
|
|
//
|
|
// BlockBased/Full FilterBlock would be called in the same way.
|
|
class FilterBlockBuilder {
|
|
public:
|
|
explicit FilterBlockBuilder() {}
|
|
virtual ~FilterBlockBuilder() {}
|
|
|
|
virtual bool IsBlockBased() = 0; // If is blockbased filter
|
|
virtual void StartBlock(uint64_t block_offset) = 0; // Start new block filter
|
|
virtual void Add(const Slice& key) = 0; // Add a key to current filter
|
|
virtual Slice Finish() = 0; // Generate Filter
|
|
|
|
private:
|
|
// No copying allowed
|
|
FilterBlockBuilder(const FilterBlockBuilder&);
|
|
void operator=(const FilterBlockBuilder&);
|
|
};
|
|
|
|
// A FilterBlockReader is used to parse filter from SST table.
|
|
// KeyMayMatch and PrefixMayMatch would trigger filter checking
|
|
//
|
|
// BlockBased/Full FilterBlock would be called in the same way.
|
|
class FilterBlockReader {
|
|
public:
|
|
explicit FilterBlockReader() : size_(0), statistics_(nullptr) {}
|
|
explicit FilterBlockReader(size_t s, Statistics* stats)
|
|
: size_(s), statistics_(stats) {}
|
|
virtual ~FilterBlockReader() {}
|
|
|
|
virtual bool IsBlockBased() = 0; // If is blockbased filter
|
|
virtual bool KeyMayMatch(const Slice& key,
|
|
uint64_t block_offset = kNotValid) = 0;
|
|
virtual bool PrefixMayMatch(const Slice& prefix,
|
|
uint64_t block_offset = kNotValid) = 0;
|
|
virtual size_t ApproximateMemoryUsage() const = 0;
|
|
virtual size_t size() const { return size_; }
|
|
virtual Statistics* statistics() const { return statistics_; }
|
|
|
|
// convert this object to a human readable form
|
|
virtual std::string ToString() const {
|
|
std::string error_msg("Unsupported filter \n");
|
|
return error_msg;
|
|
}
|
|
|
|
private:
|
|
// No copying allowed
|
|
FilterBlockReader(const FilterBlockReader&);
|
|
void operator=(const FilterBlockReader&);
|
|
size_t size_;
|
|
Statistics* statistics_;
|
|
};
|
|
|
|
} // namespace rocksdb
|