321dfdc3ae
Summary: The leveldb API is enhanced to support different compression algorithms at different levels. This adds the option min_level_to_compress to db_bench that specifies the minimum level for which compression should be done when compression is enabled. This can be used to disable compression for levels 0 and 1 which are likely to suffer from stalls because of the CPU load for memtable flushes and (L0,L1) compaction. Level 0 is special as it gets frequent memtable flushes. Level 1 is special as it frequently gets all:all file compactions between it and level 0. But all other levels could be the same. For any level N where N > 1, the rate of sequential IO for that level should be the same. The last level is the exception because it might not be full and because files from it are not read to compact with the next larger level. The same amount of time will be spent doing compaction at any level N excluding N=0, 1 or the last level. By this standard all of those levels should use the same compression. The difference is that the loss (using more disk space) from a faster compression algorithm is less significant for N=2 than for N=3. So we might be willing to trade disk space for faster write rates with no compression for L0 and L1, snappy for L2, zlib for L3. Using a faster compression algorithm for the mid levels also allows us to reclaim some cpu without trading off much loss in disk space overhead. Also note that little is to be gained by compressing levels 0 and 1. For a 4-level tree they account for 10% of the data. For a 5-level tree they account for 1% of the data. With compression enabled: * memtable flush rate is ~18MB/second * (L0,L1) compaction rate is ~30MB/second With compression enabled but min_level_to_compress=2 * memtable flush rate is ~320MB/second * (L0,L1) compaction rate is ~560MB/second This practicaly takes the same code from https://reviews.facebook.net/D6225 but makes the leveldb api more general purpose with a few additional lines of code. Test Plan: make check Differential Revision: https://reviews.facebook.net/D6261
96 lines
3.5 KiB
C++
96 lines
3.5 KiB
C++
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
//
|
|
// TableBuilder provides the interface used to build a Table
|
|
// (an immutable and sorted map from keys to values).
|
|
//
|
|
// Multiple threads can invoke const methods on a TableBuilder without
|
|
// external synchronization, but if any of the threads may call a
|
|
// non-const method, all threads accessing the same TableBuilder must use
|
|
// external synchronization.
|
|
|
|
#ifndef STORAGE_LEVELDB_INCLUDE_TABLE_BUILDER_H_
|
|
#define STORAGE_LEVELDB_INCLUDE_TABLE_BUILDER_H_
|
|
|
|
#include <stdint.h>
|
|
#include "leveldb/options.h"
|
|
#include "leveldb/status.h"
|
|
|
|
namespace leveldb {
|
|
|
|
class BlockBuilder;
|
|
class BlockHandle;
|
|
class WritableFile;
|
|
|
|
class TableBuilder {
|
|
public:
|
|
// Create a builder that will store the contents of the table it is
|
|
// building in *file. Does not close the file. It is up to the
|
|
// caller to close the file after calling Finish(). The output file
|
|
// will be part of level specified by 'level'. A value of -1 means
|
|
// that the caller does not know which level the output file will reside.
|
|
TableBuilder(const Options& options, WritableFile* file, int level=-1);
|
|
|
|
// REQUIRES: Either Finish() or Abandon() has been called.
|
|
~TableBuilder();
|
|
|
|
// Change the options used by this builder. Note: only some of the
|
|
// option fields can be changed after construction. If a field is
|
|
// not allowed to change dynamically and its value in the structure
|
|
// passed to the constructor is different from its value in the
|
|
// structure passed to this method, this method will return an error
|
|
// without changing any fields.
|
|
Status ChangeOptions(const Options& options);
|
|
|
|
// Add key,value to the table being constructed.
|
|
// REQUIRES: key is after any previously added key according to comparator.
|
|
// REQUIRES: Finish(), Abandon() have not been called
|
|
void Add(const Slice& key, const Slice& value);
|
|
|
|
// Advanced operation: flush any buffered key/value pairs to file.
|
|
// Can be used to ensure that two adjacent entries never live in
|
|
// the same data block. Most clients should not need to use this method.
|
|
// REQUIRES: Finish(), Abandon() have not been called
|
|
void Flush();
|
|
|
|
// Return non-ok iff some error has been detected.
|
|
Status status() const;
|
|
|
|
// Finish building the table. Stops using the file passed to the
|
|
// constructor after this function returns.
|
|
// REQUIRES: Finish(), Abandon() have not been called
|
|
Status Finish();
|
|
|
|
// Indicate that the contents of this builder should be abandoned. Stops
|
|
// using the file passed to the constructor after this function returns.
|
|
// If the caller is not going to call Finish(), it must call Abandon()
|
|
// before destroying this builder.
|
|
// REQUIRES: Finish(), Abandon() have not been called
|
|
void Abandon();
|
|
|
|
// Number of calls to Add() so far.
|
|
uint64_t NumEntries() const;
|
|
|
|
// Size of the file generated so far. If invoked after a successful
|
|
// Finish() call, returns the size of the final generated file.
|
|
uint64_t FileSize() const;
|
|
|
|
private:
|
|
bool ok() const { return status().ok(); }
|
|
void WriteBlock(BlockBuilder* block, BlockHandle* handle);
|
|
void WriteRawBlock(const Slice& data, CompressionType, BlockHandle* handle);
|
|
|
|
struct Rep;
|
|
Rep* rep_;
|
|
int level_;
|
|
|
|
// No copying allowed
|
|
TableBuilder(const TableBuilder&);
|
|
void operator=(const TableBuilder&);
|
|
};
|
|
|
|
} // namespace leveldb
|
|
|
|
#endif // STORAGE_LEVELDB_INCLUDE_TABLE_BUILDER_H_
|