andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fb 5789 max total WAL size clarification (#9108)

Browse Source 
Summary:
Add clarification/extension to comments on max_total_wal_size and the Java wrapper MaxTotalWalSize to better explain the effect of the option on log file sizes.

Closes https://github.com/facebook/rocksdb/issues/5789

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9108

Reviewed By: pdillinger

Differential Revision: D32066640

Pulled By: mrambacher

fbshipit-source-id: 7d5affc87e4119019054af9c884a2ea01d68f5b7
This commit is contained in:
Alan Paxton 2021-11-08 08:53:33 -08:00 committed by Facebook GitHub Bot
parent be351f4754
commit e5b34f5867
2 changed files with 49 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
include/rocksdb/options.h
View File

@ -568,8 +568,18 @@ struct DBOptions {
// (i.e. the ones that are causing all the space amplification). If set to 0 // (i.e. the ones that are causing all the space amplification). If set to 0
// (default), we will dynamically choose the WAL size limit to be // (default), we will dynamically choose the WAL size limit to be
// [sum of all write_buffer_size * max_write_buffer_number] * 4 // [sum of all write_buffer_size * max_write_buffer_number] * 4
// This option takes effect only when there are more than one column family as //
// otherwise the wal size is dictated by the write_buffer_size. // For example, with 15 column families, each with
// write_buffer_size = 128 MB
// max_write_buffer_number = 6
// max_total_wal_size will be calculated to be [15 * 128MB * 6] * 4 = 45GB
//
// The RocksDB wiki has some discussion about how the WAL interacts
// with memtables and flushing of column families.
// https://github.com/facebook/rocksdb/wiki/Column-Families
//
// This option takes effect only when there are more than one column
// family as otherwise the wal size is dictated by the write_buffer_size.
// //
// Default: 0 // Default: 0
// //

45
java/src/main/java/org/rocksdb/MutableDBOptionsInterface.java
View File

@ -202,12 +202,24 @@ public interface MutableDBOptionsInterface<T extends MutableDBOptionsInterface<T
long delayedWriteRate(); long delayedWriteRate();
/** /**
* <p>Once write-ahead logs exceed this size, we will start forcing the * <p>Set the max total write-ahead log size. Once write-ahead logs exceed this size, we will
* flush of column families whose memtables are backed by the oldest live * start forcing the flush of column families whose memtables are backed by the oldest live WAL
* WAL file (i.e. the ones that are causing all the space amplification). * file
* </p>
* <p>The oldest WAL files are the ones that are causing all the space amplification.
* </p>
* For example, with 15 column families, each with
* <code>write_buffer_size = 128 MB</code>
* <code>max_write_buffer_number = 6</code>
* <code>max_total_wal_size</code> will be calculated to be <code>[15 * 128MB * 6] * 4 =
* 45GB</code>
* <p>
* The RocksDB wiki has some discussion about how the WAL interacts
* with memtables and flushing of column families, at
* <a href="https://github.com/facebook/rocksdb/wiki/Column-Families">...</a>
* </p> * </p>
* <p>If set to 0 (default), we will dynamically choose the WAL size limit to * <p>If set to 0 (default), we will dynamically choose the WAL size limit to
* be [sum of all write_buffer_size * max_write_buffer_number] * 2</p> * be [sum of all write_buffer_size * max_write_buffer_number] * 4</p>
* <p>This option takes effect only when there are more than one column family as * <p>This option takes effect only when there are more than one column family as
* otherwise the wal size is dictated by the write_buffer_size.</p> * otherwise the wal size is dictated by the write_buffer_size.</p>
* <p>Default: 0</p> * <p>Default: 0</p>
@ -218,13 +230,30 @@ public interface MutableDBOptionsInterface<T extends MutableDBOptionsInterface<T
T setMaxTotalWalSize(long maxTotalWalSize); T setMaxTotalWalSize(long maxTotalWalSize);
/** /**
* <p>Returns the max total wal size. Once write-ahead logs exceed this size, * <p>Returns the max total write-ahead log size. Once write-ahead logs exceed this size,
* we will start forcing the flush of column families whose memtables are * we will start forcing the flush of column families whose memtables are
* backed by the oldest live WAL file (i.e. the ones that are causing all * backed by the oldest live WAL file.</p>
* the space amplification).</p> * <p>The oldest WAL files are the ones that are causing all the space amplification.
* </p>
* For example, with 15 column families, each with
* <code>write_buffer_size = 128 MB</code>
* <code>max_write_buffer_number = 6</code>
* <code>max_total_wal_size</code> will be calculated to be <code>[15 * 128MB * 6] * 4 =
* 45GB</code>
* <p>
* The RocksDB wiki has some discussion about how the WAL interacts
* with memtables and flushing of column families, at
* <a href="https://github.com/facebook/rocksdb/wiki/Column-Families">...</a>
* </p>
* <p>If set to 0 (default), we will dynamically choose the WAL size limit to
* be [sum of all write_buffer_size * max_write_buffer_number] * 4</p>
* <p>This option takes effect only when there are more than one column family as
* otherwise the wal size is dictated by the write_buffer_size.</p>
* <p>Default: 0</p>
*
* *
* <p>If set to 0 (default), we will dynamically choose the WAL size limit * <p>If set to 0 (default), we will dynamically choose the WAL size limit
* to be [sum of all write_buffer_size * max_write_buffer_number] * 2 * to be [sum of all write_buffer_size * max_write_buffer_number] * 4
* </p> * </p>
* *
* @return max total wal size * @return max total wal size