andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
abb9b95ffe
rocksdb/util/options_test.cc

637 lines
26 KiB
C++
Raw Normal View History

Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
// Copyright (c) 2013, Facebook, Inc. All rights reserved.
// This source code is licensed under the BSD-style license found in the
// LICENSE file in the root directory of this source tree. An additional grant
// of patent rights can be found in the PATENTS file in the same directory.
//
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
fix more compile warnings N/A Change-Id: I5b6f9c70aea7d3f3489328834fed323d41106d9f Signed-off-by: liuhuahang <liuhuahang@zerus.co>
2014-09-05 08:14:37 +02:00
#ifndef __STDC_FORMAT_MACROS
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
#define __STDC_FORMAT_MACROS
fix more compile warnings N/A Change-Id: I5b6f9c70aea7d3f3489328834fed323d41106d9f Signed-off-by: liuhuahang <liuhuahang@zerus.co>
2014-09-05 08:14:37 +02:00
#endif
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
#include <unordered_map>
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
#include <inttypes.h>
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
#include "rocksdb/cache.h"
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
#include "rocksdb/options.h"
Added CompatibleOptions for compatibility with LevelDB Options Summary: Created a CompatibleOptions object that can be used as a LevelDB Options object and then converted to a RocksDB Options object using the ConvertOptions() method. Test Plan: Unit test included in diff. Reviewers: ljin Reviewed By: ljin Subscribers: sdong, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28893
2014-11-21 04:24:39 +01:00
#include "rocksdb/table.h"
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
#include "rocksdb/utilities/convenience.h"
#include "rocksdb/utilities/leveldb_options.h"
Added CompatibleOptions for compatibility with LevelDB Options Summary: Created a CompatibleOptions object that can be used as a LevelDB Options object and then converted to a RocksDB Options object using the ConvertOptions() method. Test Plan: Unit test included in diff. Reviewers: ljin Reviewed By: ljin Subscribers: sdong, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28893
2014-11-21 04:24:39 +01:00
#include "table/block_based_table_factory.h"
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
#include "util/testharness.h"
Make RocksDB compile without gflags Summary: We want to make sure people without gflags can compile RocksDB. Test Plan: remove gflags, make all Reviewers: sdong, rven, yhchiang, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 21:53:23 +01:00
#ifndef GFLAGS
bool FLAGS_enable_print = false;
#else
#include <gflags/gflags.h>
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
using GFLAGS::ParseCommandLineFlags;
DEFINE_bool(enable_print, false, "Print options generated to console.");
Make RocksDB compile without gflags Summary: We want to make sure people without gflags can compile RocksDB. Test Plan: remove gflags, make all Reviewers: sdong, rven, yhchiang, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 21:53:23 +01:00
#endif // GFLAGS
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
namespace rocksdb {
class OptionsTest {};
class StderrLogger : public Logger {
public:
virtual void Logv(const char* format, va_list ap) override {
vprintf(format, ap);
printf("\n");
}
};
Options PrintAndGetOptions(size_t total_write_buffer_limit,
int read_amplification_threshold,
int write_amplification_threshold,
uint64_t target_db_size = 68719476736) {
StderrLogger logger;
if (FLAGS_enable_print) {
printf(
"---- total_write_buffer_limit: %zu "
"read_amplification_threshold: %d write_amplification_threshold: %d "
"target_db_size %" PRIu64 " ----\n",
total_write_buffer_limit, read_amplification_threshold,
write_amplification_threshold, target_db_size);
}
Options options =
GetOptions(total_write_buffer_limit, read_amplification_threshold,
write_amplification_threshold, target_db_size);
if (FLAGS_enable_print) {
options.Dump(&logger);
printf("-------------------------------------\n\n\n");
}
return options;
}
TEST(OptionsTest, LooseCondition) {
Options options;
PrintAndGetOptions(static_cast<size_t>(10) * 1024 * 1024 * 1024, 100, 100);
// Less mem table memory budget
PrintAndGetOptions(32 * 1024 * 1024, 100, 100);
// Tight read amplification
options = PrintAndGetOptions(128 * 1024 * 1024, 8, 100);
ASSERT_EQ(options.compaction_style, kCompactionStyleLevel);
// Tight write amplification
options = PrintAndGetOptions(128 * 1024 * 1024, 64, 10);
ASSERT_EQ(options.compaction_style, kCompactionStyleUniversal);
// Both tight amplifications
PrintAndGetOptions(128 * 1024 * 1024, 4, 8);
}
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
TEST(OptionsTest, GetOptionsFromMapTest) {
std::unordered_map<std::string, std::string> cf_options_map = {
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
{"write_buffer_size", "1"},
{"max_write_buffer_number", "2"},
{"min_write_buffer_number_to_merge", "3"},
{"compression", "kSnappyCompression"},
{"compression_per_level", "kNoCompression:"
"kSnappyCompression:"
"kZlibCompression:"
"kBZip2Compression:"
"kLZ4Compression:"
"kLZ4HCCompression"},
{"compression_opts", "4:5:6"},
{"num_levels", "7"},
{"level0_file_num_compaction_trigger", "8"},
{"level0_slowdown_writes_trigger", "9"},
{"level0_stop_writes_trigger", "10"},
{"max_mem_compaction_level", "11"},
{"target_file_size_base", "12"},
{"target_file_size_multiplier", "13"},
{"max_bytes_for_level_base", "14"},
{"max_bytes_for_level_multiplier", "15"},
{"max_bytes_for_level_multiplier_additional", "16:17:18"},
{"expanded_compaction_factor", "19"},
{"source_compaction_factor", "20"},
{"max_grandparent_overlap_factor", "21"},
{"soft_rate_limit", "1.1"},
{"hard_rate_limit", "2.1"},
{"arena_block_size", "22"},
{"disable_auto_compactions", "true"},
{"purge_redundant_kvs_while_flush", "1"},
{"compaction_style", "kCompactionStyleLevel"},
{"verify_checksums_in_compaction", "false"},
{"compaction_options_fifo", "23"},
{"filter_deletes", "0"},
{"max_sequential_skip_in_iterations", "24"},
{"inplace_update_support", "true"},
{"inplace_update_num_locks", "25"},
{"memtable_prefix_bloom_bits", "26"},
{"memtable_prefix_bloom_probes", "27"},
{"memtable_prefix_bloom_huge_page_tlb_size", "28"},
{"bloom_locality", "29"},
{"max_successive_merges", "30"},
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
{"min_partial_merge_operands", "31"}
};
std::unordered_map<std::string, std::string> db_options_map = {
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
{"create_if_missing", "false"},
{"create_missing_column_families", "true"},
{"error_if_exists", "false"},
{"paranoid_checks", "true"},
{"max_open_files", "32"},
{"max_total_wal_size", "33"},
{"disable_data_sync", "false"},
{"use_fsync", "true"},
{"db_log_dir", "/db_log_dir"},
{"wal_dir", "/wal_dir"},
{"delete_obsolete_files_period_micros", "34"},
{"max_background_compactions", "35"},
{"max_background_flushes", "36"},
{"max_log_file_size", "37"},
{"log_file_time_to_roll", "38"},
{"keep_log_file_num", "39"},
{"max_manifest_file_size", "40"},
{"table_cache_numshardbits", "41"},
{"table_cache_remove_scan_count_limit", "42"},
{"WAL_ttl_seconds", "43"},
{"WAL_size_limit_MB", "44"},
{"manifest_preallocation_size", "45"},
{"allow_os_buffer", "false"},
{"allow_mmap_reads", "true"},
{"allow_mmap_writes", "false"},
{"is_fd_close_on_exec", "true"},
{"skip_log_error_on_recovery", "false"},
{"stats_dump_period_sec", "46"},
{"advise_random_on_open", "true"},
{"use_adaptive_mutex", "false"},
{"bytes_per_sync", "47"},
};
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ColumnFamilyOptions base_cf_opt;
ColumnFamilyOptions new_cf_opt;
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromMap(
base_cf_opt, cf_options_map, &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 1U);
ASSERT_EQ(new_cf_opt.max_write_buffer_number, 2);
ASSERT_EQ(new_cf_opt.min_write_buffer_number_to_merge, 3);
ASSERT_EQ(new_cf_opt.compression, kSnappyCompression);
ASSERT_EQ(new_cf_opt.compression_per_level.size(), 6U);
ASSERT_EQ(new_cf_opt.compression_per_level[0], kNoCompression);
ASSERT_EQ(new_cf_opt.compression_per_level[1], kSnappyCompression);
ASSERT_EQ(new_cf_opt.compression_per_level[2], kZlibCompression);
ASSERT_EQ(new_cf_opt.compression_per_level[3], kBZip2Compression);
ASSERT_EQ(new_cf_opt.compression_per_level[4], kLZ4Compression);
ASSERT_EQ(new_cf_opt.compression_per_level[5], kLZ4HCCompression);
ASSERT_EQ(new_cf_opt.compression_opts.window_bits, 4);
ASSERT_EQ(new_cf_opt.compression_opts.level, 5);
ASSERT_EQ(new_cf_opt.compression_opts.strategy, 6);
ASSERT_EQ(new_cf_opt.num_levels, 7);
ASSERT_EQ(new_cf_opt.level0_file_num_compaction_trigger, 8);
ASSERT_EQ(new_cf_opt.level0_slowdown_writes_trigger, 9);
ASSERT_EQ(new_cf_opt.level0_stop_writes_trigger, 10);
ASSERT_EQ(new_cf_opt.max_mem_compaction_level, 11);
ASSERT_EQ(new_cf_opt.target_file_size_base, static_cast<uint64_t>(12));
ASSERT_EQ(new_cf_opt.target_file_size_multiplier, 13);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_base, 14U);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_multiplier, 15);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_multiplier_additional.size(), 3U);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_multiplier_additional[0], 16);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_multiplier_additional[1], 17);
ASSERT_EQ(new_cf_opt.max_bytes_for_level_multiplier_additional[2], 18);
ASSERT_EQ(new_cf_opt.expanded_compaction_factor, 19);
ASSERT_EQ(new_cf_opt.source_compaction_factor, 20);
ASSERT_EQ(new_cf_opt.max_grandparent_overlap_factor, 21);
ASSERT_EQ(new_cf_opt.soft_rate_limit, 1.1);
ASSERT_EQ(new_cf_opt.hard_rate_limit, 2.1);
ASSERT_EQ(new_cf_opt.arena_block_size, 22U);
ASSERT_EQ(new_cf_opt.disable_auto_compactions, true);
ASSERT_EQ(new_cf_opt.purge_redundant_kvs_while_flush, true);
ASSERT_EQ(new_cf_opt.compaction_style, kCompactionStyleLevel);
ASSERT_EQ(new_cf_opt.verify_checksums_in_compaction, false);
ASSERT_EQ(new_cf_opt.compaction_options_fifo.max_table_files_size,
Fix mac compile
2014-09-18 00:40:25 +02:00
static_cast<uint64_t>(23));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_cf_opt.filter_deletes, false);
ASSERT_EQ(new_cf_opt.max_sequential_skip_in_iterations,
Fix mac compile
2014-09-18 00:40:25 +02:00
static_cast<uint64_t>(24));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_cf_opt.inplace_update_support, true);
ASSERT_EQ(new_cf_opt.inplace_update_num_locks, 25U);
ASSERT_EQ(new_cf_opt.memtable_prefix_bloom_bits, 26U);
ASSERT_EQ(new_cf_opt.memtable_prefix_bloom_probes, 27U);
ASSERT_EQ(new_cf_opt.memtable_prefix_bloom_huge_page_tlb_size, 28U);
ASSERT_EQ(new_cf_opt.bloom_locality, 29U);
ASSERT_EQ(new_cf_opt.max_successive_merges, 30U);
ASSERT_EQ(new_cf_opt.min_partial_merge_operands, 31U);
cf_options_map["write_buffer_size"] = "hello";
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromMap(
base_cf_opt, cf_options_map, &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
cf_options_map["write_buffer_size"] = "1";
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromMap(
base_cf_opt, cf_options_map, &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
cf_options_map["unknown_option"] = "1";
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromMap(
base_cf_opt, cf_options_map, &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
DBOptions base_db_opt;
DBOptions new_db_opt;
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetDBOptionsFromMap(base_db_opt, db_options_map, &new_db_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_db_opt.create_if_missing, false);
ASSERT_EQ(new_db_opt.create_missing_column_families, true);
ASSERT_EQ(new_db_opt.error_if_exists, false);
ASSERT_EQ(new_db_opt.paranoid_checks, true);
ASSERT_EQ(new_db_opt.max_open_files, 32);
ASSERT_EQ(new_db_opt.max_total_wal_size, static_cast<uint64_t>(33));
ASSERT_EQ(new_db_opt.disableDataSync, false);
ASSERT_EQ(new_db_opt.use_fsync, true);
ASSERT_EQ(new_db_opt.db_log_dir, "/db_log_dir");
ASSERT_EQ(new_db_opt.wal_dir, "/wal_dir");
ASSERT_EQ(new_db_opt.delete_obsolete_files_period_micros,
Fix mac compile
2014-09-18 00:40:25 +02:00
static_cast<uint64_t>(34));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_db_opt.max_background_compactions, 35);
ASSERT_EQ(new_db_opt.max_background_flushes, 36);
ASSERT_EQ(new_db_opt.max_log_file_size, 37U);
ASSERT_EQ(new_db_opt.log_file_time_to_roll, 38U);
ASSERT_EQ(new_db_opt.keep_log_file_num, 39U);
ASSERT_EQ(new_db_opt.max_manifest_file_size, static_cast<uint64_t>(40));
ASSERT_EQ(new_db_opt.table_cache_numshardbits, 41);
ASSERT_EQ(new_db_opt.table_cache_remove_scan_count_limit, 42);
ASSERT_EQ(new_db_opt.WAL_ttl_seconds, static_cast<uint64_t>(43));
ASSERT_EQ(new_db_opt.WAL_size_limit_MB, static_cast<uint64_t>(44));
ASSERT_EQ(new_db_opt.manifest_preallocation_size, 45U);
ASSERT_EQ(new_db_opt.allow_os_buffer, false);
ASSERT_EQ(new_db_opt.allow_mmap_reads, true);
ASSERT_EQ(new_db_opt.allow_mmap_writes, false);
ASSERT_EQ(new_db_opt.is_fd_close_on_exec, true);
ASSERT_EQ(new_db_opt.skip_log_error_on_recovery, false);
ASSERT_EQ(new_db_opt.stats_dump_period_sec, 46U);
ASSERT_EQ(new_db_opt.advise_random_on_open, true);
ASSERT_EQ(new_db_opt.use_adaptive_mutex, false);
ASSERT_EQ(new_db_opt.bytes_per_sync, static_cast<uint64_t>(47));
}
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
TEST(OptionsTest, GetOptionsFromStringTest) {
ColumnFamilyOptions base_cf_opt;
ColumnFamilyOptions new_cf_opt;
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
base_cf_opt.table_factory.reset();
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt, "", &new_cf_opt));
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=5", &new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 5U);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_TRUE(new_cf_opt.table_factory == nullptr);
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=6;", &new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 6U);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
" write_buffer_size = 7 ", &new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 7U);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
" write_buffer_size = 8 ; ", &new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 8U);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=9;max_write_buffer_number=10", &new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 9U);
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_cf_opt.max_write_buffer_number, 10);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=11; max_write_buffer_number = 12 ;",
&new_cf_opt));
Fixed Mac compile error in util/options_test.cc Summary: Fixed the following error in Mac: ./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare] BINARY_OP(IsEq, ==) ~~~~~~~~~~~~~~~~^~~ ./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP' if (! (x op y)) { \ ^ util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here ASSERT_EQ(new_cf_opt.write_buffer_size, 5); ^ Test Plan: options_test
2014-10-10 23:19:51 +02:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 11U);
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
ASSERT_EQ(new_cf_opt.max_write_buffer_number, 12);
// Wrong name "max_write_buffer_number_"
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=13;max_write_buffer_number_=14;",
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
&new_cf_opt));
// Wrong key/value pair
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=13;max_write_buffer_number;", &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
// Error Paring value
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=13;max_write_buffer_number=;", &new_cf_opt));
convert Options from string Summary: Allow accepting Options as a string of key/value pairs Test Plan: unit test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 19:00:12 +02:00
// Missing option name
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=13; =100;", &new_cf_opt));
Options helper supports k, m, g, and t unit suffixes Summary: Add unit support in options helper so we can specify, e.g., 10m for 10 megabytes. Test Plan: Updated options_test Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D28977
2014-11-17 22:47:51 +01:00
// Units (k)
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"memtable_prefix_bloom_bits=14k;max_write_buffer_number=-15K",
&new_cf_opt));
fix clang compilation Summary: as title
2014-11-19 05:41:29 +01:00
ASSERT_EQ(new_cf_opt.memtable_prefix_bloom_bits, 14UL*1024UL);
Options helper supports k, m, g, and t unit suffixes Summary: Add unit support in options helper so we can specify, e.g., 10m for 10 megabytes. Test Plan: Updated options_test Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D28977
2014-11-17 22:47:51 +01:00
ASSERT_EQ(new_cf_opt.max_write_buffer_number, -15*1024);
// Units (m)
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"max_write_buffer_number=16m;inplace_update_num_locks=17M",
&new_cf_opt));
Options helper supports k, m, g, and t unit suffixes Summary: Add unit support in options helper so we can specify, e.g., 10m for 10 megabytes. Test Plan: Updated options_test Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D28977
2014-11-17 22:47:51 +01:00
ASSERT_EQ(new_cf_opt.max_write_buffer_number, 16*1024*1024);
fix clang compilation Summary: as title
2014-11-19 05:41:29 +01:00
ASSERT_EQ(new_cf_opt.inplace_update_num_locks, 17*1024UL*1024UL);
Options helper supports k, m, g, and t unit suffixes Summary: Add unit support in options helper so we can specify, e.g., 10m for 10 megabytes. Test Plan: Updated options_test Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D28977
2014-11-17 22:47:51 +01:00
// Units (g)
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=18g;arena_block_size=19G", &new_cf_opt));
fix clang compilation Summary: as title
2014-11-19 05:41:29 +01:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 18*1024UL*1024UL*1024UL);
ASSERT_EQ(new_cf_opt.arena_block_size, 19*1024UL*1024UL*1024UL);
Options helper supports k, m, g, and t unit suffixes Summary: Add unit support in options helper so we can specify, e.g., 10m for 10 megabytes. Test Plan: Updated options_test Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D28977
2014-11-17 22:47:51 +01:00
// Units (t)
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=20t;arena_block_size=21T", &new_cf_opt));
fix clang compilation Summary: as title
2014-11-19 05:41:29 +01:00
ASSERT_EQ(new_cf_opt.write_buffer_size, 20*1024UL*1024UL*1024UL*1024UL);
ASSERT_EQ(new_cf_opt.arena_block_size, 21*1024UL*1024UL*1024UL*1024UL);
add support for nested BlockBasedTableOptions in config string Summary: Add support to allow nested config for block-based table factory. The format looks like this: "write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2" Test Plan: unit test Reviewers: yhchiang, rven, igor, ljin, jonahcohen Reviewed By: jonahcohen Subscribers: jonahcohen, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 22:18:57 +01:00
// Nested block based table options
// Emtpy
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={};arena_block_size=1024",
&new_cf_opt));
ASSERT_TRUE(new_cf_opt.table_factory != nullptr);
// Non-empty
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={block_cache=1M;block_size=4;};"
"arena_block_size=1024",
&new_cf_opt));
ASSERT_TRUE(new_cf_opt.table_factory != nullptr);
// Last one
ASSERT_OK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={block_cache=1M;block_size=4;}",
&new_cf_opt));
ASSERT_TRUE(new_cf_opt.table_factory != nullptr);
// Mismatch curly braces
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={{{block_size=4;};"
"arena_block_size=1024",
&new_cf_opt));
// Unexpected chars after closing curly brace
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={block_size=4;}};"
"arena_block_size=1024",
&new_cf_opt));
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={block_size=4;}xdfa;"
"arena_block_size=1024",
&new_cf_opt));
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={block_size=4;}xdfa",
&new_cf_opt));
// Invalid block based table option
ASSERT_NOK(GetColumnFamilyOptionsFromString(base_cf_opt,
"write_buffer_size=10;max_write_buffer_number=16;"
"block_based_table_factory={xx_block_size=4;}",
&new_cf_opt));
}
TEST(OptionsTest, GetBlockBasedTableOptionsFromString) {
BlockBasedTableOptions table_opt;
BlockBasedTableOptions new_opt;
// make sure default values are overwritten by something else
ASSERT_OK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;index_type=kHashSearch;"
"checksum=kxxHash;hash_index_allow_collision=1;no_block_cache=1;"
"block_cache=1M;block_cache_compressed=1k;block_size=1024;"
"block_size_deviation=8;block_restart_interval=4;"
"filter_policy=bloomfilter:4:true;whole_key_filtering=1",
&new_opt));
ASSERT_TRUE(new_opt.cache_index_and_filter_blocks);
ASSERT_EQ(new_opt.index_type, BlockBasedTableOptions::kHashSearch);
ASSERT_EQ(new_opt.checksum, ChecksumType::kxxHash);
ASSERT_TRUE(new_opt.hash_index_allow_collision);
ASSERT_TRUE(new_opt.no_block_cache);
ASSERT_TRUE(new_opt.block_cache != nullptr);
ASSERT_EQ(new_opt.block_cache->GetCapacity(), 1024UL*1024UL);
ASSERT_TRUE(new_opt.block_cache_compressed != nullptr);
ASSERT_EQ(new_opt.block_cache_compressed->GetCapacity(), 1024UL);
ASSERT_EQ(new_opt.block_size, 1024UL);
ASSERT_EQ(new_opt.block_size_deviation, 8);
ASSERT_EQ(new_opt.block_restart_interval, 4);
ASSERT_TRUE(new_opt.filter_policy != nullptr);
// unknown option
ASSERT_NOK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;index_type=kBinarySearch;"
"bad_option=1",
&new_opt));
// unrecognized index type
ASSERT_NOK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;index_type=kBinarySearchXX",
&new_opt));
// unrecognized checksum type
ASSERT_NOK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;checksum=kxxHashXX",
&new_opt));
// unrecognized filter policy name
ASSERT_NOK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;"
"filter_policy=bloomfilterxx:4:true",
&new_opt));
// unrecognized filter policy config
ASSERT_NOK(GetBlockBasedTableOptionsFromString(table_opt,
"cache_index_and_filter_blocks=1;"
"filter_policy=bloomfilter:4",
&new_opt));
}
Status StringToMap(
const std::string& opts_str,
std::unordered_map<std::string, std::string>* opts_map);
TEST(OptionsTest, StringToMapTest) {
std::unordered_map<std::string, std::string> opts_map;
// Regular options
ASSERT_OK(StringToMap("k1=v1;k2=v2;k3=v3", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "v2");
ASSERT_EQ(opts_map["k3"], "v3");
// Value with '='
opts_map.clear();
ASSERT_OK(StringToMap("k1==v1;k2=v2=;", &opts_map));
ASSERT_EQ(opts_map["k1"], "=v1");
ASSERT_EQ(opts_map["k2"], "v2=");
// Overwrriten option
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k1=v2;k3=v3", &opts_map));
ASSERT_EQ(opts_map["k1"], "v2");
ASSERT_EQ(opts_map["k3"], "v3");
// Empty value
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2=;k3=v3;k4=", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_TRUE(opts_map.find("k2") != opts_map.end());
ASSERT_EQ(opts_map["k2"], "");
ASSERT_EQ(opts_map["k3"], "v3");
ASSERT_TRUE(opts_map.find("k4") != opts_map.end());
ASSERT_EQ(opts_map["k4"], "");
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2=;k3=v3;k4= ", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_TRUE(opts_map.find("k2") != opts_map.end());
ASSERT_EQ(opts_map["k2"], "");
ASSERT_EQ(opts_map["k3"], "v3");
ASSERT_TRUE(opts_map.find("k4") != opts_map.end());
ASSERT_EQ(opts_map["k4"], "");
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2=;k3=", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_TRUE(opts_map.find("k2") != opts_map.end());
ASSERT_EQ(opts_map["k2"], "");
ASSERT_TRUE(opts_map.find("k3") != opts_map.end());
ASSERT_EQ(opts_map["k3"], "");
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2=;k3=;", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_TRUE(opts_map.find("k2") != opts_map.end());
ASSERT_EQ(opts_map["k2"], "");
ASSERT_TRUE(opts_map.find("k3") != opts_map.end());
ASSERT_EQ(opts_map["k3"], "");
// Regular nested options
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2={nk1=nv1;nk2=nv2};k3=v3", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "nk1=nv1;nk2=nv2");
ASSERT_EQ(opts_map["k3"], "v3");
// Multi-level nested options
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2={nk1=nv1;nk2={nnk1=nnk2}};"
"k3={nk1={nnk1={nnnk1=nnnv1;nnnk2;nnnv2}}};k4=v4",
&opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "nk1=nv1;nk2={nnk1=nnk2}");
ASSERT_EQ(opts_map["k3"], "nk1={nnk1={nnnk1=nnnv1;nnnk2;nnnv2}}");
ASSERT_EQ(opts_map["k4"], "v4");
// Garbage inside curly braces
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2={dfad=};k3={=};k4=v4",
&opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "dfad=");
ASSERT_EQ(opts_map["k3"], "=");
ASSERT_EQ(opts_map["k4"], "v4");
// Empty nested options
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2={};", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "");
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2={{{{}}}{}{}};", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "{{{}}}{}{}");
// With random spaces
opts_map.clear();
ASSERT_OK(StringToMap(" k1 = v1 ; k2= {nk1=nv1; nk2={nnk1=nnk2}} ; "
"k3={ { } }; k4= v4 ",
&opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "nk1=nv1; nk2={nnk1=nnk2}");
ASSERT_EQ(opts_map["k3"], "{ }");
ASSERT_EQ(opts_map["k4"], "v4");
// Empty key
ASSERT_NOK(StringToMap("k1=v1;k2=v2;=", &opts_map));
ASSERT_NOK(StringToMap("=v1;k2=v2", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2v2;", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2=v2;fadfa", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2=v2;;", &opts_map));
// Mismatch curly braces
ASSERT_NOK(StringToMap("k1=v1;k2={;k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{};k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={}};k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{}{}}};k3=v3", &opts_map));
// However this is valid!
opts_map.clear();
ASSERT_OK(StringToMap("k1=v1;k2=};k3=v3", &opts_map));
ASSERT_EQ(opts_map["k1"], "v1");
ASSERT_EQ(opts_map["k2"], "}");
ASSERT_EQ(opts_map["k3"], "v3");
// Invalid chars after closing curly brace
ASSERT_NOK(StringToMap("k1=v1;k2={{}}{};k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{}}cfda;k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{}} cfda;k3=v3", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{}} cfda", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{}}{}", &opts_map));
ASSERT_NOK(StringToMap("k1=v1;k2={{dfdl}adfa}{}", &opts_map));
}
TEST(OptionsTest, StringToMapRandomTest) {
std::unordered_map<std::string, std::string> opts_map;
// Make sure segfault is not hit by semi-random strings
std::vector<std::string> bases = {
"a={aa={};tt={xxx={}}};c=defff",
"a={aa={};tt={xxx={}}};c=defff;d={{}yxx{}3{xx}}",
"abc={{}{}{}{{{}}}{{}{}{}{}{}{}{}"};
for (std::string base : bases) {
for (int rand_seed = 301; rand_seed < 401; rand_seed++) {
Random rnd(rand_seed);
for (int attempt = 0; attempt < 10; attempt++) {
std::string str = base;
// Replace random position to space
size_t pos = static_cast<size_t>(
rnd.Uniform(static_cast<int>(base.size())));
str[pos] = ' ';
Status s = StringToMap(str, &opts_map);
ASSERT_TRUE(s.ok() || s.IsInvalidArgument());
opts_map.clear();
}
}
}
// Random Construct a string
std::vector<char> chars = {'{', '}', ' ', '=', ';', 'c'};
for (int rand_seed = 301; rand_seed < 1301; rand_seed++) {
Random rnd(rand_seed);
int len = rnd.Uniform(30);
std::string str = "";
for (int attempt = 0; attempt < len; attempt++) {
// Add a random character
size_t pos = static_cast<size_t>(
rnd.Uniform(static_cast<int>(chars.size())));
str.append(1, chars[pos]);
}
Status s = StringToMap(str, &opts_map);
ASSERT_TRUE(s.ok() || s.IsInvalidArgument());
s = StringToMap("name=" + str, &opts_map);
ASSERT_TRUE(s.ok() || s.IsInvalidArgument());
opts_map.clear();
}
Options conversion function for convenience Summary: as title Test Plan: options_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23283
2014-09-17 21:46:32 +02:00
}
Added CompatibleOptions for compatibility with LevelDB Options Summary: Created a CompatibleOptions object that can be used as a LevelDB Options object and then converted to a RocksDB Options object using the ConvertOptions() method. Test Plan: Unit test included in diff. Reviewers: ljin Reviewed By: ljin Subscribers: sdong, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28893
2014-11-21 04:24:39 +01:00
TEST(OptionsTest, ConvertOptionsTest) {
LevelDBOptions leveldb_opt;
Options converted_opt = ConvertOptions(leveldb_opt);
ASSERT_EQ(converted_opt.create_if_missing, leveldb_opt.create_if_missing);
ASSERT_EQ(converted_opt.error_if_exists, leveldb_opt.error_if_exists);
ASSERT_EQ(converted_opt.paranoid_checks, leveldb_opt.paranoid_checks);
ASSERT_EQ(converted_opt.env, leveldb_opt.env);
ASSERT_EQ(converted_opt.info_log.get(), leveldb_opt.info_log);
ASSERT_EQ(converted_opt.write_buffer_size, leveldb_opt.write_buffer_size);
ASSERT_EQ(converted_opt.max_open_files, leveldb_opt.max_open_files);
ASSERT_EQ(converted_opt.compression, leveldb_opt.compression);
std::shared_ptr<BlockBasedTableFactory> table_factory =
std::dynamic_pointer_cast<BlockBasedTableFactory>(
converted_opt.table_factory);
ASSERT_TRUE(table_factory.get() != nullptr);
const BlockBasedTableOptions table_opt = table_factory->GetTableOptions();
ASSERT_EQ(table_opt.block_cache->GetCapacity(), 8UL << 20);
ASSERT_EQ(table_opt.block_size, leveldb_opt.block_size);
ASSERT_EQ(table_opt.block_restart_interval,
leveldb_opt.block_restart_interval);
ASSERT_EQ(table_opt.filter_policy.get(), leveldb_opt.filter_policy);
}
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
} // namespace rocksdb
int main(int argc, char** argv) {
Make RocksDB compile without gflags Summary: We want to make sure people without gflags can compile RocksDB. Test Plan: remove gflags, make all Reviewers: sdong, rven, yhchiang, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 21:53:23 +01:00
#ifdef GFLAGS
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
ParseCommandLineFlags(&argc, &argv, true);
Make RocksDB compile without gflags Summary: We want to make sure people without gflags can compile RocksDB. Test Plan: remove gflags, make all Reviewers: sdong, rven, yhchiang, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 21:53:23 +01:00
#endif // GFLAGS
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
return rocksdb::test::RunAllTests();
}
Reference in New Issue Copy Permalink