andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
c815351038
rocksdb/util
History
agiardullo c815351038 Support saving history in memtable_list 
Summary:
For transactions, we are using the memtables to validate that there are no write conflicts.  But after flushing, we don't have any memtables, and transactions could fail to commit.  So we want to someone keep around some extra history to use for conflict checking.  In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit.

After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure).  It seems like the best place for this is abstracted inside the memtable_list.  I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much.

This diff adds a new parameter to control how much memtable history to keep around after flushing.  However, it sounds like people aren't too fond of adding new parameters.  So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers.  This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit.  (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached).  So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit).

However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions.

Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests.  Added testing in memtablelist_test and planning on adding more testing here.

Reviewers: sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37443
2015-05-28 16:34:24 -07:00
..
allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
arena_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
arena.cc Make arena use hugepage if possible 2014-11-21 14:11:40 -08:00
arena.h Removing unnecessary kInlineSize 2015-03-12 21:13:53 +03:00
auto_roll_logger_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
auto_roll_logger.cc Add Header to logging to capture application level information 2015-02-06 10:37:45 -08:00
auto_roll_logger.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
autovector_test.cc Build for CYGWIN 2015-04-23 21:33:44 -07:00
autovector.h Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
bloom_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
bloom.cc fix typos 2015-04-25 18:14:27 +09:00
build_version.h build: do not relink every single binary just for a timestamp 2015-02-19 13:11:10 -08:00
cache_bench.cc Fix -Wshadow for tools 2014-11-07 15:04:30 -08:00
cache_test.cc Fix memory leak in cache_test introduced in the previous commit 2015-04-26 21:47:30 -07:00
cache.cc Fix CLANG build issue introduced in previous commit 2015-04-24 14:45:12 -07:00
coding_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
coding.cc Removing BitStream* functions 2014-08-19 06:48:21 -07:00
coding.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
comparator.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
compression.h Clean up compression logging 2015-04-06 12:50:44 -07:00
crc32c_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
crc32c.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
crc32c.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_info_dumper.cc Change the log-level of DB summary and options from INFO_LEVEL to WARN_LEVEL 2015-05-22 11:54:59 -07:00
db_info_dumper.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
dynamic_bloom_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
dynamic_bloom.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
dynamic_bloom.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
env_hdfs.cc fix typos 2015-04-25 18:14:27 +09:00
env_posix.cc Add an option wal_bytes_per_sync to control sync_file_range for WAL files 2015-05-18 17:03:59 -07:00
env_test.cc test: avoid vuln-inducing use of temporary directory 2015-04-23 08:00:56 -07:00
env.cc Add an option wal_bytes_per_sync to control sync_file_range for WAL files 2015-05-18 17:03:59 -07:00
event_logger_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
event_logger.cc Allow EventLogger to directly log from a JSONWriter. 2015-05-21 15:39:30 -07:00
event_logger.h Allow EventLogger to directly log from a JSONWriter. 2015-05-21 15:39:30 -07:00
file_util.cc Provide openable snapshots 2014-11-14 11:38:26 -08:00
file_util.h Provide openable snapshots 2014-11-14 11:38:26 -08:00
filelock_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
filter_policy.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
hash_cuckoo_rep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_cuckoo_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_linklist_rep.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
hash_linklist_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_skiplist_rep.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
hash_skiplist_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
hash.h Introduce GetThreadList API 2014-11-20 10:49:32 -08:00
histogram_test.cc fix typos 2015-04-25 18:14:27 +09:00
histogram.cc fix typos 2015-04-25 18:14:27 +09:00
histogram.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
instrumented_mutex.cc Perf Context to report DB mutex waiting time 2015-02-09 17:55:12 -08:00
instrumented_mutex.h Add a counter for collecting the wait time on db mutex. 2015-02-04 21:39:45 -08:00
iostats_context_imp.h Fix ios compile 2014-08-28 12:46:05 -04:00
iostats_context.cc Fix ios compile 2014-08-28 12:46:05 -04:00
ldb_cmd_execute_result.h rocksdb: Small refactoring before migrating to gtest 2015-03-16 18:08:59 -07:00
ldb_cmd.cc Build for CYGWIN 2015-04-23 21:33:44 -07:00
ldb_cmd.h rocksdb: Small refactoring before migrating to gtest 2015-03-16 18:08:59 -07:00
ldb_tool.cc Added 'dump_live_files' command to ldb tool. 2014-12-12 17:50:36 -08:00
log_buffer.cc Enlarge log size cap when printing file summary 2014-09-23 16:56:34 -07:00
log_buffer.h RocksDB on FreeBSD support 2015-02-26 15:19:17 -08:00
log_write_bench.cc Fix more gflag namespace issues 2014-05-09 08:41:02 -07:00
logging.cc Make the benchmark scripts configurable and add tests 2015-03-30 11:28:25 -07:00
logging.h Make the benchmark scripts configurable and add tests 2015-03-30 11:28:25 -07:00
manual_compaction_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
memenv_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
memenv.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
mock_env_test.cc Fix flakiness of WalManagerTest 2015-04-13 16:15:05 -07:00
mock_env.cc Fix flakiness of WalManagerTest 2015-04-13 16:15:05 -07:00
mock_env.h Fix flakiness of WalManagerTest 2015-04-13 16:15:05 -07:00
murmurhash.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
murmurhash.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
mutable_cf_options.cc options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically. 2015-03-02 22:40:41 -08:00
mutable_cf_options.h options.paranoid_file_checks to read all rows after writing to a file. 2015-04-23 11:34:35 -07:00
mutexlock.h Add separate Read/WriteUnlock methods in MutexRW. 2014-06-16 15:41:46 -07:00
options_builder.cc Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE 2014-12-04 13:56:14 -08:00
options_helper.cc Support saving history in memtable_list 2015-05-28 16:34:24 -07:00
options_helper.h Missing header in build on CentOS 2014-11-18 22:21:02 +01:00
options_test.cc Support saving history in memtable_list 2015-05-28 16:34:24 -07:00
options.cc Support saving history in memtable_list 2015-05-28 16:34:24 -07:00
perf_context_imp.h Refactor PerfStepTimer to stop on destruct 2014-09-02 12:04:22 -07:00
perf_context.cc Adding stats for the merge and filter operation 2015-03-24 14:42:04 -07:00
posix_logger.h Remove duplicated code 2015-05-18 23:44:52 +03:00
random.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
rate_limiter_test.cc Enable dynamic changing of rate limiter's bytes_per_second 2015-03-18 15:35:55 -07:00
rate_limiter.cc Enable dynamic changing of rate limiter's bytes_per_second 2015-03-18 15:35:55 -07:00
rate_limiter.h Enable dynamic changing of rate limiter's bytes_per_second 2015-03-18 15:35:55 -07:00
scoped_arena_iterator.h Remove path with arena==nullptr from NewInternalIterator 2014-09-04 17:40:41 -07:00
skiplistrep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
slice_transform_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
slice.cc rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
sst_dump_test.cc A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge 2015-04-06 10:27:21 -07:00
sst_dump_tool_imp.h Disable pre-fetching of index and filter blocks for sst_dump_tool. 2015-02-25 16:34:26 -08:00
sst_dump_tool.cc Abstract out SetMaxPossibleForUserKey() and SetMinPossibleForUserKey 2015-04-23 18:08:37 -07:00
statistics.cc Fix assert in histogramData 2015-01-23 18:10:52 -08:00
statistics.h make statistics forward-able 2014-07-28 12:10:49 -07:00
status.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
stl_wrappers.h Killing Transform Rep 2013-12-03 12:42:15 -08:00
stop_watch.h Change StopWatch interface 2014-07-28 12:22:37 -07:00
string_util.cc Clean up StringSplit 2014-11-21 11:05:28 -05:00
string_util.h Build for CYGWIN 2015-04-23 21:33:44 -07:00
sync_point.cc SyncPoint to allow a callback with an argument and use it to get DBTest.DynamicLevelCompressionPerLevel2 more straight-forward 2015-04-14 16:18:50 -07:00
sync_point.h SyncPoint to allow a callback with an argument and use it to get DBTest.DynamicLevelCompressionPerLevel2 more straight-forward 2015-04-14 16:18:50 -07:00
testharness.cc rocksdb: print status error message when (ASSERT|EXPECT)_OK fails 2015-03-19 17:32:43 -07:00
testharness.h rocksdb: print status error message when (ASSERT|EXPECT)_OK fails 2015-03-19 17:32:43 -07:00
testutil.cc Merger test 2014-09-08 22:24:40 -07:00
testutil.h rocksdb: Add missing override 2015-02-26 11:28:41 -08:00
thread_list_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
thread_local_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
thread_local.cc Use ustricter consistency in thread local operations 2015-01-27 13:56:03 -08:00
thread_local.h Improve the comment of util/thread_local.h 2014-10-21 17:28:31 -07:00
thread_operation.h update an import path to fit in with the rest of the kids 2015-05-22 22:56:32 -07:00
thread_status_impl.cc Fixed compile errors due to some gcc does not have std::map::emplace 2015-05-18 13:48:56 -07:00
thread_status_updater_debug.cc Allow GetThreadList() to indicate a thread is doing Compaction. 2015-01-13 00:04:08 -08:00
thread_status_updater.cc Allow GetThreadList() to report basic compaction operation properties. 2015-05-06 22:51:06 -07:00
thread_status_updater.h Allow GetThreadList() to report basic compaction operation properties. 2015-05-06 22:51:06 -07:00
thread_status_util_debug.cc Fix bad performance in debug mode 2015-04-13 15:58:45 -07:00
thread_status_util.cc Allow GetThreadList() to report basic compaction operation properties. 2015-05-06 22:51:06 -07:00
thread_status_util.h Allow GetThreadList() to report basic compaction operation properties. 2015-05-06 22:51:06 -07:00
vectorrep.cc assert(sorted) in vector rep 2015-04-13 17:33:24 -07:00
xfunc.cc Support saving history in memtable_list 2015-05-28 16:34:24 -07:00
xfunc.h Support saving history in memtable_list 2015-05-28 16:34:24 -07:00
xxhash.cc Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00