andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
e85f2c64cb
rocksdb/util
History
Mike Kolupaev e85f2c64cb Prevent empty memtables from using a lot of memory 
Summary:
This fixes OOMs that we (logdevice) are currently having in production.

SkipListRep constructor does a couple small allocations from ConcurrentArena (see InlineSkipList constructor). ConcurrentArena would sometimes allocate an entire block for that, which is a few megabytes (we use Options::arena_block_size = 4 MB). So an empty memtable can take take 4 MB of memory. We have ~40k column families (spread across 15 DB instances), so 4 MB per empty memtable easily OOMs a machine for us.

This PR makes ConcurrentArena always allocate from Arena's inline block when possible. So as long as InlineSkipList's initial allocations are below 2 KB there would be no blocks allocated for empty memtables.
Closes https://github.com/facebook/rocksdb/pull/2569

Differential Revision: D5404029

Pulled By: al13n321

fbshipit-source-id: 568ec22a3fd1a485c06123f6b2dfc5e9ef67cd23
2017-07-28 15:58:43 -07:00
..
aligned_buffer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
allocator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
arena_test.cc Prevent empty memtables from using a lot of memory 2017-07-28 15:58:43 -07:00
arena.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
arena.h Prevent empty memtables from using a lot of memory 2017-07-28 15:58:43 -07:00
auto_roll_logger_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
auto_roll_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
auto_roll_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
autovector_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
autovector.h Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
build_version.cc.in Makefile: generate util/build_version.cc from .in file (#1384) 2016-10-25 11:31:39 -07:00
build_version.h Change RocksDB License 2017-07-15 16:11:23 -07:00
channel.h Change RocksDB License 2017-07-15 16:11:23 -07:00
coding_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
coding.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
coding.h Change RocksDB License 2017-07-15 16:11:23 -07:00
compaction_job_stats_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
comparator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
compression.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
concurrent_arena.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
concurrent_arena.h Prevent empty memtables from using a lot of memory 2017-07-28 15:58:43 -07:00
core_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c.cc Revert "CRC32 Power Optimization Changes" 2017-07-26 19:31:36 -07:00
crc32c.h Change RocksDB License 2017-07-15 16:11:23 -07:00
delete_scheduler_test.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
delete_scheduler.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
delete_scheduler.h Change RocksDB License 2017-07-15 16:11:23 -07:00
dynamic_bloom_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
dynamic_bloom.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
dynamic_bloom.h Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
fault_injection_test_env.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
fault_injection_test_env.h Change RocksDB License 2017-07-15 16:11:23 -07:00
file_reader_writer_test.cc Fix LITE unit tests 2017-07-26 21:11:47 -07:00
file_reader_writer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_reader_writer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
file_util.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_util.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filelock_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
filename.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
filename.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filter_policy.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
hash.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
hash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
heap_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
heap.h Change RocksDB License 2017-07-15 16:11:23 -07:00
kv_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_write_bench.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
logging.h Change RocksDB License 2017-07-15 16:11:23 -07:00
memory_usage.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mpsc.h Change RocksDB License 2017-07-15 16:11:23 -07:00
murmurhash.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
murmurhash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mutexlock.h Change RocksDB License 2017-07-15 16:11:23 -07:00
random.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
random.h Change RocksDB License 2017-07-15 16:11:23 -07:00
rate_limiter_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
rate_limiter.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
rate_limiter.h Change RocksDB License 2017-07-15 16:11:23 -07:00
slice_transform_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
slice.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
sst_file_manager_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_manager_impl.h Change RocksDB License 2017-07-15 16:11:23 -07:00
status_message.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
status.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
stderr_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
stop_watch.h Change RocksDB License 2017-07-15 16:11:23 -07:00
string_util.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
string_util.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sync_point.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
sync_point.h Change RocksDB License 2017-07-15 16:11:23 -07:00
testharness.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
testharness.h Change RocksDB License 2017-07-15 16:11:23 -07:00
testutil.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
testutil.h Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
thread_list_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
thread_local_test.cc Revert "comment out unused parameters" 2017-07-21 18:26:26 -07:00
thread_local.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
thread_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
thread_operation.h Change RocksDB License 2017-07-15 16:11:23 -07:00
threadpool_imp.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
threadpool_imp.h Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_queue.h Change RocksDB License 2017-07-15 16:11:23 -07:00
transaction_test_util.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
transaction_test_util.h Change RocksDB License 2017-07-15 16:11:23 -07:00
xxhash.cc fixed typo 2017-06-05 11:27:34 -07:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00