andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
ee95cae9a4
rocksdb/util
History
Alexey Maykov ee95cae9a4 Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references 
Summary:
Currently, blocks which have more than one reference (ie referenced by something other than cache itself) are evicted from cache. This doesn't make much sense:
- blocks are still in RAM, so the RAM usage reported by the cache is incorrect
- if the same block is needed by another iterator, it will be loaded and decompressed again

This diff changes the reference counting scheme a bit. Previously, if the cache contained the block, this was accounted for in its refcount. After this change, the refcount is only used to track external references. There is a boolean flag which indicates whether or not the block is contained in the cache.
This diff also changes how LRU list is used. Previously, both hashtable and the LRU list contained all blocks. After this change, the LRU list contains blocks with the refcount==0, ie those which can be evicted from the cache.

Note that this change still allows for cache to grow beyond its capacity. This happens when all blocks are pinned (ie refcount>0). This is consistent with the current behavior. The cache's insert function never fails. I spent lots of time trying to make table_reader and other places work with the insert which might failed. It turned out to be pretty hard. It might really destabilize some customers, so finally, I decided against doing this.

table_cache_remove_scan_count_limit option will be unneeded after this change, but I will remove it in the following diff, if this one gets approved

Test Plan: Ran tests, made sure they pass

Reviewers: sdong, ljin

Differential Revision: https://reviews.facebook.net/D25503
2014-12-10 22:28:53 -08:00
..
allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
arena_test.cc Make arena use hugepage if possible 2014-11-21 14:11:40 -08:00
arena.cc Make arena use hugepage if possible 2014-11-21 14:11:40 -08:00
arena.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
auto_roll_logger_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
auto_roll_logger.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
auto_roll_logger.h Renamed InfoLogLevel::DEBUG to InfoLogLevel::DEBUG_LEVEL 2014-04-10 15:27:42 -07:00
autovector_test.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
autovector.h Replace exception by assertion in autovector 2014-12-04 11:41:56 -08:00
benchharness_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
benchharness.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
benchharness.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
blob_store_test.cc Move the compiler back to 4.8.1 + more small fixes 2013-11-18 11:40:16 -08:00
blob_store.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
blob_store.h Turn on -Wshadow 2014-10-31 11:59:54 -07:00
bloom_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
bloom.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
build_version.h Dont build version if compiling for IOS 2014-04-11 10:54:47 -07:00
cache_bench.cc Fix -Wshadow for tools 2014-11-07 15:04:30 -08:00
cache_test.cc Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references 2014-12-10 22:28:53 -08:00
cache.cc Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references 2014-12-10 22:28:53 -08:00
coding_test.cc Remove BitStream* tests 2014-08-19 09:52:54 -04:00
coding.cc Removing BitStream* functions 2014-08-19 06:48:21 -07:00
coding.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
comparator.cc Changed name of ReverseBytewiseComparator based on review comment 2014-09-27 10:06:13 +02:00
crc32c_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
crc32c.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
crc32c.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
db_info_dumper.cc Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
db_info_dumper.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
dynamic_bloom_test.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
dynamic_bloom.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
dynamic_bloom.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
env_hdfs.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
env_posix.cc Fix #434 2014-12-09 10:22:07 -08:00
env_test.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
env.cc Check InfoLogLevel earlier in Log functions. 2014-10-30 13:36:18 -07:00
file_util.cc Provide openable snapshots 2014-11-14 11:38:26 -08:00
file_util.h Provide openable snapshots 2014-11-14 11:38:26 -08:00
filelock_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
filter_policy.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
hash_cuckoo_rep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_cuckoo_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_linklist_rep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_linklist_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_skiplist_rep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash_skiplist_rep.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
hash.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
hash.h Introduce GetThreadList API 2014-11-20 10:49:32 -08:00
histogram_test.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
histogram.cc Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
histogram.h Fix iOS compile with -Wshorten-64-to-32 2014-11-13 14:39:30 -05:00
iostats_context_imp.h Fix ios compile 2014-08-28 12:46:05 -04:00
iostats_context.cc Fix ios compile 2014-08-28 12:46:05 -04:00
ldb_cmd_execute_result.h Revert "Fix lint errors and coding style of ldb related codes." 2014-10-31 19:22:49 -07:00
ldb_cmd.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
ldb_cmd.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
ldb_tool.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
log_buffer.cc Enlarge log size cap when printing file summary 2014-09-23 16:56:34 -07:00
log_buffer.h Enlarge log size cap when printing file summary 2014-09-23 16:56:34 -07:00
log_write_bench.cc Fix more gflag namespace issues 2014-05-09 08:41:02 -07:00
logging.cc remove cast, replace %llu with % PRIu64 2014-09-23 01:10:46 +08:00
logging.h remove unused variable 2014-09-21 22:20:00 +08:00
manual_compaction_test.cc Fix CompactRange to apply filter to every key 2014-01-14 16:19:09 -08:00
mock_env_test.cc Add fsync / corrupt simulation to env_mem 2014-10-31 15:16:31 -07:00
mock_env.cc Provide openable snapshots 2014-11-14 11:38:26 -08:00
mock_env.h Provide openable snapshots 2014-11-14 11:38:26 -08:00
murmurhash.cc Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
murmurhash.h Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
mutable_cf_options.cc remove all remaining references to cfd->options() 2014-11-18 10:20:10 -08:00
mutable_cf_options.h remove all remaining references to cfd->options() 2014-11-18 10:20:10 -08:00
mutexlock.h Add separate Read/WriteUnlock methods in MutexRW. 2014-06-16 15:41:46 -07:00
options_builder.cc Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE 2014-12-04 13:56:14 -08:00
options_helper.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
options_helper.h Missing header in build on CentOS 2014-11-18 22:21:02 +01:00
options_test.cc Make RocksDB compile without gflags 2014-11-24 15:53:23 -05:00
options.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
perf_context_imp.h Refactor PerfStepTimer to stop on destruct 2014-09-02 12:04:22 -07:00
perf_context.cc Add a function to return current perf level 2014-07-10 11:35:48 -07:00
posix_logger.h Fix additional -Wshorten-64-to-32 errros 2014-11-11 14:09:14 -08:00
random.h Add appropriate LICENSE and Copyright message. 2013-10-16 17:48:41 -07:00
rate_limiter_test.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
rate_limiter.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
rate_limiter.h expose RateLimiter definition 2014-07-25 15:17:06 -07:00
scoped_arena_iterator.h Remove path with arena==nullptr from NewInternalIterator 2014-09-04 17:40:41 -07:00
signal_test.cc util/signal_test.cc: suppress intentional null pointer deref 2014-09-30 23:30:32 +02:00
skiplistrep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
slice.cc Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
sst_dump_tool.cc Make db_stress built for ROCKSDB_LITE 2014-11-14 10:20:51 -08:00
statistics.cc fix more compile warnings 2014-09-05 14:14:37 +08:00
statistics.h make statistics forward-able 2014-07-28 12:10:49 -07:00
status.cc Turn on -Wshorten-64-to-32 and fix all the errors 2014-11-11 16:47:22 -05:00
stl_wrappers.h Killing Transform Rep 2013-12-03 12:42:15 -08:00
stop_watch.h Change StopWatch interface 2014-07-28 12:22:37 -07:00
string_util.cc Clean up StringSplit 2014-11-21 11:05:28 -05:00
string_util.h Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
sync_point.cc Don't compile sync_point if NDEBUG 2014-04-17 10:49:58 -07:00
sync_point.h Fix ifdef NDEBUG 2014-04-17 14:29:28 -07:00
testharness.cc introduce TestMemEnv and use it in db_test 2014-10-31 15:08:10 -07:00
testharness.h Add rocksdb::ToString() to address cases where std::to_string is not available. 2014-11-24 20:44:49 -08:00
testutil.cc Merger test 2014-09-08 22:24:40 -07:00
testutil.h Add ComparatorDBTest to test non-default comparators 2014-10-29 15:42:36 -07:00
thread_list_test.cc Introduce GetThreadList API 2014-11-20 10:49:32 -08:00
thread_local_test.cc Turn on -Wshadow 2014-10-31 11:59:54 -07:00
thread_local.cc Replace runtime_error exception by abort() in thread_local 2014-12-04 13:35:31 -08:00
thread_local.h Improve the comment of util/thread_local.h 2014-10-21 17:28:31 -07:00
thread_status_impl_debug.cc Add enable_thread_tracking to DBOptions 2014-11-20 21:13:18 -08:00
thread_status_impl.cc Fix leak when create_missing_column_families=true on ThreadStatus 2014-11-22 00:04:41 -08:00
thread_status_impl.h Fix leak when create_missing_column_families=true on ThreadStatus 2014-11-22 00:04:41 -08:00
vectorrep.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
xxhash.cc xxHash 2014-05-01 14:09:32 -04:00
xxhash.h xxHash 2014-05-01 14:09:32 -04:00