andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
cb2476a0ca
rocksdb/util
History
Aaron Gao cb2476a0ca fix rate limiter to avoid starvation 
Summary:
The current implementation of rate limiter has the possibility to introduce resource starvation when change its limit.
This diff aims to fix this problem by consuming request bytes partially.

Test Plan:
```
./rate_limiter_test
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from RateLimiterTest
[ RUN      ] RateLimiterTest.OverflowRate
[       OK ] RateLimiterTest.OverflowRate (0 ms)
[ RUN      ] RateLimiterTest.StartStop
[       OK ] RateLimiterTest.StartStop (0 ms)
[ RUN      ] RateLimiterTest.Rate
request size [1 - 1023], limit 10 KB/sec, actual rate: 10.355712 KB/sec, elapsed 2.00 seconds
request size [1 - 1023], limit 20 KB/sec, actual rate: 19.136564 KB/sec, elapsed 2.00 seconds
request size [1 - 2047], limit 20 KB/sec, actual rate: 20.783976 KB/sec, elapsed 2.10 seconds
request size [1 - 2047], limit 40 KB/sec, actual rate: 39.308144 KB/sec, elapsed 2.10 seconds
request size [1 - 4095], limit 40 KB/sec, actual rate: 40.318349 KB/sec, elapsed 2.20 seconds
request size [1 - 4095], limit 80 KB/sec, actual rate: 79.667396 KB/sec, elapsed 2.20 seconds
request size [1 - 8191], limit 80 KB/sec, actual rate: 81.807158 KB/sec, elapsed 2.30 seconds
request size [1 - 8191], limit 160 KB/sec, actual rate: 160.659761 KB/sec, elapsed 2.20 seconds
request size [1 - 16383], limit 160 KB/sec, actual rate: 160.700990 KB/sec, elapsed 3.00 seconds
request size [1 - 16383], limit 320 KB/sec, actual rate: 317.639481 KB/sec, elapsed 2.50 seconds
[       OK ] RateLimiterTest.Rate (22618 ms)
[ RUN      ] RateLimiterTest.LimitChangeTest
[COMPLETE] request size 10 KB, new limit 20KB/sec, refill period 1000 ms
[COMPLETE] request size 10 KB, new limit 5KB/sec, refill period 1000 ms
[COMPLETE] request size 20 KB, new limit 40KB/sec, refill period 1000 ms
[COMPLETE] request size 20 KB, new limit 10KB/sec, refill period 1000 ms
[COMPLETE] request size 40 KB, new limit 80KB/sec, refill period 1000 ms
[COMPLETE] request size 40 KB, new limit 20KB/sec, refill period 1000 ms
[COMPLETE] request size 80 KB, new limit 160KB/sec, refill period 1000 ms
[COMPLETE] request size 80 KB, new limit 40KB/sec, refill period 1000 ms
[COMPLETE] request size 160 KB, new limit 320KB/sec, refill period 1000 ms
[COMPLETE] request size 160 KB, new limit 80KB/sec, refill period 1000 ms
[       OK ] RateLimiterTest.LimitChangeTest (5002 ms)
[----------] 4 tests from RateLimiterTest (27620 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (27621 ms total)
[  PASSED  ] 4 tests.
```

Reviewers: sdong, IslamAbdelRahman, yiwu, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D60207
2016-07-01 00:16:29 -07:00
..
aligned_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
allocator.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
arena.cc Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
autovector_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
autovector.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
build_version.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
cache_bench.cc Cache to have an option to fail Cache::Insert() when full 2016-03-10 17:35:19 -08:00
cache_test.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
cache.cc add simulator Cache as class SimCache/SimLRUCache(with test) 2016-05-23 23:35:23 -07:00
channel.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
coding.h Fixed compile warnings in posix_logger.h and coding.h 2016-03-31 16:01:47 -07:00
compaction_job_stats_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
comparator.cc Improve BytewiseComparatorImpl::FindShortestSeparator 2016-04-25 23:02:14 -07:00
compression.h Shared dictionary compression using reference block 2016-04-27 17:36:03 -07:00
concurrent_arena.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
concurrent_arena.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
crc32c.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler_test.cc Fix name conflict in delete_shceduler_test and db_sst_test 2016-06-10 15:39:17 -07:00
delete_scheduler.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
delete_scheduler.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
dynamic_bloom.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
env_basic_test.cc Run env_basic_test on Env::Default 2016-06-30 18:34:29 -07:00
env_chroot.cc env_chroot.cc: FreeBSD likes stdlib.h for realpaht() and friends (#1154) 2016-06-05 17:22:55 -07:00
env_chroot.h Introduce chroot Env 2016-05-06 17:42:50 -07:00
env_hdfs.cc Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions. 2016-05-20 07:42:18 -07:00
env_posix.cc Direct IO fix for Mac 2016-05-23 16:38:25 -07:00
env_test.cc Fix Windows build break 2016-06-01 18:07:59 -07:00
env.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
event_logger.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
fault_injection_test_env.cc [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
fault_injection_test_env.h [rocksdb] Recovery path sequence miscount fix 2016-05-10 14:06:07 -07:00
file_reader_writer_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_reader_writer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
file_util.cc Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
file_util.h Forge current file for checkpoint 2016-03-17 10:07:21 -07:00
filelock_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
filter_policy.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
hash.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
heap_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
heap.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
histogram_test.cc Fix Build Error 2016-03-11 22:56:25 -08:00
histogram_windowing.cc Fix in HistogramWindowingImpl 2016-03-17 14:28:41 -07:00
histogram_windowing.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
histogram.cc Fix FB internal CI build failure 2016-03-15 11:38:15 -07:00
histogram.h Histogram Concurrency Improvement and Time-Windowing Support 2016-03-11 16:54:25 -08:00
instrumented_mutex.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
instrumented_mutex.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
io_posix.cc Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
io_posix.h Direct IO capability for RocksDB 2016-05-23 12:27:27 -07:00
iostats_context_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
iostats_context_test.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
iostats_context.cc IOStatsContext::ToString() add option to exclude zero counters 2016-02-23 10:26:24 -08:00
kv_map.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_buffer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
log_write_bench.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
logging.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
lru_cache_handle.h add simulator Cache as class SimCache/SimLRUCache(with test) 2016-05-23 23:35:23 -07:00
memenv.cc In-memory environment read beyond EOF 2016-05-27 12:10:26 -07:00
mock_env_test.cc Create env_basic_test [pluggable Env part 2] 2016-06-03 15:13:03 -07:00
mock_env.cc In-memory environment read beyond EOF 2016-05-27 12:10:26 -07:00
mock_env.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
murmurhash.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
mutable_cf_options.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
mutable_cf_options.h Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
mutexlock.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_helper.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options_helper.h Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options_parser.cc Add parsing of missing DB options 2016-03-02 10:34:14 -08:00
options_parser.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_sanity_check.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
options_settable_test.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options_test.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
options.cc Add a read option to enable background purge when cleaning up iterators 2016-06-21 18:41:23 -07:00
perf_context_imp.h fix ios build error 2016-02-17 20:22:40 +08:00
perf_context.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level_imp.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
perf_level.cc Assert boundary checks for SetPerfLevel() 2016-06-01 09:07:09 -07:00
perf_step_timer.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
posix_logger.h Fixed compile warnings in posix_logger.h and coding.h 2016-03-31 16:01:47 -07:00
random.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
random.h Fixes warnings and ensure correct int behavior on 32-bit platforms. 2016-03-16 22:57:57 +01:00
rate_limiter_test.cc fix rate limiter to avoid starvation 2016-07-01 00:16:29 -07:00
rate_limiter.cc fix rate limiter to avoid starvation 2016-07-01 00:16:29 -07:00
rate_limiter.h Handle overflow case of rate limiter's paramters 2016-05-27 16:15:28 -07:00
slice_transform_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
slice.cc Merge db/slice.cc into util/slice.cc 2016-06-10 16:37:36 -07:00
sst_file_manager_impl.cc Fix SstFileManager uninitialized data member 2016-02-18 11:25:19 -08:00
sst_file_manager_impl.h Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage 2016-02-17 15:20:23 -08:00
statistics.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
statistics.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status_message.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
status.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
stderr_logger.h Stderr info logger 2016-04-01 11:06:06 -07:00
stop_watch.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
string_util.h FreeBSD does not have std::to_string (#1190) 2016-06-29 07:35:17 -07:00
sync_point.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
sync_point.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testharness.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testharness.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
testutil.cc Deprectate filter_deletes 2016-06-17 10:30:47 -07:00
testutil.h db/log_test: add recycle log test 2016-05-22 22:00:15 -07:00
thread_list_test.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_local_test.cc Fix LITE build thread_local_test 2016-02-19 13:57:18 -08:00
thread_local.cc Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_local.h Fixed a dependency issue of ThreadLocalPtr 2016-02-10 16:56:01 -08:00
thread_operation.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_impl.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_updater.cc Use pure if-then check instead of assert in EraseColumnFamilyInfo 2016-03-04 16:03:31 -08:00
thread_status_updater.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util_debug.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
thread_status_util.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
threadpool.cc threadpool.cc: abort() lives in stdlib.h on FreeBSD (#1155) 2016-06-05 17:23:38 -07:00
threadpool.h Use generic threadpool for Windows environment (#1120) 2016-05-12 18:34:04 -07:00
transaction_test_util.cc Add multithreaded transaction test 2016-03-11 15:16:52 -08:00
transaction_test_util.h Fix AppVeyor build error 2016-03-15 10:57:33 -07:00
xfunc.cc Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xfunc.h Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
xxhash.cc Enable MS Warning C4804 : unsafe use of type 'bool' in operation 2015-11-18 16:23:19 -08:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00