andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
dfc61e7c24
rocksdb/util
History
Gabriel Wicke 090c78a0d7 Support lowering CPU priority of background threads 
Summary:
Background activities like compaction can negatively affect
latency of higher-priority tasks like request processing. To avoid this,
rocksdb already lowers the IO priority of background threads on Linux
systems. While this takes care of typical IO-bound systems, it does not
help much when CPU (temporarily) becomes the bottleneck. This is
especially likely when using more expensive compression settings.

This patch adds an API to allow for lowering the CPU priority of
background threads, modeled on the IO priority API. Benchmarks (see
below) show significant latency and throughput improvements when CPU
bound. As a result, workloads with some CPU usage bursts should benefit
from lower latencies at a given utilization, or should be able to push
utilization higher at a given request latency target.

A useful side effect is that compaction CPU usage is now easily visible
in common tools, allowing for an easier estimation of the contribution
of compaction vs. request processing threads.

As with IO priority, the implementation is limited to Linux, degrading
to a no-op on other systems.
Closes https://github.com/facebook/rocksdb/pull/3763

Differential Revision: D7740096

Pulled By: gwicke

fbshipit-source-id: e5d32373e8dc403a7b0c2227023f9ce4f22b413c
2018-04-24 08:41:51 -07:00
..
aligned_buffer.h Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
allocator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
arena_test.cc arena: derive alignment unit from std::max_align_t 2017-10-17 11:13:19 -07:00
arena.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
arena.h arena: derive alignment unit from std::max_align_t 2017-10-17 11:13:19 -07:00
auto_roll_logger_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
auto_roll_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
auto_roll_logger.h Fix the Logger::Close() and DBImpl::Close() design pattern 2018-02-23 13:57:26 -08:00
autovector_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
autovector.h Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
bloom.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
build_version.cc.in Makefile: generate util/build_version.cc from .in file (#1384) 2016-10-25 11:31:39 -07:00
build_version.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cast_util.h Add a missing "once" in .h 2017-07-31 12:12:03 -07:00
channel.h Change RocksDB License 2017-07-15 16:11:23 -07:00
coding_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
coding.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
coding.h Change RocksDB License 2017-07-15 16:11:23 -07:00
compaction_job_stats_impl.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
comparator.cc Move ~Comparator define to comparator.h 2017-10-17 09:58:13 -07:00
compression.h comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
concurrent_arena.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
concurrent_arena.h util: Fix coverity issues 2017-11-03 14:42:08 -07:00
core_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c_ppc_asm.S Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc_constants.h Make it explicit blob db doesn't support CF 2017-09-08 11:11:04 -07:00
crc32c_ppc.c Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_test.cc Port 3 way SSE4.2 crc32c implementation from Folly 2017-12-19 18:26:49 -08:00
crc32c.cc Suppress UBSAN error in finer guanularity 2018-02-13 12:18:07 -08:00
crc32c.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
delete_scheduler_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
delete_scheduler.cc SstFileManager: add bytes_max_delete_chunk 2018-03-22 15:58:37 -07:00
delete_scheduler.h SstFileManager: add bytes_max_delete_chunk 2018-03-22 15:58:37 -07:00
duplicate_detector.h Improve db_stress with transactions 2018-04-18 16:32:35 -07:00
dynamic_bloom_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
dynamic_bloom.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
dynamic_bloom.h Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
event_logger_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
fault_injection_test_env.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
fault_injection_test_env.h Fix FaultInjectionTestEnv to work with DirectIO 2018-03-14 00:57:24 -07:00
file_reader_writer_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
file_reader_writer.cc Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
file_reader_writer.h Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
file_util.cc check return status for Sync() and Append() calls to avoid corruption 2018-04-19 14:13:46 -07:00
file_util.h Support for Column family specific paths. 2018-04-05 19:58:20 -07:00
filelock_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
filename.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
filename.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filter_policy.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
gflags_compat.h fix gflags namespace 2017-12-01 10:42:05 -08:00
hash_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
hash.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
hash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
heap_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
heap.h Change RocksDB License 2017-07-15 16:11:23 -07:00
kv_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_write_bench.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
logging.h Change RocksDB License 2017-07-15 16:11:23 -07:00
memory_usage.h Change RocksDB License 2017-07-15 16:11:23 -07:00
murmurhash.cc Suppress UBSAN error in finer guanularity 2018-02-13 12:18:07 -08:00
murmurhash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mutexlock.h Change RocksDB License 2017-07-15 16:11:23 -07:00
ppc-opcode.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
random.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
random.h Change RocksDB License 2017-07-15 16:11:23 -07:00
rate_limiter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rate_limiter.cc rate limit auto-tuning 2017-10-04 19:15:01 -07:00
rate_limiter.h rate limit auto-tuning 2017-10-04 19:15:01 -07:00
set_comparator.h WritePrepared Txn: Move DuplicateDetector to util 2018-03-05 23:57:12 -08:00
slice_transform_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
slice.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
sst_file_manager_impl.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
sst_file_manager_impl.h Enable cancelling manual compactions if they hit the sfm size limit 2018-04-02 19:58:04 -07:00
status_message.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
status.cc make MockTimeEnv::current_time_ atomic to fix data race 2018-04-10 14:13:18 -07:00
stderr_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
stop_watch.h Change RocksDB License 2017-07-15 16:11:23 -07:00
string_util.cc Several small "fixes" 2018-02-15 16:57:37 -08:00
string_util.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sync_point_impl.cc Refactor sync_point to make implementation either customizable or replaceable 2018-03-23 12:56:52 -07:00
sync_point_impl.h Refactor sync_point to make implementation either customizable or replaceable 2018-03-23 12:56:52 -07:00
sync_point.cc Refactor sync_point to make implementation either customizable or replaceable 2018-03-23 12:56:52 -07:00
sync_point.h Refactor sync_point to make implementation either customizable or replaceable 2018-03-23 12:56:52 -07:00
testharness.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
testharness.h Change RocksDB License 2017-07-15 16:11:23 -07:00
testutil.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
testutil.h Comment out unused variables 2018-03-05 13:13:41 -08:00
thread_list_test.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
thread_local_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
thread_local.cc util: Fix coverity issues 2017-11-03 14:42:08 -07:00
thread_local.h Fix deadlock in ColumnFamilyData::InstallSuperVersion() 2018-02-16 08:13:34 -08:00
thread_operation.h Change RocksDB License 2017-07-15 16:11:23 -07:00
threadpool_imp.cc Support lowering CPU priority of background threads 2018-04-24 08:41:51 -07:00
threadpool_imp.h Support lowering CPU priority of background threads 2018-04-24 08:41:51 -07:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_queue.h Change RocksDB License 2017-07-15 16:11:23 -07:00
transaction_test_util.cc WritePrepared Txn: rollback via commit 2018-04-20 15:28:19 -07:00
transaction_test_util.h WritePrepared Txn: stress test 2017-12-06 09:42:28 -08:00
xxhash.cc fixed typo 2017-06-05 11:27:34 -07:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00