andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Levi Tamasi bd8404feff Do not schedule memtable trimming if there is no history (#6177) 
Summary:
We have observed an increase in CPU load caused by frequent calls to
`ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory`
when using `max_write_buffer_size_to_maintain` to limit the amount of
memtable history maintained for transaction conflict checking. Part of the issue
is that trimming can potentially be scheduled even if there is no memtable
history. The patch adds a check that fixes this.

See also https://github.com/facebook/rocksdb/pull/6169.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6177

Test Plan:
Compared `perf` output for

```
./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32
```

before and after the change. There is a significant reduction for the call chain
`rocksdb::DBImpl::TrimMemtableHistory` -> `rocksdb::ColumnFamilyData::InstallSuperVersion` ->
`rocksdb::ThreadLocalPtr::StaticMeta::Scrape` even without https://github.com/facebook/rocksdb/pull/6169.

Differential Revision: D19057445

Pulled By: ltamasi

fbshipit-source-id: dff81882d7b280e17eda7d9b072a2d4882c50f79
2019-12-13 19:11:19 -08:00
..
compaction
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
db_impl
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
arena_wrapped_db_iter.cc
Apply formatter on recent 45 commits. (#5827)
2019-09-19 12:34:17 -07:00
arena_wrapped_db_iter.h
Refactor ArenaWrappedDBIter into separate files (#5801)
2019-09-13 13:50:43 -07:00
blob_index.h
Mark BlobIndex::DebugString const
2019-12-11 17:19:43 -08:00
builder.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
builder.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
c_test.c
Fix c_test:filter for various CACHE_LINE_SIZEs (#6153)
2019-12-11 15:17:08 -08:00
c.cc
Apply formatter to some recent commits (#6138)
2019-12-09 15:49:49 -08:00
column_family_test.cc
Refactor trimming logic for immutable memtables (#5022)
2019-08-23 13:55:34 -07:00
column_family.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
column_family.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
compact_files_test.cc
Use aggregate initialization for FlushJobInfo/CompactionJobInfo (#5997)
2019-11-01 11:46:19 -07:00
compacted_db_impl.cc
New API to get all merge operands for a Key (#5604)
2019-08-06 14:26:44 -07:00
compacted_db_impl.h
Use delete to disable automatic generated methods. (#5009)
2019-09-11 18:09:00 -07:00
comparator_db_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
convenience.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
corruption_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
cuckoo_table_db_test.cc
Fix memory leak on error opening PlainTable (#5951)
2019-10-21 16:53:06 -07:00
db_basic_test.cc
Apply formatter to some recent commits (#6138)
2019-12-09 15:49:49 -08:00
db_blob_index_test.cc
Disable blob iterator test with max_sequential_skip_in_iterations==0 in LITE mode (#6052)
2019-11-19 15:02:41 -08:00
db_block_cache_test.cc
Charge block cache for cache internal usage (#5797)
2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc
Expose and elaborate FilterBuildingContext (#6088)
2019-11-26 18:24:10 -08:00
db_compaction_filter_test.cc
upgrade gtest 1.7.0 => 1.8.1 for json result writing
2019-09-09 11:24:11 -07:00
db_compaction_test.cc
wait pending memtable writes on file ingestion or compact range (#6113)
2019-12-12 14:08:02 -08:00
db_dynamic_level_test.cc
Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668)
2018-11-12 16:42:16 -08:00
db_encryption_test.cc
Fix EncryptedEnv assert (#5735)
2019-09-05 17:21:42 -07:00
db_filesnapshot.cc
Support concurrent CF iteration and drop (#6147)
2019-12-12 19:04:48 -08:00
db_flush_test.cc
Fix a potential bug scheduling unnecessary threads (#6104)
2019-11-27 14:48:49 -08:00
db_info_dumper.cc
Apply formatter to recent 200+ commits. (#5830)
2019-09-20 12:04:26 -07:00
db_info_dumper.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_io_failure_test.cc
Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596)
2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
db_iter_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
db_iter.cc
Fix blob context when db_iter uses seek (#6051)
2019-11-19 11:39:02 -08:00
db_iter.h
Apply formatter on recent 45 commits. (#5827)
2019-09-19 12:34:17 -07:00
db_iterator_test.cc
Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154)
2019-12-13 14:30:14 -08:00
db_log_iter_test.cc
Apply modernize-use-override (2nd iteration)
2019-02-14 14:41:36 -08:00
db_memtable_test.cc
upgrade gtest 1.7.0 => 1.8.1 for json result writing
2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc
New API to get all merge operands for a Key (#5604)
2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc
New API to get all merge operands for a Key (#5604)
2019-08-06 14:26:44 -07:00
db_options_test.cc
Make default value of options.ttl to be 30 days when it is supported. (#6073)
2019-11-26 10:00:32 -08:00
db_properties_test.cc
Refactor/consolidate legacy Bloom implementation details (#5784)
2019-09-16 16:17:09 -07:00
db_range_del_test.cc
Fix RangeDeletion bug (#6062)
2019-12-12 15:18:02 -08:00
db_sst_test.cc
Fix bugs in WAL trash file handling (#5520)
2019-07-06 21:07:32 -07:00
db_statistics_test.cc
Make statistics's stats_level change thread-safe (#5030)
2019-03-01 10:42:09 -08:00
db_table_properties_test.cc
upgrade gtest 1.7.0 => 1.8.1 for json result writing
2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc
Remove managed iterator
2018-07-17 14:43:18 -07:00
db_test2.cc
Fix a regression bug on total order seek with prefix enabled and range delete (#6028)
2019-11-13 10:11:34 -08:00
db_test_util.cc
Batched MultiGet API for multiple column families (#5816)
2019-11-12 13:52:55 -08:00
db_test_util.h
Expose and elaborate FilterBuildingContext (#6088)
2019-11-26 18:24:10 -08:00
db_test.cc
Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)
2019-11-26 21:39:56 -08:00
db_universal_compaction_test.cc
Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125)
2019-12-09 09:23:50 -08:00
db_wal_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
db_write_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
dbformat_test.cc
Move some logging related files to logging/ (#5387)
2019-05-31 17:23:59 -07:00
dbformat.cc
Apply formatter to recent 200+ commits. (#5830)
2019-09-20 12:04:26 -07:00
dbformat.h
Use delete to disable automatic generated methods. (#5009)
2019-09-11 18:09:00 -07:00
deletefile_test.cc
Refactor deletefile_test.cc (#5822)
2019-09-18 16:58:21 -07:00
error_handler_test.cc
wait pending memtable writes on file ingestion or compact range (#6113)
2019-12-12 14:08:02 -08:00
error_handler.cc
Make format
2019-05-31 15:24:43 -07:00
error_handler.h
Fix typos in comments (#4456)
2018-10-04 20:46:50 -07:00
event_helpers.cc
BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903)
2019-10-14 15:21:01 -07:00
event_helpers.h
BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903)
2019-10-14 15:21:01 -07:00
experimental.cc
Organizing rocksdb/db directory
2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc
Allow ingesting overlapping files (#5539)
2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
external_sst_file_ingestion_job.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
external_sst_file_test.cc
wait pending memtable writes on file ingestion or compact range (#6113)
2019-12-12 14:08:02 -08:00
fault_injection_test.cc
Move some logging related files to logging/ (#5387)
2019-05-31 17:23:59 -07:00
file_indexer_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
file_indexer.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
file_indexer.h
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
filename_test.cc
Move some logging related files to logging/ (#5387)
2019-05-31 17:23:59 -07:00
flush_job_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
flush_job.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
flush_job.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
flush_scheduler.cc
Support concurrent CF iteration and drop (#6147)
2019-12-12 19:04:48 -08:00
flush_scheduler.h
Refactor trimming logic for immutable memtables (#5022)
2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc
simplify include directive involving inttypes (#5402)
2019-06-06 13:56:07 -07:00
forward_iterator.cc
Add more callers for table reader. (#5454)
2019-06-20 14:31:48 -07:00
forward_iterator.h
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
import_column_family_job.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
import_column_family_job.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
import_column_family_test.cc
Apply formatter to recent 200+ commits. (#5830)
2019-09-20 12:04:26 -07:00
internal_stats.cc
Apply formatter to recent 200+ commits. (#5830)
2019-09-20 12:04:26 -07:00
internal_stats.h
Rename InternalDBStatsType enum names (#5779)
2019-09-06 17:31:10 -07:00
job_context.h
WritePrepared: Fix visible key compacted out by compaction (#4883)
2019-01-15 21:34:38 -08:00
listener_test.cc
Propagate SST and blob file numbers through the EventListener interface (#5962)
2019-10-24 14:44:15 -07:00
log_format.h
Fix an inaccurate comment (#4315)
2018-08-24 18:13:20 -07:00
log_reader.cc
Divide file_reader_writer.h and .cc (#5803)
2019-09-16 10:33:51 -07:00
log_reader.h
Divide file_reader_writer.h and .cc (#5803)
2019-09-16 10:33:51 -07:00
log_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
log_writer.cc
Divide file_reader_writer.h and .cc (#5803)
2019-09-16 10:33:51 -07:00
log_writer.h
Use delete to disable automatic generated methods. (#5009)
2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc
Skip deleted WALs during recovery
2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h
Skip deleted WALs during recovery
2018-05-03 15:43:09 -07:00
lookup_key.h
Avoid user key copying for Get/Put/Write with user-timestamp (#5502)
2019-07-25 15:27:39 -07:00
malloc_stats.cc
Support jemalloc compiled with --with-jemalloc-prefix (#5521)
2019-07-02 12:07:01 -07:00
malloc_stats.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
manual_compaction_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
memtable_list_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
memtable_list.cc
Do not schedule memtable trimming if there is no history (#6177)
2019-12-13 19:11:19 -08:00
memtable_list.h
Do not schedule memtable trimming if there is no history (#6177)
2019-12-13 19:11:19 -08:00
memtable.cc
Misc hashing updates / upgrades (#5909)
2019-10-24 17:16:46 -07:00
memtable.h
Do not schedule memtable trimming if there is no history (#6177)
2019-12-13 19:11:19 -08:00
merge_context.h
Introduce a new MultiGet batching implementation (#5011)
2019-04-11 14:28:26 -07:00
merge_helper_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
merge_helper.cc
Fix merging range tombstone covering put during flush/compaction (#5406)
2019-06-04 10:24:14 -07:00
merge_helper.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
merge_operator.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_test.cc
Make format
2019-05-31 15:24:43 -07:00
obsolete_files_test.cc
Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820)
2019-09-18 11:52:17 -07:00
options_file_test.cc
Organizing rocksdb/db directory
2019-05-31 11:57:01 -07:00
perf_context_test.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
plain_table_db_test.cc
Fix memory leak on error opening PlainTable (#5951)
2019-10-21 16:53:06 -07:00
pre_release_callback.h
WritePrepared: reduce prepared_mutex_ overhead (#5420)
2019-06-10 11:53:31 -07:00
prefix_test.cc
Organizing rocksdb/db directory
2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc
Move some memory related files from util/ to memory/ (#5382)
2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc
Move test related files under util/ to test_util/ (#5377)
2019-05-30 11:25:51 -07:00
range_del_aggregator.cc
Organizing rocksdb/db directory
2019-05-31 11:57:01 -07:00
range_del_aggregator.h
Organizing rocksdb/db directory
2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc
Move test related files under util/ to test_util/ (#5377)
2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc
Apply formatter to recent 200+ commits. (#5830)
2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h
Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720)
2019-09-05 17:30:29 -07:00
read_callback.h
WriteUnPrepared: improve read your own write functionality (#5573)
2019-07-23 08:08:19 -07:00
repair_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
repair.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
snapshot_checker.h
WritePrepared: fix issue with snapshot released during compaction (#4858)
2019-01-16 09:55:32 -08:00
snapshot_impl.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
snapshot_impl.h
Refresh snapshot list during long compactions (2nd attempt) (#5278)
2019-05-03 17:30:22 -07:00
table_cache.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
table_cache.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
table_properties_collector_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
table_properties_collector.cc
Feature for sampling and reporting compressibility (#4842)
2019-03-18 12:15:34 -07:00
table_properties_collector.h
Feature for sampling and reporting compressibility (#4842)
2019-03-18 12:15:34 -07:00
transaction_log_impl.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
transaction_log_impl.h
reuse scratch buffer in transaction_log_reader (#5702)
2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc
Support concurrent CF iteration and drop (#6147)
2019-12-12 19:04:48 -08:00
trim_history_scheduler.h
Refactor trimming logic for immutable memtables (#5022)
2019-08-23 13:55:34 -07:00
version_builder_test.cc
Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)
2019-11-26 21:39:56 -08:00
version_builder.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
version_builder.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
version_edit_test.cc
Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)
2019-11-26 21:39:56 -08:00
version_edit.cc
Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)
2019-11-26 21:39:56 -08:00
version_edit.h
Support options.max_open_files = -1 with periodic_compaction_seconds (#6090)
2019-11-26 21:39:56 -08:00
version_set_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
version_set.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
version_set.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
wal_manager_test.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
wal_manager.cc
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
wal_manager.h
Introduce a new storage specific Env API (#5761)
2019-12-13 14:48:41 -08:00
write_batch_base.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_batch_internal.h
Add insert hints for each writebatch (#5728)
2019-09-12 17:15:18 -07:00
write_batch_test.cc
upgrade gtest 1.7.0 => 1.8.1 for json result writing
2019-09-09 11:24:11 -07:00
write_batch.cc
Do not schedule memtable trimming if there is no history (#6177)
2019-12-13 19:11:19 -08:00
write_callback_test.cc
WritePrepared: reduce prepared_mutex_ overhead (#5420)
2019-06-10 11:53:31 -07:00
write_callback.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller_test.cc
Move test related files under util/ to test_util/ (#5377)
2019-05-30 11:25:51 -07:00
write_controller.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_thread.cc
Option to make write group size configurable (#5759)
2019-09-11 18:28:33 -07:00
write_thread.h
Option to make write group size configurable (#5759)
2019-09-11 18:28:33 -07:00