andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
afbfb3f567
rocksdb/db
History
Levi Tamasi afbfb3f567 Do not create/install new SuperVersion if nothing was deleted during memtable trim (#6169) 
Summary:
We have observed an increase in CPU load caused by frequent calls to
`ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory`
when using `max_write_buffer_size_to_maintain` to limit the amount of
memtable history maintained for transaction conflict checking. As it turns out,
this is caused by the code creating and installing a new `SuperVersion` even if
no memtables were actually trimmed. The patch adds a check to avoid this.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6169

Test Plan:
Compared `perf` output for

```
./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32
```

before and after the change. With the fix, the call chain `rocksdb::DBImpl::TrimMemtableHistory` ->
`rocksdb::ColumnFamilyData::InstallSuperVersion` -> `rocksdb::ThreadLocalPtr::StaticMeta::Scrape`
no longer registers in the `perf` report.

Differential Revision: D19031509

Pulled By: ltamasi

fbshipit-source-id: 02686fce594e5b50eba0710e4b28a9b808c8aa20
2020-01-10 09:53:31 -08:00
..
compaction upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_impl Do not create/install new SuperVersion if nothing was deleted during memtable trim (#6169) 2020-01-10 09:53:31 -08:00
builder.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
builder.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
c_test.c Add C binding for secondary instance (#5505) 2019-06-27 08:58:54 -07:00
c.cc Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
column_family_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
column_family.h Bump up memory order of ref counting of ColumnFamilyData (#5723) 2019-08-20 10:34:33 -07:00
compact_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Do readahead in VerifyChecksum() (#5713) 2019-08-16 16:42:56 -07:00
corruption_test.cc Fix VerifyChecksum readahead with mmap mode (#5945) 2019-10-22 11:42:36 -07:00
cuckoo_table_db_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
db_basic_test.cc Fix a buffer overrun problem in BlockBasedTable::MultiGet (#6014) 2019-11-12 10:57:32 -08:00
db_blob_index_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_block_cache_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_bloom_filter_test.cc Unordered Writes (#5218) 2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
db_flush_test.cc Ref and unref cfd before and after calling WaitForFlushMemTables (#5513) 2019-07-01 14:12:02 -07:00
db_info_dumper.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter.cc Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
db_iter.h Make format 2019-05-31 15:24:43 -07:00
db_iterator_test.cc Fix data block upper bound checking for iterator reseek case (#5883) 2019-10-07 16:39:18 -07:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_options_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_properties_test.cc Refactor some confusing logic in PlainTableReader 2019-09-13 10:26:36 -07:00
db_range_del_test.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test2.cc Support loading custom objects in unit tests (#5676) 2019-08-09 15:12:08 -07:00
db_test_util.cc exclude TEST_ENV_URI from rocksdb lite (#5686) 2019-08-10 19:15:05 -07:00
db_test_util.h Support loading custom objects in unit tests (#5676) 2019-08-09 15:12:08 -07:00
db_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_universal_compaction_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_wal_test.cc Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
db_write_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
dbformat.cc Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
deletefile_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
error_handler_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
event_helpers.cc Fix wrong info log printing for num_range_deletions (#5617) 2019-07-23 19:38:16 -07:00
event_helpers.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Introduce IngestExternalFileOptions.verify_checksums_readahead_size (#5721) 2019-08-20 10:43:39 -07:00
external_sst_file_ingestion_job.cc Fix a bug in file ingestion (#5760) 2019-08-30 18:29:07 -07:00
external_sst_file_ingestion_job.h Fix IngestExternalFile overlapping check (#5649) 2019-08-14 21:02:28 -07:00
external_sst_file_test.cc Fix a bug in file ingestion (#5760) 2019-08-30 18:29:07 -07:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job_test.cc Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
flush_job.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
flush_job.h Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
forward_iterator.cc Add more callers for table reader. (#5454) 2019-06-20 14:31:48 -07:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
import_column_family_job.cc Fix IngestExternalFile overlapping check (#5649) 2019-08-14 21:02:28 -07:00
import_column_family_job.h Export Import sst files (#5495) 2019-07-17 12:27:14 -07:00
import_column_family_test.cc Fix tsan and valgrind failures in import_column_family_test 2019-07-19 13:25:36 -07:00
internal_stats.cc Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
internal_stats.h Rename InternalDBStatsType enum names (#5779) 2019-09-06 17:31:10 -07:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Support for single-primary, multi-secondary instances (#4899) 2019-03-26 16:45:31 -07:00
log_reader.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
log_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
log_writer.cc LogWriter to only flush after finish generating whole record (#5328) 2019-05-21 12:33:17 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable_list_test.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
memtable_list.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
memtable_list.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
memtable.cc Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
memtable.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc Make format 2019-05-31 15:24:43 -07:00
obsolete_files_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Fix Appveyor build due to signed/unsigned comparison 2019-09-10 14:34:37 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_del_aggregator.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.h Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
repair.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
table_cache.cc Support row cache with batched MultiGet (#5706) 2019-08-28 16:11:56 -07:00
table_cache.h Support row cache with batched MultiGet (#5706) 2019-08-28 16:11:56 -07:00
table_properties_collector_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
transaction_log_impl.cc reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
version_builder.cc Lower the risk for users to run options.force_consistency_checks = true (#5744) 2019-08-29 14:07:37 -07:00
version_builder.h Lower the risk for users to run options.force_consistency_checks = true (#5744) 2019-08-29 14:07:37 -07:00
version_edit_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
version_edit.cc Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
version_edit.h Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
version_set_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
version_set.cc Revert "Merging iterator to avoid child iterator reseek for some cases (#5286)" (#5871) 2019-10-01 14:41:58 -07:00
version_set.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
wal_manager_test.cc Replace Corruption with TryAgain status when new tail is not visible to TransactionLogIterator (#5474) 2019-06-19 08:10:08 -07:00
wal_manager.cc do a bit less work in the normal case (#5695) 2019-09-10 09:41:45 -07:00
wal_manager.h Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765) 2019-09-04 12:10:17 -07:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch.cc Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_thread.cc Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00