andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Anand Ananthabhotla b9d6eccac1 Lock free MultiGet (#4754) 
Summary:
Avoid locking the DB mutex in order to reference SuperVersions. Instead, we get the thread local cached SuperVersion for each column family in the list. It depends on finding a sequence number that overlaps with all the open memtables. We start with the latest published sequence number, and if any of the memtables is sealed before we can get all the SuperVersions, the process is repeated. After a few times, give up and lock the DB mutex.

Tests:
1. Unit tests
2. make check
3. db_bench -

TEST_TMPDIR=/dev/shm ./db_bench -use_existing_db=true -benchmarks=readrandom -write_buffer_size=4194304 -target_file_size_base=4194304 -max_bytes_for_level_base=16777216 -num=5000000 -reads=1000000 -threads=32 -compression_type=none -cache_size=1048576000 -batch_size=1 -bloom_bits=1
readrandom   :       0.167 micros/op 5983920 ops/sec;  426.2 MB/s (1000000 of 1000000 found)

Multireadrandom with batch size 1:
multireadrandom :       0.176 micros/op 5684033 ops/sec; (1000000 of 1000000 found)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4754

Differential Revision: D13363550

Pulled By: anand1976

fbshipit-source-id: 6243e8de7dbd9c8bb490a8eca385da0c855b1dd4
2019-01-02 11:42:54 -08:00
..
builder.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
builder.h
Add compaction logic to RangeDelAggregatorV2 (#4758)
2018-12-17 13:20:51 -08:00
c_test.c
Get CompactionJobInfo from CompactFiles
2018-12-13 14:21:24 -08:00
c.cc
Support setting options on column families via C bindings (#4785)
2018-12-17 13:52:12 -08:00
column_family_test.cc
Fix spelling errors (#4827)
2019-01-02 11:17:57 -08:00
column_family.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
column_family.h
Lock free MultiGet (#4754)
2019-01-02 11:42:54 -08:00
compact_files_test.cc
Get CompactionJobInfo from CompactFiles
2018-12-13 14:21:24 -08:00
compacted_db_impl.cc
move dump stats to a separate thread (#4382)
2018-10-08 22:54:43 -07:00
compacted_db_impl.h
Enable checkpoint of read-only db (#4681)
2018-12-07 17:06:02 -08:00
compaction_iteration_stats.h
add counter for deletion dropping optimization
2017-08-19 14:10:08 -07:00
compaction_iterator_test.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
compaction_iterator.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
compaction_iterator.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
compaction_job_stats_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
compaction_job_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
compaction_job.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
compaction_job.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
compaction_picker_fifo.cc
Move FIFOCompactionPicker to a separate file (#4724)
2018-11-29 16:04:52 -08:00
compaction_picker_fifo.h
Move FIFOCompactionPicker to a separate file (#4724)
2018-11-29 16:04:52 -08:00
compaction_picker_test.cc
Move FIFOCompactionPicker to a separate file (#4724)
2018-11-29 16:04:52 -08:00
compaction_picker_universal.cc
Remove random writes from SST file ingestion (#4172)
2018-07-27 16:12:23 -07:00
compaction_picker_universal.h
Delete triggered compaction for universal style
2018-05-29 15:44:34 -07:00
compaction_picker.cc
Move FIFOCompactionPicker to a separate file (#4724)
2018-11-29 16:04:52 -08:00
compaction_picker.h
Move FIFOCompactionPicker to a separate file (#4724)
2018-11-29 16:04:52 -08:00
compaction.cc
Truncate range tombstones by leveraging InternalKeys (#4432)
2018-10-09 15:19:38 -07:00
compaction.h
Truncate range tombstones by leveraging InternalKeys (#4432)
2018-10-09 15:19:38 -07:00
comparator_db_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
convenience.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
corruption_test.cc
fix DeleteRange memory leak for mmap and block cache (#4810)
2018-12-20 21:59:49 -08:00
cuckoo_table_db_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
db_basic_test.cc
Lock free MultiGet (#4754)
2019-01-02 11:42:54 -08:00
db_blob_index_test.cc
fix lite build
2017-10-17 08:57:09 -07:00
db_block_cache_test.cc
Fix spelling errors (#4827)
2019-01-02 11:17:57 -08:00
db_bloom_filter_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
db_compaction_filter_test.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
db_compaction_test.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
db_dynamic_level_test.cc
Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668)
2018-11-12 16:42:16 -08:00
db_encryption_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
db_filesnapshot.cc
Remove redundant member var and set options (#4631)
2018-11-12 12:24:26 -08:00
db_flush_test.cc
Improve flushing multiple column families (#4708)
2018-12-13 15:12:40 -08:00
db_impl_compaction_flush.cc
Remove an unused parameter (#4816)
2019-01-02 09:59:13 -08:00
db_impl_debug.cc
Enable atomic flush (#4023)
2018-10-26 15:08:43 -07:00
db_impl_experimental.cc
Update JobContext. (#3949)
2018-08-03 17:42:34 -07:00
db_impl_files.cc
Fix #3840: only SyncClosedLogs for multiple CFs (#4460)
2018-11-13 11:32:16 -08:00
db_impl_open.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
db_impl_readonly.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
db_impl_readonly.h
Get CompactionJobInfo from CompactFiles
2018-12-13 14:21:24 -08:00
db_impl_write.cc
Fix spelling errors (#4827)
2019-01-02 11:17:57 -08:00
db_impl.cc
Lock free MultiGet (#4754)
2019-01-02 11:42:54 -08:00
db_impl.h
Fix spelling errors (#4827)
2019-01-02 11:17:57 -08:00
db_info_dumper.cc
avoid copying when iterating using range-based for (#4459)
2018-10-09 17:15:51 -07:00
db_info_dumper.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_io_failure_test.cc
Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596)
2018-10-29 14:36:31 -07:00
db_iter_stress_test.cc
Move prefix_extractor to MutableCFOptions
2018-05-21 14:43:11 -07:00
db_iter_test.cc
Fix regression bug of Prev() with upper bound (#3989)
2018-06-12 16:57:36 -07:00
db_iter.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
db_iter.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
db_iterator_test.cc
Fix ticker stat for number files closed (#4703)
2018-11-21 18:31:34 -08:00
db_log_iter_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
db_memtable_test.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
db_merge_operator_test.cc
WriteUnPrepared Txn: Disable seek to snapshot optimization (#3955)
2018-06-27 12:23:07 -07:00
db_options_test.cc
move dump stats to a separate thread (#4382)
2018-10-08 22:54:43 -07:00
db_properties_test.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
db_range_del_test.cc
Fix range tombstones written to more files than necessary (#4592)
2018-10-29 19:23:27 -07:00
db_sst_test.cc
Simplify DBWithMaxSpaceAllowedRandomized (#4235)
2018-08-08 07:27:46 -07:00
db_statistics_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
db_table_properties_test.cc
Allow dynamic modification of window size and deletion trigger (#4403)
2018-09-20 15:15:28 -07:00
db_tailing_iter_test.cc
Remove managed iterator
2018-07-17 14:43:18 -07:00
db_test2.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
db_test_util.cc
Lock free MultiGet (#4754)
2019-01-02 11:42:54 -08:00
db_test_util.h
Lock free MultiGet (#4754)
2019-01-02 11:42:54 -08:00
db_test.cc
Get CompactionJobInfo from CompactFiles
2018-12-13 14:21:24 -08:00
db_universal_compaction_test.cc
Adapt three unit tests with newer compiler/libraries (#4562)
2018-10-24 08:17:56 -07:00
db_wal_test.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
db_write_test.cc
Suppress tsan lock-order-inversion on FlushWAL
2018-05-14 21:13:35 -07:00
dbformat_test.cc
Relax VersionStorageInfo::GetOverlappingInputs check (#4050)
2018-07-13 17:42:38 -07:00
dbformat.cc
types: add kEntryBlobIndex for TablePropertiesCollector (#4233)
2018-08-06 18:27:44 -07:00
dbformat.h
s/CacheAllocator/MemoryAllocator/g (#4590)
2018-10-26 14:30:30 -07:00
deletefile_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
error_handler_test.cc
Fix regression test failures introduced by PR #4164 (#4375)
2018-09-17 13:14:07 -07:00
error_handler.cc
Fix typos in comments (#4456)
2018-10-04 20:46:50 -07:00
error_handler.h
Fix typos in comments (#4456)
2018-10-04 20:46:50 -07:00
event_helpers.cc
Auto recovery from out of space errors (#4164)
2018-09-15 13:43:04 -07:00
event_helpers.h
Auto recovery from out of space errors (#4164)
2018-09-15 13:43:04 -07:00
experimental.cc
comment unused parameters to turn on -Wunused-parameter flag
2018-04-12 17:59:16 -07:00
external_sst_file_basic_test.cc
Change directory where ExternalSSTFileBasicTest runs (#4766)
2018-12-11 10:21:37 -08:00
external_sst_file_ingestion_job.cc
Fix typos in comments (#4456)
2018-10-04 20:46:50 -07:00
external_sst_file_ingestion_job.h
Remove random writes from SST file ingestion (#4172)
2018-07-27 16:12:23 -07:00
external_sst_file_test.cc
Try to fix ExternalSSTFileTest.IngestNonExistingFile flakines (#4625)
2018-11-02 17:26:35 -07:00
fault_injection_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
file_indexer_test.cc
Comment out unused variables
2018-03-05 13:13:41 -08:00
file_indexer.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
file_indexer.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
filename_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
flush_job_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
flush_job.cc
Add compaction logic to RangeDelAggregatorV2 (#4758)
2018-12-17 13:20:51 -08:00
flush_job.h
Add support to flush multiple CFs atomically (#4262)
2018-10-15 20:01:17 -07:00
flush_scheduler.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
flush_scheduler.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
forward_iterator_bench.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
forward_iterator.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
forward_iterator.h
Comment out unused variables
2018-03-05 13:13:41 -08:00
internal_stats.cc
Add DB property for SST files kept from deletion (#4618)
2018-11-05 20:24:40 -08:00
internal_stats.h
Add DB property for SST files kept from deletion (#4618)
2018-11-05 20:24:40 -08:00
job_context.h
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
listener_test.cc
Add listener to sample file io (#3933)
2018-10-12 18:36:11 -07:00
log_format.h
Fix an inaccurate comment (#4315)
2018-08-24 18:13:20 -07:00
log_reader.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
log_reader.h
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
log_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
log_writer.cc
Pass manual_wal_flush also to the first wal file
2018-05-14 10:57:56 -07:00
log_writer.h
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
logs_with_prep_tracker.cc
Skip deleted WALs during recovery
2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h
Skip deleted WALs during recovery
2018-05-03 15:43:09 -07:00
malloc_stats.cc
Fix compile error with jemalloc (#4488)
2018-10-12 11:50:50 -07:00
malloc_stats.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
manual_compaction_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
memtable_list_test.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
memtable_list.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
memtable_list.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
memtable.cc
Change is_range_del_table_empty_ flag to atomic (#4801)
2018-12-19 17:21:14 -08:00
memtable.h
Change is_range_del_table_empty_ flag to atomic (#4801)
2018-12-19 17:21:14 -08:00
merge_context.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
merge_helper_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_helper.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
merge_helper.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
merge_operator.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
merge_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
obsolete_files_test.cc
Modify verification logic of ObsoleteOptionsFileTest (#4218)
2018-08-03 13:57:40 -07:00
options_file_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
perf_context_test.cc
Add a new per level counter for block cache hit (#4796)
2018-12-21 13:20:05 -08:00
pinned_iterators_manager.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
plain_table_db_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
pre_release_callback.h
Fix pre_release callback argument list.
2018-04-05 11:12:16 -07:00
prefix_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
range_del_aggregator_bench.cc
Fix Windows broken build error due to non-const override (#4798)
2018-12-19 13:29:51 -08:00
range_del_aggregator_test.cc
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
range_del_aggregator.cc
Remove stale TODO (#4800)
2018-12-19 15:45:37 -08:00
range_del_aggregator.h
Fix unused member compile error
2018-12-18 14:28:42 -08:00
range_tombstone_fragmenter_test.cc
Prepare FragmentedRangeTombstoneIterator for use in compaction (#4740)
2018-12-11 12:10:48 -08:00
range_tombstone_fragmenter.cc
Add compaction logic to RangeDelAggregatorV2 (#4758)
2018-12-17 13:20:51 -08:00
range_tombstone_fragmenter.h
Add compaction logic to RangeDelAggregatorV2 (#4758)
2018-12-17 13:20:51 -08:00
read_callback.h
WriteUnPrepared Txn: Disable seek to snapshot optimization (#3955)
2018-06-27 12:23:07 -07:00
repair_test.cc
Acquire lock on DB LOCK file before starting repair. (#4435)
2018-10-12 10:41:54 -07:00
repair.cc
Add compaction logic to RangeDelAggregatorV2 (#4758)
2018-12-17 13:20:51 -08:00
snapshot_checker.h
Add path to WritableFileWriter. (#4039)
2018-08-23 10:12:58 -07:00
snapshot_impl.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
snapshot_impl.h
BlobDB: Fix VisibleToActiveSnapshot() (#4236)
2018-08-06 16:57:42 -07:00
table_cache.cc
Introduce a CPU time counter in perf_context (#4741)
2018-12-20 12:03:44 -08:00
table_cache.h
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
table_properties_collector_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
table_properties_collector.cc
Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)
2018-10-30 15:34:27 -07:00
table_properties_collector.h
Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)
2018-10-30 15:34:27 -07:00
transaction_log_impl.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
transaction_log_impl.h
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
version_builder_test.cc
Remove random writes from SST file ingestion (#4172)
2018-07-27 16:12:23 -07:00
version_builder.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
version_builder.h
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
version_edit_test.cc
Update recovery code for version edits group commit. (#3945)
2018-08-20 14:58:00 -07:00
version_edit.cc
Fix spelling errors (#4827)
2019-01-02 11:17:57 -08:00
version_edit.h
Update recovery code for version edits group commit. (#3945)
2018-08-20 14:58:00 -07:00
version_set_test.cc
Improve flushing multiple column families (#4708)
2018-12-13 15:12:40 -08:00
version_set.cc
Preload some files even if options.max_open_files (#3340)
2018-12-28 18:02:28 -08:00
version_set.h
Remove v1 RangeDelAggregator (#4778)
2018-12-17 17:33:46 -08:00
wal_manager_test.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
wal_manager.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
wal_manager.h
Fix memleak when DB::DeleteFile()
2018-01-11 18:57:33 -08:00
write_batch_base.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_batch_internal.h
WriteUnPrepared: Add new WAL marker kTypeBeginUnprepareXID (#4069)
2018-06-28 18:58:29 -07:00
write_batch_test.cc
Clean up FragmentedRangeTombstoneList (#4692)
2018-11-28 15:29:02 -08:00
write_batch.cc
Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
2018-11-09 11:19:58 -08:00
write_callback_test.cc
Per-thread unique test db names (#4135)
2018-07-13 17:27:39 -07:00
write_callback.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller_test.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller.cc
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_controller.h
Change RocksDB License
2017-07-15 16:11:23 -07:00
write_thread.cc
Add more sync point to fix flaky test GroupCommitTest
2018-11-07 14:07:53 -08:00
write_thread.h
Handle mixed slowdown/no_slowdown writer properly (#4475)
2018-10-09 22:52:40 -07:00