andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
rocksdb/db
History
Mike Kolupaev 218487d8dc [wal changes 1/3] fixed unbounded wal growth in some workloads 
Summary:
This fixes the following scenario we've hit:
 - we reached max_total_wal_size, created a new wal and scheduled flushing all memtables corresponding to the old one,
 - before the last of these flushes started its column family was dropped; the last background flush call was a no-op; no one removed the old wal from alive_logs_,
 - hours have passed and no flushes happened even though lots of data was written; data is written to different column families, compactions are disabled; old column families are dropped before memtable grows big enough to trigger a flush; the old wal still sits in alive_logs_ preventing max_total_wal_size limit from kicking in,
 - a few more hours pass and we run out disk space because of one huge .log file.

Test Plan: `make check`; backported the new test, checked that it fails without this diff

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D40893
2015-07-02 14:27:00 -07:00
..
builder.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
builder.h
Add more table properties to EventLogger
2015-05-12 15:53:55 -07:00
c_test.c
Block c_test in ROCKSDB_LITE
2015-06-17 10:54:51 -07:00
c.cc
Use CompactRangeOptions for CompactRange
2015-06-17 14:36:14 -07:00
column_family_test.cc
Make column_family_test runnable in ROCKSDB_LITE
2015-06-29 14:39:01 -07:00
column_family.cc
Fail DB::Open() when the requested compression is not available
2015-06-18 14:55:05 -07:00
column_family.h
Fail DB::Open() when the requested compression is not available
2015-06-18 14:55:05 -07:00
compact_files_test.cc
Make "make all" work for CYGWIN
2015-06-09 16:36:07 -07:00
compaction_job_stats_test.cc
Skip bottommost level compaction if possible
2015-06-18 11:03:31 -07:00
compaction_job_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
compaction_job.cc
Fixed a bug of CompactionStats in multi-level universal compaction case
2015-06-17 23:40:34 -07:00
compaction_job.h
Fixed a bug of CompactionStats in multi-level universal compaction case
2015-06-17 23:40:34 -07:00
compaction_picker_test.cc
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
compaction_picker.cc
Don't let two L0->L1 compactions run in parallel
2015-06-11 15:42:16 -07:00
compaction_picker.h
Optimize GetRange Function
2015-05-05 09:57:47 -07:00
compaction.cc
Use nullptr for default compaction_filter_factory
2015-06-08 16:34:26 -07:00
compaction.h
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
comparator_db_test.cc
Make "make all" work for CYGWIN
2015-06-09 16:36:07 -07:00
corruption_test.cc
fix typos
2015-04-25 18:14:27 +09:00
cuckoo_table_db_test.cc
Allowing L0 -> L1 trivial move on sorted data
2015-06-04 16:51:25 -07:00
db_bench.cc
Implement a table-level row cache
2015-06-23 10:25:45 -07:00
db_filesnapshot.cc
Add wal files to Checkpoint for multiple column families.
2015-06-19 16:08:31 -07:00
db_impl_debug.cc
[wal changes 1/3] fixed unbounded wal growth in some workloads
2015-07-02 14:27:00 -07:00
db_impl_experimental.cc
Clean up InstallSuperVersion
2015-06-17 12:37:59 -07:00
db_impl_readonly.cc
Move GetThreadList() feature under Env.
2014-12-22 12:20:17 -08:00
db_impl_readonly.h
Use CompactRangeOptions for CompactRange
2015-06-17 14:36:14 -07:00
db_impl.cc
[wal changes 1/3] fixed unbounded wal growth in some workloads
2015-07-02 14:27:00 -07:00
db_impl.h
[wal changes 1/3] fixed unbounded wal growth in some workloads
2015-07-02 14:27:00 -07:00
db_iter_test.cc
Fix a comparison in DBIter::FindPrevUserKey()
2015-06-29 17:04:03 -07:00
db_iter.cc
Fix a comparison in DBIter::FindPrevUserKey()
2015-06-29 17:04:03 -07:00
db_iter.h
reduce references to cfd->options() in DBImpl
2014-09-08 15:04:34 -07:00
db_test.cc
[wal changes 1/3] fixed unbounded wal growth in some workloads
2015-07-02 14:27:00 -07:00
dbformat_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
dbformat.cc
Replace %llu with format macros in ParsedInternalKey::DebugString())
2015-06-17 20:44:26 -07:00
dbformat.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
deletefile_test.cc
Use CompactRangeOptions for CompactRange
2015-06-17 14:36:14 -07:00
event_helpers.cc
Add EventListener::OnTableFileDeletion()
2015-06-03 19:57:01 -07:00
event_helpers.h
Add EventListener::OnTableFileDeletion()
2015-06-03 19:57:01 -07:00
experimental.cc
Implement DB::PromoteL0 method
2015-04-23 12:10:36 -07:00
fault_injection_test.cc
Use CompactRangeOptions for CompactRange
2015-06-17 14:36:14 -07:00
file_indexer_test.cc
Fix possible SIGSEGV in CompactRange (github issue #596)
2015-04-29 10:52:31 -07:00
file_indexer.cc
Fix possible SIGSEGV in CompactRange (github issue #596)
2015-04-29 10:52:31 -07:00
file_indexer.h
Turn on -Wshorten-64-to-32 and fix all the errors
2014-11-11 16:47:22 -05:00
filename_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
filename.cc
Sync manifest file when initializing it
2015-01-22 14:32:03 -08:00
filename.h
Sync manifest file when initializing it
2015-01-22 14:32:03 -08:00
flush_job_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
flush_job.cc
Add largest sequence to FlushJobInfo
2015-06-11 15:22:22 -07:00
flush_job.h
Add largest sequence to FlushJobInfo
2015-06-11 15:22:22 -07:00
flush_scheduler.cc
Don't return (or dereference) dangling pointer
2014-10-02 14:33:16 -07:00
flush_scheduler.h
Fix data race #1
2015-01-26 11:48:07 -08:00
forward_iterator.cc
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
forward_iterator.h
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
internal_stats.cc
Fixed a bug of CompactionStats in multi-level universal compaction case
2015-06-17 23:40:34 -07:00
internal_stats.h
Fixed a bug of CompactionStats in multi-level universal compaction case
2015-06-17 23:40:34 -07:00
job_context.h
Clean up old log files in background threads
2015-03-30 15:04:10 -04:00
listener_test.cc
Use CompactRangeOptions for CompactRange
2015-06-17 14:36:14 -07:00
log_format.h
Some minor refactoring on the code
2014-01-02 16:32:31 -08:00
log_reader.cc
Introduce WAL recovery consistency levels
2015-06-22 15:28:12 -07:00
log_reader.h
Introduce WAL recovery consistency levels
2015-06-22 15:28:12 -07:00
log_test.cc
Introduce WAL recovery consistency levels
2015-06-22 15:28:12 -07:00
log_writer.cc
Fix comparison between signed and usigned integers
2015-05-19 10:59:30 -07:00
log_writer.h
Log writer record format doc.
2015-04-07 16:25:56 -07:00
managed_iterator.cc
Fix compile error on MacOS.
2015-02-24 16:24:53 -08:00
managed_iterator.h
Fixed xfunc related compile errors in ROCKSDB_LITE
2015-04-09 21:05:18 -07:00
memtable_allocator.cc
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
memtable_allocator.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00
memtable_list_test.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
memtable_list.cc
Allow GetApproximateSize() to include mem table size if it is skip list memtable
2015-06-16 18:13:23 -07:00
memtable_list.h
Allow GetApproximateSize() to include mem table size if it is skip list memtable
2015-06-16 18:13:23 -07:00
memtable.cc
Allow GetApproximateSize() to include mem table size if it is skip list memtable
2015-06-16 18:13:23 -07:00
memtable.h
Allow GetApproximateSize() to include mem table size if it is skip list memtable
2015-06-16 18:13:23 -07:00
memtablerep_bench.cc
build: avoid unused-variable warning
2015-05-02 13:19:10 -07:00
merge_context.h
API to fetch from both a WriteBatchWithIndex and the db
2015-05-11 14:51:51 -07:00
merge_helper.cc
Call merge operators with empty values
2015-06-26 11:35:46 -07:00
merge_helper.h
Helper function to time Merges
2015-04-27 20:23:50 -07:00
merge_operator.cc
Call merge operators with empty values
2015-06-26 11:35:46 -07:00
merge_test.cc
Call merge operators with empty values
2015-06-26 11:35:46 -07:00
perf_context_test.cc
Makefile minor cleanup
2015-03-30 16:05:35 -04:00
plain_table_db_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
prefix_test.cc
rocksdb: Remove #include "util/string_util.h" from util/testharness.h
2015-03-19 17:29:37 -07:00
repair.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
skiplist_test.cc
rocksdb: switch to gtest
2015-03-17 14:08:00 -07:00
skiplist.h
Allow GetApproximateSize() to include mem table size if it is skip list memtable
2015-06-16 18:13:23 -07:00
slice.cc
Create an abstract interface for write batches
2015-03-17 19:23:08 -07:00
snapshot.h
GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex
2015-06-08 21:57:02 -07:00
table_cache.cc
Implement a table-level row cache
2015-06-23 10:25:45 -07:00
table_cache.h
Implement a table-level row cache
2015-06-23 10:25:45 -07:00
table_properties_collector_test.cc
Make table_properties_collector_test runnable in ROCKSDB_LITE
2015-06-24 01:38:53 -07:00
table_properties_collector.cc
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
2015-04-06 10:27:21 -07:00
table_properties_collector.h
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
transaction_log_impl.cc
Turn -Wshadow back on
2014-11-06 11:14:28 -08:00
transaction_log_impl.h
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
version_builder_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_builder.cc
Fix deleting obsolete files
2015-02-06 08:44:30 -08:00
version_builder.h
Move VersionBuilder logic to a separate .cc file
2014-10-31 16:34:38 -07:00
version_edit_test.cc
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_edit.cc
Turn on -Wshadow
2014-10-31 11:59:54 -07:00
version_edit.h
Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2015-06-05 20:18:21 -07:00
version_set_test.cc
Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true
2015-04-03 09:04:35 -07:00
version_set.cc
Allocate LevelFileIteratorState and LevelFileNumIterator from DB iterator's arena
2015-06-30 17:30:38 -07:00
version_set.h
[wal changes 1/3] fixed unbounded wal growth in some workloads
2015-07-02 14:27:00 -07:00
wal_manager_test.cc
Fix flakiness of WalManagerTest
2015-04-13 16:15:05 -07:00
wal_manager.cc
rocksdb: Add missing override
2015-02-26 11:28:41 -08:00
wal_manager.h
Fix -Wnon-virtual-dtor errors
2014-11-10 17:39:38 -05:00
write_batch_base.cc
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_batch_internal.h
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_batch_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_batch.cc
WriteBatch.Merge w/ SliceParts support
2015-05-29 04:30:03 -07:00
write_callback_test.cc
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_callback.h
Optimistic Transactions
2015-05-29 14:36:35 -07:00
write_controller_test.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_controller.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_controller.h
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_thread.cc
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
write_thread.h
Slow down writes by bytes written
2015-06-11 20:42:18 -07:00
writebuffer.h
Enforce write buffer memory limit across column families
2014-12-02 12:09:20 -08:00