andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
b2bc1da561
rocksdb/java/rocksjni
History
Mike Kolupaev 637e64b9ac Add an option to prevent DB::Open() from querying sizes of all sst files (#6353) 
Summary:
When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open.

If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes.

We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353

Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble.

Differential Revision: D19656425

Pulled By: al13n321

fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82
2020-02-04 01:27:26 -08:00
..
backupablejni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
backupenginejni.cc Support range deletion tombstones in IngestExternalFile SSTs (#3778) 2018-07-13 22:43:09 -07:00
cassandra_compactionfilterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
cassandra_value_operator.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
checkpoint.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
clock_cache.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
columnfamilyhandle.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
compact_range_options.cc Add CompactRangeOptions for Java (#4220) 2018-08-17 10:57:25 -07:00
compaction_filter_factory_jnicallback.cc Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
compaction_filter_factory_jnicallback.h Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
compaction_filter_factory.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_filter.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
compaction_job_info.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_job_stats.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options_fifo.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options_universal.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
compaction_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
comparator.cc Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
comparatorjnicallback.cc Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
comparatorjnicallback.h Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
compression_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
env_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
env.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
filter.cc Allow fractional bits/key in BloomFilterPolicy (#6092) 2019-11-26 15:59:34 -08:00
ingest_external_file_options.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
iterator.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
jnicallback.cc Suppress lint in old files 2018-01-29 12:56:42 -08:00
jnicallback.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
loggerjnicallback.cc Remove invalid comparison of va_list and nullptr (#5836) 2019-09-23 12:15:01 -07:00
loggerjnicallback.h Added CompactionFilterFactory support to RocksJava 2017-10-12 11:12:16 -07:00
lru_cache.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
memory_util.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
memtablejni.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
merge_operator.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
native_comparator_wrapper_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
optimistic_transaction_db.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
optimistic_transaction_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
options_util.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
options.cc Add an option to prevent DB::Open() from querying sizes of all sst files (#6353) 2020-02-04 01:27:26 -08:00
persistent_cache.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
portal.h Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
ratelimiterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
remove_emptyvalue_compactionfilterjni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
restorejni.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocks_callback_object.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocksdb_exception_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
rocksjni.cc RocksDB#keyMayExist should not assume database values are unicode strings (#6186) 2019-12-20 14:27:58 -08:00
slice.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
snapshot.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
sst_file_manager.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
sst_file_reader_iterator.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
sst_file_readerjni.cc Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
sst_file_writerjni.cc Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
statistics.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
statisticsjni.cc Get CompactionJobInfo from CompactFiles 2018-12-13 14:21:24 -08:00
statisticsjni.h Suppress lint in old files 2018-01-29 12:56:42 -08:00
table_filter_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table_filter_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table_filter.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
table.cc JNI: Do not create 8M block cache for negative blockCacheSize values (#5465) 2019-06-24 11:37:04 -07:00
thread_status.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
trace_writer.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
transaction_db_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_db.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
transaction_log.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_notifier_jnicallback.cc Add TransactionDB and OptimisticTransactionDB to the Java API 2018-03-02 10:34:13 -08:00
transaction_notifier_jnicallback.h Add TransactionDB and OptimisticTransactionDB to the Java API 2018-03-02 10:34:13 -08:00
transaction_notifier.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction_options.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
transaction.cc Extend Transaction::GetForUpdate with do_validate (#4680) 2018-12-06 17:49:00 -08:00
ttl.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter_jnicallback.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter_jnicallback.h Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
wal_filter.cc Add missing functionality to RocksJava (#4833) 2019-02-22 14:46:46 -08:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_batch_with_index.cc Improve RocksJava Comparator (#6252) 2020-02-03 12:30:13 -08:00
write_batch.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
write_buffer_manager.cc Plumb WriteBufferManager through JNI (#4492) 2018-10-17 11:49:57 -07:00
writebatchhandlerjnicallback.cc Remove warnings caused by unused variables in jni (#4345) 2018-09-05 13:42:34 -07:00
writebatchhandlerjnicallback.h WriteUnPrepared: Add support for recovering WriteUnprepared transactions (#4078) 2018-07-06 17:59:13 -07:00