andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
f790558f79
rocksdb/table
History
Sagar Vemuri f790558f79 Use creation_time or mtime when file_creation_time=0 (#5184) 
Summary:
We found an issue in Periodic Compactions (introduced in #5166) where files were not being picked up for compactions as all the SST files created with older versions of RocksDB have `file_creation_time` as 0. (Note that `file_creation_time` is a new table property introduced in #5166).

To address this, Periodic compactions now fall back to looking at the `creation_time` table property or the file's modification time (as given by the Env) when `file_creation_time` table property is found to be 0.

Here how the file's modification time (and, in turn, the file age) is computed now:
1. Use `file_creation_time` table property if it is > 0.
1. If not, then use `creation_time` table property if it is > 0.
1. If not, then use file's mtime stat metadata given by the underlying Env.
Don't consider the file at all for compaction if the modification time cannot be correctly determined based on the above conditions.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5184

Differential Revision: D14907795

Pulled By: sagar0

fbshipit-source-id: 4bb2f3631f9a3e04470c674a1d13544584e1e56c
2019-05-02 11:51:30 -07:00
..
adaptive_table_factory.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
adaptive_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
block_based_filter_block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_table_builder.cc Periodic Compactions (#5166) 2019-05-02 11:48:54 -07:00
block_based_table_builder.h Periodic Compactions (#5166) 2019-05-02 11:48:54 -07:00
block_based_table_factory.cc Periodic Compactions (#5166) 2019-05-02 11:48:54 -07:00
block_based_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
block_based_table_reader.cc Evict the uncompression dictionary from the block cache upon table close (#5150) 2019-04-04 16:21:12 -07:00
block_based_table_reader.h Revert "Avoid per-key upper bound check in BlockBasedTableIterator (#5101)" (#5132) 2019-04-02 10:00:38 -07:00
block_builder.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_builder.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.h Cache dictionary used for decompressing data blocks (#4881) 2019-01-23 18:15:47 -08:00
block_prefix_index.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_prefix_index.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
bloom_block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_block.h Disallow customized hash function in DynamicBloom (#4915) 2019-01-24 10:34:30 -08:00
cleanable_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_builder.cc Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
cuckoo_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_factory.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_factory.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_reader_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_reader.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
cuckoo_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
data_block_footer.cc Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_footer.h Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_hash_index_test.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
data_block_hash_index.cc DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
data_block_hash_index.h DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
filter_block.h use user_key and iterate_upper_bound to determine compatibility of bloom filters (#3899) 2018-06-26 15:57:26 -07:00
flush_block_policy.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
format.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
format.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
full_filter_bits_builder.h Skip duplicate bloom keys when whole_key and prefix are mixed 2018-04-24 10:58:16 -07:00
full_filter_block_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
full_filter_block.cc Charging block cache more accurately (#4073) 2018-06-29 08:57:20 -07:00
full_filter_block.h Charging block cache more accurately (#4073) 2018-06-29 08:57:20 -07:00
get_context.cc PlainTable should avoid copying Get() results from immortal source. (#4924) 2019-01-25 17:12:19 -08:00
get_context.h Cache dictionary used for decompressing data blocks (#4881) 2019-01-23 18:15:47 -08:00
index_builder.cc Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
index_builder.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
internal_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
iter_heap.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
iterator_wrapper.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
iterator.cc Add back NewEmptyIterator (#5203) 2019-04-17 09:40:28 -07:00
merger_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
merging_iterator.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
merging_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
meta_blocks.cc Use creation_time or mtime when file_creation_time=0 (#5184) 2019-05-02 11:51:30 -07:00
meta_blocks.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
mock_table.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
mock_table.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
partitioned_filter_block_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
partitioned_filter_block.cc Fix bug in partition filters with format_version=4 (#4381) 2018-09-17 17:28:15 -07:00
partitioned_filter_block.h Fix bug in partition filters with format_version=4 (#4381) 2018-09-17 17:28:15 -07:00
persistent_cache_helper.cc Remove two variables from BlockContents class and don't use class Block for compressed block (#4650) 2018-11-13 17:02:55 -08:00
persistent_cache_helper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_options.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.cc Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_builder.h Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_factory.cc Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
plain_table_index.cc Fix many bugs in log statement arguments (#5089) 2019-04-04 12:12:11 -07:00
plain_table_index.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_key_coding.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_key_coding.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
plain_table_reader.cc Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
plain_table_reader.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
scoped_arena_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_reader_test.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_reader.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_writer_collectors.h Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_writer.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_builder.h Periodic Compactions (#5166) 2019-05-02 11:48:54 -07:00
table_properties_internal.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_properties.cc Periodic Compactions (#5166) 2019-05-02 11:48:54 -07:00
table_reader_bench.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_reader.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
table_test.cc Evict the uncompression dictionary from the block cache upon table close (#5150) 2019-04-04 16:21:12 -07:00
two_level_iterator.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
two_level_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00