andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Merge branch 'master' into fix-range-deletion-bug

Browse Source
This commit is contained in:
奏之章 2019-11-28 12:06:48 +08:00 committed by GitHub
commit da5c117dd8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
62 changed files with 971 additions and 340 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
HISTORY.md
View File

@ -1,17 +1,20 @@
# Rocksdb Change Log
## Unreleased
### Public API Change
* RocksDB release 4.1 or older will not be able to open DB generated by the new release. 4.2 was released on Feb 23, 2016.
* TTL Compactions in Level compaction style now initiate successive cascading compactions on a key range so that it reaches the bottom level quickly on TTL expiry. `creation_time` table property for compaction output files is now set to the minimum of the creation times of all compaction inputs.
* With FIFO compaction style, options.periodic_compaction_seconds will have the same meaning as options.ttl. Whichever stricter will be used. With the default options.periodic_compaction_seconds value with options.ttl's default of 0, RocksDB will give a default of 30 days.
* Added an API GetCreationTimeOfOldestFile(uint64_t* creation_time) to get the file_creation_time of the oldest SST file in the DB.
* An unlikely usage of FilterPolicy is no longer supported. Calling GetFilterBitsBuilder() on the FilterPolicy returned by NewBloomFilterPolicy will now cause an assertion violation in debug builds, because RocksDB has internally migrated to a more elaborate interface that is expected to evolve further. Custom implementations of FilterPolicy should work as before, except those wrapping the return of NewBloomFilterPolicy, which will require a new override of a protected function in FilterPolicy.
* NewBloomFilterPolicy now takes bits_per_key as a double instead of an int. This permits finer control over the memory vs. accuracy trade-off in the new Bloom filter implementation and should not change source code compatibility.
* The option BackupableDBOptions::max_valid_backups_to_open is now only used when opening BackupEngineReadOnly. When opening a read/write BackupEngine, anything but the default value logs a warning and is treated as the default. This change ensures that backup deletion has proper accounting of shared files to ensure they are deleted when no longer referenced by a backup.
### Default Option Changes
* Changed the default value of periodic_compaction_seconds to `UINT64_MAX - 1` which allows RocksDB to auto-tune periodic compaction scheduling. When using the default value, periodic compactions are now auto-enabled if a compaction filter is used. A value of `0` will turn off the feature completely.
* Changed the default value of ttl to `UINT64_MAX - 1` which allows RocksDB to auto-tune ttl value. When using the default value, TTL will be auto-enabled to 30 days, when the feature is supported. To revert the old behavior, you can explictly set it to 0.
### Buf Fixes
* Fix a bug that can cause unnecessary bg thread to be scheduled(#6104).
## 6.6.0 (11/25/2019)
### Bug Fixes
* Fix data corruption casued by output of intra-L0 compaction on ingested file not being placed in correct order in L0.
* Fix a data race between Version::GetColumnFamilyMetaData() and Compaction::MarkFilesBeingCompacted() for access to being_compacted (#6056). The current fix acquires the db mutex during Version::GetColumnFamilyMetaData(), which may cause regression.
* Fix a bug in DBIter that is_blob_ state isn't updated when iterating backward using seek.
* Fix a bug when format_version=3, partitioned fitlers, and prefix search are used in conjunction. The bug could result into Seek::(prefix) returning NotFound for an existing prefix.
* Revert the feature "Merging iterator to avoid child iterator reseek for some cases (#5286)" since it might cause strong results when reseek happens with a different iterator upper bound.
* Fix a bug causing a crash during ingest external file when background compaction cause severe error (file not found).
* Fix a bug when partitioned filters and prefix search are used in conjunction, ::SeekForPrev could return invalid for an existing prefix. ::SeekForPrev might be called by the user, or internally on ::Prev, or within ::Seek if the return value involves Delete or a Merge operand.
* Fix OnFlushCompleted fired before flush result persisted in MANIFEST when there's concurrent flush job. The bug exists since OnFlushCompleted was introduced in rocksdb 3.8.
* Fixed an sst_dump crash on some plain table SST files.
* Fixed a memory leak in some error cases of opening plain table SST files.
* Fix a bug when a crash happens while calling WriteLevel0TableForRecovery for multiple column families, leading to a column family's log number greater than the first corrutped log number when the DB is being opened in PointInTime recovery mode during next recovery attempt (#5856).
### New Features
* Universal compaction to support options.periodic_compaction_seconds. A full compaction will be triggered if any file is over the threshold.
@ -19,19 +22,44 @@
* A batched MultiGet API (DB::MultiGet()) that supports retrieving keys from multiple column families.
* Full and partitioned filters in the block-based table use an improved Bloom filter implementation, enabled with format_version 5 (or above) because previous releases cannot read this filter. This replacement is faster and more accurate, especially for high bits per key or millions of keys in a single (full) filter. For example, the new Bloom filter has the same false postive rate at 9.55 bits per key as the old one at 10 bits per key, and a lower false positive rate at 16 bits per key than the old one at 100 bits per key.
* Added AVX2 instructions to USE_SSE builds to accelerate the new Bloom filter and XXH3-based hash function on compatible x86_64 platforms (Haswell and later, ~2014).
* Support options.ttl with options.max_open_files = -1. File's oldest ancester time will be written to manifest. If it is availalbe, this information will be used instead of creation_time in table properties.
* Support options.ttl or options.periodic_compaction_seconds with options.max_open_files = -1. File's oldest ancester time and file creation time will be written to manifest. If it is availalbe, this information will be used instead of creation_time and file_creation_time in table properties.
* Setting options.ttl for universal compaction now has the same meaning as setting periodic_compaction_seconds.
* SstFileMetaData also returns file creation time and oldest ancester time.
* The `sst_dump` command line tool `recompress` command now displays how many blocks were compressed and how many were not, in particular how many were not compressed because the compression ratio was not met (12.5% threshold for GoodCompressionRatio), as seen in the `number.block.not_compressed` counter stat since version 6.0.0.
* The block cache usage is now takes into account the overhead of metadata per each entry. This results into more accurate managment of memory. A side-effect of this feature is that less items are fit into the block cache of the same size, which would result to higher cache miss rates. This can be remedied by increasing the block cache size or passing kDontChargeCacheMetadata to its constuctor to restore the old behavior.
* When using BlobDB, a mapping is maintained and persisted in the MANIFEST between each SST file and the oldest non-TTL blob file it references.
* `db_bench` now supports and by default issues non-TTL Puts to BlobDB. TTL Puts can be enabled by specifying a non-zero value for the `blob_db_max_ttl_range` command line parameter explicitly.
* `sst_dump` now supports printing BlobDB blob indexes in a human-readable format. This can be enabled by specifying the `decode_blob_index` flag on the command line.
* A number of new information elements are now exposed through the EventListener interface. For flushes, the file numbers of the new SST file and the oldest blob file referenced by the SST are propagated. For compactions, the level, file number, and the oldest blob file referenced are passed to the client for each compaction input and output file.
### Public API Change
* RocksDB release 4.1 or older will not be able to open DB generated by the new release. 4.2 was released on Feb 23, 2016.
* TTL Compactions in Level compaction style now initiate successive cascading compactions on a key range so that it reaches the bottom level quickly on TTL expiry. `creation_time` table property for compaction output files is now set to the minimum of the creation times of all compaction inputs.
* With FIFO compaction style, options.periodic_compaction_seconds will have the same meaning as options.ttl. Whichever stricter will be used. With the default options.periodic_compaction_seconds value with options.ttl's default of 0, RocksDB will give a default of 30 days.
* Added an API GetCreationTimeOfOldestFile(uint64_t* creation_time) to get the file_creation_time of the oldest SST file in the DB.
* FilterPolicy now exposes additional API to make it possible to choose filter configurations based on context, such as table level and compaction style. See `LevelAndStyleCustomFilterPolicy` in db_bloom_filter_test.cc. While most existing custom implementations of FilterPolicy should continue to work as before, those wrapping the return of NewBloomFilterPolicy will require overriding new function `GetBuilderWithContext()`, because calling `GetFilterBitsBuilder()` on the FilterPolicy returned by NewBloomFilterPolicy is no longer supported.
* An unlikely usage of FilterPolicy is no longer supported. Calling GetFilterBitsBuilder() on the FilterPolicy returned by NewBloomFilterPolicy will now cause an assertion violation in debug builds, because RocksDB has internally migrated to a more elaborate interface that is expected to evolve further. Custom implementations of FilterPolicy should work as before, except those wrapping the return of NewBloomFilterPolicy, which will require a new override of a protected function in FilterPolicy.
* NewBloomFilterPolicy now takes bits_per_key as a double instead of an int. This permits finer control over the memory vs. accuracy trade-off in the new Bloom filter implementation and should not change source code compatibility.
* The option BackupableDBOptions::max_valid_backups_to_open is now only used when opening BackupEngineReadOnly. When opening a read/write BackupEngine, anything but the default value logs a warning and is treated as the default. This change ensures that backup deletion has proper accounting of shared files to ensure they are deleted when no longer referenced by a backup.
* Deprecate `snap_refresh_nanos` option.
* Added DisableManualCompaction/EnableManualCompaction to stop and resume manual compaction.
* Add TryCatchUpWithPrimary() to StackableDB in non-LITE mode.
* Add a new Env::LoadEnv() overloaded function to return a shared_ptr to Env.
* Flush sets file name to "(nil)" for OnTableFileCreationCompleted() if the flush does not produce any L0. This can happen if the file is empty thus delete by RocksDB.
### Default Option Changes
* Changed the default value of periodic_compaction_seconds to `UINT64_MAX - 1` which allows RocksDB to auto-tune periodic compaction scheduling. When using the default value, periodic compactions are now auto-enabled if a compaction filter is used. A value of `0` will turn off the feature completely.
* Changed the default value of ttl to `UINT64_MAX - 1` which allows RocksDB to auto-tune ttl value. When using the default value, TTL will be auto-enabled to 30 days, when the feature is supported. To revert the old behavior, you can explictly set it to 0.
### Performance Improvements
* For 64-bit hashing, RocksDB is standardizing on a slightly modified preview version of XXH3. This function is now used for many non-persisted hashes, along with fastrange64() in place of the modulus operator, and some benchmarks show a slight improvement.
* Level iterator to invlidate the iterator more often in prefix seek and the level is filtered out by prefix bloom.
## 6.5.2 (11/15/2019)
### Bug Fixes
* Fix data corruption casued by output of intra-L0 compaction on ingested file not being placed in correct order in L0.
* Fix a data race between Version::GetColumnFamilyMetaData() and Compaction::MarkFilesBeingCompacted() for access to being_compacted (#6056). The current fix acquires the db mutex during Version::GetColumnFamilyMetaData(), which may cause regression.
* Fix a assertion failure in MultiGet() when BlockBasedTableOptions::no_block_cache is true and there is no compressed block cache
* If a call to BackupEngine::PurgeOldBackups or BackupEngine::DeleteBackup suffered a crash, power failure, or I/O error, files could be left over from old backups that could only be purged with a call to GarbageCollect. Any call to PurgeOldBackups, DeleteBackup, or GarbageCollect should now suffice to purge such files.
* Fix a buffer overrun problem in BlockBasedTable::MultiGet() when compression is enabled and no compressed block cache is configured.
* Fix a bug in DBIter that is_blob_ state isn't updated when iterating backward using seek.
* If a call to BackupEngine::PurgeOldBackups or BackupEngine::DeleteBackup suffered a crash, power failure, or I/O error, files could be left over from old backups that could only be purged with a call to GarbageCollect. Any call to PurgeOldBackups, DeleteBackup, or GarbageCollect should now suffice to purge such files.
## 6.5.1 (10/16/2019)
### Bug Fixes
@ -45,40 +73,20 @@
* Fix a bug where the compaction snapshot refresh feature is not disabled as advertised when `snap_refresh_nanos` is set to 0..
* Fix bloom filter lookups by the MultiGet batching API when BlockBasedTableOptions::whole_key_filtering is false, by checking that a key is in the perfix_extractor domain and extracting the prefix before looking up.
* Fix a bug in file ingestion caused by incorrect file number allocation when the number of column families involved in the ingestion exceeds 2.
* Fix a bug when format_version=3, partitioned fitlers, and prefix search are used in conjunction. The bug could result into Seek::(prefix) returning NotFound for an existing prefix.
* Revert the feature "Merging iterator to avoid child iterator reseek for some cases (#5286)" since it might cause strong results when reseek happens with a different iterator upper bound.
* Fix a bug causing a crash during ingest external file when background compaction cause severe error (file not found).
* Fix a bug when partitioned filters and prefix search are used in conjunction, ::SeekForPrev could return invalid for an existing prefix. ::SeekForPrev might be called by the user, or internally on ::Prev, or within ::Seek if the return value involves Delete or a Merge operand.
* Fix OnFlushCompleted fired before flush result persisted in MANIFEST when there's concurrent flush job. The bug exists since OnFlushCompleted was introduced in rocksdb 3.8.
* Fixed an sst_dump crash on some plain table SST files.
* Fixed a memory leak in some error cases of opening plain table SST files.
* Fix a bug when a crash happens while calling WriteLevel0TableForRecovery for multiple column families, leading to a column family's log number greater than the first corrutped log number when the DB is being opened in PointInTime recovery mode during next recovery attempt (#5856).
### New Features
* Introduced DBOptions::max_write_batch_group_size_bytes to configure maximum limit on number of bytes that are written in a single batch of WAL or memtable write. It is followed when the leader write size is larger than 1/8 of this limit.
* VerifyChecksum() by default will issue readahead. Allow ReadOptions to be passed in to those functions to override the readhead size. For checksum verifying before external SST file ingestion, a new option IngestExternalFileOptions.verify_checksums_readahead_size, is added for this readahead setting.
* When user uses options.force_consistency_check in RocksDb, instead of crashing the process, we now pass the error back to the users without killing the process.
* Add an option `memtable_insert_hint_per_batch` to WriteOptions. If it is true, each WriteBatch will maintain its own insert hints for each memtable in concurrent write. See include/rocksdb/options.h for more details.
* The `sst_dump` command line tool `recompress` command now displays how many blocks were compressed and how many were not, in particular how many were not compressed because the compression ratio was not met (12.5% threshold for GoodCompressionRatio), as seen in the `number.block.not_compressed` counter stat since version 6.0.0.
* The block cache usage is now takes into account the overhead of metadata per each entry. This results into more accurate managment of memory. A side-effect of this feature is that less items are fit into the block cache of the same size, which would result to higher cache miss rates. This can be remedied by increasing the block cache size or passing kDontChargeCacheMetadata to its constuctor to restore the old behavior.
* When using BlobDB, a mapping is maintained and persisted in the MANIFEST between each SST file and the oldest non-TTL blob file it references.
* `db_bench` now supports and by default issues non-TTL Puts to BlobDB. TTL Puts can be enabled by specifying a non-zero value for the `blob_db_max_ttl_range` command line parameter explicitly.
* `sst_dump` now supports printing BlobDB blob indexes in a human-readable format. This can be enabled by specifying the `decode_blob_index` flag on the command line.
* A number of new information elements are now exposed through the EventListener interface. For flushes, the file numbers of the new SST file and the oldest blob file referenced by the SST are propagated. For compactions, the level, file number, and the oldest blob file referenced are passed to the client for each compaction input and output file.
### Public API Change
* Added max_write_buffer_size_to_maintain option to better control memory usage of immutable memtables.
* Added a lightweight API GetCurrentWalFile() to get last live WAL filename and size. Meant to be used as a helper for backup/restore tooling in a larger ecosystem such as MySQL with a MyRocks storage engine.
* The MemTable Bloom filter, when enabled, now always uses cache locality. Options::bloom_locality now only affects the PlainTable SST format.
* Deprecate `snap_refresh_nanos` option.
* Added DisableManualCompaction/EnableManualCompaction to stop and resume manual compaction.
* Add TryCatchUpWithPrimary() to StackableDB in non-LITE mode.
* Add a new Env::LoadEnv() overloaded function to return a shared_ptr to Env.
* Flush sets file name to "(nil)" for OnTableFileCreationCompleted() if the flush does not produce any L0. This can happen if the file is empty thus delete by RocksDB.
### Performance Improvements
* Improve the speed of the MemTable Bloom filter, reducing the write overhead of enabling it by 1/3 to 1/2, with similar benefit to read performance.
* Level iterator to invlidate the iterator more often in prefix seek and the level is filtered out by prefix bloom.
## 6.4.0 (7/30/2019)
### Default Option Change

43
Makefile
View File

@ -1731,12 +1731,23 @@ else
ARCH := $(shell getconf LONG_BIT)
endif
ifneq (,$(filter ppc% arm64 aarch64 sparc64, $(MACHINE)))
ROCKSDBJNILIB = librocksdbjni-linux-$(MACHINE).so
else
ROCKSDBJNILIB = librocksdbjni-linux$(ARCH).so
ifeq ($(shell ldd /usr/bin/env 2>/dev/null | grep -q musl; echo $$?),0)
JNI_LIBC = musl
# GNU LibC (or glibc) is so pervasive we can assume it is the default
# else
# JNI_LIBC = glibc
endif
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux$(ARCH).jar
ifneq ($(origin JNI_LIBC), undefined)
JNI_LIBC_POSTFIX = -$(JNI_LIBC)
endif
ifneq (,$(filter ppc% arm64 aarch64 sparc64, $(MACHINE)))
ROCKSDBJNILIB = librocksdbjni-linux-$(MACHINE)$(JNI_LIBC_POSTFIX).so
else
ROCKSDBJNILIB = librocksdbjni-linux$(ARCH)$(JNI_LIBC_POSTFIX).so
endif
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux$(ARCH)$(JNI_LIBC_POSTFIX).jar
ROCKSDB_JAR_ALL = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar
ROCKSDB_JAVADOCS_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-javadoc.jar
ROCKSDB_SOURCES_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar
@ -1910,12 +1921,12 @@ rocksdbjavastatic: $(java_static_all_libobjects)
cd java/src/main/java;jar -cf ../../../target/$(ROCKSDB_SOURCES_JAR) org
rocksdbjavastaticrelease: rocksdbjavastatic
cd java/crossbuild && vagrant destroy -f && vagrant up linux32 && vagrant halt linux32 && vagrant up linux64 && vagrant halt linux64
cd java/crossbuild && (vagrant destroy -f || true) && vagrant up linux32 && vagrant halt linux32 && vagrant up linux64 && vagrant halt linux64 && vagrant up linux64-musl && vagrant halt linux64-musl
cd java;jar -cf target/$(ROCKSDB_JAR_ALL) HISTORY*.md
cd java/target;jar -uf $(ROCKSDB_JAR_ALL) librocksdbjni-*.so librocksdbjni-*.jnilib
cd java/target/classes;jar -uf ../$(ROCKSDB_JAR_ALL) org/rocksdb/*.class org/rocksdb/util/*.class
rocksdbjavastaticreleasedocker: rocksdbjavastatic rocksdbjavastaticdockerx86 rocksdbjavastaticdockerx86_64
rocksdbjavastaticreleasedocker: rocksdbjavastatic rocksdbjavastaticdockerx86 rocksdbjavastaticdockerx86_64 rocksdbjavastaticdockerx86musl rocksdbjavastaticdockerx86_64musl
cd java;jar -cf target/$(ROCKSDB_JAR_ALL) HISTORY*.md
cd java/target;jar -uf $(ROCKSDB_JAR_ALL) librocksdbjni-*.so librocksdbjni-*.jnilib
cd java/target/classes;jar -uf ../$(ROCKSDB_JAR_ALL) org/rocksdb/*.class org/rocksdb/util/*.class
@ -1936,6 +1947,22 @@ rocksdbjavastaticdockerarm64v8:
mkdir -p java/target
docker run --rm --name rocksdb_linux_arm64v8-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:centos7_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerx86musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x86-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerx86_64musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_x64-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerppc64lemusl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_ppc64le-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_ppc64le-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticdockerarm64v8musl:
mkdir -p java/target
docker run --rm --name rocksdb_linux_arm64v8-musl-be --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host:ro --volume /rocksdb-local-build --volume `pwd`/java/target:/rocksdb-java-target --env DEBUG_LEVEL=$(DEBUG_LEVEL) evolvedbinary/rocksjava:alpine3_arm64v8-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
rocksdbjavastaticpublish: rocksdbjavastaticrelease rocksdbjavastaticpublishcentral
rocksdbjavastaticpublishdocker: rocksdbjavastaticreleasedocker rocksdbjavastaticpublishcentral
@ -1945,6 +1972,8 @@ rocksdbjavastaticpublishcentral:
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar -Dclassifier=sources
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux64.jar -Dclassifier=linux64
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux32.jar -Dclassifier=linux32
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux64-musl.jar -Dclassifier=linux64-musl
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux32-musl.jar -Dclassifier=linux32-musl
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar -Dclassifier=osx
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-win64.jar -Dclassifier=win64
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar

1
db/compaction/compaction_job.cc
View File

@ -1501,6 +1501,7 @@ Status CompactionJob::OpenCompactionOutputFile(
out.meta.fd = FileDescriptor(file_number,
sub_compact->compaction->output_path_id(), 0);
out.meta.oldest_ancester_time = oldest_ancester_time;
out.meta.file_creation_time = current_time;
out.finished = false;
sub_compact->outputs.push_back(out);
}

2
db/compaction/compaction_job_test.cc
View File

@ -184,7 +184,7 @@ class CompactionJobTest : public testing::Test {
VersionEdit edit;
edit.AddFile(level, file_number, 0, 10, smallest_key, largest_key,
smallest_seqno, largest_seqno, false, oldest_blob_file_number,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
mutex_.Lock();
versions_->LogAndApply(versions_->GetColumnFamilySet()->GetDefault(),

2
db/compaction/compaction_picker_test.cc
View File

@ -95,7 +95,7 @@ class CompactionPickerTest : public testing::Test {
InternalKey(smallest, smallest_seq, kTypeValue),
InternalKey(largest, largest_seq, kTypeValue), smallest_seq,
largest_seq, /* marked_for_compact */ false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
f->compensated_file_size =
(compensated_file_size != 0) ? compensated_file_size : file_size;
vstorage_->AddFile(level, f);

178
db/db_bloom_filter_test.cc
View File

@ -648,15 +648,17 @@ TEST_F(DBBloomFilterTest, BloomFilterReverseCompatibility) {
}
namespace {
// A wrapped bloom over default FilterPolicy
class WrappedBloom : public FilterPolicy {
// A wrapped bloom over block-based FilterPolicy
class TestingWrappedBlockBasedFilterPolicy : public FilterPolicy {
public:
explicit WrappedBloom(int bits_per_key)
explicit TestingWrappedBlockBasedFilterPolicy(int bits_per_key)
: filter_(NewBloomFilterPolicy(bits_per_key, true)), counter_(0) {}
~WrappedBloom() override { delete filter_; }
~TestingWrappedBlockBasedFilterPolicy() override { delete filter_; }
const char* Name() const override { return "WrappedRocksDbFilterPolicy"; }
const char* Name() const override {
return "TestingWrappedBlockBasedFilterPolicy";
}
void CreateFilter(const rocksdb::Slice* keys, int n,
std::string* dst) const override {
@ -683,12 +685,13 @@ class WrappedBloom : public FilterPolicy {
};
} // namespace
TEST_F(DBBloomFilterTest, BloomFilterWrapper) {
TEST_F(DBBloomFilterTest, WrappedBlockBasedFilterPolicy) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
BlockBasedTableOptions table_options;
WrappedBloom* policy = new WrappedBloom(10);
TestingWrappedBlockBasedFilterPolicy* policy =
new TestingWrappedBlockBasedFilterPolicy(10);
table_options.filter_policy.reset(policy);
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
@ -718,6 +721,166 @@ TEST_F(DBBloomFilterTest, BloomFilterWrapper) {
ASSERT_EQ(2U * maxKey, policy->GetCounter());
}
namespace {
// NOTE: This class is referenced by HISTORY.md as a model for a wrapper
// FilterPolicy selecting among configurations based on context.
class LevelAndStyleCustomFilterPolicy : public FilterPolicy {
public:
explicit LevelAndStyleCustomFilterPolicy(int bpk_fifo, int bpk_l0_other,
int bpk_otherwise)
: policy_fifo_(NewBloomFilterPolicy(bpk_fifo)),
policy_l0_other_(NewBloomFilterPolicy(bpk_l0_other)),
policy_otherwise_(NewBloomFilterPolicy(bpk_otherwise)) {}
// OK to use built-in policy name because we are deferring to a
// built-in builder. We aren't changing the serialized format.
const char* Name() const override { return policy_fifo_->Name(); }
FilterBitsBuilder* GetBuilderWithContext(
const FilterBuildingContext& context) const override {
if (context.compaction_style == kCompactionStyleFIFO) {
return policy_fifo_->GetBuilderWithContext(context);
} else if (context.level_at_creation == 0) {
return policy_l0_other_->GetBuilderWithContext(context);
} else {
return policy_otherwise_->GetBuilderWithContext(context);
}
}
FilterBitsReader* GetFilterBitsReader(const Slice& contents) const override {
// OK to defer to any of them; they all can parse built-in filters
// from any settings.
return policy_fifo_->GetFilterBitsReader(contents);
}
// Defer just in case configuration uses block-based filter
void CreateFilter(const Slice* keys, int n, std::string* dst) const override {
policy_otherwise_->CreateFilter(keys, n, dst);
}
bool KeyMayMatch(const Slice& key, const Slice& filter) const override {
return policy_otherwise_->KeyMayMatch(key, filter);
}
private:
const std::unique_ptr<const FilterPolicy> policy_fifo_;
const std::unique_ptr<const FilterPolicy> policy_l0_other_;
const std::unique_ptr<const FilterPolicy> policy_otherwise_;
};
class TestingContextCustomFilterPolicy
: public LevelAndStyleCustomFilterPolicy {
public:
explicit TestingContextCustomFilterPolicy(int bpk_fifo, int bpk_l0_other,
int bpk_otherwise)
: LevelAndStyleCustomFilterPolicy(bpk_fifo, bpk_l0_other, bpk_otherwise) {
}
FilterBitsBuilder* GetBuilderWithContext(
const FilterBuildingContext& context) const override {
test_report_ += "cf=";
test_report_ += context.column_family_name;
test_report_ += ",cs=";
test_report_ +=
OptionsHelper::compaction_style_to_string[context.compaction_style];
test_report_ += ",lv=";
test_report_ += std::to_string(context.level_at_creation);
test_report_ += "\n";
return LevelAndStyleCustomFilterPolicy::GetBuilderWithContext(context);
}
std::string DumpTestReport() {
std::string rv;
std::swap(rv, test_report_);
return rv;
}
private:
mutable std::string test_report_;
};
} // namespace
TEST_F(DBBloomFilterTest, ContextCustomFilterPolicy) {
for (bool fifo : {true, false}) {
Options options = CurrentOptions();
options.statistics = rocksdb::CreateDBStatistics();
options.compaction_style =
fifo ? kCompactionStyleFIFO : kCompactionStyleLevel;
BlockBasedTableOptions table_options;
auto policy = std::make_shared<TestingContextCustomFilterPolicy>(15, 8, 5);
table_options.filter_policy = policy;
table_options.format_version = 5;
options.table_factory.reset(NewBlockBasedTableFactory(table_options));
CreateAndReopenWithCF({fifo ? "abe" : "bob"}, options);
const int maxKey = 10000;
for (int i = 0; i < maxKey / 2; i++) {
ASSERT_OK(Put(1, Key(i), Key(i)));
}
// Add a large key to make the file contain wide range
ASSERT_OK(Put(1, Key(maxKey + 55555), Key(maxKey + 55555)));
Flush(1);
EXPECT_EQ(policy->DumpTestReport(),
fifo ? "cf=abe,cs=kCompactionStyleFIFO,lv=0\n"
: "cf=bob,cs=kCompactionStyleLevel,lv=0\n");
for (int i = maxKey / 2; i < maxKey; i++) {
ASSERT_OK(Put(1, Key(i), Key(i)));
}
Flush(1);
EXPECT_EQ(policy->DumpTestReport(),
fifo ? "cf=abe,cs=kCompactionStyleFIFO,lv=0\n"
: "cf=bob,cs=kCompactionStyleLevel,lv=0\n");
// Check that they can be found
for (int i = 0; i < maxKey; i++) {
ASSERT_EQ(Key(i), Get(1, Key(i)));
}
// Since we have two tables / two filters, we might have Bloom checks on
// our queries, but no more than one "useful" per query on a found key.
EXPECT_LE(TestGetAndResetTickerCount(options, BLOOM_FILTER_USEFUL), maxKey);
// Check that we have two filters, each about
// fifo: 0.12% FP rate (15 bits per key)
// level: 2.3% FP rate (8 bits per key)
for (int i = 0; i < maxKey; i++) {
ASSERT_EQ("NOT_FOUND", Get(1, Key(i + 33333)));
}
{
auto useful_count =
TestGetAndResetTickerCount(options, BLOOM_FILTER_USEFUL);
EXPECT_GE(useful_count, maxKey * 2 * (fifo ? 0.9980 : 0.975));
EXPECT_LE(useful_count, maxKey * 2 * (fifo ? 0.9995 : 0.98));
}
if (!fifo) { // FIFO only has L0
// Full compaction
ASSERT_OK(db_->CompactRange(CompactRangeOptions(), handles_[1], nullptr,
nullptr));
EXPECT_EQ(policy->DumpTestReport(),
"cf=bob,cs=kCompactionStyleLevel,lv=1\n");
// Check that we now have one filter, about 9.2% FP rate (5 bits per key)
for (int i = 0; i < maxKey; i++) {
ASSERT_EQ("NOT_FOUND", Get(1, Key(i + 33333)));
}
{
auto useful_count =
TestGetAndResetTickerCount(options, BLOOM_FILTER_USEFUL);
EXPECT_GE(useful_count, maxKey * 0.90);
EXPECT_LE(useful_count, maxKey * 0.91);
}
}
// Destroy
ASSERT_OK(dbfull()->DropColumnFamily(handles_[1]));
dbfull()->DestroyColumnFamilyHandle(handles_[1]);
handles_[1] = nullptr;
}
}
class SliceTransformLimitedDomain : public SliceTransform {
const char* Name() const override { return "SliceTransformLimitedDomain"; }
@ -1159,6 +1322,7 @@ TEST_F(DBBloomFilterTest, OptimizeFiltersForHits) {
options.table_factory.reset(NewBlockBasedTableFactory(bbto));
options.optimize_filters_for_hits = true;
options.statistics = rocksdb::CreateDBStatistics();
get_perf_context()->Reset();
get_perf_context()->EnablePerLevelPerfContext();
CreateAndReopenWithCF({"mypikachu"}, options);

151
db/db_compaction_test.cc
View File

@ -3657,71 +3657,103 @@ TEST_F(DBCompactionTest, LevelPeriodicCompaction) {
const int kNumLevelFiles = 2;
const int kValueSize = 100;
Options options = CurrentOptions();
options.periodic_compaction_seconds = 48 * 60 * 60; // 2 days
options.max_open_files = -1; // needed for ttl compaction
env_->time_elapse_only_sleep_ = false;
options.env = env_;
for (bool if_restart : {false, true}) {
for (bool if_open_all_files : {false, true}) {
Options options = CurrentOptions();
options.periodic_compaction_seconds = 48 * 60 * 60; // 2 days
if (if_open_all_files) {
options.max_open_files = -1; // needed for ttl compaction
} else {
options.max_open_files = 20;
}
// RocksDB sanitize max open files to at least 20. Modify it back.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"SanitizeOptions::AfterChangeMaxOpenFiles", [&](void* arg) {
int* max_open_files = static_cast<int*>(arg);
*max_open_files = 0;
});
// In the case where all files are opened and doing DB restart
// forcing the file creation time in manifest file to be 0 to
// simulate the case of reading from an old version.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"VersionEdit::EncodeTo:VarintFileCreationTime", [&](void* arg) {
if (if_restart && if_open_all_files) {
std::string* encoded_fieled = static_cast<std::string*>(arg);
*encoded_fieled = "";
PutVarint64(encoded_fieled, 0);
}
});
env_->addon_time_.store(0);
DestroyAndReopen(options);
env_->time_elapse_only_sleep_ = false;
options.env = env_;
int periodic_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
if (compaction_reason == CompactionReason::kPeriodicCompaction) {
periodic_compactions++;
env_->addon_time_.store(0);
DestroyAndReopen(options);
int periodic_compactions = 0;
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"LevelCompactionPicker::PickCompaction:Return", [&](void* arg) {
Compaction* compaction = reinterpret_cast<Compaction*>(arg);
auto compaction_reason = compaction->compaction_reason();
if (compaction_reason == CompactionReason::kPeriodicCompaction) {
periodic_compactions++;
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(Put(Key(i * kNumKeysPerFile + j),
RandomString(&rnd, kValueSize)));
}
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
Flush();
}
dbfull()->TEST_WaitForCompact();
Random rnd(301);
for (int i = 0; i < kNumLevelFiles; ++i) {
for (int j = 0; j < kNumKeysPerFile; ++j) {
ASSERT_OK(
Put(Key(i * kNumKeysPerFile + j), RandomString(&rnd, kValueSize)));
ASSERT_EQ("2", FilesPerLevel());
ASSERT_EQ(0, periodic_compactions);
// Add 50 hours and do a write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("a", "1"));
Flush();
dbfull()->TEST_WaitForCompact();
// Assert that the files stay in the same level
ASSERT_EQ("3", FilesPerLevel());
// The two old files go through the periodic compaction process
ASSERT_EQ(2, periodic_compactions);
MoveFilesToLevel(1);
ASSERT_EQ("0,3", FilesPerLevel());
// Add another 50 hours and do another write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("b", "2"));
if (if_restart) {
Reopen(options);
} else {
Flush();
}
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("1,3", FilesPerLevel());
// The three old files now go through the periodic compaction process. 2
// + 3.
ASSERT_EQ(5, periodic_compactions);
// Add another 50 hours and do another write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("c", "3"));
Flush();
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("2,3", FilesPerLevel());
// The four old files now go through the periodic compaction process. 5
// + 4.
ASSERT_EQ(9, periodic_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
}
Flush();
}
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("2", FilesPerLevel());
ASSERT_EQ(0, periodic_compactions);
// Add 50 hours and do a write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("a", "1"));
Flush();
dbfull()->TEST_WaitForCompact();
// Assert that the files stay in the same level
ASSERT_EQ("3", FilesPerLevel());
// The two old files go through the periodic compaction process
ASSERT_EQ(2, periodic_compactions);
MoveFilesToLevel(1);
ASSERT_EQ("0,3", FilesPerLevel());
// Add another 50 hours and do another write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("b", "2"));
Flush();
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("1,3", FilesPerLevel());
// The three old files now go through the periodic compaction process. 2 + 3.
ASSERT_EQ(5, periodic_compactions);
// Add another 50 hours and do another write
env_->addon_time_.fetch_add(50 * 60 * 60);
ASSERT_OK(Put("c", "3"));
Flush();
dbfull()->TEST_WaitForCompact();
ASSERT_EQ("2,3", FilesPerLevel());
// The four old files now go through the periodic compaction process. 5 + 4.
ASSERT_EQ(9, periodic_compactions);
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
@ -3734,7 +3766,6 @@ TEST_F(DBCompactionTest, LevelPeriodicCompactionWithOldDB) {
const int kValueSize = 100;
Options options = CurrentOptions();
options.max_open_files = -1; // needed for ttl compaction
env_->time_elapse_only_sleep_ = false;
options.env = env_;

24
db/db_flush_test.cc
View File

@ -212,6 +212,30 @@ TEST_F(DBFlushTest, ManualFlushWithMinWriteBufferNumberToMerge) {
t.join();
}
TEST_F(DBFlushTest, ScheduleOnlyOneBgThread) {
Options options = CurrentOptions();
Reopen(options);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
int called = 0;
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::MaybeScheduleFlushOrCompaction:AfterSchedule:0", [&](void* arg) {
ASSERT_NE(nullptr, arg);
auto unscheduled_flushes = *reinterpret_cast<int*>(arg);
ASSERT_EQ(0, unscheduled_flushes);
++called;
});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put("a", "foo"));
FlushOptions flush_opts;
ASSERT_OK(dbfull()->Flush(flush_opts));
ASSERT_EQ(1, called);
SyncPoint::GetInstance()->DisableProcessing();
SyncPoint::GetInstance()->ClearAllCallBacks();
}
TEST_P(DBFlushDirectIOTest, DirectIO) {
Options options;
options.create_if_missing = true;

14
db/db_impl/db_impl_compaction_flush.cc
View File

@ -1257,7 +1257,7 @@ Status DBImpl::ReFitLevel(ColumnFamilyData* cfd, int level, int target_level) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->oldest_blob_file_number,
f->oldest_ancester_time);
f->oldest_ancester_time, f->file_creation_time);
}
ROCKS_LOG_DEBUG(immutable_db_options_.info_log,
"[%s] Apply version edit:\n%s", cfd->GetName().c_str(),
@ -1918,6 +1918,10 @@ void DBImpl::MaybeScheduleFlushOrCompaction() {
fta->thread_pri_ = Env::Priority::HIGH;
env_->Schedule(&DBImpl::BGWorkFlush, fta, Env::Priority::HIGH, this,
&DBImpl::UnscheduleFlushCallback);
--unscheduled_flushes_;
TEST_SYNC_POINT_CALLBACK(
"DBImpl::MaybeScheduleFlushOrCompaction:AfterSchedule:0",
&unscheduled_flushes_);
}
// special case -- if high-pri (flush) thread pool is empty, then schedule
@ -1932,6 +1936,7 @@ void DBImpl::MaybeScheduleFlushOrCompaction() {
fta->thread_pri_ = Env::Priority::LOW;
env_->Schedule(&DBImpl::BGWorkFlush, fta, Env::Priority::LOW, this,
&DBImpl::UnscheduleFlushCallback);
--unscheduled_flushes_;
}
}
@ -2015,8 +2020,6 @@ ColumnFamilyData* DBImpl::PopFirstFromCompactionQueue() {
DBImpl::FlushRequest DBImpl::PopFirstFromFlushQueue() {
assert(!flush_queue_.empty());
FlushRequest flush_req = flush_queue_.front();
assert(unscheduled_flushes_ >= static_cast<int>(flush_req.size()));
unscheduled_flushes_ -= static_cast<int>(flush_req.size());
flush_queue_.pop_front();
// TODO: need to unset flush reason?
return flush_req;
@ -2058,7 +2061,7 @@ void DBImpl::SchedulePendingFlush(const FlushRequest& flush_req,
cfd->Ref();
cfd->SetFlushReason(flush_reason);
}
unscheduled_flushes_ += static_cast<int>(flush_req.size());
++unscheduled_flushes_;
flush_queue_.push_back(flush_req);
}
@ -2672,7 +2675,8 @@ Status DBImpl::BackgroundCompaction(bool* made_progress,
f->fd.GetPathId(), f->fd.GetFileSize(), f->smallest,
f->largest, f->fd.smallest_seqno,
f->fd.largest_seqno, f->marked_for_compaction,
f->oldest_blob_file_number, f->oldest_ancester_time);
f->oldest_blob_file_number, f->oldest_ancester_time,
f->file_creation_time);
ROCKS_LOG_BUFFER(
log_buffer,

2
db/db_impl/db_impl_experimental.cc
View File

@ -129,7 +129,7 @@ Status DBImpl::PromoteL0(ColumnFamilyHandle* column_family, int target_level) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->oldest_blob_file_number,
f->oldest_ancester_time);
f->oldest_ancester_time, f->file_creation_time);
}
status = versions_->LogAndApply(cfd, *cfd->GetLatestMutableCFOptions(),

2
db/db_impl/db_impl_open.cc
View File

@ -1226,7 +1226,7 @@ Status DBImpl::WriteLevel0TableForRecovery(int job_id, ColumnFamilyData* cfd,
meta.fd.GetFileSize(), meta.smallest, meta.largest,
meta.fd.smallest_seqno, meta.fd.largest_seqno,
meta.marked_for_compaction, meta.oldest_blob_file_number,
meta.oldest_ancester_time);
meta.oldest_ancester_time, meta.file_creation_time);
}
InternalStats::CompactionStats stats(CompactionReason::kFlush, 1);

29
db/db_test.cc
View File

@ -1022,7 +1022,8 @@ TEST_F(DBTest, FailMoreDbPaths) {
void CheckColumnFamilyMeta(
const ColumnFamilyMetaData& cf_meta,
const std::vector<std::vector<FileMetaData>>& files_by_level) {
const std::vector<std::vector<FileMetaData>>& files_by_level,
uint64_t start_time, uint64_t end_time) {
ASSERT_EQ(cf_meta.name, kDefaultColumnFamilyName);
ASSERT_EQ(cf_meta.levels.size(), files_by_level.size());
@ -1060,6 +1061,14 @@ void CheckColumnFamilyMeta(
file_meta_from_files.largest.user_key().ToString());
ASSERT_EQ(file_meta_from_cf.oldest_blob_file_number,
file_meta_from_files.oldest_blob_file_number);
ASSERT_EQ(file_meta_from_cf.oldest_ancester_time,
file_meta_from_files.oldest_ancester_time);
ASSERT_EQ(file_meta_from_cf.file_creation_time,
file_meta_from_files.file_creation_time);
ASSERT_GE(file_meta_from_cf.file_creation_time, start_time);
ASSERT_LE(file_meta_from_cf.file_creation_time, end_time);
ASSERT_GE(file_meta_from_cf.oldest_ancester_time, start_time);
ASSERT_LE(file_meta_from_cf.oldest_ancester_time, end_time);
}
ASSERT_EQ(level_meta_from_cf.size, level_size);
@ -1113,6 +1122,11 @@ TEST_F(DBTest, MetaDataTest) {
Options options = CurrentOptions();
options.create_if_missing = true;
options.disable_auto_compactions = true;
int64_t temp_time = 0;
options.env->GetCurrentTime(&temp_time);
uint64_t start_time = static_cast<uint64_t>(temp_time);
DestroyAndReopen(options);
Random rnd(301);
@ -1139,9 +1153,12 @@ TEST_F(DBTest, MetaDataTest) {
std::vector<std::vector<FileMetaData>> files_by_level;
dbfull()->TEST_GetFilesMetaData(db_->DefaultColumnFamily(), &files_by_level);
options.env->GetCurrentTime(&temp_time);
uint64_t end_time = static_cast<uint64_t>(temp_time);
ColumnFamilyMetaData cf_meta;
db_->GetColumnFamilyMetaData(&cf_meta);
CheckColumnFamilyMeta(cf_meta, files_by_level);
CheckColumnFamilyMeta(cf_meta, files_by_level, start_time, end_time);
std::vector<LiveFileMetaData> live_file_meta;
db_->GetLiveFilesMetaData(&live_file_meta);
@ -6420,6 +6437,12 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
}
}
});
// Set file creation time in manifest all to 0.
rocksdb::SyncPoint::GetInstance()->SetCallBack(
"FileMetaData::FileMetaData", [&](void* arg) {
FileMetaData* meta = static_cast<FileMetaData*>(arg);
meta->file_creation_time = 0;
});
rocksdb::SyncPoint::GetInstance()->EnableProcessing();
Random rnd(301);
@ -6431,7 +6454,7 @@ TEST_F(DBTest, CreationTimeOfOldestFile) {
Flush();
}
// At this point there should be 2 files, oen with file_creation_time = 0 and
// At this point there should be 2 files, one with file_creation_time = 0 and
// the other non-zero. GetCreationTimeOfOldestFile API should return 0.
uint64_t creation_time;
Status s1 = dbfull()->GetCreationTimeOfOldestFile(&creation_time);

5
db/db_test_util.h
View File

@ -988,6 +988,11 @@ class DBTestBase : public testing::Test {
uint64_t TestGetTickerCount(const Options& options, Tickers ticker_type) {
return options.statistics->getTickerCount(ticker_type);
}
uint64_t TestGetAndResetTickerCount(const Options& options,
Tickers ticker_type) {
return options.statistics->getAndResetTickerCount(ticker_type);
}
};
} // namespace rocksdb

9
db/external_sst_file_ingestion_job.cc
View File

@ -246,16 +246,19 @@ Status ExternalSstFileIngestionJob::Run() {
// We use the import time as the ancester time. This is the time the data
// is written to the database.
uint64_t oldest_ancester_time = 0;
int64_t temp_current_time = 0;
uint64_t current_time = kUnknownFileCreationTime;
uint64_t oldest_ancester_time = kUnknownOldestAncesterTime;
if (env_->GetCurrentTime(&temp_current_time).ok()) {
oldest_ancester_time = static_cast<uint64_t>(temp_current_time);
current_time = oldest_ancester_time =
static_cast<uint64_t>(temp_current_time);
}
edit_.AddFile(f.picked_level, f.fd.GetNumber(), f.fd.GetPathId(),
f.fd.GetFileSize(), f.smallest_internal_key,
f.largest_internal_key, f.assigned_seqno, f.assigned_seqno,
false, kInvalidBlobFileNumber, oldest_ancester_time);
false, kInvalidBlobFileNumber, oldest_ancester_time,
current_time);
}
return status;
}

3
db/flush_job.cc
View File

@ -368,6 +368,7 @@ Status FlushJob::WriteLevel0Table() {
// It's not clear whether oldest_key_time is always available. In case
// it is not available, use current_time.
meta_.oldest_ancester_time = std::min(current_time, oldest_key_time);
meta_.file_creation_time = current_time;
s = BuildTable(
dbname_, db_options_.env, *cfd_->ioptions(), mutable_cf_options_,
@ -413,7 +414,7 @@ Status FlushJob::WriteLevel0Table() {
meta_.fd.GetFileSize(), meta_.smallest, meta_.largest,
meta_.fd.smallest_seqno, meta_.fd.largest_seqno,
meta_.marked_for_compaction, meta_.oldest_blob_file_number,
meta_.oldest_ancester_time);
meta_.oldest_ancester_time, meta_.file_creation_time);
}
#ifndef ROCKSDB_LITE
// Piggyback FlushJobInfo on the first first flushed memtable.

8
db/import_column_family_job.cc
View File

@ -135,10 +135,12 @@ Status ImportColumnFamilyJob::Run() {
// We use the import time as the ancester time. This is the time the data
// is written to the database.
uint64_t oldest_ancester_time = 0;
int64_t temp_current_time = 0;
uint64_t oldest_ancester_time = kUnknownOldestAncesterTime;
uint64_t current_time = kUnknownOldestAncesterTime;
if (env_->GetCurrentTime(&temp_current_time).ok()) {
oldest_ancester_time = static_cast<uint64_t>(temp_current_time);
current_time = oldest_ancester_time =
static_cast<uint64_t>(temp_current_time);
}
for (size_t i = 0; i < files_to_import_.size(); ++i) {
@ -149,7 +151,7 @@ Status ImportColumnFamilyJob::Run() {
f.fd.GetFileSize(), f.smallest_internal_key,
f.largest_internal_key, file_metadata.smallest_seqno,
file_metadata.largest_seqno, false, kInvalidBlobFileNumber,
oldest_ancester_time);
oldest_ancester_time, current_time);
// If incoming sequence number is higher, update local sequence number.
if (file_metadata.largest_seqno > versions_->LastSequence()) {

14
db/repair.cc
View File

@ -577,13 +577,13 @@ class Repairer {
// TODO(opt): separate out into multiple levels
for (const auto* table : cf_id_and_tables.second) {
edit.AddFile(0, table->meta.fd.GetNumber(), table->meta.fd.GetPathId(),
table->meta.fd.GetFileSize(), table->meta.smallest,
table->meta.largest, table->meta.fd.smallest_seqno,
table->meta.fd.largest_seqno,
table->meta.marked_for_compaction,
table->meta.oldest_blob_file_number,
table->meta.oldest_ancester_time);
edit.AddFile(
0, table->meta.fd.GetNumber(), table->meta.fd.GetPathId(),
table->meta.fd.GetFileSize(), table->meta.smallest,
table->meta.largest, table->meta.fd.smallest_seqno,
table->meta.fd.largest_seqno, table->meta.marked_for_compaction,
table->meta.oldest_blob_file_number,
table->meta.oldest_ancester_time, table->meta.file_creation_time);
}
assert(next_file_number_ > 0);
vset_.MarkFileNumberUsed(next_file_number_ - 1);

47
db/version_builder_test.cc
View File

@ -63,7 +63,7 @@ class VersionBuilderTest : public testing::Test {
file_number, path_id, file_size, GetInternalKey(smallest, smallest_seq),
GetInternalKey(largest, largest_seq), smallest_seqno, largest_seqno,
/* marked_for_compact */ false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
f->compensated_file_size = file_size;
f->num_entries = num_entries;
f->num_deletions = num_deletions;
@ -114,7 +114,8 @@ TEST_F(VersionBuilderTest, ApplyAndSaveTo) {
VersionEdit version_edit;
version_edit.AddFile(2, 666, 0, 100U, GetInternalKey("301"),
GetInternalKey("350"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.DeleteFile(3, 27U);
EnvOptions env_options;
@ -149,7 +150,8 @@ TEST_F(VersionBuilderTest, ApplyAndSaveToDynamic) {
VersionEdit version_edit;
version_edit.AddFile(3, 666, 0, 100U, GetInternalKey("301"),
GetInternalKey("350"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.DeleteFile(0, 1U);
version_edit.DeleteFile(0, 88U);
@ -187,7 +189,8 @@ TEST_F(VersionBuilderTest, ApplyAndSaveToDynamic2) {
VersionEdit version_edit;
version_edit.AddFile(4, 666, 0, 100U, GetInternalKey("301"),
GetInternalKey("350"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.DeleteFile(0, 1U);
version_edit.DeleteFile(0, 88U);
version_edit.DeleteFile(4, 6U);
@ -216,19 +219,24 @@ TEST_F(VersionBuilderTest, ApplyMultipleAndSaveTo) {
VersionEdit version_edit;
version_edit.AddFile(2, 666, 0, 100U, GetInternalKey("301"),
GetInternalKey("350"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 676, 0, 100U, GetInternalKey("401"),
GetInternalKey("450"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 636, 0, 100U, GetInternalKey("601"),
GetInternalKey("650"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 616, 0, 100U, GetInternalKey("501"),
GetInternalKey("550"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 606, 0, 100U, GetInternalKey("701"),
GetInternalKey("750"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
EnvOptions env_options;
@ -255,30 +263,37 @@ TEST_F(VersionBuilderTest, ApplyDeleteAndSaveTo) {
VersionEdit version_edit;
version_edit.AddFile(2, 666, 0, 100U, GetInternalKey("301"),
GetInternalKey("350"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 676, 0, 100U, GetInternalKey("401"),
GetInternalKey("450"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 636, 0, 100U, GetInternalKey("601"),
GetInternalKey("650"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 616, 0, 100U, GetInternalKey("501"),
GetInternalKey("550"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit.AddFile(2, 606, 0, 100U, GetInternalKey("701"),
GetInternalKey("750"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_builder.Apply(&version_edit);
VersionEdit version_edit2;
version_edit.AddFile(2, 808, 0, 100U, GetInternalKey("901"),
GetInternalKey("950"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_edit2.DeleteFile(2, 616);
version_edit2.DeleteFile(2, 636);
version_edit.AddFile(2, 806, 0, 100U, GetInternalKey("801"),
GetInternalKey("850"), 200, 200, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
version_builder.Apply(&version_edit2);
version_builder.SaveTo(&new_vstorage);

16
db/version_edit.cc
View File

@ -62,6 +62,7 @@ enum CustomTag : uint32_t {
kMinLogNumberToKeepHack = 3,
kOldestBlobFileNumber = 4,
kOldestAncesterTime = 5,
kFileCreationTime = 6,
kPathId = 65,
};
// If this bit for the custom tag is set, opening DB should fail if
@ -217,6 +218,14 @@ bool VersionEdit::EncodeTo(std::string* dst) const {
TEST_SYNC_POINT_CALLBACK("VersionEdit::EncodeTo:VarintOldestAncesterTime",
&varint_oldest_ancester_time);
PutLengthPrefixedSlice(dst, Slice(varint_oldest_ancester_time));
PutVarint32(dst, CustomTag::kFileCreationTime);
std::string varint_file_creation_time;
PutVarint64(&varint_file_creation_time, f.file_creation_time);
TEST_SYNC_POINT_CALLBACK("VersionEdit::EncodeTo:VarintFileCreationTime",
&varint_file_creation_time);
PutLengthPrefixedSlice(dst, Slice(varint_file_creation_time));
if (f.fd.GetPathId() != 0) {
PutVarint32(dst, CustomTag::kPathId);
char p = static_cast<char>(f.fd.GetPathId());
@ -335,6 +344,11 @@ const char* VersionEdit::DecodeNewFile4From(Slice* input) {
return "invalid oldest ancester time";
}
break;
case kFileCreationTime:
if (!GetVarint64(&field, &f.file_creation_time)) {
return "invalid file creation time";
}
break;
case kNeedCompaction:
if (field.size() != 1) {
return "need_compaction field wrong size";
@ -660,6 +674,8 @@ std::string VersionEdit::DebugString(bool hex_key) const {
}
r.append(" oldest_ancester_time:");
AppendNumberTo(&r, f.oldest_ancester_time);
r.append(" file_creation_time:");
AppendNumberTo(&r, f.file_creation_time);
}
r.append("\n ColumnFamily: ");
AppendNumberTo(&r, column_family_);

40
db/version_edit.h
View File

@ -26,6 +26,7 @@ class VersionSet;
constexpr uint64_t kFileNumberMask = 0x3FFFFFFFFFFFFFFF;
constexpr uint64_t kInvalidBlobFileNumber = 0;
constexpr uint64_t kUnknownOldestAncesterTime = 0;
constexpr uint64_t kUnknownFileCreationTime = 0;
extern uint64_t PackFileNumberAndPathId(uint64_t number, uint64_t path_id);
@ -128,7 +129,10 @@ struct FileMetaData {
// in turn be outputs for compact older SST files. We track the memtable
// flush timestamp for the oldest SST file that eventaully contribute data
// to this file. 0 means the information is not available.
uint64_t oldest_ancester_time = 0;
uint64_t oldest_ancester_time = kUnknownOldestAncesterTime;
// Unix time when the SST file is created.
uint64_t file_creation_time = kUnknownFileCreationTime;
FileMetaData() = default;
@ -136,13 +140,17 @@ struct FileMetaData {
const InternalKey& smallest_key, const InternalKey& largest_key,
const SequenceNumber& smallest_seq,
const SequenceNumber& largest_seq, bool marked_for_compact,
uint64_t oldest_blob_file, uint64_t _oldest_ancester_time)
uint64_t oldest_blob_file, uint64_t _oldest_ancester_time,
uint64_t _file_creation_time)
: fd(file, file_path_id, file_size, smallest_seq, largest_seq),
smallest(smallest_key),
largest(largest_key),
marked_for_compaction(marked_for_compact),
oldest_blob_file_number(oldest_blob_file),
oldest_ancester_time(_oldest_ancester_time) {}
oldest_ancester_time(_oldest_ancester_time),
file_creation_time(_file_creation_time) {
TEST_SYNC_POINT_CALLBACK("FileMetaData::FileMetaData", this);
}
// REQUIRED: Keys must be given to the function in sorted order (it expects
// the last key to be the largest).
@ -168,13 +176,23 @@ struct FileMetaData {
// if table reader is already pinned.
// 0 means the information is not available.
uint64_t TryGetOldestAncesterTime() {
if (oldest_ancester_time != 0) {
if (oldest_ancester_time != kUnknownOldestAncesterTime) {
return oldest_ancester_time;
} else if (fd.table_reader != nullptr &&
fd.table_reader->GetTableProperties() != nullptr) {
return fd.table_reader->GetTableProperties()->creation_time;
}
return 0;
return kUnknownOldestAncesterTime;
}
uint64_t TryGetFileCreationTime() {
if (file_creation_time != kUnknownFileCreationTime) {
return file_creation_time;
} else if (fd.table_reader != nullptr &&
fd.table_reader->GetTableProperties() != nullptr) {
return fd.table_reader->GetTableProperties()->file_creation_time;
}
return kUnknownFileCreationTime;
}
};
@ -277,14 +295,14 @@ class VersionEdit {
uint64_t file_size, const InternalKey& smallest,
const InternalKey& largest, const SequenceNumber& smallest_seqno,
const SequenceNumber& largest_seqno, bool marked_for_compaction,
uint64_t oldest_blob_file_number,
uint64_t oldest_ancester_time) {
uint64_t oldest_blob_file_number, uint64_t oldest_ancester_time,
uint64_t file_creation_time) {
assert(smallest_seqno <= largest_seqno);
new_files_.emplace_back(
level,
FileMetaData(file, file_path_id, file_size, smallest, largest,
smallest_seqno, largest_seqno, marked_for_compaction,
oldest_blob_file_number, oldest_ancester_time));
level, FileMetaData(file, file_path_id, file_size, smallest, largest,
smallest_seqno, largest_seqno,
marked_for_compaction, oldest_blob_file_number,
oldest_ancester_time, file_creation_time));
}
void AddFile(int level, const FileMetaData& f) {

21
db/version_edit_test.cc
View File

@ -37,7 +37,7 @@ TEST_F(VersionEditTest, EncodeDecode) {
InternalKey("foo", kBig + 500 + i, kTypeValue),
InternalKey("zoo", kBig + 600 + i, kTypeDeletion),
kBig + 500 + i, kBig + 600 + i, false, kInvalidBlobFileNumber,
888);
888, 678);
edit.DeleteFile(4, kBig + 700 + i);
}
@ -55,17 +55,19 @@ TEST_F(VersionEditTest, EncodeDecodeNewFile4) {
edit.AddFile(3, 300, 3, 100, InternalKey("foo", kBig + 500, kTypeValue),
InternalKey("zoo", kBig + 600, kTypeDeletion), kBig + 500,
kBig + 600, true, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
edit.AddFile(4, 301, 3, 100, InternalKey("foo", kBig + 501, kTypeValue),
InternalKey("zoo", kBig + 601, kTypeDeletion), kBig + 501,
kBig + 601, false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
edit.AddFile(5, 302, 0, 100, InternalKey("foo", kBig + 502, kTypeValue),
InternalKey("zoo", kBig + 602, kTypeDeletion), kBig + 502,
kBig + 602, true, kInvalidBlobFileNumber, 666);
kBig + 602, true, kInvalidBlobFileNumber, 666, 888);
edit.AddFile(5, 303, 0, 100, InternalKey("foo", kBig + 503, kTypeBlobIndex),
InternalKey("zoo", kBig + 603, kTypeBlobIndex), kBig + 503,
kBig + 603, true, 1001, kUnknownOldestAncesterTime);
kBig + 603, true, 1001, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
;
edit.DeleteFile(4, 700);
@ -104,10 +106,10 @@ TEST_F(VersionEditTest, ForwardCompatibleNewFile4) {
edit.AddFile(3, 300, 3, 100, InternalKey("foo", kBig + 500, kTypeValue),
InternalKey("zoo", kBig + 600, kTypeDeletion), kBig + 500,
kBig + 600, true, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
edit.AddFile(4, 301, 3, 100, InternalKey("foo", kBig + 501, kTypeValue),
InternalKey("zoo", kBig + 601, kTypeDeletion), kBig + 501,
kBig + 601, false, kInvalidBlobFileNumber, 686);
kBig + 601, false, kInvalidBlobFileNumber, 686, 868);
edit.DeleteFile(4, 700);
edit.SetComparatorName("foo");
@ -154,7 +156,7 @@ TEST_F(VersionEditTest, NewFile4NotSupportedField) {
edit.AddFile(3, 300, 3, 100, InternalKey("foo", kBig + 500, kTypeValue),
InternalKey("zoo", kBig + 600, kTypeDeletion), kBig + 500,
kBig + 600, true, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
edit.SetComparatorName("foo");
edit.SetLogNumber(kBig + 100);
@ -182,7 +184,8 @@ TEST_F(VersionEditTest, NewFile4NotSupportedField) {
TEST_F(VersionEditTest, EncodeEmptyFile) {
VersionEdit edit;
edit.AddFile(0, 0, 0, 0, InternalKey(), InternalKey(), 0, 0, false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
std::string buffer;
ASSERT_TRUE(!edit.EncodeTo(&buffer));
}

29
db/version_set.cc
View File

@ -1459,7 +1459,8 @@ void Version::GetColumnFamilyMetaData(ColumnFamilyMetaData* cf_meta) {
file->fd.largest_seqno, file->smallest.user_key().ToString(),
file->largest.user_key().ToString(),
file->stats.num_reads_sampled.load(std::memory_order_relaxed),
file->being_compacted, file->oldest_blob_file_number});
file->being_compacted, file->oldest_blob_file_number,
file->TryGetOldestAncesterTime(), file->TryGetFileCreationTime()});
files.back().num_entries = file->num_entries;
files.back().num_deletions = file->num_deletions;
level_size += file->fd.GetFileSize();
@ -1485,10 +1486,9 @@ void Version::GetCreationTimeOfOldestFile(uint64_t* creation_time) {
for (int level = 0; level < storage_info_.num_non_empty_levels_; level++) {
for (FileMetaData* meta : storage_info_.LevelFiles(level)) {
assert(meta->fd.table_reader != nullptr);
uint64_t file_creation_time =
meta->fd.table_reader->GetTableProperties()->file_creation_time;
if (file_creation_time == 0) {
*creation_time = file_creation_time;
uint64_t file_creation_time = meta->TryGetFileCreationTime();
if (file_creation_time == kUnknownFileCreationTime) {
*creation_time = 0;
return;
}
if (file_creation_time < oldest_time) {
@ -2501,8 +2501,7 @@ void VersionStorageInfo::ComputeExpiredTtlFiles(
void VersionStorageInfo::ComputeFilesMarkedForPeriodicCompaction(
const ImmutableCFOptions& ioptions,
const uint64_t periodic_compaction_seconds) {
assert(periodic_compaction_seconds > 0 &&
periodic_compaction_seconds < port::kMaxUint64);
assert(periodic_compaction_seconds > 0);
files_marked_for_periodic_compaction_.clear();
@ -2513,8 +2512,8 @@ void VersionStorageInfo::ComputeFilesMarkedForPeriodicCompaction(
}
const uint64_t current_time = static_cast<uint64_t>(temp_current_time);
// If periodic_compaction_seconds > current_time, no file possibly qualifies
// periodic compaction.
// If periodic_compaction_seconds is larger than current time, periodic
// compaction can't possibly be triggered.
if (periodic_compaction_seconds > current_time) {
return;
}
@ -2524,20 +2523,18 @@ void VersionStorageInfo::ComputeFilesMarkedForPeriodicCompaction(
for (int level = 0; level < num_levels(); level++) {
for (auto f : files_[level]) {
if (!f->being_compacted && f->fd.table_reader != nullptr &&
f->fd.table_reader->GetTableProperties() != nullptr) {
if (!f->being_compacted) {
// Compute a file's modification time in the following order:
// 1. Use file_creation_time table property if it is > 0.
// 2. Use creation_time table property if it is > 0.
// 3. Use file's mtime metadata if the above two table properties are 0.
// Don't consider the file at all if the modification time cannot be
// correctly determined based on the above conditions.
uint64_t file_modification_time =
f->fd.table_reader->GetTableProperties()->file_creation_time;
if (file_modification_time == 0) {
uint64_t file_modification_time = f->TryGetFileCreationTime();
if (file_modification_time == kUnknownFileCreationTime) {
file_modification_time = f->TryGetOldestAncesterTime();
}
if (file_modification_time == 0) {
if (file_modification_time == kUnknownOldestAncesterTime) {
auto file_path = TableFileName(ioptions.cf_paths, f->fd.GetNumber(),
f->fd.GetPathId());
status = ioptions.env->GetFileModificationTime(
@ -4980,7 +4977,7 @@ Status VersionSet::WriteCurrentStateToManifest(log::Writer* log) {
f->fd.GetFileSize(), f->smallest, f->largest,
f->fd.smallest_seqno, f->fd.largest_seqno,
f->marked_for_compaction, f->oldest_blob_file_number,
f->oldest_ancester_time);
f->oldest_ancester_time, f->file_creation_time);
}
}
edit.SetLogNumber(cfd->GetLogNumber());

7
db/version_set_test.cc
View File

@ -40,7 +40,7 @@ class GenerateLevelFilesBriefTest : public testing::Test {
InternalKey(smallest, smallest_seq, kTypeValue),
InternalKey(largest, largest_seq, kTypeValue), smallest_seq,
largest_seq, /* marked_for_compact */ false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
files_.push_back(f);
}
@ -135,7 +135,7 @@ class VersionStorageInfoTest : public testing::Test {
file_number, 0, file_size, GetInternalKey(smallest, 0),
GetInternalKey(largest, 0), /* smallest_seq */ 0, /* largest_seq */ 0,
/* marked_for_compact */ false, kInvalidBlobFileNumber,
kUnknownOldestAncesterTime);
kUnknownOldestAncesterTime, kUnknownFileCreationTime);
f->compensated_file_size = file_size;
vstorage_.AddFile(level, f);
}
@ -146,7 +146,8 @@ class VersionStorageInfoTest : public testing::Test {
FileMetaData* f = new FileMetaData(
file_number, 0, file_size, smallest, largest, /* smallest_seq */ 0,
/* largest_seq */ 0, /* marked_for_compact */ false,
kInvalidBlobFileNumber, kUnknownOldestAncesterTime);
kInvalidBlobFileNumber, kUnknownOldestAncesterTime,
kUnknownFileCreationTime);
f->compensated_file_size = file_size;
vstorage_.AddFile(level, f);
}

49
include/rocksdb/filter_policy.h
View File

@ -25,9 +25,12 @@
#include <string>
#include <vector>
#include "rocksdb/advanced_options.h"
namespace rocksdb {
class Slice;
struct BlockBasedTableOptions;
// A class that takes a bunch of keys, then generates filter
class FilterBitsBuilder {
@ -80,8 +83,25 @@ class FilterBitsReader {
}
};
// Internal type required for FilterPolicy
struct FilterBuildingContext;
// Contextual information passed to BloomFilterPolicy at filter building time.
// Used in overriding FilterPolicy::GetBuilderWithContext().
struct FilterBuildingContext {
// This constructor is for internal use only and subject to change.
FilterBuildingContext(const BlockBasedTableOptions& table_options);
// Options for the table being built
const BlockBasedTableOptions& table_options;
// Name of the column family for the table (or empty string if unknown)
std::string column_family_name;
// The compactions style in effect for the table
CompactionStyle compaction_style = kCompactionStyleLevel;
// The table level at time of constructing the SST file, or -1 if unknown.
// (The table file could later be used at a different level.)
int level_at_creation = -1;
};
// We add a new format of filter block called full filter block
// This new interface gives you more space of customization
@ -125,8 +145,21 @@ class FilterPolicy {
// Return a new FilterBitsBuilder for full or partitioned filter blocks, or
// nullptr if using block-based filter.
// NOTE: This function is only called by GetBuilderWithContext() below for
// custom FilterPolicy implementations. Thus, it is not necessary to
// override this function if overriding GetBuilderWithContext().
virtual FilterBitsBuilder* GetFilterBitsBuilder() const { return nullptr; }
// A newer variant of GetFilterBitsBuilder that allows a FilterPolicy
// to customize the builder for contextual constraints and hints.
// (Name changed to avoid triggering -Werror=overloaded-virtual.)
// If overriding GetFilterBitsBuilder() suffices, it is not necessary to
// override this function.
virtual FilterBitsBuilder* GetBuilderWithContext(
const FilterBuildingContext&) const {
return GetFilterBitsBuilder();
}
// Return a new FilterBitsReader for full or partitioned filter blocks, or
// nullptr if using block-based filter.
// As here, the input slice should NOT be deleted by FilterPolicy.
@ -134,18 +167,6 @@ class FilterPolicy {
const Slice& /*contents*/) const {
return nullptr;
}
protected:
// An internal-use-only variant of GetFilterBitsBuilder that allows
// a built-in FilterPolicy to customize the builder for contextual
// constraints and hints. (Name changed to avoid triggering
// -Werror=overloaded-virtual.)
virtual FilterBitsBuilder* GetFilterBitsBuilderInternal(
const FilterBuildingContext&) const {
return GetFilterBitsBuilder();
}
friend FilterBuildingContext;
};
// Return a new filter policy that uses a bloom filter with approximately

16
include/rocksdb/metadata.h
View File

@ -69,7 +69,8 @@ struct SstFileMetaData {
SequenceNumber _smallest_seqno, SequenceNumber _largest_seqno,
const std::string& _smallestkey,
const std::string& _largestkey, uint64_t _num_reads_sampled,
bool _being_compacted, uint64_t _oldest_blob_file_number)
bool _being_compacted, uint64_t _oldest_blob_file_number,
uint64_t _oldest_ancester_time, uint64_t _file_creation_time)
: size(_size),
name(_file_name),
file_number(_file_number),
@ -82,7 +83,9 @@ struct SstFileMetaData {
being_compacted(_being_compacted),
num_entries(0),
num_deletions(0),
oldest_blob_file_number(_oldest_blob_file_number) {}
oldest_blob_file_number(_oldest_blob_file_number),
oldest_ancester_time(_oldest_ancester_time),
file_creation_time(_file_creation_time) {}
// File size in bytes.
size_t size;
@ -105,6 +108,15 @@ struct SstFileMetaData {
uint64_t oldest_blob_file_number; // The id of the oldest blob file
// referenced by the file.
// An SST file may be generated by compactions whose input files may
// in turn be generated by earlier compactions. The creation time of the
// oldest SST file that is the compaction ancester of this file.
// The timestamp is provided Env::GetCurrentTime().
// 0 if the information is not available.
uint64_t oldest_ancester_time;
// Timestamp when the SST file is created, provided by Env::GetCurrentTime().
// 0 if the information is not available.
uint64_t file_creation_time;
};
// The full set of metadata associated with each SST file.

2
include/rocksdb/options.h
View File

@ -1181,7 +1181,7 @@ struct ReadOptions {
// "iterate_upper_bound" defines the extent upto which the forward iterator
// can returns entries. Once the bound is reached, Valid() will be false.
// "iterate_upper_bound" is exclusive ie the bound value is
// not a valid entry. If iterator_extractor is not null, the Seek target
// not a valid entry. If prefix_extractor is not null, the Seek target
// and iterate_upper_bound need to have the same prefix.
// This is because ordering is not guaranteed outside of prefix domain.
//

6
include/rocksdb/table.h
View File

@ -269,9 +269,9 @@ struct BlockBasedTableOptions {
// probably use this as it would reduce the index size.
// This option only affects newly written tables. When reading existing
// tables, the information about version is read from the footer.
// 5 -- Can be read by RocksDB's versions since X.X.X (something after 6.4.6)
// Full and partitioned filters use a generally faster and more accurate
// Bloom filter implementation, with a different schema.
// 5 -- Can be read by RocksDB's versions since 6.6.0. Full and partitioned
// filters use a generally faster and more accurate Bloom filter
// implementation, with a different schema.
uint32_t format_version = 2;
// Store index blocks on disk in compressed format. Changing this option to

12
include/rocksdb/utilities/backupable_db.h
View File

@ -104,13 +104,13 @@ struct BackupableDBOptions {
// Default: 4194304
uint64_t callback_trigger_interval_size;
// When Open() is called, it will open at most this many of the latest
// non-corrupted backups.
// For BackupEngineReadOnly, Open() will open at most this many of the
// latest non-corrupted backups.
//
// Note setting this to a non-default value prevents old files from being
// deleted in the shared directory, as we can't do proper ref-counting. If
// using this option, make sure to occasionally disable it (by resetting to
// INT_MAX) and run GarbageCollect to clean accumulated stale files.
// Note: this setting is ignored (behaves like INT_MAX) for any kind of
// writable BackupEngine because it would inhibit accounting for shared
// files for proper backup deletion, including purging any incompletely
// created backups on creation of a new backup.
//
// Default: INT_MAX
int max_valid_backups_to_open;

25
java/RELEASE.md
View File

@ -4,28 +4,33 @@ RocksDB can be built as a single self contained cross-platform JAR. The cross-pl
Building a cross-platform JAR requires:
* [Vagrant](https://www.vagrantup.com/)
* [Virtualbox](https://www.virtualbox.org/)
* [Docker](https://www.docker.com/docker-community)
* A Mac OSX machine that can compile RocksDB.
* Java 7 set as JAVA_HOME.
Once you have these items, run this make command from RocksDB's root source directory:
make jclean clean rocksdbjavastaticrelease
make jclean clean rocksdbjavastaticreleasedocker
This command will build RocksDB natively on OSX, and will then spin up two Vagrant Virtualbox Ubuntu images to build RocksDB for both 32-bit and 64-bit Linux.
This command will build RocksDB natively on OSX, and will then spin up docker containers to build RocksDB for 32-bit and 64-bit Linux with glibc, and 32-bit and 64-bit Linux with musl libc.
You can find all native binaries and JARs in the java/target directory upon completion:
librocksdbjni-linux32.so
librocksdbjni-linux64.so
librocksdbjni-linux64-musl.so
librocksdbjni-linux32-musl.so
librocksdbjni-osx.jnilib
rocksdbjni-3.5.0-javadoc.jar
rocksdbjni-3.5.0-linux32.jar
rocksdbjni-3.5.0-linux64.jar
rocksdbjni-3.5.0-osx.jar
rocksdbjni-3.5.0-sources.jar
rocksdbjni-3.5.0.jar
rocksdbjni-x.y.z-javadoc.jar
rocksdbjni-x.y.z-linux32.jar
rocksdbjni-x.y.z-linux64.jar
rocksdbjni-x.y.z-linux64-musl.jar
rocksdbjni-x.y.z-linux32-musl.jar
rocksdbjni-x.y.z-osx.jar
rocksdbjni-x.y.z-sources.jar
rocksdbjni-x.y.z.jar
Where x.y.z is the built version number of RocksDB.
## Maven publication

26
java/crossbuild/Vagrantfile vendored
View File

@ -8,10 +8,28 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.define "linux32" do |linux32|
linux32.vm.box = "bento/centos-6.10-i386"
linux32.vm.provision :shell, path: "build-linux-centos.sh"
end
config.vm.define "linux64" do |linux64|
linux64.vm.box = "bento/centos-6.10"
linux64.vm.provision :shell, path: "build-linux-centos.sh"
end
config.vm.define "linux32-musl" do |musl32|
musl32.vm.box = "alpine/alpine32"
musl32.vm.box_version = "3.6.0"
musl32.vm.provision :shell, path: "build-linux-alpine.sh"
end
config.vm.define "linux64-musl" do |musl64|
musl64.vm.box = "generic/alpine36"
## Should use the alpine/alpine64 box, but this issue needs to be fixed first - https://github.com/hashicorp/vagrant/issues/11218
# musl64.vm.box = "alpine/alpine64"
# musl64.vm.box_version = "3.6.0"
musl64.vm.provision :shell, path: "build-linux-alpine.sh"
end
config.vm.provider "virtualbox" do |v|
@ -20,7 +38,13 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
v.customize ["modifyvm", :id, "--nictype1", "virtio" ]
end
config.vm.provision :shell, path: "build-linux-centos.sh"
if Vagrant.has_plugin?("vagrant-cachier")
config.cache.scope = :box
end
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.no_install = true
end
config.vm.synced_folder "../target", "/rocksdb-build"
config.vm.synced_folder "../..", "/rocksdb", type: "rsync"
config.vm.boot_timeout = 1200

70
java/crossbuild/build-linux-alpine.sh Executable file
View File

@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
set -e
# update Alpine with latest versions
echo '@edge http://nl.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories
echo '@community http://nl.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories
apk update
apk upgrade
# install CA certificates
apk add ca-certificates
# install build tools
apk add \
build-base \
coreutils \
file \
git \
perl \
automake \
autoconf \
cmake
# install tool dependencies for building RocksDB static library
apk add \
curl \
bash \
wget \
tar \
openssl
# install RocksDB dependencies
apk add \
snappy snappy-dev \
zlib zlib-dev \
bzip2 bzip2-dev \
lz4 lz4-dev \
zstd zstd-dev \
linux-headers \
jemalloc jemalloc-dev
# install OpenJDK7
apk add openjdk7 \
&& apk add java-cacerts \
&& rm /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/cacerts \
&& ln -s /etc/ssl/certs/java/cacerts /usr/lib/jvm/java-1.7-openjdk/jre/lib/security/cacerts
# cleanup
rm -rf /var/cache/apk/*
# puts javac in the PATH
export JAVA_HOME=/usr/lib/jvm/java-1.7-openjdk
export PATH=/usr/lib/jvm/java-1.7-openjdk/bin:$PATH
# gflags from source
cd /tmp &&\
git clone -b v2.0 --single-branch https://github.com/gflags/gflags.git &&\
cd gflags &&\
./configure --prefix=/usr && make && make install &&\
rm -rf /tmp/*
# build rocksdb
cd /rocksdb
make jclean clean
PORTABLE=1 make -j8 rocksdbjavastatic
cp /rocksdb/java/target/librocksdbjni-* /rocksdb-build
cp /rocksdb/java/target/rocksdbjni-* /rocksdb-build

18
java/crossbuild/docker-build-linux-alpine.sh Executable file
View File

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
set -e
#set -x
# just in-case this is run outside Docker
mkdir -p /rocksdb-local-build
rm -rf /rocksdb-local-build/*
cp -r /rocksdb-host/* /rocksdb-local-build
cd /rocksdb-local-build
make clean-not-downloaded
PORTABLE=1 make rocksdbjavastatic
cp java/target/librocksdbjni-linux*.so java/target/rocksdbjni-*-linux*.jar /rocksdb-java-target

10
java/crossbuild/docker-build-linux-centos.sh
View File

@ -14,19 +14,19 @@ cd /rocksdb-local-build
# Use scl devtoolset if available
if hash scl 2>/dev/null; then
if scl --list | grep -q 'devtoolset-7'; then
# CentOS 7+
scl enable devtoolset-7 'make clean-not-downloaded'
# CentOS 7+
scl enable devtoolset-7 'make clean-not-downloaded'
scl enable devtoolset-7 'PORTABLE=1 make -j2 rocksdbjavastatic'
elif scl --list | grep -q 'devtoolset-2'; then
# CentOS 5 or 6
scl enable devtoolset-2 'make clean-not-downloaded'
# CentOS 5 or 6
scl enable devtoolset-2 'make clean-not-downloaded'
scl enable devtoolset-2 'PORTABLE=1 make -j2 rocksdbjavastatic'
else
echo "Could not find devtoolset"
exit 1;
fi
else
make clean-not-downloaded
make clean-not-downloaded
PORTABLE=1 make -j2 rocksdbjavastatic
fi

2
java/rocksjni.pom
View File

@ -10,7 +10,7 @@
<artifactId>rocksdbjni</artifactId>
<!-- Version will be automatically replaced -->
<version>-</version>
<description>RocksDB fat jar that contains .so files for linux32 and linux64, jnilib files
<description>RocksDB fat jar that contains .so files for linux32 and linux64 (glibc and musl-libc), jnilib files
for Mac OSX, and a .dll for Windows x64.
</description>
<licenses>

2
java/src/main/java/org/rocksdb/ReadOptions.java
View File

@ -476,7 +476,7 @@ public class ReadOptions extends RocksObject {
*
* The upper bound is exclusive i.e. the bound value is not a valid entry.
*
* If iterator_extractor is not null, the Seek target and iterate_upper_bound
* If prefix_extractor is not null, the Seek target and iterate_upper_bound
* need to have the same prefix. This is because ordering is not guaranteed
* outside of prefix domain.
*

43
java/src/main/java/org/rocksdb/util/Environment.java
View File

@ -1,9 +1,22 @@
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
package org.rocksdb.util;
import java.io.File;
import java.io.IOException;
public class Environment {
private static String OS = System.getProperty("os.name").toLowerCase();
private static String ARCH = System.getProperty("os.arch").toLowerCase();
private static boolean MUSL_LIBC;
static {
try {
final Process p = new ProcessBuilder("/usr/bin/env", "sh", "-c", "ldd /usr/bin/env | grep -q musl").start();
MUSL_LIBC = p.waitFor() == 0;
} catch (final IOException | InterruptedException e) {
MUSL_LIBC = false;
}
}
public static boolean isAarch64() {
return ARCH.contains("aarch64");
@ -38,6 +51,10 @@ public class Environment {
OS.contains("nux");
}
public static boolean isMuslLibc() {
return MUSL_LIBC;
}
public static boolean isSolaris() {
return OS.contains("sunos");
}
@ -61,15 +78,37 @@ public class Environment {
return appendLibOsSuffix("lib" + getSharedLibraryName(name), true);
}
/**
* Get the name of the libc implementation
*
* @return the name of the implementation,
* or null if the default for that platform (e.g. glibc on Linux).
*/
public static /* @Nullable */ String getLibcName() {
if (isMuslLibc()) {
return "musl";
} else {
return null;
}
}
private static String getLibcPostfix() {
final String libcName = getLibcName();
if (libcName == null) {
return "";
}
return "-" + libcName;
}
public static String getJniLibraryName(final String name) {
if (isUnix()) {
final String arch = is64Bit() ? "64" : "32";
if (isPowerPC() || isAarch64()) {
return String.format("%sjni-linux-%s", name, ARCH);
return String.format("%sjni-linux-%s%s", name, ARCH, getLibcPostfix());
} else if (isS390x()) {
return String.format("%sjni-linux%s", name, ARCH);
} else {
return String.format("%sjni-linux%s", name, arch);
return String.format("%sjni-linux%s%s", name, arch, getLibcPostfix());
}
} else if (isMac()) {
return String.format("%sjni-osx", name);

74
java/src/test/java/org/rocksdb/util/EnvironmentTest.java
View File

@ -16,14 +16,17 @@ import static org.assertj.core.api.Assertions.assertThat;
public class EnvironmentTest {
private final static String ARCH_FIELD_NAME = "ARCH";
private final static String OS_FIELD_NAME = "OS";
private final static String MUSL_LIBC_FIELD_NAME = "MUSL_LIBC";
private static String INITIAL_OS;
private static String INITIAL_ARCH;
private static boolean INITIAL_MUSL_LIBC;
@BeforeClass
public static void saveState() {
INITIAL_ARCH = getEnvironmentClassField(ARCH_FIELD_NAME);
INITIAL_OS = getEnvironmentClassField(OS_FIELD_NAME);
INITIAL_MUSL_LIBC = getEnvironmentClassField(MUSL_LIBC_FIELD_NAME);
}
@Test
@ -53,6 +56,7 @@ public class EnvironmentTest {
@Test
public void nix32() {
// Linux
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Linux", "32");
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
@ -61,7 +65,17 @@ public class EnvironmentTest {
isEqualTo("librocksdbjni-linux32.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni.so");
// Linux musl-libc (Alpine)
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, true);
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
isEqualTo(".so");
assertThat(Environment.getJniLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni-linux32-musl.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni.so");
// UNIX
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Unix", "32");
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
@ -84,6 +98,7 @@ public class EnvironmentTest {
@Test
public void nix64() {
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Linux", "x64");
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
@ -92,7 +107,17 @@ public class EnvironmentTest {
isEqualTo("librocksdbjni-linux64.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni.so");
// Linux musl-libc (Alpine)
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, true);
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
isEqualTo(".so");
assertThat(Environment.getJniLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni-linux64-musl.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).
isEqualTo("librocksdbjni.so");
// UNIX
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Unix", "x64");
assertThat(Environment.isWindows()).isFalse();
assertThat(Environment.getJniLibraryExtension()).
@ -130,8 +155,37 @@ public class EnvironmentTest {
isEqualTo("librocksdbjni.dll");
}
@Test
public void ppc64le() {
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Linux", "ppc64le");
assertThat(Environment.isUnix()).isTrue();
assertThat(Environment.isPowerPC()).isTrue();
assertThat(Environment.is64Bit()).isTrue();
assertThat(Environment.getJniLibraryExtension()).isEqualTo(".so");
assertThat(Environment.getSharedLibraryName("rocksdb")).isEqualTo("rocksdbjni");
assertThat(Environment.getJniLibraryName("rocksdb")).isEqualTo("rocksdbjni-linux-ppc64le");
assertThat(Environment.getJniLibraryFileName("rocksdb"))
.isEqualTo("librocksdbjni-linux-ppc64le.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).isEqualTo("librocksdbjni.so");
// Linux musl-libc (Alpine)
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, true);
setEnvironmentClassFields("Linux", "ppc64le");
assertThat(Environment.isUnix()).isTrue();
assertThat(Environment.isPowerPC()).isTrue();
assertThat(Environment.is64Bit()).isTrue();
assertThat(Environment.getJniLibraryExtension()).isEqualTo(".so");
assertThat(Environment.getSharedLibraryName("rocksdb")).isEqualTo("rocksdbjni");
assertThat(Environment.getJniLibraryName("rocksdb")).isEqualTo("rocksdbjni-linux-ppc64le-musl");
assertThat(Environment.getJniLibraryFileName("rocksdb"))
.isEqualTo("librocksdbjni-linux-ppc64le-musl.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).isEqualTo("librocksdbjni.so");
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
}
@Test
public void aarch64() {
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
setEnvironmentClassFields("Linux", "aarch64");
assertThat(Environment.isUnix()).isTrue();
assertThat(Environment.isAarch64()).isTrue();
@ -142,6 +196,19 @@ public class EnvironmentTest {
assertThat(Environment.getJniLibraryFileName("rocksdb"))
.isEqualTo("librocksdbjni-linux-aarch64.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).isEqualTo("librocksdbjni.so");
// Linux musl-libc (Alpine)
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, true);
setEnvironmentClassFields("Linux", "aarch64");
assertThat(Environment.isUnix()).isTrue();
assertThat(Environment.isAarch64()).isTrue();
assertThat(Environment.is64Bit()).isTrue();
assertThat(Environment.getJniLibraryExtension()).isEqualTo(".so");
assertThat(Environment.getSharedLibraryName("rocksdb")).isEqualTo("rocksdbjni");
assertThat(Environment.getJniLibraryName("rocksdb")).isEqualTo("rocksdbjni-linux-aarch64-musl");
assertThat(Environment.getJniLibraryFileName("rocksdb"))
.isEqualTo("librocksdbjni-linux-aarch64-musl.so");
assertThat(Environment.getSharedLibraryFileName("rocksdb")).isEqualTo("librocksdbjni.so");
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, false);
}
private void setEnvironmentClassFields(String osName,
@ -154,9 +221,10 @@ public class EnvironmentTest {
public static void restoreState() {
setEnvironmentClassField(OS_FIELD_NAME, INITIAL_OS);
setEnvironmentClassField(ARCH_FIELD_NAME, INITIAL_ARCH);
setEnvironmentClassField(MUSL_LIBC_FIELD_NAME, INITIAL_MUSL_LIBC);
}
private static String getEnvironmentClassField(String fieldName) {
private static <T> T getEnvironmentClassField(String fieldName) {
final Field field;
try {
field = Environment.class.getDeclaredField(fieldName);
@ -164,13 +232,13 @@ public class EnvironmentTest {
final Field modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
return (String)field.get(null);
return (T)field.get(null);
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new RuntimeException(e);
}
}
private static void setEnvironmentClassField(String fieldName, String value) {
private static void setEnvironmentClassField(String fieldName, Object value) {
final Field field;
try {
field = Environment.class.getDeclaredField(fieldName);

7
port/win/io_win.cc
View File

@ -1025,7 +1025,12 @@ Status WinRandomRWFile::Close() {
//////////////////////////////////////////////////////////////////////////
/// WinMemoryMappedBufer
WinMemoryMappedBuffer::~WinMemoryMappedBuffer() {
BOOL ret = FALSE;
BOOL ret
#if defined(_MSC_VER)
= FALSE;
#else
__attribute__((__unused__));
#endif
if (base_ != nullptr) {
ret = ::UnmapViewOfFile(base_);
assert(ret);

7
port/win/win_thread.cc
View File

@ -138,7 +138,12 @@ void WindowsThread::join() {
"WaitForSingleObjectFailed: thread join");
}
BOOL rc;
BOOL rc
#if defined(_MSC_VER)
= FALSE;
#else
__attribute__((__unused__));
#endif
rc = CloseHandle(reinterpret_cast<HANDLE>(data_->handle_));
assert(rc != 0);
data_->handle_ = 0;

39
table/block_based/block_based_table_builder.cc
View File

@ -62,13 +62,14 @@ namespace {
// Create a filter block builder based on its type.
FilterBlockBuilder* CreateFilterBlockBuilder(
const ImmutableCFOptions& /*opt*/, const MutableCFOptions& mopt,
const BlockBasedTableOptions& table_opt,
const FilterBuildingContext& context,
const bool use_delta_encoding_for_index_values,
PartitionedIndexBuilder* const p_index_builder) {
const BlockBasedTableOptions& table_opt = context.table_options;
if (table_opt.filter_policy == nullptr) return nullptr;
FilterBitsBuilder* filter_bits_builder =
FilterBuildingContext(table_opt).GetBuilder();
BloomFilterPolicy::GetBuilderFromContext(context);
if (filter_bits_builder == nullptr) {
return new BlockBasedFilterBlockBuilder(mopt.prefix_extractor.get(),
table_opt);
@ -345,6 +346,7 @@ struct BlockBasedTableBuilder::Rep {
std::string compressed_output;
std::unique_ptr<FlushBlockPolicy> flush_block_policy;
int level_at_creation;
uint32_t column_family_id;
const std::string& column_family_name;
uint64_t creation_time = 0;
@ -363,9 +365,9 @@ struct BlockBasedTableBuilder::Rep {
const CompressionType _compression_type,
const uint64_t _sample_for_compression,
const CompressionOptions& _compression_opts, const bool skip_filters,
const std::string& _column_family_name, const uint64_t _creation_time,
const uint64_t _oldest_key_time, const uint64_t _target_file_size,
const uint64_t _file_creation_time)
const int _level_at_creation, const std::string& _column_family_name,
const uint64_t _creation_time, const uint64_t _oldest_key_time,
const uint64_t _target_file_size, const uint64_t _file_creation_time)
: ioptions(_ioptions),
moptions(_moptions),
table_options(table_opt),
@ -398,6 +400,7 @@ struct BlockBasedTableBuilder::Rep {
flush_block_policy(
table_options.flush_block_policy_factory->NewFlushBlockPolicy(
table_options, data_block)),
level_at_creation(_level_at_creation),
column_family_id(_column_family_id),
column_family_name(_column_family_name),
creation_time(_creation_time),
@ -419,9 +422,13 @@ struct BlockBasedTableBuilder::Rep {
if (skip_filters) {
filter_builder = nullptr;
} else {
FilterBuildingContext context(table_options);
context.column_family_name = column_family_name;
context.compaction_style = ioptions.compaction_style;
context.level_at_creation = level_at_creation;
filter_builder.reset(CreateFilterBlockBuilder(
_ioptions, _moptions, table_options,
use_delta_encoding_for_index_values, p_index_builder_));
ioptions, moptions, context, use_delta_encoding_for_index_values,
p_index_builder_));
}
for (auto& collector_factories : *int_tbl_prop_collector_factories) {
@ -454,9 +461,9 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
const CompressionType compression_type,
const uint64_t sample_for_compression,
const CompressionOptions& compression_opts, const bool skip_filters,
const std::string& column_family_name, const uint64_t creation_time,
const uint64_t oldest_key_time, const uint64_t target_file_size,
const uint64_t file_creation_time) {
const std::string& column_family_name, const int level_at_creation,
const uint64_t creation_time, const uint64_t oldest_key_time,
const uint64_t target_file_size, const uint64_t file_creation_time) {
BlockBasedTableOptions sanitized_table_options(table_options);
if (sanitized_table_options.format_version == 0 &&
sanitized_table_options.checksum != kCRC32c) {
@ -469,12 +476,12 @@ BlockBasedTableBuilder::BlockBasedTableBuilder(
sanitized_table_options.format_version = 1;
}
rep_ =
new Rep(ioptions, moptions, sanitized_table_options, internal_comparator,
int_tbl_prop_collector_factories, column_family_id, file,
compression_type, sample_for_compression, compression_opts,
skip_filters, column_family_name, creation_time, oldest_key_time,
target_file_size, file_creation_time);
rep_ = new Rep(ioptions, moptions, sanitized_table_options,
internal_comparator, int_tbl_prop_collector_factories,
column_family_id, file, compression_type,
sample_for_compression, compression_opts, skip_filters,
level_at_creation, column_family_name, creation_time,
oldest_key_time, target_file_size, file_creation_time);
if (rep_->filter_builder != nullptr) {
rep_->filter_builder->StartBlock(0);

5
table/block_based/block_based_table_builder.h
View File

@ -47,8 +47,9 @@ class BlockBasedTableBuilder : public TableBuilder {
const CompressionType compression_type,
const uint64_t sample_for_compression,
const CompressionOptions& compression_opts, const bool skip_filters,
const std::string& column_family_name, const uint64_t creation_time = 0,
const uint64_t oldest_key_time = 0, const uint64_t target_file_size = 0,
const std::string& column_family_name, const int level_at_creation,
const uint64_t creation_time = 0, const uint64_t oldest_key_time = 0,
const uint64_t target_file_size = 0,
const uint64_t file_creation_time = 0);
// No copying allowed

2
table/block_based/block_based_table_factory.cc
View File

@ -218,7 +218,7 @@ TableBuilder* BlockBasedTableFactory::NewTableBuilder(
table_builder_options.sample_for_compression,
table_builder_options.compression_opts,
table_builder_options.skip_filters,
table_builder_options.column_family_name,
table_builder_options.column_family_name, table_builder_options.level,
table_builder_options.creation_time,
table_builder_options.oldest_key_time,
table_builder_options.target_file_size,

33
table/block_based/filter_policy.cc
View File

@ -27,7 +27,7 @@ namespace {
// See description in FastLocalBloomImpl
class FastLocalBloomBitsBuilder : public BuiltinFilterBitsBuilder {
public:
FastLocalBloomBitsBuilder(const int millibits_per_key)
explicit FastLocalBloomBitsBuilder(const int millibits_per_key)
: millibits_per_key_(millibits_per_key),
num_probes_(FastLocalBloomImpl::ChooseNumProbes(millibits_per_key_)) {
assert(millibits_per_key >= 1000);
@ -493,17 +493,17 @@ bool BloomFilterPolicy::KeyMayMatch(const Slice& key,
FilterBitsBuilder* BloomFilterPolicy::GetFilterBitsBuilder() const {
// This code path should no longer be used, for the built-in
// BloomFilterPolicy. Internal to RocksDB and outside BloomFilterPolicy,
// only get a FilterBitsBuilder with FilterBuildingContext::GetBuilder(),
// which will call BloomFilterPolicy::GetFilterBitsBuilderInternal.
// RocksDB users have been warned (HISTORY.md) that they can no longer
// call this on the built-in BloomFilterPolicy (unlikely).
// BloomFilterPolicy. Internal to RocksDB and outside
// BloomFilterPolicy, only get a FilterBitsBuilder with
// BloomFilterPolicy::GetBuilderFromContext(), which will call
// BloomFilterPolicy::GetBuilderWithContext(). RocksDB users have
// been warned (HISTORY.md) that they can no longer call this on
// the built-in BloomFilterPolicy (unlikely).
assert(false);
return GetFilterBitsBuilderInternal(
FilterBuildingContext(BlockBasedTableOptions()));
return GetBuilderWithContext(FilterBuildingContext(BlockBasedTableOptions()));
}
FilterBitsBuilder* BloomFilterPolicy::GetFilterBitsBuilderInternal(
FilterBitsBuilder* BloomFilterPolicy::GetBuilderWithContext(
const FilterBuildingContext& context) const {
Mode cur = mode_;
// Unusual code construction so that we can have just
@ -511,7 +511,7 @@ FilterBitsBuilder* BloomFilterPolicy::GetFilterBitsBuilderInternal(
for (int i = 0; i < 2; ++i) {
switch (cur) {
case kAuto:
if (context.table_options_.format_version < 5) {
if (context.table_options.format_version < 5) {
cur = kLegacyBloom;
} else {
cur = kFastLocalBloom;
@ -529,6 +529,15 @@ FilterBitsBuilder* BloomFilterPolicy::GetFilterBitsBuilderInternal(
return nullptr; // something legal
}
FilterBitsBuilder* BloomFilterPolicy::GetBuilderFromContext(
const FilterBuildingContext& context) {
if (context.table_options.filter_policy) {
return context.table_options.filter_policy->GetBuilderWithContext(context);
} else {
return nullptr;
}
}
// Read metadata to determine what kind of FilterBitsReader is needed
// and return a new one.
FilterBitsReader* BloomFilterPolicy::GetFilterBitsReader(
@ -679,6 +688,10 @@ const FilterPolicy* NewBloomFilterPolicy(double bits_per_key,
return new BloomFilterPolicy(bits_per_key, m);
}
FilterBuildingContext::FilterBuildingContext(
const BlockBasedTableOptions& _table_options)
: table_options(_table_options) {}
FilterPolicy::~FilterPolicy() { }
} // namespace rocksdb

38
table/block_based/filter_policy_internal.h
View File

@ -29,24 +29,6 @@ class BuiltinFilterBitsBuilder : public FilterBitsBuilder {
virtual uint32_t CalculateSpace(const int num_entry) = 0;
};
// Current information passed to BloomFilterPolicy at filter building
// time. Subject to change.
struct FilterBuildingContext {
explicit FilterBuildingContext(const BlockBasedTableOptions& table_options)
: table_options_(table_options) {}
// A convenience function to save boilerplate
FilterBitsBuilder* GetBuilder() const {
if (table_options_.filter_policy) {
return table_options_.filter_policy->GetFilterBitsBuilderInternal(*this);
} else {
return nullptr;
}
}
const BlockBasedTableOptions& table_options_;
};
// RocksDB built-in filter policy for Bloom or Bloom-like filters.
// This class is considered internal API and subject to change.
// See NewBloomFilterPolicy.
@ -105,6 +87,18 @@ class BloomFilterPolicy : public FilterPolicy {
FilterBitsBuilder* GetFilterBitsBuilder() const override;
// To use this function, call GetBuilderFromContext().
//
// Neither the context nor any objects therein should be saved beyond
// the call to this function, unless it's shared_ptr.
FilterBitsBuilder* GetBuilderWithContext(
const FilterBuildingContext&) const override;
// Returns a new FilterBitsBuilder from the filter_policy in
// table_options of a context, or nullptr if not applicable.
// (An internal convenience function to save boilerplate.)
static FilterBitsBuilder* GetBuilderFromContext(const FilterBuildingContext&);
// Read metadata to determine what kind of FilterBitsReader is needed
// and return a new one. This must successfully process any filter data
// generated by a built-in FilterBitsBuilder, regardless of the impl
@ -116,14 +110,6 @@ class BloomFilterPolicy : public FilterPolicy {
// Essentially for testing only: legacy whole bits/key
int GetWholeBitsPerKey() const { return whole_bits_per_key_; }
protected:
// To use this function, call FilterBuildingContext::GetBuilder().
//
// Neither the context nor any objects therein should be saved beyond
// the call to this function, unless it's shared_ptr.
FilterBitsBuilder* GetFilterBitsBuilderInternal(
const FilterBuildingContext&) const override;
private:
// Newer filters support fractional bits per key. For predictable behavior
// of 0.001-precision values across floating point implementations, we

18
table/block_based/full_filter_block_test.cc
View File

@ -110,8 +110,7 @@ class PluginFullFilterBlockTest : public mock::MockBlockBasedTableTester,
};
TEST_F(PluginFullFilterBlockTest, PluginEmptyBuilder) {
FullFilterBlockBuilder builder(
nullptr, true, FilterBuildingContext(table_options_).GetBuilder());
FullFilterBlockBuilder builder(nullptr, true, GetBuilder());
Slice slice = builder.Finish();
ASSERT_EQ("", EscapeString(slice));
@ -130,8 +129,7 @@ TEST_F(PluginFullFilterBlockTest, PluginEmptyBuilder) {
}
TEST_F(PluginFullFilterBlockTest, PluginSingleChunk) {
FullFilterBlockBuilder builder(
nullptr, true, FilterBuildingContext(table_options_).GetBuilder());
FullFilterBlockBuilder builder(nullptr, true, GetBuilder());
builder.Add("foo");
builder.Add("bar");
builder.Add("box");
@ -188,8 +186,7 @@ class FullFilterBlockTest : public mock::MockBlockBasedTableTester,
};
TEST_F(FullFilterBlockTest, EmptyBuilder) {
FullFilterBlockBuilder builder(
nullptr, true, FilterBuildingContext(table_options_).GetBuilder());
FullFilterBlockBuilder builder(nullptr, true, GetBuilder());
Slice slice = builder.Finish();
ASSERT_EQ("", EscapeString(slice));
@ -238,8 +235,7 @@ TEST_F(FullFilterBlockTest, DuplicateEntries) {
{ // empty prefixes
std::unique_ptr<const SliceTransform> prefix_extractor(
NewFixedPrefixTransform(0));
auto bits_builder = new CountUniqueFilterBitsBuilderWrapper(
FilterBuildingContext(table_options_).GetBuilder());
auto bits_builder = new CountUniqueFilterBitsBuilderWrapper(GetBuilder());
const bool WHOLE_KEY = true;
FullFilterBlockBuilder builder(prefix_extractor.get(), WHOLE_KEY,
bits_builder);
@ -262,8 +258,7 @@ TEST_F(FullFilterBlockTest, DuplicateEntries) {
// mix of empty and non-empty
std::unique_ptr<const SliceTransform> prefix_extractor(
NewFixedPrefixTransform(7));
auto bits_builder = new CountUniqueFilterBitsBuilderWrapper(
FilterBuildingContext(table_options_).GetBuilder());
auto bits_builder = new CountUniqueFilterBitsBuilderWrapper(GetBuilder());
const bool WHOLE_KEY = true;
FullFilterBlockBuilder builder(prefix_extractor.get(), WHOLE_KEY,
bits_builder);
@ -279,8 +274,7 @@ TEST_F(FullFilterBlockTest, DuplicateEntries) {
}
TEST_F(FullFilterBlockTest, SingleChunk) {
FullFilterBlockBuilder builder(
nullptr, true, FilterBuildingContext(table_options_).GetBuilder());
FullFilterBlockBuilder builder(nullptr, true, GetBuilder());
ASSERT_EQ(0, builder.NumAdded());
builder.Add("foo");
builder.Add("bar");

20
table/block_based/mock_block_based_table.h
View File

@ -4,9 +4,10 @@
// (found in the LICENSE.Apache file in the root directory).
#pragma once
#include "table/block_based/block_based_filter_block.h"
#include "rocksdb/filter_policy.h"
#include "table/block_based/block_based_filter_block.h"
#include "table/block_based/block_based_table_reader.h"
#include "table/block_based/filter_policy_internal.h"
namespace rocksdb {
namespace mock {
@ -18,6 +19,8 @@ class MockBlockBasedTable : public BlockBasedTable {
};
class MockBlockBasedTableTester {
static constexpr int kMockLevel = 0;
public:
Options options_;
ImmutableCFOptions ioptions_;
@ -33,11 +36,18 @@ class MockBlockBasedTableTester {
table_options_.filter_policy.reset(filter_policy);
constexpr bool skip_filters = false;
constexpr int level = 0;
constexpr bool immortal_table = false;
table_.reset(new MockBlockBasedTable(
new BlockBasedTable::Rep(ioptions_, env_options_, table_options_,
icomp_, skip_filters, level, immortal_table)));
table_.reset(new MockBlockBasedTable(new BlockBasedTable::Rep(
ioptions_, env_options_, table_options_, icomp_, skip_filters,
kMockLevel, immortal_table)));
}
FilterBitsBuilder* GetBuilder() const {
FilterBuildingContext context(table_options_);
context.column_family_name = "mock_cf";
context.compaction_style = ioptions_.compaction_style;
context.level_at_creation = kMockLevel;
return BloomFilterPolicy::GetBuilderFromContext(context);
}
};

3
table/block_based/partitioned_filter_block_test.cc
View File

@ -126,7 +126,8 @@ class PartitionedFilterBlockTest
const bool kValueDeltaEncoded = true;
return new PartitionedFilterBlockBuilder(
prefix_extractor, table_options_.whole_key_filtering,
FilterBuildingContext(table_options_).GetBuilder(),
BloomFilterPolicy::GetBuilderFromContext(
FilterBuildingContext(table_options_)),
table_options_.index_block_restart_interval, !kValueDeltaEncoded,
p_index_builder, partition_size);
}

17
tools/block_cache_analyzer/block_cache_trace_analyzer.cc
View File

@ -14,6 +14,7 @@
#include <fstream>
#include <iomanip>
#include <iostream>
#include <memory>
#include <random>
#include <sstream>
@ -650,7 +651,7 @@ void BlockCacheTraceAnalyzer::WriteCorrelationFeaturesToFile(
const std::map<std::string, Features>& label_features,
const std::map<std::string, Predictions>& label_predictions,
uint32_t max_number_of_values) const {
std::default_random_engine rand_engine(static_cast<unsigned int>(env_->NowMicros()));
std::default_random_engine rand_engine(static_cast<std::default_random_engine::result_type>(env_->NowMicros()));
for (auto const& label_feature_vectors : label_features) {
const Features& past = label_feature_vectors.second;
auto it = label_predictions.find(label_feature_vectors.first);
@ -1170,7 +1171,7 @@ void BlockCacheTraceAnalyzer::WriteReuseLifetime(
}
void BlockCacheTraceAnalyzer::WriteBlockReuseTimeline(
uint64_t reuse_window, bool user_access_only, TraceType block_type) const {
const uint64_t reuse_window, bool user_access_only, TraceType block_type) const {
// A map from block key to an array of bools that states whether a block is
// accessed in a time window.
std::map<uint64_t, std::vector<bool>> block_accessed;
@ -1209,11 +1210,11 @@ void BlockCacheTraceAnalyzer::WriteBlockReuseTimeline(
TraverseBlocks(block_callback);
// A cell is the number of blocks accessed in a reuse window.
uint64_t reuse_table[reuse_vector_size][reuse_vector_size];
std::unique_ptr<uint64_t[]> reuse_table(new uint64_t[reuse_vector_size * reuse_vector_size]);
for (uint64_t start_time = 0; start_time < reuse_vector_size; start_time++) {
// Initialize the reuse_table.
for (uint64_t i = 0; i < reuse_vector_size; i++) {
reuse_table[start_time][i] = 0;
reuse_table[start_time * reuse_vector_size + i] = 0;
}
// Examine all blocks.
for (auto const& block : block_accessed) {
@ -1222,7 +1223,7 @@ void BlockCacheTraceAnalyzer::WriteBlockReuseTimeline(
// This block is accessed at start time and at the current time. We
// increment reuse_table[start_time][i] since it is reused at the ith
// window.
reuse_table[start_time][i]++;
reuse_table[start_time * reuse_vector_size + i]++;
}
}
}
@ -1250,8 +1251,8 @@ void BlockCacheTraceAnalyzer::WriteBlockReuseTimeline(
if (j < start_time) {
row += "100.0";
} else {
row += std::to_string(percent(reuse_table[start_time][j],
reuse_table[start_time][start_time]));
row += std::to_string(percent(reuse_table[start_time * reuse_vector_size + j],
reuse_table[start_time * reuse_vector_size + start_time]));
}
}
out << row << std::endl;
@ -1673,7 +1674,7 @@ void BlockCacheTraceAnalyzer::PrintAccessCountStats(bool user_access_only,
if (bottom_k_index >= bottom_k) {
break;
}
std::map<TableReaderCaller, uint32_t> caller_naccesses;
std::map<TableReaderCaller, uint64_t> caller_naccesses;
uint64_t naccesses = 0;
for (auto const& block_id : naccess_it->second) {
BlockAccessInfo* block = block_info_map_.find(block_id)->second;

2
tools/block_cache_analyzer/block_cache_trace_analyzer.h
View File

@ -289,7 +289,7 @@ class BlockCacheTraceAnalyzer {
// The file is named
// "block_type_user_access_only_reuse_window_reuse_timeline". The file format
// is start_time,0,1,...,N where N equals trace_duration / reuse_window.
void WriteBlockReuseTimeline(uint64_t reuse_window, bool user_access_only,
void WriteBlockReuseTimeline(const uint64_t reuse_window, bool user_access_only,
TraceType block_type) const;
// Write the Get spatical locality into csv files saved in 'output_dir'.

2
tools/db_crashtest.py
View File

@ -66,7 +66,7 @@ default_params = {
"verify_checksum": 1,
"write_buffer_size": 4 * 1024 * 1024,
"writepercent": 35,
"format_version": lambda: random.randint(2, 4),
"format_version": lambda: random.choice([2, 3, 4, 5, 5]),
"index_block_restart_interval": lambda: random.choice(range(1, 16)),
"use_multiget" : lambda: random.randint(0, 1),
"periodic_compaction_seconds" :

3
util/bloom_test.cc
View File

@ -276,7 +276,8 @@ class FullBloomTest : public testing::TestWithParam<BloomFilterPolicy::Mode> {
}
void Reset() {
bits_builder_.reset(FilterBuildingContext(table_options_).GetBuilder());
bits_builder_.reset(BloomFilterPolicy::GetBuilderFromContext(
FilterBuildingContext(table_options_)));
bits_reader_.reset(nullptr);
buf_.reset(nullptr);
filter_size_ = 0;

2
util/filter_bench.cc
View File

@ -280,7 +280,7 @@ void FilterBench::Go() {
std::unique_ptr<FilterBitsBuilder> builder;
if (!FLAGS_use_plain_table_bloom && FLAGS_impl != 1) {
builder.reset(FilterBuildingContext(table_options_).GetBuilder());
builder.reset(GetBuilder());
}
uint32_t variance_mask = 1;

7
util/status.cc
View File

@ -21,7 +21,12 @@ const char* Status::CopyState(const char* state) {
#ifdef OS_WIN
const size_t cch = std::strlen(state) + 1; // +1 for the null terminator
char* result = new char[cch];
errno_t ret;
errno_t ret
#if defined(_MSC_VER)
;
#else
__attribute__((__unused__));
#endif
ret = strncpy_s(result, cch, state, cch - 1);
result[cch - 1] = '\0';
assert(ret == 0);

4
utilities/blob_db/blob_db_impl.cc
View File

@ -975,7 +975,7 @@ Status BlobDBImpl::PutBlobValue(const WriteOptions& /*options*/,
}
if (s.ok()) {
assert(blob_file != nullptr);
assert(blob_file->compression() == bdb_options_.compression);
assert(blob_file->GetCompressionType() == bdb_options_.compression);
s = AppendBlob(blob_file, headerbuf, key, value_compressed, expiration,
&index_entry);
}
@ -1364,7 +1364,7 @@ Status BlobDBImpl::GetRawBlobFromFile(const Slice& key, uint64_t file_number,
blob_file = it->second;
}
*compression_type = blob_file->compression();
*compression_type = blob_file->GetCompressionType();
// takes locks when called
std::shared_ptr<RandomAccessFileReader> reader;

2
utilities/blob_db/blob_file.cc
View File

@ -54,7 +54,7 @@ BlobFile::~BlobFile() {
}
}
uint32_t BlobFile::column_family_id() const { return column_family_id_; }
uint32_t BlobFile::GetColumnFamilyId() const { return column_family_id_; }
std::string BlobFile::PathName() const {
return BlobFileName(path_to_dir_, file_number_);

4
utilities/blob_db/blob_file.h
View File

@ -116,7 +116,7 @@ class BlobFile {
~BlobFile();
uint32_t column_family_id() const;
uint32_t GetColumnFamilyId() const;
// Returns log file's absolute pathname.
std::string PathName() const;
@ -204,7 +204,7 @@ class BlobFile {
void SetHasTTL(bool has_ttl) { has_ttl_ = has_ttl; }
CompressionType compression() const { return compression_; }
CompressionType GetCompressionType() const { return compression_; }
std::shared_ptr<Writer> GetWriter() const { return log_writer_; }

2
utilities/transactions/transaction_base.cc
View File

@ -323,7 +323,7 @@ void TransactionBaseImpl::MultiGet(const ReadOptions& read_options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
bool sorted_input) {
const bool sorted_input) {
write_batch_.MultiGetFromBatchAndDB(db_, read_options, column_family,
num_keys, keys, values, statuses,
sorted_input);

2
utilities/transactions/transaction_base.h
View File

@ -98,7 +98,7 @@ class TransactionBaseImpl : public Transaction {
void MultiGet(const ReadOptions& options, ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys, PinnableSlice* values,
Status* statuses, bool sorted_input = false) override;
Status* statuses, const bool sorted_input = false) override;
using Transaction::MultiGetForUpdate;
std::vector<Status> MultiGetForUpdate(

2
utilities/transactions/write_prepared_txn.cc
View File

@ -44,7 +44,7 @@ void WritePreparedTxn::MultiGet(const ReadOptions& options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
bool sorted_input) {
const bool sorted_input) {
SequenceNumber min_uncommitted, snap_seq;
const SnapshotBackup backed_by_snapshot =
wpt_db_->AssignMinMaxSeqs(options.snapshot, &min_uncommitted, &snap_seq);

2
utilities/transactions/write_prepared_txn.h
View File

@ -61,7 +61,7 @@ class WritePreparedTxn : public PessimisticTransaction {
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
bool sorted_input = false) override;
const bool sorted_input = false) override;
// Note: The behavior is undefined in presence of interleaved writes to the
// same transaction.

2
utilities/transactions/write_unprepared_txn.cc
View File

@ -837,7 +837,7 @@ void WriteUnpreparedTxn::MultiGet(const ReadOptions& options,
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
bool sorted_input) {
const bool sorted_input) {
SequenceNumber min_uncommitted, snap_seq;
const SnapshotBackup backed_by_snapshot =
wupt_db_->AssignMinMaxSeqs(options.snapshot, &min_uncommitted, &snap_seq);

2
utilities/transactions/write_unprepared_txn.h
View File

@ -194,7 +194,7 @@ class WriteUnpreparedTxn : public WritePreparedTxn {
ColumnFamilyHandle* column_family,
const size_t num_keys, const Slice* keys,
PinnableSlice* values, Status* statuses,
bool sorted_input = false) override;
const bool sorted_input = false) override;
using Transaction::GetIterator;
virtual Iterator* GetIterator(const ReadOptions& options) override;