Commit Graph

9764 Commits

Author SHA1 Message Date
Zhichao Cao
d1c510baec Handoff checksum Implementation (#7523)
Summary:
in PR https://github.com/facebook/rocksdb/issues/7419 , we introduce the new Append and PositionedAppend APIs to WritableFile at File System, which enable RocksDB to pass the data verification information (e.g., checksum of the data) to the lower layer. In this PR, we use the new API in WritableFileWriter, such that the file created via WritableFileWrite can pass the checksum to the storage layer. To control which types file should apply the checksum handoff, we add checksum_handoff_file_types to DBOptions. User can use this option to control which file types (Currently supported file tyes: kLogFile, kTableFile, kDescriptorFile.) should use the new Append and PositionedAppend APIs to handoff the verification information.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7523

Test Plan: add new unit test, pass make check/ make asan_check

Reviewed By: pdillinger

Differential Revision: D24313271

Pulled By: zhichao-cao

fbshipit-source-id: aafd69091ae85c3318e3e17cbb96fe7338da11d0
2021-02-10 22:20:32 -08:00
Peter Dillinger
e4f1e64c30 Add prefetching (batched MultiGet) for experimental Ribbon filter (#7889)
Summary:
Adds support for prefetching data in Ribbon queries,
which especially optimizes batched Ribbon queries for MultiGet
(~222ns/key to ~97ns/key) but also single key queries on cold memory
(~333ns to ~226ns) because many queries span more than one cache line.

This required some refactoring of the query algorithm, and there
does not appear to be a noticeable regression in "hot memory" query
times (perhaps from 48ns to 50ns).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7889

Test Plan:
existing unit tests, plus performance validation with
filter_bench:

Each data point is the best of two runs. I saturated the machine
CPUs with other filter_bench runs in the background.

Before:

    $ ./filter_bench -impl=3 -m_keys_total_max=200 -average_keys_per_filter=100000 -m_queries=50
    WARNING: Assertions are enabled; benchmarks unnecessarily slow
    Building...
    Build avg ns/key: 125.86
    Number of filters: 1993
    Total size (MB): 168.166
    Reported total allocated memory (MB): 183.211
    Reported internal fragmentation: 8.94626%
    Bits/key stored: 7.05341
    Prelim FP rate %: 0.951827
    ----------------------------
    Mixed inside/outside queries...
      Single filter net ns/op: 48.0111
      Batched, prepared net ns/op: 222.384
      Batched, unprepared net ns/op: 343.908
      Skewed 50% in 1% net ns/op: 252.916
      Skewed 80% in 20% net ns/op: 320.579
      Random filter net ns/op: 332.957

After:

    $ ./filter_bench -impl=3 -m_keys_total_max=200 -average_keys_per_filter=100000 -m_queries=50
    WARNING: Assertions are enabled; benchmarks unnecessarily slow
    Building...
    Build avg ns/key: 128.117
    Number of filters: 1993
    Total size (MB): 168.166
    Reported total allocated memory (MB): 183.211
    Reported internal fragmentation: 8.94626%
    Bits/key stored: 7.05341
    Prelim FP rate %: 0.951827
    ----------------------------
    Mixed inside/outside queries...
      Single filter net ns/op: 49.8812
      Batched, prepared net ns/op: 97.1514
      Batched, unprepared net ns/op: 222.025
      Skewed 50% in 1% net ns/op: 197.48
      Skewed 80% in 20% net ns/op: 212.457
      Random filter net ns/op: 226.464

Bloom comparison, for reference:

    $ ./filter_bench -impl=2 -m_keys_total_max=200 -average_keys_per_filter=100000 -m_queries=50
    WARNING: Assertions are enabled; benchmarks unnecessarily slow
    Building...
    Build avg ns/key: 35.3042
    Number of filters: 1993
    Total size (MB): 238.488
    Reported total allocated memory (MB): 262.875
    Reported internal fragmentation: 10.2255%
    Bits/key stored: 10.0029
    Prelim FP rate %: 0.965327
    ----------------------------
    Mixed inside/outside queries...
      Single filter net ns/op: 9.09931
      Batched, prepared net ns/op: 34.21
      Batched, unprepared net ns/op: 88.8564
      Skewed 50% in 1% net ns/op: 139.75
      Skewed 80% in 20% net ns/op: 181.264
      Random filter net ns/op: 173.88

Reviewed By: jay-zhuang

Differential Revision: D26378710

Pulled By: pdillinger

fbshipit-source-id: 058428967c55ed763698284cd3b4bbe3351b6e69
2021-02-10 21:04:56 -08:00
David CARLIER
14fbb43f3e db_bench: dump cpu info for Mac. (#7932)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7932

Reviewed By: jay-zhuang

Differential Revision: D26316480

Pulled By: zhichao-cao

fbshipit-source-id: 3e002e49fcb7f60bc9270550a6b3e182fe197551
2021-02-10 12:56:44 -08:00
Xavier Deguillard
7ebde3da45 Build a full RocksDB on M1 macs (#7943)
Summary:
With M1 macs being available, it is possible that RocksDB will be built on them, without the resulting artifacts to be intended for iOS, where a non-lite RocksDB is needed.

It is not clear to me why the ROCKSDB_LITE cmake option isn't used for iOS consumer, so sending this pull request as a way to foster discussion and to find a path forward to get a full RocksDB build on M1.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7943

Test Plan:
Applied the following patch:
```
 diff --git a/fbcode/opensource/fbcode_builder/manifests/rocksdb b/fbcode/opensource/fbcode_builder/manifests/rocksdb
 --- a/fbcode/opensource/fbcode_builder/manifests/rocksdb
+++ b/fbcode/opensource/fbcode_builder/manifests/rocksdb
@@ -2,8 +2,8 @@
 name = rocksdb

 [download]
-url = https://github.com/facebook/rocksdb/archive/v6.8.1.tar.gz
-sha256 = ca192a06ed3bcb9f09060add7e9d0daee1ae7a8705a3d5ecbe41867c5e2796a2
+url = https://github.com/xavierd/rocksdb/archive/master.zip
+sha256 = f93f3f92df66a8401659e35398749d5910b92bd9c14b8354a35ea8852865c422

 [dependencies]
 lz4
@@ -11,7 +11,7 @@

 [build]
 builder = cmake
-subdir = rocksdb-6.8.1
+subdir = rocksdb-master

 [cmake.defines]
 WITH_SNAPPY=ON
```

And ran `getdeps build eden` on an M1 macbook. The build used to fail at link time due to some RocksDB symbols not being found, it now fails for another reason (x86_64 Rust symbols).

Reviewed By: jay-zhuang

Differential Revision: D26324049

Pulled By: xavierd

fbshipit-source-id: 12d86f3395709c4c323f440844e3ae65672aef2d
2021-02-10 10:13:59 -08:00
Yanqin Jin
170dffac83 Use actual url instead of tinyurl.com (#7950)
Summary:
Due to offline discussion, we use actual url of the clang-format-diff.py and add a note.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7950

Reviewed By: pdillinger

Differential Revision: D26370822

Pulled By: riversand963

fbshipit-source-id: 7508e23c002d56d5c1649090438ef5f8ff2cdbe7
2021-02-10 10:08:09 -08:00
Andrew Kryczka
c16d5a4fda Makefile support to statically link external plugin code (#7918)
Summary:
Added support for detecting plugins linked in the "plugin/" directory and building them from our Makefile in a standardized way. See "plugin/README.md" for details. An example of a plugin that can be built in this way can be found in https://github.com/ajkr/dedupfs.

There will be more to do in terms of making this process more convenient and adding support for CMake.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7918

Test Plan: my own plugin (https://github.com/ajkr/dedupfs) and also heard this patch worked with ZenFS.

Reviewed By: pdillinger

Differential Revision: D26189969

Pulled By: ajkr

fbshipit-source-id: 6624d4357d0ffbaedb42f0d12a3fcb737c78f758
2021-02-10 08:35:34 -08:00
Jay Zhuang
65487da1f5 Add circleci format_compatible nightly build (#7926)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7926

Test Plan: manual trigger the test: https://app.circleci.com/pipelines/github/facebook/rocksdb/5718/workflows/0427934c-7629-4e1e-a523-1fa1ab8e8f59/jobs/91413

Reviewed By: pdillinger

Differential Revision: D26238070

Pulled By: jay-zhuang

fbshipit-source-id: 5f6ca75d922a88f9e8c8b17d6d7b026506ff8638
2021-02-09 20:48:53 -08:00
Yanqin Jin
48669be618 Update clang-format-diff.py path (#7944)
Summary:
Recent Github actions of format checking fail due to invalid location
from where clang-format-diff.py is downloaded. Update the path to point
to a stable, archived location.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7944

Test Plan: manually check the result of Github action.

Reviewed By: ltamasi

Differential Revision: D26345066

Pulled By: riversand963

fbshipit-source-id: 2b1a58c2e59c2f1eb11202d321d2ea002cb0917e
2021-02-09 12:49:38 -08:00
Peter Dillinger
91e2c151d2 Update zstd in buck build (#7923)
Summary:
Needed for internal tooling update

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7923

Test Plan: internal tests

Reviewed By: jay-zhuang

Differential Revision: D26203911

Pulled By: pdillinger

fbshipit-source-id: 223824c37e883285cbf041f52902ea60504afe72
2021-02-08 14:46:01 -08:00
Jay Zhuang
cf160b98e1 Add full_history_ts_low option to compaction (#7884)
Summary:
The full_history_ts_low is used for user-defined timestamp GC
compaction, which is introduced in https://github.com/facebook/rocksdb/issues/7740, https://github.com/facebook/rocksdb/issues/7657 and https://github.com/facebook/rocksdb/issues/7655.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7884

Reviewed By: ltamasi

Differential Revision: D25982553

Pulled By: jay-zhuang

fbshipit-source-id: 36303d412d65b5d8166b6da24fa21ad85adbabee
2021-02-08 13:45:48 -08:00
Levi Tamasi
974458891c Revert "Turn on memtable bloom filter by default. (#6584)" (#7939)
Summary:
This reverts commit ee79a28963.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7939

Reviewed By: siying

Differential Revision: D26298564

Pulled By: ltamasi

fbshipit-source-id: 6d663516e82e6de436f8d5317932ca9a98e152bd
2021-02-06 22:34:30 -08:00
sdong
e3183eae77 Stress test to allow memtable whole key filter (#7937)
Summary:
Right now, stress test cannot be configured to use memtable whole key filter without prefix filter. It doesn't appear to be necessary. remove this constraint.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7937

Test Plan: "make crash_test" to be able to run.

Reviewed By: ltamasi

Differential Revision: D26295532

fbshipit-source-id: 30c874a9dc2b672a460603a4ee32368674e0face
2021-02-05 22:50:45 -08:00
Andrew Kryczka
8d2bbdd04f Allow range deletions in *TransactionDB only when safe (#7929)
Summary:
Explicitly reject all range deletions on `TransactionDB` or `OptimisticTransactionDB`, except when the user provides sufficient promises that allow us to proceed safely. The necessary promises are described in the API doc for `TransactionDB::DeleteRange()`. There is currently no way to provide enough promises to make it safe in `OptimisticTransactionDB`.

Fixes https://github.com/facebook/rocksdb/issues/7913.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7929

Test Plan: unit tests covering the cases it's permitted/rejected

Reviewed By: ltamasi

Differential Revision: D26240254

Pulled By: ajkr

fbshipit-source-id: 2834a0ce64cc3e4c3799e35b885a5e79c2f4f6d9
2021-02-05 15:57:26 -08:00
Levi Tamasi
4fba83b4c2 Fix db_bench_tool_test (#7935)
Summary:
The patch fixes the build for `db_bench_tool_test` and makes the tests pass.
Namely, it fixes the following issues:

* https://github.com/facebook/rocksdb/issues/7703 removed the member variable `fs_` but the test case `OptionsFileMultiLevelUniversal`
was not updated.
* https://github.com/facebook/rocksdb/issues/7344 fixed the `OptionsFile` test case for the case when Snappy is *not* available but at the
same time broke it for the case when it *is* available. (The test used a default-constructed
`ColumnFamilyOptions` object, and the default value of the `compression` option is either
Snappy or no compression depending on whether Snappy is supported.)
* The test used `google::ParseCommandLineFlags` instead of
`GFLAGS_NAMESPACE::ParseCommandLineFlags`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7935

Test Plan: Ran the test both with and without Snappy support.

Reviewed By: zhichao-cao

Differential Revision: D26269765

Pulled By: ltamasi

fbshipit-source-id: b7303d8a981ab299d22ab540e0cbd12d149ed9bb
2021-02-05 15:41:48 -08:00
sdong
ee79a28963 Turn on memtable bloom filter by default. (#6584)
Summary:
Memtable bloom filter is useful in many use cases. A default value on with conservative 1.5% memory can benefit more use cases than use cases impacted.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6584

Test Plan: Run all existing tests.

Reviewed By: pdillinger

Differential Revision: D20626739

fbshipit-source-id: 1dd45532b932139552519b8c2682bd954550c2f9
2021-02-05 12:59:46 -08:00
Deepak Ojha
cbf38af705 Wal recovery failure with encryption due to zero bytes WAL size. (#7924)
Summary:
Fixes issue https://github.com/facebook/rocksdb/issues/7410

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7924

Reviewed By: jay-zhuang

Differential Revision: D26216805

Pulled By: ajkr

fbshipit-source-id: 5720635ee06003269bfaf8c8616b3671974b08a8
2021-02-05 12:40:52 -08:00
Stanislav Tkach
3feee6db17 Add get/set deadline and io_timeout C functions (read options) (#7914)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7914

Reviewed By: jay-zhuang

Differential Revision: D26184409

Pulled By: ajkr

fbshipit-source-id: 8e30faac5223ec80c22e2b617af67775322065d8
2021-02-04 17:00:58 -08:00
Jay Zhuang
a9f8f1fe38 Missed skip_format_check for lite build (#7927)
Summary:
Missed lite build in https://github.com/facebook/rocksdb/issues/7911.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7927

Test Plan: Manual build: https://www.internalfb.com/intern/sandcastle/job/4503600004036336

Reviewed By: ltamasi

Differential Revision: D26226135

Pulled By: jay-zhuang

fbshipit-source-id: e9c51a8f4b6d5469521e0d788ead971a1c1b48ff
2021-02-03 10:39:27 -08:00
Levi Tamasi
0288bdbc53 Add the integrated BlobDB to the stress/crash tests (#7900)
Summary:
The patch adds support for the options related to the new BlobDB implementation
to `db_stress`, including support for dynamically adjusting them using `SetOptions`
when `set_options_one_in` and a new flag `allow_setting_blob_options_dynamically`
are specified. (The latter is used to prevent the options from being enabled when
incompatible features are in use.)

The patch also updates the `db_stress` help messages of the existing stacked BlobDB
related options to clarify that they pertain to the old implementation. In addition, it
adds the new BlobDB to the crash test script. In order to prevent a combinatorial explosion
of jobs and still perform whitebox/blackbox testing (including under ASAN/TSAN/UBSAN),
and to also test BlobDB in conjunction with atomic flush and transactions, the script sets
the BlobDB options in 10% of normal/`cf_consistency`/`txn` crash test runs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7900

Test Plan: Ran `make check` and `db_stress`/`db_crashtest.py` with various options.

Reviewed By: jay-zhuang

Differential Revision: D26094913

Pulled By: ltamasi

fbshipit-source-id: c2ef3391a05e43a9687f24e297df05f4a5584814
2021-02-02 11:41:18 -08:00
Zhichao Cao
108e6b6354 Return Status::OK for unimplemented write batch handler in trace analyzer (#7910)
Summary:
The unimplemented handler will return Status::InvalidArgument() and caused issues when using trace analyzer for write batch record. Override with returning Status::OK()

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7910

Test Plan: tested with real trace, make check

Reviewed By: siying

Differential Revision: D26154327

Pulled By: zhichao-cao

fbshipit-source-id: bcdefd4891f839b2e89e4c079f9f430245f482fb
2021-02-02 10:52:28 -08:00
mrambacher
21218316da Fix build_version.cc generation error if GIT not found (#7916)
Summary:
(Fixes a regression introduced in the build_version generation PR https://github.com/facebook/rocksdb/issues/7866 )

In the Makefile case, needed to ignore stderr on the tag (everywhere else was fine).

In the CMAKE case, no GIT implies "changes" so that we use the system date rather than the empty GIT date.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7916

Test Plan: Built in a tree that did not contain the ".git" directory.  Validated that no errors appeared during the build process and that the build version date was not empty.

Reviewed By: jay-zhuang

Differential Revision: D26169203

Pulled By: mrambacher

fbshipit-source-id: 3288a23b48d97efed5e5b38c9aefb3ef1153fa16
2021-02-01 21:05:54 -08:00
Xiaopeng Zhang
bf6795aea0 fix java sample typo and replace deprecated code with latest (#7906)
Summary:
1. replace deprecated code in sample java with latest api
2. fix optimistictransaction sample code typo

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7906

Reviewed By: ajkr

Differential Revision: D26127429

Pulled By: jay-zhuang

fbshipit-source-id: f015ad1435f565cffb8798a4fb5afc44c72d73d7
2021-02-01 14:45:34 -08:00
Jay Zhuang
87983d442c Skip format checks for sandcastle build (#7911)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7911

Test Plan:
Manually tested most of the builds:
* asan: https://www.internalfb.com/intern/sandcastle/job/9007199627266604
* asan_crash: https://www.internalfb.com/intern/sandcastle/group/nonce/827960152876428/
* asan_crash_with_atomic_flush: https://www.internalfb.com/intern/sandcastle/group/nonce/6901727142256449/
* asan_crash_with_txn https://www.internalfb.com/intern/sandcastle/group/nonce/5745312080630484/
* blackbox_asan_crash https://www.internalfb.com/intern/sandcastle/group/nonce/939185737271884/
* clang_analyze https://www.internalfb.com/intern/sandcastle/job/31525197767943703
* code_cov https://www.internalfb.com/intern/sandcastle/job/376027170
* format_compatible https://www.internalfb.com/intern/sandcastle/group/nonce/6238410205251033/
* no_compression https://www.internalfb.com/intern/sandcastle/group/nonce/5901170225087670/
* regression https://www.internalfb.com/intern/sandcastle/group/nonce/6050737823326726/
* release https://www.internalfb.com/intern/sandcastle/group/nonce/528162753321938/
* stress_crash_with_txn https://www.internalfb.com/intern/sandcastle/group/nonce/4473519221616524/
* tsan_crash_with_atomic_flush https://www.internalfb.com/intern/sandcastle/group/nonce/6950713723315225/
* ubsan_crash https://www.internalfb.com/intern/sandcastle/group/nonce/8824481248569096/
* unit_non_shm https://www.internalfb.com/intern/sandcastle/group/nonce/3373704090128605/
* unity  https://www.internalfb.com/intern/sandcastle/group/nonce/832911990656833/
* whitebox_stress_crash https://www.internalfb.com/intern/sandcastle/group/nonce/3698026167778822/

Reviewed By: pdillinger

Differential Revision: D26165859

Pulled By: jay-zhuang

fbshipit-source-id: d1e7dfd024b82d67b1c291f87216d7fdfc01856a
2021-02-01 09:17:54 -08:00
Levi Tamasi
e5311a8ea4 Fix a SingleDelete related optimization for blob indexes (#7904)
Summary:
There is a small `SingleDelete` related optimization in the
`CompactionIterator` code: when a `SingleDelete`-`Put` pair is preserved
solely for the purposes of transaction conflict checking, the value
itself gets cleared. (This is referred to as "optimization 3" in the
`CompactionIterator` code.) Though the rest of the code got updated to
support `SingleDelete`'ing blob indexes, this chunk was apparently
missed, resulting in an assertion failure (or `ROCKS_LOG_FATAL` in release
builds) when triggered. Note: in addition to clearing the value, we also
need to update the type of the KV to regular value when dealing with
blob indexes here.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7904

Test Plan: `make check`

Reviewed By: ajkr

Differential Revision: D26118009

Pulled By: ltamasi

fbshipit-source-id: 6bf78043d20265e2b15c2e1ab8865025040c42ae
2021-01-29 12:41:25 -08:00
Andrew Kryczka
78ee8564ad Integrity protection for live updates to WriteBatch (#7748)
Summary:
This PR adds the foundation classes for key-value integrity protection and the first use case: protecting live updates from the source buffers added to `WriteBatch` through the destination buffer in `MemTable`. The width of the protection info is not yet configurable -- only eight bytes per key is supported. This PR allows users to enable protection by constructing `WriteBatch` with `protection_bytes_per_key == 8`. It does not yet expose a way for users to get integrity protection via other write APIs (e.g., `Put()`, `Merge()`, `Delete()`, etc.).

The foundation classes (`ProtectionInfo.*`) embed the coverage info in their type, and provide `Protect.*()` and `Strip.*()` functions to navigate between types with different coverage. For making bytes per key configurable (for powers of two up to eight) in the future, these classes are templated on the unsigned integer type used to store the protection info. That integer contains the XOR'd result of hashes with independent seeds for all covered fields. For integer fields, the hash is computed on the raw unadjusted bytes, so the result is endian-dependent. The most significant bytes are truncated when the hash value (8 bytes) is wider than the protection integer.

When `WriteBatch` is constructed with `protection_bytes_per_key == 8`, we hold a `ProtectionInfoKVOTC` (i.e., one that covers key, value, optype aka `ValueType`, timestamp, and CF ID) for each entry added to the batch. The protection info is generated from the original buffers passed by the user, as well as the original metadata generated internally. When writing to memtable, each entry is transformed to a `ProtectionInfoKVOTS` (i.e., dropping coverage of CF ID and adding coverage of sequence number), since at that point we know the sequence number, and have already selected a memtable corresponding to a particular CF. This protection info is verified once the entry is encoded in the `MemTable` buffer.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7748

Test Plan:
- an integration test to verify a wide variety of single-byte changes to the encoded `MemTable` buffer are caught
- add to stress/crash test to verify it works in variety of configs/operations without intentional corruption
- [deferred] unit tests for `ProtectionInfo.*` classes for edge cases like KV swap, `SliceParts` and `Slice` APIs are interchangeable, etc.

Reviewed By: pdillinger

Differential Revision: D25754492

Pulled By: ajkr

fbshipit-source-id: e481bac6c03c2ab268be41359730f1ceb9964866
2021-01-29 12:18:58 -08:00
mrambacher
4a09d632c4 Remove Legacy and Custom FileWrapper classes from header files (#7851)
Summary:
Removed the uses of the Legacy FileWrapper classes from the source code.  The wrappers were creating an additional layer of indirection/wrapping, as the Env already has a FileSystem.

Moved the Custom FileWrapper classes into the CustomEnv, as these classes are really for the private use the the CustomEnv class.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7851

Reviewed By: anand1976

Differential Revision: D26114816

Pulled By: mrambacher

fbshipit-source-id: db32840e58d969d3a0fa6c25aaf13d6dcdc74150
2021-01-28 22:10:32 -08:00
mrambacher
0a9a05ae12 Make builds reproducible (#7866)
Summary:
Closes https://github.com/facebook/rocksdb/issues/7035

Changed how build_version.cc was generated:
- Included the GIT tag/branch in the build_version file
- Changed the "Build Date" to be:
      - If the GIT branch is "clean" (no changes), the date of the last git commit
      - If the branch is not clean, the current date
 - Added APIs to access the "build information", rather than accessing the strings directly.

The build_version.cc file is now regenerated whenever the library objects are rebuilt.

Verified that the built files remain the same size across builds on a "clean build" and the same information is reported by sst_dump --version

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7866

Reviewed By: pdillinger

Differential Revision: D26086565

Pulled By: mrambacher

fbshipit-source-id: 6fcbe47f6033989d5cf26a0ccb6dfdd9dd239d7f
2021-01-28 17:42:16 -08:00
Levi Tamasi
c696f27432 Accumulate blob file additions in VersionEdit during recovery (#7903)
Summary:
During recovery, RocksDB performs a kind of dummy flush; namely, entries
from the WAL are added to memtables, which then get written to SSTs and
blob files (if enabled) just like during a regular flush. Note that
multiple memtables might be flushed during recovery for the same column
family, for example, if the DB is reopened with a lower write buffer size,
and therefore, we need to make sure to collect all SST and blob file
additions. The patch fixes a bug in the earlier logic which resulted in
later blob file additions overwriting earlier ones.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7903

Test Plan: Added a unit test and ran `db_stress`.

Reviewed By: jay-zhuang

Differential Revision: D26110847

Pulled By: ltamasi

fbshipit-source-id: eddb50a608a88f54f3cec3a423de8235aba951fd
2021-01-27 18:46:15 -08:00
Zhichao Cao
95013df278 Do not set bg error for compaction in retryable IO Error case (#7899)
Summary:
When retryable IO error occurs during compaction, it is mapped to soft error and set the BG error. However, auto resume is not called to clean the soft error since compaction will reschedule by itself. In this change, When retryable IO error occurs during compaction, BG error is not set. User will be informed the error via EventHelper.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7899

Test Plan: tested with error_handler_fs_test

Reviewed By: anand1976

Differential Revision: D26094097

Pulled By: zhichao-cao

fbshipit-source-id: c53424f11d237405592cd762f43cbbdf8da8234f
2021-01-27 17:58:12 -08:00
Peter Dillinger
19210d5715 Accurate re-generate command in TARGETS file (#7902)
Summary:
TIL we have different versions of TARGETS file generated with
options passed to buckifier. Someone thought they were totally fine to
squash the file by re-running the command to generate (pretty reasonable
assumption) but the command was incorrect due to missing the extra
argument used to generate THAT TARGETS file.

This change includes in the command written in the TARGETS header the
extra argument passed to buckify (when used).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7902

Test Plan:
manual, as in the (now fixed) comments at the top of
buckify_rocksdb.py

Reviewed By: ajkr

Differential Revision: D26108317

Pulled By: pdillinger

fbshipit-source-id: 46e93dc1465e27bd18e0e0baa8eeee1b591c765d
2021-01-27 16:21:30 -08:00
Jay Zhuang
8944a3bf27 Add support for specifying working directory in lego-determinator (#7901)
Summary:
So we could change the repo or repo structure.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7901

Test Plan:
works with existing job: https://www.internalfb.com/intern/sandcastle/job/13510799254899745/
works with internal repo: https://www.internalfb.com/intern/sandcastle/job/27021598137143731

Reviewed By: pdillinger

Differential Revision: D26108239

Pulled By: jay-zhuang

fbshipit-source-id: ac24559653d9abc012068e8accd7606505bd5b88
2021-01-27 15:24:11 -08:00
Xiaopeng Zhang
36963dc2ca fix write option typo in java samples (#7894)
Summary:
this is a trivial PR for rocksdb java samples, I think it is a typo about write options. to do sync write, WAL should not be disabled

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7894

Reviewed By: jay-zhuang

Differential Revision: D26047128

Pulled By: mrambacher

fbshipit-source-id: a06ce54cb61af0d3f2578a709c34a0b1ccecb0b2
2021-01-26 19:13:08 -08:00
Jay Zhuang
c6ff4c0b70 Fix deadlock in fs_test.WALWriteRetryableErrorAutoRecover1 (#7897)
Summary:
The recovery thread could hold the db.mutex, which is needed from sync
write in main thread.
Make sure the write is done before recovery thread starts.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7897

Test Plan: `gtest-parallel ./error_handler_fs_test --gtest_filter=DBErrorHandlingFSTest.WALWriteRetryableErrorAutoRecover1 -r 10000 --workers=200`

Reviewed By: zhichao-cao

Differential Revision: D26082933

Pulled By: jay-zhuang

fbshipit-source-id: 226fc49228c0e5903f86ff45cc3fed3080abdb1f
2021-01-26 17:02:03 -08:00
anand76
4ee991b1e6 Cleanup multiple DBs after running db_bench in multi-DB mode (#7891)
Summary:
Currently, db_bench cleanup only deletes the main DB, if there's one.
Multiple DBs that are opened when --num_multi_db is specified are not
deleted, which can lead to crashes due to running compaction threads on
process exit.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7891

Test Plan: Run regression test

Reviewed By: jay-zhuang

Differential Revision: D26049914

Pulled By: anand1976

fbshipit-source-id: acef2821001ca5e208a96a6a273c724e56353316
2021-01-26 11:12:22 -08:00
Jay Zhuang
9425acacce Fix flaky error_handler_fs_test.MultiDBCompactionError (#7896)
Summary:
The error recovery thread may out-live DBImpl object, which causing
access released DBImpl.mutex. Close SstFileManager before closing DB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7896

Test Plan:
the issue can be reproduced by adding sleep in recovery code.
Pass the tests with sleep.

Reviewed By: zhichao-cao

Differential Revision: D26076655

Pulled By: jay-zhuang

fbshipit-source-id: 0d9cc5639c12fcfc001427015e75a9736f33cd96
2021-01-26 11:00:12 -08:00
Levi Tamasi
f9a30e0a5a Add 6.16 and 6.17 to check_format_compatible.sh (#7895)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7895

Test Plan: `tools/check_format_compatible.sh`

Reviewed By: zhichao-cao

Differential Revision: D26055885

Pulled By: ltamasi

fbshipit-source-id: fd669a439e7bf924b6abd9ef209130f528768c06
2021-01-26 09:39:21 -08:00
mrambacher
12f1137355 Add a SystemClock class to capture the time functions of an Env (#7858)
Summary:
Introduces and uses a SystemClock class to RocksDB.  This class contains the time-related functions of an Env and these functions can be redirected from the Env to the SystemClock.

Many of the places that used an Env (Timer, PerfStepTimer, RepeatableThread, RateLimiter, WriteController) for time-related functions have been changed to use SystemClock instead.  There are likely more places that can be changed, but this is a start to show what can/should be done.  Over time it would be nice to migrate most (if not all) of the uses of the time functions from the Env to the SystemClock.

There are several Env classes that implement these functions.  Most of these have not been converted yet to SystemClock implementations; that will come in a subsequent PR.  It would be good to unify many of the Mock Timer implementations, so that they behave similarly and be tested similarly (some override Sleep, some use a MockSleep, etc).

Additionally, this change will allow new methods to be introduced to the SystemClock (like https://github.com/facebook/rocksdb/issues/7101 WaitFor) in a consistent manner across a smaller number of classes.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7858

Reviewed By: pdillinger

Differential Revision: D26006406

Pulled By: mrambacher

fbshipit-source-id: ed10a8abbdab7ff2e23d69d85bd25b3e7e899e90
2021-01-25 22:09:11 -08:00
Akanksha Mahajan
1d226018af In IOTracing, add filename with each operation in trace file. (#7885)
Summary:
1. In IOTracing, add filename with each IOTrace record. Filename is stored in file object (Tracing Wrappers).
         2. Change the logic of figuring out which additional information (file_size,
            length, offset etc) needs to be store with each operation
            which is different for different operations.
            When new information will be added in future (depends on operation),
            this change would make the future additions simple.

Logic: In IOTraceRecord, io_op_data is added and its
         bitwise positions represent which additional information need
         to added in the record from enum IOTraceOp. Values in IOTraceOp represent bitwise positions.
         So if length and offset needs to be stored (IOTraceOp::kIOLen
         is 1 and IOTraceOp::kIOOffset is 2), position 1 and 2 (from rightmost bit) will be set
         and io_op_data will contain 110.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7885

Test Plan: Updated io_tracer_test and verified the trace file manually.

Reviewed By: anand1976

Differential Revision: D25982353

Pulled By: akankshamahajan15

fbshipit-source-id: ebfc5539cc0e231d7794a6b42b73f5403e360b22
2021-01-25 14:37:35 -08:00
Levi Tamasi
431e8afba7 Do not explicitly flush blob files when using the integrated BlobDB (#7892)
Summary:
In the original stacked BlobDB implementation, which writes blobs to blob files
immediately and treats blob files as logs, it makes sense to flush the file after
writing each blob to protect against process crashes; however, in the integrated
implementation, which builds blob files in the background jobs, this unnecessarily
reduces performance. This patch fixes this by simply adding a `do_flush` flag to
`BlobLogWriter`, which is set to `true` by the stacked implementation and to `false`
by the new code. Note: the change itself is trivial but the tests needed some work;
since in the new implementation, blobs are now buffered, adding a blob to
`BlobFileBuilder` is no longer guaranteed to result in an actual I/O. Therefore, we can
no longer rely on `FaultInjectionTestEnv` when testing failure cases; instead, we
manipulate the return values of I/O methods directly using `SyncPoint`s.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7892

Test Plan: `make check`

Reviewed By: jay-zhuang

Differential Revision: D26022814

Pulled By: ltamasi

fbshipit-source-id: b3dce419f312137fa70d84cdd9b908fd5d60d8cd
2021-01-25 13:32:33 -08:00
Levi Tamasi
19076c95aa Update HISTORY.md for PR 7888 (#7890)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7890

Reviewed By: ajkr

Differential Revision: D26005509

Pulled By: ltamasi

fbshipit-source-id: e7eb732180d447900788d0e3a17dfd1c3f1e708a
2021-01-21 14:20:10 -08:00
Matthew Von-Maszewski
12a8be1d44 MergeHelper::FilterMerge() calling ElapsedNanosSafe() upon exit even … (#7867)
Summary:
…when unused.  Causes many calls to clock_gettime, impacting performance.

Was looking for something else via Linux "perf" command when I spotted heavy usage of clock_gettime during a compaction.  Our product heavily uses the rocksdb::Options::merge_operator.  MergeHelper::FilterMerge() properly tests if timing is enabled/disabled upon entry, but not on exit.  This patch fixes the exit.

Note:  the entry test also verifies if "nullptr!=stats_".  This test is redundant to code within ShouldReportDetailedTime().  Therefore I omitted it in my change.

merge_test.cc updated with test that shows failure before merge_helper.cc change ... and fix after change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7867

Reviewed By: jay-zhuang

Differential Revision: D25960175

Pulled By: ajkr

fbshipit-source-id: 56e66d7eb6ae5eae89c8e0d5a262bd2905a226b6
2021-01-21 13:13:02 -08:00
Andrew Kryczka
e18a4df62a workaround race conditions during PeriodicWorkScheduler registration (#7888)
Summary:
This provides a workaround for two race conditions that will be fixed in
a more sophisticated way later. This PR:

(1) Makes the client serialize calls to `Timer::Start()` and `Timer::Shutdown()` (see https://github.com/facebook/rocksdb/issues/7711). The long-term fix will be to make those functions thread-safe.
(2) Makes `PeriodicWorkScheduler` atomically add/cancel work together with starting/shutting down its `Timer`. The long-term fix will be for `Timer` API to offer more specialized APIs so the client will not need to synchronize.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7888

Test Plan: ran the repro provided in https://github.com/facebook/rocksdb/issues/7881

Reviewed By: jay-zhuang

Differential Revision: D25990891

Pulled By: ajkr

fbshipit-source-id: a97fdaebbda6d7db7ddb1b146738b68c16c5be38
2021-01-21 08:50:38 -08:00
Adam Retter
d5f5d6579a Fix compilation against musl lib C (#7875)
Summary:
See https://github.com/percona/PerconaFT/pull/450

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7875

Reviewed By: ajkr

Differential Revision: D25938020

Pulled By: jay-zhuang

fbshipit-source-id: 9014dbc7b23bf92c5e63bfbdda4565bb0d2f2b58
2021-01-21 08:39:42 -08:00
Seeker
1a6438cff5 Add artifact suffix to shared library (#7755)
Summary:
On Unix systems, `ARTIFACT_SUFFIX` was added to the static library `librocksdb.a` but not the shared library `librocksdb.so`

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7755

Reviewed By: ajkr

Differential Revision: D25988550

Pulled By: jay-zhuang

fbshipit-source-id: 8079f26802ac937d5a75cbd6d3c0544094df1b11
2021-01-21 08:35:03 -08:00
Levi Tamasi
2d37830e44 Make blob related VersionEdit tags unignorable (#7886)
Summary:
BlobFileAddition and BlobFileGarbage should not be in the ignorable tag
range, since if they are present in the MANIFEST, users cannot downgrade
to a RocksDB version that does not understand them without losing access
to the data in the blob files. The patch moves these two tags to the
unignorable range; this should still be safe at this point, since the
integrated BlobDB project is still work in progress and thus there
shouldn't be any ignorable BlobFileAddition/BlobFileGarbage tags out
there.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7886

Test Plan: `make check`

Reviewed By: cheng-chang

Differential Revision: D25980956

Pulled By: ltamasi

fbshipit-source-id: 13cf5bd61d77f049b513ecd5ad0be8c637e40a9d
2021-01-20 20:29:04 -08:00
Cheng Chang
b0c43e7081 Update HISTORY.md (#7887)
Summary:
Mention the forward compatibility fix for WAL related version edits.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7887

Reviewed By: ltamasi

Differential Revision: D25982494

Pulled By: cheng-chang

fbshipit-source-id: 4be292aa4bf7fbc8a27c0bef1e7a98ad3ea8e1fa
2021-01-20 14:33:59 -08:00
Seeker
cdd8b09b9b Fix CMakeLists.txt for Apple Silicon (#7883)
Summary:
`CMAKE_SYSTEM_PROCESSOR` defaults to `arm64` on Apple Silicon starting with CMake 3.19.2; see https://cmake.org/cmake/help/latest/variable/CMAKE_HOST_SYSTEM_PROCESSOR.html#macos-platforms

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7883

Reviewed By: ajkr

Differential Revision: D25967902

Pulled By: jay-zhuang

fbshipit-source-id: 771bcb5fe845cf5f545c226691c0c41648186de4
2021-01-20 10:50:10 -08:00
Cheng Chang
e44948295e Make it able to ignore WAL related VersionEdits in older versions (#7873)
Summary:
Although the tags for `WalAddition`, `WalDeletion` are after `kTagSafeIgnoreMask`, to actually be able to skip these entries in older versions of RocksDB, we require that they are encoded with their encoded size as the prefix. This requirement is not met in the current codebase, so a downgraded DB may fail to open if these entries exist in the MANIFEST.

If a DB wants to downgrade, and its MANIFEST contains `WalAddition` or `WalDeletion`, it can set `track_and_verify_wals_in_manifest` to `false`, then restart twice, then downgrade. On the first restart, a new MANIFEST will be created with a `WalDeletion` indicating that all previously tracked WALs are removed from MANIFEST. On the second restart, since there is  no tracked WALs in MANIFEST now, a new MANIFEST will be created with neither `WalAddition` nor `WalDeletion`. Then the DB can downgrade.

Tags for `BlobFileAddition`, `BlobFileGarbage` also have the same problem, but this PR focuses on solving the problem for WAL edits.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7873

Test Plan: Added a `VersionEditTest::IgnorableTags` unit test to verify all entries with tags larger than `kTagSafeIgnoreMask` can actually be skipped and won't affect parsing of other entries.

Reviewed By: ajkr

Differential Revision: D25935930

Pulled By: cheng-chang

fbshipit-source-id: 7a02fdba4311d6084328c14aed110a26d08c3efb
2021-01-19 19:27:53 -08:00
Cheng Chang
928dea0e32 Update HISTORY.md (#7874)
Summary:
I find that the `track_and_verify_wals_in_manifest` option was only removed from 6.15 branch's HISTORY, but still appears under 6.15 in master branch's HISTORY. It should be moved to 6.16 since that's when the feature should be available.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7874

Reviewed By: jay-zhuang

Differential Revision: D25935971

Pulled By: cheng-chang

fbshipit-source-id: fe8bf1ec111597f9207e109aa3be65f8f919f1fd
2021-01-19 16:10:13 -08:00
Cheng Chang
4aa1a19db8 Add Apache Doris to USERS (#7865)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7865

Reviewed By: ajkr

Differential Revision: D25916166

Pulled By: cheng-chang

fbshipit-source-id: 24776b0203b21a733b5358dfa5dd66f639106dad
2021-01-19 15:31:56 -08:00