Commit Graph

9385 Commits

Author SHA1 Message Date
Peter Dillinger
4202c48f80 Replace large 'rm' with 'find' (#7095)
Summary:
On some platforms like MacOS, a second 'make check' can lead to
/bin/rm: Argument list too long

This is fixed by replacing with a 'find'. Also, using '-f' for more rm calls
to avoid prompt.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7095

Test Plan: 'make check' on Linux and MacOS

Reviewed By: riversand963

Differential Revision: D22415808

Pulled By: pdillinger

fbshipit-source-id: 0fd1ebae13739c9d81f9e813e99b062715604d6b
2020-07-07 16:48:52 -07:00
Peter Dillinger
787bf79fa0 Fix build of db_stress with LIB_MODE=shared (#7098)
Summary:
by tracking and linking against runtime dependent libraries in
Makefile

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7098

Test Plan: look for fix in CircleCI

Reviewed By: riversand963

Differential Revision: D22420860

Pulled By: pdillinger

fbshipit-source-id: d211d709214bf5306db68e43b7a2f18169281022
2020-07-07 16:48:52 -07:00
Peter Dillinger
dbf5c55812 Exclude c_test from buck build opt mode (#7093)
Summary:
Fix a Facebook internal build

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7093

Test Plan:
buck build mode/opt :c_test :c_test_bin (was compilation
failure, now "not found")
buck build mode/dev :c_test :c_test_bin (still passes)

Reviewed By: ajkr

Differential Revision: D22412528

Pulled By: pdillinger

fbshipit-source-id: 8e55c43dbf95386597e4cc690c41d9cbdcee03aa
2020-07-07 11:28:22 -07:00
Peter Dillinger
92731b6b4a Major CircleCI/Linux fixes / tweaks / enhancements (#7078)
Summary:
Primarily, this change adds a way to work around a bug limiting the effective output (and therefore debugability) of the Linux builds using parallel make. We would get
make[1]: write error: stdout
probably due to a kernel bug, apparently affecting both available ubuntu 16 machine images (maybe not affecting docker images, less horsepower). https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393

Now in the CircleCI config, make output on Ubuntu is piped through a custom 'cat' that ignores EAGAIN errors, which seems to fix the problem.

Significant other changes:
* Add another linux build that combines
  * LIB_MODE=shared, to ensure this works with compile and unit test execution
  * Alternative rocksdb namespace, to ensure this works (not rely on Travis)
  * ASSERT_STATUS_CHECKED=1, but with building all unit tests and running those expected to pass with it
* Run release build with and without gflags. (Was running only without, ignore large swaths of code in a normal release build! Two regressions in this build, only with gflags, in the last week not caught by CI!)
* Use gflags with unity and LITE build, as typical case.

Debugability improvements:
* Use V=1 to show commands being executed (thanks to EAGAIN work-around)
* Print kernel version and compiler versions as part of V=1 output from Makefile

Cosmetic other changes:
* Put more commands on one line, for less clutter in CircleCI output pages
* Remove redundant "all" in "make all check" and put make command options before targets
* Change some recursive "make clean" into dependency on "clean," toward minimizing unnecessary overhead (detect platform, build version, etc.) of extra recursive makes

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7078

Reviewed By: siying

Differential Revision: D22391647

Pulled By: pdillinger

fbshipit-source-id: d446fccf5a8c568b37dc8748621c8a5c546fe135
2020-07-07 11:25:46 -07:00
Levi Tamasi
a693341604 Move the blob file format related classes to the main namespace, rename reader/writer (#7086)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7086

Test Plan: `make check`

Reviewed By: zhichao-cao

Differential Revision: D22395420

Pulled By: ltamasi

fbshipit-source-id: 088a20097bd6b73b0c433cd79725779f97ec04f2
2020-07-06 17:18:14 -07:00
Peter Dillinger
4b107ceb7e Improve code comments in EstimateLiveDataSize (#7072)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7072

Reviewed By: ajkr

Differential Revision: D22391641

Pulled By: pdillinger

fbshipit-source-id: 0ef355576454514263ab684eb1a5c06787f3242a
2020-07-06 16:17:02 -07:00
Adam Retter
899e59ecb7 Add DB::OpenAsSecondary to RocksJava (#7047)
Summary:
Closes https://github.com/facebook/rocksdb/issues/5852
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7047

Reviewed By: cheng-chang

Differential Revision: D22335162

Pulled By: pdillinger

fbshipit-source-id: 75f3c524deccea7ebc0ad288da41f1ea81406c1c
2020-07-06 11:48:57 -07:00
Peter Dillinger
bd77e34191 More Makefile clean-up (#7066)
Summary:
(a) use STRESS_LIBRARY for db_stress and make sure
STRESS_LIBRARY has other stress test dependencies (as in buck build)

(b) fix rpath option to be accepted on MacOS. It still doesn't fully work
for me e.g. to run a LIB_MODE=shared unit test binary from another
directory, as it does on Linux, but the option is now accepted, and running
unit tests from current directory works for me.

Also adding LIB_MODE=shared to Travis. (Later TBD where best to fit in
in CircleCI.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7066

Test Plan: manual

Reviewed By: cheng-chang

Differential Revision: D22364068

Pulled By: pdillinger

fbshipit-source-id: 6fa98a222f89f808ee786474de1100d92c1adec3
2020-07-06 11:19:48 -07:00
Adam Retter
0117cbfc96 Adds a function to RocksJava for retrieving the version (#7083)
Summary:
Adds the function `RocksDB#rocksdbVersion()` for retrieving the RocksDB version.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7083

Reviewed By: cheng-chang

Differential Revision: D22391628

Pulled By: pdillinger

fbshipit-source-id: e1cabcf28aa81f5ee8dcdce5c9eca6b3155a279e
2020-07-06 11:06:21 -07:00
Zitan Chen
147f7b472a Fix flakiness of BackupableDBTest.TableFileCorruptedBeforeBackup (#7082)
Summary:
If the corruption of a table file is done before flushing, then db manifest may record the checksum for the corrupted table, which results in "matching checksums" when backup engine tries to verfiy the checksum, and causes a flaky test.

Fix the issue by adding `Flush()` before trying to corrupt a table file in *db*.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7082

Test Plan:
`buck test`
Without the fix, failed 5 of 100 tests.
Suspected whether the pseudo randomness causes the issue: doubling `keys_iteration` resulted in 2 of 100 tests failed; deterministically corrupting tables file also caused 2 of 100 tests to fail.
With the fix, passed 200 of 200 tests.

Reviewed By: pdillinger

Differential Revision: D22375421

Pulled By: gg814

fbshipit-source-id: 7304618e7520684b6087e42d0b58329c5ad18329
2020-07-03 15:40:04 -07:00
Jay Zhuang
ca7659e2c4 Fix release build caused by #7067 (#7077)
Summary:
The issue is introduced by https://github.com/facebook/rocksdb/issues/7067

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7077

Test Plan: `make release`

Reviewed By: pdillinger

Differential Revision: D22370835

Pulled By: jay-zhuang

fbshipit-source-id: 44326bae07809c4518371b6a7d1f47124e24a4f3
2020-07-02 20:53:08 -07:00
sdong
e0d0b49577 Fix test in buck test (#7076)
Summary:
This is to fix special logic to run tests inside FB.
Buck test is broken after moving to cpp_unittest(). Move c_test back to the previous approach.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7076

Test Plan: Watch the Sandcastle run

Reviewed By: ajkr

Differential Revision: D22370096

fbshipit-source-id: 4a464d0903f2c76ae2de3a8ad373ffc9bedec64c
2020-07-02 20:28:45 -07:00
Jay Zhuang
00de699096 Replace reinterpret_cast with static_cast_with_check (#7067)
Summary:
Replace `reinterpret_cast` with `static_cast_with_check` for `DBImpl` and `ColumnFamilyHandleImpl`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7067

Reviewed By: siying

Differential Revision: D22361587

Pulled By: jay-zhuang

fbshipit-source-id: dfe9e8f3af39c3d27cc372c55ab9ad905eb0a5a1
2020-07-02 19:25:41 -07:00
Zitan Chen
373d5ac485 BackupEngine verifies table file checksums on creating new backups (#7015)
Summary:
When table file checksums are enabled and stored in the DB manifest by using the RocksDB default crc32c checksum function, BackupEngine will calculate the crc32c checksum of the file to be copied and compare the calculated result with the one stored in the DB manifest before copying the file to the backup directory.

After copying to the backup directory, BackupEngine will verify the checksum of the copied file with the one calculated before copying. This helps detect some rare corruption events such as bit-flips during the copying process.

No verification with checksums in DB manifest will be performed if the table file checksum function is not the RocksDB default crc32c checksum function.

In addition, If `share_table_files` and `share_files_with_checksum` are true, BackupEngine will compare the checksums computed before and after copying of the table files.

Corresponding tests are added.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7015

Test Plan: Passed make check

Reviewed By: pdillinger

Differential Revision: D22165732

Pulled By: gg814

fbshipit-source-id: ee0e8cc397c455eba64545c29380b9d9853588ec
2020-07-02 18:15:12 -07:00
Peter Dillinger
a680a7ea37 Un-revert #7049, revert #7022 (#7071)
Summary:
Even though local bisection gave me a clear signal (and still does) that reverting https://github.com/facebook/rocksdb/issues/7049 would fix the failures in MultiThreadedDBTest, https://github.com/facebook/rocksdb/issues/7022 seems to be the root cause. Reverting https://github.com/facebook/rocksdb/issues/7022 and keeping https://github.com/facebook/rocksdb/issues/7049 seems to fix the issue in local reproducer also. (Had these landed in opposite order, bisection would have found the root cause.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7071

Reviewed By: akankshamahajan15

Differential Revision: D22362857

Pulled By: pdillinger

fbshipit-source-id: ed63df3d74e9d4ce1604de8fe43b216166c7a3f0
2020-07-02 13:30:41 -07:00
Peter Dillinger
52d59e0c93 Revert "Whole DBTest to skip fsync (#7049)" (#7070)
Summary:
This reverts commit 4f1534bdb0.

This commit caused failures and deadlocks in
MultiThreadedDBTest.MultiThreaded/69 and others.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7070

Reviewed By: riversand963

Differential Revision: D22358778

Pulled By: pdillinger

fbshipit-source-id: faf8f2cb469a7063a113921c8e9c64a9f7610dac
2020-07-02 10:22:43 -07:00
sdong
4f1534bdb0 Whole DBTest to skip fsync (#7049)
Summary:
After https://github.com/facebook/rocksdb/pull/7036, we still see extra DBTest that can timeout when running 10 or 20 in parallel. Expand skip-fsync mode in whole DBTest. Still preserve other tests from doing this mode to be conservative.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7049

Test Plan: Run all existing files.

Reviewed By: pdillinger

Differential Revision: D22301700

fbshipit-source-id: f9a9e3b3b26ce640665a47cb8bff33ba0c89b565
2020-07-01 19:37:56 -07:00
Zitan Chen
b5bae48c8a Fix db_id and db_session_id nullptr warning by clang analyzer (#7063)
Summary:
GetFileDbIdentities requires either db_id non-null or db_session_id non-null.
Passing nullptr for db_id or db_session_id in CopyOrCreateFile indicates the caller does not want to obtain the value for db_id or db_session_id.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7063

Test Plan:
USE_CLANG=1 make analyze
backupable_db_test

Reviewed By: pdillinger

Differential Revision: D22338497

Pulled By: gg814

fbshipit-source-id: 2aa2dcc14d156b0f99b07d6cf3c731ee088272cd
2020-07-01 17:28:28 -07:00
Akanksha Mahajan
5edfe3a3d8 Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7022)
Summary:
When format_version is high enough to support user-key and there are index entries for same user key that spans multiple data blocks then it changes from user-key mode to internal-key mode. But the flush policy is not reset to point to Block Builder of internal-keys. After this switch, no entries are added to user key index partition result, thus it never triggers flushing the block.

Fix: After adding the entry in sub_builder_index_, if there is a switch from user-key to internal-key, then flush policy is updated to point to Block Builder of internal-keys index partition.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7022

Test Plan:
1. make check -j64
           2. Added one unit test case

Reviewed By: ajkr

Differential Revision: D22197734

Pulled By: akankshamahajan15

fbshipit-source-id: d87e9e46bccab8e896ee6979d6b79c51f73d479e
2020-07-01 14:58:08 -07:00
Andrew Kryczka
c25a014792 deflake DBCompactionTestWithParam.IntraL0Compaction test (#7065)
Summary:
This check is flaky because compaction could run between the `Flush()` and the `TestGetTickerCount()`, which would increase the `BLOCK_CACHE_INDEX_MISS` count beyond what the test expects. Verified by adding a `sleep(1)` between those two lines and observing the counter is too high every time. The solution is just to remove this check as it doesn't have any use anyways. The latter check of index miss is sufficient to conclude the newest L0 file (i.e., the one generated by intra-L0) does not have its index block pinned in cache. It'd be nice to simultaneously check the L0 files generated by flush do have their index blocks pinned in cache, but that's not what the line deleted in this PR was checking..
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7065

Reviewed By: pdillinger

Differential Revision: D22340327

Pulled By: ajkr

fbshipit-source-id: e076b2c7228b7fa763dd0c0cb13828e176c1abee
2020-07-01 14:53:10 -07:00
Peter Dillinger
e2fd501d44 Stabilize DBTest.ApproximateSizesMemTable (#7064)
Summary:
Random memtable layouts could cause random failure,
reproducible with command below running for a while. Test now using
deterministic behavior.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7064

Test Plan: while ./db_test --gtest_filter=*SizesMemTable*; do true; done

Reviewed By: siying

Differential Revision: D22339442

Pulled By: pdillinger

fbshipit-source-id: 8e74e5a9b5e88f7030854045a22c12cf561d5de6
2020-07-01 13:52:20 -07:00
Peter Dillinger
8e6ff044e1 Fix release build and fbcode+clang+shared (#7062)
Summary:
Follow-up to https://github.com/facebook/rocksdb/issues/6660. Release build had linker error. fbcode+clang+shared build was erroring on unused parameter '-nostdinc'.

Fixes https://github.com/facebook/rocksdb/issues/7061

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7062

Test Plan: make release, USE_CLANG=1 LIB_MODE=shared make check, etc.

Reviewed By: siying

Differential Revision: D22335663

Pulled By: pdillinger

fbshipit-source-id: 261cd959ca1f6c273dc763a70020a535ba8e81de
2020-07-01 10:30:55 -07:00
mrambacher
80f71b5863 Use Libraries in the RocksDB Makefile Build (#6660)
Summary:
Change the linking of tests/tools to be against a library rather than a list of objects.  This change substantially reduces the size of the objects produced.

peterd clean repo size: 264M
Before this change, with make all: 40G
After this change, with make all: 28G
With make LIB_MODE=shared all: 7.0G

The list of TESTS was changed from being hard-coded to generated from the test sources variable.  Note that there are some test sources that are not built as tests (though the set of tests is identical to the previous version).

Added OBJ_DIR option to Makefile to allow objects to be placed in an alternative location.  By default, OBJ_DIR is the same as before ("./").

This change is a precursor to being able to build/run the tests/tools linked against static libraries.  Additionally, it should be possible to clean up and merge some of the rules for building tests and the like if so desired.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6660

Reviewed By: riversand963

Differential Revision: D22244463

Pulled By: pdillinger

fbshipit-source-id: db9c6341d81ed62c2270374f4ede02fb9604c754
2020-06-30 19:33:31 -07:00
Zitan Chen
6a243b3ade Generalize BackupEngine naming option for share_files_with_checksum SSTs and revert BackupEngine::VerifyBackup to check only file sizes by default (#7032)
Summary:
`bool BackupableDBOptions::new_naming_for_backup_files` is updated to `BackupTableNameOption BackupableDBOptions::share_files_with_checksum_naming`, where `BackupTableNameOption` is an `enum` type with two enumerators `kChecksumAndFileSize` and `kChecksumAndFileSize`. This opens up possibilities of extenting the current naming scheme for backup table files. By default, `BackupTableNameOption BackupableDBOptions::share_files_with_checksum_naming` is set to `kChecksumAndDbSessionId`.

Revert `BackupEngine::VerifyBackup` to only check file sizes by default.

Also fix the construction of the `SstFileDumper` in `GetFileDbIdentities` by setting a proper `Env` of the `Options` passed in the constructor.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7032

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D22237763

Pulled By: gg814

fbshipit-source-id: 466902a4e731babd64e30f0e82ca1aa82962e52e
2020-06-30 18:47:16 -07:00
Yanqin Jin
f8bfd66b97 Fix python in format check script for Centos8 (#7057)
Summary:
As title.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7057

Test Plan: ./tools/check_format_compatible.sh

Reviewed By: pdillinger

Differential Revision: D22319831

Pulled By: riversand963

fbshipit-source-id: 82653a525a5296ef65a6a7a439cdd6bff88f498e
2020-06-30 16:37:21 -07:00
Andrew Kryczka
8458532d58 Skip unnecessary allocation for mmap reads under 5000 bytes (#7043)
Summary:
With mmap enabled on an uncompressed file, we were previously always doing a heap allocation to obtain the scratch buffer for `RandomAccessFileReader::Read()`. However, that allocation was unnecessary as the underlying file reader returned a pointer into its mapped memory, not the provided scratch buffer. This PR makes passes the `BlockFetcher`'s inline buffer as the scratch buffer if the data block is small enough (less than `kDefaultStackBufferSize` bytes, currently 5000). Ideally we would not pass a scratch buffer at all for an mmap read; however, the `RandomAccessFile::Read()` API guarantees such a buffer is provided, and non-standard implementations may be relying on it even when `Options::allow_mmap_reads == true`. In that case, this PR still works but introduces an extra copy from the inline buffer to a heap buffer.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7043

Reviewed By: cheng-chang

Differential Revision: D22320606

Pulled By: ajkr

fbshipit-source-id: ad964dd23df34e07d979c6032c2dfe5454c98b52
2020-06-30 15:40:40 -07:00
Levi Tamasi
e367bc7f4b Clean up blob files based on the linked SST set (#7001)
Summary:
The earlier `VersionBuilder` code only cleaned up blob files that were
marked as entirely consisting of garbage using `VersionEdits` with
`BlobFileGarbage`. This covers the cases when table files go through
regular compaction, where we iterate through the KVs and thus have an
opportunity to calculate the amount of garbage (that is, most cases).
However, it does not help when table files are simply dropped (e.g. deletion
compactions or the `DeleteFile` API). To deal with such cases, the patch
adds logic that cleans up all blob files at the head of the list until the first
one with linked SSTs is found. (As an example, let's assume we have blob files
with numbers 1..10, and the first one with any linked SSTs is number 8.
This means that SSTs in the `Version` only rely on blob files with numbers >= 8,
and thus 1..7 are no longer needed.)

The code change itself is pretty small; however, changing the logic like this
necessitated changes to some tests that have been added recently (namely
to the ones that use blob files in isolation, i.e. without any table files referring
to them). Some of these cases were fixed by bypassing `VersionBuilder` altogether
in order to keep the tests simple (which actually makes them more proper unit tests
as well), while the `VersionBuilder` unit tests were fixed by adding dummy table
files to the test cases as needed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7001

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D22119474

Pulled By: ltamasi

fbshipit-source-id: c6547141355667d4291d9661d6518eb741e7b54a
2020-06-30 15:31:21 -07:00
Yanqin Jin
f5554fd7b6 Add recent versions to format compatibility check (#7059)
Summary:
as title.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7059

Test Plan: ./tools/check_format_compatible.sh

Reviewed By: siying

Differential Revision: D22320774

Pulled By: riversand963

fbshipit-source-id: 124d13b08703d077a7aab3678e1eb639fcbcceca
2020-06-30 15:07:41 -07:00
Cheng Chang
f045ee6422 Increase transaction timeout and enable deadlock detection in stress test (#7056)
Summary:
There are errors like `Transaction put: Operation timed out: Timeout waiting to lock key
terminate called without an active exception`, based on experiment on devserver, increasing timeouts can resolve the issue.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7056

Test Plan: watch stress test with txn.

Reviewed By: anand1976

Differential Revision: D22317265

Pulled By: cheng-chang

fbshipit-source-id: 2dc3352def5e78d2c39a18d7262a3a65ca98bbba
2020-06-30 14:29:17 -07:00
sdong
80b107a0a9 Divide WriteCallbackTest.WriteWithCallbackTest (#7037)
Summary:
WriteCallbackTest.WriteWithCallbackTest has a deep for-loop and in some cases runs very long. Parameterimized it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7037

Test Plan: Run the test and see it passes.

Reviewed By: ltamasi

Differential Revision: D22269259

fbshipit-source-id: a1b6687b5bf4609754833d14cf383d68bc7ab27a
2020-06-30 12:31:30 -07:00
sdong
2d1d51d385 db_stress: deep clean directory before checkpoint (#7039)
Summary:
We see crash test occassionally fails with "A checkpoint operation failed with: Invalid argument: Directory exists". The suspicious is that the directory fails to be deleted because some trash files. Deep clean the directory after a DestroyDB() call.

Also add more debugging printf in case it fails.
Also, preserve the DB if verification fails.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7039

Test Plan: Run db_stress with low --checkpoint_one_in value

Reviewed By: riversand963

Differential Revision: D22271694

fbshipit-source-id: 6a9b2abb664fc69a4dc666741df4f6b23703cd6d
2020-06-30 12:01:34 -07:00
Burton Li
5be2cb6948 Compaction filter support for BlobDB (#6850)
Summary:
Added compaction filter support for BlobDB non-TTL values. Same as vanilla RocksDB, user compaction filter applies to all k/v pairs of the compaction for non-TTL values. It honors `min_blob_size`, which potentially results value transitions between inlined data and stored-in-blob data when size of value is changed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6850

Reviewed By: siying

Differential Revision: D22263487

Pulled By: ltamasi

fbshipit-source-id: 8fc03f8cde2a5c831e63b436b3dbf1b7f90939e8
2020-06-29 17:32:14 -07:00
sdong
58547e533b Disable fsync in some tests to speed them up (#7036)
Summary:
Fsyncing files is not providing more test coverage in many tests. Provide an option in SpecialEnv to turn it off to speed it up and enable this option in some tests with relatively long run time.
Most of those tests can be divided as parameterized gtest too. This two speed up approaches are orthogonal and we can do both if needed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7036

Test Plan: Run all tests and make sure they pass.

Reviewed By: ltamasi

Differential Revision: D22268084

fbshipit-source-id: 6d4a838a1b7328c13931a2a5d93de57aa02afaab
2020-06-29 16:56:59 -07:00
Anand Ananthabhotla
9a5886bd8c Extend Get/MultiGet deadline support to table open (#6982)
Summary:
Current implementation of the ```read_options.deadline``` option only checks the deadline for random file reads during point lookups. This PR extends the checks to file opens, prefetches and preloads as part of table open.

The main changes are in the ```BlockBasedTable```, partitioned index and filter readers, and ```TableCache``` to take ReadOptions as an additional parameter. In ```BlockBasedTable::Open```, in order to retain existing behavior w.r.t checksum verification and block cache usage, we filter out most of the options in ```ReadOptions``` except ```deadline```. However, having the ```ReadOptions``` gives us more flexibility to honor other options like verify_checksums, fill_cache etc. in the future.

Additional changes in callsites due to function signature changes in ```NewTableReader()``` and ```FilePrefetchBuffer```.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6982

Test Plan: Add new unit tests in db_basic_test

Reviewed By: riversand963

Differential Revision: D22219515

Pulled By: anand1976

fbshipit-source-id: 8a3b92f4a889808013838603aa3ca35229cd501b
2020-06-29 14:53:17 -07:00
sdong
d809ae9a2d Remove 2019 from appveyor (#7038)
Summary:
VS2019 is covered in CircleCI. The only thing missing there is -DCMAKE_CXX_STANDARD=20 option. Add the option there and remove VS2019 build from Appveyor.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7038

Test Plan: Watch build results.

Reviewed By: pdillinger, ltamasi

Differential Revision: D22270010

fbshipit-source-id: 77d30be49d38b41516fa8a12be45395c27b12761
2020-06-29 14:31:41 -07:00
Stanislav Tkach
1b85d57cf5 Expose KeyMayExist in the C API (#7021)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7021

Reviewed By: ajkr

Differential Revision: D22246297

Pulled By: pdillinger

fbshipit-source-id: 81dfd0a49e4d5ce0c9f00772c17cca425757ea24
2020-06-29 12:21:53 -07:00
Yanqin Jin
d47c871190 Fix data race to VersionSet::io_status_ (#7034)
Summary:
After https://github.com/facebook/rocksdb/issues/6949 , VersionSet::io_status_ can be concurrently accessed by multiple
threads without lock, causing tsan test to fail. For example, a bg flush thread
resets io_status_ before calling LogAndApply(), while another thread already in
the process of LogAndApply() reads io_status_. This is a bug.

We do not have to reset io_status_ each time we call LogAndApply(). io_status_
is part of the state of VersionSet, and it indicates the outcome of preceding
MANIFEST/CURRENT files IO operations. Its value should be updated only when:

1. MANIFEST/CURRENT files IO fail for the first time.
2. MANIFEST/CURRENT files IO succeed as part of recovering from a prior
   failure without process restart, e.g. calling Resume().

Test Plan (devserver):
COMPILE_WITH_TSAN=1 make check
COMPILE_WITH_TSAN=1 make db_test2
./db_test2 --gtest_filter=DBTest2.CompactionStall
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7034

Reviewed By: zhichao-cao

Differential Revision: D22247137

Pulled By: riversand963

fbshipit-source-id: 77b83e05390f3ee3cd2d96d3fdd6fe4f225e3216
2020-06-27 08:57:31 -07:00
Akanksha Mahajan
b9d51b8684 Fix for TSAN failure in DeleteScheduler (#7029)
Summary:
TSAN failure caused by setting statistics in SstFileManager and DeleteScheduler.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7029

Test Plan:
1. make check -j64
           2. COMPILE_WITH_TSAN=1 make check -j64

Reviewed By: siying, zhichao-cao

Differential Revision: D22223418

Pulled By: akankshamahajan15

fbshipit-source-id: c5bf336d711b787908dfeb6166cab4aa2e494d61
2020-06-26 15:37:22 -07:00
Zitan Chen
1569dc48f5 BackupEngine::VerifyBackup verifies checksum by default (#7014)
Summary:
A parameter `verify_with_checksum` is added to `BackupEngine::VerifyBackup`, which is true by default. So now `BackupEngine::VerifyBackup` verifies backup files with checksum AND file size by default. When `verify_with_checksum` is false, `BackupEngine::VerifyBackup` only compares file sizes to verify backup files.

Also add a test for the case when corruption does not change the file size.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7014

Test Plan: Passed backupable_db_test

Reviewed By: zhichao-cao

Differential Revision: D22165590

Pulled By: gg814

fbshipit-source-id: 606a7450714e868bceb38598c89fd356c6004f4f
2020-06-26 11:42:12 -07:00
sdong
f9817201af Add unity build to CircleCI (#7026)
Summary:
We are still keeping unity build working. So it's a good idea to add to a pre-commit CI.
A latest GCC docker image just to get a little bit more coverage. Fix three small issues to make it pass.
Also make unity_test to run db_basic_test rather than db_test to cut the test time. There is no point to run expensive tests here. It was set to run db_test before db_basic_test was separated out.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7026

Test Plan: watch tests to pass.

Reviewed By: zhichao-cao

Differential Revision: D22223197

fbshipit-source-id: baa3b6cbb623bf359829b63ce35715c75bcb0ed4
2020-06-26 11:14:08 -07:00
sdong
7006997e12 Add ASAN CircleCI Run (#7027)
Summary:
ASAN run is powerful in finding memory leak bugs. Running it as a part of the pre-merge CI can help contributors avoid to merge some code with bugs.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7027

Test Plan: Watch the test result.

Reviewed By: pdillinger

Differential Revision: D22222371

fbshipit-source-id: 92f9ce19e01a94ba5f9b765e154f7bcdece5c2a9
2020-06-25 17:41:50 -07:00
Daniel Black
ce332f8c5e freebsd: malloc_usable_size check malloc_np.h (#7009)
Summary:
Per https://www.unix.com/man-page/freebsd/3/malloc_usable_size/
malloc_usable_size is in malloc_np.h as its a non-standard API.

Without patch it just fails to detect from ./CMakeFiles/CMakeError.log

In file included from /home/dan/build-rocksdb/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx:2:
/usr/include/malloc.h:3:2: error: "<malloc.h> has been replaced by <stdlib.h>"
 ^
/home/dan/build-rocksdb/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx:8:19: error: use of undeclared identifier 'malloc_usable_size'
  return ((int*)(&malloc_usable_size))[argc];
                  ^
2 errors generated.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7009

Reviewed By: riversand963

Differential Revision: D22176093

Pulled By: ajkr

fbshipit-source-id: da980f3d343b6d9b0c70d7827c6df495f3fb1ade
2020-06-25 17:30:27 -07:00
Daniel Black
c2b0b696c4 filelock_test: add freebsd headers for waitpid (#7010)
Summary:
Per manual https://www.unix.com/man-page/FreeBSD/2/waitpid
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7010

Reviewed By: siying

Differential Revision: D22176164

Pulled By: ajkr

fbshipit-source-id: 0a850ae6f1791d10951d5e4a79cfee01a3981d5a
2020-06-25 17:25:42 -07:00
Daniel Black
741b9ba96b gflags: freebsd include path + links (#7011)
Summary:
The include path from find_package(gflags) needed to be included to
compile.

Because gflags got included in THIRDPARTY_LIBS as a PRIVATE library
to ROCKSDB_{SHARED|STATIC}_LIB, its functions aren't accessible to
the all the tools an utilities that use gflags directly.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7011

Reviewed By: siying

Differential Revision: D22176303

Pulled By: ajkr

fbshipit-source-id: 0a94523fc69e82d8f686bc0b43dc3eafc51ad84f
2020-06-25 17:23:01 -07:00
Adam Retter
df5fbe6408 Portable backward compatibility with MacOS 10.12+ (#7016)
Summary:
When `PORTABLE=1` is set, RocksDB will now be built with backwards compatibility for MacOS as far back as 10.12 (i.e. 2016).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7016

Reviewed By: ajkr

Differential Revision: D22211312

Pulled By: pdillinger

fbshipit-source-id: 7b0858d9b55d6265d3ea27bf5ea1673639b6538c
2020-06-25 13:57:00 -07:00
Stanislav Tkach
70b5d95dc7 Add (more) getters for options to the C API (#6998)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6998

Reviewed By: ajkr

Differential Revision: D22211700

Pulled By: pdillinger

fbshipit-source-id: 1141c20527dee5e13205059bf8e83927063c4c1e
2020-06-25 13:53:33 -07:00
Adam Retter
82d98447e4 Use official CMake snap on Travis Linux (#6892)
Summary:
Switch to using the official CMake snap rather than our own from S3.
See https://gitlab.kitware.com/cmake/cmake/-/issues/20122#note_769266
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6892

Reviewed By: ajkr

Differential Revision: D22211822

Pulled By: pdillinger

fbshipit-source-id: baf20005a4064d7f86b46087a41af4dfdea5510f
2020-06-25 13:50:38 -07:00
sdong
d64cf0e4ee Move away from direct TmpDir() call in some tests (#7030)
Summary:
Some tests directly uses TmpDir() as temporary directory without adding any randomize factor. This would cause failures when tests run in parallel. Fix it by moving some of them to test::PerThreadDBPath()
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7030

Test Plan: Watch existing tests pass

Reviewed By: zhichao-cao

Differential Revision: D22224710

fbshipit-source-id: 28c9932fede0a4a64670e5b5fdb08f4fb5dccdd0
2020-06-25 12:09:57 -07:00
Siying Dong
63b59f21d9 Add CircleCI gadget (#7028)
Summary:
CircleCI is stably running. Need to add a gadget.
Also since Circle builds some Windows and Linux, rename Travis and Appveyor builds to their names.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7028

Reviewed By: ajkr

Differential Revision: D22223756

fbshipit-source-id: 8f9dcea0b614f083e6d2a2ebf84dc6ab7c4d1601
2020-06-25 10:30:33 -07:00
Zitan Chen
95fbb62c44 Update HISTORY.md to include the Public API Change for DB::OpenForReadonly introduced earlier (#7023)
Summary:
`DB::OpenForReadOnly()` now returns `Status::NotFound` when the specified DB directory does not exist. Previously the error returned depended on the underlying `Env`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7023

Reviewed By: ajkr

Differential Revision: D22207845

Pulled By: gg814

fbshipit-source-id: f35830811a0e67efb0ee82eda3a9739bc526baba
2020-06-25 06:14:29 -07:00