Commit Graph

80 Commits

Author SHA1 Message Date
Peter Dillinger
1601433b3a Misc CI improvements / additions (#9859)
Summary:
* Add valgrind test to nightly CircleCI (in case it can catch something that
ASAN/UBSAN does not)
* Add clang13+asan+ubsan+folly test to nightly CircleCI, for broader testing
* Consolidate many copies of ASAN_OPTIONS= while also allowing it to be
inherited from parent environment rather than always overridden.
* Move UBSAN exclusion from Makefile into options_settable_test.cc

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9859

Test Plan: CI

Reviewed By: jay-zhuang

Differential Revision: D35730903

Pulled By: pdillinger

fbshipit-source-id: 6f5464034e8115f9a07f6f7aec1de9219ec2837c
2022-04-18 20:26:37 -07:00
Peter Dillinger
41237dd306 Add "no compression" job to CircleCI (#9850)
Summary:
Since they operate at distinct abstraction layers, I thought it
was prudent to combine with EncryptedEnv CI test for each PR, for efficiency
in testing. Also added supported compressions to sst_dump --help output
so that CI job can verify no compiled-in compression support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850

Test Plan: CI, some manual stuff

Reviewed By: riversand963

Differential Revision: D35682346

Pulled By: pdillinger

fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
2022-04-18 12:47:16 -07:00
Peter Dillinger
efd035164b Meta-internal folly integration with F14FastMap (#9546)
Summary:
Especially after updating to C++17, I don't see a compelling case for
*requiring* any folly components in RocksDB. I was able to purge the existing
hard dependencies, and it can be quite difficult to strip out non-trivial components
from folly for use in RocksDB. (The prospect of doing that on F14 has changed
my mind on the best approach here.)

But this change creates an optional integration where we can plug in
components from folly at compile time, starting here with F14FastMap to replace
std::unordered_map when possible (probably no public APIs for example). I have
replaced the biggest CPU users of std::unordered_map with compile-time
pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set.
USE_FOLLY is always set in the Meta-internal buck build, and a simulation of
that is in the Makefile for public CI testing. A full folly build is not needed, but
checking out the full folly repo is much simpler for getting the dependency,
and anything else we might want to optionally integrate in the future.

Some picky details:
* I don't think the distributed mutex stuff is actually used, so it was easy to remove.
* I implemented an alternative to `folly::constexpr_log2` (which is much easier
in C++17 than C++11) so that I could pull out the hard dependencies on
`ConstexprMath.h`
* I had to add noexcept move constructors/operators to some types to make
F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a
macro to make that easier in some common cases.
* Updated Meta-internal buck build to use folly F14Map (always)

No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a
production integration for open source users.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546

Test Plan:
CircleCI tests updated so that a couple of them use folly.

Most internal unit & stress/crash tests updated to use Meta-internal latest folly.
(Note: they should probably use buck but they currently use Makefile.)

Example performance improvement: when filter partitions are pinned in cache,
they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build
a test that exercises that heavily. Build DB with

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters
```

and test with (simultaneous runs with & without folly, ~20 times each to see
convergence)

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache
```

Average ops/s no folly: 26229.2
Average ops/s with folly: 26853.3 (+2.4%)

Reviewed By: ajkr

Differential Revision: D34181736

Pulled By: pdillinger

fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
2022-04-13 07:34:01 -07:00
Jay Zhuang
d5c34fa8f4 Upgrade gbenchmark to 1.6.1 (#9775)
Summary:
Upgrade google benchmark to the latest 1.6.1.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9775

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D35252889

Pulled By: jay-zhuang

fbshipit-source-id: 4d60dd1c6f522d0af0b3942ae8fa88e5ae17f34a
2022-03-30 10:09:49 -07:00
Jay Zhuang
81d1cdca7f Fix make clean fail after java build (#9710)
Summary:
Seems clean-rocksjava and clean-rocks conflict.
Also remove unnecessary step in java CI build, otherwise it will rebuild
the code again as java make sample do clean up first.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9710

Test Plan: `make rocksdbjava && make clean` should return success

Reviewed By: riversand963

Differential Revision: D35122872

Pulled By: jay-zhuang

fbshipit-source-id: 2a15b83e7a763c0fc0e42e1f35aac9551f951ece
2022-03-24 13:39:15 -07:00
Jay Zhuang
661e03294c Enable detect_stack_use_after_return for ASAN (#9714)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9714

Reviewed By: ajkr

Differential Revision: D34983675

Pulled By: jay-zhuang

fbshipit-source-id: 0252ec6ee38a0b960df4c92791c7c2bcbfba5ad8
2022-03-21 10:34:11 -07:00
Peter Dillinger
fe9a344c55 crash_test Makefile refactoring, add to CircleCI (#9702)
Summary:
some Makefile refactoring to support Meta-internal workflows,
and add a basic crash_test flow to CircleCI

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9702

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D34934315

Pulled By: pdillinger

fbshipit-source-id: 67f17280096d8968d8e44459293f72fb6fe339f3
2022-03-16 15:58:06 -07:00
Jay Zhuang
91372328ef Reduce Windows build parallelism number (#9687)
Summary:
To avoid OOM issue for VS2007 build.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9687

Test Plan: Run VS2007 build 5 times, seems fine.

Reviewed By: ajkr

Differential Revision: D34845073

Pulled By: jay-zhuang

fbshipit-source-id: 60f84885e391e878ee6f3b1945376323baf47ec5
2022-03-12 11:45:10 -08:00
Peter Dillinger
9b8b8b1504 Remove remaining SKIP_LINK=1 in circleci config (#9669)
Summary:
Should be unnecessary

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9669

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D34689277

Pulled By: pdillinger

fbshipit-source-id: 5d44de1f851503fd1777b869c06c330f3c4deade
2022-03-07 15:23:25 -08:00
Jay Zhuang
67542bfab5 Improve build speed (#9605)
Summary:
Improve the CI build speed:
- split the macos tests to 2 parallel jobs
- split tsan tests to 2 parallel jobs
- move non-shm tests to nightly build
- slow jobs use lager machine
- fast jobs use smaller machine
- add microbench to no-test jobs
- add run-microbench to nightly build

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9605

Reviewed By: riversand963

Differential Revision: D34358982

Pulled By: jay-zhuang

fbshipit-source-id: d5091b3f4ef6d25c5c37920fb614f3342ee60e4a
2022-03-03 11:58:51 -08:00
Levi Tamasi
2ee25e8846 Remove cat_ignore_eagain (#9531)
Summary:
... since it was only necessary to work around a bug on certain Ubuntu
16.04 images (and we now use 20.04 across the board).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9531

Test Plan: Watch CI.

Reviewed By: ajkr

Differential Revision: D34089424

Pulled By: ltamasi

fbshipit-source-id: f15f86332c119099f61b9bdc74604657fc5d964e
2022-02-08 17:51:59 -08:00
Levi Tamasi
1f96bc6864 Upgrade to Ubuntu 20.04 in our CircleCI config (#9486)
Summary:
Ubuntu 16.04 has reached EOL. The patch upgrades the image for all of
our CircleCI jobs to the latest, namely `ubuntu-2004:202111-02`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9486

Test Plan: Watch the CI build results.

Reviewed By: ajkr

Differential Revision: D34029339

Pulled By: ltamasi

fbshipit-source-id: a266b631c04d227fe29b8156be61229605eb9dd7
2022-02-06 17:37:12 -08:00
Peter Dillinger
fd3e0f43b3 Require C++17 (#9481)
Summary:
Drop support for some old compilers by requiring C++17 standard
(or higher). See https://github.com/facebook/rocksdb/issues/9388

First modification based on this is to remove some conditional compilation in slice.h (also
better for ODR)

Also in this PR:
* Fix some Makefile formatting that seems to affect ASSERT_STATUS_CHECKED config in
some cases
* Add c_test to NON_PARALLEL_TEST in Makefile
* Fix a clang-analyze reported "potential leak" in lru_cache_test
* Better "compatibility" definition of DEFINE_uint32 for old versions of gflags
* Fix a linking problem with shared libraries in Makefile (`./random_test: error while loading shared libraries: librocksdb.so.6.29: cannot open shared object file: No such file or directory`)
* Always set ROCKSDB_SUPPORT_THREAD_LOCAL and use thread_local (from C++11)
  * TODO in later PR: clean up that obsolete flag
* Fix a cosmetic typo in c.h (https://github.com/facebook/rocksdb/issues/9488)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9481

Test Plan:
CircleCI config substantially updated.

* Upgrade to latest Ubuntu images for each release
* Generally prefer Ubuntu 20, but keep a couple Ubuntu 16 builds with oldest supported
compilers, to ensure compatibility
* Remove .circleci/cat_ignore_eagain except for Ubuntu 16 builds, because this is to work
around a kernel bug that should not affect anything but Ubuntu 16.
* Remove designated gcc-9 build, because the default linux build now uses GCC 9 from
Ubuntu 20.
* Add some `apt-key add` to fix some apt "couldn't be verified" errors
* Generally drop SKIP_LINK=1; work-around no longer needed
* Generally `add-apt-repository` before `apt-get update` as manual testing indicated the
reverse might not work.

Travis:
* Use gcc-7 by default (remove specific gcc-7 and gcc-4.8 builds)
* TODO in later PR: fix s390x "Assembler messages: Error: invalid switch -march=z14" failure

AppVeyor:
* Completely dropped because we are dropping VS2015 support and CircleCI covers
VS >= 2017

Also local testing with old gflags (out of necessity when using ROCKSDB_NO_FBCODE=1).

Reviewed By: mrambacher

Differential Revision: D33946377

Pulled By: pdillinger

fbshipit-source-id: ae077c823905b45370a26c0103ada119459da6c1
2022-02-04 17:13:10 -08:00
Yanqin Jin
c58c5596e7 Fix compilation errors and add fuzzers to CircleCI (#9420)
Summary:
This PR does the following:
- Fix compilation and linking errors when building fuzzer
- Add the above to CircleCI
- Update documentation

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9420

Test Plan: CI

Reviewed By: jay-zhuang

Differential Revision: D33849452

Pulled By: riversand963

fbshipit-source-id: 0794e5d04a3f53bfd2216fe2b3cd827ca2083ac3
2022-02-01 10:32:15 -08:00
Jay Zhuang
980b9ff385 Add more micro-benchmark tests (#9436)
Summary:
* Add more micro-benchmark tests
* Expose an API in DBImpl for waiting for compactions (still not visible to the user)
* Add argument name for ribbon_bench
* remove benchmark run from CI, as it runs too long.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9436

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D33777836

Pulled By: jay-zhuang

fbshipit-source-id: c05de3bc082cc05b5d019f00b324e774bf4bbd96
2022-02-01 09:01:55 -08:00
Yanqin Jin
fa52376117 Move RADOS support to separate repo (#9206)
Summary:
This PR moves RADOS support from RocksDB repo to a separate repo. The new (temporary?) repo
in this PR serves as an example before we finalize the decision on where and who to host RADOS support. At this point,
people can start from the example repo and fork.

The goal is to include this commit in RocksDB 7.0 release.

Reference:
https://github.com/ajkr/dedupfs by ajkr

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9206

Test Plan:
Follow instructions in https://github.com/riversand963/rocksdb-rados-env/blob/main/README.md and build
test binary `env_librados_test` and run it.

Also, make check

Reviewed By: ajkr

Differential Revision: D33751690

Pulled By: riversand963

fbshipit-source-id: 30466c62afa9e4619847a48567ed158e62835e35
2022-01-24 22:50:07 -08:00
Jay Zhuang
7711f8cbb4 Remove pyenv installation and use deps from S3 (#9406)
Summary:
* remove pyenv installation step which is not needed (it takes 3 minutes to install for every job and fail from time to time)
* download compression lib fail from time to time, Uploaded the libs to S3 and download from them for CI, which should be more stable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9406

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D33700158

Pulled By: jay-zhuang

fbshipit-source-id: be7b172d7cd059c9d7b3139fd7a34f8070460e31
2022-01-21 09:33:24 -08:00
Jay Zhuang
cd50078ae0 Update circleci xcode version (#9405)
Summary:
xcode 11.3.1 is deprecated https://circleci.com/docs/2.0/testing-ios/ , jobs are failing:
```
failed to create host: Image xcode:11.3.0 is not supported
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9405

Test Plan: CI

Reviewed By: ajkr, hx235

Differential Revision: D33674462

Pulled By: jay-zhuang

fbshipit-source-id: 85dd27aad84d26eaaa5c5375015344182b2c50b9
2022-01-20 09:41:37 -08:00
Eric Thérond
5602b1d3d9 Add support for Apple Silicon to RocksJava (#9254)
Summary:
Fixes facebook/rocksdb#7720

Updated Makefile with flags to define target architecture when compiling/linking,
and added goal `rocksdbjavastaticosxub` to build a OS X Universal Binary native library.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9254

Reviewed By: mrambacher

Differential Revision: D33551160

Pulled By: pdillinger

fbshipit-source-id: 9ce9962e03aacf55014545a6cdf638b5b14b8fa9
2022-01-12 17:20:58 -08:00
Jay Zhuang
9c6fb26033 Fix clang13 build error (#9374)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9374

Test Plan: Add CI for clang13 build

Reviewed By: riversand963

Differential Revision: D33522867

Pulled By: jay-zhuang

fbshipit-source-id: 642756825cf0b51e35861fb847ebaee4611b76ca
2022-01-11 10:36:22 -08:00
Peter Dillinger
d561432de8 Fix some CI output (#9193)
Summary:
Address some issues with https://github.com/facebook/rocksdb/issues/9188
* Internal CI doesn't render \r as anything, so use \n for "not
connected to terminal" case
* CircleCI apparently uses a pseudo-tty for output and although
rerdirect stdout (because of EAGAIN bug) we don't redirect stderr, so it
is detected as a terminal-connected case. Fix by redirecting stderr
also.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9193

Test Plan: manual, CI

Reviewed By: akankshamahajan15

Differential Revision: D32581128

Pulled By: pdillinger

fbshipit-source-id: 5ae7c3209128d8dbd4153c5b9fdb2b810e6deb2e
2021-11-20 10:21:58 -08:00
Peter Dillinger
3ce4d4f558 Print failures in parallel make check (#9188)
Summary:
Generating megabytes of successful test output has caused
issues / inconveniences for CI and internal sandcastle runs. This
changes their configuration to only print output from failed tests.
(Successful test output is still available in files under t/.)

This likewise changes default behavior of parallel `make check` as
a quick team poll showed interest in that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188

Test Plan:
Seed some test failures and observe
* `make -j24 check` (new behavior)
* `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior)
* `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior)

Reviewed By: siying

Differential Revision: D32567392

Pulled By: pdillinger

fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
2021-11-19 16:55:45 -08:00
Peter Dillinger
92e2399669 Fix EnvLibrados and add to CI (#9088)
Summary:
This feature was not part of any common or CI build, so no
surprise it broke. Now we can at least ensure compilation. I don't know
how to run the test successfully (missing config file) so it is bypassed
for now.

Fixes https://github.com/facebook/rocksdb/issues/9078

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9088

Test Plan: CI

Reviewed By: mrambacher

Differential Revision: D32009467

Pulled By: pdillinger

fbshipit-source-id: 3e0d1e5fde7f0ece703d48a81479e1cc7392c25c
2021-10-29 08:19:03 -07:00
Jay Zhuang
b4326b5273 Fix gcc-11 compile error (#9043)
Summary:
gcc11 added new static check.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9043

Test Plan: Added CI for gcc11 build

Reviewed By: zhichao-cao

Differential Revision: D31716005

Pulled By: jay-zhuang

fbshipit-source-id: 9f53be6f2f9e58e39b83359f6bbf66f945d57429
2021-10-18 12:22:37 -07:00
Peter Dillinger
bda8d93ba9 Fix and detect headers with missing dependencies (#8893)
Summary:
It's always annoying to find a header does not include its own
dependencies and only works when included after other includes. This
change adds `make check-headers` which validates that each header can
be included at the top of a file. Some headers are excluded e.g. because
of platform or external dependencies.

rocksdb_namespace.h had to be re-worked slightly to enable checking for
failure to include it. (ROCKSDB_NAMESPACE is a valid namespace name.)

Fixes mostly involve adding and cleaning up #includes, but for
FileTraceWriter, a constructor was out-of-lined to make a forward
declaration sufficient.

This check is not currently run with `make check` but is added to
CircleCI build-linux-unity since that one is already relatively fast.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8893

Test Plan: existing tests and resolving issues detected by new check

Reviewed By: mrambacher

Differential Revision: D30823300

Pulled By: pdillinger

fbshipit-source-id: 9fff223944994c83c105e2e6496d24845dc8e572
2021-09-10 10:00:26 -07:00
Levi Tamasi
2e09a54c4f Update branch name to "main" in CircleCI config (#8726)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8726

Reviewed By: jay-zhuang

Differential Revision: D30675182

Pulled By: ltamasi

fbshipit-source-id: d5746931f6d942ed3a9d21325335cfc9e111a7f3
2021-09-01 12:26:42 -07:00
Jay Zhuang
31193a73a4 Add missing steps for cmake build (#8524)
Summary:
Some cmake and test configuration are set in pre-steps
enviroment variables. Add the missing steps.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524

Test Plan: CI pass

Reviewed By: siying

Differential Revision: D29682731

Pulled By: jay-zhuang

fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d
2021-07-15 13:37:49 -07:00
Jay Zhuang
5dd18a8d8e Add micro-benchmark support (#8493)
Summary:
Add google benchmark for microbench.
Add ribbon_bench for benchmark ribbon filter vs. other filters.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493

Test Plan:
added test to CI
To run the benchmark on devhost:
Install benchmark: `$ sudo dnf install google-benchmark-devel`
Build and run:
`$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench`
or with cmake:
`$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench`

Reviewed By: pdillinger

Differential Revision: D29589649

Pulled By: jay-zhuang

fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
2021-07-08 18:22:45 -07:00
Akanksha Mahajan
d96b0127a0 Change Windows image back to stable (#8424)
Summary:
Since windows timeout issue has been fixed. Change the image back to
stable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8424

Test Plan: Check CircleCI jobs

Reviewed By: zhichao-cao

Differential Revision: D29235219

Pulled By: akankshamahajan15

fbshipit-source-id: 2c111f96e216dac4ae3d7ec3b4cdd8e459575d37
2021-06-18 12:32:01 -07:00
Akanksha Mahajan
9047fd7766 Fix Windows build failure (#8400)
Summary:
Fix window build failure by reverting to previous tag as suggested by CircleCI.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8400

Test Plan: Watch CircleCI builds for a day or two for failure

Reviewed By: jay-zhuang

Differential Revision: D29104458

Pulled By: akankshamahajan15

fbshipit-source-id: 5df03092e4b0c221ee12daad7d1fdf8d35eb1082
2021-06-14 09:35:32 -07:00
Jay Zhuang
eda83eaac0 Fix cmake build failure with gflags (#8324)
Summary:
- Fix cmake build failure with gflags.
- Add CI tests for both gflags 2.1 and 2.2.
- Fix ctest config with gtest.
- Add CI to run test with ctest.

One benefit of ctest is it support timeout, it's set to 5min in our CI, so we will know which test is hang.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8324

Test Plan: CI pass

Reviewed By: ajkr

Differential Revision: D28762517

Pulled By: jay-zhuang

fbshipit-source-id: 09063c5af5f9f33abfcdeb48593acbd9826cd199
2021-06-01 14:43:15 -07:00
Jay Zhuang
6c7c3e8cb3 Use large macos instance (#8320)
Summary:
Macos build is taking more than 1 hour, bump the instance type from the
default medium to large (large macos instance was not available before).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8320

Test Plan: watch CI pass

Reviewed By: ajkr

Differential Revision: D28589456

Pulled By: jay-zhuang

fbshipit-source-id: cff78dae5aaf9de90ade3468469290176de5ff32
2021-05-21 18:17:03 -07:00
Jay Zhuang
e9a0bc14dd Fix cmake failed to build db_bench (#8289)
Summary:
And change the cmake build on macos with GFLAGS on to cover more cases.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8289

Reviewed By: zhichao-cao

Differential Revision: D28372467

Pulled By: jay-zhuang

fbshipit-source-id: ad7fbe523c3fb135ef5281adbaf2070ca5d0873d
2021-05-12 11:39:01 -07:00
Jay Zhuang
a6e425dc44 Fix a minor clang release build failure (#8290)
Summary:
Error message:
```
cache/clock_cache.cc:434:14: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'uint32_t' (aka 'unsigned int') [-Werror,-Wshorten-64-to-32]
    *state = end_idx;
           ~ ^~~~~~~
```
Make circleci to cover this case by install tbb.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8290

Test Plan: `USE_CLANG=1 make -j1 release`

Reviewed By: akankshamahajan15

Differential Revision: D28374672

Pulled By: jay-zhuang

fbshipit-source-id: e8c3ee46f2a008e8a599413292e5a4b5151365df
2021-05-12 10:45:29 -07:00
Jay Zhuang
8a92564a82 Update CircleCI MacOS Xcode version to 11.3.0 (#8256)
Summary:
To fix CircleCI pyenv installation failure.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8256

Reviewed By: ajkr

Differential Revision: D28191772

Pulled By: jay-zhuang

fbshipit-source-id: 2bbb1d5ded473e510c11c8ed27884c4ad073973f
2021-05-04 10:34:31 -07:00
anand76
8fe33a0a9f Change CircleCI Windows to previous known good image (#8220)
Summary:
This is to try to resolve the VS2015 install failure in CircleCI Windows builds.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8220

Reviewed By: jay-zhuang

Differential Revision: D28061834

Pulled By: anand1976

fbshipit-source-id: b2663eb60babee603669a2c2cb55f182df1cc7b1
2021-04-28 11:30:30 -07:00
Jay Zhuang
a89740fbc6 Fix unittest no space issue (#8204)
Summary:
Unittest reports no space from time to time, which can be reproduced on a small memory machine with SHM. It's caused by large WAL files generated during the test, which is preallocated, but didn't truncate during close(). Adding the missing APIs to set preallocation.
It added arm test as nightly build, as the test runs more than 1 hour.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8204

Test Plan: test on small memory arm machine

Reviewed By: mrambacher

Differential Revision: D27873145

Pulled By: jay-zhuang

fbshipit-source-id: f797c429d6bc13cbcc673bc03fcc72adda55f506
2021-04-20 08:42:28 -07:00
Jay Zhuang
a345b4d60d Move arm build from travis to circleci (#8203)
Summary:
Moving ARM build from travis to CircleCI.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8203

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D27861753

Pulled By: jay-zhuang

fbshipit-source-id: 5e36a67f6fbb921c2ed80b284ba2de485411937b
2021-04-19 20:07:02 -07:00
Peter Dillinger
bd7ddf58cb Make tests "parallel" and "passing ASC" by default (#8146)
Summary:
New tests should by default be expected to be parallelizeable
and passing with ASSERT_STATUS_CHECKED. Thus, I'm changing those two
lists to exclusions rather than inclusions.

For the set of exclusions, I only listed things that currently failed
for me when attempting not to exclude, or had some other documented
reason. This marks many more tests as "parallel," which will potentially
cause some failures from self-interference, but we can address those as
they are discovered.

Also changed CircleCI ASC test to be parallelized; the easy way to do
that is to exclude building tests that don't pass ASC, which is now a
small set.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8146

Test Plan: Watch CI, etc.

Reviewed By: riversand963

Differential Revision: D27542782

Pulled By: pdillinger

fbshipit-source-id: bdd74bcd912a963ee33f3fc0d2cad2567dc7740f
2021-04-04 20:10:11 -07:00
Jay Zhuang
65487da1f5 Add circleci format_compatible nightly build (#7926)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7926

Test Plan: manual trigger the test: https://app.circleci.com/pipelines/github/facebook/rocksdb/5718/workflows/0427934c-7629-4e1e-a523-1fa1ab8e8f59/jobs/91413

Reviewed By: pdillinger

Differential Revision: D26238070

Pulled By: jay-zhuang

fbshipit-source-id: 5f6ca75d922a88f9e8c8b17d6d7b026506ff8638
2021-02-09 20:48:53 -08:00
Adam Retter
e4bdf9e0cb Small improvements to CircleCI Windows jobs (#7852)
Summary:
* Clearer indication of which versions of msbuild and Visual Studio is used
* Explicit naming of the build jobs within the Windows workflows

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7852

Reviewed By: akankshamahajan15

Differential Revision: D25864444

Pulled By: jay-zhuang

fbshipit-source-id: 0d618ad8a8892d5a2575cdfaa59d61a989c4df4b
2021-01-11 13:32:07 -08:00
Adam Retter
c22e619f7e Store test logs as artifacts if the build fails in CircleCI (#7812)
Summary:
If a workflow fails in CircleCI this will ensure that the `t/` directory is tar'd up and added to the workflow as an artifact. This allows us to download the detailed logs and see what went wrong.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7812

Reviewed By: zhichao-cao

Differential Revision: D25761003

Pulled By: jay-zhuang

fbshipit-source-id: 41cfd16c6385bfcc9fb35fb63df84f97d4b8b80b
2021-01-07 17:13:19 -08:00
Peter Dillinger
4d897e51df Migrate away from Travis+Linux+amd64 (#7791)
Summary:
This disables Linux/amd64 builds in Travis for PRs, and adds a
gcc-10+c++20 build in CircleCI, which should fill out sufficient coverage
vs. what we had in Travis

Fixed a use of std::is_pod, which is deprecated in c++20

Fixed ++ on a volatile in db_repl_stress.cc, with bigger refactoring.
Although ++ on this volatile was probably ok with one thread writer and
one thread reader, the code was still overly complex. There was a
deadcode check for error
`if (replThread.no_read < dataPump.no_records)` which can be proven
never to happen based on the structure of the code. It infinite loops
instead for the case intended to be checked. I just simplified the code
for what should be the same checking power.

Also most configurations seem to be using make parallelism = 2 * vcores,
so fixing / using that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7791

Test Plan:
CI
and `while ./db_repl_stress; do echo again; done` for a while

Reviewed By: siying

Differential Revision: D25669834

Pulled By: pdillinger

fbshipit-source-id: b2c688053d0b1d52c989903449d3cd27a04130d6
2020-12-22 00:20:57 -08:00
Adam Retter
29d12748b0 Fix failing RocksJava test compilation and add CI (#7769)
Summary:
* Fixes a Java test compilation issue on macOS
* Cleans up CircleCI RocksDBJava build config
* Adds CircleCI for RocksDBJava on MacOS
* Ensures backwards compatibility with older macOS via CircleCI
* Fixes RocksJava static builds ordering
* Adds missing RocksJava static builds to CircleCI for Mac and Linux
* Improves parallelism in RocksJava builds
* Reduces the size of the machines used for RocksJava CircleCI as they don't need to be so large (Saves credits)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7769

Reviewed By: akankshamahajan15

Differential Revision: D25601293

Pulled By: pdillinger

fbshipit-source-id: 0a0bb9906f65438fe143487d78e37e1947364d08
2020-12-16 16:00:02 -08:00
Jay Zhuang
07c0fc002a Fix pyenv-version-name not found issue (#7768)
Summary:
Remove the workaround for that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7768

Reviewed By: cheng-chang

Differential Revision: D25487150

Pulled By: jay-zhuang

fbshipit-source-id: 4514dfc64eb56d2dad48f9a24176c21af8107def
2020-12-10 20:37:51 -08:00
Peter Dillinger
e34b2e9f41 Migrate away from broken macos on Travis (#7745)
Summary:
Add macos+cmake build on CircleCI instead.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7745

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D25352864

Pulled By: pdillinger

fbshipit-source-id: 6b0a328cbe715bc3b43d70e919a27c834edcf079
2020-12-04 23:24:27 -08:00
Akanksha Mahajan
6cacb0d3a1 Add cmake-mingw in circle-build (#7144)
Summary:
Add cmake-mignw in circle-build

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7144

Test Plan: watch circle cmake-mingw build

Reviewed By: jay-zhuang

Differential Revision: D25039744

Pulled By: akankshamahajan15

fbshipit-source-id: 92584c9d5ad161b93d5e5a1303aac306e7985108
2020-11-17 18:19:50 -08:00
Peter Dillinger
c57f914482 Use NPHash64 in more places (#7632)
Summary:
Since the hashes should not be persisted in output_validator
nor mock_env.

Also updated NPHash64 to use 64-bit seed, and comments.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7632

Test Plan:
make check, and new build setting that enables modification
to NPHash64, to check for behavior depending on specific values. Added
that setting to one of the CircleCI configurations.

Reviewed By: jay-zhuang

Differential Revision: D24833780

Pulled By: pdillinger

fbshipit-source-id: 02a57652ccf1ac105fbca79e77875bb7bf7c071f
2020-11-10 23:42:13 -08:00
mrambacher
f35f7f2704 Fix many tests to run with MEM_ENV and ENCRYPTED_ENV; Introduce a MemoryFileSystem class (#7566)
Summary:
This PR does a few things:

1.  The MockFileSystem class was split out from the MockEnv.  This change would theoretically allow a MockFileSystem to be used by other Environments as well (if we created a means of constructing one).  The MockFileSystem implements a FileSystem in its entirety and does not rely on any Wrapper implementation.

2.  Make the RocksDB test suite work when MOCK_ENV=1 and ENCRYPTED_ENV=1 are set.  To accomplish this, a few things were needed:
- The tests that tried to use the "wrong" environment (Env::Default() instead of env_) were updated
- The MockFileSystem was changed to support the features it was missing or mishandled (such as recursively deleting files in a directory or supporting renaming of a directory).

3.  Updated the test framework to have a ROCKSDB_GTEST_SKIP macro.  This can be used to flag tests that are skipped.  Currently, this defaults to doing nothing (marks the test as SUCCESS) but will mark the tests as SKIPPED when RocksDB is upgraded to a version of gtest that supports this (gtest-1.10).

I have run a full "make check" with MEM_ENV, ENCRYPTED_ENV,  both, and neither under both MacOS and RedHat.  A few tests were disabled/skipped for the MEM/ENCRYPTED cases.  The error_handler_fs_test fails/hangs for MEM_ENV (presumably a timing problem) and I will introduce another PR/issue to track that problem.  (I will also push a change to disable those tests soon).  There is one more test in DBTest2 that also fails which I need to investigate or skip before this PR is merged.

Theoretically, this PR should also allow the test suite to run against an Env loaded from the registry, though I do not have one to try it with currently.

Finally, once this is accepted, it would be nice if there was a CircleCI job to run these tests on a checkin so this effort does not become stale.  I do not know how to do that, so if someone could write that job, it would be appreciated :)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7566

Reviewed By: zhichao-cao

Differential Revision: D24408980

Pulled By: jay-zhuang

fbshipit-source-id: 911b1554a4d0da06fd51feca0c090a4abdcb4a5f
2020-10-27 10:33:09 -07:00
Cheng Chang
cb2581031a Add macos tests to circleci (#7536)
Summary:
as title

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7536

Test Plan: see the new `build-macos` tests pass in circleci

Reviewed By: jay-zhuang

Differential Revision: D24243218

Pulled By: cheng-chang

fbshipit-source-id: 9b5f8a859e54c99a9ebe7efff6f336458a5d42de
2020-10-12 10:46:40 -07:00