Commit Graph

412 Commits

Author SHA1 Message Date
Peter Dillinger
ea3aa60dcc More improvements to output for CircleCI (#9201)
Summary:
More follow-up to https://github.com/facebook/rocksdb/issues/9193 + https://github.com/facebook/rocksdb/issues/9188
* Even though we need to print ETA updates to avoid hitting the 10min
timeout, we need to avoid printing an update if there's no actual
progress, so that hung tests will timeout after 10 min rather than 5
hours.
* When there is a hung test, it's really annoying to track down which
test is hung, so if no progress is observed for 1 minute, we run ps once
to show what is running.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9201

Test Plan: manual and CircleCI

Reviewed By: jay-zhuang

Differential Revision: D32612028

Pulled By: pdillinger

fbshipit-source-id: 00f8ea70fc5fec9ede28ff74287d90fc73854aad
2021-11-23 22:10:27 -08:00
Peter Dillinger
d561432de8 Fix some CI output (#9193)
Summary:
Address some issues with https://github.com/facebook/rocksdb/issues/9188
* Internal CI doesn't render \r as anything, so use \n for "not
connected to terminal" case
* CircleCI apparently uses a pseudo-tty for output and although
rerdirect stdout (because of EAGAIN bug) we don't redirect stderr, so it
is detected as a terminal-connected case. Fix by redirecting stderr
also.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9193

Test Plan: manual, CI

Reviewed By: akankshamahajan15

Differential Revision: D32581128

Pulled By: pdillinger

fbshipit-source-id: 5ae7c3209128d8dbd4153c5b9fdb2b810e6deb2e
2021-11-20 10:21:58 -08:00
Peter Dillinger
3ce4d4f558 Print failures in parallel make check (#9188)
Summary:
Generating megabytes of successful test output has caused
issues / inconveniences for CI and internal sandcastle runs. This
changes their configuration to only print output from failed tests.
(Successful test output is still available in files under t/.)

This likewise changes default behavior of parallel `make check` as
a quick team poll showed interest in that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188

Test Plan:
Seed some test failures and observe
* `make -j24 check` (new behavior)
* `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior)
* `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior)

Reviewed By: siying

Differential Revision: D32567392

Pulled By: pdillinger

fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
2021-11-19 16:55:45 -08:00
Peter Dillinger
d95ffbaf4f Parallelize sandcastle tests (#9178)
Summary:
Some tests are timing out, so increase parallelism but also
keep test output on console

Large credit to jay-zhuang who is currently unavailable to revise & land https://github.com/facebook/rocksdb/issues/9135

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9178

Test Plan:
make check and valgrind_test, with and without PRINT_PARALLEL_OUTPUTS=1
https://www.internalfb.com/intern/sandcastle/group/nonce/6211105410048528/

Reviewed By: riversand963

Differential Revision: D32476680

Pulled By: pdillinger

fbshipit-source-id: 8844947416e5baf4435e1ef6e33aeecc45621a89
2021-11-16 16:45:13 -08:00
Sahir Hoda
1178d348aa Fix portable mac shared_library ld flags (#9149)
Summary:
Move the 'macosx-version-min' arg to the front of PLATFORM_SHARED_LDFLAGS so that it doesn't get concatenated with the library name. Fixes https://github.com/facebook/rocksdb/issues/9146

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9149

Reviewed By: mrambacher

Differential Revision: D32396101

Pulled By: pdillinger

fbshipit-source-id: aefcf53384e64d399049f158779acc3a4e54a8fe
2021-11-16 12:17:17 -08:00
Jack Feng
7d74d3471c Fixing more loosely formatted issue (#9140)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9140

Upon checking, some jobs still have loosely formatted issue.
Also, some started failing due to my previous buggy change.

https://fburl.com/scuba/sandcastle/e9mhejsc
https://fburl.com/scuba/duplo_events/dbxx4215

Reviewed By: jay-zhuang

Differential Revision: D32213703

fbshipit-source-id: 806f872b820afe275cf62459988c9d6f668af35c
2021-11-05 12:21:38 -07:00
Jack Feng
8be121bb80 Fixing loosely formatted child job specs, double quotes (#9125)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9125

See task for more context

Reviewed By: jay-zhuang

Differential Revision: D32157345

fbshipit-source-id: be6631c6400fd66e6891e71dc0235798c594010a
2021-11-04 15:34:51 -07:00
Jack Feng
56810142a5 Fixing child jobs of rocksdb for duplo deprecation (#9117)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9117

This fixes the loosely formatted json child job spec. I.e., trailing commas

Reviewed By: jay-zhuang

Differential Revision: D32119606

fbshipit-source-id: 0ef571ccbef0e638df18f17288611d3b5774f129
2021-11-03 14:56:26 -07:00
Peter Dillinger
4ec31dc8ac Make format-diff.sh locale-independent (#9079)
Summary:
Force POSIX locale for calls to 'git remote' that might have
locale-dependent formatting, as shown in https://github.com/facebook/rocksdb/issues/8731 comment

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9079

Test Plan:
manual (haven't tried on a machine with non-english default
locale)

Reviewed By: ltamasi

Differential Revision: D31943092

Pulled By: pdillinger

fbshipit-source-id: 7dbe5915824f39f73b412cc3d1a86a2521cf76c1
2021-10-27 12:26:36 -07:00
Jonathan Albrecht
e970248602 Add support for building on s390x platform (#8962)
Summary:
This PR adds support for building on s390x including updating travis CI. It uses the previous work in https://github.com/facebook/rocksdb/pull/6168 and adds some more changes to get all current tests (make check and jni tests) to pass. The tests were run with snappy, lz4, bzip2 and zstd all compiled in.

There are a few pieces still needed to get the travis build working that I don't think I can do. adamretter is this something you could help with?

1. A prebuilt https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-s390x.deb package
2. A https://hub.docker.com/r/evolvedbinary/rocksjava s390x image

Not sure if there is more required for travis. Happy to help in any way I can.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8962

Reviewed By: mrambacher

Differential Revision: D31802198

Pulled By: pdillinger

fbshipit-source-id: 683511466fa6b505f85ba5a9964a268c6151f0c2
2021-10-22 10:13:15 -07:00
mikael
c246c9c6e2 Remove -Wshorten-64-to-32 FreeBSD on aarch64 as it breaks compilation. (#9010)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9010

Reviewed By: pdillinger

Differential Revision: D31613434

Pulled By: ajkr

fbshipit-source-id: 305a84fa715d0121fa65abaea3d32bad41233957
2021-10-14 14:38:47 -07:00
Huachao Huang
e5aa7deae1 Fix format script for Ubuntu (#9028)
Summary:
I get `clang-format-diff` after running `apt install clang-format` on Ubuntu instead of `clang-format-diff.py`. So I think it makes sense to make the format script compatible with this behavior.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9028

Reviewed By: ajkr

Differential Revision: D31634041

Pulled By: jay-zhuang

fbshipit-source-id: b936de791ddcafa6ff304039ef33936e1e04864d
2021-10-14 12:27:00 -07:00
anand76
418831cf45 Fix wrong tmp dir name in fbcode stress test lego command (#9000)
Summary:
Change the directory to /dev/shm/rocksdb_fbcode_crash_test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9000

Reviewed By: akankshamahajan15

Differential Revision: D31511886

Pulled By: anand1976

fbshipit-source-id: 4e48fd2338638fd9223b55d5f7496c26d331bfc7
2021-10-08 13:39:41 -07:00
anand76
2f1296ef48 Allow more flexible invocation of fbcode_stress_crash (#8985)
Summary:
Instead of hardcoding the stress test type and some args, allow it to be passed through env variables.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8985

Reviewed By: akankshamahajan15

Differential Revision: D31349495

Pulled By: anand1976

fbshipit-source-id: 585c8fcb0232d0a95925b1a8c4e42a0940227e8b
2021-10-08 10:28:05 -07:00
Jay Zhuang
0595101c4f Add microbench build support for fbcode (#8954)
Summary:
For internal build enviroment only. Developer could run the
microbenchmark without `ROCKSDB_NO_FBCODE=1`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8954

Test Plan: `$ make microbench` on dev server

Reviewed By: ajkr

Differential Revision: D31163717

Pulled By: jay-zhuang

fbshipit-source-id: 1ff59f660ca05afd0fd5c7c7dcdfd831ac365462
2021-09-24 10:23:35 -07:00
sdong
64ca0d9b46 Adjust contrun name (#8924)
Summary:
One contrun name is incorrect, which mixed error reporting with another one. Fix it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8924

Reviewed By: ltamasi

Differential Revision: D30999477

fbshipit-source-id: 46a04b2e4b48f755181aa9a47c353d91f1128469
2021-09-16 15:06:30 -07:00
Peter Dillinger
cb5b851ff8 Add (& fix) some simple source code checks (#8821)
Summary:
* Don't hardcode namespace rocksdb (use ROCKSDB_NAMESPACE)
* Don't #include <rocksdb/...> (use double quotes)
* Support putting NOCOMMIT (any case) in source code that should not be
committed/pushed in current state.

These will be run with `make check` and in GitHub actions

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8821

Test Plan: existing tests, manually try out new checks

Reviewed By: zhichao-cao

Differential Revision: D30791726

Pulled By: pdillinger

fbshipit-source-id: 399c883f312be24d9e55c58951d4013e18429d92
2021-09-07 21:19:27 -07:00
Jay Zhuang
6cca9fab7c Remove asan_symbolize.py for internal asan build (#8737)
Summary:
asan_symbolize.py is not compatible with python3. Also make it
consistent with public CI, which doesn't use asan_symbolize.py
And update coverage_test.sh to use python3.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8737

Test Plan: CI

Reviewed By: pdillinger

Differential Revision: D30702430

Pulled By: jay-zhuang

fbshipit-source-id: ef09947b1232294d31b09a855c2f0ce149097dd9
2021-09-07 15:39:11 -07:00
Jay Zhuang
e8eb02145a Fix a minor regression script issue (#8755)
Summary:
The system default `time` doesn't support option -v

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8755

Test Plan: CI: https://www.internalfb.com/intern/sandcastle/job/13510799359724405

Reviewed By: ltamasi

Differential Revision: D30757119

Pulled By: jay-zhuang

fbshipit-source-id: 093e5084f3b7cc71f6795b1062f48d4e77ed4518
2021-09-04 17:35:22 -07:00
Levi Tamasi
972e3400e1 Update branch name in rocksdb-lego-determinator (#8754)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8754

Reviewed By: jay-zhuang

Differential Revision: D30754986

Pulled By: ltamasi

fbshipit-source-id: 4c879a8deaaae07a921a35ff5ed1939d4262f982
2021-09-03 21:13:39 -07:00
Jay Zhuang
0c942a9c0d Fix regression test script (#8753)
Summary:
Regression test is broken and not running:
1. failed test is not reporting, fix it by add `set -e`
2. internal regression test is not run inside github, removing that
3. fix a few minor issues to pass the test
4. delete unused binary size build, and regression test is reporting binary size now.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8753

Test Plan: CI: https://www.internalfb.com/intern/sandcastle/job/13510799359573861

Reviewed By: ltamasi

Differential Revision: D30754380

Pulled By: jay-zhuang

fbshipit-source-id: 0cfa008327fff31bc61118a3fe642924090d28e1
2021-09-03 19:05:33 -07:00
Peter Dillinger
ac48e0d589 Fix fbcode linker error with make shared_lib (#8742)
Summary:
Need proper linker path for linking shared library

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8742

Test Plan: `make shared_lib` on Facebook machine

Reviewed By: jay-zhuang

Differential Revision: D30709012

Pulled By: pdillinger

fbshipit-source-id: 6d17e281204c359aced85e18212e59758cec3313
2021-09-01 15:26:34 -07:00
Levi Tamasi
f756448f28 Make format-diff.sh branch name agnostic (#8731)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8731

Reviewed By: jay-zhuang

Differential Revision: D30678124

Pulled By: ltamasi

fbshipit-source-id: 0131b6707f0c5d1d887bcd45781623143b5ccae0
2021-09-01 12:26:42 -07:00
Jay Zhuang
35dd251187 build with platform assembler (#8733)
Summary:
Required for platform009, which is incompat with the centos
assembler.
author: pbrady@fb.com D29099768

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8733

Test Plan: CI

Reviewed By: pixelb

Differential Revision: D30687156

Pulled By: jay-zhuang

fbshipit-source-id: 796f90842cbf0ca11bad07e7d654bce1fafc4ba0
2021-09-01 10:27:32 -07:00
Andrew Kryczka
f484a60d1f Fix legocastle Python commands for CentOS 8 (#8701)
Summary:
There is no longer an unversioned `python` command that refers to Python
3; the recommended alternative is `/usr/bin/env python3`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8701

Test Plan: - [internal link] https://www.internalfb.com/intern/sandcastle/group/nonce/5100000000000001/

Reviewed By: riversand963

Differential Revision: D30520380

Pulled By: ajkr

fbshipit-source-id: 2af459a64a15fb2a011e98b156f31d322f6b2d25
2021-08-25 16:53:54 -07:00
anand76
ddf439c561 Pass extra db_stress args to fbcode crash tests (#8587)
Summary:
Allow extra arguments to be passed to db_stress in fbcode crash tests by the ```rocksdb-lego-determinator``` invoker.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8587

Reviewed By: zhichao-cao

Differential Revision: D29940217

Pulled By: anand1976

fbshipit-source-id: 17cbcd2def60eff2a895553f917694496c4742aa
2021-07-27 12:46:47 -07:00
Jay Zhuang
daf7e77a6b Disable DistributedMutex test by default (#8584)
Summary:
DistributedMutex hasn't been used in the code base and enabling
`USE_FOLLY_DISTRIBUTED_MUTEX` only runs the mutex tests from third-party
lib. So disabling it for now.
The implementation may also out of date, should re-sync with folly before
using.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8584

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D29888960

Pulled By: jay-zhuang

fbshipit-source-id: 3e75f73386c6ed03efb96a1400258d602a724f17
2021-07-23 15:55:23 -07:00
Jay Zhuang
5dd18a8d8e Add micro-benchmark support (#8493)
Summary:
Add google benchmark for microbench.
Add ribbon_bench for benchmark ribbon filter vs. other filters.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493

Test Plan:
added test to CI
To run the benchmark on devhost:
Install benchmark: `$ sudo dnf install google-benchmark-devel`
Build and run:
`$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench`
or with cmake:
`$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench`

Reviewed By: pdillinger

Differential Revision: D29589649

Pulled By: jay-zhuang

fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
2021-07-08 18:22:45 -07:00
anand76
41d32152ce Enable crash test to run using fbcode components (#8471)
Summary:
Add a new test ```fbcode_crash_test``` to rocksdb-lego-determinator. This test allows the crash test to be run on Facebook Sandcastle infra using fbcode components. Also use the default Env in db_stress to access the expected values path as it requires a memory mapped file and may not work with custom Envs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8471

Reviewed By: ajkr

Differential Revision: D29474722

Pulled By: anand1976

fbshipit-source-id: 7d086d82dd7091ae48e08cb4ace763ce3e3b87ef
2021-07-01 12:23:01 -07:00
Peter Dillinger
32a4d59477 Remove TSAN crash test opt-in to platform007 (#8406)
Summary:
Facebook internal test failure

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8406

Test Plan: COMPILE_WITH_TSAN=1 make crash_test_with_txn

Reviewed By: akankshamahajan15

Differential Revision: D29124055

Pulled By: pdillinger

fbshipit-source-id: f0f15d6a71435043b19f2170e7500823de1cc43a
2021-06-15 08:17:01 -07:00
Peter Dillinger
3981430f54 Fix runtime linkage with libasan in Facebook platform009 (#8402)
Summary:
Was seeing

    ./cache_test: error while loading shared libraries: libasan.so.5: cannot open shared object file: No such file or directory

etc. using COMPILE_WITH_ASAN=1 without USE_CLANG=1

Now including compiler libs in runtime ld path.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8402

Test Plan: reproduced with local builds

Reviewed By: akankshamahajan15

Differential Revision: D29107729

Pulled By: pdillinger

fbshipit-source-id: 13805b87b846b39522c9dd6a231ca245c58f1c71
2021-06-14 12:16:11 -07:00
Peter Dillinger
79407a8a61 Fix^2 use of binutils in Facebook platform009 (#8399) (#8401)
Summary:
Internal builds still failing, this time with ld

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8401

Test Plan: Like https://github.com/facebook/rocksdb/issues/8399 but letting build run to completion

Reviewed By: bjlemaire

Differential Revision: D29103512

Pulled By: pdillinger

fbshipit-source-id: 0fcad2c63518cf2b721e749881da40b90f5d3133
2021-06-14 08:45:21 -07:00
Peter Dillinger
0d0aa578ff Fix use of binutils in Facebook platform009 (#8399)
Summary:
Internal builds failing

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8399

Test Plan:
I can reproduce a failure by putting a bad version of `as` in
my PATH. This indicates that before this change, the custom compiler is
falsely relying on host `as`. This change fixes that, ignoring the bad
`as` on PATH.

Reviewed By: akankshamahajan15

Differential Revision: D29094159

Pulled By: pdillinger

fbshipit-source-id: c432e90404ea4d39d885a685eebbb08be9eda1c8
2021-06-13 23:33:31 -07:00
Peter Dillinger
a42a342a7a Make platform009 default for FB developers (#8389)
Summary:
platform007 being phased out and sometimes broken

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8389

Test Plan: `make V=1` to see which compiler is being used

Reviewed By: jay-zhuang

Differential Revision: D29067183

Pulled By: pdillinger

fbshipit-source-id: d1b07267cbc55baa9395f2f4fe3967cc6dad52f7
2021-06-11 11:37:05 -07:00
sdong
bd3d080ef8 Try to build with liburing by default. (#8322)
Summary:
By default, try to build with liburing. For make, if ROCKSDB_USE_IO_URING is not set, treat as 1, which means RocksDB will try to build with liburing. For cmake, add WITH_LIBURING to control it, with default on.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8322

Test Plan: Build using cmake and make.

Reviewed By: anand1976

Differential Revision: D28586498

fbshipit-source-id: cfd39159ab697f4b93a9293a59c07f839b1e7ed5
2021-05-21 10:21:53 -07:00
Peter Dillinger
35af0433cf Fix crash test with backup as read-only DB (#8161)
Summary:
Forgot to re-test crash test after adding read-only filesystem
enforcement to https://github.com/facebook/rocksdb/issues/8142. The problem is ReadOnlyFileSystem would reject
CreateDirIfMissing whenever DBOptions::create_if_missing=true. The fix
that is better for users is to allow CreateDirIfMissing in
ReadOnlyFileSystem if the directory exists, so that they don't cause a
failure on using create_if_missing with opening backups as read-only
DBs. Added this option test to the unit test (in addition to being in the
crash test).

Also fixed a couple of lints.

And some better messaging from 'make format' so that when you run it
with uncommitted changes, it's clear that it's only checking the
uncommitted changes.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8161

Test Plan: local blackbox_crash_test with amplified backup_one_in

Reviewed By: ajkr

Differential Revision: D27614409

Pulled By: pdillinger

fbshipit-source-id: 63ccb626c7e34c200d61c6bca2a8f60da9015179
2021-04-06 23:31:51 -07:00
mrambacher
493a4e28d9 Pass PLATFORM_FLAGS in build_detect_platform (#8111)
Summary:
At least under MacOS, some things were excluded from the build (like Snappy) because the compilation flags were not passed in correctly.  This PR does a few things:
- Passes the EXTRA_CXX/LDFLAGS into build_detect_platform.  This means that if some tool (like TBB for example) is not installed in a standard place, it could still be detected by build_detect_platform.  In this case, the developer would invoke: "EXTRA_CXXFLAGS=<path to TBB include> EXTRA_LDFLAGS=<path to TBB library> make", and the build script would find the tools in the extra location.
- Changes the compilation tests to use PLATFORM_CXXFLAGS.  This change causes the EXTRA_FLAGS passed in to the script to be included in the compilation check.  Additionally, flags set by the script itself (like --std=c++11) will be used during the checks.

Validated that the make_platform.mk file generated on Linux does not change with this change.  On my MacOS machine, the SNAPPY libraries are now available (they were not before as they required --std=c++11 to build).

I also verified that I can build against TBB installed on my Mac by passing in the EXTRA CXX and LD FLAGS to the location in which TBB is installed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8111

Reviewed By: jay-zhuang

Differential Revision: D27353516

Pulled By: mrambacher

fbshipit-source-id: b6b378c96dbf678bab1479556dcbcb49c47e807d
2021-03-31 07:40:46 -07:00
Adam Retter
24b7ebee80 range_tree requires GNU libc on ppc64 (#8070)
Summary:
If the platform is ppc64 and the libc is not GNU libc, then we exclude the range_tree from compilation.

See https://jira.percona.com/browse/PS-7559

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8070

Reviewed By: jay-zhuang

Differential Revision: D27246004

Pulled By: mrambacher

fbshipit-source-id: 59d8433242ce7ce608988341becb4f83312445f5
2021-03-29 16:32:08 -07:00
Yanqin Jin
469164dc3c Add stress crash test with timestamp to lego determinator (#8104)
Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8104

Test Plan: build_tools/rocksdb-lego-determinator stress_crash_with_ts

Reviewed By: ltamasi

Differential Revision: D27312265

Pulled By: riversand963

fbshipit-source-id: 3175a9d9074bdb282137c6518402d622436931d6
2021-03-24 17:58:31 -07:00
sherriiiliu
f91fd0c944 Extract test cases correctly in run_ci_db_test.ps1 script (#7989)
Summary:
Extract test cases correctly in run_ci_db_test.ps1 script.

There are some new test group that are ended with # comments. Previously in the script when trying to extract test groups and test cases, the regex rule did not apply to this case so the concatenation of some test group and test case failed, see examples in comments.

Also removed useless trailing whitespaces in the script.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7989

Reviewed By: jay-zhuang

Differential Revision: D26615909

Pulled By: ajkr

fbshipit-source-id: 8e68d599994f17d6fefde0daa925c3018179521a
2021-02-23 14:25:42 -08:00
Yanqin Jin
170dffac83 Use actual url instead of tinyurl.com (#7950)
Summary:
Due to offline discussion, we use actual url of the clang-format-diff.py and add a note.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7950

Reviewed By: pdillinger

Differential Revision: D26370822

Pulled By: riversand963

fbshipit-source-id: 7508e23c002d56d5c1649090438ef5f8ff2cdbe7
2021-02-10 10:08:09 -08:00
Yanqin Jin
48669be618 Update clang-format-diff.py path (#7944)
Summary:
Recent Github actions of format checking fail due to invalid location
from where clang-format-diff.py is downloaded. Update the path to point
to a stable, archived location.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7944

Test Plan: manually check the result of Github action.

Reviewed By: ltamasi

Differential Revision: D26345066

Pulled By: riversand963

fbshipit-source-id: 2b1a58c2e59c2f1eb11202d321d2ea002cb0917e
2021-02-09 12:49:38 -08:00
Jay Zhuang
a9f8f1fe38 Missed skip_format_check for lite build (#7927)
Summary:
Missed lite build in https://github.com/facebook/rocksdb/issues/7911.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7927

Test Plan: Manual build: https://www.internalfb.com/intern/sandcastle/job/4503600004036336

Reviewed By: ltamasi

Differential Revision: D26226135

Pulled By: jay-zhuang

fbshipit-source-id: e9c51a8f4b6d5469521e0d788ead971a1c1b48ff
2021-02-03 10:39:27 -08:00
Jay Zhuang
87983d442c Skip format checks for sandcastle build (#7911)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7911

Test Plan:
Manually tested most of the builds:
* asan: https://www.internalfb.com/intern/sandcastle/job/9007199627266604
* asan_crash: https://www.internalfb.com/intern/sandcastle/group/nonce/827960152876428/
* asan_crash_with_atomic_flush: https://www.internalfb.com/intern/sandcastle/group/nonce/6901727142256449/
* asan_crash_with_txn https://www.internalfb.com/intern/sandcastle/group/nonce/5745312080630484/
* blackbox_asan_crash https://www.internalfb.com/intern/sandcastle/group/nonce/939185737271884/
* clang_analyze https://www.internalfb.com/intern/sandcastle/job/31525197767943703
* code_cov https://www.internalfb.com/intern/sandcastle/job/376027170
* format_compatible https://www.internalfb.com/intern/sandcastle/group/nonce/6238410205251033/
* no_compression https://www.internalfb.com/intern/sandcastle/group/nonce/5901170225087670/
* regression https://www.internalfb.com/intern/sandcastle/group/nonce/6050737823326726/
* release https://www.internalfb.com/intern/sandcastle/group/nonce/528162753321938/
* stress_crash_with_txn https://www.internalfb.com/intern/sandcastle/group/nonce/4473519221616524/
* tsan_crash_with_atomic_flush https://www.internalfb.com/intern/sandcastle/group/nonce/6950713723315225/
* ubsan_crash https://www.internalfb.com/intern/sandcastle/group/nonce/8824481248569096/
* unit_non_shm https://www.internalfb.com/intern/sandcastle/group/nonce/3373704090128605/
* unity  https://www.internalfb.com/intern/sandcastle/group/nonce/832911990656833/
* whitebox_stress_crash https://www.internalfb.com/intern/sandcastle/group/nonce/3698026167778822/

Reviewed By: pdillinger

Differential Revision: D26165859

Pulled By: jay-zhuang

fbshipit-source-id: d1e7dfd024b82d67b1c291f87216d7fdfc01856a
2021-02-01 09:17:54 -08:00
Jay Zhuang
8944a3bf27 Add support for specifying working directory in lego-determinator (#7901)
Summary:
So we could change the repo or repo structure.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7901

Test Plan:
works with existing job: https://www.internalfb.com/intern/sandcastle/job/13510799254899745/
works with internal repo: https://www.internalfb.com/intern/sandcastle/job/27021598137143731

Reviewed By: pdillinger

Differential Revision: D26108239

Pulled By: jay-zhuang

fbshipit-source-id: ac24559653d9abc012068e8accd7606505bd5b88
2021-01-27 15:24:11 -08:00
Peter Dillinger
4d897e51df Migrate away from Travis+Linux+amd64 (#7791)
Summary:
This disables Linux/amd64 builds in Travis for PRs, and adds a
gcc-10+c++20 build in CircleCI, which should fill out sufficient coverage
vs. what we had in Travis

Fixed a use of std::is_pod, which is deprecated in c++20

Fixed ++ on a volatile in db_repl_stress.cc, with bigger refactoring.
Although ++ on this volatile was probably ok with one thread writer and
one thread reader, the code was still overly complex. There was a
deadcode check for error
`if (replThread.no_read < dataPump.no_records)` which can be proven
never to happen based on the structure of the code. It infinite loops
instead for the case intended to be checked. I just simplified the code
for what should be the same checking power.

Also most configurations seem to be using make parallelism = 2 * vcores,
so fixing / using that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7791

Test Plan:
CI
and `while ./db_repl_stress; do echo again; done` for a while

Reviewed By: siying

Differential Revision: D25669834

Pulled By: pdillinger

fbshipit-source-id: b2c688053d0b1d52c989903449d3cd27a04130d6
2020-12-22 00:20:57 -08:00
Adam Retter
303d283420 RocksJava static lib dependencies should support MacOS 10.12+ (#7683)
Summary:
Expands on https://github.com/facebook/rocksdb/pull/7016 so that when `PORTABLE=1` is set the dependencies for RocksJava static target will also be built with backwards compatibility for MacOS as far back as 10.12 (i.e. 2016).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7683

Reviewed By: ajkr

Differential Revision: D25034164

Pulled By: pdillinger

fbshipit-source-id: dc9e51828869ed9ec336a8a86683e4d0bfe04f27
2020-11-17 15:34:05 -08:00
Xie Yanbo
c9c9709a1a Update clang-format-diff.py (#7609)
Summary:
`llvm-mirror/clang` is archived. Get the `clang-format-diff.py` file from the active source.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7609

Reviewed By: ajkr

Differential Revision: D24711608

Pulled By: pdillinger

fbshipit-source-id: b115d8765ff23fbb8190290a170de21565daba84
2020-11-04 16:09:01 -08:00
Peter Dillinger
0e2e67562f Give instructions instead of broken 2to3 for clang-format-diff.py (#7603)
Summary:
My previous change to use lib2to3 to migrate clang-format-diff.py
for Python 2 only works if there's nothing to reformat. Instead, give
instructions to download to REPO_ROOT.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7603

Test Plan: Try the instructions on a fresh CentOS 8 devserver

Reviewed By: riversand963

Differential Revision: D24569608

Pulled By: pdillinger

fbshipit-source-id: 1410ba163e016b226e883dec93fae3df9ed0eab2
2020-10-27 11:58:17 -07:00
Peter Dillinger
c4d8838a2b New bit manipulation functions and 128-bit value library (#7338)
Summary:
These new functions and 128-bit value bit operations are
expected to be used in a forthcoming Bloom filter alternative.

No functional changes to production code, just new code only called by
unit tests, cosmetic changes to existing headers, and fix an existing
function for a yet-unused template instantiation (BitsSetToOne on
something signed and smaller than 32 bits).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7338

Test Plan:
Unit tests included. Works with and without
TEST_UINT128_COMPAT=1 to check compatibility with and without
__uint128_t. Also added that parameter to the CircleCI build
build-linux-shared_lib-alt_namespace-status_checked.

Reviewed By: jay-zhuang

Differential Revision: D23494945

Pulled By: pdillinger

fbshipit-source-id: 5c0dc419100d9df5d4d9abb153b2855d5aea39e8
2020-09-03 09:32:59 -07:00