Commit Graph

744 Commits

Author SHA1 Message Date
Andrew Kryczka
8115eb520d add Status check assertions for repair_test (#7455)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7455

Reviewed By: riversand963

Differential Revision: D23985283

Pulled By: ajkr

fbshipit-source-id: 5dd2be62350f6e31d13a1e7821cb848a37699c93
2020-09-29 16:30:08 -07:00
sdong
d08a9005b7 Make db_basic_test pass assert status checked (#7452)
Summary:
Add db_basic_test status check list. Some of the warnings are suppressed. It is possible that some of them are due to real bugs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7452

Test Plan: See CI tests pass.

Reviewed By: zhichao-cao

Differential Revision: D23979764

fbshipit-source-id: 6151570c2a9b931b0fbb3fe939a94b2bd1583cbe
2020-09-29 09:49:04 -07:00
Andrew Kryczka
7a23a2175b enable Status check assertions for sst_file_reader_test (#7448)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7448

Reviewed By: jay-zhuang

Differential Revision: D23973281

Pulled By: ajkr

fbshipit-source-id: 90fe55f1db904f1a236f3f7994d0806b8d24282c
2020-09-28 15:13:15 -07:00
Zhichao Cao
d71cfe04e4 Add flush_job_test to the list of ASSERT_STATUS_CHECKED tests (#7445)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7445

Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 flush_job_test

Reviewed By: akankshamahajan15

Differential Revision: D23969372

Pulled By: zhichao-cao

fbshipit-source-id: 498ff45ef84e07ec27a8f35d0874d3371412afe9
2020-09-28 14:59:02 -07:00
Levi Tamasi
1abbc56aba Add version_builder_test to the list of ASSERT_STATUS_CHECKED tests (#7444)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7444

Test Plan: `ASSERT_STATUS_CHECKED=1 make version_builder_test -j24`

Reviewed By: jay-zhuang

Differential Revision: D23965793

Pulled By: ltamasi

fbshipit-source-id: 8beaf66548379f21146189cda699d5f6fbb35a1b
2020-09-28 12:12:40 -07:00
Akanksha Mahajan
9a63bbd391 Add few unit test cases in ASSERT_STATUS_CHECKED build (#7427)
Summary:
Fix few test cases and add them in ASSERT_STATUS_CHECKED build.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7427

Test Plan:
1.  ASSERT_STATUS_CHECKED=1 make -j48 check,
                 2. travis build for ASSERT_STATUS_CHECKED,
                 3. Without ASSERT_STATUS_CHECKED:  make check -j64, CircleCI build and travis build

Reviewed By: pdillinger

Differential Revision: D23909983

Pulled By: akankshamahajan15

fbshipit-source-id: 42d7e4aea972acb9fcddb7ca73fcb82f93272434
2020-09-24 21:48:57 -07:00
Akanksha Mahajan
98ac6b646a Add IO Tracer Parser (#7333)
Summary:
Implement a parsing tool io_tracer_parser that takes IO trace file (binary file) with command line argument --io_trace_file and output file with --output_file and dumps the IO trace records in outputfile in human readable form.

Also added unit test cases that generates IO trace records and calls io_tracer_parse to parse those records.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7333

Test Plan:
make check -j64,
 Add unit test cases.

Reviewed By: anand1976

Differential Revision: D23772360

Pulled By: akankshamahajan15

fbshipit-source-id: 9c20519c189362e6663352d08863326f3e496271
2020-09-23 15:50:26 -07:00
Peter Dillinger
31d1cea4f3 Add and fix clang -Wshift-sign-overflow (#7431)
Summary:
This option is apparently used by some teams within Facebook
(internal ref T75998621)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7431

Test Plan: USE_CLANG=1 make check before (fails) and after

Reviewed By: jay-zhuang

Differential Revision: D23876584

Pulled By: pdillinger

fbshipit-source-id: abb8b67a1f1aac75327944d266e284b2b6727191
2020-09-23 15:26:17 -07:00
Jay Zhuang
6980fff445 CircleCI build improvements (#7392)
Summary:
* add slack integration
* parallelize `check_some` tests with gtest-parallel
* upgrade build image to the latest one: ubuntu-1604:202007-01
* clean up the config

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7392

Reviewed By: pdillinger

Differential Revision: D23845379

Pulled By: jay-zhuang

fbshipit-source-id: f3af82dc2daeba6441f61c858e08d3936c6ccd10
2020-09-23 14:42:16 -07:00
mrambacher
a08d6f18f0 Add more tests to ASSERT_STATUS_CHECKED (#7367)
Summary:
db_options_test
options_file_test
auto_roll_logger_test
options_util_test
persistent_cache_test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7367

Reviewed By: jay-zhuang

Differential Revision: D23712520

Pulled By: zhichao-cao

fbshipit-source-id: 99b331e357f5d6a6aabee89d1bd933002cbb3908
2020-09-16 15:48:07 -07:00
Andrew Kryczka
d4993b9b60 Makefile support subset/individual valgrind tests (#7379)
Summary:
Introduced `valgrind_check_some`, which is analogous to the `check_some` target for non-valgrind tests. It simplifies the process for running a single valgrind test or subset of valgrind tests when trying to repro a failure.

I also added a `ROCKSDBTESTS_ONLY` parameter, which simplifies selecting a single test to run. Previously the user would have to use `ROCKSDBTESTS_START` and `ROCKSDBTESTS_END`, but it was difficult to determine the end variable since it is an exclusive endpoint and must match an actual test name.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7379

Reviewed By: pdillinger

Differential Revision: D23673608

Pulled By: ajkr

fbshipit-source-id: 87ed81f1a671d46c2dff6a701f85f1891c725b3f
2020-09-14 19:46:46 -07:00
mrambacher
7d472accdc Bring the Configurable options together (#5753)
Summary:
This PR merges the functionality of making the ColumnFamilyOptions, TableFactory, and DBOptions into Configurable into a single PR, resolving any merge conflicts

Pull Request resolved: https://github.com/facebook/rocksdb/pull/5753

Reviewed By: ajkr

Differential Revision: D23385030

Pulled By: zhichao-cao

fbshipit-source-id: 8b977a7731556230b9b8c5a081b98e49ee4f160a
2020-09-14 17:01:01 -07:00
Peter Dillinger
a7fde8727b Fix platform_dependent in Travis, rebalance groups (#7360)
Summary:
Was broken by https://github.com/facebook/rocksdb/issues/6660

Travis times before this change, after 6660:
platform_dependent: 17 min
group 1: 15 min
group 2: 44 min (often timeout on non-x86 or non-Linux)
group 3: 31 min
group 4: 21 min

After this change:
TODO

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7360

Test Plan: CI inspection

Reviewed By: ajkr

Differential Revision: D23586917

Pulled By: pdillinger

fbshipit-source-id: 4c67cf33180b0b833c39a817e6c1f128727941d2
2020-09-09 09:49:24 -07:00
Peter Dillinger
9de912de3f Fix some errors showing up in Travis builds (#7359)
Summary:
Also enables a pull request to trigger all the Travis
configurations by writing FULL_CI in the commit message. (See what I did
there?)

First issue

    make: *** No rule to make target 'jl/util/crc32c_ppc_asm.o', needed by 'rocksdbjava'.  Stop.

Second issue

    tools/db_bench_tool.cc:5514:38: error: ‘gen_exp.rocksdb::Benchmark::GenerateTwoTermExpKeys::keyrange_size_’ may be used uninitialized in this function

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7359

Test Plan: CI

Reviewed By: zhichao-cao

Differential Revision: D23582132

Pulled By: pdillinger

fbshipit-source-id: 06d794673fd522ba11cf6398385387e6bd97ef89
2020-09-08 15:11:47 -07:00
Jay Zhuang
27aa443a15 Add sst_file_dumper status check (#7315)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7315

Test Plan:
`ASSERT_STATUS_CHECKED=1 make sst_dump_test && ./sst_dump_test`
And manually run `./sst_dump --file=*.sst` before and after the change.

Reviewed By: pdillinger

Differential Revision: D23361669

Pulled By: jay-zhuang

fbshipit-source-id: 5bf51a2a90ee35c8c679e5f604732ec2aef5949a
2020-09-04 19:26:42 -07:00
Peter Dillinger
c4d8838a2b New bit manipulation functions and 128-bit value library (#7338)
Summary:
These new functions and 128-bit value bit operations are
expected to be used in a forthcoming Bloom filter alternative.

No functional changes to production code, just new code only called by
unit tests, cosmetic changes to existing headers, and fix an existing
function for a yet-unused template instantiation (BitsSetToOne on
something signed and smaller than 32 bits).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7338

Test Plan:
Unit tests included. Works with and without
TEST_UINT128_COMPAT=1 to check compatibility with and without
__uint128_t. Also added that parameter to the CircleCI build
build-linux-shared_lib-alt_namespace-status_checked.

Reviewed By: jay-zhuang

Differential Revision: D23494945

Pulled By: pdillinger

fbshipit-source-id: 5c0dc419100d9df5d4d9abb153b2855d5aea39e8
2020-09-03 09:32:59 -07:00
Jonny Heggheim
1e59800718 Use standard variables for installing/uninstalling with make (#7187)
Summary:
Fixes https://github.com/facebook/rocksdb/issues/7185.

Standard for GNU and FreeBSD. See
https://www.freebsd.org/doc/en/books/porters-handbook/porting-prefix.html
https://www.gnu.org/prep/standards/html_node/DESTDIR.html#DESTDIR
https://www.gnu.org/prep/standards/html_node/Directory-Variables.html

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7187

Reviewed By: cheng-chang

Differential Revision: D23333233

Pulled By: ajkr

fbshipit-source-id: f704d23852c4516cf5fa00df73ff57687b2ddffb
2020-08-28 14:47:31 -07:00
Jay Zhuang
c2485f2d81 Add buffer prefetch support for non directIO usecase (#7312)
Summary:
A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312

Reviewed By: anand1976

Differential Revision: D23329847

Pulled By: jay-zhuang

fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
2020-08-27 18:16:53 -07:00
Levi Tamasi
5043960623 Add a blob file builder class that can be used in background jobs (#7306)
Summary:
The patch adds a class called `BlobFileBuilder` that can be used to build
and cut blob files in background jobs (flushes/compactions). The class
enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined
in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`),
as well as compression (`blob_compression_type`) and checksums for blob files.
It also keeps track of the generated blob files and their associated `BlobFileAddition`
metadata, which can be applied as part of the background job's `VersionEdit`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D23298817

Pulled By: ltamasi

fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
2020-08-27 11:55:54 -07:00
mrambacher
b7e1c5213f Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305)
Summary:
More tests now pass.  When in doubt, I added a TODO comment to check what should happen with an ignored error.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7305

Reviewed By: akankshamahajan15

Differential Revision: D23301262

Pulled By: ajkr

fbshipit-source-id: 5f120edc7393560aefc0633250277bbc7e8de9e6
2020-08-24 16:43:31 -07:00
John Goerzen
59ebab654e Generate and install a pkg-config file (#7244)
Summary:
pkg-config files are quite useful for communicating to users of a
library how to compile against them. This commit generates and installs
a pkg-config file that can be used for both static and dynamic builds
against the RocksDB library. This should make life easier for developers
of client programs, language bindings, etc.

Example usage:

```
g++ `pkg-config --cflags rocksdb` -o simple_example simple_example.cc `pkg-config --libs rocksdb`

g++ `pkg-config --cflags --static rocksdb` -static \
   -o simple_example simple_example.cc `pkg-config --libs --static rocksdb`
```

The commit also adds the generated file to .gitignore, to the uninstall
target, and to clean.

No additional dependencies are added to RocksDB itself, and this does
not make RocksDB use pkg-config as part of its build process.

Resolves https://github.com/facebook/rocksdb/issues/4452

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7244

Reviewed By: siying

Differential Revision: D23146153

Pulled By: ajkr

fbshipit-source-id: 3045aa650d68bd5ac42d40ed709570e9584ef004
2020-08-17 11:53:11 -07:00
Jay Zhuang
69760b4d05 Introduce a global StatsDumpScheduler for stats dumping (#7223)
Summary:
Have a global StatsDumpScheduler for all DB instance stats dumping, including `DumpStats()` and `PersistStats()`. Before this, there're 2 dedicate threads for every DB instance, one for DumpStats() one for PersistStats(), which could create lots of threads if there're hundreds DB instances.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7223

Reviewed By: riversand963

Differential Revision: D23056737

Pulled By: jay-zhuang

fbshipit-source-id: 0faa2311142a73433ebb3317361db7cbf43faeba
2020-08-14 20:12:44 -07:00
Yanqin Jin
e9daedec84 Add CircleCI for tests on non-shm (#7229)
Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7229

Test Plan: Watch for CircleCI results.

Reviewed By: siying

Differential Revision: D23023943

Pulled By: riversand963

fbshipit-source-id: 41a989a3ffdfd2decd309185e3b963e810419577
2020-08-11 18:30:47 -07:00
Andrew Kryczka
6e99de6d3c Allow optimization level to be set in Makefile (#7202)
Summary:
`-O3` is already adopted widely, so we should make it easier to configure
for development/open source. This PR adds an `OPTIMIZE_LEVEL` variable
that users can set to override the `-O` flag chosen in the Makefile.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7202

Test Plan: built a few different ways and verified correct value is passed for `-O` flag

Reviewed By: pdillinger

Differential Revision: D22845291

Pulled By: ajkr

fbshipit-source-id: 84471362e7d627dd606b25bf5f6a3d796817fa1c
2020-08-10 11:25:20 -07:00
mrambacher
56f468b356 Add more tests to ASSERT_STATUS_CHECKED (#7211)
Summary:
Added 4 more tests to those which pass ASSERT_STATUS_CHECKED (cache_test, lru_cache_test, filename_test, filelock_test).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7211

Reviewed By: ajkr

Differential Revision: D22982858

Pulled By: zhichao-cao

fbshipit-source-id: acdd071582ed6aa7447ed96c5732f10bf720d783
2020-08-06 17:19:41 -07:00
Adam Retter
3356187617 Automatically number the Maven artifacts (#7219)
Summary:
Improvements to the RocksJava release process:
* Generates the Maven artifact version number as part of the release step
* Also generates appropriate checksum files to speed the deploy and publish step

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7219

Reviewed By: ltamasi

Differential Revision: D22983481

Pulled By: pdillinger

fbshipit-source-id: 7b8ffaf46471cd3cda181eb830c962b317d2e688
2020-08-06 14:13:30 -07:00
Cheng Chang
cd48ecaa1a Define WAL related classes to be used in VersionEdit and VersionSet (#7164)
Summary:
`WalAddition`, `WalDeletion` are defined in `wal_version.h` and used in `VersionEdit`.
`WalAddition` is used to represent events of creating a new WAL (no size, just log number), or closing a WAL (with size).
`WalDeletion` is used to represent events of deleting or archiving a WAL, it means the WAL is no longer alive (won't be replayed during recovery).

`WalSet` is the set of alive WALs kept in `VersionSet`.

1. Why use `WalDeletion` instead of relying on `MinLogNumber` to identify outdated WALs

On recovery, we can compute `MinLogNumber()` based on the log numbers kept in MANIFEST, any log with number < MinLogNumber can be ignored. So it seems that we don't need to persist `WalDeletion` to MANIFEST, since we can ignore the WALs based on MinLogNumber.

But the `MinLogNumber()` is actually a lower bound, it does not exactly mean that logs starting from MinLogNumber must exist. This is because in a corner case, when a column family is empty and never flushed, its log number is set to the largest log number, but not persisted in MANIFEST. So let's say there are 2 column families, when creating the DB, the first WAL has log number 1, so it's persisted to MANIFEST for both column families. Then CF 0 is empty and never flushed, CF 1 is updated and flushed, so a new WAL with log number 2 is created and persisted to MANIFEST for CF 1. But CF 0's log number in MANIFEST is still 1. So on recovery, MinLogNumber is 1, but since log 1 only contains data for CF 1, and CF 1 is flushed, log 1 might have already been deleted from disk.

We can make `MinLogNumber()` be the exactly minimum log number that must exist, by persisting the most recent log number for empty column families that are not flushed. But if there are N such column families, then every time a new WAL is created, we need to add N records to MANIFEST.

In current design, a record is persisted to MANIFEST only when WAL is created, closed, or deleted/archived, so the number of WAL related records are bounded to 3x number of WALs.

2. Why keep `WalSet` in `VersionSet` instead of applying the `VersionEdit`s to `VersionStorageInfo`

`VersionEdit`s are originally designed to track the addition and deletion of SST files. The SST files are related to column families, each column family has a list of `Version`s, and each `Version` keeps the set of active SST files in `VersionStorageInfo`.

But WALs are a concept of DB, they are not bounded to specific column families. So logically it does not make sense to store WALs in a column family's `Version`s.
Also, `Version`'s purpose is to keep reference to SST / blob files, so that they are not deleted until there is no version referencing them. But a WAL is deleted regardless of version references.
So we keep the WALs in `VersionSet`  for the purpose of writing out the DB state's snapshot when creating new MANIFESTs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7164

Test Plan:
make version_edit_test && ./version_edit_test
make wal_edit_test && ./wal_edit_test

Reviewed By: ltamasi

Differential Revision: D22677936

Pulled By: cheng-chang

fbshipit-source-id: 5a3b6890140e572ffd79eb37e6e4c3c32361a859
2020-08-05 16:34:38 -07:00
mrambacher
d9d190742c Make env*_test work with ASSERT_STATUS_CHECKED (#7176)
Summary:
Make (most of) the env*_test pass when ASSERT_STATUS_CHECKED is enabled.

One test that opens a database is currently disabled in this mode, as there are many errors that need revisited for DB tests and status checks.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7176

Reviewed By: cheng-chang

Differential Revision: D22799278

Pulled By: ajkr

fbshipit-source-id: 16d8a02eaeecd6df1060249b6a5811292801f2ed
2020-07-28 22:59:48 -07:00
Andrew Kryczka
c0c33a4854 Makefile support for link-time optimization (#7181)
Summary:
`USE_LTO=1` in `make` commands now enables LTO. The archiver (`ar`) needed
to change in this PR to use a wrapper that enables the LTO plugin.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7181

Test Plan:
build a few ways
```
$ make clean && USE_LTO=1 make -j48 db_bench
$ make clean && USE_CLANG=1 USE_LTO=1 make -j48 db_bench
$ make clean && ROCKSDB_NO_FBCODE=1 USE_LTO=1 make -j48 db_bench
```

Reviewed By: cheng-chang

Differential Revision: D22784994

Pulled By: ajkr

fbshipit-source-id: 9c45333bd49bf4615aa04c85b7c6fd3925421152
2020-07-28 13:10:44 -07:00
Yanqin Jin
954ee56571 Add job specs for blackbox/whitebox stress tests (#7170)
Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7170

Test Plan: Manually invoke the commands.

Reviewed By: siying

Differential Revision: D22732256

Pulled By: riversand963

fbshipit-source-id: d331e5ee84658ac079814292ff1a1eacfd14bfdf
2020-07-24 13:42:53 -07:00
sdong
0f487cc35f Exclude two tests in CircleCI TSAN tests (#7152)
Summary:
Two TSAN tests occaionaly fail. Exclude them for now:

[ RUN      ] DeleteFileTest.BackgroundPurgeCFDropTest
db/deletefile_test.cc:122: Failure
Expected equality of these values:
  required_manifest
    Which is: 1
  manifest_cnt
    Which is: 2

[ RUN      ] FormatLatest/ColumnFamilyTest.FlushCloseWALFiles/0
db/column_family_test.cc:3004: Failure
Expected equality of these values:
  2
  env.num_open_wal_file_.load()
    Which is: 1

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7152

Test Plan: Watch CircleCI restuls

Reviewed By: ajkr

Differential Revision: D22632285

fbshipit-source-id: 29fa348e8be917be0237c74812a8b0b04978e84e
2020-07-20 15:01:17 -07:00
Jay Zhuang
77062cf13e Store the test results to CircleCI (#7137)
Summary:
To have test report.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7137

Reviewed By: siying

Differential Revision: D22630798

Pulled By: jay-zhuang

fbshipit-source-id: bc07ba673c0bceed5a4829b4af2d9a74435379c7
2020-07-20 11:16:19 -07:00
sdong
1cc9b0eb02 Fix parallel test sometimes doesn't fail with failed tests. (#7147)
Summary:
In CircleCI tests, we failed to fail tests properly if parallel doesn't return an error code. It's probably would happen when unit tests fail with signals, rather than return values. Fix them.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7147

Test Plan: Manually ingest a failure and see it to fail.

Reviewed By: jay-zhuang

Differential Revision: D22611594

fbshipit-source-id: 88a42425a41d1213d29bd2e7c80731d2bdd5644b
2020-07-17 18:07:08 -07:00
Adam Retter
de8c92a596 Only check for python location once (#7123)
Summary:
This fixes an issue introduced in 0c56fc4 whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6.

The location is now only checked once.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123

Reviewed By: zhichao-cao

Differential Revision: D22532274

Pulled By: ajkr

fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970
2020-07-14 16:01:21 -07:00
sdong
e930ba1d18 Run a subset of TSAN tests in CIrcleCI (#7122)
Summary:
It is helpful to add some TSAN coverage before a pull request is committed. This diff adds some of them.
Some slow tests are excluded for the running speed. Some are blacklisted because they show warnings. Will investigate these warnings and see whether we can fix or suppress them.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7122

Test Plan: Watch CIrcleCI runs

Reviewed By: riversand963

Differential Revision: D22532133

fbshipit-source-id: 81ddd02d9df19c513a12811979e8ddabae911354
2020-07-14 15:33:10 -07:00
mrambacher
c7c7b07f06 More Makefile Cleanup (#7097)
Summary:
Cleans up some of the dependencies on test code in the Makefile while building tools:
- Moves the test::RandomString, DBBaseTest::RandomString into Random
- Moves the test::RandomHumanReadableString into Random
- Moves the DestroyDir method into file_utils
- Moves the SetupSyncPointsToMockDirectIO into sync_point.
- Moves the FaultInjection Env and FS classes under env

These changes allow all of the tools to build without dependencies on test_util, thereby simplifying the build dependencies.  By moving the FaultInjection code, the dependency in db_stress on different libraries for debug vs release was eliminated.

Tested both release and debug builds via Make and CMake for both static and shared libraries.

More work remains to clean up how the tools are built and remove some unnecessary dependencies.  There is also more work that should be done to get the Makefile and CMake to align in their builds -- what is in the libraries and the sizes of the executables are different.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7097

Reviewed By: riversand963

Differential Revision: D22463160

Pulled By: pdillinger

fbshipit-source-id: e19462b53324ab3f0b7c72459dbc73165cc382b2
2020-07-09 14:35:17 -07:00
Peter Dillinger
4202c48f80 Replace large 'rm' with 'find' (#7095)
Summary:
On some platforms like MacOS, a second 'make check' can lead to
/bin/rm: Argument list too long

This is fixed by replacing with a 'find'. Also, using '-f' for more rm calls
to avoid prompt.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7095

Test Plan: 'make check' on Linux and MacOS

Reviewed By: riversand963

Differential Revision: D22415808

Pulled By: pdillinger

fbshipit-source-id: 0fd1ebae13739c9d81f9e813e99b062715604d6b
2020-07-07 16:48:52 -07:00
Peter Dillinger
787bf79fa0 Fix build of db_stress with LIB_MODE=shared (#7098)
Summary:
by tracking and linking against runtime dependent libraries in
Makefile

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7098

Test Plan: look for fix in CircleCI

Reviewed By: riversand963

Differential Revision: D22420860

Pulled By: pdillinger

fbshipit-source-id: d211d709214bf5306db68e43b7a2f18169281022
2020-07-07 16:48:52 -07:00
Peter Dillinger
92731b6b4a Major CircleCI/Linux fixes / tweaks / enhancements (#7078)
Summary:
Primarily, this change adds a way to work around a bug limiting the effective output (and therefore debugability) of the Linux builds using parallel make. We would get
make[1]: write error: stdout
probably due to a kernel bug, apparently affecting both available ubuntu 16 machine images (maybe not affecting docker images, less horsepower). https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393

Now in the CircleCI config, make output on Ubuntu is piped through a custom 'cat' that ignores EAGAIN errors, which seems to fix the problem.

Significant other changes:
* Add another linux build that combines
  * LIB_MODE=shared, to ensure this works with compile and unit test execution
  * Alternative rocksdb namespace, to ensure this works (not rely on Travis)
  * ASSERT_STATUS_CHECKED=1, but with building all unit tests and running those expected to pass with it
* Run release build with and without gflags. (Was running only without, ignore large swaths of code in a normal release build! Two regressions in this build, only with gflags, in the last week not caught by CI!)
* Use gflags with unity and LITE build, as typical case.

Debugability improvements:
* Use V=1 to show commands being executed (thanks to EAGAIN work-around)
* Print kernel version and compiler versions as part of V=1 output from Makefile

Cosmetic other changes:
* Put more commands on one line, for less clutter in CircleCI output pages
* Remove redundant "all" in "make all check" and put make command options before targets
* Change some recursive "make clean" into dependency on "clean," toward minimizing unnecessary overhead (detect platform, build version, etc.) of extra recursive makes

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7078

Reviewed By: siying

Differential Revision: D22391647

Pulled By: pdillinger

fbshipit-source-id: d446fccf5a8c568b37dc8748621c8a5c546fe135
2020-07-07 11:25:46 -07:00
Peter Dillinger
bd77e34191 More Makefile clean-up (#7066)
Summary:
(a) use STRESS_LIBRARY for db_stress and make sure
STRESS_LIBRARY has other stress test dependencies (as in buck build)

(b) fix rpath option to be accepted on MacOS. It still doesn't fully work
for me e.g. to run a LIB_MODE=shared unit test binary from another
directory, as it does on Linux, but the option is now accepted, and running
unit tests from current directory works for me.

Also adding LIB_MODE=shared to Travis. (Later TBD where best to fit in
in CircleCI.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7066

Test Plan: manual

Reviewed By: cheng-chang

Differential Revision: D22364068

Pulled By: pdillinger

fbshipit-source-id: 6fa98a222f89f808ee786474de1100d92c1adec3
2020-07-06 11:19:48 -07:00
Peter Dillinger
8e6ff044e1 Fix release build and fbcode+clang+shared (#7062)
Summary:
Follow-up to https://github.com/facebook/rocksdb/issues/6660. Release build had linker error. fbcode+clang+shared build was erroring on unused parameter '-nostdinc'.

Fixes https://github.com/facebook/rocksdb/issues/7061

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7062

Test Plan: make release, USE_CLANG=1 LIB_MODE=shared make check, etc.

Reviewed By: siying

Differential Revision: D22335663

Pulled By: pdillinger

fbshipit-source-id: 261cd959ca1f6c273dc763a70020a535ba8e81de
2020-07-01 10:30:55 -07:00
mrambacher
80f71b5863 Use Libraries in the RocksDB Makefile Build (#6660)
Summary:
Change the linking of tests/tools to be against a library rather than a list of objects.  This change substantially reduces the size of the objects produced.

peterd clean repo size: 264M
Before this change, with make all: 40G
After this change, with make all: 28G
With make LIB_MODE=shared all: 7.0G

The list of TESTS was changed from being hard-coded to generated from the test sources variable.  Note that there are some test sources that are not built as tests (though the set of tests is identical to the previous version).

Added OBJ_DIR option to Makefile to allow objects to be placed in an alternative location.  By default, OBJ_DIR is the same as before ("./").

This change is a precursor to being able to build/run the tests/tools linked against static libraries.  Additionally, it should be possible to clean up and merge some of the rules for building tests and the like if so desired.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6660

Reviewed By: riversand963

Differential Revision: D22244463

Pulled By: pdillinger

fbshipit-source-id: db9c6341d81ed62c2270374f4ede02fb9604c754
2020-06-30 19:33:31 -07:00
sdong
f9817201af Add unity build to CircleCI (#7026)
Summary:
We are still keeping unity build working. So it's a good idea to add to a pre-commit CI.
A latest GCC docker image just to get a little bit more coverage. Fix three small issues to make it pass.
Also make unity_test to run db_basic_test rather than db_test to cut the test time. There is no point to run expensive tests here. It was set to run db_test before db_basic_test was separated out.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7026

Test Plan: watch tests to pass.

Reviewed By: zhichao-cao

Differential Revision: D22223197

fbshipit-source-id: baa3b6cbb623bf359829b63ce35715c75bcb0ed4
2020-06-26 11:14:08 -07:00
Peter Dillinger
88b4210701 Remove racially charged terms "whitelist" and "blacklist" (#7008)
Summary:
We don't need them.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7008

Test Plan: "make check" and ensure "make crash_test" starts

Reviewed By: ajkr

Differential Revision: D22143838

Pulled By: pdillinger

fbshipit-source-id: 72c8e16603abc59f4954e304466bc4dc1f58f94e
2020-06-19 15:27:32 -07:00
Akanksha Mahajan
552fd765b3 Add IOTracer reader, writer classes for reading/writing IO operations in a binary file (#6958)
Summary:
1. As part of IOTracing project, Add a class IOTracer,
IOTraceReader and IOTracerWriter that writes the file operations
information in a binary file. IOTrace Record contains record information
and right now it contains access_timestamp, file_operation, file_name,
io_status, len, offset and later other options will be added when file
system APIs will be call IOTracer.

2. Add few unit test cases that verify that reading and writing to a IO
Trace file is working properly and before start trace and after ending
trace nothing is added to the binary file.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6958

Test Plan:
1. make check -j64
                 2. New testcases for IOTracer.

Reviewed By: anand1976

Differential Revision: D21943375

Pulled By: akankshamahajan15

fbshipit-source-id: 3532204e2a3eab0104bf411ab142e3fdd4fbce54
2020-06-18 10:46:11 -07:00
Yanqin Jin
15d9f28da5 Add stress test for best-efforts recovery (#6819)
Summary:
Add crash test for the case of best-efforts recovery.
After a certain amount of time, we kill the db_stress process, randomly delete some certain table files and restart db_stress. Given the randomness of file deletion, it is difficult to verify against a reference for data correctness. Therefore, we just check that the db can restart successfully.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6819

Test Plan:
```
./db_stress -best_efforts_recovery=true -disable_wal=1 -reopen=0
./db_stress -best_efforts_recovery=true -disable_wal=0 -skip_verifydb=1 -verify_db_one_in=0 -continuous_verification_interval=0
make crash_test_with_best_efforts_recovery
```

Reviewed By: anand1976

Differential Revision: D21436753

Pulled By: riversand963

fbshipit-source-id: 0b3605c922a16c37ed17d5ab6682ca4240e47926
2020-06-12 19:27:46 -07:00
sdong
6a8ddd374d Introduce some Linux build to CircleCI (#6937)
Summary:
Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ
ed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937

Test Plan: Watch CI build results.

Reviewed By: pdillinger

Differential Revision: D21941605

fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad
2020-06-08 19:34:31 -07:00
Cheng Chang
1bee0fca05 Make DestroyDir destroy directories recursively (#6934)
Summary:
Currently, `DeleteDir` only deletes the directory if there are no other directories under the target dir. This PR makes it delete directories recursively.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6934

Test Plan:
Added a new unit test in testutil_test.cc.
`make testutil_test`

Reviewed By: zhichao-cao

Differential Revision: D21884211

Pulled By: cheng-chang

fbshipit-source-id: 0b9a48a200f494ee007aef5d1763b4aa331f8b5a
2020-06-05 10:55:22 -07:00
sdong
afa3518839 Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923)
Summary:
This reverts commit 8d87e9cea1.

Based on offline discussions, it's too early to upgrade to gtest 1.10, as it prevents some developers from using an older version of gtest to integrate to some other systems. Revert it for now.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6923

Reviewed By: pdillinger

Differential Revision: D21864799

fbshipit-source-id: d0726b1ff649fc911b9378f1763316200bd363fc
2020-06-03 15:55:03 -07:00
Peter Dillinger
43f8a9dcce Some fixes for gcc 4.8 and add to Travis (#6915)
Summary:
People keep breaking the gcc 4.8 compilation due to different
warnings for shadowing member functions with locals. Adding to Travis
to keep compatibility. (gcc 4.8 is default on CentOS 7.)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915

Test Plan: local and Travis

Reviewed By: siying

Differential Revision: D21842894

Pulled By: pdillinger

fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4
2020-06-03 11:39:25 -07:00