Commit Graph

282 Commits

Author SHA1 Message Date
sdong
7efd9c25c4 Increse timeout in some tests
Summary: Increase test timeout to some tests to unblock CI.

Test Plan: watch how it runs.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61263
2016-07-27 11:39:36 -07:00
woonhak.kang
5c858ddd20 fix errata in libnuma test (#1244) 2016-07-26 19:28:45 -07:00
krad
a4955b39ac Run sandcastle tests in /dev/shm 2016-07-25 15:10:24 -07:00
sdong
f85df120f2 Re-enable tsan crash white-box test with reduced killing odds
Summary:
Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds.
Add a parameter in crash test script to allow we pass it through an environment variable.

Test Plan: Run it manually.

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61053
2016-07-22 12:51:27 -07:00
sdong
b50632920d Add unit test not on /dev/shm as part of the pre-commit tests
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system.

Test Plan: See the tests launched by Phabricator

Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60963
2016-07-21 12:53:23 -07:00
sdong
f9b14be495 Re-enable TSAN crash test but only with black box crash test
Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job.

Test Plan: Run it locally.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60951
2016-07-20 13:52:25 -07:00
sdong
68f3eb746c Run release build for CLANG and GCC 4.8.1 in pre-commit tests too
Summary: As Title.

Test Plan: See how the diff works.

Reviewers: kradhakrishnan, andrewkr, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60933
2016-07-20 13:46:32 -07:00
Gunnar Kudrjavets
61dbfbb6ce Add release build to RocksDB per-diff/post-commit tests
Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening.

Test Plan:
- `[p]arc diff --preview`
- Observe the execution in Sandcastle and make sure that release build and tests are executed.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60441
2016-07-08 16:02:29 -07:00
Gunnar Kudrjavets
aa432be4b9 Workarounds for continuous build implementation 2016-06-23 19:38:29 -07:00
Gunnar Kudrjavets
8cd9f04feb Test change to verify new commit detection 2016-06-23 12:08:44 -07:00
Gunnar Kudrjavets
8a4ee7e90b Trivial change to test cont. build 2016-06-23 11:53:38 -07:00
Gunnar Kudrjavets
6576fa05af Fix minor typos and PHP source file name used to trigger the builds
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.

Test Plan: - Testing with a private Sandcastle instance.

Reviewers: sdong, mung

Reviewed By: mung

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59919
2016-06-22 11:05:52 -07:00
Gunnar Kudrjavets
a52e4d7d02 Framework for enabling continuous RocksDB build and tests
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.

The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:

- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.

- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.

- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.

Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.

Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.

Reviewers: sdong, yhchiang, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59811
2016-06-21 11:38:54 -07:00
krad
550bf895e9 Minor bug fix with log name 2016-06-14 12:20:58 -07:00
Islam AbdelRahman
8100ec2cd1 Fix libgcc broken lib path
Summary: current libgcc lib path is broken, update it and make sure the new one exists

Test Plan:
verify that the new path exists
make check -j64

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59535
2016-06-12 00:11:08 -07:00
Islam AbdelRahman
bde7d1055f Fix clang_analyze path in fbcode_config.sh
Summary: current clang_analyze path is broken, update it

Test Plan:
USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make J=64 analyze -j64
make sure it works

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59505
2016-06-10 14:36:38 -07:00
krad
e9c1face6c Minor fix to precommit-check.py 2016-06-10 10:53:02 -07:00
krad
fcc47fa5f1 New features to precommit check script
Summary:
Added new features to help run the entire suite of tests on a diff
locally leveraging parallelism

Test Plan: Basic command line test

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59445
2016-06-10 10:33:22 -07:00
Islam AbdelRahman
edc764e91b Use valgrind built with gcc-4.9-glibc-2.20
Summary:
Right now we use valgrind built with gcc-5-glibc-2.23 which keep generating this error everywhere

```
==3224756== Conditional jump or move depends on uninitialised value(s)
==3224756==    at 0x401AD68: index (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4008933: _dl_map_object (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000ECC: map_doit (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40101C7: _dl_catch_error (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000B34: do_preload (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x400458C: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40185DB: _dl_sysdep_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4004F39: _dl_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000CE7: ??? (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==  Uninitialised value was created by a stack allocation
==3224756==    at 0x4004519: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==
```

Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64

Reviewers: andrewkr, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59343
2016-06-08 13:22:16 -07:00
Islam AbdelRahman
54db29b8f0 Use gvfs links in dependencies.sh
Summary:
Update `build_tools/update_dependencies.sh` to use /mnt/gvfs path in dependencies.sh
This should prevent us from having errors because of changing the path of a compiler or a library

Test Plan:
make all -j64
USE_CLANG make all -j64
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j64

Reviewers: andrewkr, sdong, yiwu

Reviewed By: yiwu

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59295
2016-06-07 14:22:52 -07:00
Yi Wu
10d46b9c8f Update tp2 clang path to fix clang build
Summary: clang was removed from tp2 so change to llvm-fb.

Test Plan:
  USE_CLANG=1 make all -j64

Reviewers: yhchiang, andrewkr, kradhakrishnan, sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59259
2016-06-06 18:29:39 -07:00
Andrew Kryczka
e42dc9192e Update paths for fbcode dependencies
Summary: The path to g++ was outdated. Ran ./build_tools/update_dependencies.sh.

Test Plan: make -j64 all

Reviewers: sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59229
2016-06-06 11:16:51 -07:00
Yi Wu
8f65feafc0 Have sandcastle run lite_test for every diff
Summary: Have sandcastle run unit test in lite mode for every diff.

Test Plan: seems sandcastle picked up changes here and running lite_test for this diff.

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57741
2016-05-06 14:51:20 -07:00
sdong
992a8f83b7 Not enable jemalloc status printing if USE_CLANG=1
Summary: Warning is printed out with USE_CLANG=1 when including jemalloc.h. Disable it in that case.

Test Plan: Run db_bench with USE_CLANG=1 and not. Make sure they can all build and jemalloc status is printed out in the case where USE_CLANG is not set.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57399
2016-04-28 16:16:14 -07:00
Islam AbdelRahman
af70f9ac6d Fix typo in build_tools/fbcode_config.sh
Summary: Fix typo in build_tools/fbcode_config.sh

Test Plan: none

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57363
2016-04-27 21:00:22 -07:00
Islam AbdelRahman
0850bc5147 Fix build on machines without jemalloc
Summary: It looks like we mistakenly enable JEMALLOC even if it's not available on the machine, that's why travis is failing

Test Plan:
check on my devserver
check on my mac

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57345
2016-04-27 18:25:19 -07:00
Sergey Makarenko
1c80dfab24 Print memory allocation counters
Summary:
Introduced option to dump malloc statistics using new option flag.
    Added new command line option to db_bench tool to enable this
    funtionality.
    Also extended build to support environments with/without jemalloc.

Test Plan:
1) Build rocksdb using `make` command. Launch the following command
    `./db_bench --benchmarks=fillrandom --dump_malloc_stats=true
    --num=10000000` end verified that jemalloc dump is present in LOG file.
    2) Build rocksdb using `DISABLE_JEMALLOC=1  make db_bench -j32` and ran
    the same db_bench tool and found the following message in LOG file:
    "Please compile with jemalloc to enable malloc dump".
    3) Also built rocksdb using `make` command on MacOS to verify behavior
    in non-FB environment.
    Also to debug build configuration change temporary changed
    AM_DEFAULT_VERBOSITY = 1 in Makefile to see compiler and build
    tools output. For case 1) -DROCKSDB_JEMALLOC was present in compiler
    command line. For both 2) and 3) this flag was not present.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57321
2016-04-27 16:23:33 -07:00
dx9
b71c4e613f Alpine Linux Build (#990)
* Musl libc does not provide adaptive mutex. Added feature test for PTHREAD_MUTEX_ADAPTIVE_NP.

* Musl libc does not provide backtrace(3). Added a feature check for backtrace(3).

* Fixed compiler error.

* Musl libc does not implement backtrace(3). Added platform check for libexecinfo.

* Alpine does not appear to support gcc -pg option. By default (gcc has PIE option enabled) it fails with:

gcc: error: -pie and -pg|p|profile are incompatible when linking

When -fno-PIE and -nopie are used it fails with:

/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find gcrt1.o: No such file or directory

Added gcc -pg platform test and output PROFILING_FLAGS accordingly. Replaced pg var in Makefile with PROFILING_FLAGS.

* fix segfault when TEST_IOCTL_FRIENDLY_TMPDIR is undefined and default candidates are not suitable

* use ASSERT_DOUBLE_EQ instead of ASSERT_EQ

* When compiled with ROCKSDB_MALLOC_USABLE_SIZE UniversalCompactionFourPaths and UniversalCompactionSecondPathRatio tests fail due to premature memtable flushes on systems with 16-byte alignment. Arena runs out of block space before GenerateNewFile() completes.

Increased options.write_buffer_size.
2016-04-22 16:49:12 -07:00
Andrew Kryczka
ec84bef24a New legocastle output parsing
Summary:
Added a python script to parse combined stdout/stderr of legocastle
steps. Previously we just matched words like 'Failure', which didn't work since
even our test names matched that pattern.

I went through all the legocastle steps to come up with strict failure regexes
for the common failure cases. There is also some more complex logic to present
gtest failures, since the test name and failure message are not on the same
line.

There will definitely be error cases that don't match any of these patterns, so
we can iterate on it over time.

Test Plan:
no end-to-end test. I ran the legocastle steps locally and piped to
my script, then verified output, e.g.,

  $ set -o pipefail && TEST_TMPDIR=/dev/shm/rocksdb COMPILE_WITH_ASAN=1 OPT=-g make J=1 asan_check |& /usr/facebook/ops/scripts/asan_symbolize.py -d |& python build_tools/error_filter.py asan
  ==2058029==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000a414 at pc 0x4c12f6 bp 0x7ffcfb7a0520 sp 0x7ffcfb7a0518

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56691
2016-04-18 13:01:10 -07:00
krad
083cadc7e9 Minor fix to Java sandcastle job definition
Summary:
proxy config tools are not available in sandcastle environment. Hard
coding the proxy information in the job.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56775
2016-04-14 15:05:54 -07:00
krad
7c14d11eb9 Minor fix to java build job
Summary:
Java build is downloading junit when not available. Without the proxy
settings exported that would not work.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56751
2016-04-14 10:46:02 -07:00
krad
09be5cad5b Minor fix to sandcastle java job definition
Summary: System was not able to locate javac

Test Plan: Run command manually and observe output

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56709
2016-04-13 15:59:05 -07:00
krad
c2c8fe47f0 Add Java job for sandcastle
Summary: Adding java build job definition to sandcastle.

Test Plan: ./build_tools/rocksdb-lego-determinator java_build

Reviewers: yhchiang, sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56679
2016-04-13 11:40:51 -07:00
Gunnar Kudrjavets
994b3bd693 Add support for UBsan builds to RocksDB
Summary:
Undefined Behavior Sanitizer (ubsan) is //a good thing// which will help us to find sneaky bugs with low cost. Please see http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ for more details and official GCC documentation for more context: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html.

Changes itself are quite simple and pretty much imitating whatever is implemented for ASan.

Hooking the UBsan validation build to Sandcastle is a separate step and will be dealt as separate diff because code is in internal repository.

Test Plan: Make sure that that there no regressions when it comes to builds and test pass rate.

Reviewers: leveldb, sdong, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56049
2016-03-30 15:59:24 -07:00
sdong
07bb12d97d Update internal jemalloc and other versions
Summary: Update jemalloc to newer versions, as well as some other dependencies. Only effective for FB internal.

Test Plan: See all tests run

Reviewers: IslamAbdelRahman, MarkCallaghan

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55947
2016-03-25 10:42:20 -07:00
Dmitri Smirnov
e7cc49cbdf Add support for db_test2 for dev and CI runs 2016-03-23 16:33:57 -07:00
Islam AbdelRahman
171c8e80b1 Update dependencies / Fix Clang
Summary:
Update dependencies and fix broken clang symlink

version changes
```
zstd 0.4.7 => 0.5.1
jemalloc 4.0.3 => master
```

Test Plan:
make check -j64
USE_CLANG=1 make check -j64
ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64

Reviewers: sdong, yhchiang, anthony, andrewkr

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D55179
2016-03-07 12:57:19 -08:00
root
3373c81fa8 Modify build_tools/build_detect_platform to detect and set -march=z10 on Linux s390x. 2016-02-29 15:02:52 -05:00
root
21f17aaa60 Modified Makefile and build_tools/build_detect_platform to compile on Linux s390x. 2016-02-26 18:03:07 -05:00
Andrew Kryczka
6a2b4fcb80 Add flag to forcibly disable fallocate
Summary:
see https://github.com/facebook/rocksdb/issues/977; there are issues
with fallocate() on certain filesystems/kernel versions that can lead it to pre-
allocating blocks but never freeing them, even if they're unused.

Test Plan:
verified build commands omit DROCKSDB_FALLOCATE_PRESENT when this env
variable is set.

without disabling it:

  $ ROCKSDB_NO_FBCODE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  0

with disabling it:

  $ ROCKSDB_NO_FBCODE=1 DISABLE_FALLOCATE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  1

Reviewers: kradhakrishnan, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54069
2016-02-11 17:00:01 -08:00
Baraa Hamodi
21e95811d1 Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
Islam AbdelRahman
77ef87ccb0 Update fbcode_config4.8.1.sh to use update_dependencies.sh
Summary:
This patch update fbcode_config4.8.1.sh to get it's dependencies the same way we updated fbcode_config.sh in D53037

as a result zstd is upgraded to 0.4.7 instead of 0.4.5

Test Plan:
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make check -j64

Reviewers: yhchiang, andrewkr, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53355
2016-01-26 12:13:27 -08:00
Islam AbdelRahman
f7ef1a6132 Include rest of dependencies in dependencies.sh
Summary:
This diff

  - Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh
  - updtade zst to 0.4.7
  - It also fix a problem in clang scan build

Test Plan:
make check
USE_CLANG=1 make check
USE_CLANG=1 make analyze
coverage_test.sh

Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53301
2016-01-25 14:35:04 -08:00
Islam AbdelRahman
538eec0661 Update fbcode_config.sh to use latest versions automatically
Summary:
Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated

```
Snappy: 1.0.3 => 1.1.3
GFLAGS: 1.6 => 2.1.1
JEMALLOC: 3.6.0 => 4.0.3
```

I have also updated clang from 3.7 to 3.7.1

```
Clang 3.7 => 3.7.1
```

Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library

Test Plan:
make check -j64
USE_CLANG=1 make check -j64

DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running)

Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53037
2016-01-22 11:56:32 -08:00
krad
b0a15e7fb9 Mechanism to run CI jobs on local branch via commit_prereq
Summary: This patch provides a mechanism to run pre commit tests on the local
branch before committing. This can help prevent frequent build breaks.

The tests can be run in parallel by specifying the J=<..> environment
variable.

Test Plan: Run manually

Reviewers: sdong rven tec

CC: leveldb@

Task ID: #9689218

Blame Rev:
2016-01-21 16:22:21 -08:00
bcbrock
f423f05dcd Simple changes to support builds for ppc64[le] consistent with X86
These simple changes are required to allow builds on ppc64[le] systems
consistent with X86. The Makefile now recognizes both ppc64 and ppc64le, and
in the absence of PORTABLE=1, the code will be built analogously to the X86
-march=native.

Note that although GCC supports -mcpu=native -mtune=native on POWER, it
doesn't work correctly on all systems. This is why we need to get the actual
machine model from the AUX vector.
2016-01-19 09:08:19 -06:00
agiardullo
83e1de92af move internal build to use zstd 0.4.5
Summary: update internal build scripts to use zstd 0.4.5.

Test Plan: built and ran tests with and without ROCKSDB_FBCODE_BUILD_WITH_481

Reviewers: yhchiang, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52833
2016-01-15 13:27:35 -08:00
Mark Callaghan
4041903ecd Enhance db_bench write rate limit
Summary:
1) changes tools/{benchmark,run_flash_bench}.sh to optionally use the write rate limit
2) removes code for --writes_per_second and switches the 'background' write rate limit
to use --benchmark_write_rate_limit

Replaces https://reviews.facebook.net/D49113

Task ID: #9555881

Blame Rev:

Test Plan:
tools/run_flash_bench.sh

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D52485
2016-01-04 12:01:27 -08:00
sdong
d8677a8d2c Upgrade internal CLANG version for FB-internal gcc 4.8.1
Summary: After removing two move operations, we can make CLANG 3.7 build pass under GCC 4.8.1.

Test Plan: USE_CLANG=1 ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32

Reviewers: yhchiang, IslamAbdelRahman, rven, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52365
2015-12-29 10:33:23 -08:00
Andrew Kryczka
9c176ef90d Update liblz4 to r131
Summary:
It was already built in third-party2 but the include/library paths in
rocksdb hadn't been updated accordingly.

Test Plan:
verified build works

  $ make clean && make -j32 all
  $ make clean && USE_CLANG=1 make -j32 all

Reviewers: cyan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52299
2015-12-23 17:13:31 -08:00
krad
362d819a14 Improving parser
Summary: Improving the parser string to make better error report. Currently the
error report fails to capture the assert details. This fix addresses the issue.

Test Plan: None

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-12-11 11:06:42 -08:00
charsyam
c30b499541 fix typos in comments 2015-12-11 01:54:48 +09:00
sdong
758dbec7f0 Fix fb-only build for gcc 4.8.1
Summary: fb build for gcc 4.8.1 is broken for the ZSTD dependency is not fixed after ea11923550. Fixing it.

Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make -j40 OPT=-g

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51741
2015-12-08 17:40:59 -08:00
sdong
ea11923550 Upgrade to ZSTD 0.4.2
Summary: Change to call the new compression function.

Test Plan: build and run db_bench with the compression to make sure it compresses.

Reviewers: anthony, rven, kradhakrishnan, IslamAbdelRahman, igor, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51603
2015-12-08 16:33:26 -08:00
sdong
d5239f8709 build_tools/fbcode_config4.8.1.sh: upgrade versions of some dependencies
Summary: Upgrade version of some dependencies in build_tools/fbcode_config4.8.1.sh. I didn't upgrade version for CLANG because some warnings are shown.

Test Plan:
build:

ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make all -j32
as well as
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32

Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, igor

Reviewed By: igor

Subscribers: igor, kradhakrishnan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51003
2015-11-18 16:18:11 -08:00
Dmitri Smirnov
7cb1293b60 Fix log names when scheduling runs and reruns 2015-11-17 18:41:29 -08:00
Dmitri Smirnov
94e39e2364 Exclude DBTest.FileCreationRandomFailure as a long running test
Increase concurrency to 18
  Fix exclusion but in the ps script
2015-11-17 13:54:13 -08:00
Dmitri Smirnov
2a0510c9f4 Failed tests must be rerun with concurrency 1 2015-11-17 11:23:09 -08:00
yuslepukhin
935d1495c5 Run tests imporvements
Add sequential rerun for any failed tests. Add env_test case.
  Limit concurrency
  Allow to specify individual tests
  Take $Limit into account when displaying number of tests
2015-11-12 14:42:38 -08:00
Nathan Bronson
986230b8c8 Revert "Fix TSAN build for fbcode"
Summary:
Reverting c745f1d2c4 because it
was based on an incorrect understanding of the correct way to enable
TSAN tests (it assumes "make COMPILE_WITH_TSAN=1 check" but in fact only
"COMPILE_WITH_TSAN=1 make check" is supported).

Test Plan: COMPILE_WITH_TSAN=1 make check

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50445
2015-11-09 11:51:10 -08:00
Nathan Bronson
c745f1d2c4 Fix TSAN build for fbcode
Summary:
TSAN builds for gcc 4.9 need a PIC version of the libraries
taken from the fbcode platform.  This is accomplished by assuming every
.a has a _pic.a sibling, and by fixing the third-party2 zlib build.

Test Plan: make COMPILE_WITH_TSAN=1 check

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50331
2015-11-06 09:05:12 -08:00
Dmitri Smirnov
3277d172ba Improve concurrency when running tests
PowerShell seems to have a hard time when a flood of async tasks is
  scheduled at the same time.
  I speculated that WaitForMultipleObjects() in Windows can only take up
  to 64 process handles and if you want to handle more than you should write
  some additional code which can be sub-optimal. I.e to implement Wait-Job -Any.
  I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62.
  So in the new version the script fires up up to $Concurrency value
  and wait for anything to complete before starting any more processes.
  This improved matters greatly. Individual tests against ramdrive now
  run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency
  values of 8-16. About 48 is required to load a CPU on my box running against HD
  but that does not improve running times much.

  Other changes include respect -EnableJE for the individual test exes.
  Enforce exclusions for the individual tests.
2015-11-05 14:03:41 -08:00
Igor Canadi
14c6e1a04c Add write_stress to RocksDB Legocastle runs
Summary: As title. Let's run it for 1 hour.

Test Plan: How can I test legocastle changes?

Reviewers: IslamAbdelRahman, yhchiang, rven, sdong, anthony, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49653
2015-11-05 12:07:39 -08:00
Islam AbdelRahman
581f20fd8b Add LITE tests to Legocastle
Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE

Test Plan: will be tested after landing in fbcode

Reviewers: sdong, yhchiang, igor, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49065
2015-10-26 11:50:29 -07:00
Javier González
6e6dd5f6f9 Split posix storage backend into Env and library
Summary: This patch splits the posix storage backend into Env and
the actual *File implementations. The motivation is to allow other Envs
to use posix as a library. This enables a storage backend different from
posix to split its secondary storage between a normal file system
partition managed by posix, and it own media.

Test Plan: No new functionality is added to posix Env or the library,
thus the current tests should suffice.
2015-10-22 17:31:31 +02:00
krad
7717ad1afe Adding artifacts to stress_crash CI job
Summary: Adding the ability to upload logs and db content to storage after the
completion of the job

Test Plan: Manual run

Reviewers:

CC: leveldb@

Task ID: #8754201

Blame Rev:
2015-10-19 20:13:14 -07:00
krad
bb64d6da46 Disabling TSAN crash test
Summary: The test is failing with OOM. Temporarily disabled.

Test Plan: Manual check

Reviewers:

CC: leveldb@

Task ID: #8732029

Blame Rev:
2015-10-15 13:42:34 -07:00
Yueh-Hsuan Chiang
c4366165e7 Merge pull request #759 from jwlent55/statically-load-compression-libraries
Ensure that the compression libraries are statically linked into dyna…
2015-10-09 20:03:39 -07:00
krad
64546af83e Adding parser to CI jobs
Summary: The parser will help parse the output in order to send meaningful
notifications.

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-09 17:58:31 -07:00
James Lent
5a7222782a Ensure that the compression libraries are statically linked into dynamic libraries included
in the Java jar.  Also build the linux libraries using the portable flag to fix a problem with
the linux32 build and improve the general portability of the RocksDB dynamic libraries.
==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
2015-10-09 14:41:40 -04:00
krad
77318ee14b Enable crash CI jobs
Summary: Enabling CI crash test jobs

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #8635140

Blame Rev:
2015-10-08 10:14:43 -07:00
Igor Canadi
f925208ffc Create Makefile target unity_test
Summary: unity_test will compile db_test with unity.a instead of librocksdb.a. This will test both the compilation and some small amount of runtime.

Test Plan: This is a test :)

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48297
2015-10-07 14:46:18 -07:00
krad
726d9ce1a6 Disabling unity
Summary: Unity CI job is broken. Disabling.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #8632483

Blame Rev:
2015-10-06 17:51:31 -07:00
krad
60b1c0a701 Fix to CI job definition
Summary: Sandcastle does not raise a task if assignee is specified. Removing the
assignee will force the system to use the oncall.

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-06 14:21:10 -07:00
krad
a39897369a Adding features to CI job description
Summary: Added following capability
(1) Ability to do configurable reports using args
(2) Support to generating a task on failure
(3) Ability to configure email trigger condition

Test Plan: Manual testing

Reviewers:

CC:leveldb@

Task ID: #6968635

Blame Rev:
2015-10-02 12:36:58 -07:00
Igor Canadi
a47bf325c6 Merge pull request #748 from yuslepukhin/improve_test_runs
Improve test running scripts
2015-10-02 11:42:53 -07:00
Dmitri Smirnov
65324a16dc Improve test running scripts
Introduce proper command line arguments so we can control the script
  Add appveyor support
  Add an ability to run all other (non db_tests) test executables in parallel
  Use .NET HashSet instead of empty valued hashtable to improve the looks
  TODO: Some of the tests do not use GTests and need to improve log parsing
2015-10-02 11:33:35 -07:00
Evan Shaw
7a23e4d8ca New amalgamation target
This commit adds two new targets to the Makefile: rocksdb.cc and rocksdb.h

These files, when combined with the c.h header, are a self-contained RocksDB
source distribution called an amalgamation. (The name comes from SQLite's, which
is similar in concept.)

The main benefit of an amalgamation is that it's very easy to drop into a
new project. It also compiles faster compared to compiling individual source
files and potentially gives the compiler more opportunity to make optimizations
since it can see all functions at once.

rocksdb.cc and rocksdb.h are generated by a new script, amalgamate.py.
A detailed description of how amalgamate.py works is in a comment at the top of
the file.

There are also some small changes to existing files to enable the amalgamation:
* Use quotes for includes in unity build
* Fix an old header inclusion in util/xfunc.cc
* Move some includes outside ifdef in util/env_hdfs.cc
* Separate out tool sources in Makefile so they won't be included in unity.cc
* Unity build now produces a static library

Closes #733
2015-10-01 08:29:31 +13:00
Igor Canadi
75134f7562 Merge pull request #741 from yuslepukhin/test_appveyor_baseline
Improve CI build and build switches
2015-09-29 13:16:40 -07:00
Dmitri Smirnov
2e7506d82c Improve CI build and build switches
Add an optimized build config switch for faster test runs
  Change compiler options to introduce more opitmizations and be more inline with MS internal switches.
  Make appveyor build to utilize all the avaiable cores on the VM (parallel)
  Introduce new appveyor configuration for daily test runs as it would take too long
  to run db_test after each checkin even in paralell.
  With some exclusions we make it in 38 minutes. We currently fail to install ramdisk during the build.
  Add a powershell script to faicilitate paralell run for db_test cases.
2015-09-29 12:22:48 -07:00
krad
7df348b405 Minor fix to CI job definition
Summary: Added email notification for a few jobs and fixed a bug in
no_compression script

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-25 14:54:51 -07:00
krad
e01f32cc3c Parameterizing email id
Summary: Removing the leveldb@fb.com and making it a command line argument

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-24 15:18:36 -07:00
krad
628216fc1f Simplifying valgrind testing
Summary: Our valgrind testing is buggy and cumbersome in terms of locating the
error. It originates from the fact we accumulate all output for the tests. It is
extremely hard to locate the point of error.

The communication between valgrind and the script is not sturdy. We are
experiencing bugs.

Simplifying to stop on first error.

Test Plan: Run manually

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-23 15:08:58 -07:00
krad
03dd8f3ca7 Fixing punit job description
Summary: The job errors with log print step. Removing the step and re-enabling
the test case.

Test Plan: Manual test run

Reviewers:

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-23 07:35:29 -07:00
krad
a6c22e3e41 Disabling parallel test CI job
Summary: Parallel test CI is failing. Reducing noise by temporarily disabling
the job

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-22 10:37:56 -07:00
krad
dd2e1eeb35 Disabling log running jobs
Summary: Long running jobs are blocking the queue due to lack of adequate
machines in the CI

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-21 12:58:33 -07:00
krad
4d6eb52d1a Fix to CI jobs.
Summary: The approach to serialize long scripts as string is flawed. Instead
adding args to invoke the long scripts.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-19 17:33:06 -07:00
krad
3bcc072d24 Added more CI jobs
Summary: Migrating the long script jobs and the left overs from Jenkin.

Test Plan: Run command manually

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-18 13:24:09 -07:00
Andres Noetzli
1b598213aa Check formatting in Travis
Summary:
Check code formatting before running tests. Goal is to make it easier to deal
with external pull requests.

Test Plan: See https://travis-ci.org/facebook/rocksdb/builds/80952963

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47211
2015-09-17 22:02:05 -07:00
krad
2819a1db32 Minor fix to CI job definition
Summary: Timeout needs to be set for command and job for it to work.

Test Plan: Manual

Reviewers: sdong igor

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-15 10:56:48 -07:00
Andres Noetzli
0bfe0573e8 Add gflags dependency to Travis script
Summary:
Travis is failing due to missing gflags
(https://travis-ci.org/facebook/rocksdb/jobs/80307921). Adding libgflags-dev
should help.

Test Plan: Run tests on Travis.

Reviewers: rven, anthony, yhchiang, aekmekji, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46905
2015-09-14 14:30:17 -07:00
krad
e2d6011f47 Minor fix to CI job definition
Summary: Command level timeouts are not working. Moving it to step level.

Test Plan: Manual testing sending commands

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-14 12:10:18 -07:00
krad
9f3a66a936 Improvements to CI jobs
Summary: Fixed
- Added timeouts for crash tests. They take around 9hrs.
- Added oncall so we can get notifications for timeout etc.
- Fixed a bug in the valgrind script
- Cosmetic fix

Test Plan: Manual run

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-11 16:09:01 -07:00
krad
7db1471cc1 Minor fix to CI job
Summary: Fix the issue where compilation error will not result in log file not
found error

Test Plan: None

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-11 13:00:55 -07:00
krad
41bce05869 CI job improvements
Summary: Added more jobs and refactored code express the jobs more cleanly

Test Plan: Manual test

Reviewers: igor sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-10 14:29:41 -07:00
krad
52386a1e73 Minor fix to sandcastle jobs commands
Summary: Dropping parallel compilation since that seen to introduce compilation
errors spuriously

Test Plan: Run the command manually

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-09 10:54:50 -07:00
krad
778cf4449b Adding email notification.
Summary: The email notifications needs to be at command level and the job level.
Adding command level notification.

Test Plan: Run command manually

Reviewers: igor sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-08 14:19:03 -07:00
krad
20ef64cae0 Moving jobs to use gcc-4.9
Summary: Fixed the glitch in Sandcastle and added gcc-4.9 support. Moving the
jobs to 4.9

Test Plan: Manually run the script

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-03 15:20:36 -07:00
krad
b8a962d4f5 Adding commands for few more CI jobs.
Summary: Added commands to spawn crash, stress, asan, asan_crash, clang CI jobs

Test Plan: Manual test

Reviewers: sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-02 14:06:03 -07:00
krad
c6d870ffb0 Merge branch 'arcpatch-D45741' 2015-08-31 12:42:56 -07:00