Summary:
Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people.
Closes https://github.com/facebook/rocksdb/pull/3346
Differential Revision: D6690544
Pulled By: siying
fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49
Summary:
**# Summary**
RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain.
This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used.
**# Performance Test Results**
We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm.
Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s.
1) ReadRandom in db_bench overall metrics
PER RUN
Algorithm | run | micros/op | ops/sec |Throughput (MB/s)
3-way | 1 | 4.143 | 241387 | 26.7
3-way | 2 | 3.775 | 264872 | 29.3
3-way | 3 | 4.116 | 242929 | 26.9
FastCrc32c|1 | 4.037 | 247727 | 27.4
FastCrc32c|2 | 4.648 | 215166 | 23.8
FastCrc32c|3 | 4.352 | 229799 | 25.4
AVG
Algorithm | Average of micros/op | Average of ops/sec | Average of Throughput (MB/s)
3-way | 4.01 | 249,729 | 27.63
FastCrc32c | 4.35 | 230,897 | 25.53
2) Crc32c computation CPU cost (inclusive samples percentage)
PER RUN
Implementation | run | TotalSamples | Crc32c percentage
3-way | 1 | 4,572,250,000 | 4.37%
3-way | 2 | 3,779,250,000 | 4.62%
3-way | 3 | 4,129,500,000 | 4.48%
FastCrc32c | 1 | 4,663,500,000 | 11.24%
FastCrc32c | 2 | 4,047,500,000 | 12.34%
FastCrc32c | 3 | 4,366,750,000 | 11.68%
**# Test Plan**
make -j64 corruption_test && ./corruption_test
By default it uses 3-way SSE algorithm
NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test
make clean && DEBUG_LEVEL=0 make -j64 db_bench
make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench
Closes https://github.com/facebook/rocksdb/pull/3173
Differential Revision: D6330882
Pulled By: yingsu00
fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9
Summary:
I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google".
Closes https://github.com/facebook/rocksdb/pull/3212
Differential Revision: D6456973
Pulled By: ajkr
fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656
Summary:
If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2.
Closes https://github.com/facebook/rocksdb/pull/3156
Differential Revision: D6304703
Pulled By: ajkr
fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef
Summary:
Add per-exe execution capability
Add fix parsing of groups/tests
Add timer test exclusion
Fix unit tests
Ifdef threadpool specific tests that do not pass on Vista threadpool.
Remove spurious outout from prefix_test so test case listing works
properly.
Fix not using standard test directories results in file creation errors
in sst_dump_test.
BlobDb fixes:
In C++ end() iterators can not be dereferenced. They are not valid.
When deleting blob_db_ set it to nullptr before any other code executes.
Not fixed:. On Windows you can not delete a file while it is open.
[ RUN ] BlobDBTest.ReadWhileGC
d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options)
IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied
d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options)
IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied
write_batch
Should not call front() if there is a chance the container is empty
Closes https://github.com/facebook/rocksdb/pull/3152
Differential Revision: D6293274
Pulled By: sagar0
fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956
Summary:
* make `checksum_type_string_map` available for lite
* comment out `FilesPerLevel` in lite mode.
* travis and legocastle lite build also build `all` target and run tests
Closes https://github.com/facebook/rocksdb/pull/3015
Differential Revision: D6069822
Pulled By: yiwu-arbug
fbshipit-source-id: 9fe92ac220e711e9e6ed4e921bd25ef4314796a0
Summary:
Update dependencies.sh. Also update tbb to 4.3, which is the latest available in TP2.
Closes https://github.com/facebook/rocksdb/pull/2812
Differential Revision: D5741394
Pulled By: yiwu-arbug
fbshipit-source-id: cafa0b7179f9a44669e5ccace818a02b42336781
Summary:
Most of the data used here in shell commands is not generated directly from user input but some data (ie: from environment variables) may have been external influenced. It is a good practice to escape this data before using it in a shell command.
Originally D4800264 but we never quite got it merged.
Reviewed By: yiwu-arbug
Differential Revision: D5595052
fbshipit-source-id: c09d8b47fe35fc6a47afb4933ccad9d56ca8d7be
Summary:
With some compilers, `-std=c++11` is necessary for <cstdint> to be
available. Pass this flag via $PLATFORM_CXXFLAGS. Fixes#2488.
Closes https://github.com/facebook/rocksdb/pull/2545
Differential Revision: D5620610
Pulled By: yiwu-arbug
fbshipit-source-id: 2f975b8c1ad52e283e677d9a33543abd064f13ce
Summary:
TSAN shows error when we grab too many locks at the same time. In TSAN crash test, make one shard key cover 2^22 keys so that no many keys will be hold at the same time.
Closes https://github.com/facebook/rocksdb/pull/2719
Differential Revision: D5609035
Pulled By: siying
fbshipit-source-id: 930e5d63fff92dbc193dc154c4c615efbdf06c6a
Summary:
Seems the only function of the script is to create a new branch, which can be done easily. I'm removing it.
Closes https://github.com/facebook/rocksdb/pull/2623
Differential Revision: D5468681
Pulled By: yiwu-arbug
fbshipit-source-id: 87dea5ecc4c85e06941ccbc36993f7f589063878
Summary:
We don't need this format compatibility hack anymore. We should remove it to make things simpler.
Closes https://github.com/facebook/rocksdb/pull/2607
Differential Revision: D5444107
Pulled By: siying
fbshipit-source-id: 7ef587dd0cacfc15a4083a137adba8e6bfddac7e
Summary:
We forgot to add the new flag in internal build script. Add it.
Closes https://github.com/facebook/rocksdb/pull/2360
Differential Revision: D5121428
Pulled By: siying
fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f
Summary:
We've had a couple CockroachDB users fail to build RocksDB on exotic platforms, so I figured I'd try my hand at solving these issues upstream. The problems stem from a) `USE_SSE=1` being too aggressive about turning on SSE4.2, even on toolchains that don't support SSE4.2 and b) RocksDB attempting to detect support for thread-local storage based on OS, even though it can vary by compiler on the same OS.
See the individual commit messages for details. Regarding SSE support, this PR should change virtually nothing for non-CMake based builds. `make`, `PORTABLE=1 make`, `USE_SSE=1 make`, and `PORTABLE=1 USE_SSE=1 make` function exactly as before, except that SSE support will be automatically disabled when a simple SSE4.2-using test program fails to compile, as it does on OpenBSD. (OpenBSD's ports GCC supports SSE4.2, but its binutils do not, so `__SSE_4_2__` is defined but an SSE4.2-using program will fail to assemble.) A warning is emitted in this case. The CMake build is modified to support the same set of options, except that `USE_SSE` is spelled `FORCE_SSE42` because `USE_SSE` is rather useless now that we can automatically detect SSE support, and I figure changing options in the CMake build is less disruptive than changing the non-CMake build.
I've tested these changes on all the platforms I can get my hands on (macOS, Windows MSVC, Windows MinGW, and OpenBSD) and it all works splendidly. Let me know if there's anything you object to—I obviously don't mean to break any of your build pipelines in the process of fixing ours downstream.
Closes https://github.com/facebook/rocksdb/pull/2199
Differential Revision: D5054042
Pulled By: yiwu-arbug
fbshipit-source-id: 938e1fc665c049c02ae15698e1409155b8e72171
Summary:
Currently, the RPM package will install the lib and header files into `/usr/package/lib` and `/usr/package/include` which is not in the default search paths. It is reasonable to install them under `/usr/lib` and `/usr/include` so that no extra configuration is required.
Closes https://github.com/facebook/rocksdb/pull/2221
Differential Revision: D5054030
Pulled By: yiwu-arbug
fbshipit-source-id: 1d23de5ff21f07e6738c9dfa04429acd7a839143
Summary:
- added a feature test in build_detect_platform to check whether sched_getcpu() is available. glibc offers it only on some platforms (e.g., linux but not mac); this way should be easier than maintaining a list of platforms on which it's available.
- refactored PhysicalCoreID() to be simpler / less repetitive. ordered the conditional compilation clauses from most-to-least preferred
Closes https://github.com/facebook/rocksdb/pull/2272
Differential Revision: D5038093
Pulled By: ajkr
fbshipit-source-id: 81d7db3cc620250de220bdeb3194b2b3d7673de7
Summary:
VALGRIND_VER was left empty after moving the environment to GCC-5. Set it back.
Closes https://github.com/facebook/rocksdb/pull/2234
Differential Revision: D4978534
Pulled By: siying
fbshipit-source-id: f0640d58e8f575f75fb3f8b92e686c9e0b6a59bb
Summary:
Replacement of #2147
The change was squashed due to a lot of conflicts.
Closes https://github.com/facebook/rocksdb/pull/2194
Differential Revision: D4929799
Pulled By: siying
fbshipit-source-id: 5cd49c254737a1d5ac13f3c035f128e86524c581
Summary:
When people are working off of a rocksdb fork, i.e. when their 'origin'
points to github.com/<username>/rocksdb, the script creates a new branch
and pushes to their origin. The new branch created by this script should
instead be pushed to github.com/facebook/rocksdb. Many people might
have named facebook/rocksdb remote as 'upstream' (or something else).
This fix provides an option to specify the remote to push the branch to.
The default is still 'origin'
More context:
When I created 5.4 branch using this script, it got pushed to sagar0/rocksdb instead of facebook/rocksdb, as I was working off of a fork. My 'origin' was pointing to sagar0/rocksdb. My 'upstream' was set to 'facebook/rocksdb'. So, I had to manually push the branch to my 'upstream'.
Closes https://github.com/facebook/rocksdb/pull/2156
Differential Revision: D4885333
Pulled By: sagar0
fbshipit-source-id: 9410eab5bd9bbefc340059800bd6b8434406729d
Summary:
Rework test running script.
New options SuiteRun - runs specified executables as google suite
test cases in parallel.
Run - this option now runs executables in parallel the same as 'tests'
RunAll - scans for test executables and attempts to run them all
as suites except those mentiones in RunOnly (hardcoded in the script)
or specified either in $ExcludeTestCases $ExcludeTestExe
Closes https://github.com/facebook/rocksdb/pull/2089
Differential Revision: D4832212
Pulled By: yiwu-arbug
fbshipit-source-id: 954990c
Summary:
Since non_shn CI was made to run in parallel, /dev/shm is automatically used. It defeated the purpose of the test to cover a non-ramfs file system.
Closes https://github.com/facebook/rocksdb/pull/2031
Differential Revision: D4764804
Pulled By: siying
fbshipit-source-id: 5666bda
Summary:
This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off.
Also adjust slightly appveyor script to run db_basic_test cases concurrently.
Closes https://github.com/facebook/rocksdb/pull/1959
Differential Revision: D4671972
Pulled By: IslamAbdelRahman
fbshipit-source-id: 1886620
Summary:
The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version.
Closes https://github.com/facebook/rocksdb/pull/1912
Differential Revision: D4609854
Pulled By: siying
fbshipit-source-id: 3dc04f2
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859
Differential Revision: D4541292
Pulled By: sagar0
fbshipit-source-id: 5b3a6ca
Summary:
The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed
soon.
Closes https://github.com/facebook/rocksdb/pull/1839
Differential Revision: D4509684
Pulled By: siying
fbshipit-source-id: 3efe296
Summary:
Update clang-format script to format diff since last commit from master,
instead of just last commit. In our common workflow we usually endup
with multiple commits for a single PR. This change make it easier to
format all stacking changes.
Closes https://github.com/facebook/rocksdb/pull/1684
Differential Revision: D4340597
Pulled By: yiwu-arbug
fbshipit-source-id: c18949e
Summary:
Previously:
$ make format
Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production
build_tools/format-diff.sh
You didn't have clang-format-diff.py available in your computer!
You can download it by running:
curl http://goo.gl/iUW1u2
Makefile:868: recipe for target 'format' failed
make: *** [format] Error 128
$ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 276 0 276 0 0 148 0 --:--:-- 0:00:01 --:--:-- 148m
$ more ~/bin/clang-format-diff.py
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>.
</BODY>
</HTML>
Closes https://github.com/facebook/rocksdb/pull/1680
Differential Revision: D4338495
Pulled By: yiwu-arbug
fbshipit-source-id: e2b24d8
Summary:
This diff includes an implementation of CompactionFilter that allows
users to write CompactionFilter in Lua. With this ability, users can
dynamically change compaction filter logic without requiring building
the rocksdb binary and restarting the database.
To compile, WITH_LUA_PATH must be specified to the base directory
of lua.
Closes https://github.com/facebook/rocksdb/pull/1478
Differential Revision: D4150138
Pulled By: yhchiang
fbshipit-source-id: ed84222
Summary:
Rocksdb currently has many references to std::map.emplace_back()
which is not implemented in gcc 4.7, but valid in gcc 4.8. Confirmed that
it did not build with gcc 4.7, but builds fine with gcc 4.8
Closes https://github.com/facebook/rocksdb/pull/1272
Differential Revision: D4101385
Pulled By: IslamAbdelRahman
fbshipit-source-id: f6af453
Summary:
valgrind_test is the correct way to run valgrind tests.
this is becasue we need to force DISABLE_JEMALLOC
Test Plan: Running sandcastle and contrun
Reviewers: IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D65451
Summary:
ZSTD 1.0.0 is coming. We can finally add a support of ZSTD without worrying about compatibility.
Still keep ZSTDNotFinal for compatibility reason.
Test Plan: Run all tests. Run db_bench with ZSTD version with RocksDB built with ZSTD 1.0 and older.
Reviewers: andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: cyan, igor, IslamAbdelRahman, leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D63141
Summary: Splitting the makefile part of D55581.
Test Plan:
make all check -j32
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all check -j32
ROCKSDB_NO_FBCODE=1 make all check -j32
export TBB_BASE=/mnt/gvfs/third-party2/tbb/afa54b33cfcf93f1d90a3160cdb894d6d63d5dca/4.0_update2/gcc-4.9-glibc-2.20/e9936bf;
ROCKSDB_NO_FBCODE=1 CFLAGS="-I $TBB_BASE/include" LDFLAGS="-L $TBB_BASE/lib -Wl,-rpath=$TBB_BASE/lib" make all check -j32
Reviewers: IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: kradhakrishnan, yhchiang, IslamAbdelRahman, andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56979
Summary:
We have replaced LEVELDB_PLATFORM_POSIX with ROCKSDB_PLATFORM_POSIX in our code
replace it in Makefile
Test Plan: make check
Reviewers: sdong, yiwu, andrewkr, lightmark
Reviewed By: lightmark
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61365
Summary:
We may not have permission on all /dev/shm to fix the sticky bit.
Making the sticky bit fix advisory.
Test Plan: Run CI job locally
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D61371
Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure.
Test Plan:
- Land the changes.
- Trigger a RocksDB contrun - observe the results.
- Look at the results from the nightly runs and fix issues as necessary.
Current testing so far has been in isolation for various components.
Reviewers: andrewkr, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60909
Summary:
Removing moreutils from sandcastle and adding gnu parallel.
Then passing in J= nproc command
Test Plan: Testing on sandcastle
Reviewers: sdong, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61017
Summary: Increase test timeout to some tests to unblock CI.
Test Plan: watch how it runs.
Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61263
Summary:
Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds.
Add a parameter in crash test script to allow we pass it through an environment variable.
Test Plan: Run it manually.
Reviewers: IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D61053
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system.
Test Plan: See the tests launched by Phabricator
Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku
Reviewed By: gunnarku
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60963
Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job.
Test Plan: Run it locally.
Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60951
Summary: As Title.
Test Plan: See how the diff works.
Reviewers: kradhakrishnan, andrewkr, gunnarku
Reviewed By: gunnarku
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60933
Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening.
Test Plan:
- `[p]arc diff --preview`
- Observe the execution in Sandcastle and make sure that release build and tests are executed.
Reviewers: sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D60441
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.
Test Plan: - Testing with a private Sandcastle instance.
Reviewers: sdong, mung
Reviewed By: mung
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59919
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.
The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:
- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.
- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.
- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.
Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.
Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.
Reviewers: sdong, yhchiang, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59811
Summary: current libgcc lib path is broken, update it and make sure the new one exists
Test Plan:
verify that the new path exists
make check -j64
Reviewers: yiwu, sdong, andrewkr
Reviewed By: andrewkr
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59535
Summary: current clang_analyze path is broken, update it
Test Plan:
USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make J=64 analyze -j64
make sure it works
Reviewers: yiwu, sdong, andrewkr
Reviewed By: andrewkr
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59505
Summary:
Added new features to help run the entire suite of tests on a diff
locally leveraging parallelism
Test Plan: Basic command line test
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D59445
Summary:
Right now we use valgrind built with gcc-5-glibc-2.23 which keep generating this error everywhere
```
==3224756== Conditional jump or move depends on uninitialised value(s)
==3224756== at 0x401AD68: index (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x4008933: _dl_map_object (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x4000ECC: map_doit (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x40101C7: _dl_catch_error (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x4000B34: do_preload (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x400458C: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x40185DB: _dl_sysdep_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x4004F39: _dl_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== by 0x4000CE7: ??? (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756== Uninitialised value was created by a stack allocation
==3224756== at 0x4004519: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==
```
Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64
Reviewers: andrewkr, yiwu, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59343
Summary:
Update `build_tools/update_dependencies.sh` to use /mnt/gvfs path in dependencies.sh
This should prevent us from having errors because of changing the path of a compiler or a library
Test Plan:
make all -j64
USE_CLANG make all -j64
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j64
Reviewers: andrewkr, sdong, yiwu
Reviewed By: yiwu
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D59295
Summary: clang was removed from tp2 so change to llvm-fb.
Test Plan:
USE_CLANG=1 make all -j64
Reviewers: yhchiang, andrewkr, kradhakrishnan, sdong, lightmark, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D59259
Summary: The path to g++ was outdated. Ran ./build_tools/update_dependencies.sh.
Test Plan: make -j64 all
Reviewers: sdong, lightmark, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D59229
Summary: Have sandcastle run unit test in lite mode for every diff.
Test Plan: seems sandcastle picked up changes here and running lite_test for this diff.
Reviewers: kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D57741
Summary: Warning is printed out with USE_CLANG=1 when including jemalloc.h. Disable it in that case.
Test Plan: Run db_bench with USE_CLANG=1 and not. Make sure they can all build and jemalloc status is printed out in the case where USE_CLANG is not set.
Reviewers: andrewkr, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D57399
Summary: It looks like we mistakenly enable JEMALLOC even if it's not available on the machine, that's why travis is failing
Test Plan:
check on my devserver
check on my mac
Reviewers: sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D57345
Summary:
Introduced option to dump malloc statistics using new option flag.
Added new command line option to db_bench tool to enable this
funtionality.
Also extended build to support environments with/without jemalloc.
Test Plan:
1) Build rocksdb using `make` command. Launch the following command
`./db_bench --benchmarks=fillrandom --dump_malloc_stats=true
--num=10000000` end verified that jemalloc dump is present in LOG file.
2) Build rocksdb using `DISABLE_JEMALLOC=1 make db_bench -j32` and ran
the same db_bench tool and found the following message in LOG file:
"Please compile with jemalloc to enable malloc dump".
3) Also built rocksdb using `make` command on MacOS to verify behavior
in non-FB environment.
Also to debug build configuration change temporary changed
AM_DEFAULT_VERBOSITY = 1 in Makefile to see compiler and build
tools output. For case 1) -DROCKSDB_JEMALLOC was present in compiler
command line. For both 2) and 3) this flag was not present.
Reviewers: sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D57321
* Musl libc does not provide adaptive mutex. Added feature test for PTHREAD_MUTEX_ADAPTIVE_NP.
* Musl libc does not provide backtrace(3). Added a feature check for backtrace(3).
* Fixed compiler error.
* Musl libc does not implement backtrace(3). Added platform check for libexecinfo.
* Alpine does not appear to support gcc -pg option. By default (gcc has PIE option enabled) it fails with:
gcc: error: -pie and -pg|p|profile are incompatible when linking
When -fno-PIE and -nopie are used it fails with:
/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find gcrt1.o: No such file or directory
Added gcc -pg platform test and output PROFILING_FLAGS accordingly. Replaced pg var in Makefile with PROFILING_FLAGS.
* fix segfault when TEST_IOCTL_FRIENDLY_TMPDIR is undefined and default candidates are not suitable
* use ASSERT_DOUBLE_EQ instead of ASSERT_EQ
* When compiled with ROCKSDB_MALLOC_USABLE_SIZE UniversalCompactionFourPaths and UniversalCompactionSecondPathRatio tests fail due to premature memtable flushes on systems with 16-byte alignment. Arena runs out of block space before GenerateNewFile() completes.
Increased options.write_buffer_size.
Summary:
Added a python script to parse combined stdout/stderr of legocastle
steps. Previously we just matched words like 'Failure', which didn't work since
even our test names matched that pattern.
I went through all the legocastle steps to come up with strict failure regexes
for the common failure cases. There is also some more complex logic to present
gtest failures, since the test name and failure message are not on the same
line.
There will definitely be error cases that don't match any of these patterns, so
we can iterate on it over time.
Test Plan:
no end-to-end test. I ran the legocastle steps locally and piped to
my script, then verified output, e.g.,
$ set -o pipefail && TEST_TMPDIR=/dev/shm/rocksdb COMPILE_WITH_ASAN=1 OPT=-g make J=1 asan_check |& /usr/facebook/ops/scripts/asan_symbolize.py -d |& python build_tools/error_filter.py asan
==2058029==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000a414 at pc 0x4c12f6 bp 0x7ffcfb7a0520 sp 0x7ffcfb7a0518
Reviewers: kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56691
Summary:
proxy config tools are not available in sandcastle environment. Hard
coding the proxy information in the job.
Test Plan: Run manually
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56775
Summary:
Java build is downloading junit when not available. Without the proxy
settings exported that would not work.
Test Plan: Run manually
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56751
Summary: System was not able to locate javac
Test Plan: Run command manually and observe output
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D56709
Summary:
Undefined Behavior Sanitizer (ubsan) is //a good thing// which will help us to find sneaky bugs with low cost. Please see http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ for more details and official GCC documentation for more context: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html.
Changes itself are quite simple and pretty much imitating whatever is implemented for ASan.
Hooking the UBsan validation build to Sandcastle is a separate step and will be dealt as separate diff because code is in internal repository.
Test Plan: Make sure that that there no regressions when it comes to builds and test pass rate.
Reviewers: leveldb, sdong, IslamAbdelRahman, yhchiang
Reviewed By: yhchiang
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D56049
Summary: Update jemalloc to newer versions, as well as some other dependencies. Only effective for FB internal.
Test Plan: See all tests run
Reviewers: IslamAbdelRahman, MarkCallaghan
Reviewed By: IslamAbdelRahman
Subscribers: leveldb, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D55947
Summary:
see https://github.com/facebook/rocksdb/issues/977; there are issues
with fallocate() on certain filesystems/kernel versions that can lead it to pre-
allocating blocks but never freeing them, even if they're unused.
Test Plan:
verified build commands omit DROCKSDB_FALLOCATE_PRESENT when this env
variable is set.
without disabling it:
$ ROCKSDB_NO_FBCODE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
0
with disabling it:
$ ROCKSDB_NO_FBCODE=1 DISABLE_FALLOCATE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
1
Reviewers: kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D54069
Summary:
This patch update fbcode_config4.8.1.sh to get it's dependencies the same way we updated fbcode_config.sh in D53037
as a result zstd is upgraded to 0.4.7 instead of 0.4.5
Test Plan:
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make check -j64
Reviewers: yhchiang, andrewkr, rven, kradhakrishnan, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D53355
Summary:
This diff
- Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh
- updtade zst to 0.4.7
- It also fix a problem in clang scan build
Test Plan:
make check
USE_CLANG=1 make check
USE_CLANG=1 make analyze
coverage_test.sh
Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D53301
Summary:
Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated
```
Snappy: 1.0.3 => 1.1.3
GFLAGS: 1.6 => 2.1.1
JEMALLOC: 3.6.0 => 4.0.3
```
I have also updated clang from 3.7 to 3.7.1
```
Clang 3.7 => 3.7.1
```
Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library
Test Plan:
make check -j64
USE_CLANG=1 make check -j64
DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running)
Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong
Reviewed By: sdong
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D53037
Summary: This patch provides a mechanism to run pre commit tests on the local
branch before committing. This can help prevent frequent build breaks.
The tests can be run in parallel by specifying the J=<..> environment
variable.
Test Plan: Run manually
Reviewers: sdong rven tec
CC: leveldb@
Task ID: #9689218
Blame Rev:
These simple changes are required to allow builds on ppc64[le] systems
consistent with X86. The Makefile now recognizes both ppc64 and ppc64le, and
in the absence of PORTABLE=1, the code will be built analogously to the X86
-march=native.
Note that although GCC supports -mcpu=native -mtune=native on POWER, it
doesn't work correctly on all systems. This is why we need to get the actual
machine model from the AUX vector.
Summary: update internal build scripts to use zstd 0.4.5.
Test Plan: built and ran tests with and without ROCKSDB_FBCODE_BUILD_WITH_481
Reviewers: yhchiang, rven, kradhakrishnan, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D52833
Summary:
1) changes tools/{benchmark,run_flash_bench}.sh to optionally use the write rate limit
2) removes code for --writes_per_second and switches the 'background' write rate limit
to use --benchmark_write_rate_limit
Replaces https://reviews.facebook.net/D49113
Task ID: #9555881
Blame Rev:
Test Plan:
tools/run_flash_bench.sh
Revert Plan:
Database Impact:
Memcache Impact:
Other Notes:
EImportant:
- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -
Reviewers: igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D52485
Summary: After removing two move operations, we can make CLANG 3.7 build pass under GCC 4.8.1.
Test Plan: USE_CLANG=1 ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32
Reviewers: yhchiang, IslamAbdelRahman, rven, anthony
Reviewed By: anthony
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D52365
Summary:
It was already built in third-party2 but the include/library paths in
rocksdb hadn't been updated accordingly.
Test Plan:
verified build works
$ make clean && make -j32 all
$ make clean && USE_CLANG=1 make -j32 all
Reviewers: cyan, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D52299
Summary: Improving the parser string to make better error report. Currently the
error report fails to capture the assert details. This fix addresses the issue.
Test Plan: None
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: fb build for gcc 4.8.1 is broken for the ZSTD dependency is not fixed after ea11923550. Fixing it.
Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make -j40 OPT=-g
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D51741
Summary: Change to call the new compression function.
Test Plan: build and run db_bench with the compression to make sure it compresses.
Reviewers: anthony, rven, kradhakrishnan, IslamAbdelRahman, igor, yhchiang
Reviewed By: yhchiang
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D51603
Summary: Upgrade version of some dependencies in build_tools/fbcode_config4.8.1.sh. I didn't upgrade version for CLANG because some warnings are shown.
Test Plan:
build:
ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make all -j32
as well as
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32
Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, igor
Reviewed By: igor
Subscribers: igor, kradhakrishnan, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D51003
Add sequential rerun for any failed tests. Add env_test case.
Limit concurrency
Allow to specify individual tests
Take $Limit into account when displaying number of tests
Summary:
Reverting c745f1d2c4 because it
was based on an incorrect understanding of the correct way to enable
TSAN tests (it assumes "make COMPILE_WITH_TSAN=1 check" but in fact only
"COMPILE_WITH_TSAN=1 make check" is supported).
Test Plan: COMPILE_WITH_TSAN=1 make check
Reviewers: kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D50445
Summary:
TSAN builds for gcc 4.9 need a PIC version of the libraries
taken from the fbcode platform. This is accomplished by assuming every
.a has a _pic.a sibling, and by fixing the third-party2 zlib build.
Test Plan: make COMPILE_WITH_TSAN=1 check
Reviewers: sdong, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D50331
PowerShell seems to have a hard time when a flood of async tasks is
scheduled at the same time.
I speculated that WaitForMultipleObjects() in Windows can only take up
to 64 process handles and if you want to handle more than you should write
some additional code which can be sub-optimal. I.e to implement Wait-Job -Any.
I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62.
So in the new version the script fires up up to $Concurrency value
and wait for anything to complete before starting any more processes.
This improved matters greatly. Individual tests against ramdrive now
run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency
values of 8-16. About 48 is required to load a CPU on my box running against HD
but that does not improve running times much.
Other changes include respect -EnableJE for the individual test exes.
Enforce exclusions for the individual tests.
Summary: As title. Let's run it for 1 hour.
Test Plan: How can I test legocastle changes?
Reviewers: IslamAbdelRahman, yhchiang, rven, sdong, anthony, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D49653
Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE
Test Plan: will be tested after landing in fbcode
Reviewers: sdong, yhchiang, igor, kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D49065
Summary: This patch splits the posix storage backend into Env and
the actual *File implementations. The motivation is to allow other Envs
to use posix as a library. This enables a storage backend different from
posix to split its secondary storage between a normal file system
partition managed by posix, and it own media.
Test Plan: No new functionality is added to posix Env or the library,
thus the current tests should suffice.
Summary: Adding the ability to upload logs and db content to storage after the
completion of the job
Test Plan: Manual run
Reviewers:
CC: leveldb@
Task ID: #8754201
Blame Rev:
Summary: The parser will help parse the output in order to send meaningful
notifications.
Test Plan: Manual testing
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
in the Java jar. Also build the linux libraries using the portable flag to fix a problem with
the linux32 build and improve the general portability of the RocksDB dynamic libraries.
==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
Summary: unity_test will compile db_test with unity.a instead of librocksdb.a. This will test both the compilation and some small amount of runtime.
Test Plan: This is a test :)
Reviewers: kradhakrishnan
Reviewed By: kradhakrishnan
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D48297
Summary: Sandcastle does not raise a task if assignee is specified. Removing the
assignee will force the system to use the oncall.
Test Plan: Manual
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Added following capability
(1) Ability to do configurable reports using args
(2) Support to generating a task on failure
(3) Ability to configure email trigger condition
Test Plan: Manual testing
Reviewers:
CC:leveldb@
Task ID: #6968635
Blame Rev:
Introduce proper command line arguments so we can control the script
Add appveyor support
Add an ability to run all other (non db_tests) test executables in parallel
Use .NET HashSet instead of empty valued hashtable to improve the looks
TODO: Some of the tests do not use GTests and need to improve log parsing
This commit adds two new targets to the Makefile: rocksdb.cc and rocksdb.h
These files, when combined with the c.h header, are a self-contained RocksDB
source distribution called an amalgamation. (The name comes from SQLite's, which
is similar in concept.)
The main benefit of an amalgamation is that it's very easy to drop into a
new project. It also compiles faster compared to compiling individual source
files and potentially gives the compiler more opportunity to make optimizations
since it can see all functions at once.
rocksdb.cc and rocksdb.h are generated by a new script, amalgamate.py.
A detailed description of how amalgamate.py works is in a comment at the top of
the file.
There are also some small changes to existing files to enable the amalgamation:
* Use quotes for includes in unity build
* Fix an old header inclusion in util/xfunc.cc
* Move some includes outside ifdef in util/env_hdfs.cc
* Separate out tool sources in Makefile so they won't be included in unity.cc
* Unity build now produces a static library
Closes#733
Add an optimized build config switch for faster test runs
Change compiler options to introduce more opitmizations and be more inline with MS internal switches.
Make appveyor build to utilize all the avaiable cores on the VM (parallel)
Introduce new appveyor configuration for daily test runs as it would take too long
to run db_test after each checkin even in paralell.
With some exclusions we make it in 38 minutes. We currently fail to install ramdisk during the build.
Add a powershell script to faicilitate paralell run for db_test cases.
Summary: Added email notification for a few jobs and fixed a bug in
no_compression script
Test Plan: Manual test
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Our valgrind testing is buggy and cumbersome in terms of locating the
error. It originates from the fact we accumulate all output for the tests. It is
extremely hard to locate the point of error.
The communication between valgrind and the script is not sturdy. We are
experiencing bugs.
Simplifying to stop on first error.
Test Plan: Run manually
Reviewers: sdong igor
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: The job errors with log print step. Removing the step and re-enabling
the test case.
Test Plan: Manual test run
Reviewers:
CC: levledb@
Task ID: #6968635
Blame Rev:
Summary: Parallel test CI is failing. Reducing noise by temporarily disabling
the job
Test Plan: Manual
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Long running jobs are blocking the queue due to lack of adequate
machines in the CI
Test Plan: Manual testing
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: The approach to serialize long scripts as string is flawed. Instead
adding args to invoke the long scripts.
Test Plan: Manual test
Reviewers:
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Migrating the long script jobs and the left overs from Jenkin.
Test Plan: Run command manually
Reviewers: sdon igor
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Command level timeouts are not working. Moving it to step level.
Test Plan: Manual testing sending commands
Reviewers: sdong igor
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Fixed
- Added timeouts for crash tests. They take around 9hrs.
- Added oncall so we can get notifications for timeout etc.
- Fixed a bug in the valgrind script
- Cosmetic fix
Test Plan: Manual run
Reviewers: sdong igor
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Fix the issue where compilation error will not result in log file not
found error
Test Plan: None
Reviewers: sdong igor
CC: leveldb@
Task ID: #6968635
Blame Rev:
Summary: Added more jobs and refactored code express the jobs more cleanly
Test Plan: Manual test
Reviewers: igor sdong
CC: leveldb@
Task ID: #6968635
Blame Rev:
