Commit Graph

417 Commits

Author SHA1 Message Date
hyunwoo
c7662a44a4 fixed typo
Summary:
fixed typo
Closes https://github.com/facebook/rocksdb/pull/2376

Differential Revision: D5183630

Pulled By: ajkr

fbshipit-source-id: 133cfd0445959e70aa2cd1a12151bf3c0c5c3ac5
2017-06-05 11:27:34 -07:00
Aaron Gao
e7612798b5 update buckifer/TARGETS
Summary:
update targets file for release
Closes https://github.com/facebook/rocksdb/pull/2358

Differential Revision: D5115705

Pulled By: lightmark

fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794
2017-05-24 11:56:57 -07:00
Siying Dong
5068034666 range sync should be enabled
Summary:
We forgot to add the new flag in internal build script. Add it.
Closes https://github.com/facebook/rocksdb/pull/2360

Differential Revision: D5121428

Pulled By: siying

fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f
2017-05-24 09:58:24 -07:00
Nikhil Benesch
11c5d4741a cross-platform compatibility improvements
Summary:
We've had a couple CockroachDB users fail to build RocksDB on exotic platforms, so I figured I'd try my hand at solving these issues upstream. The problems stem from a) `USE_SSE=1` being too aggressive about turning on SSE4.2, even on toolchains that don't support SSE4.2 and b) RocksDB attempting to detect support for thread-local storage based on OS, even though it can vary by compiler on the same OS.

See the individual commit messages for details. Regarding SSE support, this PR should change virtually nothing for non-CMake based builds. `make`, `PORTABLE=1 make`, `USE_SSE=1 make`, and `PORTABLE=1 USE_SSE=1 make` function exactly as before, except that SSE support will be automatically disabled when a simple SSE4.2-using test program fails to compile, as it does on OpenBSD. (OpenBSD's ports GCC supports SSE4.2, but its binutils do not, so `__SSE_4_2__` is defined but an SSE4.2-using program will fail to assemble.) A warning is emitted in this case. The CMake build is modified to support the same set of options, except that `USE_SSE` is spelled `FORCE_SSE42` because `USE_SSE` is rather useless now that we can automatically detect SSE support, and I figure changing options in the CMake build is less disruptive than changing the non-CMake build.

I've tested these changes on all the platforms I can get my hands on (macOS, Windows MSVC, Windows MinGW, and OpenBSD) and it all works splendidly. Let me know if there's anything you object to—I obviously don't mean to break any of your build pipelines in the process of fixing ours downstream.
Closes https://github.com/facebook/rocksdb/pull/2199

Differential Revision: D5054042

Pulled By: yiwu-arbug

fbshipit-source-id: 938e1fc665c049c02ae15698e1409155b8e72171
2017-05-15 16:15:38 -07:00
Jeff Li
d004333021 Put lib files into suitable path in RPM package
Summary:
Currently, the RPM package will install the lib and header files into `/usr/package/lib` and `/usr/package/include` which is not in the default search paths. It is reasonable to install them under `/usr/lib` and `/usr/include` so that no extra configuration is required.
Closes https://github.com/facebook/rocksdb/pull/2221

Differential Revision: D5054030

Pulled By: yiwu-arbug

fbshipit-source-id: 1d23de5ff21f07e6738c9dfa04429acd7a839143
2017-05-15 14:05:46 -07:00
Andrew Kryczka
be421b0b16 portable sched_getcpu calls
Summary:
- added a feature test in build_detect_platform to check whether sched_getcpu() is available. glibc offers it only on some platforms (e.g., linux but not mac); this way should be easier than maintaining a list of platforms on which it's available.
- refactored PhysicalCoreID() to be simpler / less repetitive. ordered the conditional compilation clauses from most-to-least preferred
Closes https://github.com/facebook/rocksdb/pull/2272

Differential Revision: D5038093

Pulled By: ajkr

fbshipit-source-id: 81d7db3cc620250de220bdeb3194b2b3d7673de7
2017-05-10 12:29:23 -07:00
Dmitri Smirnov
35df23fe82 Fix suite exclisions
Summary: Closes https://github.com/facebook/rocksdb/pull/2250

Differential Revision: D5037141

Pulled By: siying

fbshipit-source-id: 53a0ab5553422839bb2df6a7badde54810774a3e
2017-05-10 11:12:38 -07:00
Siying Dong
0b90aa9515 Set VALGRIND_VER
Summary:
VALGRIND_VER was left empty after moving the environment to GCC-5. Set it back.
Closes https://github.com/facebook/rocksdb/pull/2234

Differential Revision: D4978534

Pulled By: siying

fbshipit-source-id: f0640d58e8f575f75fb3f8b92e686c9e0b6a59bb
2017-05-01 16:35:15 -07:00
Tomas Kolda
04d58970cb AIX and Solaris Sparc Support
Summary:
Replacement of #2147

The change was squashed due to a lot of conflicts.
Closes https://github.com/facebook/rocksdb/pull/2194

Differential Revision: D4929799

Pulled By: siying

fbshipit-source-id: 5cd49c254737a1d5ac13f3c035f128e86524c581
2017-04-21 20:48:04 -07:00
Siying Dong
ca96654d85 Change Build Env to gcc-5
Summary:
Default to build using gcc-5. Only apply to Facebook-only environments.
Closes https://github.com/facebook/rocksdb/pull/2158

Differential Revision: D4887568

Pulled By: siying

fbshipit-source-id: 53496c9af3273ccd44441bd0bef9d29beefbc00b
2017-04-14 11:12:56 -07:00
Sagar Vemuri
6799c7e00d Pass in remote as a param to branch creation script
Summary:
When people are working off of a rocksdb fork, i.e. when their 'origin'
points to github.com/<username>/rocksdb, the script creates a new branch
and pushes to their origin. The new branch created by this script should
instead be pushed to github.com/facebook/rocksdb. Many people might
have named facebook/rocksdb remote as 'upstream' (or something else).
This fix provides an option to specify the remote to push the branch to.
The default is still 'origin'

More context:
When I created 5.4 branch using this script, it got pushed to sagar0/rocksdb instead of facebook/rocksdb, as I was working off of a fork. My 'origin' was pointing to sagar0/rocksdb. My 'upstream' was set to 'facebook/rocksdb'. So, I had to manually push the branch to my 'upstream'.
Closes https://github.com/facebook/rocksdb/pull/2156

Differential Revision: D4885333

Pulled By: sagar0

fbshipit-source-id: 9410eab5bd9bbefc340059800bd6b8434406729d
2017-04-13 16:12:04 -07:00
Dmitri Smirnov
e5a1372b24 Rework test running script.
Summary:
Rework test running script.
  New options SuiteRun - runs specified executables as google suite
  test cases in parallel.
  Run - this option now runs executables in parallel the same as 'tests'
  RunAll - scans for test executables and attempts to run them all
  as suites except those mentiones in RunOnly (hardcoded in the script)
  or specified either in $ExcludeTestCases $ExcludeTestExe
Closes https://github.com/facebook/rocksdb/pull/2089

Differential Revision: D4832212

Pulled By: yiwu-arbug

fbshipit-source-id: 954990c
2017-04-05 11:39:20 -07:00
Siying Dong
88bb6f6bfa non_shm CI should run tests on /tmp
Summary:
Since non_shn CI was made to run in parallel, /dev/shm is automatically used. It defeated the purpose of the test to cover a non-ramfs file system.
Closes https://github.com/facebook/rocksdb/pull/2031

Differential Revision: D4764804

Pulled By: siying

fbshipit-source-id: 5666bda
2017-03-23 15:24:12 -07:00
Raza Hussain
4b04addfce updated solution if "make format" command fails
Summary: Closes https://github.com/facebook/rocksdb/pull/2016

Differential Revision: D4747003

Pulled By: yiwu-arbug

fbshipit-source-id: c1e2c5a
2017-03-21 11:09:10 -07:00
Dmitri Smirnov
c9df05d1e4 Fix random access alignment
Summary:
This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off.
Also adjust slightly appveyor script to run db_basic_test cases concurrently.
Closes https://github.com/facebook/rocksdb/pull/1959

Differential Revision: D4671972

Pulled By: IslamAbdelRahman

fbshipit-source-id: 1886620
2017-03-08 17:09:11 -08:00
Siying Dong
96c7e1504a Fix Java build
Summary:
The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version.
Closes https://github.com/facebook/rocksdb/pull/1912

Differential Revision: D4609854

Pulled By: siying

fbshipit-source-id: 3dc04f2
2017-02-23 18:39:13 -08:00
Sagar Vemuri
eb912a927e Remove disableDataSync option
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859

Differential Revision: D4541292

Pulled By: sagar0

fbshipit-source-id: 5b3a6ca
2017-02-13 11:09:13 -08:00
Rhys Parry
9fc23c55f2 Use gcc-4.9-glibc-2.20-fb python in precommit_checker
Summary:
The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed
soon.
Closes https://github.com/facebook/rocksdb/pull/1839

Differential Revision: D4509684

Pulled By: siying

fbshipit-source-id: 3efe296
2017-02-03 13:39:18 -08:00
Arun Sharma
fba726e555 Version librocksdb.so
Summary:
After make install, I see a directory hierarchy that looks like

```
./usr
./usr/include
./usr/include/rocksdb
./usr/include/rocksdb/filter_policy.h
[..]
./usr/include/rocksdb/iterator.h
./usr/include/rocksdb/utilities
./usr/include/rocksdb/utilities/ldb_cmd_execute_result.h
./usr/include/rocksdb/utilities/lua
./usr/include/rocksdb/utilities/lua/rocks_lua_custom_library.h
./usr/include/rocksdb/utilities/lua/rocks_lua_util.h
./usr/include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h
./usr/include/rocksdb/utilities/backupable_db.h
[..]
./usr/include/rocksdb/utilities/env_registry.h
[..]
./usr/include/rocksdb/env.h
./usr/lib64
./usr/lib64/librocksdb.so.5
./usr/lib64/librocksdb.so.5.0.0
./usr/lib64/librocksdb.so
./usr/lib64/librocksdb.a
```
Closes https://github.com/facebook/rocksdb/pull/1798

Differential Revision: D4456536

Pulled By: yiwu-arbug

fbshipit-source-id: 5494e91
2017-01-24 11:09:10 -08:00
Gunnar Kudrjavets
335981d473 Fix the directory path for RocksDB repo
fbshipit-source-id: 285548d
2016-12-27 22:24:16 -08:00
Yi Wu
fc0c6fd984 "make format" format diff since last commit from master
Summary:
Update clang-format script to format diff since last commit from master,
instead of just last commit. In our common workflow we usually endup
with multiple commits for a single PR. This change make it easier to
format all stacking changes.
Closes https://github.com/facebook/rocksdb/pull/1684

Differential Revision: D4340597

Pulled By: yiwu-arbug

fbshipit-source-id: c18949e
2016-12-16 11:24:18 -08:00
Daniel Black
a8bf4d63f9 Make format shows wrong curl command for retrieving clang-format-diff.py
Summary:
Previously:

$ make format
Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production
build_tools/format-diff.sh
You didn't have clang-format-diff.py available in your computer!
You can download it by running:
    curl http://goo.gl/iUW1u2
Makefile:868: recipe for target 'format' failed
make: *** [format] Error 128

$ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   276    0   276    0     0    148      0 --:--:--  0:00:01 --:--:--   148m

$ more ~/bin/clang-format-diff.py
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>.
</BODY>
</HTML>
Closes https://github.com/facebook/rocksdb/pull/1680

Differential Revision: D4338495

Pulled By: yiwu-arbug

fbshipit-source-id: e2b24d8
2016-12-15 20:24:12 -08:00
Anirban Rahut
aad1191765 pass rocksdb oncall to mysql_mtr_filter otherwise tasks get created w…
Summary:
…rong owner

 mysql_mtr_filter script needs proper oncall
Closes https://github.com/facebook/rocksdb/pull/1586

Differential Revision: D4245150

Pulled By: anirbanr-fb

fbshipit-source-id: fd8577c
2016-11-29 12:09:12 -08:00
Yueh-Hsuan Chiang
647eafdc21 Introduce Lua Extension: RocksLuaCompactionFilter
Summary:
This diff includes an implementation of CompactionFilter that allows
users to write CompactionFilter in Lua.  With this ability, users can
dynamically change compaction filter logic without requiring building
the rocksdb binary and restarting the database.

To compile, WITH_LUA_PATH must be specified to the base directory
of lua.
Closes https://github.com/facebook/rocksdb/pull/1478

Differential Revision: D4150138

Pulled By: yhchiang

fbshipit-source-id: ed84222
2016-11-16 15:39:12 -08:00
Nipunn Koorapati
25f5742f0b Update documentation to point at gcc 4.8
Summary:
Rocksdb currently has many references to std::map.emplace_back()
which is not implemented in gcc 4.7, but valid in gcc 4.8. Confirmed that
it did not build with gcc 4.7, but builds fine with gcc 4.8
Closes https://github.com/facebook/rocksdb/pull/1272

Differential Revision: D4101385

Pulled By: IslamAbdelRahman

fbshipit-source-id: f6af453
2016-10-29 12:09:17 -07:00
Anirban Rahut
2e8004e608 Changing the legocastle run to use valgrind_test instead of _check
Summary:
valgrind_test is the correct way to run valgrind tests.
this is becasue we need to force DISABLE_JEMALLOC

Test Plan: Running sandcastle and contrun

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D65451
2016-10-24 16:23:19 -07:00
sdong
607628d349 Support ZSTD with finalized format
Summary:
ZSTD 1.0.0 is coming. We can finally add a support of ZSTD without worrying about compatibility.
Still keep ZSTDNotFinal for compatibility reason.

Test Plan: Run all tests. Run db_bench with ZSTD version with RocksDB built with ZSTD 1.0 and older.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: cyan, igor, IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D63141
2016-09-06 12:22:16 -07:00
Alex Robinson
b10d65c2a4 Update and slightly clarify instructions in build_detect_platform (#1301) 2016-08-25 10:40:38 -07:00
Yi Wu
99c4af7160 Make ClockCache available with TSAN build
Summary: ..to unblock TSAN contbuild.

Test Plan:
  COMPILE_WITH_TSAN=1 make db_stress -j64
  ./db_stress --use_clock_cache=true

Reviewers: sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D62397
2016-08-23 13:40:47 -07:00
Yi Wu
ff17a2abf3 Adding TBB as dependency.
Summary: Splitting the makefile part of D55581.

Test Plan:
  make all check -j32
  ROCKSDB_FBCODE_BUILD_WITH_481=1 make all check -j32
  ROCKSDB_NO_FBCODE=1 make all check -j32

  export TBB_BASE=/mnt/gvfs/third-party2/tbb/afa54b33cfcf93f1d90a3160cdb894d6d63d5dca/4.0_update2/gcc-4.9-glibc-2.20/e9936bf;
  ROCKSDB_NO_FBCODE=1 CFLAGS="-I $TBB_BASE/include" LDFLAGS="-L $TBB_BASE/lib -Wl,-rpath=$TBB_BASE/lib" make all check -j32

Reviewers: IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: kradhakrishnan, yhchiang, IslamAbdelRahman, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56979
2016-08-18 10:44:29 -07:00
Islam AbdelRahman
4990c0d1a5 Remove deprecated LEVELDB_PLATFORM_POSIX
Summary:
We have replaced LEVELDB_PLATFORM_POSIX with ROCKSDB_PLATFORM_POSIX in our code
replace it in Makefile

Test Plan: make check

Reviewers: sdong, yiwu, andrewkr, lightmark

Reviewed By: lightmark

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61365
2016-08-03 11:07:53 -07:00
krad
c49ea68c55 Fix to enable running CI jobs locally
Summary:
We may not have permission on all /dev/shm to fix the sticky bit.
Making the sticky bit fix advisory.

Test Plan: Run CI job locally

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D61371
2016-08-01 16:31:49 -07:00
Gunnar Kudrjavets
6920cde893 Remove an extra apostrophe 2016-07-27 21:16:11 -07:00
Gunnar Kudrjavets
9c8ac144bd Avoid duplicate task creation for RocksDB contruns
Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure.

Test Plan:
- Land the changes.
- Trigger a RocksDB contrun - observe the results.
- Look at the results from the nightly runs and fix issues as necessary.

Current testing so far has been in isolation for various components.

Reviewers: andrewkr, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60909
2016-07-27 16:37:12 -07:00
Anirban Rahut
d3bfd33972 Testing out parallel sandcastle changes
Summary:
Removing moreutils from sandcastle and adding gnu parallel.
Then passing in J= nproc command

Test Plan: Testing on sandcastle

Reviewers: sdong, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61017
2016-07-27 11:58:57 -07:00
sdong
7efd9c25c4 Increse timeout in some tests
Summary: Increase test timeout to some tests to unblock CI.

Test Plan: watch how it runs.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61263
2016-07-27 11:39:36 -07:00
woonhak.kang
5c858ddd20 fix errata in libnuma test (#1244) 2016-07-26 19:28:45 -07:00
krad
a4955b39ac Run sandcastle tests in /dev/shm 2016-07-25 15:10:24 -07:00
sdong
f85df120f2 Re-enable tsan crash white-box test with reduced killing odds
Summary:
Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds.
Add a parameter in crash test script to allow we pass it through an environment variable.

Test Plan: Run it manually.

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61053
2016-07-22 12:51:27 -07:00
sdong
b50632920d Add unit test not on /dev/shm as part of the pre-commit tests
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system.

Test Plan: See the tests launched by Phabricator

Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60963
2016-07-21 12:53:23 -07:00
sdong
f9b14be495 Re-enable TSAN crash test but only with black box crash test
Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job.

Test Plan: Run it locally.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60951
2016-07-20 13:52:25 -07:00
sdong
68f3eb746c Run release build for CLANG and GCC 4.8.1 in pre-commit tests too
Summary: As Title.

Test Plan: See how the diff works.

Reviewers: kradhakrishnan, andrewkr, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60933
2016-07-20 13:46:32 -07:00
Gunnar Kudrjavets
61dbfbb6ce Add release build to RocksDB per-diff/post-commit tests
Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening.

Test Plan:
- `[p]arc diff --preview`
- Observe the execution in Sandcastle and make sure that release build and tests are executed.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60441
2016-07-08 16:02:29 -07:00
Gunnar Kudrjavets
aa432be4b9 Workarounds for continuous build implementation 2016-06-23 19:38:29 -07:00
Gunnar Kudrjavets
8cd9f04feb Test change to verify new commit detection 2016-06-23 12:08:44 -07:00
Gunnar Kudrjavets
8a4ee7e90b Trivial change to test cont. build 2016-06-23 11:53:38 -07:00
Gunnar Kudrjavets
6576fa05af Fix minor typos and PHP source file name used to trigger the builds
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.

Test Plan: - Testing with a private Sandcastle instance.

Reviewers: sdong, mung

Reviewed By: mung

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59919
2016-06-22 11:05:52 -07:00
Gunnar Kudrjavets
a52e4d7d02 Framework for enabling continuous RocksDB build and tests
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.

The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:

- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.

- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.

- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.

Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.

Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.

Reviewers: sdong, yhchiang, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59811
2016-06-21 11:38:54 -07:00
krad
550bf895e9 Minor bug fix with log name 2016-06-14 12:20:58 -07:00
Islam AbdelRahman
8100ec2cd1 Fix libgcc broken lib path
Summary: current libgcc lib path is broken, update it and make sure the new one exists

Test Plan:
verify that the new path exists
make check -j64

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59535
2016-06-12 00:11:08 -07:00
Islam AbdelRahman
bde7d1055f Fix clang_analyze path in fbcode_config.sh
Summary: current clang_analyze path is broken, update it

Test Plan:
USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make J=64 analyze -j64
make sure it works

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59505
2016-06-10 14:36:38 -07:00
krad
e9c1face6c Minor fix to precommit-check.py 2016-06-10 10:53:02 -07:00
krad
fcc47fa5f1 New features to precommit check script
Summary:
Added new features to help run the entire suite of tests on a diff
locally leveraging parallelism

Test Plan: Basic command line test

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59445
2016-06-10 10:33:22 -07:00
Islam AbdelRahman
edc764e91b Use valgrind built with gcc-4.9-glibc-2.20
Summary:
Right now we use valgrind built with gcc-5-glibc-2.23 which keep generating this error everywhere

```
==3224756== Conditional jump or move depends on uninitialised value(s)
==3224756==    at 0x401AD68: index (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4008933: _dl_map_object (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000ECC: map_doit (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40101C7: _dl_catch_error (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000B34: do_preload (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x400458C: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40185DB: _dl_sysdep_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4004F39: _dl_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000CE7: ??? (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==  Uninitialised value was created by a stack allocation
==3224756==    at 0x4004519: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==
```

Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64

Reviewers: andrewkr, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59343
2016-06-08 13:22:16 -07:00
Islam AbdelRahman
54db29b8f0 Use gvfs links in dependencies.sh
Summary:
Update `build_tools/update_dependencies.sh` to use /mnt/gvfs path in dependencies.sh
This should prevent us from having errors because of changing the path of a compiler or a library

Test Plan:
make all -j64
USE_CLANG make all -j64
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j64

Reviewers: andrewkr, sdong, yiwu

Reviewed By: yiwu

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59295
2016-06-07 14:22:52 -07:00
Yi Wu
10d46b9c8f Update tp2 clang path to fix clang build
Summary: clang was removed from tp2 so change to llvm-fb.

Test Plan:
  USE_CLANG=1 make all -j64

Reviewers: yhchiang, andrewkr, kradhakrishnan, sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59259
2016-06-06 18:29:39 -07:00
Andrew Kryczka
e42dc9192e Update paths for fbcode dependencies
Summary: The path to g++ was outdated. Ran ./build_tools/update_dependencies.sh.

Test Plan: make -j64 all

Reviewers: sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59229
2016-06-06 11:16:51 -07:00
Yi Wu
8f65feafc0 Have sandcastle run lite_test for every diff
Summary: Have sandcastle run unit test in lite mode for every diff.

Test Plan: seems sandcastle picked up changes here and running lite_test for this diff.

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57741
2016-05-06 14:51:20 -07:00
sdong
992a8f83b7 Not enable jemalloc status printing if USE_CLANG=1
Summary: Warning is printed out with USE_CLANG=1 when including jemalloc.h. Disable it in that case.

Test Plan: Run db_bench with USE_CLANG=1 and not. Make sure they can all build and jemalloc status is printed out in the case where USE_CLANG is not set.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57399
2016-04-28 16:16:14 -07:00
Islam AbdelRahman
af70f9ac6d Fix typo in build_tools/fbcode_config.sh
Summary: Fix typo in build_tools/fbcode_config.sh

Test Plan: none

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57363
2016-04-27 21:00:22 -07:00
Islam AbdelRahman
0850bc5147 Fix build on machines without jemalloc
Summary: It looks like we mistakenly enable JEMALLOC even if it's not available on the machine, that's why travis is failing

Test Plan:
check on my devserver
check on my mac

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57345
2016-04-27 18:25:19 -07:00
Sergey Makarenko
1c80dfab24 Print memory allocation counters
Summary:
Introduced option to dump malloc statistics using new option flag.
    Added new command line option to db_bench tool to enable this
    funtionality.
    Also extended build to support environments with/without jemalloc.

Test Plan:
1) Build rocksdb using `make` command. Launch the following command
    `./db_bench --benchmarks=fillrandom --dump_malloc_stats=true
    --num=10000000` end verified that jemalloc dump is present in LOG file.
    2) Build rocksdb using `DISABLE_JEMALLOC=1  make db_bench -j32` and ran
    the same db_bench tool and found the following message in LOG file:
    "Please compile with jemalloc to enable malloc dump".
    3) Also built rocksdb using `make` command on MacOS to verify behavior
    in non-FB environment.
    Also to debug build configuration change temporary changed
    AM_DEFAULT_VERBOSITY = 1 in Makefile to see compiler and build
    tools output. For case 1) -DROCKSDB_JEMALLOC was present in compiler
    command line. For both 2) and 3) this flag was not present.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D57321
2016-04-27 16:23:33 -07:00
dx9
b71c4e613f Alpine Linux Build (#990)
* Musl libc does not provide adaptive mutex. Added feature test for PTHREAD_MUTEX_ADAPTIVE_NP.

* Musl libc does not provide backtrace(3). Added a feature check for backtrace(3).

* Fixed compiler error.

* Musl libc does not implement backtrace(3). Added platform check for libexecinfo.

* Alpine does not appear to support gcc -pg option. By default (gcc has PIE option enabled) it fails with:

gcc: error: -pie and -pg|p|profile are incompatible when linking

When -fno-PIE and -nopie are used it fails with:

/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find gcrt1.o: No such file or directory

Added gcc -pg platform test and output PROFILING_FLAGS accordingly. Replaced pg var in Makefile with PROFILING_FLAGS.

* fix segfault when TEST_IOCTL_FRIENDLY_TMPDIR is undefined and default candidates are not suitable

* use ASSERT_DOUBLE_EQ instead of ASSERT_EQ

* When compiled with ROCKSDB_MALLOC_USABLE_SIZE UniversalCompactionFourPaths and UniversalCompactionSecondPathRatio tests fail due to premature memtable flushes on systems with 16-byte alignment. Arena runs out of block space before GenerateNewFile() completes.

Increased options.write_buffer_size.
2016-04-22 16:49:12 -07:00
Andrew Kryczka
ec84bef24a New legocastle output parsing
Summary:
Added a python script to parse combined stdout/stderr of legocastle
steps. Previously we just matched words like 'Failure', which didn't work since
even our test names matched that pattern.

I went through all the legocastle steps to come up with strict failure regexes
for the common failure cases. There is also some more complex logic to present
gtest failures, since the test name and failure message are not on the same
line.

There will definitely be error cases that don't match any of these patterns, so
we can iterate on it over time.

Test Plan:
no end-to-end test. I ran the legocastle steps locally and piped to
my script, then verified output, e.g.,

  $ set -o pipefail && TEST_TMPDIR=/dev/shm/rocksdb COMPILE_WITH_ASAN=1 OPT=-g make J=1 asan_check |& /usr/facebook/ops/scripts/asan_symbolize.py -d |& python build_tools/error_filter.py asan
  ==2058029==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000a414 at pc 0x4c12f6 bp 0x7ffcfb7a0520 sp 0x7ffcfb7a0518

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56691
2016-04-18 13:01:10 -07:00
krad
083cadc7e9 Minor fix to Java sandcastle job definition
Summary:
proxy config tools are not available in sandcastle environment. Hard
coding the proxy information in the job.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56775
2016-04-14 15:05:54 -07:00
krad
7c14d11eb9 Minor fix to java build job
Summary:
Java build is downloading junit when not available. Without the proxy
settings exported that would not work.

Test Plan: Run manually

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56751
2016-04-14 10:46:02 -07:00
krad
09be5cad5b Minor fix to sandcastle java job definition
Summary: System was not able to locate javac

Test Plan: Run command manually and observe output

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56709
2016-04-13 15:59:05 -07:00
krad
c2c8fe47f0 Add Java job for sandcastle
Summary: Adding java build job definition to sandcastle.

Test Plan: ./build_tools/rocksdb-lego-determinator java_build

Reviewers: yhchiang, sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56679
2016-04-13 11:40:51 -07:00
Gunnar Kudrjavets
994b3bd693 Add support for UBsan builds to RocksDB
Summary:
Undefined Behavior Sanitizer (ubsan) is //a good thing// which will help us to find sneaky bugs with low cost. Please see http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ for more details and official GCC documentation for more context: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html.

Changes itself are quite simple and pretty much imitating whatever is implemented for ASan.

Hooking the UBsan validation build to Sandcastle is a separate step and will be dealt as separate diff because code is in internal repository.

Test Plan: Make sure that that there no regressions when it comes to builds and test pass rate.

Reviewers: leveldb, sdong, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56049
2016-03-30 15:59:24 -07:00
sdong
07bb12d97d Update internal jemalloc and other versions
Summary: Update jemalloc to newer versions, as well as some other dependencies. Only effective for FB internal.

Test Plan: See all tests run

Reviewers: IslamAbdelRahman, MarkCallaghan

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D55947
2016-03-25 10:42:20 -07:00
Dmitri Smirnov
e7cc49cbdf Add support for db_test2 for dev and CI runs 2016-03-23 16:33:57 -07:00
Islam AbdelRahman
171c8e80b1 Update dependencies / Fix Clang
Summary:
Update dependencies and fix broken clang symlink

version changes
```
zstd 0.4.7 => 0.5.1
jemalloc 4.0.3 => master
```

Test Plan:
make check -j64
USE_CLANG=1 make check -j64
ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64

Reviewers: sdong, yhchiang, anthony, andrewkr

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D55179
2016-03-07 12:57:19 -08:00
root
3373c81fa8 Modify build_tools/build_detect_platform to detect and set -march=z10 on Linux s390x. 2016-02-29 15:02:52 -05:00
root
21f17aaa60 Modified Makefile and build_tools/build_detect_platform to compile on Linux s390x. 2016-02-26 18:03:07 -05:00
Andrew Kryczka
6a2b4fcb80 Add flag to forcibly disable fallocate
Summary:
see https://github.com/facebook/rocksdb/issues/977; there are issues
with fallocate() on certain filesystems/kernel versions that can lead it to pre-
allocating blocks but never freeing them, even if they're unused.

Test Plan:
verified build commands omit DROCKSDB_FALLOCATE_PRESENT when this env
variable is set.

without disabling it:

  $ ROCKSDB_NO_FBCODE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  0

with disabling it:

  $ ROCKSDB_NO_FBCODE=1 DISABLE_FALLOCATE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  1

Reviewers: kradhakrishnan, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54069
2016-02-11 17:00:01 -08:00
Baraa Hamodi
21e95811d1 Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
Islam AbdelRahman
77ef87ccb0 Update fbcode_config4.8.1.sh to use update_dependencies.sh
Summary:
This patch update fbcode_config4.8.1.sh to get it's dependencies the same way we updated fbcode_config.sh in D53037

as a result zstd is upgraded to 0.4.7 instead of 0.4.5

Test Plan:
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64
make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make check -j64

Reviewers: yhchiang, andrewkr, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53355
2016-01-26 12:13:27 -08:00
Islam AbdelRahman
f7ef1a6132 Include rest of dependencies in dependencies.sh
Summary:
This diff

  - Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh
  - updtade zst to 0.4.7
  - It also fix a problem in clang scan build

Test Plan:
make check
USE_CLANG=1 make check
USE_CLANG=1 make analyze
coverage_test.sh

Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53301
2016-01-25 14:35:04 -08:00
Islam AbdelRahman
538eec0661 Update fbcode_config.sh to use latest versions automatically
Summary:
Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated

```
Snappy: 1.0.3 => 1.1.3
GFLAGS: 1.6 => 2.1.1
JEMALLOC: 3.6.0 => 4.0.3
```

I have also updated clang from 3.7 to 3.7.1

```
Clang 3.7 => 3.7.1
```

Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library

Test Plan:
make check -j64
USE_CLANG=1 make check -j64

DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running)

Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53037
2016-01-22 11:56:32 -08:00
krad
b0a15e7fb9 Mechanism to run CI jobs on local branch via commit_prereq
Summary: This patch provides a mechanism to run pre commit tests on the local
branch before committing. This can help prevent frequent build breaks.

The tests can be run in parallel by specifying the J=<..> environment
variable.

Test Plan: Run manually

Reviewers: sdong rven tec

CC: leveldb@

Task ID: #9689218

Blame Rev:
2016-01-21 16:22:21 -08:00
bcbrock
f423f05dcd Simple changes to support builds for ppc64[le] consistent with X86
These simple changes are required to allow builds on ppc64[le] systems
consistent with X86. The Makefile now recognizes both ppc64 and ppc64le, and
in the absence of PORTABLE=1, the code will be built analogously to the X86
-march=native.

Note that although GCC supports -mcpu=native -mtune=native on POWER, it
doesn't work correctly on all systems. This is why we need to get the actual
machine model from the AUX vector.
2016-01-19 09:08:19 -06:00
agiardullo
83e1de92af move internal build to use zstd 0.4.5
Summary: update internal build scripts to use zstd 0.4.5.

Test Plan: built and ran tests with and without ROCKSDB_FBCODE_BUILD_WITH_481

Reviewers: yhchiang, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52833
2016-01-15 13:27:35 -08:00
Mark Callaghan
4041903ecd Enhance db_bench write rate limit
Summary:
1) changes tools/{benchmark,run_flash_bench}.sh to optionally use the write rate limit
2) removes code for --writes_per_second and switches the 'background' write rate limit
to use --benchmark_write_rate_limit

Replaces https://reviews.facebook.net/D49113

Task ID: #9555881

Blame Rev:

Test Plan:
tools/run_flash_bench.sh

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D52485
2016-01-04 12:01:27 -08:00
sdong
d8677a8d2c Upgrade internal CLANG version for FB-internal gcc 4.8.1
Summary: After removing two move operations, we can make CLANG 3.7 build pass under GCC 4.8.1.

Test Plan: USE_CLANG=1 ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32

Reviewers: yhchiang, IslamAbdelRahman, rven, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52365
2015-12-29 10:33:23 -08:00
Andrew Kryczka
9c176ef90d Update liblz4 to r131
Summary:
It was already built in third-party2 but the include/library paths in
rocksdb hadn't been updated accordingly.

Test Plan:
verified build works

  $ make clean && make -j32 all
  $ make clean && USE_CLANG=1 make -j32 all

Reviewers: cyan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52299
2015-12-23 17:13:31 -08:00
krad
362d819a14 Improving parser
Summary: Improving the parser string to make better error report. Currently the
error report fails to capture the assert details. This fix addresses the issue.

Test Plan: None

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-12-11 11:06:42 -08:00
charsyam
c30b499541 fix typos in comments 2015-12-11 01:54:48 +09:00
sdong
758dbec7f0 Fix fb-only build for gcc 4.8.1
Summary: fb build for gcc 4.8.1 is broken for the ZSTD dependency is not fixed after ea11923550. Fixing it.

Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make -j40 OPT=-g

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51741
2015-12-08 17:40:59 -08:00
sdong
ea11923550 Upgrade to ZSTD 0.4.2
Summary: Change to call the new compression function.

Test Plan: build and run db_bench with the compression to make sure it compresses.

Reviewers: anthony, rven, kradhakrishnan, IslamAbdelRahman, igor, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51603
2015-12-08 16:33:26 -08:00
sdong
d5239f8709 build_tools/fbcode_config4.8.1.sh: upgrade versions of some dependencies
Summary: Upgrade version of some dependencies in build_tools/fbcode_config4.8.1.sh. I didn't upgrade version for CLANG because some warnings are shown.

Test Plan:
build:

ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make all -j32
as well as
ROCKSDB_FBCODE_BUILD_WITH_481=1 make all -j32

Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, igor

Reviewed By: igor

Subscribers: igor, kradhakrishnan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D51003
2015-11-18 16:18:11 -08:00
Dmitri Smirnov
7cb1293b60 Fix log names when scheduling runs and reruns 2015-11-17 18:41:29 -08:00
Dmitri Smirnov
94e39e2364 Exclude DBTest.FileCreationRandomFailure as a long running test
Increase concurrency to 18
  Fix exclusion but in the ps script
2015-11-17 13:54:13 -08:00
Dmitri Smirnov
2a0510c9f4 Failed tests must be rerun with concurrency 1 2015-11-17 11:23:09 -08:00
yuslepukhin
935d1495c5 Run tests imporvements
Add sequential rerun for any failed tests. Add env_test case.
  Limit concurrency
  Allow to specify individual tests
  Take $Limit into account when displaying number of tests
2015-11-12 14:42:38 -08:00
Nathan Bronson
986230b8c8 Revert "Fix TSAN build for fbcode"
Summary:
Reverting c745f1d2c4 because it
was based on an incorrect understanding of the correct way to enable
TSAN tests (it assumes "make COMPILE_WITH_TSAN=1 check" but in fact only
"COMPILE_WITH_TSAN=1 make check" is supported).

Test Plan: COMPILE_WITH_TSAN=1 make check

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50445
2015-11-09 11:51:10 -08:00
Nathan Bronson
c745f1d2c4 Fix TSAN build for fbcode
Summary:
TSAN builds for gcc 4.9 need a PIC version of the libraries
taken from the fbcode platform.  This is accomplished by assuming every
.a has a _pic.a sibling, and by fixing the third-party2 zlib build.

Test Plan: make COMPILE_WITH_TSAN=1 check

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D50331
2015-11-06 09:05:12 -08:00
Dmitri Smirnov
3277d172ba Improve concurrency when running tests
PowerShell seems to have a hard time when a flood of async tasks is
  scheduled at the same time.
  I speculated that WaitForMultipleObjects() in Windows can only take up
  to 64 process handles and if you want to handle more than you should write
  some additional code which can be sub-optimal. I.e to implement Wait-Job -Any.
  I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62.
  So in the new version the script fires up up to $Concurrency value
  and wait for anything to complete before starting any more processes.
  This improved matters greatly. Individual tests against ramdrive now
  run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency
  values of 8-16. About 48 is required to load a CPU on my box running against HD
  but that does not improve running times much.

  Other changes include respect -EnableJE for the individual test exes.
  Enforce exclusions for the individual tests.
2015-11-05 14:03:41 -08:00
Igor Canadi
14c6e1a04c Add write_stress to RocksDB Legocastle runs
Summary: As title. Let's run it for 1 hour.

Test Plan: How can I test legocastle changes?

Reviewers: IslamAbdelRahman, yhchiang, rven, sdong, anthony, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D49653
2015-11-05 12:07:39 -08:00
Islam AbdelRahman
581f20fd8b Add LITE tests to Legocastle
Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE

Test Plan: will be tested after landing in fbcode

Reviewers: sdong, yhchiang, igor, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D49065
2015-10-26 11:50:29 -07:00
Javier González
6e6dd5f6f9 Split posix storage backend into Env and library
Summary: This patch splits the posix storage backend into Env and
the actual *File implementations. The motivation is to allow other Envs
to use posix as a library. This enables a storage backend different from
posix to split its secondary storage between a normal file system
partition managed by posix, and it own media.

Test Plan: No new functionality is added to posix Env or the library,
thus the current tests should suffice.
2015-10-22 17:31:31 +02:00
krad
7717ad1afe Adding artifacts to stress_crash CI job
Summary: Adding the ability to upload logs and db content to storage after the
completion of the job

Test Plan: Manual run

Reviewers:

CC: leveldb@

Task ID: #8754201

Blame Rev:
2015-10-19 20:13:14 -07:00
krad
bb64d6da46 Disabling TSAN crash test
Summary: The test is failing with OOM. Temporarily disabled.

Test Plan: Manual check

Reviewers:

CC: leveldb@

Task ID: #8732029

Blame Rev:
2015-10-15 13:42:34 -07:00
Yueh-Hsuan Chiang
c4366165e7 Merge pull request #759 from jwlent55/statically-load-compression-libraries
Ensure that the compression libraries are statically linked into dyna…
2015-10-09 20:03:39 -07:00
krad
64546af83e Adding parser to CI jobs
Summary: The parser will help parse the output in order to send meaningful
notifications.

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-09 17:58:31 -07:00
James Lent
5a7222782a Ensure that the compression libraries are statically linked into dynamic libraries included
in the Java jar.  Also build the linux libraries using the portable flag to fix a problem with
the linux32 build and improve the general portability of the RocksDB dynamic libraries.
==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
2015-10-09 14:41:40 -04:00
krad
77318ee14b Enable crash CI jobs
Summary: Enabling CI crash test jobs

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #8635140

Blame Rev:
2015-10-08 10:14:43 -07:00
Igor Canadi
f925208ffc Create Makefile target unity_test
Summary: unity_test will compile db_test with unity.a instead of librocksdb.a. This will test both the compilation and some small amount of runtime.

Test Plan: This is a test :)

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48297
2015-10-07 14:46:18 -07:00
krad
726d9ce1a6 Disabling unity
Summary: Unity CI job is broken. Disabling.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #8632483

Blame Rev:
2015-10-06 17:51:31 -07:00
krad
60b1c0a701 Fix to CI job definition
Summary: Sandcastle does not raise a task if assignee is specified. Removing the
assignee will force the system to use the oncall.

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-06 14:21:10 -07:00
krad
a39897369a Adding features to CI job description
Summary: Added following capability
(1) Ability to do configurable reports using args
(2) Support to generating a task on failure
(3) Ability to configure email trigger condition

Test Plan: Manual testing

Reviewers:

CC:leveldb@

Task ID: #6968635

Blame Rev:
2015-10-02 12:36:58 -07:00
Igor Canadi
a47bf325c6 Merge pull request #748 from yuslepukhin/improve_test_runs
Improve test running scripts
2015-10-02 11:42:53 -07:00
Dmitri Smirnov
65324a16dc Improve test running scripts
Introduce proper command line arguments so we can control the script
  Add appveyor support
  Add an ability to run all other (non db_tests) test executables in parallel
  Use .NET HashSet instead of empty valued hashtable to improve the looks
  TODO: Some of the tests do not use GTests and need to improve log parsing
2015-10-02 11:33:35 -07:00
Evan Shaw
7a23e4d8ca New amalgamation target
This commit adds two new targets to the Makefile: rocksdb.cc and rocksdb.h

These files, when combined with the c.h header, are a self-contained RocksDB
source distribution called an amalgamation. (The name comes from SQLite's, which
is similar in concept.)

The main benefit of an amalgamation is that it's very easy to drop into a
new project. It also compiles faster compared to compiling individual source
files and potentially gives the compiler more opportunity to make optimizations
since it can see all functions at once.

rocksdb.cc and rocksdb.h are generated by a new script, amalgamate.py.
A detailed description of how amalgamate.py works is in a comment at the top of
the file.

There are also some small changes to existing files to enable the amalgamation:
* Use quotes for includes in unity build
* Fix an old header inclusion in util/xfunc.cc
* Move some includes outside ifdef in util/env_hdfs.cc
* Separate out tool sources in Makefile so they won't be included in unity.cc
* Unity build now produces a static library

Closes #733
2015-10-01 08:29:31 +13:00
Igor Canadi
75134f7562 Merge pull request #741 from yuslepukhin/test_appveyor_baseline
Improve CI build and build switches
2015-09-29 13:16:40 -07:00
Dmitri Smirnov
2e7506d82c Improve CI build and build switches
Add an optimized build config switch for faster test runs
  Change compiler options to introduce more opitmizations and be more inline with MS internal switches.
  Make appveyor build to utilize all the avaiable cores on the VM (parallel)
  Introduce new appveyor configuration for daily test runs as it would take too long
  to run db_test after each checkin even in paralell.
  With some exclusions we make it in 38 minutes. We currently fail to install ramdisk during the build.
  Add a powershell script to faicilitate paralell run for db_test cases.
2015-09-29 12:22:48 -07:00
krad
7df348b405 Minor fix to CI job definition
Summary: Added email notification for a few jobs and fixed a bug in
no_compression script

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-25 14:54:51 -07:00
krad
e01f32cc3c Parameterizing email id
Summary: Removing the leveldb@fb.com and making it a command line argument

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-24 15:18:36 -07:00
krad
628216fc1f Simplifying valgrind testing
Summary: Our valgrind testing is buggy and cumbersome in terms of locating the
error. It originates from the fact we accumulate all output for the tests. It is
extremely hard to locate the point of error.

The communication between valgrind and the script is not sturdy. We are
experiencing bugs.

Simplifying to stop on first error.

Test Plan: Run manually

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-23 15:08:58 -07:00
krad
03dd8f3ca7 Fixing punit job description
Summary: The job errors with log print step. Removing the step and re-enabling
the test case.

Test Plan: Manual test run

Reviewers:

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-23 07:35:29 -07:00
krad
a6c22e3e41 Disabling parallel test CI job
Summary: Parallel test CI is failing. Reducing noise by temporarily disabling
the job

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-22 10:37:56 -07:00
krad
dd2e1eeb35 Disabling log running jobs
Summary: Long running jobs are blocking the queue due to lack of adequate
machines in the CI

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-21 12:58:33 -07:00
krad
4d6eb52d1a Fix to CI jobs.
Summary: The approach to serialize long scripts as string is flawed. Instead
adding args to invoke the long scripts.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-19 17:33:06 -07:00
krad
3bcc072d24 Added more CI jobs
Summary: Migrating the long script jobs and the left overs from Jenkin.

Test Plan: Run command manually

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-18 13:24:09 -07:00
Andres Noetzli
1b598213aa Check formatting in Travis
Summary:
Check code formatting before running tests. Goal is to make it easier to deal
with external pull requests.

Test Plan: See https://travis-ci.org/facebook/rocksdb/builds/80952963

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47211
2015-09-17 22:02:05 -07:00
krad
2819a1db32 Minor fix to CI job definition
Summary: Timeout needs to be set for command and job for it to work.

Test Plan: Manual

Reviewers: sdong igor

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-15 10:56:48 -07:00
Andres Noetzli
0bfe0573e8 Add gflags dependency to Travis script
Summary:
Travis is failing due to missing gflags
(https://travis-ci.org/facebook/rocksdb/jobs/80307921). Adding libgflags-dev
should help.

Test Plan: Run tests on Travis.

Reviewers: rven, anthony, yhchiang, aekmekji, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46905
2015-09-14 14:30:17 -07:00
krad
e2d6011f47 Minor fix to CI job definition
Summary: Command level timeouts are not working. Moving it to step level.

Test Plan: Manual testing sending commands

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-14 12:10:18 -07:00
krad
9f3a66a936 Improvements to CI jobs
Summary: Fixed
- Added timeouts for crash tests. They take around 9hrs.
- Added oncall so we can get notifications for timeout etc.
- Fixed a bug in the valgrind script
- Cosmetic fix

Test Plan: Manual run

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-11 16:09:01 -07:00
krad
7db1471cc1 Minor fix to CI job
Summary: Fix the issue where compilation error will not result in log file not
found error

Test Plan: None

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-11 13:00:55 -07:00
krad
41bce05869 CI job improvements
Summary: Added more jobs and refactored code express the jobs more cleanly

Test Plan: Manual test

Reviewers: igor sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-10 14:29:41 -07:00
krad
52386a1e73 Minor fix to sandcastle jobs commands
Summary: Dropping parallel compilation since that seen to introduce compilation
errors spuriously

Test Plan: Run the command manually

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-09 10:54:50 -07:00
krad
778cf4449b Adding email notification.
Summary: The email notifications needs to be at command level and the job level.
Adding command level notification.

Test Plan: Run command manually

Reviewers: igor sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-08 14:19:03 -07:00
krad
20ef64cae0 Moving jobs to use gcc-4.9
Summary: Fixed the glitch in Sandcastle and added gcc-4.9 support. Moving the
jobs to 4.9

Test Plan: Manually run the script

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-03 15:20:36 -07:00
krad
b8a962d4f5 Adding commands for few more CI jobs.
Summary: Added commands to spawn crash, stress, asan, asan_crash, clang CI jobs

Test Plan: Manual test

Reviewers: sdong

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-02 14:06:03 -07:00
krad
c6d870ffb0 Merge branch 'arcpatch-D45741' 2015-08-31 12:42:56 -07:00
krad
57b3a87731 Adding sandcastle determinator for RocksDB
Summary: This fuels commands to be executed for different sandcastle jobs. This is a nice way to separate RocksDB specific commands from Sandcastle specific job definition.

Test Plan: None. Will be tested when we add Sandcastle job.

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45741
2015-08-31 12:37:26 -07:00
sdong
7a0dbdf3ac Add ZSTD (not final format) compression type
Summary: Add ZSTD compression type. The same way as adding LZ4.

Test Plan: run all tests. Generate files in db_bench. Make sure reads succeed. But the SST files cannot be opened in older versions. Also some other adhoc tests.

Reviewers: rven, anthony, IslamAbdelRahman, kradhakrishnan, igor

Reviewed By: igor

Subscribers: MarkCallaghan, maykov, yoshinorim, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45747
2015-08-28 11:01:13 -07:00
Yoshinori Matsunobu
18ba58a943 Upgrading jemalloc from 3.6.0 to the latest for fbcode+gcc 4.8.1
Summary:
MyRocks is using jemalloc latest version, not 3.6.0.
Combining multiple versions (3.6.0 in RocksDB and latest in MyRocks)
broke some features -- for example, getting SIGSEGV when heap profiling
was enabled.
This diff switches to use jemalloc latest, if
env variable ROCKSDB_FBCODE_BUILD_WITH_481=1 was set.
My understanding is this env was used by MyRocks only so it would be
safe to change.

Test Plan: building MyRocks then verified jemalloc heap profiling worked

Reviewers: igor, rven, yhchiang, jtolmer, maykov, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D43479
2015-08-04 16:35:26 -07:00
Igor Canadi
436ed904da Add rpath option to production builds for 4.8.1 toolchain
Summary: Copy change from D37533 to gcc 4.8.1 config

Test Plan: make db_bench, `ldd db_bench`, try running it

Reviewers: MarkCallaghan, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40845
2015-06-30 13:30:54 -07:00
Igor Canadi
09f5a4b486 set -e in fb_compile_mongo.sh
Summary: Based on @anthony's feedback, we want to fail early if our static linking fails.

Test Plan: none

Reviewers: anthony

Reviewed By: anthony

Subscribers: dhruba, anthony, leveldb

Differential Revision: https://reviews.facebook.net/D40839
2015-06-29 11:43:25 -07:00
Igor Canadi
0a019d74a0 Use malloc_usable_size() for accounting block cache size
Summary:
Currently, when we insert something into block cache, we say that the block cache capacity decreased by the size of the block. However, size of the block might be less than the actual memory used by this object. For example, 4.5KB block will actually use 8KB of memory. So even if we configure block cache to 10GB, our actually memory usage of block cache will be 20GB!

This problem showed up a lot in testing and just recently also showed up in MongoRocks production where we were using 30GB more memory than expected.

This diff will fix the problem. Instead of counting the block size, we will count memory used by the block. That way, a block cache configured to be 10GB will actually use only 10GB of memory.

I'm using non-portable function and I couldn't find info on portability on Google. However, it seems to work on Linux, which will cover majority of our use-cases.

Test Plan:
1. fill up mongo instance with 80GB of data
2. restart mongo with block cache size configured to 10GB
3. do a table scan in mongo
4. memory usage before the diff: 12GB. memory usage after the diff: 10.5GB

Reviewers: sdong, MarkCallaghan, rven, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40635
2015-06-26 11:48:09 -07:00
Igor Canadi
472e64d39e Improve fb_compile_mongo.sh
Summary: If we create a new temp directory for each build, scons will recompile everything because we have different parameters. Instead, let's set up a constant path to our static lib. That way we won't have to recompile.

Test Plan: Run fb_compile_mongo.sh twice -- second time it didn't recompile everything

Reviewers: MarkCallaghan, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40707
2015-06-26 10:24:08 -07:00
Igor Canadi
dadc429767 Reproducible MongoRocks compile with FB toolchain
Summary:
Added a script that will compile MongoRocks with the same flags as RocksDB binary. On FB infra, we can now do:

  cd ~/rocksdb; make static_lib
  cd ~/mongo; ~/rocksdb/build_tools/fb_compile_mongo.sh

No need to upgrade the g++ on the devbox (like Aaron and I did) or maintain a separate script to compile (like Mark did)

fb_compile_mongo.sh gets the settings from fbcode_config.sh, so it also makes it easier to upgrade the environment one day.

Test Plan: Compiled mongod with new script. Also, ldd output looks good: https://phabricator.fb.com/P19891602

Reviewers: AaronFeldman, MarkCallaghan, anthony

Reviewed By: anthony

Subscribers: anthony, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40659
2015-06-24 15:09:55 -07:00
Yueh-Hsuan Chiang
0b1ffe2e1d Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS
Summary:
Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS as
those environment doe not have compile option --no-as-needed

  ld: unknown option: --no-as-needed
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

Test Plan: make shared_lib

Reviewers: meyering, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40353
2015-06-23 16:32:59 -07:00
Igor Canadi
c89369f57c Move dockerbuild.sh to build_tools/
Summary: That's where we keep build tools :)

Test Plan: none

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D39741
2015-06-17 14:09:12 -07:00
sdong
46296cc869 Cygwin build not to use -fPIC
Summary:
Cygwin doesn't support -fPIC. Remove it.
Not sure whether we can build shared library in Cygwin but at least it can build without warning.

Test Plan: Build under Cygwin

Reviewers: yhchiang, rven, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D40077
2015-06-12 13:59:59 -07:00
sdong
98a44559d5 Build for CYGWIN
Summary:
Make it build for CYGWIN.
Need to define "-std=gnu++11" instead of "-std=c++11" and use some replacement functions.

Test Plan: Build it and run some unit tests in CYGWIN

Reviewers: yhchiang, rven, anthony, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D37605
2015-04-23 21:33:44 -07:00
Igor Canadi
fd7a357318 Enable open source users to use jemalloc (github issue #438)
Summary: Currently open source rocksdb only builds with tcmalloc. This diff first checks if jemalloc is available. If it is, it compiles with jemalloc. If it isn't, it checks for tcmalloc.

Test Plan: Tried this out on my Ubuntu virtual machine and confirms that jemalloc is correctly detected and compiled.

Reviewers: MarkCallaghan, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: adamretter, meyering, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36789
2015-04-23 17:48:18 -07:00
Mark Callaghan
6e359419fe Add rpath for production builds
Summary:
This lets the production toolchain libraries get used on devservers and
in production.

Task ID: #6849362

Blame Rev:

Test Plan:
Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37533
2015-04-22 17:17:50 -07:00
Igor Canadi
6a5ffee0cc Fix gflags Makefile
Summary: `echo` correctly interpretes \n on mac, but not on linux. On linux you have to give it `-e` to interpret \n. Unfortunately, `-e` options is not available on Mac. Go back to old way of checking gflags

Test Plan: build_tools/build_detect_platform on mac and linux

Reviewers: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37515
2015-04-22 12:50:28 -07:00
Igor Canadi
7d136994c9 Get rid of error output
Summary: We should send error output to /dev/null

Test Plan: none

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37449
2015-04-20 19:44:32 -07:00
Igor Canadi
742fa9e316 Fix compile with two gflags
Summary:
If the system has gflags with both `google` and `gflags` namespaces installed, we try to define GFLAGS as two things. This breaks the compile.

Fix: Use `else if` -- try compiling with `google` namespace only if compile with `gflags` failed

Test Plan: build_tools/build_detect_platform correctly identifies gflags

Reviewers: lgalanis

Reviewed By: lgalanis

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37389
2015-04-20 10:55:17 -07:00
Igor Canadi
98ef21d2ff Merge pull request #584 from pshareghi/rocksdb-3.10-falloch
Added falloc.h in build_detect_platform
2015-04-15 14:56:36 -07:00
Pooya Shareghi
e8808b9128 Added falloc.h in build_detect_platform
On Centos 6, you need to explicitely include linux/falloc.h which is
whele the  FALLOC_FL_* flags are defined. Otherwise, the fallocate()
support test defined in build_detect_platform will fail.

Signed-off-by: Pooya Shareghi <shareghi@gmail.com>
2015-04-13 17:56:12 -07:00
Igor Canadi
91df4e969d Remove use of whole-archive to include jemalloc
Summary: I don't think we need to use whole-archive to include jemalloc. This change only affects our development builds -- it does not affect our open source builds (which don't support jemalloc) or our fbcode third-party2 builds (which use open-source build codepaths).

Test Plan:
make
verify that jemalloc is running by running `MALLOC_CONF="prof:true" ./cache_test` and observing that file was created

Reviewers: MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36783
2015-04-09 15:10:53 -07:00
Igor Canadi
c66483c132 Fix github issue #563
Summary:
As described in https://github.com/facebook/rocksdb/issues/563, we should add minor version to SONAME, since we break ABI with minor releases.

I also turned PLATFORM_SHARED_VERSIONED to true by default. This is true in LevelDB and it was switched to false by D15117 for no apparent reason. It should only be false for iOS.

Test Plan: `make shared_lib` produced librocksdb.dylib.3.10.0

Reviewers: sdong, yhchiang, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36573
2015-04-07 13:22:22 -07:00
Igor Canadi
de22c7bd1f Integrate Jenkins with Phabricator
Summary:
After this diff, when a user submits a diff from Facebook's VPN
network, we'll automatically trigger a jenkins test. Once jenkins test
is done, we'll update the diff with test results.

Test Plan:
Made sure that jenkins build is triggered on `arc diff` and
that result is reflected back on the diff

Reviewers: sdong, rven, kradhakrishnan, anthony, yhchiang

Reviewed By: anthony

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36555
2015-04-07 11:56:29 -07:00
Igor Canadi
df71c6b9ed Script to trigger jenkins test
Summary: After you run `arc diff`, just run `build_tools/trigger_jenkins_test.sh` and Jenkins will test your diff!

Test Plan: Triggered a build to jenkins

Reviewers: sdong, rven, IslamAbdelRahman, anthony, yhchiang, meyering

Reviewed By: meyering

Subscribers: meyering, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36021
2015-03-31 08:56:40 -07:00
Igor Sugak
67d995808a rocksdb: fix make unity
Summary:
changed make unity target to use $LIB_SOURCES as a source of library source code. In the old way (using find) table/mock_table.h was added to a list of library objects and this was a course of `make unity` break.

`build_tools/unity` contains some redundant code, I deleted it and moved the functionality in the Makefile.

Test Plan:
Make sure unity completes with no errors.
```lang=bash
% make unity
```

Reviewers: sdong, rven, igor, meyering

Reviewed By: igor, meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35385
2015-03-18 19:12:23 -07:00
Igor Sugak
bd4963e64d rocksdb: update reference to the latest version of clang dev in fbcode
Summary: Updated reference to the latest clang dev in fbcode. Since RocksDB already uses recent version of clang dev no code changes need.

Test Plan:
Make sure can be build with clang.
```lang=bash
% USE_CLANG=1 make all
```

Reviewers: meyering, sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D35121
2015-03-16 17:08:05 -07:00
Venkatesh Radhakrishnan
480b284766 Fix make_new_version.sh
Summary:
One of the commands in the script make_new_version.sh was
incorrect. Fixed it.

Test Plan: Try the script out to verify that it works.

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D34791
2015-03-10 17:24:33 -07:00
Jim Meyering
ebc647de87 build: fix missing dependency problems
Summary:
Any time one would modify a dependent of any *test*.cc file,
"make" would fail to rebuild the affected test binaries,
e.g., db_test.  That was due to the fact that we deliberately
excluded those test-related files from the definition of SOURCES
and only $(SOURCES) was used to create the automatically-generated
.d dependency files.  The fix is to generate a .d file for every
source file.
* src.mk: New file.  Defines LIB_SOURCES, MOCK_SOURCES
and TEST_BENCH_SOURCES.
* Makefile: Include src.mk.
Reflect s/SOURCES/LIB_SOURCES/ renaming.
* build_tools/build_detect_platform: Remove the code
that was used to generate SOURCES= and MOCK_SOURCES=
definitions in make_config.mk. Those lists of files
are now hard-coded in src.mk. Hard-coding this list of
sources is desirable, because without that, one risks
including stray .cc files in a build.  Not reproducible.

Test Plan:
Touch a file used by db_test's dependent .o files and ensure that
they are all recompiled.  Before, none would be:

  $ touch db/db_impl.h && make db_test
    CC       db/db_test.o
    CC       db/column_family.o
    CC       db/db_filesnapshot.o
    CC       db/db_impl.o
    CC       db/db_impl_debug.o
    CC       db/db_impl_readonly.o
    CC       db/forward_iterator.o
    CC       db/internal_stats.o
    CC       db/managed_iterator.o
    CC       db/repair.o
    CC       db/write_batch.o
    CC       utilities/compacted_db/compacted_db_impl.o
    CC       utilities/ttl/db_ttl_impl.o
    CC       util/ldb_cmd.o
    CC       util/ldb_tool.o
    CC       util/sst_dump_tool.o
    CC       util/xfunc.o
    CCLD     db_test

Reviewers: ljin, igor.sugak, igor, rven, sdong

Reviewed By: sdong

Subscribers: yhchiang, adamretter, fyrz, dhruba

Differential Revision: https://reviews.facebook.net/D33849
2015-03-06 10:55:11 -08:00
Igor Canadi
f9c14a42e8 Fix compile on Mac 2015-02-27 09:31:29 -08:00
Islam AbdelRahman
ba9d1737a8 RocksDB on FreeBSD support
Summary:
This patch will update the Makefile and source code so that we can build RocksDB successfully on FreeBSD 10 and 11 (64-bit and 32-bit)
I have also encountered some problems when running tests on FreeBSD, I will try to fix them individually in different diffs

Notes:

  - FreeBSD uses clang as it's default compiler (http://lists.freebsd.org/pipermail/freebsd-current/2012-September/036480.html)
  - GNU C++ compiler have C++ 11 problems on FreeBSD (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193528)
  - make is not gmake on FreeBSD (http://www.khmere.com/freebsd_book/html/ch01.html)

Test Plan:
Using VMWare Fusion Create 4 VM machines (FreeBSD 11 64-bit, FreeBSD 11 32-bit, FreeBSD 10 64-bit, FreeBSD 10 32-bit)

  - pkg install git gmake gflags archivers/snappy
  - git clone https://github.com/facebook/rocksdb.git
  - apply this patch
  - setenv CXX c++
  - setenv CPATH /usr/local/include/
  - setenv LIBRARY_PATH  /usr/local/lib/
  - gmake db_bench
  - make sure compilation is successful and db_bench is running
  - gmake all
  - make sure compilation is successful

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D33891
2015-02-26 15:19:17 -08:00
Igor Sugak
4ba119df5c rocksdb: Update path to dev clang in fbcode
Summary: Current clang path in fbcode points to clang 3.5. This diff updates clang path to clang 3.7.

Test Plan:
Make sure clang 3.7 is in use and all tests are passing.
```lang=bash
% USE_CLANG make check
```

Reviewers: meyering, sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D34119
2015-02-26 15:12:08 -08:00
Mark Callaghan
1e06a40685 Support builds for MongoDB+RocksDB
Summary:
This changes the RocksDB build to share headers provided by MongoDB. It is invoked as:
  ROCKSDB_FOR_MONGO=/path/to/mongodb/root make static_lib

Task ID: #

Blame Rev:

Test Plan:
Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D34071
2015-02-26 10:50:03 -08:00
Jim Meyering
a42324e370 build: do not relink every single binary just for a timestamp
Summary:
Prior to this change, "make check" would always waste a lot of
time relinking 60+ binaries. With this change, it does that
only when the generated file, util/build_version.cc, changes,
and that happens only when the date changes or when the
current git SHA changes.

This change makes some other improvements: before, there was no
rule to build a deleted util/build_version.cc. If it was somehow
removed, any attempt to link a program would fail.
There is no longer any need for the separate file,
build_tools/build_detect_version.  Its functionality is
now in the Makefile.

* Makefile (DEPFILES): Don't filter-out util/build_version.cc.
No need, and besides, removing that dependency was wrong.
(date, git_sha, gen_build_version): New helper variables.
(util/build_version.cc): New rule, to create this file
and update it only if it would contain new information.
* build_tools/build_detect_platform: Remove file.
* db/db_impl.cc: Now, print only date (not the time).
* util/build_version.h (rocksdb_build_compile_time): Remove
declaration.  No longer used.

Test Plan:
- Run "make check" twice, and note that the second time no linking is performed.
- Remove util/build_version.cc and ensure that any "make"
command regenerates it before doing anything else.
- Run this: strings librocksdb.a|grep _build_.
That prints output including the following:

  rocksdb_build_git_date:2015-02-19
  rocksdb_build_git_sha:2.8.fb-1792-g3cb6cc0

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D33591
2015-02-19 13:11:10 -08:00
Igor Sugak
4d98e29352 rocksdb: Enable scan-build static analysis
Summary:
Added new target ##make analyze## into Makefile. This command runs clang static analyzer and builds the sources as ##make all##. The result report is put into ##$(RocksDbSourceRoot)/can_build_report/##

If the development environment is a Facebook devserver and ##ROCKSDB_NO_FBCODE## is not set, then scan-build is used from fbcode. If it is run not on a Facebook devserver, scan-build should be available in ##$PATH##. I'll add details to wiki how to install scan-build on a non Facebook devserver environment.

Test Plan:
Run the fallowing commands on a Facebook devserver and Mac OS, and ensure no build or test errors.

```
% make all check -j32
% make clean
% USE_CLANG=1 make all -j32
% make analyze
% USE_CLANG=1 make analyze
```

Reviewers: sdong, lgalanis, leveldb, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D32799
2015-02-03 21:43:06 -08:00
Igor Canadi
2fd8f750ab Compile MemEnv with standard RocksDB library
Summary: This was a feature request by osquery. See task t5617758

Test Plan: compiles and memenv_test runs

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32115
2015-01-29 16:33:11 -08:00
Yueh-Hsuan Chiang
c4fb83441c Update the comment for the removal of mac-install-gflags.sh
Summary: Update the comment for the removal of mac-install-gflags.sh

Test Plan: n/a

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D32295
2015-01-27 10:30:35 -08:00
Igor Canadi
58f34edfce Fix valgrind 2015-01-26 13:17:28 -08:00
Igor Canadi
910186c278 Return the build with 4.8.1
Summary: We need this because we build MySQL with 4.8.1.

Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make check

Reviewers: sdong, yhchiang, rven, yoshinorim

Reviewed By: yoshinorim

Subscribers: jonahcohen, yoshinorim, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32073
2015-01-23 14:51:27 -08:00
Igor Canadi
401d4205ef Add thread sanitizer
Summary: When you compile with COMPILE_WITH_TSAN=1, we will compile the code with -fsanitize=thread. This will resolve bunch of data race issues we might have.

Test Plan: COMPILE_WITH_TSAN=1 m db_test

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32019
2015-01-23 11:22:20 -08:00
Igor Canadi
b068f0a673 Upgrade our compilers
Summary:
Upgrade gcc to 4.9.1 and clang to dev.

With new compilers I succeeded to run thread sanitizer, too. I'll post output (doesn't look good) and fix some things in separate diffs.

Test Plan: compiles with both g++ and clang

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32007
2015-01-23 11:12:37 -08:00
Igor Canadi
155bec4cf2 fallocate also tests FALLOC_FL_KEEP_SIZE 2015-01-15 14:15:43 -08:00
Igor Canadi
a3001b1d3d Remove -mtune=native because it's redundant 2014-12-19 09:06:45 -08:00
Igor Canadi
06eed650a0 Optimize default compile to compilation platform by default
Summary:
This diff changes compile to optimize for native platform by default. This will automatically turn on crc32 optimizations for modern processors, which greatly improves rocksdb's performance.

I also did some more changes to compilation documentation.

Test Plan:
compile with `make`, observe -march=native
compile with `PORTABLE=1 make`, observe no -march=native

Reviewers: sdong, rven, yhchiang, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D30225
2014-12-15 11:29:41 +01:00
Igor Canadi
767777c2bd Turn on -Wshorten-64-to-32 and fix all the errors
Summary:
We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.

This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.

Test Plan: compiles

Reviewers: ljin, rven, yhchiang, sdong

Reviewed By: yhchiang

Subscribers: bobbaldwin, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28689
2014-11-11 16:47:22 -05:00
Igor Canadi
5c93090530 Turn on -Wshadow
Summary:
Only one more try, I promise.

I talked to Jim and he mentioned that if we include our system includes with -isystem rather than with -I, that signals to the compile that those are system includes and thus no warnings are issued. So I turned our glibc includes into system includes and now we no longer get the warning from there, making us shadow-warning-free!

Test Plan: compiles with both clang and gcc

Reviewers: sdong, yhchiang, rven, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28479
2014-11-07 12:57:08 -08:00
Igor Canadi
30ca3752ba Revamp our build tools
Summary:
This diff is revamping our build tools:
1) Use third-party2 instead of third-party
2) consolidate clang and gcc scripts together, lots of duplication there
3) remove hdfs libs, we never compile rocksdb with them

clang compilation doesn't work yet. It doesn't work in master either. I plan to fix it soon, but I just spent 2 hours trying to make it work and failed. I'll ask experts.

Test Plan: compiles with gcc

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28047
2014-11-03 14:00:45 -08:00
Lei Jin
8db24f4b35 exclude mock test file from MOCK_SOURCES
Summary: as title

Test Plan: build with mock_env_test.cc

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28107
2014-10-31 15:07:27 -07:00
Igor Canadi
c082853340 Include all the mocks 2014-10-28 17:55:08 -07:00
Igor Canadi
abac3d6476 TableMock + framework for mock classes
Summary:
This diff replaces BlockBasedTable in flush_job_test with TableMock, making it depend on less things and making it closer to an unit test than integration test.

It also introduces a framework to compile mock classes -- Any file named *mock.cc will not be compiled into the build. It will only get compiled into the tests. What way we can mock out most other classes, Version, VersionSet, DBImpl, etc.

Test Plan: flush_job_test

Reviewers: ljin, rven, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27681
2014-10-28 17:52:32 -07:00
Igor Canadi
48842ab316 Deprecate AtomicPointer
Summary: RocksDB already depends on C++11, so we might as well all the goodness that C++11 provides. This means that we don't need AtomicPointer anymore. The less things in port/, the easier it will be to port to other platforms.

Test Plan: make check + careful visual review verifying that NoBarried got memory_order_relaxed, while Acquire/Release methods got memory_order_acquire and memory_order_release

Reviewers: rven, yhchiang, ljin, sdong

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D27543
2014-10-27 14:50:21 -07:00
fyrz
69d4c5123e Cross-platform fix version.sh
version.sh now works also on linux properly.
2014-10-04 11:17:06 +02:00
Igor Canadi
079a612b5e Fix build_tools/version.sh 2014-10-02 14:03:49 -07:00
Igor Canadi
0908ddcea5 Don't keep managing two rocksdb version
Summary:
Before this diff, there are two places with rocksdb versions. After the diff:
1. we only have one source of truth for rocksdb version
2. we have a script that we can use to get the version that we can use in other compilations (java, go, etc).

Test Plan: make

Reviewers: yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24333
2014-10-02 11:59:22 -07:00
mike@arpaia.co
ee1f3ccb06 Package generation for Ubuntu and CentOS
Summary:
I put together a script to assist in the generation of deb's and
rpm's. I've tested that this works on ubuntu via vagrant. I've included the
Vagrantfile here, but I can remove it if it's not useful. The package.sh
script should work on any ubuntu or centos machine, I just added a bit of
logic in there to allow a base Ubuntu or Centos machine to be able to build
RocksDB from scratch.

Example output on Ubuntu 14.04:

```
root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh
[+] g++-4.7 is already installed. skipping.
[+] libgflags-dev is already installed. skipping.
[+] ruby-all-dev is already installed. skipping.
[+] fpm is already installed. skipping.
Created package {:path=>"rocksdb_3.5_amd64.deb"}
root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb
 new debian package, version 2.0.
 size 17392022 bytes: control archive=1518 bytes.
     275 bytes,    11 lines      control
    2911 bytes,    38 lines      md5sums
 Package: rocksdb
 Version: 3.5
 License: BSD
 Vendor: Facebook
 Architecture: amd64
 Maintainer: rocksdb@fb.com
 Installed-Size: 83358
 Section: default
 Priority: extra
 Homepage: http://rocksdb.org/
 Description: RocksDB is an embeddable persistent key-value store for fast storage.
 ```

 Example output on CentOS 6.5:

 ```
 [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm
 Name        : rocksdb                      Relocations: /usr
 Version     : 3.5                               Vendor: Facebook
 Release     : 1                             Build Date: Mon 29 Sep 2014 01:26:11 AM UTC
 Install Date: (not installed)               Build Host: localhost
 Group       : default                       Source RPM: rocksdb-3.5-1.src.rpm
 Size        : 96231106                         License: BSD
 Signature   : (none)
 Packager    : rocksdb@fb.com
 URL         : http://rocksdb.org/
 Summary     : RocksDB is an embeddable persistent key-value store for fast storage.
 Description :
 RocksDB is an embeddable persistent key-value store for fast storage.
 ```

Test Plan:
How this gets used is really up to the RocksDB core team. If you
want to actually get this into mainline, you might have to change `make
install` such that it install the RocksDB shared object file as well, which
would require you to link against gflags (maybe?) and that would require some
potential modifications to the script here (basically add a depends on that
package).

Currently, this will install the headers and a pre-compiled statically linked
object file. If that's what you want out of life, than this requires no
modifications.

Reviewers: ljin, yhchiang, igor

Reviewed By: igor

Differential Revision: https://reviews.facebook.net/D24141
2014-09-29 16:09:46 -07:00
Igor Canadi
3b897cddd7 Enable no-fbcode RocksDB build
Summary: I want to use open source build rather than fbcode one. This enables me to run `ROCKSDB_NO_FBCODE=1 make` and run it with my system g++.

Test Plan:
ROCKSDB_NO_FBCODE=1 make
make

Reviewers: sdong, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23613
2014-09-19 09:27:16 -07:00
Igor Canadi
8de151bb99 Add db_bench with lots of column families to regression tests
Summary:
That way we can see when this graph goes up and be happy.

Couple of changes:
1. title
2. fix db_bench to delete column families before deleting the DB. this was asserting when compiled in debug mode
3. don't sync manifest when disableDataSync. We discussed this offline. I can move it to separate diff if you'd like

Test Plan: ran it

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22815
2014-09-05 14:20:18 -07:00
miguelportilla
93e6b5e9d9 Changes to support unity build:
* Script for building the unity.cc file via Makefile
* Unity executable Makefile target for testing builds
* Source code changes to fix compilation of unity build
2014-08-11 13:22:47 -04:00
Yueh-Hsuan Chiang
ae7743f226 Fixed some make and linking issues of RocksDBJava
Summary:
Fixed some make and linking issues of RocksDBJava. Specifically:
* Add JAVA_LDFLAGS, which does not include gflags
* rocksdbjava library now uses JAVA_LDFLAGS instead of LDFLAGS
* java/Makefile now includes build_config.mk
* rearrange make rocksdbjava workflow to ensure the library file is correctly
  included in the jar file.

Test Plan:
make rocksdbjava
make jdb_bench
java/jdb_bench.sh

Reviewers: dhruba, swapnilghike, zzbennett, rsumbaly, ankgup87

Reviewed By: ankgup87

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D20289
2014-07-21 22:41:54 -07:00
Radheshyam Balasundaram
f0660d5253 Adding NUMA support to db_bench tests
Summary:
Changes:
- Adding numa_aware flag to db_bench.cc
- Using numa.h library to bind memory and cpu of threads to a fixed NUMA node
Result: There seems to be no significant change in the micros/op time with numa_aware enabled. I also tried this with other implementations, including a combination of pthread_setaffinity_np, sched_setaffinity and set_mempolicy methods. It'd be great if someone could point out where I'm going wrong and if we can achieve a better micors/op.

Test Plan:
Ran db_bench tests using following command:
./db_bench --db=/mnt/tmp --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=/mnt/tmp --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --duration=300 --benchmarks=readwhilewriting --use_existing_db=1 --num=157286400 --threads=24 --writes_per_second=10240 --numa_aware=[False/True]

The tests were run in private devserver with 24 cores and the db was prepopulated using filluniquerandom test. The tests resulted in 0.145 us/op with numa_aware=False and 0.161 us/op with numa_aware=True.

Reviewers: sdong, yhchiang, ljin, igor

Reviewed By: ljin, igor

Subscribers: igor, leveldb

Differential Revision: https://reviews.facebook.net/D19353
2014-07-07 10:53:31 -07:00
nawu
fb54eef744 escaped the special characters and added a dot 2014-06-24 00:14:02 -05:00
nawu
c9ad282e4a added a dot 2014-06-24 00:05:03 -05:00
Igor Canadi
d4a8423334 Remove seek compaction
Summary:
As discussed in our internal group, we don't get much use of seek compaction at the moment, while it's making code more complicated and slower in some cases.

This diff removes seek compaction and (hopefully) all code that was introduced to support seek compaction.

There is one test case that relied on didIO information. I'll try to find another way to implement it.

Test Plan: make check

Reviewers: sdong, haobo, yhchiang, ljin, dhruba

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19161
2014-06-20 10:23:02 +02:00
Lei Jin
578cf84ddf give correct metric name for grep in regression script
Summary: as title

Test Plan: jenkin

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D19005
2014-06-10 16:32:09 -07:00
Lei Jin
c4b3817d7c fix regression test
Summary: as title

Test Plan: push

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18999
2014-06-10 11:24:24 -07:00
Lei Jin
e2d3101cf1 collect metrics for in memory workload get/seek
Summary:
collect in-memory workload get/seek metrics so that we can alert on
regression

Test Plan: ran locally

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18969
2014-06-10 09:59:16 -07:00
Igor Canadi
457bae6911 Fix regression test
Summary:
388d2054c7
added extra line to db_bench output, breaking regression tests. This diff makes it more robust and fixes the issue

Test Plan: ran it

Reviewers: ljin, sdong

Reviewed By: sdong

Subscribers: sdong, leveldb

Differential Revision: https://reviews.facebook.net/D18897
2014-06-04 09:59:44 -07:00