Commit Graph

363 Commits

Author SHA1 Message Date
Andrew Kryczka
186b3afaa8 Use fallocate even if hole-punching unsupported (#5023)
Summary:
The compiler flag `-DROCKSDB_FALLOCATE_PRESENT` was only set when
`fallocate`, `FALLOC_FL_KEEP_SIZE`, and `FALLOC_FL_PUNCH_HOLE` were all
present. However, the last of the three is not really necessary for the
primary `fallocate` use case; furthermore, it was introduced only in later
Linux kernel versions (2.6.38+).

This PR changes the flag `-DROCKSDB_FALLOCATE_PRESENT` to only require
`fallocate` and `FALLOC_FL_KEEP_SIZE` to be present. There is a separate
check for `FALLOC_FL_PUNCH_HOLE` only in the place where it is used.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5023

Differential Revision: D14248487

Pulled By: siying

fbshipit-source-id: a10ed0b902fa755988e957bd2dcec9081ec0502e
2019-03-04 15:43:17 -08:00
Yanqin Jin
7d23210226 Separate crash test with atomic flush (#4945)
Summary:
Currently crash test covers cases with and without atomic flush, but takes too
long to finish. Therefore it may be a better idea to put crash test with atomic
flush in a separate set of tests.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4945

Differential Revision: D13947548

Pulled By: riversand963

fbshipit-source-id: 177c6de865290fd650b0103408339eaa3f801d8c
2019-02-19 14:08:39 -08:00
Andrew Kryczka
34b55dd8c0 Fix no compression CI test config (#4982)
Summary:
We should strip `-DZSTD` to prevent ZSTD from being used in the no compression tests, similarly to how we prevent all other compression libraries from being used.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4982

Differential Revision: D14075349

Pulled By: ajkr

fbshipit-source-id: 8bd861516cf28a568c2b701ad33d0bb658db93b2
2019-02-13 16:47:01 -08:00
Andrew Kryczka
e242fa4664 Add latest toolchain (gcc-8, etc.) build support for fbcode users (#4923)
Summary:
- When building with internal dependencies, specify this toolchain by setting `ROCKSDB_FBCODE_BUILD_WITH_PLATFORM007=1`
- It is not enabled by default. However, it is enabled for TSAN builds in CI since there is a known problem with TSAN in gcc-5: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71090
- I did not add support for Lua since (1) we agreed to deprecate it, and (2) we only have an internal build for v5.3 with this toolchain while that has breaking changes compared to our current version (v5.2).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4923

Differential Revision: D13827226

Pulled By: ajkr

fbshipit-source-id: 9aa3388ed3679777cfb15ef8cbcb83c07f62f947
2019-01-28 11:26:32 -08:00
Andrew Kryczka
8ec3e72551 Cache dictionary used for decompressing data blocks (#4881)
Summary:
- If block cache disabled or not used for meta-blocks, `BlockBasedTableReader::Rep::uncompression_dict` owns the `UncompressionDict`. It is preloaded during `PrefetchIndexAndFilterBlocks`.
- If block cache is enabled and used for meta-blocks, block cache owns the `UncompressionDict`, which holds dictionary and digested dictionary when needed. It is never prefetched though there is a TODO for this in the code. The cache key is simply the compression dictionary block handle.
- New stats for compression dictionary accesses in block cache: "BLOCK_CACHE_COMPRESSION_DICT_*" and "compression_dict_block_read_count"
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4881

Differential Revision: D13663801

Pulled By: ajkr

fbshipit-source-id: bdcc54044e180855cdcc57639b493b0e016c9a3f
2019-01-23 18:15:47 -08:00
Faustin Lammler
7d65bd5ce4 Fix spelling errors (#4827)
Summary:
Hi, Lintian, the Debian package checker complains about spelling error (spelling-error-in-binary).

See https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs/98380
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4827

Differential Revision: D13566362

Pulled By: riversand963

fbshipit-source-id: cd4e9212133c73b0591030de6cdedaa47575968d
2019-01-02 11:17:57 -08:00
Adam Retter
75a333ad2d Fix build on macOS when jemalloc is installed via HomeBrew (#4767)
Summary:
Closes https://github.com/facebook/rocksdb/issues/2954

This doesn't necessarily mean that Rocks runs correctly with jemalloc on macOS... but at least now the build completes!
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4767

Differential Revision: D13495195

Pulled By: sagar0

fbshipit-source-id: d3fda0d6358a93e9803d215e3406f117be367c42
2018-12-17 16:30:20 -08:00
Maggie Moss
659d0e602e Run Define codemod in fbcode
Summary:
Found a callsite for `moo_translate` in the Scuba warnings and realized we have a few calls to `define()` left in fbcode.

- I ran the `DefineCodemod` script against fbcode
- Fixed broken tests, and ensured that tests that are explicitly testing the behaviour of `define()` were not changed.

bypass-lint

Reviewed By: kmeht

Differential Revision: D12968447

fbshipit-source-id: d8fd3649a2ce9868b8938d293e1bebf1a6d2fad8
2018-11-19 11:59:15 -08:00
Yi Wu
8c2a48742a Use -Os for lite release build (#4652)
Summary:
Set `-Os` for lite release build to minimize binary size.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4652

Differential Revision: D12965427

Pulled By: yiwu-arbug

fbshipit-source-id: c8b647642c24b3e5df6a2cd13112e452a08e8398
2018-11-07 22:10:28 -08:00
Yi Wu
742302a1a3 Fix compile error with aligned-new (#4576)
Summary:
In fbcode when we build with clang7++, although -faligned-new is available in compile phase, we link with an older version of libstdc++.a and it doesn't come with aligned-new support (e.g. `nm libstdc++.a | grep align_val_t` return empty). In this case the previous -faligned-new detection can pass but will end up with link error. Fixing it by only have the detection for non-fbcode build.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4576

Differential Revision: D10500008

Pulled By: yiwu-arbug

fbshipit-source-id: b375de4fbb61d2a08e54ab709441aa8e7b4b08cf
2018-10-23 10:55:41 -07:00
Dmitry Alimov
e13d8dcbbb Fix typos in comments (#4456)
Summary:
Fix some typos in the comments
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4456

Differential Revision: D10209214

Pulled By: miasantreble

fbshipit-source-id: dff857ba60396bc95126e635db96d7dc8330d2cb
2018-10-04 20:46:50 -07:00
Anand Ananthabhotla
c9a0419413 Release 5.16 (#4298)
Summary:
Update HISTORY.md for 5.16.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4298

Differential Revision: D9433868

Pulled By: anand1976

fbshipit-source-id: e7880a1c952210b1e9d7466eed72a6cb5018096b
2018-08-21 14:43:08 -07:00
Andrew Kryczka
63f265c383 Update FB internal dependencies (#4244)
Summary:
I noticed we were building against zstd 1.3.0 which is missing optimizations that our fbcode customers have (they're on zstd 1.3.5). Ran `./build_tools/update_dependencies.sh` to catch us up. Omitted the changes it made for gcc-4.8 since it's broken.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4244

Differential Revision: D9230111

Pulled By: ajkr

fbshipit-source-id: 3e8ec1d8a961f98ec77c8c6580bde4caacf2d437
2018-08-09 16:58:40 -07:00
Anand Ananthabhotla
72e69497fc Fix error parsing in build_tools/error_filter.py (#4247)
Summary:
The error_filter.py script parses the output of the "Build and run" stage of continuous tests to check for errors. It is currently only detecting compile errors and not link errors. This change fixes that.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4247

Differential Revision: D9233735

Pulled By: anand1976

fbshipit-source-id: 16e5a04950891cd9aba5cb3efcb6abc2a2e0d5ae
2018-08-09 14:43:34 -07:00
Andrew Kryczka
9e8ef45066 Link to FB internal build of ZSTD with -fPIC (#4249)
Summary:
TSAN requires the code is built with -fPIC. This PR links against a libzstd built with -fPIC when necessary, which enables ZSTD compression to be used in TSAN builds.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4249

Differential Revision: D9244746

Pulled By: ajkr

fbshipit-source-id: 8c6a8fadd6c8643b2077afcbc3626779e1d73b63
2018-08-09 12:42:43 -07:00
Anand Ananthabhotla
853a5cd3b2 Fix sandcastle rocksdb-contrun-tsan_crash job json (#4243)
Summary:
Fix the nested quotes for CRASH_TEST_EXT_ARGS, as the generated json could not be parsed by the sandcastle job.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4243

Differential Revision: D9228429

Pulled By: anand1976

fbshipit-source-id: 3c2bcac34870e377949d8a79c55e33b8363b25dd
2018-08-08 15:57:11 -07:00
Andrew Kryczka
6175b4b294 Support dictionary compression in stress/crash tests (#4234)
Summary:
- Add `--compression_max_dict_bytes` and `--compression_zstd_max_train_bytes` flags to stress test
- Randomly enable/disable the above flags in crash test
- Set `--compression_type=zstd` in FB-specific crash test runs
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4234

Differential Revision: D9187207

Pulled By: ajkr

fbshipit-source-id: 8d78cf8d8e1165f2cd1c32e069b73726b5bc1fd2
2018-08-06 15:27:29 -07:00
Andrew Kryczka
9b89479e64 Pass -latomic to linker when using clang
Summary:
clang compilation is failing due to a4fb1f8c04. In that commit I added a call to `std::atomic::is_lock_free` which was evidently relying on a compiler builtin only present in gcc.

Drawbacks to this fix are:

- users may need to install libatomic
- there might be cases where clang is used even though USE_CLANG is unset (e.g., when clang is the only available compiler). I didn't figure out how to add -latomic in those cases...

An alternative fix mentioned in http://lists.llvm.org/pipermail/llvm-bugs/2017-August/057263.html is using -stdlib=libc++ with clang.
Closes https://github.com/facebook/rocksdb/pull/3769

Differential Revision: D7756261

Pulled By: ajkr

fbshipit-source-id: 26888300683fa9970ab5950239d1aa217e8efd49
2018-04-25 12:13:41 -07:00
Adam Retter
12b400e814 Some small improvements to the build_tools
Summary: Closes https://github.com/facebook/rocksdb/pull/3664

Differential Revision: D7459433

Pulled By: sagar0

fbshipit-source-id: 3817e5d45fc70e83cb26f9800eaa0f4566c8dc0e
2018-04-02 23:57:41 -07:00
Tobias Tschinkowitz
ccb761364d Enable compilation on OpenBSD
Summary:
I modified the Makefile so that we can compile rocksdb on OpenBSD.
The instructions for building have been added to INSTALL.md.
The whole compilation process works fine like this on OpenBSD-current
Closes https://github.com/facebook/rocksdb/pull/3617

Differential Revision: D7323754

Pulled By: siying

fbshipit-source-id: 990037d1cc69138d22f85bd77ef4dc8c1ba9edea
2018-03-19 12:30:05 -07:00
Fosco Marotto
d518fe1da6 uint64_t and size_t changes to compile for iOS
Summary:
In attempting to build a static lib for use in iOS, I ran in to lots of type errors between uint64_t and size_t.  This PR contains the changes I made to get `TARGET_OS=IOS make static_lib` to succeed while also getting Xcode to build successfully with the resulting `librocksdb.a` library imported.

This also compiles for me on macOS and tests fine, but I'm really not sure if I made the correct decisions about where to `static_cast` and where to change types.

Also up for discussion: is iOS worth supporting?  Getting the static lib is just part one, we aren't providing any bridging headers or wrappers like the ObjectiveRocks project, it won't be a great experience.
Closes https://github.com/facebook/rocksdb/pull/3503

Differential Revision: D7106457

Pulled By: gfosco

fbshipit-source-id: 82ac2073de7e1f09b91f6b4faea91d18bd311f8e
2018-03-06 12:43:51 -08:00
Dmitri Smirnov
c364eb42b5 Windows cumulative patch
Summary:
This patch addressed several issues.
  Portability including db_test std::thread -> port::Thread Cc: @
  and %z to ROCKSDB portable macro. Cc: maysamyabandeh

  Implement Env::AreFilesSame

  Make the implementation of file unique number more robust

  Get rid of C-runtime and go directly to Windows API when dealing
  with file primitives.

  Implement GetSectorSize() and aling unbuffered read on the value if
  available.

  Adjust Windows Logger for the new interface, implement CloseImpl() Cc: anand1976

  Fix test running script issue where $status var was of incorrect scope
  so the failures were swallowed and not reported.

  DestroyDB() creates a logger and opens a LOG file in the directory
  being cleaned up. This holds a lock on the folder and the cleanup is
  prevented. This fails one of the checkpoin tests. We observe the same in production.
  We close the log file in this change.

 Fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose failure where the test
 attempts to open a directory with NewRandomAccessFile which does not
 work on Windows.
  Fix DBTest.SoftLimit as it is dependent on thread timing. CC: yiwu-arbug
Closes https://github.com/facebook/rocksdb/pull/3552

Differential Revision: D7156304

Pulled By: siying

fbshipit-source-id: 43db0a757f1dfceffeb2b7988043156639173f5b
2018-03-06 11:57:43 -08:00
Istvan Szukacs
d6336563e5 Adding CentOS 7 Vagrantfile & build script
Summary:
I have updated the Vagrantfile to have an entry for CentOS 7. Also created a simple build script which is pretty similar to the one in Beringei.

How to test:
```
vagrant up centos7
```
Todo:

Implement -j X for the build.
Closes https://github.com/facebook/rocksdb/pull/3530

Differential Revision: D7090739

Pulled By: ajkr

fbshipit-source-id: 9f9eda5b507568993543d08de7ce168dfc12282e
2018-02-26 15:27:17 -08:00
Yi Wu
989d12313c Legocastle job to report lite build binary size to scuba
Summary:
Add a legocastle job to continuously build the last 10 commits every 4 hours and report lite build binary size to scuba.
Closes https://github.com/facebook/rocksdb/pull/3511

Differential Revision: D7001730

Pulled By: yiwu-arbug

fbshipit-source-id: 7c8ca87c46d663c786a0d32be69ebbe7b19a5eb9
2018-02-15 17:27:24 -08:00
Mark Isaacson
b8eb32f8cf Suppress lint in old files
Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.

Reviewed By: yiwu-arbug

Differential Revision: D6821806

fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
2018-01-29 12:56:42 -08:00
Adam Retter
a53c571d2d FreeBSD build support for RocksDB and RocksJava
Summary:
Tested on a clean FreeBSD 11.01 x64.

Closes https://github.com/facebook/rocksdb/pull/1423
Closes https://github.com/facebook/rocksdb/pull/3357

Differential Revision: D6705868

Pulled By: sagar0

fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd
2018-01-11 13:29:55 -08:00
Siying Dong
a478e85697 Remove GCC parameter "-march=native" for ARM
Summary:
Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people.
Closes https://github.com/facebook/rocksdb/pull/3346

Differential Revision: D6690544

Pulled By: siying

fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49
2018-01-09 18:27:03 -08:00
yingsu00
f54d7f5fea Port 3 way SSE4.2 crc32c implementation from Folly
Summary:
**# Summary**

RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain.

This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used.

**# Performance Test Results**
We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm.

Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s.

1) ReadRandom in db_bench overall metrics

    PER RUN
    Algorithm | run | micros/op | ops/sec |Throughput (MB/s)
    3-way      |  1   | 4.143   | 241387 | 26.7
    3-way      |  2   | 3.775   | 264872 | 29.3
    3-way      | 3    | 4.116   | 242929 | 26.9
    FastCrc32c|1  | 4.037   | 247727 | 27.4
    FastCrc32c|2  | 4.648   | 215166 | 23.8
    FastCrc32c|3  | 4.352   | 229799 | 25.4

     AVG
    Algorithm     |    Average of micros/op |   Average of ops/sec |    Average of Throughput (MB/s)
    3-way           |     4.01                               |      249,729                 |      27.63
    FastCrc32c  |     4.35                              |     230,897                  |      25.53

 2)   Crc32c computation CPU cost (inclusive samples percentage)
    PER RUN
    Implementation | run |  TotalSamples   | Crc32c percentage
    3-way                 |  1    |  4,572,250,000 | 4.37%
    3-way                 |  2    |  3,779,250,000 | 4.62%
    3-way                 |  3    |  4,129,500,000 | 4.48%
    FastCrc32c       |  1    |  4,663,500,000 | 11.24%
    FastCrc32c       |  2    |  4,047,500,000 | 12.34%
    FastCrc32c       |  3    |  4,366,750,000 | 11.68%

 **# Test Plan**
     make -j64 corruption_test && ./corruption_test
      By default it uses 3-way SSE algorithm

     NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test

    make clean && DEBUG_LEVEL=0 make -j64 db_bench
    make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench
Closes https://github.com/facebook/rocksdb/pull/3173

Differential Revision: D6330882

Pulled By: yingsu00

fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9
2017-12-19 18:26:49 -08:00
Andrew Kryczka
e3814a8608 revert fbcode build behavior
Summary: Closes https://github.com/facebook/rocksdb/pull/3242

Differential Revision: D6514255

Pulled By: ajkr

fbshipit-source-id: c39fa8e745866b052649d02bf339e794d77e96a3
2017-12-07 16:12:52 -08:00
Adam Novak
a37d734596 Add ROCKSDB_DISABLE_* environment variables
Summary:
Should fix #3036.
Closes https://github.com/facebook/rocksdb/pull/3042

Differential Revision: D6452921

Pulled By: sagar0

fbshipit-source-id: eaf11e43fee1f8747006530cfc0c7a358f1c2f0f
2017-12-05 15:12:46 -08:00
Andrew Kryczka
63f1c0a57d fix gflags namespace
Summary:
I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google".
Closes https://github.com/facebook/rocksdb/pull/3212

Differential Revision: D6456973

Pulled By: ajkr

fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656
2017-12-01 10:42:05 -08:00
Andrew Kryczka
07c2738ffa prefer enabling cpu features via -march/-mcpu
Summary:
If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2.
Closes https://github.com/facebook/rocksdb/pull/3156

Differential Revision: D6304703

Pulled By: ajkr

fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef
2017-11-10 16:57:11 -08:00
Dmitri Smirnov
f8e2db0717 Fix crashes, address test issues and adjust windows test script
Summary:
Add per-exe execution capability
  Add fix parsing of groups/tests
  Add timer test exclusion

 Fix unit tests
  Ifdef threadpool specific tests that do not pass on Vista threadpool.
  Remove spurious outout from prefix_test so test case listing works
  properly.
  Fix not using standard test directories results in file creation errors
  in sst_dump_test.

  BlobDb fixes:
    In C++ end() iterators can not be dereferenced. They are not valid.
	When deleting blob_db_ set it to nullptr before any other code executes.
	Not fixed:. On Windows you can not delete a file while it is open.
	[ RUN      ] BlobDBTest.ReadWhileGC
	d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options)
	IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied
	d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options)
	IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied

  write_batch
    Should not call front() if there is a chance the container is empty
Closes https://github.com/facebook/rocksdb/pull/3152

Differential Revision: D6293274

Pulled By: sagar0

fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956
2017-11-10 10:41:57 -08:00
Yi Wu
8e63cad078 fix lite build
Summary:
* make `checksum_type_string_map` available for lite
* comment out `FilesPerLevel` in lite mode.
* travis and legocastle lite build also build `all` target and run tests
Closes https://github.com/facebook/rocksdb/pull/3015

Differential Revision: D6069822

Pulled By: yiwu-arbug

fbshipit-source-id: 9fe92ac220e711e9e6ed4e921bd25ef4314796a0
2017-10-17 08:57:09 -07:00
Yi Wu
725bc403f8 update dependencies.sh
Summary:
Update dependencies.sh. Also update tbb to 4.3, which is the latest available in TP2.
Closes https://github.com/facebook/rocksdb/pull/2812

Differential Revision: D5741394

Pulled By: yiwu-arbug

fbshipit-source-id: cafa0b7179f9a44669e5ccace818a02b42336781
2017-08-31 15:26:24 -07:00
Neal Poole
dfa6c23c4b Update RocksDBCommonHelper to use escapeshellarg
Summary:
Most of the data used here in shell commands is not generated directly from user input but some data (ie: from environment variables) may have been external influenced. It is a good practice to escape this data before using it in a shell command.

Originally D4800264 but we never quite got it merged.

Reviewed By: yiwu-arbug

Differential Revision: D5595052

fbshipit-source-id: c09d8b47fe35fc6a47afb4933ccad9d56ca8d7be
2017-08-15 06:56:31 -07:00
Nikhil Benesch
c5f0c6cc66 compile with correct flags to determine SSE4.2 support
Summary:
With some compilers, `-std=c++11` is necessary for <cstdint> to be
available. Pass this flag via $PLATFORM_CXXFLAGS. Fixes #2488.
Closes https://github.com/facebook/rocksdb/pull/2545

Differential Revision: D5620610

Pulled By: yiwu-arbug

fbshipit-source-id: 2f975b8c1ad52e283e677d9a33543abd064f13ce
2017-08-13 21:47:45 -07:00
Siying Dong
b87ee6f773 Use more keys per lock in daily TSAN crash test
Summary:
TSAN shows error when we grab too many locks at the same time. In TSAN crash test, make one shard key cover 2^22 keys so that no many keys will be hold at the same time.
Closes https://github.com/facebook/rocksdb/pull/2719

Differential Revision: D5609035

Pulled By: siying

fbshipit-source-id: 930e5d63fff92dbc193dc154c4c615efbdf06c6a
2017-08-10 17:56:57 -07:00
Maysam Yabandeh
627c9f1abb Don't add -ljemalloc when DISABLE_JEMALLOC is set
Summary:
fixes #2555
Closes https://github.com/facebook/rocksdb/pull/2684

Differential Revision: D5560527

Pulled By: maysamyabandeh

fbshipit-source-id: 6e1d874ae0b4e699a77203d9d52d0bb8f59013b0
2017-08-04 10:42:32 -07:00
Alan Somers
5883a1ae24 Fix /bin/bash shebangs
Summary:
"/bin/bash" is a Linuxism.  "/usr/bin/env bash" is portable.
Closes https://github.com/facebook/rocksdb/pull/2646

Differential Revision: D5556259

Pulled By: ajkr

fbshipit-source-id: cbffd38ecdbfffb2438969ec007ab345ed893ccb
2017-08-03 15:56:46 -07:00
Andres Suarez
3ce20e985b Fix use of RocksDBCommonHelper in cont_integration.sh
Reviewed By: mzlee

Differential Revision: D5472936

fbshipit-source-id: cf75858f879f1192b468a9020005634a5afad880
2017-07-26 19:31:36 -07:00
Yi Wu
63163a8c6e Remove make_new_version.sh
Summary:
Seems the only function of the script is to create a new branch, which can be done easily. I'm removing it.
Closes https://github.com/facebook/rocksdb/pull/2623

Differential Revision: D5468681

Pulled By: yiwu-arbug

fbshipit-source-id: 87dea5ecc4c85e06941ccbc36993f7f589063878
2017-07-20 20:45:20 -07:00
Siying Dong
33b1de82a7 Remove format compatibility hack
Summary:
We don't need this format compatibility hack anymore. We should remove it to make things simpler.
Closes https://github.com/facebook/rocksdb/pull/2607

Differential Revision: D5444107

Pulled By: siying

fbshipit-source-id: 7ef587dd0cacfc15a4083a137adba8e6bfddac7e
2017-07-18 11:12:32 -07:00
Siying Dong
3c327ac2d0 Change RocksDB License
Summary: Closes https://github.com/facebook/rocksdb/pull/2589

Differential Revision: D5431502

Pulled By: siying

fbshipit-source-id: 8ebf8c87883daa9daa54b2303d11ce01ab1f6f75
2017-07-15 16:11:23 -07:00
Maysam Yabandeh
4267eb00d4 Remove punit tests
Summary: Closes https://github.com/facebook/rocksdb/pull/2577

Differential Revision: D5420460

Pulled By: maysamyabandeh

fbshipit-source-id: 83671ec65513f4223044316e28cd6d33e9d5b531
2017-07-13 17:12:20 -07:00
Siying Dong
afbef65187 Bug fix: Fast CRC Support printing is not honest
Summary:
11c5d4741a introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts.
Closes https://github.com/facebook/rocksdb/pull/2513

Differential Revision: D5343802

Pulled By: yiwu-arbug

fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05
2017-06-28 21:41:42 -07:00
hyunwoo
c7662a44a4 fixed typo
Summary:
fixed typo
Closes https://github.com/facebook/rocksdb/pull/2376

Differential Revision: D5183630

Pulled By: ajkr

fbshipit-source-id: 133cfd0445959e70aa2cd1a12151bf3c0c5c3ac5
2017-06-05 11:27:34 -07:00
Aaron Gao
e7612798b5 update buckifer/TARGETS
Summary:
update targets file for release
Closes https://github.com/facebook/rocksdb/pull/2358

Differential Revision: D5115705

Pulled By: lightmark

fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794
2017-05-24 11:56:57 -07:00
Siying Dong
5068034666 range sync should be enabled
Summary:
We forgot to add the new flag in internal build script. Add it.
Closes https://github.com/facebook/rocksdb/pull/2360

Differential Revision: D5121428

Pulled By: siying

fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f
2017-05-24 09:58:24 -07:00
Nikhil Benesch
11c5d4741a cross-platform compatibility improvements
Summary:
We've had a couple CockroachDB users fail to build RocksDB on exotic platforms, so I figured I'd try my hand at solving these issues upstream. The problems stem from a) `USE_SSE=1` being too aggressive about turning on SSE4.2, even on toolchains that don't support SSE4.2 and b) RocksDB attempting to detect support for thread-local storage based on OS, even though it can vary by compiler on the same OS.

See the individual commit messages for details. Regarding SSE support, this PR should change virtually nothing for non-CMake based builds. `make`, `PORTABLE=1 make`, `USE_SSE=1 make`, and `PORTABLE=1 USE_SSE=1 make` function exactly as before, except that SSE support will be automatically disabled when a simple SSE4.2-using test program fails to compile, as it does on OpenBSD. (OpenBSD's ports GCC supports SSE4.2, but its binutils do not, so `__SSE_4_2__` is defined but an SSE4.2-using program will fail to assemble.) A warning is emitted in this case. The CMake build is modified to support the same set of options, except that `USE_SSE` is spelled `FORCE_SSE42` because `USE_SSE` is rather useless now that we can automatically detect SSE support, and I figure changing options in the CMake build is less disruptive than changing the non-CMake build.

I've tested these changes on all the platforms I can get my hands on (macOS, Windows MSVC, Windows MinGW, and OpenBSD) and it all works splendidly. Let me know if there's anything you object to—I obviously don't mean to break any of your build pipelines in the process of fixing ours downstream.
Closes https://github.com/facebook/rocksdb/pull/2199

Differential Revision: D5054042

Pulled By: yiwu-arbug

fbshipit-source-id: 938e1fc665c049c02ae15698e1409155b8e72171
2017-05-15 16:15:38 -07:00
Jeff Li
d004333021 Put lib files into suitable path in RPM package
Summary:
Currently, the RPM package will install the lib and header files into `/usr/package/lib` and `/usr/package/include` which is not in the default search paths. It is reasonable to install them under `/usr/lib` and `/usr/include` so that no extra configuration is required.
Closes https://github.com/facebook/rocksdb/pull/2221

Differential Revision: D5054030

Pulled By: yiwu-arbug

fbshipit-source-id: 1d23de5ff21f07e6738c9dfa04429acd7a839143
2017-05-15 14:05:46 -07:00
Andrew Kryczka
be421b0b16 portable sched_getcpu calls
Summary:
- added a feature test in build_detect_platform to check whether sched_getcpu() is available. glibc offers it only on some platforms (e.g., linux but not mac); this way should be easier than maintaining a list of platforms on which it's available.
- refactored PhysicalCoreID() to be simpler / less repetitive. ordered the conditional compilation clauses from most-to-least preferred
Closes https://github.com/facebook/rocksdb/pull/2272

Differential Revision: D5038093

Pulled By: ajkr

fbshipit-source-id: 81d7db3cc620250de220bdeb3194b2b3d7673de7
2017-05-10 12:29:23 -07:00
Dmitri Smirnov
35df23fe82 Fix suite exclisions
Summary: Closes https://github.com/facebook/rocksdb/pull/2250

Differential Revision: D5037141

Pulled By: siying

fbshipit-source-id: 53a0ab5553422839bb2df6a7badde54810774a3e
2017-05-10 11:12:38 -07:00
Siying Dong
0b90aa9515 Set VALGRIND_VER
Summary:
VALGRIND_VER was left empty after moving the environment to GCC-5. Set it back.
Closes https://github.com/facebook/rocksdb/pull/2234

Differential Revision: D4978534

Pulled By: siying

fbshipit-source-id: f0640d58e8f575f75fb3f8b92e686c9e0b6a59bb
2017-05-01 16:35:15 -07:00
Tomas Kolda
04d58970cb AIX and Solaris Sparc Support
Summary:
Replacement of #2147

The change was squashed due to a lot of conflicts.
Closes https://github.com/facebook/rocksdb/pull/2194

Differential Revision: D4929799

Pulled By: siying

fbshipit-source-id: 5cd49c254737a1d5ac13f3c035f128e86524c581
2017-04-21 20:48:04 -07:00
Siying Dong
ca96654d85 Change Build Env to gcc-5
Summary:
Default to build using gcc-5. Only apply to Facebook-only environments.
Closes https://github.com/facebook/rocksdb/pull/2158

Differential Revision: D4887568

Pulled By: siying

fbshipit-source-id: 53496c9af3273ccd44441bd0bef9d29beefbc00b
2017-04-14 11:12:56 -07:00
Sagar Vemuri
6799c7e00d Pass in remote as a param to branch creation script
Summary:
When people are working off of a rocksdb fork, i.e. when their 'origin'
points to github.com/<username>/rocksdb, the script creates a new branch
and pushes to their origin. The new branch created by this script should
instead be pushed to github.com/facebook/rocksdb. Many people might
have named facebook/rocksdb remote as 'upstream' (or something else).
This fix provides an option to specify the remote to push the branch to.
The default is still 'origin'

More context:
When I created 5.4 branch using this script, it got pushed to sagar0/rocksdb instead of facebook/rocksdb, as I was working off of a fork. My 'origin' was pointing to sagar0/rocksdb. My 'upstream' was set to 'facebook/rocksdb'. So, I had to manually push the branch to my 'upstream'.
Closes https://github.com/facebook/rocksdb/pull/2156

Differential Revision: D4885333

Pulled By: sagar0

fbshipit-source-id: 9410eab5bd9bbefc340059800bd6b8434406729d
2017-04-13 16:12:04 -07:00
Dmitri Smirnov
e5a1372b24 Rework test running script.
Summary:
Rework test running script.
  New options SuiteRun - runs specified executables as google suite
  test cases in parallel.
  Run - this option now runs executables in parallel the same as 'tests'
  RunAll - scans for test executables and attempts to run them all
  as suites except those mentiones in RunOnly (hardcoded in the script)
  or specified either in $ExcludeTestCases $ExcludeTestExe
Closes https://github.com/facebook/rocksdb/pull/2089

Differential Revision: D4832212

Pulled By: yiwu-arbug

fbshipit-source-id: 954990c
2017-04-05 11:39:20 -07:00
Siying Dong
88bb6f6bfa non_shm CI should run tests on /tmp
Summary:
Since non_shn CI was made to run in parallel, /dev/shm is automatically used. It defeated the purpose of the test to cover a non-ramfs file system.
Closes https://github.com/facebook/rocksdb/pull/2031

Differential Revision: D4764804

Pulled By: siying

fbshipit-source-id: 5666bda
2017-03-23 15:24:12 -07:00
Raza Hussain
4b04addfce updated solution if "make format" command fails
Summary: Closes https://github.com/facebook/rocksdb/pull/2016

Differential Revision: D4747003

Pulled By: yiwu-arbug

fbshipit-source-id: c1e2c5a
2017-03-21 11:09:10 -07:00
Dmitri Smirnov
c9df05d1e4 Fix random access alignment
Summary:
This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off.
Also adjust slightly appveyor script to run db_basic_test cases concurrently.
Closes https://github.com/facebook/rocksdb/pull/1959

Differential Revision: D4671972

Pulled By: IslamAbdelRahman

fbshipit-source-id: 1886620
2017-03-08 17:09:11 -08:00
Siying Dong
96c7e1504a Fix Java build
Summary:
The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version.
Closes https://github.com/facebook/rocksdb/pull/1912

Differential Revision: D4609854

Pulled By: siying

fbshipit-source-id: 3dc04f2
2017-02-23 18:39:13 -08:00
Sagar Vemuri
eb912a927e Remove disableDataSync option
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859

Differential Revision: D4541292

Pulled By: sagar0

fbshipit-source-id: 5b3a6ca
2017-02-13 11:09:13 -08:00
Rhys Parry
9fc23c55f2 Use gcc-4.9-glibc-2.20-fb python in precommit_checker
Summary:
The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed
soon.
Closes https://github.com/facebook/rocksdb/pull/1839

Differential Revision: D4509684

Pulled By: siying

fbshipit-source-id: 3efe296
2017-02-03 13:39:18 -08:00
Arun Sharma
fba726e555 Version librocksdb.so
Summary:
After make install, I see a directory hierarchy that looks like

```
./usr
./usr/include
./usr/include/rocksdb
./usr/include/rocksdb/filter_policy.h
[..]
./usr/include/rocksdb/iterator.h
./usr/include/rocksdb/utilities
./usr/include/rocksdb/utilities/ldb_cmd_execute_result.h
./usr/include/rocksdb/utilities/lua
./usr/include/rocksdb/utilities/lua/rocks_lua_custom_library.h
./usr/include/rocksdb/utilities/lua/rocks_lua_util.h
./usr/include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h
./usr/include/rocksdb/utilities/backupable_db.h
[..]
./usr/include/rocksdb/utilities/env_registry.h
[..]
./usr/include/rocksdb/env.h
./usr/lib64
./usr/lib64/librocksdb.so.5
./usr/lib64/librocksdb.so.5.0.0
./usr/lib64/librocksdb.so
./usr/lib64/librocksdb.a
```
Closes https://github.com/facebook/rocksdb/pull/1798

Differential Revision: D4456536

Pulled By: yiwu-arbug

fbshipit-source-id: 5494e91
2017-01-24 11:09:10 -08:00
Gunnar Kudrjavets
335981d473 Fix the directory path for RocksDB repo
fbshipit-source-id: 285548d
2016-12-27 22:24:16 -08:00
Yi Wu
fc0c6fd984 "make format" format diff since last commit from master
Summary:
Update clang-format script to format diff since last commit from master,
instead of just last commit. In our common workflow we usually endup
with multiple commits for a single PR. This change make it easier to
format all stacking changes.
Closes https://github.com/facebook/rocksdb/pull/1684

Differential Revision: D4340597

Pulled By: yiwu-arbug

fbshipit-source-id: c18949e
2016-12-16 11:24:18 -08:00
Daniel Black
a8bf4d63f9 Make format shows wrong curl command for retrieving clang-format-diff.py
Summary:
Previously:

$ make format
Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production
build_tools/format-diff.sh
You didn't have clang-format-diff.py available in your computer!
You can download it by running:
    curl http://goo.gl/iUW1u2
Makefile:868: recipe for target 'format' failed
make: *** [format] Error 128

$ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   276    0   276    0     0    148      0 --:--:--  0:00:01 --:--:--   148m

$ more ~/bin/clang-format-diff.py
<HTML>
<HEAD>
<TITLE>Moved Permanently</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Moved Permanently</H1>
The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>.
</BODY>
</HTML>
Closes https://github.com/facebook/rocksdb/pull/1680

Differential Revision: D4338495

Pulled By: yiwu-arbug

fbshipit-source-id: e2b24d8
2016-12-15 20:24:12 -08:00
Anirban Rahut
aad1191765 pass rocksdb oncall to mysql_mtr_filter otherwise tasks get created w…
Summary:
…rong owner

 mysql_mtr_filter script needs proper oncall
Closes https://github.com/facebook/rocksdb/pull/1586

Differential Revision: D4245150

Pulled By: anirbanr-fb

fbshipit-source-id: fd8577c
2016-11-29 12:09:12 -08:00
Yueh-Hsuan Chiang
647eafdc21 Introduce Lua Extension: RocksLuaCompactionFilter
Summary:
This diff includes an implementation of CompactionFilter that allows
users to write CompactionFilter in Lua.  With this ability, users can
dynamically change compaction filter logic without requiring building
the rocksdb binary and restarting the database.

To compile, WITH_LUA_PATH must be specified to the base directory
of lua.
Closes https://github.com/facebook/rocksdb/pull/1478

Differential Revision: D4150138

Pulled By: yhchiang

fbshipit-source-id: ed84222
2016-11-16 15:39:12 -08:00
Nipunn Koorapati
25f5742f0b Update documentation to point at gcc 4.8
Summary:
Rocksdb currently has many references to std::map.emplace_back()
which is not implemented in gcc 4.7, but valid in gcc 4.8. Confirmed that
it did not build with gcc 4.7, but builds fine with gcc 4.8
Closes https://github.com/facebook/rocksdb/pull/1272

Differential Revision: D4101385

Pulled By: IslamAbdelRahman

fbshipit-source-id: f6af453
2016-10-29 12:09:17 -07:00
Anirban Rahut
2e8004e608 Changing the legocastle run to use valgrind_test instead of _check
Summary:
valgrind_test is the correct way to run valgrind tests.
this is becasue we need to force DISABLE_JEMALLOC

Test Plan: Running sandcastle and contrun

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D65451
2016-10-24 16:23:19 -07:00
sdong
607628d349 Support ZSTD with finalized format
Summary:
ZSTD 1.0.0 is coming. We can finally add a support of ZSTD without worrying about compatibility.
Still keep ZSTDNotFinal for compatibility reason.

Test Plan: Run all tests. Run db_bench with ZSTD version with RocksDB built with ZSTD 1.0 and older.

Reviewers: andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: cyan, igor, IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D63141
2016-09-06 12:22:16 -07:00
Alex Robinson
b10d65c2a4 Update and slightly clarify instructions in build_detect_platform (#1301) 2016-08-25 10:40:38 -07:00
Yi Wu
99c4af7160 Make ClockCache available with TSAN build
Summary: ..to unblock TSAN contbuild.

Test Plan:
  COMPILE_WITH_TSAN=1 make db_stress -j64
  ./db_stress --use_clock_cache=true

Reviewers: sdong, lightmark, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D62397
2016-08-23 13:40:47 -07:00
Yi Wu
ff17a2abf3 Adding TBB as dependency.
Summary: Splitting the makefile part of D55581.

Test Plan:
  make all check -j32
  ROCKSDB_FBCODE_BUILD_WITH_481=1 make all check -j32
  ROCKSDB_NO_FBCODE=1 make all check -j32

  export TBB_BASE=/mnt/gvfs/third-party2/tbb/afa54b33cfcf93f1d90a3160cdb894d6d63d5dca/4.0_update2/gcc-4.9-glibc-2.20/e9936bf;
  ROCKSDB_NO_FBCODE=1 CFLAGS="-I $TBB_BASE/include" LDFLAGS="-L $TBB_BASE/lib -Wl,-rpath=$TBB_BASE/lib" make all check -j32

Reviewers: IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: kradhakrishnan, yhchiang, IslamAbdelRahman, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56979
2016-08-18 10:44:29 -07:00
Islam AbdelRahman
4990c0d1a5 Remove deprecated LEVELDB_PLATFORM_POSIX
Summary:
We have replaced LEVELDB_PLATFORM_POSIX with ROCKSDB_PLATFORM_POSIX in our code
replace it in Makefile

Test Plan: make check

Reviewers: sdong, yiwu, andrewkr, lightmark

Reviewed By: lightmark

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61365
2016-08-03 11:07:53 -07:00
krad
c49ea68c55 Fix to enable running CI jobs locally
Summary:
We may not have permission on all /dev/shm to fix the sticky bit.
Making the sticky bit fix advisory.

Test Plan: Run CI job locally

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D61371
2016-08-01 16:31:49 -07:00
Gunnar Kudrjavets
6920cde893 Remove an extra apostrophe 2016-07-27 21:16:11 -07:00
Gunnar Kudrjavets
9c8ac144bd Avoid duplicate task creation for RocksDB contruns
Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure.

Test Plan:
- Land the changes.
- Trigger a RocksDB contrun - observe the results.
- Look at the results from the nightly runs and fix issues as necessary.

Current testing so far has been in isolation for various components.

Reviewers: andrewkr, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60909
2016-07-27 16:37:12 -07:00
Anirban Rahut
d3bfd33972 Testing out parallel sandcastle changes
Summary:
Removing moreutils from sandcastle and adding gnu parallel.
Then passing in J= nproc command

Test Plan: Testing on sandcastle

Reviewers: sdong, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61017
2016-07-27 11:58:57 -07:00
sdong
7efd9c25c4 Increse timeout in some tests
Summary: Increase test timeout to some tests to unblock CI.

Test Plan: watch how it runs.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61263
2016-07-27 11:39:36 -07:00
woonhak.kang
5c858ddd20 fix errata in libnuma test (#1244) 2016-07-26 19:28:45 -07:00
krad
a4955b39ac Run sandcastle tests in /dev/shm 2016-07-25 15:10:24 -07:00
sdong
f85df120f2 Re-enable tsan crash white-box test with reduced killing odds
Summary:
Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds.
Add a parameter in crash test script to allow we pass it through an environment variable.

Test Plan: Run it manually.

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61053
2016-07-22 12:51:27 -07:00
sdong
b50632920d Add unit test not on /dev/shm as part of the pre-commit tests
Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system.

Test Plan: See the tests launched by Phabricator

Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60963
2016-07-21 12:53:23 -07:00
sdong
f9b14be495 Re-enable TSAN crash test but only with black box crash test
Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job.

Test Plan: Run it locally.

Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60951
2016-07-20 13:52:25 -07:00
sdong
68f3eb746c Run release build for CLANG and GCC 4.8.1 in pre-commit tests too
Summary: As Title.

Test Plan: See how the diff works.

Reviewers: kradhakrishnan, andrewkr, gunnarku

Reviewed By: gunnarku

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60933
2016-07-20 13:46:32 -07:00
Gunnar Kudrjavets
61dbfbb6ce Add release build to RocksDB per-diff/post-commit tests
Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening.

Test Plan:
- `[p]arc diff --preview`
- Observe the execution in Sandcastle and make sure that release build and tests are executed.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60441
2016-07-08 16:02:29 -07:00
Gunnar Kudrjavets
aa432be4b9 Workarounds for continuous build implementation 2016-06-23 19:38:29 -07:00
Gunnar Kudrjavets
8cd9f04feb Test change to verify new commit detection 2016-06-23 12:08:44 -07:00
Gunnar Kudrjavets
8a4ee7e90b Trivial change to test cont. build 2016-06-23 11:53:38 -07:00
Gunnar Kudrjavets
6576fa05af Fix minor typos and PHP source file name used to trigger the builds
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed.

Test Plan: - Testing with a private Sandcastle instance.

Reviewers: sdong, mung

Reviewed By: mung

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59919
2016-06-22 11:05:52 -07:00
Gunnar Kudrjavets
a52e4d7d02 Framework for enabling continuous RocksDB build and tests
Summary:
The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs.

The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things:

- Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only.

- Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed.

- Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen.

Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff.

Test Plan:
- Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass.
- Private testing of triggering the continuous build script.
- Once the changes will land then author an internal job which will use the script and verify its validity.

Reviewers: sdong, yhchiang, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59811
2016-06-21 11:38:54 -07:00
krad
550bf895e9 Minor bug fix with log name 2016-06-14 12:20:58 -07:00
Islam AbdelRahman
8100ec2cd1 Fix libgcc broken lib path
Summary: current libgcc lib path is broken, update it and make sure the new one exists

Test Plan:
verify that the new path exists
make check -j64

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59535
2016-06-12 00:11:08 -07:00
Islam AbdelRahman
bde7d1055f Fix clang_analyze path in fbcode_config.sh
Summary: current clang_analyze path is broken, update it

Test Plan:
USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make J=64 analyze -j64
make sure it works

Reviewers: yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59505
2016-06-10 14:36:38 -07:00
krad
e9c1face6c Minor fix to precommit-check.py 2016-06-10 10:53:02 -07:00
krad
fcc47fa5f1 New features to precommit check script
Summary:
Added new features to help run the entire suite of tests on a diff
locally leveraging parallelism

Test Plan: Basic command line test

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D59445
2016-06-10 10:33:22 -07:00
Islam AbdelRahman
edc764e91b Use valgrind built with gcc-4.9-glibc-2.20
Summary:
Right now we use valgrind built with gcc-5-glibc-2.23 which keep generating this error everywhere

```
==3224756== Conditional jump or move depends on uninitialised value(s)
==3224756==    at 0x401AD68: index (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4008933: _dl_map_object (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000ECC: map_doit (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40101C7: _dl_catch_error (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000B34: do_preload (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x400458C: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x40185DB: _dl_sysdep_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4004F39: _dl_start (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==    by 0x4000CE7: ??? (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==  Uninitialised value was created by a stack allocation
==3224756==    at 0x4004519: dl_main (in /usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld-2.20.so)
==3224756==
```

Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64

Reviewers: andrewkr, yiwu, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59343
2016-06-08 13:22:16 -07:00