Commit Graph

4631 Commits

Author SHA1 Message Date
Yueh-Hsuan Chiang
955ecf8b49 Fix an ASAN error in compact_files_test
Summary:
compact_files_test enables SyncPoint but never disable it before
the test terminates.  As a result, it might cause heap-use-after-free
error when some code path trying to access the static variable of
SyncPoint when it has already gone out of scope after the main thread
dies.

Test Plan:
COMPILE_WITH_ASAN=1 make compact_files_test -j32
./compact_files_test

Reviewers: sdong, anthony, kradhakrishnan, rven, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53379
2016-01-26 11:30:30 -08:00
Islam AbdelRahman
b0afcdeeac Fix bug in block based tables with full filter block and prefix_extractor
Summary:
Right now when we are creating a BlockBasedTable with fill filter block
we add to the filter all the prefixes that are InDomain() based on the prefix_extractor

the problem is that when we read a key from the file, we check the filter block for the prefix whether or not it's InDomain()

Test Plan: unit tests

Reviewers: yhchiang, rven, anthony, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53385
2016-01-26 11:07:08 -08:00
Andrew Kryczka
167bd8856d [directory includes cleanup] Finish removing util->db dependencies 2016-01-26 10:49:24 -08:00
Andrew Kryczka
acd7d58695 [directory includes cleanup] Remove util->db dependency for ThreadStatusUtil
Summary:
We can avoid the dependency by forward-declaring ColumnFamilyData and
then treating it as a black box. That means callers of ThreadStatusUtil need to
explicitly provide more options, even if they can be derived from the
ColumnFamilyData, since ThreadStatusUtil doesn't include the definition.

This is part of a series of diffs to eliminate circular dependencies between
directories (e.g., db/* files depending on util/* files and vice-versa).

Test Plan:
  $ ./db_test --gtest_filter=DBTest.GetThreadStatus
  $ make -j32 commit-prereq

Reviewers: sdong, yhchiang, IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53361
2016-01-26 10:49:16 -08:00
Andrew Kryczka
46f9cd46af [directory includes cleanup] Move cross-function test points
Summary:
I split the db-specific test points out into a separate file under db/
directory. There were also a few bugs to fix in xfunc.{h,cc} that prevented it
from compiling previously; see https://reviews.facebook.net/D36825.

Test Plan:
compilation works now, below command works, will also run "make xfunc".

  $ make check ROCKSDB_XFUNC_TEST='managed_new' tests-regexp='DBTest' -j32

Reviewers: sdong, yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53343
2016-01-26 10:49:05 -08:00
krad
22ecb752db Add valgrind to pre-commit sandcastle testing
Summary:
Initially I removed "valgrind" from the list since it take too much
time (3+hr) compared to tsan (40 min) when the tests are run in parallel. It is
not effective to run the tests in parallel in sandcastle and tsan takes about
3hrs as well.

Adding valgrind to the list.

Test Plan: Submit this diff and watch the run

Reviewers: sdong, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53337
2016-01-26 10:40:57 -08:00
Venkatesh Radhakrishnan
b7ecf3d214 Fix intermittent hang in ColumnFamilyTest.FlushAndDropRaceCondition
Summary:
ColumnFamilyTest.FlushAndDropRaceCondition sometimes
hangs because the sync point, "FlushJob::InstallResults", sleeps
holding the DB mutex. Fixing it by releasing the mutex before sleeping.

Test Plan:
seq 1000 |parallel --gnu --eta 't=/dev/shm/rdb-{}; rm -rf $t;
mkdir $t && export TEST_TMPDIR=$t; ./column_family_test
-gtest_filter=*FlushAndDropRaceCondition* > $t/log-{}'

Reviewers: IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53349
2016-01-26 09:12:20 -08:00
sdong
38e1d7fea3 ldb to support --column_family option
Summary:
Add an option --column_family option, so that users can query or update specific column family.
Also add an create column family parameter to make unit test easier.
Still need to add unit tests.

Test Plan: Will add a test case in ldb python test.

Reviewers: yhchiang, rven, andrewkr, IslamAbdelRahman, kradhakrishnan, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53265
2016-01-25 14:58:18 -08:00
sdong
da33dfe188 Parameterize DBTest.Randomized
Summary: Break down DBTest.Randomized to multiple gtest tests based on config type

Test Plan: Run the test and all tests. Make sure configurations are correctly set

Reviewers: yhchiang, IslamAbdelRahman, rven, kradhakrishnan, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53247
2016-01-25 14:49:08 -08:00
sdong
fb9811ee9b Add a perf context level that doesn't measure time for mutex operations
Summary: Timing mutex operations can impact scalability of the system. Add a new perf context level that can measure time counters except for mutex.

Test Plan: Add a new unit test case to make sure it is not set.

Reviewers: IslamAbdelRahman, rven, kradhakrishnan, yhchiang, anthony

Reviewed By: anthony

Subscribers: MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53199
2016-01-25 14:45:03 -08:00
Islam AbdelRahman
f7ef1a6132 Include rest of dependencies in dependencies.sh
Summary:
This diff

  - Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh
  - updtade zst to 0.4.7
  - It also fix a problem in clang scan build

Test Plan:
make check
USE_CLANG=1 make check
USE_CLANG=1 make analyze
coverage_test.sh

Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53301
2016-01-25 14:35:04 -08:00
Andrew Kryczka
3e9209a078 Updated GetProperty documentation
Summary: As titled. Also added the kBaseLevel string, which was missing earlier.

Test Plan: built

Reviewers: yhchiang, anthony, rven, kradhakrishnan, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53271
2016-01-25 14:04:55 -08:00
Venkatesh Radhakrishnan
40911e0b32 Run unit tests in parallel to find failing tests
Summary:
Added make targets parallel_test and parallel_dbtest to run
tests in parallel. Each test is run 32 times in parallel. There is a
timeout to catch hangs. The test continues after a failure and reports
non-zero status on failure

Test Plan: Run the two make targets

Reviewers: anthony, yhchiang, IslamAbdelRahman, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53079
2016-01-25 14:04:27 -08:00
Islam AbdelRahman
2fbc59a348 Disallow SstFileWriter from creating empty sst files
Summary:
SstFileWriter may create an sst file with no entries
Right now this will fail when being ingested using DB::AddFile() saying that the keys are corrupted

Test Plan: make check

Reviewers: yhchiang, rven, anthony, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D52815
2016-01-25 13:47:07 -08:00
krad
f53c95f81b Cosmetic fixes and comments for the reader
Summary:
Cosmetic fixes and some comments for the script. It is one big hack and
hopefully the comments will make it easy to maintain.

Test Plan: Run manual tests

Reviewers: sdong, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53277
2016-01-25 10:45:29 -08:00
sdong
f1ed170105 Add tests to make sure new DB or ColumnFamily options are settable through string
Summary: Add a test to fail if someone adds a DB options.

Test Plan: Run the test, run the test with valgrind. Add an option to DB option in the middle or in the end and make sure it fails.

Reviewers: yhchiang, anthony, IslamAbdelRahman, kradhakrishnan, rven, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53097
2016-01-22 16:18:44 -08:00
krad
f57596b0cb Improvements to pre-commit
Summary:
- UI is enhanced to lists the tests, status and the results
- We are using the same pre-commit tool as the make equivalent
- No more emails to user on failure
- Dropped valgrind from the list since it can be a time hogger (and can hurt
  scheduling for others)
- Patching bug fix
- Made the jobs run in parallel in sandcastle

Test Plan: Manual test

Reviewers: sdong, rven, igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53217

Making parallel requests to sandcastle

Test Plan: Run manual tests

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53259
2016-01-22 15:40:32 -08:00
Islam AbdelRahman
538eec0661 Update fbcode_config.sh to use latest versions automatically
Summary:
Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated

```
Snappy: 1.0.3 => 1.1.3
GFLAGS: 1.6 => 2.1.1
JEMALLOC: 3.6.0 => 4.0.3
```

I have also updated clang from 3.7 to 3.7.1

```
Clang 3.7 => 3.7.1
```

Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library

Test Plan:
make check -j64
USE_CLANG=1 make check -j64

DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running)

Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53037
2016-01-22 11:56:32 -08:00
agiardullo
8019aa9b55 improve test for manifest write failure
Summary: Improve testing per discussion in D52989

Test Plan: ran test

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53211
2016-01-22 11:47:59 -08:00
agiardullo
bcd4ccbc33 Revert D7809
Summary: Revert the functionaility of D7809 (but I'm keeping the logging and test code).  We decided it was dangerous to ignore sync failures based on attempting to read the data written.  The read does not tell us whether the data was synced.

Test Plan: There was no test for the particular functionaility that was reverted.  Keeping the test code from D7809 that tests whether we set the DB to be readonly when paranoid checks are enabled.

Reviewers: rven, yhchiang, kradhakrishnan, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52989
2016-01-21 17:14:43 -08:00
krad
b0a15e7fb9 Mechanism to run CI jobs on local branch via commit_prereq
Summary: This patch provides a mechanism to run pre commit tests on the local
branch before committing. This can help prevent frequent build breaks.

The tests can be run in parallel by specifying the J=<..> environment
variable.

Test Plan: Run manually

Reviewers: sdong rven tec

CC: leveldb@

Task ID: #9689218

Blame Rev:
2016-01-21 16:22:21 -08:00
Andrew Kryczka
bb2888738c Cleanup property-related variable names
Summary:
I noticed these names were quite confusing while updating GetProperty
documentation.

Test Plan: running "make commit-prereq -j32"

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53073
2016-01-21 11:38:15 -08:00
Andrew Kryczka
29289333d0 Add named constants for remaining properties
Summary:
There were just these two properties that didn't have any named
constant.

Test Plan:
build and below test

  $ ./db_properties_test --gtest_filter=DBPropertiesTest.NumImmutableMemTable

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53103
2016-01-21 10:59:36 -08:00
sdong
2c2b72218d Disable OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable under CLANG
Summary: OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable is failiong under CLANG. Disable the test to unblock the build.

Test Plan: Run it both of CLANG and GCC

Reviewers: kradhakrishnan, rven, andrewkr, anthony, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53157
2016-01-21 10:56:34 -08:00
krad
a300d9928e Added sandcastle pre-commit
Test Plan:
Lately we have been breaking our builds too often. This changes adds
the capability to schedule tests in sandcastle for every diff created. This will
help us increase the pre-commit testing bar.

This patch will dispatch signals to sandcastle to start running tests on the
diff. The test failures are reported to the user via email.

The user can also manually check the progress of test in sandcastle via the URL
provided.

Reviewers: sdong, rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53001
2016-01-21 08:53:55 -08:00
sdong
202be23e46 Add test that verifies all options in BlockBasedTableOptions is settable through GetBlockBasedTableOptionsFromString()
Summary: Add a test OptionsParserTest.BlockBasedTableOptionsAdded, which will fail if a new option is added to BlockBasedTableOptions but is not settable through GetBlockBasedTableOptionsFromString().

Test Plan: Run the test. Also manually remove and add options and make sure it fails.

Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, rven, yhchiang, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52953
2016-01-20 21:59:28 -08:00
Andrew Kryczka
eceb5cb1b7 Split db_test.cc (part 1: properties)
Summary:
Moved all the tests that verify property correctness into a separate
file. The goal is to reduce compile time and complexity of db_test. I didn't
add parallelism for db_properties_test, even though these tests were
parallelized in db_test, since the file is small enough that it won't matter.

Some of these moves may be controversial since it's hard to say whether the
test is "verifying property correctness," or "using properties to verify
rocksdb's correctness." I'm interested in any opinions.

Test Plan: ran db_properties_test, also waiting on "make commit-prereq -j32"

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52995
2016-01-20 15:17:52 -08:00
sdong
94918ae84b db_bench: explicitly clear buffer in compress benchmark
Summary: It is reported that in compress benchmark in db_bench, zlib will cause an OOM. The suggestd fix was to clear the buffer.

Test Plan: Build and run compress benchmark.

Reviewers: IslamAbdelRahman, yhchiang, rven, andrewkr, kradhakrishnan, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52857
2016-01-19 18:11:46 -08:00
sdong
fdbff42391 Crash test to make kill decision for every kill point
Summary:
In crash test, when coming to each kill point, we start a random class using seed as current second. With this approach, for every second, the random number used is the same. However, in each second, there are multiple kill points with different frequency. It makes it hard to reason about chance of kill point to trigger. With this commit, we use thread local random seed to generate the random number, so that it will take different values per second, hoping it makes chances of killing much easier to reason about.

Also significantly reduce the kill odd to make sure time before kiling is similar as before.

Test Plan: Run white box crash test and see the killing happens as expected and the run time time before killing reasonable.

Reviewers: kradhakrishnan, IslamAbdelRahman, rven, yhchiang, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52971
2016-01-19 18:11:24 -08:00
Siying Dong
39c3e94ff4 Merge pull request #954 from davidbernard/solaris_build
Solaris Build
2016-01-19 18:05:02 -08:00
David Bernard
df7c2f3b59 As per google coding standard replace "using" in option_builder.cc and
geodb_impl.cc
2016-01-20 00:27:29 +00:00
David Bernard
12809b44b4 Revert "Change notification email for travis"
This reverts commit eaa5637568.
2016-01-19 22:46:39 +00:00
Mike Kolupaev
34704d5c7b [easy] Fixed a crash in LogAndApply() when CF creation failed
Summary: That line used to dereference `column_family_data`, which is nullptr if we're creating a column family.

Test Plan: `make -j check`

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D52881
2016-01-19 11:46:52 -08:00
Siying Dong
791dbafa93 Merge pull request #953 from sselva/master
Add Rakuten Marketing to USERS.md
2016-01-19 11:04:04 -08:00
Igor Canadi
594a5ef021 Merge pull request #955 from bcbrock/ppc64-build
Simple changes to support builds for ppc64[le] consistent with X86
2016-01-19 09:05:44 -08:00
bcbrock
f423f05dcd Simple changes to support builds for ppc64[le] consistent with X86
These simple changes are required to allow builds on ppc64[le] systems
consistent with X86. The Makefile now recognizes both ppc64 and ppc64le, and
in the absence of PORTABLE=1, the code will be built analogously to the X86
-march=native.

Note that although GCC supports -mcpu=native -mtune=native on POWER, it
doesn't work correctly on all systems. This is why we need to get the actual
machine model from the AUX vector.
2016-01-19 09:08:19 -06:00
David Bernard
3f12e16f27 Make alloca.h optional 2016-01-19 06:17:31 +00:00
David Bernard
eaa5637568 Change notification email for travis 2016-01-19 04:59:39 +00:00
David Bernard
d78c6b28c4 Changes for build on solaris
Makefile adjust paths for solaris build
Makefile enable _GLIBCXX_USE_C99 so that std::to_string is available
db_compaction_test.cc Initialise a variable to avoid a compilation error
db_impl.cc Include <alloca.h>
db_test.cc Include <alloca.h>
Environment.java recognise solaris envrionment
options_bulder.cc Make log unambiguous
geodb_impl.cc Make log and floor unambiguous
2016-01-19 04:45:21 +00:00
Selva Sarangan
2e9fae3f2a Add Rakuten Marketing to USERS.md 2016-01-18 15:24:52 -08:00
agiardullo
83e1de92af move internal build to use zstd 0.4.5
Summary: update internal build scripts to use zstd 0.4.5.

Test Plan: built and ran tests with and without ROCKSDB_FBCODE_BUILD_WITH_481

Reviewers: yhchiang, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52833
2016-01-15 13:27:35 -08:00
Gunnar Kudrjavets
aec10f734b Guard falloc.h inclusion to avoid build breaks
Summary: Depending on the order of include paths and versions of various headers we may end up in a situation where we'll encounter a build break caused by redefinition of constants. gcc-4.9-glibc-2.20 header update to include/bits/fcntl-linux.h introduced the definitions of FALLOC_FL_* constants. However, linux/falloc.h from kernel-headers also has FALLOC_FL_* constants defined. Therefore during the compilation we'll get "previously defined" errors.

Test Plan:
Both in the environment where the build break manifests (to make sure that the change fixed the problem) and in the environment where everything builds fine (to make sure that there are no regressions):

make clean
make -j 32

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52821
2016-01-14 22:47:15 -08:00
agiardullo
f7ebc2f346 Update HISTORY.mc for 4.4.0
Summary: Prepare to cut release

Test Plan: no code change.

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D52827
2016-01-14 16:57:06 -08:00
Siying Dong
addd9545f8 Merge pull request #947 from yuslepukhin/align_and_yield
Align statistics
2016-01-14 10:28:40 -08:00
Dmitri Smirnov
ac50fd3a71 Align statistics
Use Yield macro to make it a little more portable between platforms.
2016-01-13 14:53:23 -08:00
sdong
b54d4dd435 tools/sst_dump_tool_imp.h not to depend on "util/testutil.h"
Summary:
util/testutil.h doesn't seem to be used in tools/sst_dump_tool_imp.h. Remove it.
Also move some other include to tools/sst_dump_tool.cc instead.

Test Plan: Build with GCC, CLANG and with GCC 4.81 and 4.9.

Reviewers: yuslepukhin, yhchiang, rven, anthony, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52791
2016-01-13 11:34:53 -08:00
Igor Canadi
48a8667c30 Merge pull request #929 from warrenfalk/fix32
fix a compile error on 32-bit (fixes #634)
2016-01-12 11:02:36 -08:00
Islam AbdelRahman
d9bca1e14c Reduce iterator deletion overhead
Summary:
After introducing Iterator::PinData(), we have extra overhead of deleting the pinned iterators that we track in a std::set
This patch avoid inserting to the std::set if we have only one iterator (normal use case when no iterators are pinned)

Before this change
```
DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="newiterator" --db="/tmp/rocksdbtest-8616/dbbench" --use_existing_db --disable_auto_compactions
newiterator  :       1.006 micros/op 994013 ops/sec;
newiterator  :       0.994 micros/op 1006295 ops/sec;
newiterator  :       0.990 micros/op 1010422 ops/sec;
```

After change

```
DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="newiterator" --db="/tmp/rocksdbtest-8616/dbbench" --use_existing_db --disable_auto_compactions
newiterator  :       0.754 micros/op 1326588 ops/sec;
newiterator  :       0.759 micros/op 1317394 ops/sec;
newiterator  :       0.691 micros/op 1446704 ops/sec;
```

Test Plan: make check -j64

Reviewers: yhchiang, rven, anthony, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D52761
2016-01-11 16:48:15 -08:00
Igor Canadi
45d794068c Merge pull request #940 from yuslepukhin/fix_windows_build_signed_unsigned
Fix compile error.
2016-01-11 16:16:17 -08:00
Dmitri Smirnov
20d7902df9 Fix compile error.
Use constructor style initialization instead of a cast for
  simplicity.
2016-01-11 16:10:48 -08:00