Commit Graph

4702 Commits

Author SHA1 Message Date
Adam Retter
0f2d2fcff6 Refactored tests to use try-with-resources 2016-03-01 16:00:45 +00:00
Adam Retter
f8e02c7825 Deprecate org.rocksdb.AbstractNativeReference#dispose() and implement java.lang.AutoCloseable 2016-03-01 16:00:45 +00:00
Adam Retter
0f2fdfe23a Fix the javadoc and the formatting of the base-classes for objects with native references 2016-03-01 16:00:44 +00:00
Adam Retter
c5af85ecad Fix a memory leak of Slice objects from org.rocksdb.WBWIRocksIterator#entry1 2016-02-10 14:44:39 +00:00
Adam Retter
e84137c8ae Remove unnessecary java.util.List expense in JNI 2016-02-10 14:44:39 +00:00
Adam Retter
76e8beeeb9 Pass by pointer from/to Java from JNI not by object 2016-02-10 14:44:39 +00:00
Adam Retter
0e7e6f6e4b Improve Javadoc 2016-02-10 14:44:39 +00:00
Adam Retter
18eb563058 Improve the speed and synchronization around the construction of Java/JNI objects 2016-02-10 14:44:39 +00:00
Yueh-Hsuan Chiang
2a04268be3 Temporarily disable unstable tests in memory_test.cc
Summary:
memory_test.cc has some tests that are not unstable but
hard to reproduce, and the cause is the test itself not
the code.  Temporarily disable the tests until
we have a good fix.

Test Plan: memory_test

Reviewers: sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54009
2016-02-09 16:28:34 -08:00
Igor Canadi
08a78b6b44 Merge pull request #979 from facebook/update_licenses
Updated all copyright headers to the new format.
2016-02-09 15:46:18 -08:00
Baraa Hamodi
21e95811d1 Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
Andrew Kryczka
59b3ee658f Env function for bulk metadata retrieval
Summary:
Added this new function, which returns filename, size, and modified
timestamp for each file in the provided directory. The default implementation
retrieves the metadata sequentially using existing functions. In the next diff
I'll make HdfsEnv override this function to use libhdfs's bulk get function.

This won't work on windows due to the path separator.

Test Plan:
new unit test

  $ ./env_test --gtest_filter=EnvPosixTest.ConsistentChildrenMetadata

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: IslamAbdelRahman, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53781
2016-02-09 14:54:32 -08:00
Yueh-Hsuan Chiang
4a8cbf4e31 Allows Get and MultiGet to read directly from SST files.
Summary:
Add kSstFileTier to ReadTier, which allows Get and MultiGet to
read only directly from SST files and skip mem-tables.

    kSstFileTier = 0x2      // data in SST files.
                          // Note that this ReadTier currently only supports
                          // Get and MultiGet and does not support iterators.

Test Plan: add new test in db_test.

Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: igor, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53511
2016-02-09 11:20:22 -08:00
agiardullo
fe93bf9b5d Transaction::UndoGetForUpdate
Summary: MyRocks wants to be able to un-lock a key that was just locked by GetForUpdate().  To do this safely, I am now keeping track of the number of reads(for update) and writes for each key in a transaction.  UndoGetForUpdate() will only unlock a key if it hasn't been written and the read count reaches 0.

Test Plan: more unit tests

Reviewers: igor, rven, yhchiang, spetrunia, sdong

Reviewed By: spetrunia, sdong

Subscribers: spetrunia, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47043
2016-02-09 10:46:11 -08:00
sdong
2608219cc9 crash_test: cover concurrent memtable insert in default crash test
Summary: Default crash test uses prefix hash memtable, which is not compatible to concurrent memtable. Allow prefix test run with skip list and use skip list memtable when concurrent insert is used.

Test Plan: Run "python -u tools/db_crashtest.py whitebox" and watch sometimes skip list is used.

Reviewers: anthony, yhchiang, kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53907
2016-02-09 08:02:38 -08:00
sdong
a76e9093f0 Fix LITE db_test build broken by previous commit
Summary: Previous commit introduces a test that is not supported in LITE. Fix it.

Test Plan: Build the test with ROCKSDB_LITE.

Reviewers: kradhakrishnan, IslamAbdelRahman, anthony, yhchiang, andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53901
2016-02-05 14:29:09 -08:00
sdong
b1887c5dd9 Explictly fail when memtable doesn't support concurrent insert
Summary: If users turn on concurrent insert but the memtable doesn't support it, they might see unexcepted crash. Fix it by explicitly fail.

Test Plan:
Run different setting of stress_test and make sure it fails correctly.
Will add a unit test too.

Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, andrewkr, ngbronson

Reviewed By: ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53895
2016-02-05 14:15:50 -08:00
Gunnar Kudrjavets
8ed3438778 Add option to run fillseq with WAL enabled in addition to WAL disabled
Summary: This set of changes is part of the work to introduce benchmark for universal style compaction in RocksDB. It's conceptually separate from the compaction work, so sending it out as a separate diff to get it out of the way.

Test Plan:
  - Run `./tools/run_flash_bench.sh`.
  - Look at the contents of `report.txt` and `report2.txt` to make sure that data is reported and attributed correctly.
  - During `db_bench` execution time make sure that the correct flags are passed to `--disable_wal` depending on the benchmark being executed.

Reviewers: MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53865
2016-02-05 13:20:56 -08:00
sdong
73a9b0f4ba Update version to 4.5
Summary: Time to cut branch for release 4.5. Change the versions.

Test Plan: Not needed

Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53883
2016-02-05 13:05:54 -08:00
reid horuff
6f71d3b68b Improve perf of Pessimistic Transaction expirations (and optimistic transactions)
Summary:
copy from task 8196669:

1) Optimistic transactions do not support batching writes from different threads.
2) Pessimistic transactions do not support batching writes if an expiration time is set.

In these 2 cases, we currently do not do any write batching in DBImpl::WriteImpl() because there is a WriteCallback that could decide at the last minute to abort the write.  But we could support batching write operations with callbacks if we make sure to process the callbacks correctly.

To do this, we would first need to modify write_thread.cc to stop preventing writes with callbacks from being batched together.  Then we would need to change DBImpl::WriteImpl() to call all WriteCallback's in a batch, only write the batches that succeed, and correctly set the state of each batch's WriteThread::Writer.

Test Plan: Added test WriteWithCallbackTest to write_callback_test.cc which creates multiple client threads and verifies that writes are batched and executed properly.

Reviewers: hermanlee4, anthony, ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52863
2016-02-05 10:44:13 -08:00
Islam AbdelRahman
8e6172bc57 Add BlockBasedTableOptions::index_block_restart_interval
Summary: Add a new option to BlockBasedTableOptions that will allow us to change the restart interval for the index block

Test Plan: unit tests

Reviewers: yhchiang, anthony, andrewkr, sdong

Reviewed By: sdong

Subscribers: march, dhruba

Differential Revision: https://reviews.facebook.net/D53721
2016-02-05 10:22:37 -08:00
sdong
34a40bf911 Add --allow_concurrent_memtable_write in stress test and run it in crash_test
Summary: Add an option of --allow_concurrent_memtable_write in stress test and cover it in crash test

Test Plan: Run crash test and make sure three combinations of the two options show up randomly.

Reviewers: IslamAbdelRahman, yhchiang, andrewkr, anthony, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53811
2016-02-04 16:15:18 -08:00
Siying Dong
73bf330c79 Merge pull request #973 from yuslepukhin/moveout_testcode
Move out of release build some test code
2016-02-04 15:50:21 -08:00
sdong
f7c0f4e3ef perf_context.cc and iostats_context.cc use different output macro (fix unity build)
Summary: Unity build because perf_context.cc and iostats_context.cc define a different OUTPUT macro. Fix it.

Test Plan: Build unity

Reviewers: kradhakrishnan, anthony, yhchiang, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: divchenko, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53769
2016-02-03 17:27:08 -08:00
Dmitri Smirnov
9656eab005 This partially addresses issue https://github.com/facebook/rocksdb/issues/935
testutil.cc and testharness.cc could not be moved out at this time
  as they are used by 4 benchmarks in release builds.
2016-02-03 17:06:10 -08:00
Jonathan Wiepert
14a322033f Remove references to files deleted in commit abb4052278
Summary:
Remove obolete references to files in src.mk
Fix incorrect path for reference in source.mk

Test Plan: Ran build to ensure changes do not break anything.

Reviewers: leveldb, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53733
2016-02-03 16:47:45 -08:00
krad
8445e53807 Add a mechanism to run all tests in sandcastle
Summary:
When making environment specific changes, it is better to run all CI
tests. This diff provides a mechanism to do that

Format is:

ROCKSDB_CHECK_ALL=1 arc diff

Test Plan: Submit request for diff

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53631
2016-02-03 16:34:08 -08:00
Igor Canadi
461cec4e8e Merge pull request #972 from adamretter/wb-threads
Threaded tests for WriteBatch
2016-02-03 12:32:46 -08:00
Adam Retter
9ab269ab39 Threaded tests for WriteBatch 2016-02-03 20:15:49 +00:00
krad
bf767c6411 Minor fix to makefile
Summary: unit_481 is misspelt. Fixing it.

Test Plan: Running make commit_prereq

Reviewers: leveldb

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53757
2016-02-03 11:41:23 -08:00
Nathan Bronson
2c1db5ea51 always invalidate sequential-insertion cache for concurrent skiplist adds
Summary:
InlineSkipList::InsertConcurrently should invalidate the
sequential-insertion cache prev_[] for all inserts of multi-level nodes,
not just those that increase the height of the skip list.  The invariant
for prev_ is that prev_[i] (i > 0) is supposed to be the predecessor of
prev_[0] at level i.  Before this diff InsertConcurrently could violate
this constraint when inserting a multi-level node after prev_[i] but
before prev_[0].

This diff also reenables kConcurrentSkipList as db_test's
MultiThreaded/MultiThreadedDBTest.MultiThreaded/29.

Test Plan:
1. unit tests
2. temporarily hack kConcurrentSkipList timing so that it is fast but has a 1.5% failure rate on my dev box (1ms stagger on thread launch, 1s test duration, failure rate baseline over 1000 runs)
3. observe 1000 passes post-fix

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D53751
2016-02-03 11:08:16 -08:00
Siying Dong
c12ff20ab2 Merge pull request #965 from koldat/jni_for_windows
Adding support for Windows JNI build
2016-02-03 10:41:57 -08:00
Gunnar Kudrjavets
a09ce4fcd3 Skip some of the non-critical tests in ./tools/run_flash_bench.sh
Summary:
Some of the tests aren't considered to be critical when it comes to getting key benchmarking data for RocksDB. Therefore we'll introduce an environment variable `SKIP_LOW_PRI_TESTS` which enables skipping those test cases. By default all the tests will be run. If you want to optimize the test-case execution then do the following:

`
$ export SKIP_LOW_PRI_TESTS=1
$ ./tools/run_flash_bench.sh
`

Test Plan: Verified that when  `SKIP_LOW_PRI_TESTS` is not set then `benchmark.sh` is called for all the scenarios and when `SKIP_LOW_PRI_TESTS` is set to `1` then `benchmark.sh` is called only for the test-cases which are critical.

Reviewers: MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53739
2016-02-03 09:56:56 -08:00
Andrew Kryczka
284aa613a7 Eliminate duplicated property constants
Summary:
Before this diff, there were duplicated constants to refer to properties (user-
facing API had strings and InternalStats had an enum). I noticed these were
inconsistent in terms of which constants are provided, names of constants, and
documentation of constants. Overall it seemed annoying/error-prone to maintain
these duplicated constants.

So, this diff gets rid of InternalStats's constants and replaces them with a map
keyed on the user-facing constant. The value in that map contains a function
pointer to get the property value, so we don't need to do string matching while
holding db->mutex_. This approach has a side benefit of making many small
handler functions rather than a giant switch-statement.

Test Plan: db_properties_test passes, running "make commit-prereq -j32"

Reviewers: sdong, yhchiang, kradhakrishnan, IslamAbdelRahman, rven, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53253
2016-02-02 19:14:56 -08:00
Gabriela Jacques da Silva
94be872eab Merge branch 'master' of github.com:facebook/rocksdb 2016-02-02 18:38:16 -08:00
Gabriela Jacques da Silva
0c2bd5cb4b Removing data race from expirable transactions
Summary:
Doing inline checking of transaction expiration instead of
using a callback.

Test Plan: To be added

Reviewers: anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53673
2016-02-02 18:37:44 -08:00
Nathan Bronson
5fcd1ba30a disable kConcurrentSkipList multithreaded test
Summary: Disable test that is intermittently failing

Test Plan: unit tests

Reviewers: igor, andrewkr, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53715
2016-02-02 18:24:47 -08:00
Andrew Kryczka
466c2c1bf7 Generate tags for *.c files
Summary:
db/c_test.c uses the functions in db/c.cc. If we have tags generated
for one but not the other, it's easy to make mistakes like updating a function
signature and missing a call site.

Test Plan:
  $ make tags

in vim:

  :cscope find s rocksdb_options_set_compression_options
  ...
  3    325  db/c_test.c <<main>>
            rocksdb_options_set_compression_options(options, -14, -1, 0);

Reviewers: sdong, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53685
2016-02-02 14:52:49 -08:00
Siying Dong
70c068c97a Merge pull request #960 from koldat/masterFixes2
Making use of GetSystemTimePreciseAsFileTime dynamic
2016-02-02 13:43:09 -08:00
Tomas Kolda
a62c519bb6 RollLogFile tries to find non conflicting file until there is no conflict. 2016-02-02 10:33:49 +01:00
Tomas Kolda
57a95a7001 Making use of GetSystemTimePreciseAsFileTime dynamic - code review fixes 2016-02-02 10:23:56 +01:00
Tomas Kolda
502d41f150 Making use of GetSystemTimePreciseAsFileTime dynamic to not
break compatibility with Windows 7. The issue with rotated logs
was fixed other way.
2016-02-02 10:23:56 +01:00
Tomas Kolda
52153930d8 Adding support for Windows JNI build - fix Java unit test for release build of JNI DLL 2016-02-02 09:45:18 +01:00
Tomas Kolda
e2972803a6 Adding support for Windows JNI build 2016-02-02 09:11:37 +01:00
Nathan Bronson
9c2cf9479b Fix for --allow_concurrent_memtable_write with batching
Summary:
Concurrent memtable adds were incorrectly computing
the last sequence number for a write batch group when the
write batches were not solitary.  This is the cause of
https://github.com/facebook/mysql-5.6/issues/155

Test Plan:
1. unit tests
2. new unit test
3. parallel db_bench stress tests with batch size of 10 and asserts enabled

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: IslamAbdelRahman, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D53595
2016-02-01 20:41:57 -08:00
sdong
ac3fa9a6fe Travis CI to disable ROCKSDB_LITE tests
Summary: Travis CI fails most of the times because of timing out. To unblock it, disable LITE tests in Travis CI.

Test Plan: Will see.

Reviewers: anthony, yhchiang, kradhakrishnan, IslamAbdelRahman, andrewkr

Reviewed By: andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53679
2016-02-01 18:42:01 -08:00
Siying Dong
7b943da1b2 Merge pull request #967 from SherlockNoMad/ValueSize
Add histogram for value size per operation
2016-02-01 17:59:08 -08:00
Siying Dong
b5750790e0 Merge pull request #968 from yuslepukhin/one_shot_buffer
Enable per-request buffer allocation in RandomAccessFile
2016-02-01 16:39:55 -08:00
Islam AbdelRahman
1ad8182950 Fix WriteBatchTest.ManyUpdates, WriteBatchTest.LargeKeyValue under clang
Summary:
Fix current clang failure
https://ci-builds.fb.com/view/rocksdb/job/rocksdb_clang_build/1398/console

Test Plan:
make sure that both clang and g++ compilation succeed

USE_CLANG=1 make check -j64
make check -j64

Reviewers: anthony, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53667
2016-02-01 16:07:53 -08:00
sdong
ad7ecca72d Add unit tests to verify large key/value
Summary:
Add unit tests:
(1) insert entries of 8MB key and 3GB value to DB
(2) insert entry of 3GB key and 3GB value into write batch and make sure we can read it.
(3) insert 3 billions of key-value pairs into write batch and make sure we can read it.
Disable them because not all platform can run it.

Test Plan: Run the tests

Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53619
2016-02-01 15:09:50 -08:00