Commit Graph

2476 Commits

Author SHA1 Message Date
Adam Retter
700f6ec3ff Ignore IntelliJ idea project files and ignore java/out folder 2014-10-21 15:52:27 +01:00
sdong
5bfb7f5d0b db_bench: seekrandom can specify --seek_nexts to read specific keys after seek.
Summary:
Add a function as tittle.
Also use the same parameter to fillseekseq too.

Test Plan: Run seekrandom using the new parameter

Reviewers: ljin, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: rven, igor, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D25035
2014-10-20 11:55:33 -07:00
Lei Jin
ff8f74c204 remove checking lower bound of level size
Summary:
as title

Test Plan:
make db_test
2014-10-17 21:18:51 -07:00
Lei Jin
2dd9bfe3a8 Sanitize block-based table index type and check prefix_extractor
Summary:
Respond to issue reported
https://www.facebook.com/groups/rocksdb.dev/permalink/651090261656158/
Change the Sanitize signature to take both DBOptions and CFOptions

Test Plan: unit test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25041
2014-10-17 21:18:36 -07:00
Yueh-Hsuan Chiang
6c66918645 Speed up DB::Open() and Version creation by limiting the number of FileMetaData initialization.
Summary:
This diff speeds up DB::Open() and Version creation by limiting the number of FileMetaData initialization. The behavior of Version::UpdateAccumulatedStats() is changed as follows:

* It only initializes the first 20 uninitialized FileMetaData from file.  This guarantees the size of the latest 20 files will always be compensated when they have any deletion entries.  Previously it may initialize all FileMetaData by loading all files at DB::Open().
* In case none the first 20 files has any data entry, UpdateAccumulatedStats() will initialize the FileMetaData of the oldest file.

Test Plan: db_test

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24255
2014-10-17 14:58:30 -07:00
Yueh-Hsuan Chiang
5db9e76644 Fix Mac compile error: C++11 forbids default arguments for lambda expressions
Summary:
Fix the following Mac compile error.
db/db_test.cc:8686:52: error: C++11 forbids default arguments for lambda expressions [-Werror,-Wlambda-extensions]
  auto gen_l0_kb = [this](int start, int size, int stride = 1) {
                                                   ^        ~
Test Plan:
db_test
2014-10-17 14:47:26 -07:00
Lei Jin
f4363fb81c Fix DynamicMemtableOptions test
Summary: as title

Test Plan: make release

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25029
2014-10-17 10:09:45 -07:00
Yueh-Hsuan Chiang
8f01bf80cc Merge pull request #353 from fyrz/FixBloomfilterRocksJava
[RocksJava] Filters getting disposed by System.gc before EOL
2014-10-17 10:08:38 -07:00
Igor Canadi
ee80fb4b4a Total memtables size counter
Summary: Added one new counter for GetProperty

Test Plan: Not sure if needs a test case. compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25023
2014-10-17 09:26:27 -07:00
Igor Canadi
c12f571d31 Fix mac compile, second try 2014-10-17 12:09:45 -04:00
Igor Canadi
d2e60f5cee Fix mac compile 2014-10-17 12:05:32 -04:00
Lei Jin
274dc81c92 fix build failure
Summary: missed default value during merge

Test Plan: ./db_test

Reviewers: igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24975
2014-10-16 17:33:09 -07:00
Lei Jin
d6c8dba727 Log MutableCFOptions in SetOptions
Summary: as title

Test Plan: make release

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24903
2014-10-16 17:22:28 -07:00
Lei Jin
4d5708aa56 dynamic soft_rate_limit and hard_rate_limit
Summary: as title

Test Plan:
unit test
I am only able to build the test case for hard_rate_limit.
soft_rate_limit is essentially the same thing as hard_rate_limit

Reviewers: igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24759
2014-10-16 17:21:31 -07:00
Lei Jin
065a67c4f0 dynamic disable_auto_compactions
Summary: Add more tests as well

Test Plan: unit test

Reviewers: igor, sdong, yhchiang

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24747
2014-10-16 17:14:17 -07:00
Lei Jin
dc50a1a593 make max_write_buffer_number dynamic
Summary: as title

Test Plan: unit test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24729
2014-10-16 16:57:59 -07:00
sdong
6a150c0118 ldb: support --fix_prefix_len
Summary:
ldb to support --fix_prefix_len to allow us to verify more cases.
Also fix a small issue that --bloom_bits might not be applied if --block_size is not given.

Test Plan: run ldb tool against an example DB.

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24819
2014-10-16 13:42:34 -07:00
fyrz
bafbc23baa Filters getting disposed by System.gc before EOL
Previous to this commit Filters passed as parameters to the
BlockTableConfig are disposed before they should be disposed.

Further Smart pointer usage was corrected.

Java holds now the smart pointer to the FilterPolicy correctly
and cares about freeing underlying c++ structures.
2014-10-15 20:59:25 +02:00
Igor Canadi
ca250d71a1 Move logging out of mutex
Summary: As title

Test Plan: compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24897
2014-10-15 10:56:50 -07:00
sdong
5cc9adf5ba WriteBatchWithIndex's Iterator bug of SeekToFirst() and SeekToLast()
Summary: WriteBatchWithIndex's iterator's SeekToFirst() and SeekToLast() use offset=0 to indicate it is smaller than all the keys, which is wrong. offset=0 will decode a key "" (the header decodes like that). It could be larger than other keys in non-default comparators. Fix it by using a special flag of offset to indicate searching to the beginning of the CF.

Test Plan: Add a unit test that used to fail. Also, add some more tests to related cases, though they don't fail for now.

Reviewers: igor

Reviewed By: igor

Subscribers: rven, yhchiang, ljin, leveldb

Differential Revision: https://reviews.facebook.net/D24873
2014-10-15 07:33:32 -07:00
Yueh-Hsuan Chiang
7658bcc1e5 Merge pull request #352 from fyrz/OptionsTestMergeProblem
Options correction related to merge
2014-10-13 22:56:06 -07:00
fyrz
1b97934a2c Options correction 2014-10-14 07:33:57 +02:00
Igor Canadi
cc6c883f59 Stop stopping writes on bg_error_
Summary: This might have caused https://github.com/facebook/rocksdb/issues/345. If we're stopping writes and bg_error comes along, we will never unblock the write.

Test Plan: compiles

Reviewers: ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24807
2014-10-13 14:25:55 -07:00
Igor Canadi
4942f6efb6 Merge pull request #350 from fyrz/reenable_accidentally_disabled_test
[RocksJava] Options-Test was accidentally disabled
2014-10-13 12:28:21 -07:00
fyrz
ee28f431d0 With the last commit a Test was accidentally disabled. This commit solves this. 2014-10-13 21:12:42 +02:00
Yueh-Hsuan Chiang
b5dd7eed68 Merge pull request #319 from fyrz/column_families_java
Column family support in RocksJava
2014-10-13 10:25:01 -07:00
fyrz
18004d2f20 [RocksJava] Column family support
This commit includes the support for the following functionalities:

 - Single Get/Put operations
 - WriteBatch operations
 - Single iterator functionality
 - Open database with column families
 - Open database with column families Read/Only
 - Create column family
 - Drop column family
 - Properties of column families
 - Listing of column families
 - Fully backwards comptabile implementation
 - Multi Iterator support
 - MultiGet
 - KeyMayExist
 - Option to create missing column families on open

In addition there is are two new Tests:

 - Test of ColumnFamily functionality
 - Test of Read only feature to open subsets of column families
 - Basic test to test the KeyMayExist feature

What is not supported currently using RocksJava:

 - Custom ColumnFamilyOptions

The following targets work as expected:

 - make rocksdbjava
 - make jtest

Test environment: Ubuntu 14.04(LTS, x64), Java 1.7.0_65(OpenJDK IcedTea 2.5.2), g++ 4.8.2, kernel 3.13.0-35-generix
2014-10-13 10:34:52 +02:00
Yueh-Hsuan Chiang
5908d08a0b Merge pull request #336 from fyrz/32BitRocksJavaBug
32 Bit RocksJava Fix for overflowing jlongs
2014-10-12 19:48:54 -07:00
ankgup87
9e5f2a952b Merge pull request #344 from fyrz/java-doc-enhancements
JavaDoc improvements on RocksJava
2014-10-12 19:19:14 -07:00
fyrz
4f5a687254 32-Bit RocksJava resolution for jlong overflows
Summary:
This pull request solves the jlong overflow problem on 32-Bit machines as described in https://github.com/facebook/rocksdb/issues/278:

1. There is a new org.rocksdb.test.PlatformRandomHelper to assist in getting random values. For 32 Bit the getLong method is overriden by xpromaches code above. For 64 Bit it behaves as is.
2. The detection should be cross-platform (Windows is supported though it is not ported completely yet).
3. Every JNI method which sets jlong values must check if the value fits into size_t. If it overflows size_t a InvalidArgument Status object will be returned. If its ok a OK Status will be returned.
4. Setters which have this check will throw a RocksDBException if its no OK Status.

Additionally some other parts of code were corrected using the wrong type casts.

Test Plan:
make rocksdbjava
make jtest

Differential Revision: https://reviews.facebook.net/D24531
2014-10-12 22:42:18 +02:00
fyrz
16d2ebdbcf Minor adjustment to prevent two warnings 2014-10-12 22:41:59 +02:00
fyrz
70294c9114 JavaDoc improvements on RocksJava
Added some more documentation improvements and readability
improvements.
2014-10-12 22:41:59 +02:00
Igor Canadi
833357402c WriteBatchWithIndex supports an iterator that merge its change with a base iterator.
Summary: Add an iterator that combines base_iterator of type Iterator* with delta iterator of type WBWIIterator*.

Test Plan: nothing yet. work in progress

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: rven, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D24741
2014-10-10 19:02:58 -07:00
sdong
4f65fbd197 WriteBatchWithIndex's iterator to support SeekToFirst(), SeekToLast() and Prev()
Summary: Support SeekToFirst(), SeekToLast() and Prev() in WBWIIterator, returned by WriteBatchWithIndex::NewIterator().

Test Plan: Write unit test cases to cover the case.

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: rven, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D24765
2014-10-10 16:19:34 -07:00
sdong
f441b273ae WriteBatchWithIndex to support an option to overwrite rows when operating the same key
Summary: With a new option, when accepting a new key, WriteBatchWithIndex will find an existing index of the same key, and replace the content of it.

Test Plan: Add a unit test case.

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24753
2014-10-10 15:19:21 -07:00
Yueh-Hsuan Chiang
3ead857a0d Fixed Mac compile error in util/options_test.cc
Summary:
Fixed the following error in Mac:

./util/testharness.h:93:19: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare]
  BINARY_OP(IsEq, ==)
  ~~~~~~~~~~~~~~~~^~~
./util/testharness.h:86:14: note: expanded from macro 'BINARY_OP'
    if (! (x op y)) {                                   \
             ^
util/options_test.cc:269:3: note: in instantiation of function template specialization 'rocksdb::test::Tester::IsEq<unsigned long, int>' requested here
  ASSERT_EQ(new_cf_opt.write_buffer_size, 5);
  ^

Test Plan:
options_test
2014-10-10 14:20:59 -07:00
Yueh-Hsuan Chiang
5a76186340 Fixed compile error on Mac: default arguments for lambda expressions
Summary:
Fixed the following compile error on Mac.

db/db_test.cc:8618:52: error: C++11 forbids default arguments for lambda expressions [-Werror,-Wlambda-extensions]
  auto gen_l0_kb = [this](int start, int size, int stride = 1) {
                                                   ^        ~
1 error generated.

Test Plan:
db_test
2014-10-10 14:10:16 -07:00
Lei Jin
cd0d581ff5 convert Options from string
Summary: Allow accepting Options as a string of key/value pairs

Test Plan: unit test

Reviewers: yhchiang, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24597
2014-10-10 10:00:12 -07:00
Lei Jin
f18b4a4847 minor update to benchmark script
Summary: Try to match some parameters from Dhruba's benchmarks on github

Test Plan: ran it

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24687
2014-10-10 09:55:40 -07:00
sdong
b7d3d6ebc5 db_bench: set thread pool size according to max_background_flushes
Summary: option max_background_flushes doesn't make sense if thread pool size is not set accordingly. Set the thread pool size as what we do for max_background_compactions.

Test Plan: Run db_bench with max_background_flushes > 1

Reviewers: yhchiang, igor, rven, ljin

Reviewed By: ljin

Subscribers: MarkCallaghan, leveldb

Differential Revision: https://reviews.facebook.net/D24717
2014-10-09 20:38:15 -07:00
ankgup87
c5f54a8f33 Merge pull request #339 from fyrz/bloomFilterSupportNewFormat
RocksJava should support not only BlockBased Bloomfilter
2014-10-08 12:42:59 -07:00
fyrz
ced6129574 Improved JavaDoc 2014-10-07 22:21:34 +02:00
fyrz
5e43155b3f RocksJava should support not only BlockBased Bloomfilter 2014-10-07 22:21:34 +02:00
Igor Canadi
1d525891bd Update HISTORY for 3.6 2014-10-07 11:59:30 -07:00
Tomislav Novak
88edfd90ae SkipListRep::LookaheadIterator
Summary:
This diff introduces the `lookahead` argument to `SkipListFactory()`. This is an
optimization for the tailing use case which includes many seeks. E.g. consider
the following operations on a skip list iterator:

   Seek(x), Next(), Next(), Seek(x+2), Next(), Seek(x+3), Next(), Next(), ...

If `lookahead` is positive, `SkipListRep` will return an iterator which also
keeps track of the previously visited node. Seek() then first does a linear
search starting from that node (up to `lookahead` steps). As in the tailing
example above, this may require fewer than ~log(n) comparisons as with regular
skip list search.

Test Plan:
Added a new benchmark (`fillseekseq`) which simulates the usage pattern. It
first writes N records (with consecutive keys), then measures how much time it
takes to read them by calling `Seek()` and `Next()`.

   $ time ./db_bench -num 10000000 -benchmarks fillseekseq -prefix_size 1 \
      -key_size 8 -write_buffer_size $[1024*1024*1024] -value_size 50 \
      -seekseq_next 2 -skip_list_lookahead=0
   [...]
   DB path: [/dev/shm/rocksdbtest/dbbench]
   fillseekseq  :       0.389 micros/op 2569047 ops/sec;

   real    0m21.806s
   user    0m12.106s
   sys     0m9.672s

   $ time ./db_bench [...] -skip_list_lookahead=2
   [...]
   DB path: [/dev/shm/rocksdbtest/dbbench]
   fillseekseq  :       0.153 micros/op 6540684 ops/sec;

   real    0m19.469s
   user    0m10.192s
   sys     0m9.252s

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb, march, lovro

Differential Revision: https://reviews.facebook.net/D23997
2014-10-07 11:48:23 -07:00
Igor Canadi
6a443309d8 Merge pull request #342 from fyrz/java_makefile_fix
RocksJava Makefile includes incorrect paths to version.h
2014-10-07 11:47:23 -07:00
fyrz
4f272408c5 RocksJava Makefile includes incorrect paths to version.h 2014-10-07 20:43:04 +02:00
Igor Canadi
f78b832e5d Log RocksDB version
Summary: This will be much easier than reviewing git sha's we currently have in our LOGs

Test Plan: none

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24591
2014-10-07 10:40:57 -07:00
Lei Jin
25f6a852e4 add db_test for changing memtable size
Summary:
The test only covers changing write_buffer_size. Other changable
parameters such bloom bits/probes are not obvious how to test.
Suggestions are welcome

Test Plan: db_test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24429
2014-10-07 10:40:45 -07:00
ankgup87
daab6dc511 Merge pull request #318 from criccomini/master
Build RocksDB JNI cross-platform fat jar
2014-10-07 09:53:20 -07:00