Commit Graph

2499 Commits

Author SHA1 Message Date
Lei Jin
122f98e0b9 dynamic max_mem_compact_level
Summary: as title

Test Plan: unit test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25347
2014-10-23 15:37:14 -07:00
Lei Jin
1fee591e74 comments for DynamicCompactionOptions test
Summary: as title

Test Plan: n/a

Reviewers: yhchiang, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24963
2014-10-23 15:35:10 -07:00
Lei Jin
574028679b dynamic max_sequential_skip_in_iterations
Summary:
This is not a critical options. Making it dynamic so that we can remove
more reference to cfd->options()

Test Plan: unit test

Reviewers: yhchiang, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24957
2014-10-23 15:34:21 -07:00
Igor Canadi
4b1786e959 Fix SIGSEGV when declaring Arena after ScopedArenaIterator 2014-10-23 12:03:19 -07:00
Igor Canadi
05b2e60ddd Merge pull request #362 from adamretter/travis-jcheck
Added Java build and tests to Travis-CI build
2014-10-23 11:41:51 -07:00
Adam Retter
9383922cc3 Added java tests to travis build 2014-10-23 19:35:58 +01:00
Igor Canadi
90f156402c Fix CompactBetweenSnapshots 2014-10-23 11:18:33 -07:00
Igor Canadi
2717422e21 Merge pull request #361 from adamretter/fix-yosemite-build
Fix for building RocksDB Java on Mac OS X Yosemite
2014-10-23 11:10:01 -07:00
Yueh-Hsuan Chiang
3b5fe3a1f3 Correct the log message in VersionEdit
Summary:
When VersionEdit fails in kNewFile3, previously it logs "new-file2 entry".
However, it should be "new-file3 entry."

Test Plan:
make
2014-10-23 10:44:53 -07:00
Adam Retter
c584d2b538 Fix for building RocksDB Java on Mac OS X Yosemite 2014-10-23 15:39:48 +01:00
sdong
2a8e5203d8 db_bench: --batch_size used for write benchmarks too
Summary: Now --bench_size is only used in multireadrandom tests, although the codes allow it to run in all write tests. I don't see a reason why we can't enable it.

Test Plan:
Run
   ./db_bench -benchmarks multirandomwrite --threads=5 -batch_size=16
and see the stats printed out in LOG to make sure batching really happened.

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25509
2014-10-22 18:54:12 -07:00
sdong
d755e53b87 Printing number of keys in DB Stats
Summary: It is useful to print out number of keys in DB Stats

Test Plan:
./db_bench --benchmarks fillrandom --num 1000000 -threads 16 -batch_size=16

and watch the outputs in LOG files

Reviewers: MarkCallaghan, ljin, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24513
2014-10-22 18:41:33 -07:00
Lei Jin
839c376bd1 fix table_test
Summary:
SaveValue expects an internal key but I previously added to table a
user key

Test Plan:
ran the test
2014-10-22 13:53:35 -07:00
Lei Jin
0fd985f427 Avoid reloading filter on Get() if cache_index_and_filter_blocks == false
Summary:
This fixes the case that filter policy is missing in SST file, but we
open the table with filter policy on and cache_index_and_filter_blocks =
false. The current behavior is that we will try to load it every time on
Get() but fail.

Test Plan: unit test

Reviewers: yhchiang, igor, rven, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25455
2014-10-22 11:52:35 -07:00
Yueh-Hsuan Chiang
e11a5e776f Improve the comment of util/thread_local.h
Summary: Improve the comment of util/thread_local.h

Test Plan: n/a

Reviewers: ljin

Reviewed By: ljin

Differential Revision: https://reviews.facebook.net/D25449
2014-10-21 17:28:31 -07:00
Igor Canadi
6398e6a6a5 Fix DeleteFile() + enable deleting files oldest files in level 0
Summary:
DeleteFile() call was broken for non-default column family. This fixes it. We might need this feature for mongo.

I also introduced a possibility of deleting oldest file in level 0.

Test Plan: added unit test to deletefile_test

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24909
2014-10-21 11:23:06 -07:00
Yueh-Hsuan Chiang
0d3145198e Merge pull request #231 from adamretter/master
Java API for Comparator and Slice
2014-10-21 10:26:01 -07:00
Adam Retter
a6fb7f312d Fix code review comments raised in https://reviews.facebook.net/D22779 2014-10-21 15:52:28 +01:00
Adam Retter
c63494fb61 Tests for ComparatorOptions, Comparator and DirectComparator, and by
proxy we also exercise Slice and DirectSlice
2014-10-21 15:52:27 +01:00
Adam Retter
5e25274110 Fix code style problems identified by lint 2014-10-21 15:52:27 +01:00
Adam Retter
25641bfc9c Fix to memory dealocation when creating a slice from a byte buffer 2014-10-21 15:52:27 +01:00
Adam Retter
fc12cb83f2 Add locking to comparator jni callback methods which must be thread-safe 2014-10-21 15:52:27 +01:00
Adam Retter
d6fe8dacc8 Feature - Implement Java API for Comparator and Slice. Allows use of
either byte[] or DirectByteBuffer for accessing underlying data.
2014-10-21 15:52:27 +01:00
Adam Retter
700f6ec3ff Ignore IntelliJ idea project files and ignore java/out folder 2014-10-21 15:52:27 +01:00
sdong
5bfb7f5d0b db_bench: seekrandom can specify --seek_nexts to read specific keys after seek.
Summary:
Add a function as tittle.
Also use the same parameter to fillseekseq too.

Test Plan: Run seekrandom using the new parameter

Reviewers: ljin, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: rven, igor, yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D25035
2014-10-20 11:55:33 -07:00
Lei Jin
ff8f74c204 remove checking lower bound of level size
Summary:
as title

Test Plan:
make db_test
2014-10-17 21:18:51 -07:00
Lei Jin
2dd9bfe3a8 Sanitize block-based table index type and check prefix_extractor
Summary:
Respond to issue reported
https://www.facebook.com/groups/rocksdb.dev/permalink/651090261656158/
Change the Sanitize signature to take both DBOptions and CFOptions

Test Plan: unit test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25041
2014-10-17 21:18:36 -07:00
Yueh-Hsuan Chiang
6c66918645 Speed up DB::Open() and Version creation by limiting the number of FileMetaData initialization.
Summary:
This diff speeds up DB::Open() and Version creation by limiting the number of FileMetaData initialization. The behavior of Version::UpdateAccumulatedStats() is changed as follows:

* It only initializes the first 20 uninitialized FileMetaData from file.  This guarantees the size of the latest 20 files will always be compensated when they have any deletion entries.  Previously it may initialize all FileMetaData by loading all files at DB::Open().
* In case none the first 20 files has any data entry, UpdateAccumulatedStats() will initialize the FileMetaData of the oldest file.

Test Plan: db_test

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24255
2014-10-17 14:58:30 -07:00
Yueh-Hsuan Chiang
5db9e76644 Fix Mac compile error: C++11 forbids default arguments for lambda expressions
Summary:
Fix the following Mac compile error.
db/db_test.cc:8686:52: error: C++11 forbids default arguments for lambda expressions [-Werror,-Wlambda-extensions]
  auto gen_l0_kb = [this](int start, int size, int stride = 1) {
                                                   ^        ~
Test Plan:
db_test
2014-10-17 14:47:26 -07:00
Lei Jin
f4363fb81c Fix DynamicMemtableOptions test
Summary: as title

Test Plan: make release

Reviewers: igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25029
2014-10-17 10:09:45 -07:00
Yueh-Hsuan Chiang
8f01bf80cc Merge pull request #353 from fyrz/FixBloomfilterRocksJava
[RocksJava] Filters getting disposed by System.gc before EOL
2014-10-17 10:08:38 -07:00
Igor Canadi
ee80fb4b4a Total memtables size counter
Summary: Added one new counter for GetProperty

Test Plan: Not sure if needs a test case. compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D25023
2014-10-17 09:26:27 -07:00
Igor Canadi
c12f571d31 Fix mac compile, second try 2014-10-17 12:09:45 -04:00
Igor Canadi
d2e60f5cee Fix mac compile 2014-10-17 12:05:32 -04:00
Lei Jin
274dc81c92 fix build failure
Summary: missed default value during merge

Test Plan: ./db_test

Reviewers: igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24975
2014-10-16 17:33:09 -07:00
Lei Jin
d6c8dba727 Log MutableCFOptions in SetOptions
Summary: as title

Test Plan: make release

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24903
2014-10-16 17:22:28 -07:00
Lei Jin
4d5708aa56 dynamic soft_rate_limit and hard_rate_limit
Summary: as title

Test Plan:
unit test
I am only able to build the test case for hard_rate_limit.
soft_rate_limit is essentially the same thing as hard_rate_limit

Reviewers: igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24759
2014-10-16 17:21:31 -07:00
Lei Jin
065a67c4f0 dynamic disable_auto_compactions
Summary: Add more tests as well

Test Plan: unit test

Reviewers: igor, sdong, yhchiang

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24747
2014-10-16 17:14:17 -07:00
Lei Jin
dc50a1a593 make max_write_buffer_number dynamic
Summary: as title

Test Plan: unit test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24729
2014-10-16 16:57:59 -07:00
sdong
6a150c0118 ldb: support --fix_prefix_len
Summary:
ldb to support --fix_prefix_len to allow us to verify more cases.
Also fix a small issue that --bloom_bits might not be applied if --block_size is not given.

Test Plan: run ldb tool against an example DB.

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24819
2014-10-16 13:42:34 -07:00
fyrz
bafbc23baa Filters getting disposed by System.gc before EOL
Previous to this commit Filters passed as parameters to the
BlockTableConfig are disposed before they should be disposed.

Further Smart pointer usage was corrected.

Java holds now the smart pointer to the FilterPolicy correctly
and cares about freeing underlying c++ structures.
2014-10-15 20:59:25 +02:00
Igor Canadi
ca250d71a1 Move logging out of mutex
Summary: As title

Test Plan: compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24897
2014-10-15 10:56:50 -07:00
sdong
5cc9adf5ba WriteBatchWithIndex's Iterator bug of SeekToFirst() and SeekToLast()
Summary: WriteBatchWithIndex's iterator's SeekToFirst() and SeekToLast() use offset=0 to indicate it is smaller than all the keys, which is wrong. offset=0 will decode a key "" (the header decodes like that). It could be larger than other keys in non-default comparators. Fix it by using a special flag of offset to indicate searching to the beginning of the CF.

Test Plan: Add a unit test that used to fail. Also, add some more tests to related cases, though they don't fail for now.

Reviewers: igor

Reviewed By: igor

Subscribers: rven, yhchiang, ljin, leveldb

Differential Revision: https://reviews.facebook.net/D24873
2014-10-15 07:33:32 -07:00
Yueh-Hsuan Chiang
7658bcc1e5 Merge pull request #352 from fyrz/OptionsTestMergeProblem
Options correction related to merge
2014-10-13 22:56:06 -07:00
fyrz
1b97934a2c Options correction 2014-10-14 07:33:57 +02:00
Igor Canadi
cc6c883f59 Stop stopping writes on bg_error_
Summary: This might have caused https://github.com/facebook/rocksdb/issues/345. If we're stopping writes and bg_error comes along, we will never unblock the write.

Test Plan: compiles

Reviewers: ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D24807
2014-10-13 14:25:55 -07:00
Igor Canadi
4942f6efb6 Merge pull request #350 from fyrz/reenable_accidentally_disabled_test
[RocksJava] Options-Test was accidentally disabled
2014-10-13 12:28:21 -07:00
fyrz
ee28f431d0 With the last commit a Test was accidentally disabled. This commit solves this. 2014-10-13 21:12:42 +02:00
Yueh-Hsuan Chiang
b5dd7eed68 Merge pull request #319 from fyrz/column_families_java
Column family support in RocksJava
2014-10-13 10:25:01 -07:00
fyrz
18004d2f20 [RocksJava] Column family support
This commit includes the support for the following functionalities:

 - Single Get/Put operations
 - WriteBatch operations
 - Single iterator functionality
 - Open database with column families
 - Open database with column families Read/Only
 - Create column family
 - Drop column family
 - Properties of column families
 - Listing of column families
 - Fully backwards comptabile implementation
 - Multi Iterator support
 - MultiGet
 - KeyMayExist
 - Option to create missing column families on open

In addition there is are two new Tests:

 - Test of ColumnFamily functionality
 - Test of Read only feature to open subsets of column families
 - Basic test to test the KeyMayExist feature

What is not supported currently using RocksJava:

 - Custom ColumnFamilyOptions

The following targets work as expected:

 - make rocksdbjava
 - make jtest

Test environment: Ubuntu 14.04(LTS, x64), Java 1.7.0_65(OpenJDK IcedTea 2.5.2), g++ 4.8.2, kernel 3.13.0-35-generix
2014-10-13 10:34:52 +02:00