Summary:
Before this change overflowing size_t values led to a checked Exception.
After that change:
size_t overflows on 32-Bit architecture throw now an IllegalArgumentException,
which removes the necessity for a developer to catch these Exceptions explicitly.
This is especially an advantage for developers targeting 64-Bit systems because
it is not necessary anymore to catch exceptions which are never thrown on a 64-Bit
system.
Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34923
Summary:
table_cache_remove_scan_count_limit is marked as deprecated in RocksDB C++.
(see rocksdb/options.h). This patch made necessary changes on RocksJava
side.
Test Plan:
make rocksdbjava -j32
make jtest
Reviewers: rven, igor, fyrz, adamretter, ankgup87, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35355
Summary: It is no longer used by the implementation, so we should also remove it from the public API.
Test Plan: make check
Reviewers: sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D34971
Summary:
Within this commit a new AbstractLogger was introduced
which allows to handle log messages at an application level.
Log messages are passed up to Java using a JNI callback.
This allows a Java-Developer to use common Java APIs for log
messages e.g. SLF4J, LOG4J, etc. Within this commit no new
dependencies were introduced, which keeps the RocksDB API clean
and doesn`t force a developer to use a predefined high-level Java API.
Another feature is to dynamically set a custom loggers verbosity at
runtime using its public method `setInfoLogLevel` and to retrieve
the currently active level using the `infoLogLevel` method.
Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34755
Summary:
Cleaned up some tests regarding disposal order and tests
which were failing when C++ assertions were enabled.
Test Plan:
- Enable C++ Assertions (remove e.g. -DNDebug in rocksdbjava target)
- make rocksdbjava jtest
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34569
In the current implementation DefaultColumnFamily will not disown
the native handle. As the database handles the lease on the native
handle this should be the case.
Summary:
Introduced final keyword to parameters with immutable values and classes
which should not be derived.
Test Plan:
make rocksdbjava
make jtest
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33417
Summary:
Bytes are currently misinterpreted by the Java if the
byte array contains zero bytes within its content. For Strings
thats usually not useful. As the Java API allows every kind
of byte array values it might be the case that zero padding might
happen.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33165
from either java.library.path or from extracting from the Jar. Means
that the test in the build do not need to rely on the Jar, useful when
creating similar builds (and executing tests) from Maven
Summary:
As the C++ part exposes now SequenceNumber retrieval
for Snapshots we want this obviously also in the Java API.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom test
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32571
Summary:
This differential describes further changes to the Java-API
New methods:
* GetUpdatesSince
* GetLatestSequenceNumber
* EnableFileDeletions
* DisableFileDeletions
This pull requests depends on: https://github.com/facebook/rocksdb/pull/472
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32151
Summary:
Simple Java Native Objects usually are represented using
the same functionality but within different classes.
With this commit a template class was introduced to remove
the redundant impelementation to a certain extent.
[RocksJava] Removed todo comment in portal.h
As jclass instances shall not be cached, both
todos are obsolete and can be removed.
[RocksJava] Add missing test to Makefile
[RocksJava] Added tests for uncovered methods
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32145
Simple Java Native Objects usually are represented using
the same functionality but within different classes.
With this commit a template class was introduced to remove
the redundant impelementation to a certain extent.
Summary:
Prior to this the native library loader instance didn`t
care about a state. So if library loading was called multiple
times, multiple copies of the shared object were put into
the tmp folder and loaded into the JVM.
This changed within this commit to the following behavior:
- library loading is now synchronized
- library is loaded within the first call
- if loading was successful the library loaded sets a flag
- every subsequent call checks for a boolean flag indicating if there was
already a successful attempt
Test Plan:
- Execute example and watch tmp folder while the example is running
- After this patch only one shared object will be in the tmp folder
Usual tests:
- make rocksdbjava jtest
- mvn -f rocksjni.pom package
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32133
The methods:
- newIterator
- iterators
support now also ReadOptions. That allows a user of the Java API
to retrieve RocksIterator instances on a snapshot.
Summary:
Previous to this commit there was a problem with unterminated
String usage as jByteArrays are not zero terminated.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D31809
Summary:
TTLDB Support exposed in Java-API. It is now
possible to open a datbase using the RocksDB time
to live feature.
Test Plan:
make rocksdbjava
make test
mvn -f rocksjni.pom package
@Adam please test mac osx compile
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba, adam
Differential Revision: https://reviews.facebook.net/D31449
Summary:
Previous to this commit ColumnFamilyDescriptor took a String as name for the ColumnFamily name. String is however encoding dependent which is bad because listColumnFamilies returns byte arrays without any encoding information.
All public API call were deprecated and flagged to be removed in 3.10.0
Test Plan:
make rocksdbjava
make test
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D30525
Summary:
Add support to allow nested config for block-based table factory. The format looks like this:
"write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2"
Test Plan: unit test
Reviewers: yhchiang, rven, igor, ljin, jonahcohen
Reviewed By: jonahcohen
Subscribers: jonahcohen, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29223
Summary:
- AssertionError when initialized with Non-Direct Buffer
- Tests + coverage for DirectSlice
- Slice sigsegv fixes when initializing from String and byte arrays
- Slice Tests
Test Plan: Run tests without source modifications.
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D30081
Summary: Manual range compaction support in RocksJava.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D29283
Summary:
- Addressed some FindBugs issues.
- Remove obsolete dbFolder cleanup
- Comparator tests for CF
- Added AbstractComparatorTest.
- Fixed a bug in the JNI Part about Java comparators
- Minor test improvements
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D29571
Summary:
RocksDB supports two ways of saving snapshots. In
memory and on disk. The later was added with this
pull request to RocksJava.
Test Plan:
Reviewers:
Subscribers:
Previous to this commit too much targets got dependencies
on javadocs target.
Introduced one additional target "javalib" which resolves
that situation. JavaDoc will now be generated once while
executing a task with prefix "rocksdbjava".
RocksDB introduced in 3.7.0 convenience methods
for getting ColumnFamilyOptions and DBOptions
instances from predefined configuration structures.
There is now also a method in RocksJava to load DBOptions
as well as ColumnFamilyOptions from a predefined Properties
based configuration.
Summary:
The very last reference happens in DBImpl::GetOptions()
I built with both DBImpl::GetOptions() and ColumnFamilyData::options() commented out
Test Plan: make all check
Reviewers: sdong, yhchiang, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29073
Expose GetIntProperty methods to RocksJava. As the integer(64-Bit)
value is no integer in Java the method is aligned with the return
type which is long.
Summary:
As we had the discussion some weeks ago. Java needs a test framework and should support code coverage analysis. This pull request includes:
Move Tests from main method functionality to Junit4
Move WriteBatchTest to test package
Adjust the Makefile to run Junit4
Download dependencies from Make (once if not-present)
Adjustment of the rocksjni.pom to run coverage analysis using jacoco
Javadoc excludes now tests
Two bugfixes regarding GC cleanup which came up within the test runs
Make can be used as beforehand to build and run RocksJava. make test runs tests using the command-line version of Junit4.
Maven can be used to retrieve code coverage reports using mvn -f rocksjni.pom package. Code coverage reports can then be found as usual in the site folder.
Testing libraries available within Java
Junit4 (incl. hamcrest-core dependency)
AssertJ (providing fluent syntax for assertions, cglib dependency)
Mockito to provide mocktests
Libraries as said before are not statically within this commit or filesystem instead they are downloaded using curl. Make checks if files are present, if so it will perform tests without downloading the libraries again.
Note: Libraries are only necessary to compile & run tests.
Next steps after merge:
Get the maven build into travis-ci and coveralls.io
Filling up the missing test spots (based on coverage data)
Test Plan:
make rocksdbjava
make jtest
Reviewers: yhchiang, ankgup87, adamretter
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D28209
Summary: So iOS size_t is 32-bit, so we need to static_cast<size_t> any uint64_t :(
Test Plan: TARGET_OS=IOS make static_lib
Reviewers: dhruba, ljin, yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D28743
Summary:
We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.
This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.
Test Plan: compiles
Reviewers: ljin, rven, yhchiang, sdong
Reviewed By: yhchiang
Subscribers: bobbaldwin, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D28689
Summary:
ColumnFamilyHandles face the same problem as RocksIterator previously
so used methods were also applied for ColumnFamilyHandles.
Another problem with CF was that Options passed to CFs were
always filled with default values. To enable Merge, all parts
of the database must share the same merge functionality which
is not possible using default values. So from now on every
CF will inherit from db options.
Changes to RocksDB:
- merge can now take also a cfhandle
Changes to MergeTest:
- Corrected formatting
- Included also GC tests
- Extended tests to cover CF related parts
- Corrected paths to cleanup properly within the test process
- Reduced verbosity of the test
Test Plan:
make rocksdbjava
make jtest
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D27999
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:
Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).
To align BuiltinComparator with every other Enum it was moved to
a separate file.
Test Plan:
make rocksdbjava
make jtest
Summary:
- BackupableDB deleteBackup method
- BackupableDB purgeOldBackups bugfix
- BackupInfos now available in Restorable-/BackupableDB
- Extended BackupableDBTest to cover more of the currently implemented functionality.
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D27027
RocksIterator will sometimes Sigsegv on dispose. Mainly thats related
to dispose order. If the related RocksDB instance is freed beforehand
RocksIterator.dispose() will fail.
Within this commit there is a major change to RocksIterator. RocksIterator
will hold a private reference to the RocksDB instance which created the
RocksIterator. So even if RocksDB is freed in the same GC cycle the
RocksIterator instances will be freed prior to related RocksDB instances.
Another aspect targets the dispose logic if the RocksDB is freed previously
and already gc`ed. On dispose of a RocksIterator the dispose logic will check
if the RocksDB instance points to an initialized DB. If not the dispose logic
will not perform any further action.
The crash can be reproduced by using the related test provided within this
commit.
Related information: This relates to @adamretter`s facebook rocksdb-dev group
post about SigSegv on RocksIterator.dispose().
Summary:
Snapshots integration into RocksJava. Added support for the following functionalities:
- getSnapshot
- releaseSnapshot
- ReadOptions support to set a Snapshot
- ReadOptions support to retrieve Snapshot
- SnapshotTest
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D24801
Previous to this commit Filters passed as parameters to the
BlockTableConfig are disposed before they should be disposed.
Further Smart pointer usage was corrected.
Java holds now the smart pointer to the FilterPolicy correctly
and cares about freeing underlying c++ structures.
Summary:
Added support for the merge operation to RocksJava.
You can specify a merge function to be used on the current database.
The merge function can either be one of the functions defined in
utilities/merge_operators.h, which can be specified through its
corresponding name, or a user-created function that needs to be
encapsulated in a JNI object in order to be used. Examples are
provided for both use cases.
Test Plan: There are unit test in MergeTest.java
Reviewers: ankgup87
Subscribers: vladb38
Differential Revision: https://reviews.facebook.net/D24525
This commit includes the support for the following functionalities:
- Single Get/Put operations
- WriteBatch operations
- Single iterator functionality
- Open database with column families
- Open database with column families Read/Only
- Create column family
- Drop column family
- Properties of column families
- Listing of column families
- Fully backwards comptabile implementation
- Multi Iterator support
- MultiGet
- KeyMayExist
- Option to create missing column families on open
In addition there is are two new Tests:
- Test of ColumnFamily functionality
- Test of Read only feature to open subsets of column families
- Basic test to test the KeyMayExist feature
What is not supported currently using RocksJava:
- Custom ColumnFamilyOptions
The following targets work as expected:
- make rocksdbjava
- make jtest
Test environment: Ubuntu 14.04(LTS, x64), Java 1.7.0_65(OpenJDK IcedTea 2.5.2), g++ 4.8.2, kernel 3.13.0-35-generix
Summary:
This pull request solves the jlong overflow problem on 32-Bit machines as described in https://github.com/facebook/rocksdb/issues/278:
1. There is a new org.rocksdb.test.PlatformRandomHelper to assist in getting random values. For 32 Bit the getLong method is overriden by xpromaches code above. For 64 Bit it behaves as is.
2. The detection should be cross-platform (Windows is supported though it is not ported completely yet).
3. Every JNI method which sets jlong values must check if the value fits into size_t. If it overflows size_t a InvalidArgument Status object will be returned. If its ok a OK Status will be returned.
4. Setters which have this check will throw a RocksDBException if its no OK Status.
Additionally some other parts of code were corrected using the wrong type casts.
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D24531
