Summary:
When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.
We have a special case that when you set max_background_flushes to 0, we
schedule the flush to execute on the compaction thread.
Test Plan: make check (there might be some unit tests that depend on this behavior)
Reviewers: IslamAbdelRahman, yhchiang, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D41931
Summary:
ROCKSDB_WARNING is only defined if either ROCKSDB_PLATFORM_POSIX or OS_WIN is defined. This works well for building rocksdb with its own build scripts. But this won't work when an outside project(like mongodb) doesn't define ROCKSDB_PLATFORM_POSIX.
This fix defines ROCKSDB_WARNING for all platforms. No idea if its defined correctly on non-posix,non-windows platforms but this is no worse that the current situation where this macro is missing on unexpected platforms.
This fix should hopefully fix anyone whose build broke now that we've switched from using #warning to Pragma (to support windows). Unfortunately, while mongo-rocks compiles, it ignores the Pragma and doesn't print a warning. I have not been able to figure out a way to implement this portably on all platforms.
Of course, an alternate solution would be to just get rid of ROCKSDB_WARNING and live with include file redirects indefinitely. Thoughts?
Test Plan: build rocks, build mongorocks
Reviewers: igor, kradhakrishnan, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D42477
Summary:
The option bottommost_level_compaction was introduced lately.
This option breaks the Java API behavior. To prevent the library
from doing so we set that option to a fixed value in Java.
In future we are going to remove that portion and replace the
hardcoded options using a more flexible way.
Fixed bug introduced by WriteBatchWithIndex Patch
Lately icanadi changed the behavior of WriteBatchWithIndex.
See commit: 821cff114e
This commit solves problems introduced by above mentioned commit.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, ankgup87, yhchiang
Reviewed By: yhchiang
Subscribers: igor, dhruba
Differential Revision: https://reviews.facebook.net/D40647
Summary:
Rewrite Java tests compactRangeToLevel and compactRangeToLevelColumnFamily
to make them more deterministic and robust.
Test Plan:
make rocksdbjava
make jtest
Reviewers: anthony, fyrz, adamretter, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D40941
Summary:
This diff update DB::CompactRange to use RangeCompactionOptions instead of using multiple parameters
Old CompactRange is still available but deprecated
Test Plan:
make all check
make rocksdbjava
USE_CLANG=1 make all
OPT=-DROCKSDB_LITE make release
Reviewers: sdong, yhchiang, igor
Reviewed By: igor
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D40209
Summary: Optimistic transactions supporting begin/commit/rollback semantics. Currently relies on checking the memtable to determine if there are any collisions at commit time. Not yet implemented would be a way of enuring the memtable has some minimum amount of history so that we won't fail to commit when the memtable is empty. You should probably start with transaction.h to get an overview of what is currently supported.
Test Plan: Added a new test, but still need to look into stress testing.
Reviewers: yhchiang, igor, rven, sdong
Reviewed By: sdong
Subscribers: adamretter, MarkCallaghan, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D33435
Summary:
For transactions, we are using the memtables to validate that there are no write conflicts. But after flushing, we don't have any memtables, and transactions could fail to commit. So we want to someone keep around some extra history to use for conflict checking. In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit.
After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure). It seems like the best place for this is abstracted inside the memtable_list. I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much.
This diff adds a new parameter to control how much memtable history to keep around after flushing. However, it sounds like people aren't too fond of adding new parameters. So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers. This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit. (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached). So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit).
However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions.
Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests. Added testing in memtablelist_test and planning on adding more testing here.
Reviewers: sdong, rven, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D37443
Summary: Update HISTORY.md for 3.10.0
Test Plan: no code chagne.
Reviewers: sdong, rven, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35871
Summary:
In 3.10 the C++ code was extended with a MemEnv implementation. This
is now also available in RocksJava.
Changes:
- Extraced abstract super class Env
- Introduced RocksMemEnv
- Remove unnecessary disposeInternal method. The disposal of the default environment is managed by C++ so there needs to be no disposeInternal method in Java.
- Introduced a RocksMemEnvTest, which is aligned with the C++ equivalent.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35619
Summary:
RocksDB offers the possibility to set different compression types
on a per level basis. This shall be also available using RocksJava.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D35577
Summary:
Before this change overflowing size_t values led to a checked Exception.
After that change:
size_t overflows on 32-Bit architecture throw now an IllegalArgumentException,
which removes the necessity for a developer to catch these Exceptions explicitly.
This is especially an advantage for developers targeting 64-Bit systems because
it is not necessary anymore to catch exceptions which are never thrown on a 64-Bit
system.
Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34923
Summary:
table_cache_remove_scan_count_limit is marked as deprecated in RocksDB C++.
(see rocksdb/options.h). This patch made necessary changes on RocksJava
side.
Test Plan:
make rocksdbjava -j32
make jtest
Reviewers: rven, igor, fyrz, adamretter, ankgup87, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D35355
Summary: It is no longer used by the implementation, so we should also remove it from the public API.
Test Plan: make check
Reviewers: sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D34971
Summary:
Within this commit a new AbstractLogger was introduced
which allows to handle log messages at an application level.
Log messages are passed up to Java using a JNI callback.
This allows a Java-Developer to use common Java APIs for log
messages e.g. SLF4J, LOG4J, etc. Within this commit no new
dependencies were introduced, which keeps the RocksDB API clean
and doesn`t force a developer to use a predefined high-level Java API.
Another feature is to dynamically set a custom loggers verbosity at
runtime using its public method `setInfoLogLevel` and to retrieve
the currently active level using the `infoLogLevel` method.
Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34755
Summary:
Cleaned up some tests regarding disposal order and tests
which were failing when C++ assertions were enabled.
Test Plan:
- Enable C++ Assertions (remove e.g. -DNDebug in rocksdbjava target)
- make rocksdbjava jtest
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D34569
In the current implementation DefaultColumnFamily will not disown
the native handle. As the database handles the lease on the native
handle this should be the case.
Summary:
Introduced final keyword to parameters with immutable values and classes
which should not be derived.
Test Plan:
make rocksdbjava
make jtest
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33417
Summary:
Bytes are currently misinterpreted by the Java if the
byte array contains zero bytes within its content. For Strings
thats usually not useful. As the Java API allows every kind
of byte array values it might be the case that zero padding might
happen.
Test Plan:
make rocksdbjava
make jtest
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D33165
from either java.library.path or from extracting from the Jar. Means
that the test in the build do not need to rely on the Jar, useful when
creating similar builds (and executing tests) from Maven
Summary:
As the C++ part exposes now SequenceNumber retrieval
for Snapshots we want this obviously also in the Java API.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom test
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32571
Summary:
This differential describes further changes to the Java-API
New methods:
* GetUpdatesSince
* GetLatestSequenceNumber
* EnableFileDeletions
* DisableFileDeletions
This pull requests depends on: https://github.com/facebook/rocksdb/pull/472
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32151
Summary:
Simple Java Native Objects usually are represented using
the same functionality but within different classes.
With this commit a template class was introduced to remove
the redundant impelementation to a certain extent.
[RocksJava] Removed todo comment in portal.h
As jclass instances shall not be cached, both
todos are obsolete and can be removed.
[RocksJava] Add missing test to Makefile
[RocksJava] Added tests for uncovered methods
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32145
Simple Java Native Objects usually are represented using
the same functionality but within different classes.
With this commit a template class was introduced to remove
the redundant impelementation to a certain extent.
Summary:
Prior to this the native library loader instance didn`t
care about a state. So if library loading was called multiple
times, multiple copies of the shared object were put into
the tmp folder and loaded into the JVM.
This changed within this commit to the following behavior:
- library loading is now synchronized
- library is loaded within the first call
- if loading was successful the library loaded sets a flag
- every subsequent call checks for a boolean flag indicating if there was
already a successful attempt
Test Plan:
- Execute example and watch tmp folder while the example is running
- After this patch only one shared object will be in the tmp folder
Usual tests:
- make rocksdbjava jtest
- mvn -f rocksjni.pom package
Reviewers: adamretter, ankgup87, yhchiang
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D32133
The methods:
- newIterator
- iterators
support now also ReadOptions. That allows a user of the Java API
to retrieve RocksIterator instances on a snapshot.
Summary:
Previous to this commit there was a problem with unterminated
String usage as jByteArrays are not zero terminated.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D31809
Summary:
TTLDB Support exposed in Java-API. It is now
possible to open a datbase using the RocksDB time
to live feature.
Test Plan:
make rocksdbjava
make test
mvn -f rocksjni.pom package
@Adam please test mac osx compile
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba, adam
Differential Revision: https://reviews.facebook.net/D31449
Summary:
Previous to this commit ColumnFamilyDescriptor took a String as name for the ColumnFamily name. String is however encoding dependent which is bad because listColumnFamilies returns byte arrays without any encoding information.
All public API call were deprecated and flagged to be removed in 3.10.0
Test Plan:
make rocksdbjava
make test
mvn -f rocksjni.pom package
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D30525
Summary:
Add support to allow nested config for block-based table factory. The format looks like this:
"write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2"
Test Plan: unit test
Reviewers: yhchiang, rven, igor, ljin, jonahcohen
Reviewed By: jonahcohen
Subscribers: jonahcohen, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29223
Summary:
- AssertionError when initialized with Non-Direct Buffer
- Tests + coverage for DirectSlice
- Slice sigsegv fixes when initializing from String and byte arrays
- Slice Tests
Test Plan: Run tests without source modifications.
Reviewers: yhchiang, adamretter, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D30081
Summary: Manual range compaction support in RocksJava.
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D29283
Summary:
- Addressed some FindBugs issues.
- Remove obsolete dbFolder cleanup
- Comparator tests for CF
- Added AbstractComparatorTest.
- Fixed a bug in the JNI Part about Java comparators
- Minor test improvements
Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package
Reviewers: adamretter, yhchiang, ankgup87
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D29571
Summary:
RocksDB supports two ways of saving snapshots. In
memory and on disk. The later was added with this
pull request to RocksJava.
Test Plan:
Reviewers:
Subscribers:
Previous to this commit too much targets got dependencies
on javadocs target.
Introduced one additional target "javalib" which resolves
that situation. JavaDoc will now be generated once while
executing a task with prefix "rocksdbjava".
RocksDB introduced in 3.7.0 convenience methods
for getting ColumnFamilyOptions and DBOptions
instances from predefined configuration structures.
There is now also a method in RocksJava to load DBOptions
as well as ColumnFamilyOptions from a predefined Properties
based configuration.
Summary:
The very last reference happens in DBImpl::GetOptions()
I built with both DBImpl::GetOptions() and ColumnFamilyData::options() commented out
Test Plan: make all check
Reviewers: sdong, yhchiang, igor
Reviewed By: igor
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D29073
Expose GetIntProperty methods to RocksJava. As the integer(64-Bit)
value is no integer in Java the method is aligned with the return
type which is long.
Summary:
As we had the discussion some weeks ago. Java needs a test framework and should support code coverage analysis. This pull request includes:
Move Tests from main method functionality to Junit4
Move WriteBatchTest to test package
Adjust the Makefile to run Junit4
Download dependencies from Make (once if not-present)
Adjustment of the rocksjni.pom to run coverage analysis using jacoco
Javadoc excludes now tests
Two bugfixes regarding GC cleanup which came up within the test runs
Make can be used as beforehand to build and run RocksJava. make test runs tests using the command-line version of Junit4.
Maven can be used to retrieve code coverage reports using mvn -f rocksjni.pom package. Code coverage reports can then be found as usual in the site folder.
Testing libraries available within Java
Junit4 (incl. hamcrest-core dependency)
AssertJ (providing fluent syntax for assertions, cglib dependency)
Mockito to provide mocktests
Libraries as said before are not statically within this commit or filesystem instead they are downloaded using curl. Make checks if files are present, if so it will perform tests without downloading the libraries again.
Note: Libraries are only necessary to compile & run tests.
Next steps after merge:
Get the maven build into travis-ci and coveralls.io
Filling up the missing test spots (based on coverage data)
Test Plan:
make rocksdbjava
make jtest
Reviewers: yhchiang, ankgup87, adamretter
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D28209
Summary: So iOS size_t is 32-bit, so we need to static_cast<size_t> any uint64_t :(
Test Plan: TARGET_OS=IOS make static_lib
Reviewers: dhruba, ljin, yhchiang, rven, sdong
Reviewed By: sdong
Subscribers: dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D28743
Summary:
We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.
This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.
Test Plan: compiles
Reviewers: ljin, rven, yhchiang, sdong
Reviewed By: yhchiang
Subscribers: bobbaldwin, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D28689
Summary:
ColumnFamilyHandles face the same problem as RocksIterator previously
so used methods were also applied for ColumnFamilyHandles.
Another problem with CF was that Options passed to CFs were
always filled with default values. To enable Merge, all parts
of the database must share the same merge functionality which
is not possible using default values. So from now on every
CF will inherit from db options.
Changes to RocksDB:
- merge can now take also a cfhandle
Changes to MergeTest:
- Corrected formatting
- Included also GC tests
- Extended tests to cover CF related parts
- Corrected paths to cleanup properly within the test process
- Reduced verbosity of the test
Test Plan:
make rocksdbjava
make jtest
Subscribers: dhruba
Differential Revision: https://reviews.facebook.net/D27999
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:
Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).
To align BuiltinComparator with every other Enum it was moved to
a separate file.
Test Plan:
make rocksdbjava
make jtest
Summary:
- BackupableDB deleteBackup method
- BackupableDB purgeOldBackups bugfix
- BackupInfos now available in Restorable-/BackupableDB
- Extended BackupableDBTest to cover more of the currently implemented functionality.
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D27027
RocksIterator will sometimes Sigsegv on dispose. Mainly thats related
to dispose order. If the related RocksDB instance is freed beforehand
RocksIterator.dispose() will fail.
Within this commit there is a major change to RocksIterator. RocksIterator
will hold a private reference to the RocksDB instance which created the
RocksIterator. So even if RocksDB is freed in the same GC cycle the
RocksIterator instances will be freed prior to related RocksDB instances.
Another aspect targets the dispose logic if the RocksDB is freed previously
and already gc`ed. On dispose of a RocksIterator the dispose logic will check
if the RocksDB instance points to an initialized DB. If not the dispose logic
will not perform any further action.
The crash can be reproduced by using the related test provided within this
commit.
Related information: This relates to @adamretter`s facebook rocksdb-dev group
post about SigSegv on RocksIterator.dispose().
Summary:
Snapshots integration into RocksJava. Added support for the following functionalities:
- getSnapshot
- releaseSnapshot
- ReadOptions support to set a Snapshot
- ReadOptions support to retrieve Snapshot
- SnapshotTest
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D24801
Previous to this commit Filters passed as parameters to the
BlockTableConfig are disposed before they should be disposed.
Further Smart pointer usage was corrected.
Java holds now the smart pointer to the FilterPolicy correctly
and cares about freeing underlying c++ structures.
Summary:
Added support for the merge operation to RocksJava.
You can specify a merge function to be used on the current database.
The merge function can either be one of the functions defined in
utilities/merge_operators.h, which can be specified through its
corresponding name, or a user-created function that needs to be
encapsulated in a JNI object in order to be used. Examples are
provided for both use cases.
Test Plan: There are unit test in MergeTest.java
Reviewers: ankgup87
Subscribers: vladb38
Differential Revision: https://reviews.facebook.net/D24525
This commit includes the support for the following functionalities:
- Single Get/Put operations
- WriteBatch operations
- Single iterator functionality
- Open database with column families
- Open database with column families Read/Only
- Create column family
- Drop column family
- Properties of column families
- Listing of column families
- Fully backwards comptabile implementation
- Multi Iterator support
- MultiGet
- KeyMayExist
- Option to create missing column families on open
In addition there is are two new Tests:
- Test of ColumnFamily functionality
- Test of Read only feature to open subsets of column families
- Basic test to test the KeyMayExist feature
What is not supported currently using RocksJava:
- Custom ColumnFamilyOptions
The following targets work as expected:
- make rocksdbjava
- make jtest
Test environment: Ubuntu 14.04(LTS, x64), Java 1.7.0_65(OpenJDK IcedTea 2.5.2), g++ 4.8.2, kernel 3.13.0-35-generix
Summary:
This pull request solves the jlong overflow problem on 32-Bit machines as described in https://github.com/facebook/rocksdb/issues/278:
1. There is a new org.rocksdb.test.PlatformRandomHelper to assist in getting random values. For 32 Bit the getLong method is overriden by xpromaches code above. For 64 Bit it behaves as is.
2. The detection should be cross-platform (Windows is supported though it is not ported completely yet).
3. Every JNI method which sets jlong values must check if the value fits into size_t. If it overflows size_t a InvalidArgument Status object will be returned. If its ok a OK Status will be returned.
4. Setters which have this check will throw a RocksDBException if its no OK Status.
Additionally some other parts of code were corrected using the wrong type casts.
Test Plan:
make rocksdbjava
make jtest
Differential Revision: https://reviews.facebook.net/D24531
Extended Built-in comparators with ReverseBytewiseComparator.
Reverse key handling is under certain conditions essential. E.g. while
using timestamp versioned data.
As native-comparators were not available using JAVA-API. Both built-in comparators
were exposed via JNI to be set upon database creation time.
Summary: Fix JNI link error caused by the removal of options.db_stats_log_interval in https://reviews.facebook.net/D21915.
Test Plan:
make rocksdbjava
make jtest
Reviewers: ljin, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D23505
Summary:
Fixed 32-bit overflowing issue when converting jlong to size_t by
capping jlong to std::numeric_limits<size_t>::max().
Test Plan:
make rocksdbjava
make jtest
Reviewers: ankgup87, ljin, sdong, igor
Reviewed By: igor
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D23511
Summary:
Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
Previously DbBenchmark.java does not use WriteBatch when benchmarks is set to fillbatch.
Test Plan:
make rocksdbjava -j32
make jtest
make jdb_bench
cd java
./jdb_bench.sh --benchmarks=fillbatch
Reviewers: naveenatceg, ljin, sdong, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D22983
Summary: as title
Test Plan: make rocksdbjava
Reviewers: sdong, yhchiang, igor
Reviewed By: igor
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D23193
Summary:
Simply code by removing code path which does not use Arena
from NewInternalIterator
Test Plan:
make all check
make valgrind_check
Reviewers: sdong
Reviewed By: sdong
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D22395
Summary: as title
Test Plan:
tested on my mac
make rocksdbjava
make jtest
Reviewers: sdong, igor, yhchiang
Reviewed By: yhchiang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D21963
Summary:
I will move compression related options in a separate diff since this
diff is already pretty lengthy.
I guess I will also need to change JNI accordingly :(
Test Plan: make all check
Reviewers: yhchiang, igor, sdong
Reviewed By: igor
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D21915
Summary:
1. Check status of CreateNewBackup. If status is not OK, then throw.
2. Add purgeOldBackups API
Test Plan:
make test
make sample
Reviewers: haobo, sdong, zzbennett, swapnilghike, yhchiang
Reviewed By: yhchiang
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D21753
Summary:
Update header inclusion as include/utilities/*.h has been moved to
include/rocksdb/utilities/*.h
Test Plan: make rocksdbjava
Reviewers: ljin, sdong, igor
Reviewed By: igor
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20517
Summary:
This diff merges pull request #208. Contributor: ankgup87
[Java] Optimize statistics collector
* Optimize statistics collector by collecting statistics of multiple DBs in a single thread rather than starting up a new thread for each DB.
* Also, fix packaging of jnilib file on OS_X platform.
* Diff review: https://reviews.facebook.net/D20265
[Java] Add documentation on interdependency of dispose call of RocksObjects
* Remove transferCppRawPointersOwnershipFrom function.
- This function was setting opt.filter_ and thus filter_ to be null. This way there is no
one holding reference for filter object and can thus be GC'd which is not the intention.
Replaced it with storeOptionsInstace which stores options instance. Options class
internally holds Filter instance. Thus when Options is GC'd, filter reference
will be GC'd automatically.
* Added documentation explaining interdependency of Filter, Options and DB.
* Diff review: https://reviews.facebook.net/D20379
Test Plan:
described in their diff reviews
Reviewers: haobo sdong swapnilghike zzbennett rsumbaly yhchiang
Reviewed by: yhchiang
Summary:
Add setCacheNumShardBits() and cacheNumShardBits() to Options. This allows
developers to control the number of shards for the block cache.
Test Plan:
make rocksdbjava
cd java
make db_bench
./jdb_bench.sh --cache_size=1048576 --cache_numshardbits=6
Reviewers: sdong, ljin, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D19347
Summary:
Fixed some make and linking issues of RocksDBJava. Specifically:
* Add JAVA_LDFLAGS, which does not include gflags
* rocksdbjava library now uses JAVA_LDFLAGS instead of LDFLAGS
* java/Makefile now includes build_config.mk
* rearrange make rocksdbjava workflow to ensure the library file is correctly
included in the jar file.
Test Plan:
make rocksdbjava
make jdb_bench
java/jdb_bench.sh
Reviewers: dhruba, swapnilghike, zzbennett, rsumbaly, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20289
Summary: Fixed the Java compilation error caused by PlainTableFactory API change.
Test Plan:
make rocksdbjava
make jdb_bench
cd java;./jdb_bench.sh
Reviewers: sdong, ljin, ankgup87, swapnilghike, zzbennett, rsumbaly, igor
Reviewed By: igor
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D20295
Summary:
Correct the library loading for zlib in RocksJava: zlib should
be loaded by loadLibrary("z") instead of loadLibrary("zlib").
Test Plan:
make rocksdbjava
cd java
make db_bench
./jdb_bench.sh --compression_type=zlib
Reviewers: sdong, ljin, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D19341
Summary:
Enable the random values in Java DB Bench to be generated based
on the compression_ratio specified in the command-line arguments.
Test Plan:
make rocksdbjava
java/jdb_bench.sh
Reviewers: sdong, ankgup87, haobo
Reviewed By: haobo
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D19101
Summary:
Add HISTORY-JAVA.md and include both C++ and Java history files in .jar file
that describes important chagnes of RocksJava.
Test Plan: make rocksdbjava and make sure HISTORY.md is inside the .jar file
Reviewers: haobo, sdong, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D19185
Summary:
Improve the Java Doc of RocksObject, which explains more about the life-cycle
of a native handle.
Test Plan: no code change
Reviewers: haobo, sdong, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D19179
Summary:
Improve documentation for RocksEnv and its C++ resource. Specifically,
the result of RocksEnv::Default() is a singleton, and the ownership
of its c++ resource belongs to rocksdb c++. As a result, calling
dispose() of the return value of RocksDB::Default() will be no-op.
Test Plan: no code change.
Reviewers: haobo, ankgup87
Reviewed By: ankgup87
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D18945
Summary:
1. Move disOwnNativeHandle() function from RocksDB to RocksObject
to allow other RocksObject to use disOwnNativeHandle() when its
ownership of native handle has been transferred.
2. RocksObject now has an abstract implementation of dispose(),
which does the following two things. First, it checks whether
both isOwningNativeHandle() and isInitialized() return true.
If so, it will call the protected abstract function dispose0(),
which all the subclasses of RocksObject should implement. Second,
it sets nativeHandle_ = 0. This redesign ensure all subclasses
of RocksObject have the same dispose behavior.
3. All subclasses of RocksObject now should implement dispose0()
instead of dispose(), and dispose0() will be called only when
isInitialized() returns true.
Test Plan:
make rocksdbjava
make jtest
Reviewers: dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett, haobo
Reviewed By: haobo
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D18801
Summary:
Originally, the boolean arguments in Java DB benchmark only takes
true and false as valid input. This diff allows boolean arguments
to be set using 0 or 1.
Test Plan:
make rocksdbjava
make jdb_bench
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=0
java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1
Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett
Reviewed By: haobo
Subscribers: leveldb
Differential Revision: https://reviews.facebook.net/D18687
Summary: Fixed a bug in Java DB Benchmark where random reads does not consider full key range.
Test Plan:
make rocksdbjava
make jdb_bench
cd java
jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=fillseq --use_existing_db=false --num=100000
jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=readrandom --use_existing_db=true --num=100000 --reads=1000000
Reviewers: haobo, sdong
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18693
Summary: Make read benchmarks print out found / not-found information.
Test Plan:
make rocksdbjava
make jdb_bench
cd java
./jdb_bench.sh
Reviewers: haobo, sdong
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18699
Summary:
Before the Java binding for Env is ready, Java developers have no way to
control the number of background threads. This diff provides a temporary
solution where RocksDB.setMaxBackgroundCompactions() will affect the
number of background threads.
Note that once Env is ready. Changes made in this diff should be reverted.
Test Plan:
make rocksdbjava
make jtest
make jdb_bench
java/jdb_bench.sh
Reviewers: haobo, sdong
Reviewed By: sdong
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18681
Summary:
Add RocksObject, the base class of all java objects which has a c++ pointer.
While the finalizer of a RocksObject will release its c++ resource, it is
suggested to call its RocksObject.dispose() to manually release its
c++ resource.
Existing RocksDB java classes are now extending RocksObject.
Test Plan:
make rocksdbjava
make jtest
make jdb_bench
Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18411
Summary:
* Add filluniquerandom
* Add readseq, implemented using iterator.
* Realize most command-line-arguments from db_bench.cc (70+).
* Some code are commented out as some of the options in Options
not yet have Java bindings.
* Add default option to DbBenchmark.
* RocksDB will now take the ownership of all c++ raw-pointers from Options, which includes a c++ raw-pointer for Filter.
Test Plan: ./jdb_bench.sh --db=/tmp/rocksjava-bench/db --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=/tmp/rocksjava-bench/wal --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --benchmarks=filluniquerandom,readseq,readrandom --use_existing_db=0 --threads=4
Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18267
Summary:
Add Java bindings and Java tests for 30+ rocksdb::Options. Codes are
machine-genearted based on include/rocksdb/options.h with manual-polishment.
Test Plan:
make rocksdbjava
make jtest
Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18273
Summary: Add Java binding and test for rocksdb::ReadOptions.
Test Plan:
make rocksdbjava
make jtest
Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18129
Summary:
Add Java bindings for memtables and sst format. Specifically,
add two abstract Java classses --- MemTableConfig and SstFormatConfig.
Each MemTable / SST implementation should has its own config class
extends MemTableConfig / SstFormatConfig respectively and pass it
to Options via setMemTableConfig / setSstConfig.
Test Plan:
make rocksdbjava
make jdb_test
make jdb_bench
java/jdb_bench.sh \
--benchmarks=fillseq,readrandom,readwhilewriting \
--memtablerep=hash_skiplist \
--use_plain_table=1 \
--key_size=20 \
--prefix_size=12 \
--value_size=100 \
--cache_size=17179869184 \
--disable_wal=0 \
--sync=0 \
Reviewers: haobo, ankgup87, sdong
Reviewed By: haobo
CC: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D17997
Summary:
Add a skeleton binding and test for BackupableDB which shows that BackupableDB
and RocksDB can share the same JNI calls.
Test Plan:
make rocksdbjava
make jtest
Reviewers: haobo, ankgup87, sdong, dhruba
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17793
Summary:
* Add a class SizeUnit to store frequently used consts. Currently
it has KB, MB, GB, TB, and PB.
* Change the parameter type of Options.writeBufferSize and Options.blockSize
from int to long.
Test Plan:
make rocksdbjava
make jtest
Reviewers: haobo, ankgup87, sdong, dhruba
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17703
Summary:
Add java bindings for Options.block_cache and allow DbBenchmark to
set cache_size.
Test Plan:
make rocksdbjava
make jtest
make jdb_Bench
Reviewers: haobo, sdong, ankgup87
Reviewed By: ankgup87
CC: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D17481
Summary:
Make each set function of Options / WriteOptions return its option instance.
Java developers can now easier specify each option like the following:
options.setCreateIfMissing(true)
.setWriteBufferSize(8 * 1024)
.setMaxWriteBufferNumber(3)
.setDisableSeekCompaction(true)
.setBlockSize(64 * 1024)
.setMaxBackgroundCompactions(10);
Test Plan:
make rocksdbjava
make jtest
Reviewers: haobo, ankgup87, sdong, dhruba
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17661
Summary:
* Add a benchmark for java binding for rocksdb. The java benchmark
is a complete rewrite based on the c++ db/db_bench.cc and the
DbBenchmark in dain's java leveldb.
* Support multithreading.
* 'readseq' is currently not supported as it requires RocksDB Iterator.
* usage:
--benchmarks
Comma-separated list of operations to run in the specified order
Actual benchmarks:
fillseq -- write N values in sequential key order in async mode
fillrandom -- write N values in random key order in async mode
fillbatch -- write N/1000 batch where each batch has 1000 values
in random key order in sync mode
fillsync -- write N/100 values in random key order in sync mode
fill100K -- write N/1000 100K values in random order in async mode
readseq -- read N times sequentially
readrandom -- read N times in random order
readhot -- read N times in random order from 1% section of DB
Meta Operations:
delete -- delete DB
DEFAULT: [fillseq, readrandom, fillrandom]
--compression_ratio
Arrange to generate values that shrink to this fraction of
their original size after compression
DEFAULT: 0.5
--use_existing_db
If true, do not destroy the existing database. If you set this
flag and also specify a benchmark that wants a fresh database, that benchmark will fail.
DEFAULT: false
--num
Number of key/values to place in database.
DEFAULT: 1000000
--threads
Number of concurrent threads to run.
DEFAULT: 1
--reads
Number of read operations to do. If negative, do --nums reads.
--key_size
The size of each key in bytes.
DEFAULT: 16
--value_size
The size of each value in bytes.
DEFAULT: 100
--write_buffer_size
Number of bytes to buffer in memtable before compacting
(initialized to default value by 'main'.)
DEFAULT: 4194304
--cache_size
Number of bytes to use as a cache of uncompressed data.
Negative means use default settings.
DEFAULT: -1
--seed
Seed base for random number generators.
DEFAULT: 0
--db
Use the db with the following name.
DEFAULT: /tmp/rocksdbjni-bench
* Add RocksDB.write().
Test Plan: make jbench
Reviewers: haobo, sdong, dhruba, ankgup87
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17433
Summary:
Fixed the following column family related compile error.
./java/rocksjni/write_batch.cc:211:66: error: cannot initialize a parameter of type 'rocksdb::ColumnFamilyMemTables *' with an lvalue of type 'rocksdb::MemTable *'
rocksdb::Status s = rocksdb::WriteBatchInternal::InsertInto(b, mem, &options);
^~~
Test Plan:
make jni
make jtest
Reviewers: igor, haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17577
Summary: Avoid a JNI call to GetByteArrayElements, which may introduce a byte-array-copy.
Test Plan: make jtest
Reviewers: haobo, sdong, dhruba
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17451
Summary:
Improve the internal interface between java and c++ for basic db operations
by including the RocksDB native handle (i.e., c++ pointer of rocksdb::DB)
as a input parameter of the internal interface.
This improvement reduces one JNI call per db operation from c++.
Test Plan: make test
Reviewers: haobo, sdong, dhruba
Reviewed By: sdong
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17445
Summary:
* Add java api for rocksdb::WriteBatch and rocksdb::WriteOptions, which are necessary components
for running benchmark.
* Add java test for org.rocksdb.WriteBatch and org.rocksdb.WriteOptions.
* Add remove() to org.rocksdb.RocksDB, and add put() and remove() to RocksDB which take
org.rocksdb.WriteOptions.
Test Plan: make jtest
Reviewers: haobo, sdong, dhruba
Reviewed By: sdong
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17373
Summary:
* [java] Add a java api for rocksdb::Options, currently only supports create_if_missing.
* [java] Add a test for RocksDBException in RocksDBSample.
Test Plan: make jtest
Reviewers: haobo, sdong
Reviewed By: haobo
CC: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D17385
Summary:
* [java] RocksDB now does not implement Closeable.
* [java] RocksDB.close() is now synchronized.
* [c++] Fix a bug in rocksjni.cc that does not release a java reference before
throwing an exception.
Test Plan:
make jni
make jtest
Reviewers: haobo, sdong
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17355
Summary: Fix a bug in RocksDBSample.java and minor code improvement in rocksjni.cc.
Test Plan:
make jni
make jtest
Reviewers: haobo, sdong
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D17325
Summary:
This diff contains a simple jni library for rocksdb which supports open, get,
put and closeusing default options (including Options, ReadOptions, and
WriteOptions.) In the usual case, Java developers can use the c++ rocksdb
library in the way similar to the following:
RocksDB db = RocksDB.open(path_to_db);
...
db.put("hello".getBytes(), "world".getBytes();
byte[] value = db.get("hello".getBytes());
...
db.close();
Specifically, this diff has the following major classes:
* RocksDB: a Java wrapper class which forwards the operations
from the java side to c++ rocksdb library.
* RocksDBException: ncapsulates the error of an operation.
This exception type is used to describe an internal error from
the c++ rocksdb library.
This diff also include a simple java sample code calling c++ rocksdb library.
To build the rocksdb jni library, simply run make jni, and make jtest will try to
build and run the sample code.
Note that if the rocksdb is not built with the default glibc that Java uses,
java will try to load the wrong glibc during the run time. As a result,
the sample code might not work properly during the run time.
Test Plan:
* make jni
* make jtest
Reviewers: haobo, dhruba, sdong, igor, ljin
Reviewed By: dhruba
CC: leveldb, xjin
Differential Revision: https://reviews.facebook.net/D17109
Summary:
Change namespace from leveldb to rocksdb. This allows a single
application to link in open-source leveldb code as well as
rocksdb code into the same process.
Test Plan: compile rocksdb
Reviewers: emayanke
Reviewed By: emayanke
CC: leveldb
Differential Revision: https://reviews.facebook.net/D13287
Summary:
Debug and ported changes from the Open Source Github repo to our repo.
Wrote a script to easy build the java Library. future compiling java lib should just be running this script.
Test Plan: it compiles.
Reviewers: dhruba, leveldb
Reviewed By: dhruba
Differential Revision: https://reviews.facebook.net/D7323
Summary:
Porting various options, mostly related to Multi-threaded compaction to
Java.
Test Plan:
mvn test.
No clear plan on how else test.
Reviewers: dhruba
Reviewed By: dhruba
CC: leveldb, emayanke
Differential Revision: https://reviews.facebook.net/D7221
Summary: This implement the Java interface by using JNI
Test Plan: compile test
Reviewers: dhruba
Reviewed By: dhruba
Differential Revision: https://reviews.facebook.net/D5925
