Commit Graph

247 Commits

Author SHA1 Message Date
Adam Retter
f4d986364e Added SetOptions support to RocksJava (#1243)
* [refactor] Split Java ColumnFamilyOptions into mutable and immutable and implement any missing immutable options

* [feature] Implement RocksDB#setOptions
2016-08-06 12:03:47 -07:00
Islam AbdelRahman
7c01d65344 [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv
Summary: Remove duplicate cases in LoggerJniCallback::Logv and fix complication error

Test Plan: make jclean rocksdbjava jtest -j4

Reviewers: adamretter, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61335
2016-07-29 12:47:15 -07:00
Adam Retter
8796934af7 Added missing Java ReadOptions settings (#1109)
* Minor reorganisation

* Java API - Added missing ReadOptions settings
2016-07-29 12:12:03 -07:00
Adam Retter
811ee21112 Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106) 2016-07-28 16:54:06 -07:00
Islam AbdelRahman
8745f013ff [Fix java build] Stop using non standard std::make_unique
Summary: std::make_unique is not standard and not always available, remove it

Test Plan: Run "make clean jclean rocksdbjava jtest -j8" on my mac

Reviewers: yhchiang, yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61143
2016-07-26 13:18:31 -07:00
Alexander Jipa
12767b3130 fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223)
* fixes 1220: rocksjni build fails on Windows due to variable-size array declaration

using new keyword to create variable-size arrays in order to satisfy most of the compilers

* fixes 1220: rocksjni build fails on Windows due to variable-size array declaration

using unique_ptr keyword to create variable-size arrays in order to satisfy most of the compilers
2016-07-22 10:17:10 -07:00
sdong
32df9733d1 Add options.write_buffer_manager: control total memtable size across DB instances
Summary: Add option write_buffer_manager to help users control total memory spent on memtables across multiple DB instances.

Test Plan: Add a new unit test.

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: adela, benj, sumeet, muthu, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59925
2016-07-05 18:11:25 -07:00
sdong
7b79238b65 Deprectate filter_deletes
Summary: filter_deltes is not a frequently used feature. Remove it.

Test Plan: Run all test suites.

Reviewers: igor, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59427
2016-06-17 10:30:47 -07:00
Adam Retter
e3b1e3dfa6 Expose save points in Java WriteBatch and WBWI (#1092)
* Java API - Expose SetSavePoint and RollbackToRestorePoint for WriteBatch and WriteBatchWithIndex

* Minor cleanup
2016-06-14 09:51:14 -07:00
sdong
20699df843 memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
Summary:
memtable_prefix_bloom_probes is not a critical option. Remove it to reduce number of options.
It's easier for users to make mistakes with memtable_prefix_bloom_bits, turn it to memtable_prefix_bloom_bits_ratio

Test Plan: Run all existing tests

Reviewers: yhchiang, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: gunnarku, yoshinorim, MarkCallaghan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59199
2016-06-10 12:12:10 -07:00
Adam Retter
972c895c35 Previously WARN level logging became FATAL level logging in the Java API (#1089) 2016-06-07 22:35:57 -07:00
Adam Retter
774a6aa296 Java API - Rename geHistogramData -> getHistogramData (#1107)
Closes https://github.com/facebook/rocksdb/issues/835
2016-06-06 16:04:16 -07:00
Adam Retter
a791a2cf2d Java API - Bugfix for native linking of Compaction Filter (#1099) 2016-06-03 10:24:03 -07:00
Adam Retter
72c73cdc8b Java API - Add missing HEADER_LEVEL logging (#1104) 2016-05-06 15:06:12 -07:00
Adam Retter
4d02bfa3a6 Add support for PauseBackgroundWork and ContinueBackgroundWork to the Java API (#1087)
Closes https://github.com/facebook/rocksdb/issues/1071
2016-05-06 15:04:13 -07:00
Andrew Kryczka
9385fd72c0 Delete deprecated backup classes in Java
Summary: Corresponding change to D56331.

Test Plan:
Now build succeeds:

  $ make jclean && make rocksdbjava

Reviewers: yhchiang, IslamAbdelRahman, adamretter

Reviewed By: adamretter

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56913
2016-04-19 10:32:31 -07:00
Marton Trencseni
9b51987521 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
Summary:
When a block based table file is opened, if prefetch_index_and_filter is true, it will prefetch the index and filter blocks, putting them into the block cache.
What this feature adds: when a L0 block based table file is opened, if pin_l0_filter_and_index_blocks_in_cache is true in the options (and prefetch_index_and_filter is true), then the filter and index blocks aren't released back to the block cache at the end of BlockBasedTableReader::Open(). Instead the table reader takes ownership of them, hence pinning them, ie. the LRU cache will never push them out. Meanwhile in the table reader, further accesses will not hit the block cache, thus avoiding lock contention.

Test Plan:
'export TEST_TMPDIR=/dev/shm/ && DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32' is OK.
I didn't run the Java tests, I don't have Java set up on my devserver.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56133
2016-04-01 10:42:39 -07:00
Adam Retter
188bb2e7ad Fix formatting identified by arc lint 2016-03-01 16:00:45 +00:00
Adam Retter
c5af85ecad Fix a memory leak of Slice objects from org.rocksdb.WBWIRocksIterator#entry1 2016-02-10 14:44:39 +00:00
Adam Retter
e84137c8ae Remove unnessecary java.util.List expense in JNI 2016-02-10 14:44:39 +00:00
Adam Retter
76e8beeeb9 Pass by pointer from/to Java from JNI not by object 2016-02-10 14:44:39 +00:00
Adam Retter
18eb563058 Improve the speed and synchronization around the construction of Java/JNI objects 2016-02-10 14:44:39 +00:00
Baraa Hamodi
21e95811d1 Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
Tomas Kolda
52153930d8 Adding support for Windows JNI build - fix Java unit test for release build of JNI DLL 2016-02-02 09:45:18 +01:00
Tomas Kolda
e2972803a6 Adding support for Windows JNI build 2016-02-02 09:11:37 +01:00
sdong
b99d4276f2 Fix java test buid broken by 7d87f02799
Summary: 7d87f02799 changed WriteBatch::InsertInto(). Need to change it.

Test Plan: Run it.

Reviewers: yhchiang, ngbronson

Reviewed By: ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52353
2015-12-28 11:51:39 -08:00
Sage Weil
543c12ab06 options: add recycle_log_file_num option
Signed-off-by: Sage Weil <sage@redhat.com>
2015-10-18 21:21:24 -04:00
sdong
35ad531be3 Seperate InternalIterator from Iterator
Summary:
Separate a new class InternalIterator from class Iterator, when the look-up is done internally, which also means they operate on key with sequence ID and type.

This change will enable potential future optimizations but for now InternalIterator's functions are still the same as Iterator's.
At the same time, separate the cleanup function to a separate class and let both of InternalIterator and Iterator inherit from it.

Test Plan: Run all existing tests.

Reviewers: igor, yhchiang, anthony, kradhakrishnan, IslamAbdelRahman, rven

Reviewed By: rven

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D48549
2015-10-13 15:32:13 -07:00
Yueh-Hsuan Chiang
0e6e5472e1 Fixed a compile warning in rocksjni/loggerjnicallback.cc
Summary:
This patch fixes the following compile warning.
    java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’:
    java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^

Test Plan: make rocksdbjavastaticrelease

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45981
2015-09-02 15:07:44 -07:00
Yueh-Hsuan Chiang
40cd91b7fe Fixed compile warning in rocksdbjava
Summary:
Fixed the following compile warning in rocksdbjava:

    java/rocksjni/comparatorjnicallback.cc:20:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^
    java/rocksjni/comparatorjnicallback.cc: In member function ‘JNIEnv* rocksdb::BaseComparatorJniCallback::getJniEnv() const’:
    java/rocksjni/comparatorjnicallback.cc:45:8: warning: unused variable ‘rs’ [-Wunused-variable]
       jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL);
            ^
    java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’:
    java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable]
       const jint rs = env->GetJavaVM(&m_jvm);
                  ^
    java/rocksjni/loggerjnicallback.cc: In member function ‘JNIEnv* rocksdb::LoggerJniCallback::getJniEnv() const’:
    java/rocksjni/loggerjnicallback.cc:33:8: warning: unused variable ‘rs’ [-Wunused-variable]
       jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL);
            ^

Test Plan: make rocksdbjava -j32

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D45891
2015-08-31 18:36:17 -07:00
Yueh-Hsuan Chiang
f5d072ae6c Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor
Summary:
Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor
by adding the missing native implementation of
useCappedPrefixExtractor.

Test Plan:
make jclean
make rocksdbjava -j32
make jtest

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43551
2015-08-04 17:01:08 -07:00
Yueh-Hsuan Chiang
f39cbcb0a5 Merge pull request #654 from adamretter/remove-emptyvalue-compactionfilter
RemoveEmptyValueCompactionFilter
2015-08-04 16:54:14 -07:00
ashishn
f0b5bcc7b5 add support for capped prefix extractor in java 2015-08-04 16:46:23 -07:00
Yueh-Hsuan Chiang
ce21afd205 Expose the BackupEngine from the Java API
Summary:
Merge pull request #665 by adamretter

Exposes BackupEngine from C++ to the Java API. Previously only BackupableDB was available

Test Plan: BackupEngineTest.java

Reviewers: fyrz, igor, ankgup87, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D42873
2015-08-04 16:08:54 -07:00
Andres Noetzli
f73c801432 Fixing Java tests.
Summary:
While working on https://reviews.facebook.net/D43017 , I realized
that some Java tests are failing due to a deprecated option.
This patch removes the offending tests, adds @Deprecated annotations
to the Java interface and removes the corresponding functions in
rocksjni

Test Plan: make jtest (all tests are passing now)

Reviewers: rven, igor, sdong, anthony, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43035
2015-07-24 16:56:26 -07:00
Igor Canadi
35ca59364c Don't let flushes preempt compactions
Summary:
When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.

We have a special case that when you set max_background_flushes to 0, we
schedule the flush to execute on the compaction thread.

Test Plan: make check (there might be some unit tests that depend on this behavior)

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D41931
2015-07-17 12:02:52 -07:00
agiardullo
79373c372d Fix ROCKSDB_WARNING
Summary:
ROCKSDB_WARNING is only defined if either ROCKSDB_PLATFORM_POSIX or OS_WIN is defined.  This works well for building rocksdb with its own build scripts.  But this won't work when an outside project(like mongodb) doesn't define ROCKSDB_PLATFORM_POSIX.

This fix defines ROCKSDB_WARNING for all platforms.  No idea if its defined correctly on non-posix,non-windows platforms but this is no worse that the current situation where this macro is missing on unexpected platforms.

This fix should hopefully fix anyone whose build broke now that we've switched from using #warning to Pragma (to support windows).  Unfortunately, while mongo-rocks compiles, it ignores the Pragma and doesn't print a warning.  I have not been able to figure out a way to implement this portably on all platforms.

Of course, an alternate solution would be to just get rid of ROCKSDB_WARNING and live with include file redirects indefinitely.  Thoughts?

Test Plan: build rocks, build mongorocks

Reviewers: igor, kradhakrishnan, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42477
2015-07-17 11:04:55 -07:00
Adam Retter
91bf1b80ef Java facility to use the RemoveEmptyValueCompactionFilter 2015-07-16 11:50:11 +01:00
Adam Retter
3d00271e40 The ability to specify a compaction filter via the Java API 2015-07-16 11:50:10 +01:00
Jörg Maier
05e194158c Merge pull request #639 from cleaton/setMaxTableFileSize
Add JNI interface to set max_table_files_size for FIFO compaction
2015-07-14 23:23:06 +02:00
Yueh-Hsuan Chiang
03d433ee65 [RocksJava] Fixed test failures
Summary:
The option bottommost_level_compaction was introduced lately.
This option breaks the Java API behavior. To prevent the library
from doing so we set that option to a fixed value in Java.

In future we are going to remove that portion and replace the
hardcoded options using a more flexible way.

Fixed bug introduced by WriteBatchWithIndex Patch

Lately icanadi changed the behavior of WriteBatchWithIndex.
See commit: 821cff114e

This commit solves problems introduced by above mentioned commit.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, ankgup87, yhchiang

Reviewed By: yhchiang

Subscribers: igor, dhruba

Differential Revision: https://reviews.facebook.net/D40647
2015-07-01 23:22:03 -07:00
Jesper Lundgren
d62b6ed838 add setMaxTableFilesSize to JNI interface 2015-06-20 16:23:22 +08:00
Islam AbdelRahman
12e030a992 Use CompactRangeOptions for CompactRange
Summary:
This diff update DB::CompactRange to use RangeCompactionOptions instead of using multiple parameters
Old CompactRange is still available but deprecated

Test Plan:
make all check
make rocksdbjava
USE_CLANG=1 make all
OPT=-DROCKSDB_LITE make release

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D40209
2015-06-17 14:36:14 -07:00
agiardullo
dc9d70de65 Optimistic Transactions
Summary: Optimistic transactions supporting begin/commit/rollback semantics.  Currently relies on checking the memtable to determine if there are any collisions at commit time.  Not yet implemented would be a way of enuring the memtable has some minimum amount of history so that we won't fail to commit when the memtable is empty.  You should probably start with transaction.h to get an overview of what is currently supported.

Test Plan: Added a new test, but still need to look into stress testing.

Reviewers: yhchiang, igor, rven, sdong

Reviewed By: sdong

Subscribers: adamretter, MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D33435
2015-05-29 14:36:35 -07:00
agiardullo
c815351038 Support saving history in memtable_list
Summary:
For transactions, we are using the memtables to validate that there are no write conflicts.  But after flushing, we don't have any memtables, and transactions could fail to commit.  So we want to someone keep around some extra history to use for conflict checking.  In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit.

After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure).  It seems like the best place for this is abstracted inside the memtable_list.  I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much.

This diff adds a new parameter to control how much memtable history to keep around after flushing.  However, it sounds like people aren't too fond of adding new parameters.  So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers.  This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit.  (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached).  So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit).

However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions.

Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests.  Added testing in memtablelist_test and planning on adding more testing here.

Reviewers: sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37443
2015-05-28 16:34:24 -07:00
fyrz
019ecd1932 [RocksJava] Remove deprecated methods
Summary:
- Removed deprecated ColumnFamilyDescript constructor methods
- Removed deprecated skipLogErrorOnRecovery methods
- Removed deprecated tableCacheRemoveScanCountLimit methods

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, adamretter, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37191
2015-04-16 00:31:43 +02:00
fyrz
5b7131c751 [RocksJava] Removed deprecated skipLogErrorOnRecovery methods.
As annonunced these options are not used anymore so these are
removed from code.
2015-04-15 23:37:24 +02:00
Yueh-Hsuan Chiang
afc51649e2 Merge pull request #546 from fyrz/RocksJava-MemEnv
[RocksJava] Expose MemEnv in RocksJava
2015-03-24 14:01:12 -07:00
fyrz
fd8804f979 [RocksJava] Expose MemEnv in RocksJava
Summary:
In 3.10 the C++ code was extended with a MemEnv implementation. This
is now also available in RocksJava.

Changes:
- Extraced abstract super class Env
- Introduced RocksMemEnv
- Remove unnecessary disposeInternal method. The disposal of the default environment is managed by C++ so there needs to be no disposeInternal method in Java.
- Introduced a RocksMemEnvTest, which is aligned with the C++ equivalent.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35619
2015-03-23 18:59:31 +01:00
fyrz
004b89fba4 [RocksJava] Add compression per level to API
Summary:
RocksDB offers the possibility to set different compression types
on a per level basis. This shall be also available using RocksJava.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35577
2015-03-23 18:58:56 +01:00