Commit Graph

64 Commits

Author SHA1 Message Date
agiardullo
c815351038 Support saving history in memtable_list
Summary:
For transactions, we are using the memtables to validate that there are no write conflicts.  But after flushing, we don't have any memtables, and transactions could fail to commit.  So we want to someone keep around some extra history to use for conflict checking.  In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit.

After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure).  It seems like the best place for this is abstracted inside the memtable_list.  I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much.

This diff adds a new parameter to control how much memtable history to keep around after flushing.  However, it sounds like people aren't too fond of adding new parameters.  So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers.  This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit.  (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached).  So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit).

However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions.

Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests.  Added testing in memtablelist_test and planning on adding more testing here.

Reviewers: sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37443
2015-05-28 16:34:24 -07:00
fyrz
019ecd1932 [RocksJava] Remove deprecated methods
Summary:
- Removed deprecated ColumnFamilyDescript constructor methods
- Removed deprecated skipLogErrorOnRecovery methods
- Removed deprecated tableCacheRemoveScanCountLimit methods

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, adamretter, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37191
2015-04-16 00:31:43 +02:00
fyrz
5b7131c751 [RocksJava] Removed deprecated skipLogErrorOnRecovery methods.
As annonunced these options are not used anymore so these are
removed from code.
2015-04-15 23:37:24 +02:00
fyrz
004b89fba4 [RocksJava] Add compression per level to API
Summary:
RocksDB offers the possibility to set different compression types
on a per level basis. This shall be also available using RocksJava.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35577
2015-03-23 18:58:56 +01:00
fyrz
5615e23d8c [RocksJava] 32-Bit adjustments
Summary:
Before this change overflowing size_t values led to a checked Exception.

After that change:

size_t overflows on 32-Bit architecture throw now an IllegalArgumentException,
which removes the necessity for a developer to catch these Exceptions explicitly.

This is especially an advantage for developers targeting 64-Bit systems because
it is not necessary anymore to catch exceptions which are never thrown on a 64-Bit
system.

Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D34923
2015-03-19 20:45:55 +01:00
fyrz
12350115da [RocksJava] Added LevelCompactionDynamicLevelBytes to Options
Summary: Added LevelCompactionDynamicLevelBytes to Options.

Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package

Reviewers: adamretter, ankgup87, yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D34857
2015-03-18 21:38:17 +01:00
Igor Canadi
c88ff4ca76 Deprecate removeScanCountLimit in NewLRUCache
Summary: It is no longer used by the implementation, so we should also remove it from the public API.

Test Plan: make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D34971
2015-03-17 15:04:37 -07:00
Adam Retter
eafa1bfc3c Merge pull request #529 from fyrz/RocksJava-Logger
[RockJava] Custom logger addition
2015-03-16 10:38:15 +00:00
fyrz
04778a94c5 [RocksJava] OptimizeFiltersForHits
Summary: Added optimize_filters_for_hits option.

Test Plan:
make clean jclean rocksdbjava jtest
mvn -f rocksjni.pom package

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35013
2015-03-14 21:00:15 +01:00
fyrz
a3bd4142f2 [RocksJava] Logging JNI callback
Within this commit a new AbstractLogger was introduced
which pushes info log messages all the way up to Java.
2015-03-14 20:57:18 +01:00
Lei Jin
5045c43944 add support for nested BlockBasedTableOptions in config string
Summary:
Add support to allow nested config for block-based table factory. The format looks like this:

"write_buffer_size=1024;block_based_table_factory={block_size=4k};max_write_buffer_num=2"

Test Plan: unit test

Reviewers: yhchiang, rven, igor, ljin, jonahcohen

Reviewed By: jonahcohen

Subscribers: jonahcohen, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29223
2014-12-22 16:34:21 -08:00
Adam Retter
eda0dcdd97 Exposed IncreasedParallelism option to Java API as
setIncreasedParallelism
2014-12-16 14:15:41 +00:00
fyrz
e002a6122f [RocksJava] Comparator tests for CF
- Added AbstractComparatorTest.
- Fixed a bug in the JNI Part about Java comparators
2014-12-02 09:58:19 +01:00
fyrz
a280af2a57 [RocksJava] Sigsegv fix for MergerOperatorByName 2014-11-25 22:26:02 +01:00
fyrz
5249d0db50 [RocksJava] Convenience methods for Options
RocksDB introduced in 3.7.0 convenience methods
for getting ColumnFamilyOptions and DBOptions
instances from predefined configuration structures.

There is now also a method in RocksJava to load DBOptions
as well as ColumnFamilyOptions from a predefined Properties
based configuration.
2014-11-18 21:43:20 +01:00
fyrz
07cd3c42a2 [RocksJava] LogLevel support in Options
It's now possible to set a LogLevel in Options and
DBOptions to control LOG verbosity.
2014-11-14 23:40:20 +01:00
fyrz
628e39e97d [RocksJava] Integrated review comments from D28209 2014-11-14 20:59:36 +01:00
fyrz
75010d2084 [RocksJava] ColumnFamily custom Options API extension
*********************
                   ***************************
                 ******** ************* ********
                ********   ***********   ********
               ********     *********     ********
              *************************************
              *************************************
              *************************************
               ******     ***       ***     ******
                ******    ***  ***  ***    ******
                 ******        ***        ******
                   ***************************
                      *********************
2014-11-12 19:28:24 +01:00
fyrz
0345c2156f [RocksJava] Extend Options with ColumnFamilyOptions implementation ColumnFamilyOptions implementation with tests
[RocksJava] Extended ColumnFamilyTest

Summary: Options Refactoring split part 3

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28023
2014-11-12 19:28:23 +01:00
fyrz
fc6fcbab9e [RocksJava] Flush functionality
RocksJava now supports also flush functionality of
RocksDB.
2014-11-11 07:47:38 +01:00
fyrz
94e31ac227 [RocksJava] Extend Options with DBOptions implementation [RocksJava] Included DBOptionsTest and refactored OptionsTest
Summary: Options refactoring - Split Part2

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28017
2014-11-03 00:13:57 +01:00
fyrz
b060d30065 [RocksJava] Build fix after options refactoring 2014-11-02 23:48:40 +01:00
fyrz
df7abb4e8d [RocksJava] Integrated code review comments
- Added TODO comments for disabled methods
2014-10-31 00:06:52 +01:00
fyrz
39464a990f [RocksJava] Options Refactoring 3.6
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:

Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).

To align BuiltinComparator with every other Enum it was moved to
a separate file.

Test Plan:
make rocksdbjava
make jtest
2014-10-30 22:26:39 +01:00
Yueh-Hsuan Chiang
db52419cf0 Merge pull request #290 from vladb38/master
Added merge operators to RocksJava
2014-10-27 16:41:33 -07:00
Vlad Balan
a04929aa49 fixed conflict in java/Makefile 2014-10-27 10:27:02 -07:00
fyrz
b8ce526487 [RocksJava] Support Snapshots
Summary:
Snapshots integration into RocksJava. Added support for the following functionalities:

- getSnapshot
- releaseSnapshot
- ReadOptions support to set a Snapshot
- ReadOptions support to retrieve Snapshot
- SnapshotTest

Test Plan:
make rocksdbjava
make jtest

Differential Revision: https://reviews.facebook.net/D24801
2014-10-25 11:10:19 +02:00
Adam Retter
5e25274110 Fix code style problems identified by lint 2014-10-21 15:52:27 +01:00
Adam Retter
fc12cb83f2 Add locking to comparator jni callback methods which must be thread-safe 2014-10-21 15:52:27 +01:00
Adam Retter
d6fe8dacc8 Feature - Implement Java API for Comparator and Slice. Allows use of
either byte[] or DirectByteBuffer for accessing underlying data.
2014-10-21 15:52:27 +01:00
Vlad Balan
2ef3ed86f3 Integrated feedback from ankgup87
Test Plan: tested using unit tests

Reviewers: ankgup87

Differential Revision: https://reviews.facebook.net/D24573
2014-10-13 15:37:29 -07:00
Vlad Balan
a40ce219b9 Adding merge functions to RocksDBJava
Summary:
Added support for the merge operation to RocksJava.
You can specify a merge function to be used on the current database.
The merge function can either be one of the functions defined in
utilities/merge_operators.h, which can be specified through its
corresponding name, or a user-created function that needs to be
encapsulated in a JNI object in order to be used. Examples are
provided for both use cases.

Test Plan: There are unit test in MergeTest.java

Reviewers: ankgup87

Subscribers: vladb38

Differential Revision: https://reviews.facebook.net/D24525
2014-10-13 14:40:16 -07:00
fyrz
18004d2f20 [RocksJava] Column family support
This commit includes the support for the following functionalities:

 - Single Get/Put operations
 - WriteBatch operations
 - Single iterator functionality
 - Open database with column families
 - Open database with column families Read/Only
 - Create column family
 - Drop column family
 - Properties of column families
 - Listing of column families
 - Fully backwards comptabile implementation
 - Multi Iterator support
 - MultiGet
 - KeyMayExist
 - Option to create missing column families on open

In addition there is are two new Tests:

 - Test of ColumnFamily functionality
 - Test of Read only feature to open subsets of column families
 - Basic test to test the KeyMayExist feature

What is not supported currently using RocksJava:

 - Custom ColumnFamilyOptions

The following targets work as expected:

 - make rocksdbjava
 - make jtest

Test environment: Ubuntu 14.04(LTS, x64), Java 1.7.0_65(OpenJDK IcedTea 2.5.2), g++ 4.8.2, kernel 3.13.0-35-generix
2014-10-13 10:34:52 +02:00
fyrz
4f5a687254 32-Bit RocksJava resolution for jlong overflows
Summary:
This pull request solves the jlong overflow problem on 32-Bit machines as described in https://github.com/facebook/rocksdb/issues/278:

1. There is a new org.rocksdb.test.PlatformRandomHelper to assist in getting random values. For 32 Bit the getLong method is overriden by xpromaches code above. For 64 Bit it behaves as is.
2. The detection should be cross-platform (Windows is supported though it is not ported completely yet).
3. Every JNI method which sets jlong values must check if the value fits into size_t. If it overflows size_t a InvalidArgument Status object will be returned. If its ok a OK Status will be returned.
4. Setters which have this check will throw a RocksDBException if its no OK Status.

Additionally some other parts of code were corrected using the wrong type casts.

Test Plan:
make rocksdbjava
make jtest

Differential Revision: https://reviews.facebook.net/D24531
2014-10-12 22:42:18 +02:00
Yueh-Hsuan Chiang
0e516a75da Fix lint errors in java/rocksjni/options.cc
Summary:
Fix lint errors in java/rocksjni/options.cc

Test Plan:
make rocksdbjava
2014-10-03 00:10:58 -07:00
fyrz
5340484266 Built-in comparator(s) in RocksJava
Extended Built-in comparators with ReverseBytewiseComparator.

Reverse key handling is under certain conditions essential. E.g. while
using timestamp versioned data.

As native-comparators were not available using JAVA-API. Both built-in comparators
were exposed via JNI to be set upon database creation time.
2014-09-26 10:35:12 +02:00
Igor Canadi
21ddcf6e4f Remove allow_thread_local
Summary: See https://reviews.facebook.net/D19365

Test Plan: compiles

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23907
2014-09-24 13:12:16 -07:00
Ankit Gupta
bfeef94d31 Add rate limiter 2014-09-19 16:11:59 -07:00
Yueh-Hsuan Chiang
94e43a1dfe [Java] Fixed 32-bit overflowing issue when converting jlong to size_t
Summary:
Fixed 32-bit overflowing issue when converting jlong to size_t by
capping jlong to std::numeric_limits<size_t>::max().

Test Plan:
make rocksdbjava
make jtest

Reviewers: ankgup87, ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23511
2014-09-17 12:30:06 -07:00
Lei Jin
66f62e5c78 JNI changes corresponding to BlockBasedTableOptions migration
Summary: as title

Test Plan:
tested on my mac
make rocksdbjava
make jtest

Reviewers: sdong, igor, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21963
2014-08-25 14:22:55 -07:00
Lei Jin
384400128f move block based table related options BlockBasedTableOptions
Summary:
I will move compression related options in a separate diff since this
diff is already pretty lengthy.
I guess I will also need to change JNI accordingly :(

Test Plan: make all check

Reviewers: yhchiang, igor, sdong

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21915
2014-08-25 14:22:05 -07:00
Yueh-Hsuan Chiang
10fc6c7d25 [Java] Add compaction style to options
Summary:
Add compression type to options

make rocksdbjava
make sample

Reviewers: haobo, yhchiang, sdong, dhruba, rsumbaly, zzbennett, swapnilghike
Reviewed By: yhchiang, sdong
CC: leveldb

Differential Revision: https://reviews.facebook.net/D20463
2014-07-23 15:45:30 -07:00
Ankit Gupta
97bfcd6a16 Update doc of options.cc 2014-07-07 12:26:06 -07:00
Ankit Gupta
4efca96bb3 Change enum type from int to byte 2014-07-07 12:18:55 -07:00
Ankit Gupta
54d7a2c0c5 Add compression type to options 2014-07-07 09:58:54 -07:00
Yueh-Hsuan Chiang
ab3e566699 [Java] Generalize dis-own native handle and refine dispose framework.
Summary:
1. Move disOwnNativeHandle() function from RocksDB to RocksObject
to allow other RocksObject to use disOwnNativeHandle() when its
ownership of native handle has been transferred.

2. RocksObject now has an abstract implementation of dispose(),
which does the following two things.  First, it checks whether
both isOwningNativeHandle() and isInitialized() return true.
If so, it will call the protected abstract function dispose0(),
which all the subclasses of RocksObject should implement.  Second,
it sets nativeHandle_ = 0.  This redesign ensure all subclasses
of RocksObject have the same dispose behavior.

3. All subclasses of RocksObject now should implement dispose0()
instead of dispose(), and dispose0() will be called only when
isInitialized() returns true.

Test Plan:
make rocksdbjava
make jtest

Reviewers: dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett, haobo

Reviewed By: haobo

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D18801
2014-05-28 18:16:29 -07:00
Yueh-Hsuan Chiang
9895465c6a [Java] Fixed compile error due to the removal of ReadOptions.prefix_seek, minor improvement on DbBenchmark.java. 2014-04-27 21:49:45 -07:00
Yueh-Hsuan Chiang
05979493d6 [Java] Enable filluniquerandom, readseq, BloomFilter, and 70+ command-line options to DbBenchmark.java
Summary:
* Add filluniquerandom
* Add readseq, implemented using iterator.
* Realize most command-line-arguments from db_bench.cc (70+).
* Some code are commented out as some of the options in Options
  not yet have Java bindings.
* Add default option to DbBenchmark.
* RocksDB will now take the ownership of all c++ raw-pointers from Options, which includes a c++ raw-pointer for Filter.

Test Plan: ./jdb_bench.sh --db=/tmp/rocksjava-bench/db --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=/tmp/rocksjava-bench/wal --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --benchmarks=filluniquerandom,readseq,readrandom --use_existing_db=0 --threads=4

Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18267
2014-04-24 13:43:57 -07:00
Yueh-Hsuan Chiang
d8fe006056 [Java] Add Java bindings and Java tests for 30+ rocksdb::Options.
Summary:
Add Java bindings and Java tests for 30+ rocksdb::Options.  Codes are
machine-genearted based on include/rocksdb/options.h with manual-polishment.

Test Plan:
make rocksdbjava
make jtest

Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18273
2014-04-23 20:54:35 -07:00
Ankit Gupta
677b0d6d3f Refactor filter impl 2014-04-22 00:04:56 -07:00