Commit Graph

641 Commits

Author SHA1 Message Date
anand1976
d394a6bb48 Add a ticker stat for number of keys skipped during iteration
Summary:
This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the
number of internal keys skipped during iteration. Keys can be skipped
due to deletes, or lower sequence number, or higher sequence number
than the one requested.

Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary,
padding becomes a zero size array, which the Windows compiler doesn't
like. So add a cacheline worth of padding in that case to keep it happy.
We cannot conditionally add padding as gcc doesn't allow using sizeof
in preprocessor directives.
Closes https://github.com/facebook/rocksdb/pull/3177

Differential Revision: D6353897

Pulled By: anand1976

fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2
2017-11-20 21:26:37 -08:00
Sagar Vemuri
76c3fbd651 Add Memtable Read Tier to RocksJava
Summary:
This options was introduced in the C++ API in #1953 .
Closes https://github.com/facebook/rocksdb/pull/3064

Differential Revision: D6139010

Pulled By: sagar0

fbshipit-source-id: 164de11d539d174cf3afe7cd40e667049f44b0bc
2017-11-02 17:27:37 -07:00
zawlazaw
57fcdc264a added missing subcodes and improved error message for missing enum values
Summary:
Java's `Status.SubCode` was out of sync with `include/rocksdb/status.h:SubCode`.

When running out of disc space this led to an `IllegalArgumentException` because of an invalid status code, rather than just returning the corresponding status code without an exception.

I added the missing status codes.

By this, we keep the behaviour of throwing an `IllegalArgumentException` in case of newly added status codes that are defined in C but not in Java.

We could think of an alternative strategy: add in Java another code "UnknownCode" which acts as a catch-all for all those status codes that are not yet mirrored from C to Java. This approach would never throw an exception but simply return a non-OK status-code.

I think the current approach of throwing an Exception in case of a C/Java inconsistency is fine, but if you have some opinion on the alternative strategy, then feel free to comment here.
Closes https://github.com/facebook/rocksdb/pull/3050

Differential Revision: D6129682

Pulled By: sagar0

fbshipit-source-id: f2bf44caad650837cffdcb1f93eb793b43580c66
2017-10-23 16:42:07 -07:00
Sagar Vemuri
cc67b22d6e Add OptionsUtil class to java/CMakeLists.txt
Summary:
Adding OptionsUtil java class and options_util.cc to java/CMakeLists.txt, which were missed accidentally when they were introduced in #2898.
Closes https://github.com/facebook/rocksdb/pull/2985

Differential Revision: D6015878

Pulled By: sagar0

fbshipit-source-id: 1abbd46db4aebad1e07ea53523eacbdcb12823e1
2017-10-12 16:57:05 -07:00
Adam Retter
560e984995 Added CompactionFilterFactory support to RocksJava
Summary:
This PR also includes some cleanup, bugfixes and refactoring of the Java API. However these are really pre-cursors on the road to CompactionFilterFactory support.
Closes https://github.com/facebook/rocksdb/pull/1241

Differential Revision: D6012778

Pulled By: sagar0

fbshipit-source-id: 0774465940ee99001a78906e4fed4ef57068ad5c
2017-10-12 11:12:16 -07:00
Sagar Vemuri
bb38cd03a9 Limit number of merge operands in Cassandra merge operator
Summary:
Now that RocksDB supports conditional merging during point lookups (introduced in #2923), Cassandra value merge operator can be updated to pass in a limit. The limit needs to be passed in from the Cassandra code.
Closes https://github.com/facebook/rocksdb/pull/2947

Differential Revision: D5938454

Pulled By: sagar0

fbshipit-source-id: d64a72d53170d8cf202b53bd648475c3952f7d7f
2017-10-02 16:11:40 -07:00
Adam Retter
983028f097 RocksJava build target for Docker on ppc64le
Summary:
This enables us to crossbuild pcc64le RocksJava binaries with a suitably old version of glibc (2.17) on CentOS 7.
Closes https://github.com/facebook/rocksdb/pull/2491

Differential Revision: D5955301

Pulled By: sagar0

fbshipit-source-id: 69ef9746f1dc30ffde4063dc764583d8c7ae937e
2017-10-02 11:11:56 -07:00
Sagar Vemuri
0806801dc8 DestroyDB API
Summary:
Expose DestroyDB API in RocksJava.
Closes https://github.com/facebook/rocksdb/pull/2934

Differential Revision: D5914775

Pulled By: sagar0

fbshipit-source-id: 84af6ea0d2bccdcfb9fe8c07b2f87373f0d5bab6
2017-09-26 16:42:11 -07:00
Sagar Vemuri
c8f3606731 Expose LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs in RocksJava
Summary:
JNI wrappers for LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs.
Closes https://github.com/facebook/rocksdb/pull/2898

Differential Revision: D5857934

Pulled By: sagar0

fbshipit-source-id: 68b79e83eab8de9416e3f1fef73e11cf7947e90a
2017-09-21 17:29:13 -07:00
Sagar Vemuri
96a13b4f4b Use jemalloc in rocksdbjni library built via vagrant
Summary:
Problem:
During RocksJava performance testing we found that the rocksdb jni library is not built with jemalloc; instead it was getting built with the default glibc malloc. We saw quite a bit of memory bloat due to this.

Addressed this by installing jemalloc-devel package in the vm that we use to build release jars.
Closes https://github.com/facebook/rocksdb/pull/2916

Differential Revision: D5887018

Pulled By: sagar0

fbshipit-source-id: ace0b5d60234b3a30dcd5d39633e7827a5982a50
2017-09-21 16:42:06 -07:00
Sagar Vemuri
3fc08fa88e Expose max_background_jobs option in RocksJava
Summary:
This option was introduced in the C++ API in RocksDB 5.6 in bb01c1880c . Now, exposing it through RocksJava API.
Closes https://github.com/facebook/rocksdb/pull/2908

Differential Revision: D5864224

Pulled By: sagar0

fbshipit-source-id: 140aa55dcf74b14e4d11219d996735c7fdddf513
2017-09-20 10:26:37 -07:00
Pengchao Wang
e4234fbdcf collecting kValue type tombstone
Summary:
In our testing cluster, we found large amount tombstone has been promoted to kValue type from kMerge after reaching the top level of compaction. Since we used to only collecting tombstone in merge operator, those tombstones can never be collected.

This PR addresses the issue by adding a GC step in compaction filter, which is only for kValue type records. Since those record already reached the top of compaction (no earlier data exists) we can safely remove them in compaction filter without worrying old data appears.

This PR also removes an old optimization in cassandra merge operator for single merge operands.  We need to do GC even on a single operand, so the optimation does not make sense anymore.
Closes https://github.com/facebook/rocksdb/pull/2855

Reviewed By: sagar0

Differential Revision: D5806445

Pulled By: wpc

fbshipit-source-id: 6eb25629d4ce917eb5e8b489f64a6aa78c7d270b
2017-09-18 16:27:12 -07:00
Ben Clay
382277d0fe JNI support for ReadOptions::iterate_upper_bound
Summary:
Plumbed ReadOptions::iterate_upper_bound through JNI.

Made the following design choices:
* Used Slice instead of AbstractSlice due to the anticipated usecase (key / key prefix). Can change this if anyone disagrees.
* Used Slice instead of raw byte[] which seemed cleaner but necessitated the package-private handle-based Slice constructor. Followed WriteBatch as an example.
* We need a copy constructor for ReadOptions, as we create one base ReadOptions for a particular usecase and clone -> change the iterate_upper_bound on each slice operation. Shallow copy seemed cleanest.
* Hold a reference to the upper bound slice on ReadOptions, in contrast to Snapshot.

Signed a Facebook CLA this morning.
Closes https://github.com/facebook/rocksdb/pull/2872

Differential Revision: D5824446

Pulled By: sagar0

fbshipit-source-id: 74fc51313a10a81ecd348625e2a50ca5b7766888
2017-09-14 18:28:20 -07:00
zawlazaw
044a71e27e Add iterator's SeekForPrev functionality to the java-api
Summary:
As discussed in #2742 , this pull-requests brings the iterator's [SeekForPrev()](https://github.com/facebook/rocksdb/wiki/SeekForPrev) functionality to the java-api. It affects all locations in the code where previously only Seek() was supported.

All code changes are essentially a copy & paste of the already existing implementations for Seek().
**Please Note**: the changes to the C++ code were applied without fully understanding its effect, so please take a closer look. However, since Seek() and SeekForPrev() provide exactly the same signature, I do not expect any mistake here.

The java-tests are extended by new tests for the additional functionality.

Compilation (`make rocksdbjavastatic`) and test (`java/make test`) run without errors.
Closes https://github.com/facebook/rocksdb/pull/2747

Differential Revision: D5721011

Pulled By: sagar0

fbshipit-source-id: c1f951cddc321592c70dd2d32bc04892f3f119f8
2017-09-12 10:56:29 -07:00
Pengchao Wang
825a22c00c garbage collect tombstones in merge operator
Summary:
Remove cassandra tombstone when reaching the max compaction level (full merge). if all columns collected key will be removed in next compaction via compaction filter
Closes https://github.com/facebook/rocksdb/pull/2791

Reviewed By: sagar0

Differential Revision: D5722465

Pulled By: wpc

fbshipit-source-id: 61e9898a5686551653a16383255aeaab3197e65e
2017-08-31 10:11:54 -07:00
Chen Shen
78cb6b6112 Provide byte[] version of SstFileWriter.merge to reduce GC Stall
Summary:
In Java API, `SstFileWriter.put/merge/delete` takes `Slice` type of key and value, which is a Java wrapper object around C++ Slice object.  The Slice object inherited [ `finalize`](3c327ac2d0/java/src/main/java/org/rocksdb/AbstractNativeReference.java (L69)) method, which [added huge overhead](https://softwareengineering.stackexchange.com/questions/288715/is-overriding-object-finalize-really-bad/288753#288753) to JVM while creating new SstFile.

To address this issue, this PR overload the merge method to take Java byte array instead of the Slice object, and added unit test for it.

We also benchmark these two different merge function, where we could see GC Stall reduced from 50%  to 1%, and the throughput increased from 50MB to 200MB.
Closes https://github.com/facebook/rocksdb/pull/2746

Reviewed By: sagar0

Differential Revision: D5653145

Pulled By: scv119

fbshipit-source-id: b55ea58554b573d0b1c6f6170f8d9223811bc4f5
2017-08-22 12:55:24 -07:00
Adam Retter
a144a9782d Fix for CMakeLists.txt on Windows for RocksJava
Summary: Closes https://github.com/facebook/rocksdb/pull/2730

Differential Revision: D5619256

Pulled By: ajkr

fbshipit-source-id: c80d697eeceab91964259132e58f5cd2219efb93
2017-08-12 16:44:12 -07:00
Sagar Vemuri
3453870677 Fix statistics in RocksJava sample
Summary:
I observed while doing a `make jtest` that the java sample was broken, due to the changes in #2551 .
Closes https://github.com/facebook/rocksdb/pull/2674

Differential Revision: D5539807

Pulled By: sagar0

fbshipit-source-id: 2c7e9d84778099dfa1c611996b444efe3c9fd466
2017-08-01 16:58:26 -07:00
Sagar Vemuri
aace46516b Fix license headers in Cassandra related files
Summary:
I might have missed these while doing some recent cassandra code reviews.
Closes https://github.com/facebook/rocksdb/pull/2663

Differential Revision: D5520138

Pulled By: sagar0

fbshipit-source-id: 340930afe9efe03c75f535a1da1f89bd3e53c1f9
2017-07-28 13:56:56 -07:00
Daniel Black
1d8aa2961c Gcc 7 ParsedInternalKey replace memset with clear function.
Summary:
I haven't looked to see if a class variable inside a loop like this is always initialised.
Closes https://github.com/facebook/rocksdb/pull/2602

Differential Revision: D5475937

Pulled By: IslamAbdelRahman

fbshipit-source-id: 8570b308f9a4b49e2a56ccc9e9b84d7c46568c15
2017-07-24 11:31:15 -07:00
Pengchao Wang
534c255c7a Cassandra compaction filter for purge expired columns and rows
Summary:
Major changes in this PR:
* Implement CassandraCompactionFilter to remove expired columns and rows (if all column expired)
* Move cassandra related code from utilities/merge_operators/cassandra to utilities/cassandra/*
* Switch to use shared_ptr<> from uniqu_ptr for Column membership management in RowValue. Since columns do have multiple owners in Merge and GC process, use shared_ptr helps make RowValue immutable.
* Rename cassandra_merge_test to cassandra_functional_test and add two TTL compaction related tests there.
Closes https://github.com/facebook/rocksdb/pull/2588

Differential Revision: D5430010

Pulled By: wpc

fbshipit-source-id: 9566c21e06de17491d486a68c70f52d501f27687
2017-07-21 14:57:44 -07:00
Siying Dong
6e3ee015fb Update java/rocksjni.pom
Summary: Closes https://github.com/facebook/rocksdb/pull/2610

Differential Revision: D5445705

Pulled By: siying

fbshipit-source-id: d5f97ffdf5bc18b853c3a106755aca96a650e428
2017-07-18 13:12:23 -07:00
Siying Dong
3c327ac2d0 Change RocksDB License
Summary: Closes https://github.com/facebook/rocksdb/pull/2589

Differential Revision: D5431502

Pulled By: siying

fbshipit-source-id: 8ebf8c87883daa9daa54b2303d11ce01ab1f6f75
2017-07-15 16:11:23 -07:00
Daniel Black
7550255c54 Add JAVAC_ARGS as Makefile variable
Summary:
Set to "-Xlint:deprecation -Xlint:unchecked" for non-zero
DEBUG_LEVEL

Shows up a few warnings for later perusal however passes ok.
Closes https://github.com/facebook/rocksdb/pull/2557

Differential Revision: D5408104

Pulled By: sagar0

fbshipit-source-id: 02908602e09b46d33b0dfc0ffd4b11954db03c53
2017-07-12 12:12:43 -07:00
Adam Retter
000bf0af38 Improve the design and native object management of Stats in RocksJava
Summary: Closes https://github.com/facebook/rocksdb/pull/2551

Differential Revision: D5399288

Pulled By: sagar0

fbshipit-source-id: dd3df2ed6cc5ae612db0998ea746cc29fccf568e
2017-07-11 23:31:00 -07:00
Siying Dong
e517bfa2c2 CLANG Tidy
Summary: Closes https://github.com/facebook/rocksdb/pull/2502

Differential Revision: D5326498

Pulled By: siying

fbshipit-source-id: 2f0ac6dc6ca5ddb23cecf67a278c086e52646714
2017-06-27 11:00:59 -07:00
Yi Wu
2a9cd87997 Fix jni WriteBatchThreadedTest
Summary:
WriteBatchThreadedTest is failing, at least on Mac. The problem seems to be `wb` is getting GC before we finish write. Explicitly close it seems to fix it.
Closes https://github.com/facebook/rocksdb/pull/2482

Differential Revision: D5307379

Pulled By: yiwu-arbug

fbshipit-source-id: 8ff7f8170451078c941951f5aafae83afffb7933
2017-06-26 15:27:17 -07:00
Chen Shen
cbd825deea Create a MergeOperator for Cassandra Row Value
Summary:
This PR implements the MergeOperator for Cassandra Row Values.
Closes https://github.com/facebook/rocksdb/pull/2289

Differential Revision: D5055464

Pulled By: scv119

fbshipit-source-id: 45f276ef8cbc4704279202f6a20c64889bc1adef
2017-06-16 14:27:00 -07:00
Ben Torfs
6a3377f454 Synchronize statistic enumeration values between statistics.h and java API
Summary: Closes https://github.com/facebook/rocksdb/pull/2209

Differential Revision: D5251951

Pulled By: sagar0

fbshipit-source-id: 03a73d025a7b4a322bb8d8d86f5d249fcd7dd00e
2017-06-14 16:59:42 -07:00
hyunwoo
6b5a5dc5d8 fixed typo
Summary:
fixed typo
Closes https://github.com/facebook/rocksdb/pull/2430

Differential Revision: D5242471

Pulled By: IslamAbdelRahman

fbshipit-source-id: 832eb3a4c70221444ccd2ae63217823fec56c748
2017-06-13 16:58:01 -07:00
Sagar Vemuri
27b450165e Update HistogramTypes in the Java API
Summary:
This diff syncs the Histogram Types in the Java API with the ones in C++ API (`statistics.h`), and brings it up-to-date.

I also found that the enum ordering between Java and C++ has gotten out-of-sync, a few years back, with the addition of `SUBCOMPACTION_SETUP_TIME`. So updated the order as well.

`READ_NUM_MERGE_OPERANDS` added in #2373 is needed for Cassandra-on-RocksDB work.
Closes https://github.com/facebook/rocksdb/pull/2429

Differential Revision: D5215623

Pulled By: sagar0

fbshipit-source-id: bd136698c48197e53693275eb52acc9198ee5a4e
2017-06-12 12:57:49 -07:00
Sagar Vemuri
c2012d488f Java APIs for put, merge and delete in file ingestion
Summary:
Adding SSTFileWriter's newly introduced put, merge and delete apis to the Java api. The C++ APIs were first introduced in #2361.

Add is deprecated in favor of Put.
Merge is especially needed to support streaming for Cassandra-on-RocksDB work in https://issues.apache.org/jira/browse/CASSANDRA-13476.
Closes https://github.com/facebook/rocksdb/pull/2392

Differential Revision: D5165091

Pulled By: sagar0

fbshipit-source-id: 6f0ad396a7cbd2e27ca63e702584784dd72acaab
2017-06-08 17:11:57 -07:00
Sagar Vemuri
b172a3f1a2 Fix warnings while generating RocksJava documentation
Summary:
There are a couple of warnings while building RocksJava, coming from Javadoc generation.
```
Generating target/apidocs/org/rocksdb/RocksDB.html...
src/main/java/org/rocksdb/RocksDB.java:2139: warning: no throws for org.rocksdb.RocksDBException
  public void ingestExternalFile(final List<String> filePathList,
              ^
src/main/java/org/rocksdb/RocksDB.java:2162: warning: no throws for org.rocksdb.RocksDBException
  public void ingestExternalFile(final ColumnFamilyHandle columnFamilyHandle,
              ^
```
Closes https://github.com/facebook/rocksdb/pull/2396

Differential Revision: D5178388

Pulled By: sagar0

fbshipit-source-id: a0ab6696d6de78d089a9a860a559f64cc320019e
2017-06-05 15:28:00 -07:00
Adam Retter
26a8a80711 Switch from CentOS 5 to CentOS 6 for crossbuilding RocksJava
Summary:
Updates the statically linked libraries from linking against glibc 2.5, to linking against glibc 2.12.
Closes https://github.com/facebook/rocksdb/pull/2405

Differential Revision: D5184132

Pulled By: sagar0

fbshipit-source-id: 7a8ad4cf7e737ca62f29e58938bd49fa02114541
2017-06-05 12:27:24 -07:00
hyunwoo
c7662a44a4 fixed typo
Summary:
fixed typo
Closes https://github.com/facebook/rocksdb/pull/2376

Differential Revision: D5183630

Pulled By: ajkr

fbshipit-source-id: 133cfd0445959e70aa2cd1a12151bf3c0c5c3ac5
2017-06-05 11:27:34 -07:00
Adam Retter
7e8d95cc0e Fix the Java build which was broken by a4d9c02
Summary: Closes https://github.com/facebook/rocksdb/pull/2406

Differential Revision: D5181091

Pulled By: ajkr

fbshipit-source-id: fd72525da4fb1d50143080a210f8d824cbb968d6
2017-06-04 18:41:33 -07:00
Aaron Gao
f7bb1a0060 support merge and delete in file ingestion
Summary:
Previously sst_file_writer only supports kTypeValue, we need kTypeMerge and kTypeDeletion also as user requested.
Closes https://github.com/facebook/rocksdb/pull/2361

Differential Revision: D5139402

Pulled By: lightmark

fbshipit-source-id: 092a60756d01692539d817a3765ebfd58a8d7f88
2017-05-26 12:11:21 -07:00
Adam Retter
5a25304627 Fix the CMakeLists for RocksJava
Summary: Closes https://github.com/facebook/rocksdb/pull/2356

Differential Revision: D5122240

Pulled By: sagar0

fbshipit-source-id: 3764797dd261d5b2a04d58b3967969e66c541bbb
2017-05-24 11:29:08 -07:00
Adam Retter
88c818e437 Replace deprecated RocksDB#addFile with RocksDB#ingestExternalFile
Summary:
Previously the Java implementation of `RocksDB#addFile` was both incomplete and not inline with the C++ API.

Rather than fix it, as I see that `rocksdb::DB::AddFile` is now deprecated in favour of `rocksdb::DB::IngestExternalFile`, I have removed the old broken implementation and implemented `RocksDB#ingestExternalFile`.

Closes https://github.com/facebook/rocksdb/issues/2261
Closes https://github.com/facebook/rocksdb/pull/2291

Differential Revision: D5061264

Pulled By: sagar0

fbshipit-source-id: 85df0899fa1b1fc3535175cac4f52353511d4104
2017-05-22 10:27:23 -07:00
Adam Retter
254c468009 Fix the RocksJava Release on Windows
Summary:
This was previously broken accidentally by https://github.com/facebook/rocksdb/pull/2107

Closes https://github.com/facebook/rocksdb/issues/2293
Closes https://github.com/facebook/rocksdb/pull/2296

Differential Revision: D5061248

Pulled By: sagar0

fbshipit-source-id: 7f58fee754723a7052d2a7f9d3d0369051c3cc5c
2017-05-15 12:37:01 -07:00
Adam Retter
a5cc7ecec4 Facility for cross-building RocksJava using Docker
Summary:
As an alternative to Vagrant, we can now also use Docker to cross-build RocksDB. The advantages are:

1. The Docker images are fixed; they include all the latest updates and build tools.
2. The Vagrant image, required scripts that ran for every build that would update CentOS and install the buildtools. This lead to slow repeatable builds, we don't need to do this with Docker as they are already in the provided images.

The Docker images I have used have their Docker build files here: https://github.com/evolvedbinary/docker-rocksjava and the images themselves are available from Docker hub: https://hub.docker.com/r/evolvedbinary/rocksjava/

I have added the following targets to the `Makefile`:
1. `rocksdbjavastaticreleasedocker` this uses Docker to perform the cross-builds. It is basically the Docker version of the existing Vagrant `rocksdbjavastaticrelease` target.
2. `rocksdbjavastaticpublishdocker` delegates to `rocksdbjavastaticreleasedocker` and then `rocksdbjavastaticpublishcentral` to upload the artiacts to Maven Central. Equivalent to the existing Vagrant target: `rocksdbjavastaticpublish`
Closes https://github.com/facebook/rocksdb/pull/2278

Differential Revision: D5048206

Pulled By: yiwu-arbug

fbshipit-source-id: 78fa96ef9d966fe09638ed01de282cd4e31961a9
2017-05-12 11:41:21 -07:00
Min Wei
3a04a254ac Flink state
Summary:
This is to address the issue reported in
    https://github.com/facebook/rocksdb/issues/1988
The fix is simple. A typo.
Closes https://github.com/facebook/rocksdb/pull/2267

Differential Revision: D5037149

Pulled By: siying

fbshipit-source-id: 1bb585c7a753ef77c81c4b92deafbed8e21fe8ff
2017-05-10 11:12:38 -07:00
Adam Retter
e7cea86f77 Fixes the CentOS 5 cross-building of RocksJava
Summary:
Updates to CentOS 5 have been archived as CentOS 5 is EOL. We now pull the updates from the vault. This is a stop gap solution, I will send a PR in a couple days which uses fixed Docker containers (with the updates pre-installed) instead.

sagar0 Here you go :-)
Closes https://github.com/facebook/rocksdb/pull/2270

Differential Revision: D5033637

Pulled By: sagar0

fbshipit-source-id: a9312dd1bc18bfb8653f06ffa0a1512b4415720d
2017-05-10 08:57:54 -07:00
Adam Retter
498693cf3e Remove orphaned Java classes
Summary:
Remove orphaned Java classes that were missed from commit 9385fd7
Closes https://github.com/facebook/rocksdb/issues/2210
Closes https://github.com/facebook/rocksdb/pull/2212

Differential Revision: D4986199

Pulled By: maysamyabandeh

fbshipit-source-id: 74e866cd8661c172a4754c2c945e30d650f6fa05
2017-05-02 21:13:49 -07:00
Tomas Kolda
04d58970cb AIX and Solaris Sparc Support
Summary:
Replacement of #2147

The change was squashed due to a lot of conflicts.
Closes https://github.com/facebook/rocksdb/pull/2194

Differential Revision: D4929799

Pulled By: siying

fbshipit-source-id: 5cd49c254737a1d5ac13f3c035f128e86524c581
2017-04-21 20:48:04 -07:00
Aaron Gao
44fa8ece9b change use_direct_writes to use_direct_io_for_flush_and_compaction
Summary:
Replace Options::use_direct_writes with Options::use_direct_io_for_flush_and_compaction
Now if Options::use_direct_io_for_flush_and_compaction = true, we will enable direct io for both reads and writes for flush and compaction job. Whereas Options::use_direct_reads controls user reads like iterator and Get().
Closes https://github.com/facebook/rocksdb/pull/2117

Differential Revision: D4860912

Pulled By: lightmark

fbshipit-source-id: d93575a8a5e780cf7e40797287edc425ee648c19
2017-04-13 16:12:04 -07:00
Daniel Black
3eab41d7c4 java dependencies test -s -> use test -d
Summary:
To correct a build process where the JAVA_TEST_LIBDIR is a symlink to a cache directory.

Test -s (size 0) on symlinks returns true, resulting in a mkdir over the top of the symlink resulting in failure.

As a solution -d checks if it is a directory (or the symlink refers to a directory), which works in the case of real directories and symlinks to directories.

Trivial I know but it was really easy for me to use a symlink here to prevent frequent downloads in a CI environment.

Thanks for your consideration.
Closes https://github.com/facebook/rocksdb/pull/1917

Differential Revision: D4612263

Pulled By: siying

fbshipit-source-id: 4d458f8e1760068cdd6b5eae4bce6e12c400df41
2017-04-12 15:13:41 -07:00
Sagar Vemuri
343b59d6ee Move various string utility functions into string_util
Summary:
This is an effort to club all string related utility functions into one common place, in string_util, so that it is easier for everyone to know what string processing functions are available. Right now they seem to be spread out across multiple modules, like logging and options_helper.

Check the sub-commits for easier reviewing.
Closes https://github.com/facebook/rocksdb/pull/2094

Differential Revision: D4837730

Pulled By: sagar0

fbshipit-source-id: 344278a
2017-04-06 14:54:12 -07:00
Ayappan
1d068f6067 Fix CompactRange incorrect buffer release
Summary:
While running `make jtest` using IBM Java, it fails at compactRangeToLevel with the below error.

```
Run: org.rocksdb.RocksDBTest testing now -> compactRangeToLevel
JVMJNCK056E JNI error in ReleaseByteArrayElements: Got memory 0x00003FFF94AA8908 from object 0x00000000000C7F78, releasing from 0x00000000000C7F68
JVMJNCK077E Error detected in org/rocksdb/RocksDB.compactRange0(J[BI[BIZII)V

JVMJNCK024E JNI error detected. Aborting.
JVMJNCK025I Use -Xcheck:jni:nonfatal to continue running when errors are detected.

Fatal error: JNI error
Makefile:205: recipe for target 'run_test' failed
make[1]: *** [run_test] Error 87
make[1]: Leaving directory '/home/ubuntu/rocksdb/java'
Makefile:1542: recipe for target 'jtest' failed
make: *** [jtest] Error 2
```

After checking the code, it is vivid that we are messing up the `ReleaseByteArrayElements` args in `rocksdb_compactrange_helper`.

```
   .................
   1959     s = db->CompactRange(compact_options, &begin_slice, &end_slice);
   1960   }
Closes https://github.com/facebook/rocksdb/pull/2060

Differential Revision: D4831427

Pulled By: yiwu-arbug

fbshipit-source-id: dd02037
2017-04-06 14:09:13 -07:00
Siying Dong
d2dce5611a Move some files under util/ to separate dirs
Summary:
Move some files under util/ to new directories env/, monitoring/ options/ and cache/
Closes https://github.com/facebook/rocksdb/pull/2090

Differential Revision: D4833681

Pulled By: siying

fbshipit-source-id: 2fd8bef
2017-04-05 19:09:16 -07:00
Adam Retter
4ab4049f27 gflags has moved to GitHub
Summary:
Closes https://github.com/facebook/rocksdb/issues/2068
Closes https://github.com/facebook/rocksdb/pull/2072

Differential Revision: D4810855

Pulled By: ajkr

fbshipit-source-id: 288ddb9
2017-03-31 10:09:21 -07:00
Adam Retter
0ee7f04039 Added missing options to RocksJava
Summary:
This adds almost all missing options to RocksJava
Closes https://github.com/facebook/rocksdb/pull/2039

Differential Revision: D4779991

Pulled By: siying

fbshipit-source-id: 4a1bf28
2017-03-30 12:09:21 -07:00
Mikhail Antonov
f9813b853d Added SstFileWriter construtor without explicit comparator to JNI api
Summary:
Adding API missing after 1ffbdfd9a7 (diff-b94146418eed4a9c1bf324041b95b279).

adamretter  IslamAbdelRahman

Tested locally.
Closes https://github.com/facebook/rocksdb/pull/2028

Differential Revision: D4762817

Pulled By: IslamAbdelRahman

fbshipit-source-id: 833f478
2017-03-29 17:24:14 -07:00
Ayappan
69c8d524a3 Fix jni library name for PowerPC Architecture
Summary:
Right now, building rocksdbjava in PowerPC is broken due to JNI library name. I figured it out that "uname -m" and java's os.arch matches in PowerPC architecture. I made use of this advantage to fix the issue. More info can found from this issue  --> https://github.com/facebook/rocksdb/issues/1317
Closes https://github.com/facebook/rocksdb/pull/2040

Differential Revision: D4779967

Pulled By: siying

fbshipit-source-id: 259f939
2017-03-27 14:09:11 -07:00
Ayappan
203136e793 Fix Compilation errors when using IBM Java
Summary:
PR to fix this issue -> https://github.com/facebook/rocksdb/issues/1926
Closes https://github.com/facebook/rocksdb/pull/1965

Differential Revision: D4682411

Pulled By: siying

fbshipit-source-id: a519be1
2017-03-22 18:09:10 -07:00
Raza Hussain
6908e24b56 dynamic setting of stats_dump_period_sec through SetDBOption()
Summary:
Resolved the following issue: https://github.com/facebook/rocksdb/issues/1930
Closes https://github.com/facebook/rocksdb/pull/2004

Differential Revision: D4736764

Pulled By: yiwu-arbug

fbshipit-source-id: 64fe0b7
2017-03-20 22:54:13 -07:00
Matteo Merli
5f65dc8778 Expose DB::DeleteRange and WriteBath::DeleteRange in Java
Summary:
Added JNI wrapper from `DeleteRange` methods
Closes https://github.com/facebook/rocksdb/pull/1951

Differential Revision: D4657746

Pulled By: yiwu-arbug

fbshipit-source-id: 3fc7ab8
2017-03-06 22:24:26 -08:00
Adam Retter
c6d464a9da Fixed various memory leaks and Java 8 JNI Compatibility
Summary:
I have manually audited the entire RocksJava code base.

Sorry for the large pull-request, I have broken it down into many small atomic commits though.

My initial intention was to fix the warnings that appear when running RocksJava on Java 8 with `-Xcheck:jni`, for example when running `make jtest` you would see many errors similar to:

```
WARNING in native method: JNI call made without checking exceptions when required to from CallObjectMethod
WARNING in native method: JNI call made without checking exceptions when required to from CallVoidMethod
WARNING in native method: JNI call made without checking exceptions when required to from CallStaticVoidMethod
...
```

A few of those warnings still remain, however they seem to come directly from the JVM and are not directly related to RocksJava; I am in contact with the OpenJDK hostpot-dev mailing list about these - http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-February/025981.html.

As a result of fixing these, I realised we were not r
Closes https://github.com/facebook/rocksdb/pull/1890

Differential Revision: D4591758

Pulled By: siying

fbshipit-source-id: 7f7fdf4
2017-02-27 16:39:12 -08:00
Siying Dong
8efb5ffa2a [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp…
Summary:
…action

 The two options, min_partial_merge_operands and verify_checksums_in_compaction, are not seldom used. Remove them to reduce the total number of options. Also remove them from Java and C interface.
Closes https://github.com/facebook/rocksdb/pull/1902

Differential Revision: D4601219

Pulled By: siying

fbshipit-source-id: aad4cb2
2017-02-23 15:09:12 -08:00
Sagar Vemuri
eb912a927e Remove disableDataSync option
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859

Differential Revision: D4541292

Pulled By: sagar0

fbshipit-source-id: 5b3a6ca
2017-02-13 11:09:13 -08:00
Adam Retter
421ce7c2ba Add support for JNI Library on Linux on PowerPC.
Summary:
Closes https://github.com/facebook/rocksdb/issues/1317
Closes https://github.com/facebook/rocksdb/pull/1845

Differential Revision: D4546491

Pulled By: siying

fbshipit-source-id: 86190b4
2017-02-10 15:09:10 -08:00
Maysam Yabandeh
69d5262c81 Two-level Indexes
Summary:
Partition Index blocks and use a Partition-index as a 2nd level index.

The two-level index can be used by setting
BlockBasedTableOptions::kTwoLevelIndexSearch as the index type and
configuring BlockBasedTableOptions::index_per_partition

t15539501
Closes https://github.com/facebook/rocksdb/pull/1814

Differential Revision: D4473535

Pulled By: maysamyabandeh

fbshipit-source-id: bffb87e
2017-02-06 16:39:12 -08:00
Min Wei
af6ec4d78e fix batchresult handle leak
Summary:
This is related to PR https://github.com/facebook/rocksdb/pull/1642

Sorry about this extra PR, as my workflow was messed up when I checked in another work item by accident.
Closes https://github.com/facebook/rocksdb/pull/1681

Differential Revision: D4379513

Pulled By: yiwu-arbug

fbshipit-source-id: a668d4c
2017-01-20 13:24:12 -08:00
Adam Retter
85ac1a320a Fix rocksdb::Status::getState
Summary:
This fixes the Java API for Status#getState use in Native code and also simplifies the implementation of rocksdb::Status::getState.
Closes https://github.com/facebook/rocksdb/issues/1688
Closes https://github.com/facebook/rocksdb/pull/1714

Differential Revision: D4364181

Pulled By: yiwu-arbug

fbshipit-source-id: 8e073b4
2017-01-03 18:39:14 -08:00
Aaron Gao
972f96b3fb direct io write support
Summary:
rocksdb direct io support

```
[gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 5.0
Date:       Wed Nov 23 13:17:43 2016
CPU:        40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
CPUCache:   25600 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :       4.393 micros/op 227639 ops/sec;   25.2 MB/s

[gzh@dev11575.prn2 ~/roc
Closes https://github.com/facebook/rocksdb/pull/1564

Differential Revision: D4241093

Pulled By: lightmark

fbshipit-source-id: 98c29e3
2016-12-22 13:09:19 -08:00
Daniel Black
816c1e30ca gcc-7 requires include <functional> for std::function
Summary:
Fixes compile error:

In file included from ./util/statistics.h:17:0,
                 from ./util/stop_watch.h:8,
                 from ./util/perf_step_timer.h:9,
                 from ./util/iostats_context_imp.h:8,
                 from ./util/posix_logger.h:27,
                 from ./port/util_logger.h:18,
                 from ./db/auto_roll_logger.h:15,
                 from db/auto_roll_logger.cc:6:
./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type
   typedef std::function<void(void*, void*)> FoldFunc;
Closes https://github.com/facebook/rocksdb/pull/1656

Differential Revision: D4318702

Pulled By: yiwu-arbug

fbshipit-source-id: 8c5d17a
2016-12-16 11:24:18 -08:00
Adam Retter
3cdfaeca3f Fixes for MSVC compilation
Summary: Closes https://github.com/facebook/rocksdb/pull/1669

Differential Revision: D4327421

Pulled By: yiwu-arbug

fbshipit-source-id: 661ee0b
2016-12-14 10:24:13 -08:00
Yi Wu
36e4762ce0 Remove Ticker::SEQUENCE_NUMBER
Summary:
Remove the ticker count because:
* Having to reset the ticker count in WriteImpl is ineffiecent;
* It doesn't make sense to have it as a ticker count if multiple db
  instance share a statistics object.
Closes https://github.com/facebook/rocksdb/pull/1531

Differential Revision: D4194442

Pulled By: yiwu-arbug

fbshipit-source-id: e2110a9
2016-11-16 22:39:09 -08:00
Adam Retter
24bceb0963 Java API - Implement GetFromBatch and GetFromBatchAndDB in WBWI
Summary:
Needed for working with `get` after `merge` on a WBWI.
Closes https://github.com/facebook/rocksdb/pull/1093

Differential Revision: D4137978

Pulled By: yhchiang

fbshipit-source-id: e18d50d
2016-11-06 02:44:07 -08:00
Benoit Girard
2b16d664cb Change max_bytes_for_level_multiplier to double
Summary: Closes https://github.com/facebook/rocksdb/pull/1427

Differential Revision: D4094732

Pulled By: yiwu-arbug

fbshipit-source-id: b9b79e9
2016-11-01 21:09:23 -07:00
Adam Retter
ce22ea99a0 Fix casts for MSVC
Summary:
I am not sure if this is the best way to fix this?
Closes https://github.com/facebook/rocksdb/pull/1452

Differential Revision: D4109338

Pulled By: yiwu-arbug

fbshipit-source-id: ca40809
2016-11-01 10:09:16 -07:00
Islam AbdelRahman
869ae5d786 Support IngestExternalFile (remove AddFile restrictions)
Summary:
Changes in the diff

API changes:
- Introduce IngestExternalFile to replace AddFile (I think this make the API more clear)
- Introduce IngestExternalFileOptions (This struct will encapsulate the options for ingesting the external file)
- Deprecate AddFile() API

Logic changes:
- If our file overlap with the memtable we will flush the memtable
- We will find the first level in the LSM tree that our file key range overlap with the keys in it
- We will find the lowest level in the LSM tree above the the level we found in step 2 that our file can fit in and ingest our file in it
- We will assign a global sequence number to our new file
- Remove AddFile restrictions by using global sequence numbers

Other changes:
- Refactor all AddFile logic to be encapsulated in ExternalSstFileIngestionJob

Test Plan:
unit tests (still need to add more)
addfile_stress (https://reviews.facebook.net/D65037)

Reviewers: yiwu, andrewkr, lightmark, yhchiang, sdong

Reviewed By: sdong

Subscribers: jkedgar, hcz, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D65061
2016-10-20 17:05:32 -07:00
Adam Retter
48fd619a47 Minor fixes to RocksJava Native Library initialization (#1287)
* [bugfix] Make sure the Native Library is initialized. Closes https://github.com/facebook/rocksdb/issues/989

* [bugfix] Just load the native libraries once
2016-10-19 18:21:22 -07:00
Siying Dong
257de78d9b Remove "-Xcheck:jni" from Java tests (#1402)
Summary:
Junit and our code generate lots of warning if "-Xcheck:jni" is on and force Travis to fail as the logs are too long.

Test Plan: "make jtest" and see the warnings go away.
2016-10-18 09:18:24 -04:00
Siying Dong
fea6fdd676 Fix @see in two Java functions (#1396) 2016-10-14 23:03:17 -07:00
Adam Retter
2e4b5cab0e Add missing RateLimiter class to the Windows build (#1382) 2016-10-12 15:00:37 -07:00
Adam Retter
5cd28833a2 [RocksJava] Adjusted RateLimiter to 3.10.0 (#1368)
Summary:
- Deprecated RateLimiterConfig and GenericRateLimiterConfig
- Introduced RateLimiter

It is now possible to use all C++ related methods also in RocksJava.
A noteable method is setBytesPerSecond which can change the allowed
number of bytes per second at runtime.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35715
2016-10-07 12:32:21 -07:00
Adam Retter
d346ba2468 Minor fixes around Windows 64 Java Artifacts (#1366) 2016-10-03 11:58:08 -07:00
Adam Retter
3c21c64c78 Use size hint for HashMap in multiGet. Similar to https://github.com/facebook/rocksdb/pull/1344 (#1367) 2016-09-29 15:55:53 -07:00
Tomas Kolda
13f7a01f61 Fixing JNI release build for gcc (#975) 2016-09-29 17:11:32 -04:00
Injun Song
7260662b39 Add Java API for SstFileWriter
Add Java API for SstFileWriter. Closes https://github.com/facebook/rocksdb/issues/1248
2016-09-29 17:04:41 -04:00
Yi Wu
41a9070f84 Fix java makefile dependencies
Summary: Fix dependencies in java makefile, to avoid java build failure.

Test Plan: run "make rocksdbjava" multiple times.

Reviewers: IslamAbdelRahman, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D64023
2016-09-16 10:54:31 -07:00
Yi Wu
a958c26435 Rename jvalue to jval in rocksjni
Summary: jvalue shadows a global name in <jni.h>. Rename it to jval to fix java build.

Test Plan:
    JAVA_HOME=/usr/local/jdk-7u10-64 make rocksdbjava -j64

Reviewers: adamretter, yhchiang, IslamAbdelRahman

Reviewed By: yhchiang, IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63981
2016-09-14 13:12:55 -07:00
xh931076284
3639f32889 Fix bug in UnScSigned-off-by: xh931076284 <931076284@qq.com> (#1336)
Fix HdfsEnv::UnSchedule() API error
2016-09-14 10:17:34 -07:00
Yi Wu
81747f1be6 Refactor MutableCFOptions
Summary:
* Change constructor of MutableCFOptions to depends only on ColumnFamilyOptions.
* Move `max_subcompactions`, `compaction_options_fifo` and `compaction_pri` to ImmutableCFOptions to make it clear that they are immutable.

Test Plan: existing unit tests.

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63945
2016-09-13 21:11:59 -07:00
Adam Retter
22d88e24db Allow an offset as well as a length to be specified for byte[] operations in RocksJava JNI (#1264)
Test Plan: Execute the Java test suite

Reviewers: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61971
2016-09-12 11:51:08 -07:00
Yi Wu
a88677d2cf Remove ImmutableCFOptions from public API
Summary: There's no reference to ImmutableCFOptions elsewhere in /include/rocksdb. ImmutableCFOptions was introduced in this commit (5665e5e285) but later its reference in /include/rocksdb/table.h is removed.

Test Plan:
  make all check

Reviewers: IslamAbdelRahman, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: yhchiang, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63177
2016-09-02 14:16:31 -07:00
sdong
32149059f9 Merge options source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes
Summary: To reduce number of options, merge source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes.

Test Plan: Add two new unit tests. Run all existing tests, including jtest.

Reviewers: yhchiang, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59829
2016-09-01 14:33:24 -07:00
Adam Retter
f85f99bf69 Fix the Windows build of RocksDB Java. Similar to https://github.com/facebook/rocksdb/issues/1220 (#1284) 2016-08-25 10:16:26 -07:00
Adam Retter
817eeb29b4 Add singleDelete to RocksJava (#1275)
* Rename RocksDB#remove -> RocksDB#delete to match C++ API; Added deprecated versions of RocksDB#remove for backwards compatibility.

* Add missing experimental feature RocksDB#singleDelete
2016-08-22 11:02:31 -07:00
Adam Retter
ffdf6eee19 Add Status to RocksDBException so that meaningful function result Status from the C++ API isn't lost (#1273) 2016-08-22 11:01:42 -07:00
Alexander Jipa
0b63f51fbc fixes 1215: execute_process(COMMAND mkdir ${DIR}) fails to create a directory with cmake on Windows (#1219)
CMake - Use a platform-neutral mkdir function
2016-08-18 00:59:22 +01:00
Islam AbdelRahman
dd7a748cff Fix java build
Summary: Fix the java build

Test Plan: make rocksdbjava -j64

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D62097
2016-08-15 14:03:33 -07:00
Adam Retter
d367555027 Added further Java API options for controlling concurrent writes 2016-08-11 16:33:20 -07:00
Adam Retter
ebdfe34cc4 Exposed further Java API options for controlling compaction 2016-08-11 16:33:20 -07:00
Adam Retter
f4d986364e Added SetOptions support to RocksJava (#1243)
* [refactor] Split Java ColumnFamilyOptions into mutable and immutable and implement any missing immutable options

* [feature] Implement RocksDB#setOptions
2016-08-06 12:03:47 -07:00
Adam Retter
9253767a6a Correct geHistogramData() -> getHistogramData() (#1257) 2016-08-04 20:16:16 -07:00
Adam Retter
9ae92f50b2 More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237) 2016-07-29 12:55:54 -07:00
Islam AbdelRahman
7c01d65344 [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv
Summary: Remove duplicate cases in LoggerJniCallback::Logv and fix complication error

Test Plan: make jclean rocksdbjava jtest -j4

Reviewers: adamretter, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61335
2016-07-29 12:47:15 -07:00
Adam Retter
8796934af7 Added missing Java ReadOptions settings (#1109)
* Minor reorganisation

* Java API - Added missing ReadOptions settings
2016-07-29 12:12:03 -07:00