Commit Graph

280 Commits

Author SHA1 Message Date
Sagar Vemuri
343b59d6ee Move various string utility functions into string_util
Summary:
This is an effort to club all string related utility functions into one common place, in string_util, so that it is easier for everyone to know what string processing functions are available. Right now they seem to be spread out across multiple modules, like logging and options_helper.

Check the sub-commits for easier reviewing.
Closes https://github.com/facebook/rocksdb/pull/2094

Differential Revision: D4837730

Pulled By: sagar0

fbshipit-source-id: 344278a
2017-04-06 14:54:12 -07:00
Ayappan
1d068f6067 Fix CompactRange incorrect buffer release
Summary:
While running `make jtest` using IBM Java, it fails at compactRangeToLevel with the below error.

```
Run: org.rocksdb.RocksDBTest testing now -> compactRangeToLevel
JVMJNCK056E JNI error in ReleaseByteArrayElements: Got memory 0x00003FFF94AA8908 from object 0x00000000000C7F78, releasing from 0x00000000000C7F68
JVMJNCK077E Error detected in org/rocksdb/RocksDB.compactRange0(J[BI[BIZII)V

JVMJNCK024E JNI error detected. Aborting.
JVMJNCK025I Use -Xcheck:jni:nonfatal to continue running when errors are detected.

Fatal error: JNI error
Makefile:205: recipe for target 'run_test' failed
make[1]: *** [run_test] Error 87
make[1]: Leaving directory '/home/ubuntu/rocksdb/java'
Makefile:1542: recipe for target 'jtest' failed
make: *** [jtest] Error 2
```

After checking the code, it is vivid that we are messing up the `ReleaseByteArrayElements` args in `rocksdb_compactrange_helper`.

```
   .................
   1959     s = db->CompactRange(compact_options, &begin_slice, &end_slice);
   1960   }
Closes https://github.com/facebook/rocksdb/pull/2060

Differential Revision: D4831427

Pulled By: yiwu-arbug

fbshipit-source-id: dd02037
2017-04-06 14:09:13 -07:00
Siying Dong
d2dce5611a Move some files under util/ to separate dirs
Summary:
Move some files under util/ to new directories env/, monitoring/ options/ and cache/
Closes https://github.com/facebook/rocksdb/pull/2090

Differential Revision: D4833681

Pulled By: siying

fbshipit-source-id: 2fd8bef
2017-04-05 19:09:16 -07:00
Adam Retter
0ee7f04039 Added missing options to RocksJava
Summary:
This adds almost all missing options to RocksJava
Closes https://github.com/facebook/rocksdb/pull/2039

Differential Revision: D4779991

Pulled By: siying

fbshipit-source-id: 4a1bf28
2017-03-30 12:09:21 -07:00
Mikhail Antonov
f9813b853d Added SstFileWriter construtor without explicit comparator to JNI api
Summary:
Adding API missing after 1ffbdfd9a7 (diff-b94146418eed4a9c1bf324041b95b279).

adamretter  IslamAbdelRahman

Tested locally.
Closes https://github.com/facebook/rocksdb/pull/2028

Differential Revision: D4762817

Pulled By: IslamAbdelRahman

fbshipit-source-id: 833f478
2017-03-29 17:24:14 -07:00
Ayappan
203136e793 Fix Compilation errors when using IBM Java
Summary:
PR to fix this issue -> https://github.com/facebook/rocksdb/issues/1926
Closes https://github.com/facebook/rocksdb/pull/1965

Differential Revision: D4682411

Pulled By: siying

fbshipit-source-id: a519be1
2017-03-22 18:09:10 -07:00
Matteo Merli
5f65dc8778 Expose DB::DeleteRange and WriteBath::DeleteRange in Java
Summary:
Added JNI wrapper from `DeleteRange` methods
Closes https://github.com/facebook/rocksdb/pull/1951

Differential Revision: D4657746

Pulled By: yiwu-arbug

fbshipit-source-id: 3fc7ab8
2017-03-06 22:24:26 -08:00
Adam Retter
c6d464a9da Fixed various memory leaks and Java 8 JNI Compatibility
Summary:
I have manually audited the entire RocksJava code base.

Sorry for the large pull-request, I have broken it down into many small atomic commits though.

My initial intention was to fix the warnings that appear when running RocksJava on Java 8 with `-Xcheck:jni`, for example when running `make jtest` you would see many errors similar to:

```
WARNING in native method: JNI call made without checking exceptions when required to from CallObjectMethod
WARNING in native method: JNI call made without checking exceptions when required to from CallVoidMethod
WARNING in native method: JNI call made without checking exceptions when required to from CallStaticVoidMethod
...
```

A few of those warnings still remain, however they seem to come directly from the JVM and are not directly related to RocksJava; I am in contact with the OpenJDK hostpot-dev mailing list about these - http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-February/025981.html.

As a result of fixing these, I realised we were not r
Closes https://github.com/facebook/rocksdb/pull/1890

Differential Revision: D4591758

Pulled By: siying

fbshipit-source-id: 7f7fdf4
2017-02-27 16:39:12 -08:00
Siying Dong
8efb5ffa2a [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp…
Summary:
…action

 The two options, min_partial_merge_operands and verify_checksums_in_compaction, are not seldom used. Remove them to reduce the total number of options. Also remove them from Java and C interface.
Closes https://github.com/facebook/rocksdb/pull/1902

Differential Revision: D4601219

Pulled By: siying

fbshipit-source-id: aad4cb2
2017-02-23 15:09:12 -08:00
Sagar Vemuri
eb912a927e Remove disableDataSync option
Summary:
Remove disableDataSync, and another similarly named disable_data_sync options.
This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods.
Closes https://github.com/facebook/rocksdb/pull/1859

Differential Revision: D4541292

Pulled By: sagar0

fbshipit-source-id: 5b3a6ca
2017-02-13 11:09:13 -08:00
Adam Retter
85ac1a320a Fix rocksdb::Status::getState
Summary:
This fixes the Java API for Status#getState use in Native code and also simplifies the implementation of rocksdb::Status::getState.
Closes https://github.com/facebook/rocksdb/issues/1688
Closes https://github.com/facebook/rocksdb/pull/1714

Differential Revision: D4364181

Pulled By: yiwu-arbug

fbshipit-source-id: 8e073b4
2017-01-03 18:39:14 -08:00
Aaron Gao
972f96b3fb direct io write support
Summary:
rocksdb direct io support

```
[gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 5.0
Date:       Wed Nov 23 13:17:43 2016
CPU:        40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
CPUCache:   25600 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Write rate: 0 bytes/second
Compression: Snappy
Memtablerep: skip_list
Perf Level: 1
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
DB path: [/tmp/rocksdbtest-112628/dbbench]
fillseq      :       4.393 micros/op 227639 ops/sec;   25.2 MB/s

[gzh@dev11575.prn2 ~/roc
Closes https://github.com/facebook/rocksdb/pull/1564

Differential Revision: D4241093

Pulled By: lightmark

fbshipit-source-id: 98c29e3
2016-12-22 13:09:19 -08:00
Daniel Black
816c1e30ca gcc-7 requires include <functional> for std::function
Summary:
Fixes compile error:

In file included from ./util/statistics.h:17:0,
                 from ./util/stop_watch.h:8,
                 from ./util/perf_step_timer.h:9,
                 from ./util/iostats_context_imp.h:8,
                 from ./util/posix_logger.h:27,
                 from ./port/util_logger.h:18,
                 from ./db/auto_roll_logger.h:15,
                 from db/auto_roll_logger.cc:6:
./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type
   typedef std::function<void(void*, void*)> FoldFunc;
Closes https://github.com/facebook/rocksdb/pull/1656

Differential Revision: D4318702

Pulled By: yiwu-arbug

fbshipit-source-id: 8c5d17a
2016-12-16 11:24:18 -08:00
Adam Retter
3cdfaeca3f Fixes for MSVC compilation
Summary: Closes https://github.com/facebook/rocksdb/pull/1669

Differential Revision: D4327421

Pulled By: yiwu-arbug

fbshipit-source-id: 661ee0b
2016-12-14 10:24:13 -08:00
Adam Retter
24bceb0963 Java API - Implement GetFromBatch and GetFromBatchAndDB in WBWI
Summary:
Needed for working with `get` after `merge` on a WBWI.
Closes https://github.com/facebook/rocksdb/pull/1093

Differential Revision: D4137978

Pulled By: yhchiang

fbshipit-source-id: e18d50d
2016-11-06 02:44:07 -08:00
Benoit Girard
2b16d664cb Change max_bytes_for_level_multiplier to double
Summary: Closes https://github.com/facebook/rocksdb/pull/1427

Differential Revision: D4094732

Pulled By: yiwu-arbug

fbshipit-source-id: b9b79e9
2016-11-01 21:09:23 -07:00
Adam Retter
ce22ea99a0 Fix casts for MSVC
Summary:
I am not sure if this is the best way to fix this?
Closes https://github.com/facebook/rocksdb/pull/1452

Differential Revision: D4109338

Pulled By: yiwu-arbug

fbshipit-source-id: ca40809
2016-11-01 10:09:16 -07:00
Islam AbdelRahman
869ae5d786 Support IngestExternalFile (remove AddFile restrictions)
Summary:
Changes in the diff

API changes:
- Introduce IngestExternalFile to replace AddFile (I think this make the API more clear)
- Introduce IngestExternalFileOptions (This struct will encapsulate the options for ingesting the external file)
- Deprecate AddFile() API

Logic changes:
- If our file overlap with the memtable we will flush the memtable
- We will find the first level in the LSM tree that our file key range overlap with the keys in it
- We will find the lowest level in the LSM tree above the the level we found in step 2 that our file can fit in and ingest our file in it
- We will assign a global sequence number to our new file
- Remove AddFile restrictions by using global sequence numbers

Other changes:
- Refactor all AddFile logic to be encapsulated in ExternalSstFileIngestionJob

Test Plan:
unit tests (still need to add more)
addfile_stress (https://reviews.facebook.net/D65037)

Reviewers: yiwu, andrewkr, lightmark, yhchiang, sdong

Reviewed By: sdong

Subscribers: jkedgar, hcz, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D65061
2016-10-20 17:05:32 -07:00
Adam Retter
5cd28833a2 [RocksJava] Adjusted RateLimiter to 3.10.0 (#1368)
Summary:
- Deprecated RateLimiterConfig and GenericRateLimiterConfig
- Introduced RateLimiter

It is now possible to use all C++ related methods also in RocksJava.
A noteable method is setBytesPerSecond which can change the allowed
number of bytes per second at runtime.

Test Plan:
make rocksdbjava
make jtest

Reviewers: adamretter, yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35715
2016-10-07 12:32:21 -07:00
Tomas Kolda
13f7a01f61 Fixing JNI release build for gcc (#975) 2016-09-29 17:11:32 -04:00
Injun Song
7260662b39 Add Java API for SstFileWriter
Add Java API for SstFileWriter. Closes https://github.com/facebook/rocksdb/issues/1248
2016-09-29 17:04:41 -04:00
Yi Wu
a958c26435 Rename jvalue to jval in rocksjni
Summary: jvalue shadows a global name in <jni.h>. Rename it to jval to fix java build.

Test Plan:
    JAVA_HOME=/usr/local/jdk-7u10-64 make rocksdbjava -j64

Reviewers: adamretter, yhchiang, IslamAbdelRahman

Reviewed By: yhchiang, IslamAbdelRahman

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63981
2016-09-14 13:12:55 -07:00
xh931076284
3639f32889 Fix bug in UnScSigned-off-by: xh931076284 <931076284@qq.com> (#1336)
Fix HdfsEnv::UnSchedule() API error
2016-09-14 10:17:34 -07:00
Yi Wu
81747f1be6 Refactor MutableCFOptions
Summary:
* Change constructor of MutableCFOptions to depends only on ColumnFamilyOptions.
* Move `max_subcompactions`, `compaction_options_fifo` and `compaction_pri` to ImmutableCFOptions to make it clear that they are immutable.

Test Plan: existing unit tests.

Reviewers: yhchiang, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63945
2016-09-13 21:11:59 -07:00
Adam Retter
22d88e24db Allow an offset as well as a length to be specified for byte[] operations in RocksJava JNI (#1264)
Test Plan: Execute the Java test suite

Reviewers: yhchiang

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61971
2016-09-12 11:51:08 -07:00
Yi Wu
a88677d2cf Remove ImmutableCFOptions from public API
Summary: There's no reference to ImmutableCFOptions elsewhere in /include/rocksdb. ImmutableCFOptions was introduced in this commit (5665e5e285) but later its reference in /include/rocksdb/table.h is removed.

Test Plan:
  make all check

Reviewers: IslamAbdelRahman, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: yhchiang, andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D63177
2016-09-02 14:16:31 -07:00
sdong
32149059f9 Merge options source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes
Summary: To reduce number of options, merge source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes.

Test Plan: Add two new unit tests. Run all existing tests, including jtest.

Reviewers: yhchiang, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59829
2016-09-01 14:33:24 -07:00
Adam Retter
f85f99bf69 Fix the Windows build of RocksDB Java. Similar to https://github.com/facebook/rocksdb/issues/1220 (#1284) 2016-08-25 10:16:26 -07:00
Adam Retter
817eeb29b4 Add singleDelete to RocksJava (#1275)
* Rename RocksDB#remove -> RocksDB#delete to match C++ API; Added deprecated versions of RocksDB#remove for backwards compatibility.

* Add missing experimental feature RocksDB#singleDelete
2016-08-22 11:02:31 -07:00
Adam Retter
ffdf6eee19 Add Status to RocksDBException so that meaningful function result Status from the C++ API isn't lost (#1273) 2016-08-22 11:01:42 -07:00
Islam AbdelRahman
dd7a748cff Fix java build
Summary: Fix the java build

Test Plan: make rocksdbjava -j64

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D62097
2016-08-15 14:03:33 -07:00
Adam Retter
d367555027 Added further Java API options for controlling concurrent writes 2016-08-11 16:33:20 -07:00
Adam Retter
ebdfe34cc4 Exposed further Java API options for controlling compaction 2016-08-11 16:33:20 -07:00
Adam Retter
f4d986364e Added SetOptions support to RocksJava (#1243)
* [refactor] Split Java ColumnFamilyOptions into mutable and immutable and implement any missing immutable options

* [feature] Implement RocksDB#setOptions
2016-08-06 12:03:47 -07:00
Islam AbdelRahman
7c01d65344 [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv
Summary: Remove duplicate cases in LoggerJniCallback::Logv and fix complication error

Test Plan: make jclean rocksdbjava jtest -j4

Reviewers: adamretter, andrewkr, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61335
2016-07-29 12:47:15 -07:00
Adam Retter
8796934af7 Added missing Java ReadOptions settings (#1109)
* Minor reorganisation

* Java API - Added missing ReadOptions settings
2016-07-29 12:12:03 -07:00
Adam Retter
811ee21112 Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106) 2016-07-28 16:54:06 -07:00
Islam AbdelRahman
8745f013ff [Fix java build] Stop using non standard std::make_unique
Summary: std::make_unique is not standard and not always available, remove it

Test Plan: Run "make clean jclean rocksdbjava jtest -j8" on my mac

Reviewers: yhchiang, yiwu, sdong, andrewkr

Reviewed By: andrewkr

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61143
2016-07-26 13:18:31 -07:00
Alexander Jipa
12767b3130 fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223)
* fixes 1220: rocksjni build fails on Windows due to variable-size array declaration

using new keyword to create variable-size arrays in order to satisfy most of the compilers

* fixes 1220: rocksjni build fails on Windows due to variable-size array declaration

using unique_ptr keyword to create variable-size arrays in order to satisfy most of the compilers
2016-07-22 10:17:10 -07:00
sdong
32df9733d1 Add options.write_buffer_manager: control total memtable size across DB instances
Summary: Add option write_buffer_manager to help users control total memory spent on memtables across multiple DB instances.

Test Plan: Add a new unit test.

Reviewers: yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: adela, benj, sumeet, muthu, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59925
2016-07-05 18:11:25 -07:00
sdong
7b79238b65 Deprectate filter_deletes
Summary: filter_deltes is not a frequently used feature. Remove it.

Test Plan: Run all test suites.

Reviewers: igor, yhchiang, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59427
2016-06-17 10:30:47 -07:00
Adam Retter
e3b1e3dfa6 Expose save points in Java WriteBatch and WBWI (#1092)
* Java API - Expose SetSavePoint and RollbackToRestorePoint for WriteBatch and WriteBatchWithIndex

* Minor cleanup
2016-06-14 09:51:14 -07:00
sdong
20699df843 memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
Summary:
memtable_prefix_bloom_probes is not a critical option. Remove it to reduce number of options.
It's easier for users to make mistakes with memtable_prefix_bloom_bits, turn it to memtable_prefix_bloom_bits_ratio

Test Plan: Run all existing tests

Reviewers: yhchiang, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: gunnarku, yoshinorim, MarkCallaghan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D59199
2016-06-10 12:12:10 -07:00
Adam Retter
972c895c35 Previously WARN level logging became FATAL level logging in the Java API (#1089) 2016-06-07 22:35:57 -07:00
Adam Retter
774a6aa296 Java API - Rename geHistogramData -> getHistogramData (#1107)
Closes https://github.com/facebook/rocksdb/issues/835
2016-06-06 16:04:16 -07:00
Adam Retter
a791a2cf2d Java API - Bugfix for native linking of Compaction Filter (#1099) 2016-06-03 10:24:03 -07:00
Adam Retter
72c73cdc8b Java API - Add missing HEADER_LEVEL logging (#1104) 2016-05-06 15:06:12 -07:00
Adam Retter
4d02bfa3a6 Add support for PauseBackgroundWork and ContinueBackgroundWork to the Java API (#1087)
Closes https://github.com/facebook/rocksdb/issues/1071
2016-05-06 15:04:13 -07:00
Andrew Kryczka
9385fd72c0 Delete deprecated backup classes in Java
Summary: Corresponding change to D56331.

Test Plan:
Now build succeeds:

  $ make jclean && make rocksdbjava

Reviewers: yhchiang, IslamAbdelRahman, adamretter

Reviewed By: adamretter

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D56913
2016-04-19 10:32:31 -07:00
Marton Trencseni
9b51987521 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
Summary:
When a block based table file is opened, if prefetch_index_and_filter is true, it will prefetch the index and filter blocks, putting them into the block cache.
What this feature adds: when a L0 block based table file is opened, if pin_l0_filter_and_index_blocks_in_cache is true in the options (and prefetch_index_and_filter is true), then the filter and index blocks aren't released back to the block cache at the end of BlockBasedTableReader::Open(). Instead the table reader takes ownership of them, hence pinning them, ie. the LRU cache will never push them out. Meanwhile in the table reader, further accesses will not hit the block cache, thus avoiding lock contention.

Test Plan:
'export TEST_TMPDIR=/dev/shm/ && DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32' is OK.
I didn't run the Java tests, I don't have Java set up on my devserver.

Reviewers: sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D56133
2016-04-01 10:42:39 -07:00