Commit Graph

728 Commits

Author SHA1 Message Date
Siying Dong
000b9ec217 Move some logging related files to logging/ (#5387)
Summary:
Many logging related source files are under util/. It will be more structured if they are together.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5387

Differential Revision: D15579036

Pulled By: siying

fbshipit-source-id: 3850134ed50b8c0bb40a0c8ae1f184fa4081303f
2019-05-31 17:23:59 -07:00
Siying Dong
8843129ece Move some memory related files from util/ to memory/ (#5382)
Summary:
Move arena, allocator, and memory tools under util to a separate memory/ directory.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5382

Differential Revision: D15564655

Pulled By: siying

fbshipit-source-id: 9cd6b5d0d3d52b39606e19221fa154596e5852a5
2019-05-30 17:44:09 -07:00
Siying Dong
e9e0101ca4 Move test related files under util/ to test_util/ (#5377)
Summary:
There are too many types of files under util/. Some test related files don't belong to there or just are just loosely related. Mo
ve them to a new directory test_util/, so that util/ is cleaner.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5377

Differential Revision: D15551366

Pulled By: siying

fbshipit-source-id: 0f5c8653832354ef8caa31749c0143815d719e2c
2019-05-30 11:25:51 -07:00
Vijay Nadimpalli
931c9df886 Use separate status code for column family drop and db shutdown in progress (#5275)
Summary:
Currently RocksDB uses Status::ShutdownInProgress to inform about column family drop. I would like to have a separate Status code for this event.
https://github.com/facebook/rocksdb/blob/master/include/rocksdb/status.h#L55
Comment on this:
abc4202e47/db/version_set.cc (L2742):L2743
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5275

Differential Revision: D15204583

Pulled By: vjnadimpalli

fbshipit-source-id: 95e99e34b27bc165b554ecb8a48a7f8e60f21e2a
2019-05-20 10:47:32 -07:00
Fosco Marotto
6c2bf9e916 Add copyright headers per FB open-source checkup tool. (#5199)
Summary:
internal task: T35568575
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5199

Differential Revision: D14962794

Pulled By: gfosco

fbshipit-source-id: 93838ede6d0235eaecff90d200faed9a8515bbbe
2019-04-18 10:55:01 -07:00
Zhongyi Xie
baa5302447 Avoid double-compacting data in bottom level in manual compactions (#5138)
Summary:
Depending on the config, manual compaction (leveled compaction style) does following compactions:
L0->L1
L1->L2
...
Ln-1 -> Ln
Ln -> Ln
The final Ln -> Ln compaction is partly unnecessary as it recompacts all the files that were just generated by the Ln-1 -> Ln. We should avoid recompacting such files. This rule should be applied to Lmax only.
Resolves issue https://github.com/facebook/rocksdb/issues/4995
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5138

Differential Revision: D14940106

Pulled By: miasantreble

fbshipit-source-id: 8d3cf5507a17e76f3333cfd4bac5256d005636e5
2019-04-16 23:32:20 -07:00
ableegoldman
1966a7c055 Expose JavaAPI for getting the filter policy of a BlockBasedTableConfig (#5186)
Summary:
I would like to be able to read out the current Filter that has been set (or not) for a BlockBasedTableConfig. Added one public method to BlockBasedTableConfig:

public Filter filterPolicy() {
    return filterPolicy;
}
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5186

Differential Revision: D14921415

Pulled By: siying

fbshipit-source-id: 2a63c8685480197862b49fc48916c757cd6daf95
2019-04-12 14:01:36 -07:00
Andrew Audibert
e50326f327 Document the interaction between disableWAL and BackupEngine (#5071)
Summary:
BackupEngine relies on write-ahead logs to back up the memtable. Disabling write-ahead logs
can result in backups failing to preserve unflushed keys. This PR updates the documentation to specify this behavior, and suggest always flushing the memtable when write-ahead logs are disabled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5071

Differential Revision: D14524124

Pulled By: miasantreble

fbshipit-source-id: 635f855f8a42ad60273b5efd226139b511e3e5d5
2019-03-19 14:58:14 -07:00
SeterKwok
a283800616 Move some RocksObject into try-with-resources in Test (#5037)
Summary:
Fix #5008
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5037

Differential Revision: D14302474

Pulled By: riversand963

fbshipit-source-id: dcd9dda5d4d6d459315692f355499a39e546d518
2019-03-04 14:56:15 -08:00
Sagar Vemuri
9a31b8dd2c Disable getApproximateSizes test (#5035)
Summary:
Disabling `org.rocksdb.RocksDBTest.getApproximateSizes` test as it is frequently crashing on travis (#5020). It will be re-enabled once the root-cause is found and fixed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5035

Differential Revision: D14294736

Pulled By: sagar0

fbshipit-source-id: e28bff0d143a58ad6c82991fec3d4cf8c0209995
2019-03-01 23:05:32 -08:00
Siying Dong
0920bf4e68 Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034)
Summary:
This reverts commit ee1818081f.

We are not ready to deprecate this feature. revert it for now.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5034

Differential Revision: D14287246

Pulled By: siying

fbshipit-source-id: e4beafdeaee1c94364fdaa6ba198218d158339f7
2019-03-01 15:45:45 -08:00
Siying Dong
aef763b6d6 Make statistics's stats_level change thread-safe (#5030)
Summary:
Right now, users can change statistics.stats_level while DB is running, but TSAN may report
data race. We make stats_level_ to be atomic, and access them using accessors.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5030

Differential Revision: D14267519

Pulled By: siying

fbshipit-source-id: 37d7ebeff7a43a406230143422a16af899163f73
2019-03-01 10:42:09 -08:00
Sagar Vemuri
06ea73d60b Fix DefaultEnvTest.incBackgroundThreadsIfNeeded test (#5021)
Summary:
`DefaultEnvTest.incBackgroundThreadsIfNeeded` jtest should assert that the number of threads is greater than or equal to the minimum number of threads.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5021

Differential Revision: D14268311

Pulled By: sagar0

fbshipit-source-id: 01fb32b5b3ce636451d162fa1a2bbc5bd1974682
2019-02-28 14:48:57 -08:00
Adam Retter
bb474e9a02 Add missing functionality to RocksJava (#4833)
Summary:
This is my latest round of changes to add missing items to RocksJava. More to come in future PRs.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4833

Differential Revision: D14152266

Pulled By: sagar0

fbshipit-source-id: d6cff67e26da06c131491b5cf6911a8cd0db0775
2019-02-22 14:46:46 -08:00
Zhongyi Xie
c4f5d0aa15 add GetStatsHistory to retrieve stats snapshots (#4748)
Summary:
This PR adds public `GetStatsHistory` API to retrieve stats history in the form of an std map. The key of the map is the timestamp in microseconds when the stats snapshot is taken, the value is another std map from stats name to stats value (stored in std string). Two DBOptions are introduced: `stats_persist_period_sec` (default 10 minutes) controls the intervals between two snapshots are taken; `max_stats_history_count` (default 10) controls the max number of history snapshots to keep in memory. RocksDB will stop collecting stats snapshots if `stats_persist_period_sec` is set to 0.

(This PR is the in-memory part of https://github.com/facebook/rocksdb/pull/4535)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4748

Differential Revision: D13961471

Pulled By: miasantreble

fbshipit-source-id: ac836d401ecb84ea92216bf9966f969dedf4ad04
2019-02-20 15:52:54 -08:00
Siying Dong
c2affccc18 Header logger should call LogHeader() (#4980)
Summary:
The info log header feature never worked well, because log level Header was not
translated to Logger::LogHeader() call. Fix it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4980

Differential Revision: D14087283

Pulled By: siying

fbshipit-source-id: 7e7d03ce35fa8d13d4ee549f46f7326f7bc0006d
2019-02-15 16:59:36 -08:00
Aubin Sanyal
3231a2e581 Deprecate ttl option from CompactionOptionsFIFO (#4965)
Summary:
We introduced ttl option in CompactionOptionsFIFO when ttl-based file
deletion (compaction) was supported only as part of FIFO Compaction. But
with the extension of ttl semantics even to Level compaction,
CompactionOptionsFIFO.ttl can now be deprecated. Instead we will start
using ColumnFamilyOptions.ttl for FIFO compaction as well.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4965

Differential Revision: D14072960

Pulled By: sagar0

fbshipit-source-id: c98cc2ae695a28136295787cd88d36a220fc219e
2019-02-15 09:51:41 -08:00
Adam Retter
33b33235ff Add Java multiGet API for returning List<byte[]> (#1570)
Summary:
Closes https://github.com/facebook/rocksdb/pull/1570
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4797

Differential Revision: D13961770

Pulled By: sagar0

fbshipit-source-id: e34fd6250d0cd3ebb0bd688e8801fe8947fd464d
2019-02-12 17:04:48 -08:00
Siying Dong
ee1818081f Remove PlainTable's feature store_index_in_file (#4914)
Summary:
Store_index_in_file is a less useful feature. To simplify the code to maintain, we are dropping the feature.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4914

Differential Revision: D13791883

Pulled By: siying

fbshipit-source-id: d187c5d662584866103e4b77d09dfb925509ae2e
2019-01-28 12:50:22 -08:00
Max
b1288cdc24 Fix typos in comments (#4819)
Summary:
Fix some typos in comments.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4819

Differential Revision: D13548543

Pulled By: siying

fbshipit-source-id: ca2e128fa47bef32892fc3627a7541fd9e2d5c3f
2018-12-26 09:43:56 -08:00
Adam Singer
a914a1c6dc Add getMin, getMax, getCount, getSum to HistogramData class object. (#4742)
Summary:
Expose common stats min,max,count,sum via statistics JNI. These stats are not fully exposed on the Java side as is, but are available on the native side.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4742

Differential Revision: D13403766

Pulled By: ajkr

fbshipit-source-id: 5b70f7bd3fb7490aab73dcbd09f13490fce5c773
2018-12-14 14:28:44 -08:00
Adam Singer
d6dfe516ff Synchronize ticker and histogram metrics for Java API (#4733)
Summary:
Updating the `HistogramType.java` and `TickerType.java` to expose and correct metrics for statistics callbacks.

Moved `NO_ITERATOR_CREATED` to the proper stat name and deprecated `NO_ITERATORS`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4733

Differential Revision: D13466936

Pulled By: sagar0

fbshipit-source-id: a58d1edcc07c7b68c3525b1aa05828212c89c6c7
2018-12-14 11:37:05 -08:00
DorianZheng
2670fe8c73 Get CompactionJobInfo from CompactFiles
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4716

Differential Revision: D13207677

Pulled By: ajkr

fbshipit-source-id: d0ccf5a66df6cbb07288b0c5ebad81fd9df3926b
2018-12-13 14:21:24 -08:00
Tathagata Das
49666d76cf Fix swallowing of exception in Java RocksDB when loading native library (#4728)
Summary:
This PR fixes #4721. When an exception is caught and thrown as a different exception, then the original exception should be  inserted as a cause of the new exception. This bug in RocksDB was swallowing the underlying exception from `NativeLibraryLoader` and throwing the following exception
```
...
Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared libraryjava.nio.channels.ClosedByInterruptException
  at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67)
  at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
  ... 73 more
```

The fix is simple and self-explanatory.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4728

Differential Revision: D13418371

Pulled By: sagar0

fbshipit-source-id: d76c25af2a83a0f8ba62cc8d7b721bfddc85fdf1
2018-12-11 12:18:44 -08:00
Adam Retter
d3daa0db8b RocksJava must compile on JDK7 (#4768)
Summary:
Fixes some RocksJava regressions recently introduced, whereby RocksJava would not build on JDK 7.
These should have been visible on Travis-CI!
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4768

Differential Revision: D13418173

Pulled By: sagar0

fbshipit-source-id: 57bf223188887f84d9e072031af2e0d2c8a69c30
2018-12-11 11:40:23 -08:00
Adam Retter
f8943ec003 Fix issues with RocksJava dropColumnFamily (#4770)
Summary:
Closes https://github.com/facebook/rocksdb/issues/4409
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4770

Differential Revision: D13416802

Pulled By: ajkr

fbshipit-source-id: 8a351e9b80dc9eeb6073467fbc67cd2f544917b0
2018-12-11 09:17:57 -08:00
Ben Clay
8261e0026b Promote CompactionFilter* accessors to ColumnFamilyOptionsInterface (#3461)
Summary:
When adding CompactionFilter and CompactionFilterFactory settings to the Java layer, ColumnFamilyOptions was modified directly instead of ColumnFamilyOptionsInterface. This meant that the old-stye Options monolith was left behind.

This patch fixes that, by:
- promoting the CompactionFilter + CompactionFilterFactory setters from ColumnFamilyOptions -> ColumnFamilyOptionsInterface
- adding getters in ColumnFamilyOptionsInterface
- implementing setters in Options
- implementing getters in both ColumnFamilyOptions and Options
- adding testcases
- reusing a test CompactionFilterFactory by moving it to a common location
Pull Request resolved: https://github.com/facebook/rocksdb/pull/3461

Differential Revision: D13278788

Pulled By: sagar0

fbshipit-source-id: 72602c6eb97dc80734e718abb5e2e9958d3c753b
2018-12-10 15:54:52 -08:00
Adam Singer
10e7de7705 Reduce javadoc warnings (#4764)
Summary:
Compile logs have a bit of noise due to missing javadoc annotations. Updating docs to reduce.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4764

Differential Revision: D13400193

Pulled By: sagar0

fbshipit-source-id: 65c7efb70747cc3bb35a336a6881ea6536ae5ff4
2018-12-10 11:08:38 -08:00
Adam Retter
4048762cbe Updated the CentOS 6 Docker build for RocksJava to a newer GCC toolchain (#4756)
Summary:
Uses a newer build toolchain but the same old GLIBC when building releases of RocksJava for Linux x64 in the Docker Container.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4756

Differential Revision: D13383575

Pulled By: sagar0

fbshipit-source-id: 27c58814876e434d5fa61395e6664cfc5f6830b1
2018-12-07 14:37:26 -08:00
Maysam Yabandeh
b878f93c70 Extend Transaction::GetForUpdate with do_validate (#4680)
Summary:
Transaction::GetForUpdate is extended with a do_validate parameter with default value of true. If false it skips validating the snapshot (if there is any) before doing the read. After the read it also returns the latest value (expects the ReadOptions::snapshot to be nullptr). This allows RocksDB applications to use GetForUpdate similarly to how InnoDB does. Similarly ::Merge, ::Put, ::Delete, and ::SingleDelete are extended with assume_exclusive_tracked with default value of false. It true it indicates that call is assumed to be after a ::GetForUpdate(do_validate=false).
The Java APIs are accordingly updated.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4680

Differential Revision: D13068508

Pulled By: maysamyabandeh

fbshipit-source-id: f0b59db28f7f6a078b60844d902057140765e67d
2018-12-06 17:49:00 -08:00
Zhongyi Xie
2f1ca4e838 Revert "BaseDeltaIterator: always check valid() before accessing key(… (#4744)
Summary:
…) (#4702)"

This reverts commit 3a18bb3e15.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4744

Differential Revision: D13311869

Pulled By: miasantreble

fbshipit-source-id: 6300b12cc34828d8b9274e907a3aef1506d5d553
2018-12-03 23:38:27 -08:00
Zhongyi Xie
3a18bb3e15 BaseDeltaIterator: always check valid() before accessing key() (#4702)
Summary:
Current implementation of `current_over_upper_bound_` fails to take into consideration that keys might be invalid in either base iterator or delta iterator. Calling key() in such scenario will lead to assertion failure and runtime errors.
This PR addresses the bug by adding check for valid keys before calling `IsOverUpperBound()`, also added test coverage for iterate_upper_bound usage in BaseDeltaIterator
Also recommit https://github.com/facebook/rocksdb/pull/4656 (It was reverted earlier due to bugs)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4702

Differential Revision: D13146643

Pulled By: miasantreble

fbshipit-source-id: 6d136929da12d0f2e2a5cea474a8038ec5cdf1d0
2018-11-30 15:35:13 -08:00
Adam Singer
3fa80f0e85 Remove enable_internal_stats (#4714)
Summary:
Simple patch to address comments in [statistics.h#L65](https://github.com/facebook/rocksdb/blob/master/monitoring/statistics.h#L65|statistics.h#L65)  `TODO(ajkr): clean this up since there are no internal stats anymore`
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4714

Differential Revision: D13208093

Pulled By: ajkr

fbshipit-source-id: 4468badb850592411147539f859082644f5296f6
2018-11-27 12:58:58 -08:00
Sagar Vemuri
a2dec2ed08 Fix Java to C++ ticker conversions (#4719)
Summary:
Added back `NO_ITERATORS` and moved `NO_ITERATOR_CREATED` to the end of `toCppTickers`.

This is a leftover fix which is needed in addition to a138e351bc to correctly convert java tickers to c++ tickers. a138e351bc only updated `toJavaTickerType` but both `toJavaTickerType` and `toCppTickers` need to be changed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4719

Differential Revision: D13208847

Pulled By: sagar0

fbshipit-source-id: 53a42f3d6ffe04034acfde972d73040b92b4c1af
2018-11-27 10:17:07 -08:00
Sagar Vemuri
0d65315cb1 RocksJava: Add the missing FIFO compaction options (#4609)
Summary:
Make CompactionOptionsFIFO's ttl and allow_compaction options to be available in RocksJava.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4609

Differential Revision: D12849503

Pulled By: sagar0

fbshipit-source-id: 47baa97918d252370f234c36c1af15ff2dad7658
2018-11-26 17:02:08 -08:00
Zhongyi Xie
a21cb22ee3 Revert "apply ReadOptions.iterate_upper_bound to transaction iterator… (#4705)
Summary:
… (#4656)"

This reverts commit b76398a82b.

Will add test coverage for iterate_upper_bound before re-commit b76398
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4705

Differential Revision: D13148592

Pulled By: miasantreble

fbshipit-source-id: 4d1ce0bfd9f7a5359a7688bd780eb06a66f45b1f
2018-11-24 10:46:28 -08:00
Andrew Kryczka
a138e351bc Fix compatibility of public ticker stats (#4701)
Summary:
- Added back the `NO_ITERATORS` that was removed in 5945e16dfc.
- Marked it as deprecated since it is no longer populated, but kept for API compatibility.
- Made sure the new tickers, `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETED`, are appended at the end of the enum, in case people are relying on the int values.

The change where `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETED` were introduced is unreleased so I believe it is ok to change their ordering.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4701

Differential Revision: D13142887

Pulled By: ajkr

fbshipit-source-id: 29a336ce5b46632ce50ad42ccc4a29013f71d6d6
2018-11-20 13:13:16 -08:00
thisisbaozi
5cf5f4724d Expose underlying Read/Write APIs for avoiding unnecessary memory copy (#2303)
Summary:
adamretter

As you already mentioned at #1247 .
Pull Request resolved: https://github.com/facebook/rocksdb/pull/2303

Differential Revision: D10209001

Pulled By: sagar0

fbshipit-source-id: bcbce004112c2edeaff116968d79c6f90aab4b6c
2018-11-13 17:33:09 -08:00
Zhongyi Xie
b76398a82b apply ReadOptions.iterate_upper_bound to transaction iterator (#4656)
Summary:
Currently transaction iterator does not apply `ReadOptions.iterate_upper_bound` when iterating. This PR attempts to fix the problem by having `BaseDeltaIterator` enforcing the upper bound check when iterator state is changed.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4656

Differential Revision: D13039257

Pulled By: miasantreble

fbshipit-source-id: 909eb9f6b4597a4d80418fb139f32ec82c6ec1d1
2018-11-13 15:44:15 -08:00
Soli Como
5945e16dfc Divide NO_ITERATORS into two counters NO_ITERATOR_CREATED and NO_ITERATOR_DELETE (#4498)
Summary:
Currently, `Statistics` can record tick by `recordTick()` whose second parameter is an `uint64_t`.
That means tick can only increase.
If we want to reduce tick, we have to work around like `RecordTick(statistics_, NO_ITERATORS, uint64_t(-1));`.
That's kind of a hack.

So, this PR divide `NO_ITERATORS` into two counters `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETE`, making the counters increase only.

Fixes #3013 .
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4498

Differential Revision: D10395010

Pulled By: sagar0

fbshipit-source-id: cfb523b22a37411c794b4e9da090f1ae30293db2
2018-11-13 11:46:32 -08:00
Sagar Vemuri
dc3528077a Update all unique/shared_ptr instances to be qualified with namespace std (#4638)
Summary:
Ran the following commands to recursively change all the files under RocksDB:
```
find . -type f -name "*.cc" -exec sed -i 's/ unique_ptr/ std::unique_ptr/g' {} +
find . -type f -name "*.cc" -exec sed -i 's/<unique_ptr/<std::unique_ptr/g' {} +
find . -type f -name "*.cc" -exec sed -i 's/ shared_ptr/ std::shared_ptr/g' {} +
find . -type f -name "*.cc" -exec sed -i 's/<shared_ptr/<std::shared_ptr/g' {} +
```
Running `make format` updated some formatting on the files touched.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4638

Differential Revision: D12934992

Pulled By: sagar0

fbshipit-source-id: 45a15d23c230cdd64c08f9c0243e5183934338a8
2018-11-09 11:19:58 -08:00
jiachun.fjc
55c0349274 Thread.sleep() in StatisticsCollector (#4588)
Summary:
In  'StatisticsCollector', the call of Thread.sleep() might be better outside the loop?
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4588

Differential Revision: D12903406

Pulled By: sagar0

fbshipit-source-id: 1647ed779e9972bc2cea03f4c38e37ab3ad7c361
2018-11-02 10:55:10 -07:00
Ben Clay
72afdf20fc RocksJava: Add more flags to BlockBasedTableConfig (#4589)
Summary:
Punch through more flags for BlockBasedTableConfig, mostly around caching index + filter blocks and partitioned filters.

sagar0 adamretter
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4589

Differential Revision: D12840626

Pulled By: sagar0

fbshipit-source-id: 3c289d367ceb2a012023aa791b990a437dd1393a
2018-10-30 13:11:56 -07:00
Jigar Bhati
6ecd26af27 WriteBufferManager JNI fixes (#4579)
Summary:
1. `WriteBufferManager` should have a reference alive in Java side through `Options`/`DBOptions` otherwise, if it's GC'ed at java side, native side can seg fault.
2. native method `setWriteBufferManager()` in `DBOptions.java` doesn't have it's jni method invocation in rocksdbjni which is added in this PR
3. `DBOptionsTest.java` is referencing object of `Options`. Instead it should be testing against `DBOptions`. Seems like a copy paste error.
4. Add a getter for WriteBufferManager.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4579

Differential Revision: D10561150

Pulled By: sagar0

fbshipit-source-id: 139a15c7f051a9f77b4200215b88267b48fbc487
2018-10-24 12:40:52 -07:00
Jigar Bhati
a4d9aa6b18 Plumb WriteBufferManager through JNI (#4492)
Summary:
Allow rocks java to explicitly create WriteBufferManager by plumbing it to the native code through JNI.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4492

Differential Revision: D10428506

Pulled By: sagar0

fbshipit-source-id: cd9dd8c2ef745a0303416b44e2080547bdcca1fd
2018-10-17 11:49:57 -07:00
John Calcote
9c20797136 Add UInt64AddOperator to rocksjava (#4448)
Summary:
Closes https://github.com/facebook/rocksdb/issues/4447
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4448

Differential Revision: D10351852

Pulled By: ajkr

fbshipit-source-id: 18287b5190ae0b8153ce425da9a0bdfe1af88c34
2018-10-12 17:35:47 -07:00
moozzyk
f45c0d20de JNI support for ReadOptions::iterate_lower_bound (#4444)
Summary:
Fixes: #4401
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4444

Differential Revision: D10282120

Pulled By: sagar0

fbshipit-source-id: d9ddcc1b132208ae7f806fa2106add6fec1baa11
2018-10-09 17:13:56 -07:00
Ben Clay
c9048021ad RocksJava: memory_util support (#4446)
Summary:
JNI passthrough for utilities/memory/memory_util.cc

sagar0 adamretter
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4446

Differential Revision: D10174578

Pulled By: sagar0

fbshipit-source-id: d1d196d771dff22afb7ef7500f308233675696f8
2018-10-08 11:05:27 -07:00
Chen, You
02dc074916 add GetAggregatedLongProperty for Java API (#4379)
Summary:
Add Java API `getAggregatedLongProperty(final String property)`
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4379

Differential Revision: D9921463

Pulled By: sagar0

fbshipit-source-id: a02512e1b2aff4765a10b77de9a7bf7b1909d954
2018-09-19 17:46:59 -07:00
Yanqin Jin
90f5048207 Remove warnings caused by unused variables in jni (#4345)
Summary:
Test plan
```
$make clean jclean
$make -j32 rocksdbjavastatic
$make -j32 rocksdbjava
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4345

Differential Revision: D9661256

Pulled By: riversand963

fbshipit-source-id: aed316c53b29d02fbdd3fa1063a3e832b8a66469
2018-09-05 13:42:34 -07:00
Christian Esken
c7cf981a85 Add CompactRangeOptions for Java (#4220)
Summary:
Closes https://github.com/facebook/rocksdb/issues/4195

CompactRangeOptions are available the CPP API, but not in the Java API. This PR adds CompactRangeOptions to the Java API and adds an overloaded compactRange() method. See https://github.com/facebook/rocksdb/issues/4195 for the original discussion.

This change supports all fields of CompactRangeOptions, including the required enum converters in the JNI portal.

Significant changes:
- Make CompactRangeOptions available in the compactRange() for Java.
- Deprecate other compactRange() methods that have individual option params, like in the CPP code.
- Migrate rocksdb_compactrange_helper() to  CompactRangeOptions.
- Add Java unit tests for CompactRangeOptions.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4220

Differential Revision: D9380007

Pulled By: sagar0

fbshipit-source-id: 6af6c334f221427f1997b33fb24c3986b092fed6
2018-08-17 10:57:25 -07:00
Maysam Yabandeh
e0906eb785 Clarify max_total_wal_size's scope (#4194)
Summary:
max_total_wal_size takes effect only when there are more than one column families. The patch clarify that in the inline docs

Closes https://github.com/facebook/rocksdb/issues/4180
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4194

Differential Revision: D9028767

Pulled By: maysamyabandeh

fbshipit-source-id: 8d730ca7f15e76e7ee9ff88b2b48030b2d1b7078
2018-07-27 09:29:44 -07:00
Adam Retter
c6d2a7f821 Build improvements: Split docker targets and parallelize java builds
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4165

Differential Revision: D8955531

Pulled By: sagar0

fbshipit-source-id: 97d5a1375e200bde3c6414f94703504a4ed7536a
2018-07-23 13:28:37 -07:00
Siying Dong
a5e851e113 Reformatting some recent changes (#4161)
Summary:
Lint is not happy with some new code recently committed. Format them.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4161

Differential Revision: D8940582

Pulled By: siying

fbshipit-source-id: c9b43b1ef8c88b5e923911058b44eb77234b36b7
2018-07-20 14:43:38 -07:00
Tomas Kolda
80afa84903 Windows JNI build fixes (#4015)
Summary:
Fixing compilation, unsatisfied link exceptions (updated list of files that needs to be linked) and warnings for Windows build.
```C++
//MSVC 2015 does not support dynamic arrays like:
  rocksdb::Slice key_parts[jkey_parts_len];
//I have converted to:
  std::vector<rocksdb::Slice> key_parts;
```
Also reusing `free_key_parts` that does the same as `free_key_value_parts` that was removed.

Java elapsedTime unit test increase of sleep to 2 ms. Otherwise it was failing.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4015

Differential Revision: D8558215

Pulled By: sagar0

fbshipit-source-id: d3c34f846343f9218424da2402a2bd367bbd0aa2
2018-07-18 12:31:48 -07:00
Nathan VanBenschoten
ef7815b803 Support range deletion tombstones in IngestExternalFile SSTs (#3778)
Summary:
Fixes #3391.

This change adds a `DeleteRange` method to `SstFileWriter` and adds
support for ingesting SSTs with range deletion tombstones. This is
important for applications that need to atomically ingest SSTs while
clearing out any existing keys in a given key range.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/3778

Differential Revision: D8821836

Pulled By: anand1976

fbshipit-source-id: ca7786c1947ff129afa703dab011d524c7883844
2018-07-13 22:43:09 -07:00
Siying Dong
35b38a232c Update comments of WriteBatchWithIndex
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4113

Differential Revision: D8814172

Pulled By: siying

fbshipit-source-id: cabc31db2c74803af9b2f99329155a1086eb1b22
2018-07-11 17:42:50 -07:00
Nicolas Pépin-Perreault
cfee7fb51a Allow storing metadata with backups for Java API (#4111)
Summary:
Exposes BackupEngine::CreateNewBackupWithMetadata and BackupInfo metadata to the Java API.

Full disclaimer, I'm not familiar with JNI stuff, so I might have forgotten something (hopefully no memory leaks!). I also tried to find contributing guidelines but didn't see any, but I hope the PR style is consistent with the rest of the code base.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4111

Differential Revision: D8811180

Pulled By: ajkr

fbshipit-source-id: e38b3e396c7574328c2a1a0e55acc8d092b6a569
2018-07-11 15:57:49 -07:00
Siying Dong
926f3a78a6 In delete scheduler, before ftruncate file for slow delete, check whether there is other hard links (#4093)
Summary:
Right now slow deletion with ftruncate doesn't work well with checkpoints because it ruin hard linked files in checkpoints. To fix it, check the file has no other hard link before ftruncate it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4093

Differential Revision: D8730360

Pulled By: siying

fbshipit-source-id: 756eea5bce8a87b9a2ea3a5bfa190b2cab6f75df
2018-07-09 15:28:12 -07:00
Manuel Ung
b9846370e9 WriteUnPrepared: Add support for recovering WriteUnprepared transactions (#4078)
Summary:
This adds support for recovering WriteUnprepared transactions through the following changes:
- The information in `RecoveredTransaction` is extended so that it can reference multiple batches.
- `MarkBeginPrepare` is extended with a bool indicating whether it is an unprepared begin, and this is passed down to `InsertRecoveredTransaction` to indicate whether the current transaction is prepared or not.
- `WriteUnpreparedTxnDB::Initialize` is overridden so that it will rollback unprepared transactions from the recovered transactions. This can be done without updating the prepare heap/commit map, because this is before the DB has finished initializing, and after writing the rollback batch, those data structures should not contain information about the rolled back transaction anyway.

Commit/Rollback of live transactions is still unimplemented and will come later.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4078

Differential Revision: D8703382

Pulled By: lth

fbshipit-source-id: 7e0aada6c23bd39299f1f20d6c060492e0e6b60a
2018-07-06 17:59:13 -07:00
Adam Retter
0d234dfce4 Remove unused arg which causes compilation failure (#4080)
Summary:
It seems that compilation has been made stricter about unused args.
Closes https://github.com/facebook/rocksdb/pull/4080

Differential Revision: D8712049

Pulled By: sagar0

fbshipit-source-id: 984af1982638af3568aac1a167f565f4741badee
2018-07-02 10:27:36 -07:00
Andrey Zagrebin
e099c2dd55 check if data size exceeds java array vm limit when it is copied in jni (#3850)
Summary:
to address issue #3849
Closes https://github.com/facebook/rocksdb/pull/3850

Differential Revision: D8695487

Pulled By: sagar0

fbshipit-source-id: 04baeb2127663934ed1321fe6d9a9ec23c86e16b
2018-06-29 16:13:06 -07:00
Zhichao Cao
1f6efabe23 Add bottommost_compression_opts to for bottommost_compression (#3985)
Summary:
…ression

 For `CompressionType` we have options `compression` and `bottommost_compression`. Thus, to make the compression options consitent with the compression type when bottommost_compression is enabled, we add the bottommost_compression_opts
Closes https://github.com/facebook/rocksdb/pull/3985

Reviewed By: riversand963

Differential Revision: D8385911

Pulled By: zhichao-cao

fbshipit-source-id: 07bc533dd61bcf1cef5927d8d62901c13d38d5fc
2018-06-27 17:42:38 -07:00
Tomas Kolda
7b4b43febb zLinux build error with gcc and IBM Java headers (#4013)
Summary:
`SetByteArrayRegion` does not have const source buffer thus compilation error. I have made that same as in other JNI files (const_cast). It was missing for new transaction functionality added recently.
Closes https://github.com/facebook/rocksdb/pull/4013

Differential Revision: D8493290

Pulled By: sagar0

fbshipit-source-id: 14afedf365b111121bd11e68a8d546a1cae68b26
2018-06-18 13:58:28 -07:00
Tomas Kolda
e5bee404ce zLinux s390x support in JNI (#4009)
Summary:
Adding support for zLinux on s390x architecture in JNI.
Closes https://github.com/facebook/rocksdb/pull/4009

Differential Revision: D8483750

Pulled By: siying

fbshipit-source-id: e681657c27e7a28f1731e08e8570382de5deff44
2018-06-18 09:57:02 -07:00
Tomas Kolda
e750dacffb Crash on Windows, because of shared_ptr reinterpret cast (#3999)
Summary:
For more details see #3998
Closes https://github.com/facebook/rocksdb/pull/3999

Differential Revision: D8458905

Pulled By: sagar0

fbshipit-source-id: d6e09182933253a08eaf81ac7cfe50ed3b6576c5
2018-06-17 20:56:33 -07:00
Adam Retter
45434178ee Fix an issue with unnecessary capture in lambda expressions
Summary:
Closes https://github.com/facebook/rocksdb/issues/3900
Replaces https://github.com/facebook/rocksdb/pull/3901

I needed this to build v5.12.4 on Mac OS X (10.13.3).
Closes https://github.com/facebook/rocksdb/pull/3904

Differential Revision: D8169357

Pulled By: sagar0

fbshipit-source-id: 85faac42168796e7def9250d0c221a9a03b84476
2018-05-25 15:12:44 -07:00
Anand Ananthabhotla
dbdaa4662e Add a stat for MultiGet keys found, update memtable hit/miss stats
Summary:
1. Add a new ticker stat rocksdb.number.multiget.keys.found to track the
number of keys successfully read
2. Update rocksdb.memtable.hit/miss in DBImpl::MultiGet(). It was being done in
DBImpl::GetImpl(), but not MultiGet
Closes https://github.com/facebook/rocksdb/pull/3730

Differential Revision: D7677364

Pulled By: anand1976

fbshipit-source-id: af22bd0ef8ddc5cf2b4244b0a024e539fe48bca5
2018-04-20 15:28:19 -07:00
David Lai
3be9b36453 comment unused parameters to turn on -Wunused-parameter flag
Summary:
This PR comments out the rest of the unused arguments which allow us to turn on the -Wunused-parameter flag. This is the second part of a codemod relating to https://github.com/facebook/rocksdb/pull/3557.
Closes https://github.com/facebook/rocksdb/pull/3662

Differential Revision: D7426121

Pulled By: Dayvedde

fbshipit-source-id: 223994923b42bd4953eb016a0129e47560f7e352
2018-04-12 17:59:16 -07:00
Adam Retter
ca87aef82d Added support for SstFileManager to RocksJava
Summary: Closes https://github.com/facebook/rocksdb/pull/3666

Differential Revision: D7457634

Pulled By: sagar0

fbshipit-source-id: 47741e2ee66e9255c580f4e38cfb86b284c27c2f
2018-04-06 21:26:32 -07:00
Adam Retter
8917eee962 Fixed small typos
Summary: Closes https://github.com/facebook/rocksdb/pull/3667

Differential Revision: D7470060

Pulled By: miasantreble

fbshipit-source-id: 8e8545cda38f0805f35ccdb8841666a2d7a965f5
2018-04-01 17:14:46 -07:00
Sagar Vemuri
d5585bb605 Shared block cache in RocksJava
Summary:
Changes to support sharing block cache using the Java API.

Previously DB instances could share the block cache only when the same Options instance is passed to all the DB instances. But now, with this change, it is possible to explicitly create a cache and pass it to multiple options instances, to share the block cache.

Implementing this for [Rocksandra](https://github.com/instagram/cassandra/tree/rocks_3.0), but this feature has been requested by many java api users over the years.
Closes https://github.com/facebook/rocksdb/pull/3623

Differential Revision: D7305794

Pulled By: sagar0

fbshipit-source-id: 03e4e8ed7aeee6f88bada4a8365d4279ede2ad71
2018-03-21 18:43:05 -07:00
Tobias Tschinkowitz
ccb761364d Enable compilation on OpenBSD
Summary:
I modified the Makefile so that we can compile rocksdb on OpenBSD.
The instructions for building have been added to INSTALL.md.
The whole compilation process works fine like this on OpenBSD-current
Closes https://github.com/facebook/rocksdb/pull/3617

Differential Revision: D7323754

Pulled By: siying

fbshipit-source-id: 990037d1cc69138d22f85bd77ef4dc8c1ba9edea
2018-03-19 12:30:05 -07:00
Adam Retter
bef95be5d8 Improve the output of the RocksJava JUnit runner
Summary:
This changes the console output when the RocksJava tests are run. It makes spotting the errors and failures much easier; perviously the output was malformed with results like "ERun" where the "E" represented an error in the preceding test.
Closes https://github.com/facebook/rocksdb/pull/3621

Differential Revision: D7306172

Pulled By: sagar0

fbshipit-source-id: 3fa6f6e1ca6c6ea7ceef55a23ca81903716132b7
2018-03-16 13:27:55 -07:00
Javeme Lee
f6156fb558 Support StringAppendOperator(delimiter_char) constructor in java-api
Summary:
Fixes #3336
Closes https://github.com/facebook/rocksdb/pull/3337

Differential Revision: D7196585

Pulled By: sagar0

fbshipit-source-id: a854f3fc906862ecba685b31946e4ef7c0b421c5
2018-03-08 16:17:47 -08:00
Adam Retter
c5302a8a58 Java wrapper for Native Comparators
Summary:
This is an abstraction for working with custom Comparators implemented in native C++ code from Java. Native code must directly extend `rocksdb::Comparator`. When the native code comparator is compiled into the RocksDB codebase, you can then create a Java Class, and JNI stub to wrap it.

Useful if the C++/JNI barrier overhead is too much for your applications comparator performance.

An example is provided in `java/rocksjni/native_comparator_wrapper_test.cc` and `java/src/main/java/org/rocksdb/NativeComparatorWrapperTest.java`.
Closes https://github.com/facebook/rocksdb/pull/3334

Differential Revision: D7172605

Pulled By: miasantreble

fbshipit-source-id: e24b7eb267a3bcb6afa214e0379a1d5e8a2ceabe
2018-03-08 11:27:42 -08:00
Bruce Mitchener
a3a3f5497c Fix some typos in comments and docs.
Summary: Closes https://github.com/facebook/rocksdb/pull/3568

Differential Revision: D7170953

Pulled By: siying

fbshipit-source-id: 9cfb8dd88b7266da920c0e0c1e10fb2c5af0641c
2018-03-08 10:27:25 -08:00
Adam Retter
db2445ad24 Brings the Java API for WriteBatch inline with the C++ API
Summary:
* Exposes status
* Corrects some method naming
* Adds missing functionality
Closes https://github.com/facebook/rocksdb/pull/3550

Differential Revision: D7140790

Pulled By: sagar0

fbshipit-source-id: cbdab6c5a7ae4f3030fb46739e9060e381b26fa6
2018-03-02 15:44:10 -08:00
Adam Retter
2ac988c67e Add TransactionDB and OptimisticTransactionDB to the Java API
Summary:
Closes https://github.com/facebook/rocksdb/issues/697
Closes https://github.com/facebook/rocksdb/issues/1151
Closes https://github.com/facebook/rocksdb/pull/1298

Differential Revision: D7131402

Pulled By: sagar0

fbshipit-source-id: bcd34ce95ed88cc641786089ff4232df7b2f089f
2018-03-02 10:34:13 -08:00
Ben Clay
a247617e6f Java: Add copy constructors for various option classes
Summary:
Add Java-side copy constructors for:
- Options
- DBOptions
- ColumnFamilyOptions
- WriteOptions

along with unit tests to assert the copy worked.

NOTE: Unit tests are failing in travis but it looks like a global timeout issue. These tests pass.
Closes https://github.com/facebook/rocksdb/pull/3450

Differential Revision: D6874425

Pulled By: sagar0

fbshipit-source-id: 5bde68ea5b5225e071faea2628bf8bbf10bd65ab
2018-02-02 10:57:28 -08:00
Mark Isaacson
b8eb32f8cf Suppress lint in old files
Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.

Reviewed By: yiwu-arbug

Differential Revision: D6821806

fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
2018-01-29 12:56:42 -08:00
Adam Retter
a53c571d2d FreeBSD build support for RocksDB and RocksJava
Summary:
Tested on a clean FreeBSD 11.01 x64.

Closes https://github.com/facebook/rocksdb/pull/1423
Closes https://github.com/facebook/rocksdb/pull/3357

Differential Revision: D6705868

Pulled By: sagar0

fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd
2018-01-11 13:29:55 -08:00
Sagar Vemuri
677f249d6d Fix Travis build failures in CMake RocksJava
Summary:
Fixed RocksJava travis build failure due to a missing file in java/CMakeLists.txt. (from #3332).
Closes https://github.com/facebook/rocksdb/pull/3344

Differential Revision: D6686472

Pulled By: sagar0

fbshipit-source-id: dd3281dff1342c3a7235c402890420aa56db0fe3
2018-01-09 16:42:15 -08:00
Adam Retter
398d72fa61 Add autotune and #getBytesPerSecond() to RocksJava RateLimiter
Summary: Closes https://github.com/facebook/rocksdb/pull/3332

Differential Revision: D6667680

Pulled By: ajkr

fbshipit-source-id: b2bb6889257850a4eb6f6cbd7106f62df7b82730
2018-01-08 12:30:52 -08:00
Adam Retter
00e5e1ef7f Fixes the build on Windows
Summary:
As discovered during v5.9.2 release, and forward-ported.
Closes https://github.com/facebook/rocksdb/pull/3323

Differential Revision: D6657209

Pulled By: siying

fbshipit-source-id: b560d9f8ddb89e0ffaff7c895ec80f68ddf7dab4
2018-01-03 12:27:12 -08:00
Anand Ananthabhotla
fccc12f386 Add a histogram stat for memtable flush
Summary:
Add a new histogram stat called rocksdb.db.flush.micros for memtable
flush
Closes https://github.com/facebook/rocksdb/pull/3269

Differential Revision: D6559496

Pulled By: anand1976

fbshipit-source-id: f5c771ba2568630458751795e8c37a493ff9b14d
2017-12-15 18:57:00 -08:00
Tamir Duberstein
cf0d6aa007 CMake cross platform Java support and add JNI to travis
Summary:
Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform
Java support.

adamretter
Closes https://github.com/facebook/rocksdb/pull/2301

Differential Revision: D5070724

Pulled By: sagar0

fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7
2017-11-28 12:27:53 -08:00
anand1976
d394a6bb48 Add a ticker stat for number of keys skipped during iteration
Summary:
This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the
number of internal keys skipped during iteration. Keys can be skipped
due to deletes, or lower sequence number, or higher sequence number
than the one requested.

Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary,
padding becomes a zero size array, which the Windows compiler doesn't
like. So add a cacheline worth of padding in that case to keep it happy.
We cannot conditionally add padding as gcc doesn't allow using sizeof
in preprocessor directives.
Closes https://github.com/facebook/rocksdb/pull/3177

Differential Revision: D6353897

Pulled By: anand1976

fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2
2017-11-20 21:26:37 -08:00
Sagar Vemuri
76c3fbd651 Add Memtable Read Tier to RocksJava
Summary:
This options was introduced in the C++ API in #1953 .
Closes https://github.com/facebook/rocksdb/pull/3064

Differential Revision: D6139010

Pulled By: sagar0

fbshipit-source-id: 164de11d539d174cf3afe7cd40e667049f44b0bc
2017-11-02 17:27:37 -07:00
zawlazaw
57fcdc264a added missing subcodes and improved error message for missing enum values
Summary:
Java's `Status.SubCode` was out of sync with `include/rocksdb/status.h:SubCode`.

When running out of disc space this led to an `IllegalArgumentException` because of an invalid status code, rather than just returning the corresponding status code without an exception.

I added the missing status codes.

By this, we keep the behaviour of throwing an `IllegalArgumentException` in case of newly added status codes that are defined in C but not in Java.

We could think of an alternative strategy: add in Java another code "UnknownCode" which acts as a catch-all for all those status codes that are not yet mirrored from C to Java. This approach would never throw an exception but simply return a non-OK status-code.

I think the current approach of throwing an Exception in case of a C/Java inconsistency is fine, but if you have some opinion on the alternative strategy, then feel free to comment here.
Closes https://github.com/facebook/rocksdb/pull/3050

Differential Revision: D6129682

Pulled By: sagar0

fbshipit-source-id: f2bf44caad650837cffdcb1f93eb793b43580c66
2017-10-23 16:42:07 -07:00
Sagar Vemuri
cc67b22d6e Add OptionsUtil class to java/CMakeLists.txt
Summary:
Adding OptionsUtil java class and options_util.cc to java/CMakeLists.txt, which were missed accidentally when they were introduced in #2898.
Closes https://github.com/facebook/rocksdb/pull/2985

Differential Revision: D6015878

Pulled By: sagar0

fbshipit-source-id: 1abbd46db4aebad1e07ea53523eacbdcb12823e1
2017-10-12 16:57:05 -07:00
Adam Retter
560e984995 Added CompactionFilterFactory support to RocksJava
Summary:
This PR also includes some cleanup, bugfixes and refactoring of the Java API. However these are really pre-cursors on the road to CompactionFilterFactory support.
Closes https://github.com/facebook/rocksdb/pull/1241

Differential Revision: D6012778

Pulled By: sagar0

fbshipit-source-id: 0774465940ee99001a78906e4fed4ef57068ad5c
2017-10-12 11:12:16 -07:00
Sagar Vemuri
bb38cd03a9 Limit number of merge operands in Cassandra merge operator
Summary:
Now that RocksDB supports conditional merging during point lookups (introduced in #2923), Cassandra value merge operator can be updated to pass in a limit. The limit needs to be passed in from the Cassandra code.
Closes https://github.com/facebook/rocksdb/pull/2947

Differential Revision: D5938454

Pulled By: sagar0

fbshipit-source-id: d64a72d53170d8cf202b53bd648475c3952f7d7f
2017-10-02 16:11:40 -07:00
Adam Retter
983028f097 RocksJava build target for Docker on ppc64le
Summary:
This enables us to crossbuild pcc64le RocksJava binaries with a suitably old version of glibc (2.17) on CentOS 7.
Closes https://github.com/facebook/rocksdb/pull/2491

Differential Revision: D5955301

Pulled By: sagar0

fbshipit-source-id: 69ef9746f1dc30ffde4063dc764583d8c7ae937e
2017-10-02 11:11:56 -07:00
Sagar Vemuri
0806801dc8 DestroyDB API
Summary:
Expose DestroyDB API in RocksJava.
Closes https://github.com/facebook/rocksdb/pull/2934

Differential Revision: D5914775

Pulled By: sagar0

fbshipit-source-id: 84af6ea0d2bccdcfb9fe8c07b2f87373f0d5bab6
2017-09-26 16:42:11 -07:00
Sagar Vemuri
c8f3606731 Expose LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs in RocksJava
Summary:
JNI wrappers for LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs.
Closes https://github.com/facebook/rocksdb/pull/2898

Differential Revision: D5857934

Pulled By: sagar0

fbshipit-source-id: 68b79e83eab8de9416e3f1fef73e11cf7947e90a
2017-09-21 17:29:13 -07:00
Sagar Vemuri
96a13b4f4b Use jemalloc in rocksdbjni library built via vagrant
Summary:
Problem:
During RocksJava performance testing we found that the rocksdb jni library is not built with jemalloc; instead it was getting built with the default glibc malloc. We saw quite a bit of memory bloat due to this.

Addressed this by installing jemalloc-devel package in the vm that we use to build release jars.
Closes https://github.com/facebook/rocksdb/pull/2916

Differential Revision: D5887018

Pulled By: sagar0

fbshipit-source-id: ace0b5d60234b3a30dcd5d39633e7827a5982a50
2017-09-21 16:42:06 -07:00
Sagar Vemuri
3fc08fa88e Expose max_background_jobs option in RocksJava
Summary:
This option was introduced in the C++ API in RocksDB 5.6 in bb01c1880c . Now, exposing it through RocksJava API.
Closes https://github.com/facebook/rocksdb/pull/2908

Differential Revision: D5864224

Pulled By: sagar0

fbshipit-source-id: 140aa55dcf74b14e4d11219d996735c7fdddf513
2017-09-20 10:26:37 -07:00
Pengchao Wang
e4234fbdcf collecting kValue type tombstone
Summary:
In our testing cluster, we found large amount tombstone has been promoted to kValue type from kMerge after reaching the top level of compaction. Since we used to only collecting tombstone in merge operator, those tombstones can never be collected.

This PR addresses the issue by adding a GC step in compaction filter, which is only for kValue type records. Since those record already reached the top of compaction (no earlier data exists) we can safely remove them in compaction filter without worrying old data appears.

This PR also removes an old optimization in cassandra merge operator for single merge operands.  We need to do GC even on a single operand, so the optimation does not make sense anymore.
Closes https://github.com/facebook/rocksdb/pull/2855

Reviewed By: sagar0

Differential Revision: D5806445

Pulled By: wpc

fbshipit-source-id: 6eb25629d4ce917eb5e8b489f64a6aa78c7d270b
2017-09-18 16:27:12 -07:00
Ben Clay
382277d0fe JNI support for ReadOptions::iterate_upper_bound
Summary:
Plumbed ReadOptions::iterate_upper_bound through JNI.

Made the following design choices:
* Used Slice instead of AbstractSlice due to the anticipated usecase (key / key prefix). Can change this if anyone disagrees.
* Used Slice instead of raw byte[] which seemed cleaner but necessitated the package-private handle-based Slice constructor. Followed WriteBatch as an example.
* We need a copy constructor for ReadOptions, as we create one base ReadOptions for a particular usecase and clone -> change the iterate_upper_bound on each slice operation. Shallow copy seemed cleanest.
* Hold a reference to the upper bound slice on ReadOptions, in contrast to Snapshot.

Signed a Facebook CLA this morning.
Closes https://github.com/facebook/rocksdb/pull/2872

Differential Revision: D5824446

Pulled By: sagar0

fbshipit-source-id: 74fc51313a10a81ecd348625e2a50ca5b7766888
2017-09-14 18:28:20 -07:00