Commit Graph

3011 Commits

Author SHA1 Message Date
Igor Canadi
b068f0a673 Upgrade our compilers
Summary:
Upgrade gcc to 4.9.1 and clang to dev.

With new compilers I succeeded to run thread sanitizer, too. I'll post output (doesn't look good) and fix some things in separate diffs.

Test Plan: compiles with both g++ and clang

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32007
2015-01-23 11:12:37 -08:00
Yueh-Hsuan Chiang
a76d928629 Merge pull request #466 from fyrz/RocksJava-Support-ReadOptions-Iterator
[RocksJava] Allow using ReadOptions with iterators
2015-01-23 00:20:40 -08:00
fyrz
bef7821f07 [RocksJava] ReadOptions support in Iterators
The methods:

- newIterator
- iterators

support now also ReadOptions. That allows a user of the Java API
to retrieve RocksIterator instances on a snapshot.
2015-01-23 08:02:55 +01:00
Yueh-Hsuan Chiang
3b494a6103 Make options_test runnable on ROCKSDB_LITE
Summary:
Make options_test runnable on ROCKSDB_LITE by blocking
those tests that require non-ROCKSDB_LITE feature.

Test Plan:
make options_test OPT=-DROCKSDB_LITE -j32
./options_test
make clean
make options_test -j32
./options_test

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32025
2015-01-22 16:57:16 -08:00
Yueh-Hsuan Chiang
912c52e825 Merge pull request #465 from fyrz/RocksJava-BlockBasedTable-FormatVersion
[RocksJava] Format version support in BlockBasedTableConfig
2015-01-22 16:00:42 -08:00
Yueh-Hsuan Chiang
cd4c071973 Update HISTORY.md for GetThreadStatus() support on compaction. 2015-01-22 15:46:56 -08:00
Yueh-Hsuan Chiang
46a7048dcd Reduce false alarm in ThreadStatusMultipleCompaction test 2015-01-22 15:45:02 -08:00
Yueh-Hsuan Chiang
aed0286984 Merge pull request #462 from fyrz/RocksJava-JNI-allocation-correction
[RocksJava] ColumnFamily name JNI correction
2015-01-22 15:41:28 -08:00
Igor Canadi
e5df90f5d0 Fix comment (minor) 2015-01-22 14:59:36 -08:00
fyrz
dd53428f8b Incorporated review comments
- added spaces between operators
- removed obsolete text in JavaDoc
2015-01-22 23:47:31 +01:00
fyrz
908258a4f2 [RocksJava] BlockBasedTableConfig 3.10
- Added support for format version in BlockBasedTableConfig
2015-01-22 23:46:38 +01:00
fyrz
2efe228499 [RocksJava] Incorporated changes for D31809 2015-01-22 23:37:45 +01:00
sdong
4e48753b73 Sync manifest file when initializing it
Summary: Now we don't sync manifest file when initializing it, so DB cannot be safely reopened before the first mem table flush. Fix it by syncing it. This fixes fault_injection_test.

Test Plan: make all check

Reviewers: rven, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D32001
2015-01-22 14:32:03 -08:00
fyrz
e204a5a16c [RocksJava] ColumnFamily name JNI correction
Summary:
Previous to this commit there was a problem with unterminated
String usage as jByteArrays are not zero terminated.

Test Plan:
make rocksdbjava
make jtest
mvn -f rocksjni.pom package

Reviewers: yhchiang, adamretter, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D31809
2015-01-22 23:29:45 +01:00
fyrz
96264784d9 [RocksJava] ColumnFamily name JNI correction
Previous to this commit there was a problem with unterminated
String usage as jByteArrays are not zero terminated.
2015-01-22 23:29:45 +01:00
Igor Canadi
ae82849bc9 Fix build failure 2015-01-21 18:23:12 -08:00
Igor Canadi
423dee8418 Abort db_bench if Get() returns error
Summary:
I saw this when running readrandom benchmark with corrupted database -- benchmark worked!

If a Get() returns corruption we should probably abort.

Test Plan: compiles

Reviewers: yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31701
2015-01-21 18:18:15 -08:00
sdong
206237d121 DBImpl::CheckConsistency() shouldn't create path name with double "/"
Summary: GetLiveFilesMetaData() already adds a leading "/" in file name. No need to add one extra "/" in DBImpl::CheckConsistency()

Test Plan: make all check

Reviewers: yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D31779
2015-01-21 16:36:13 -08:00
Yueh-Hsuan Chiang
5e98e53249 Merge pull request #458 from fyrz/RocksJava-TTLDB-Support
[RocksJava] TTL DB support
2015-01-19 20:55:48 -08:00
fyrz
4ffe0be414 [RocksJava] Integrated changes for D31449 2015-01-19 22:19:58 +01:00
fyrz
e828567541 [RocksJava] Integrated changes from D31449 2015-01-17 23:52:43 +01:00
fyrz
859c54a03d [RocksJava] TTL-Support
Summary:
TTLDB Support exposed in Java-API. It is now
possible to open a datbase using the RocksDB time
to live feature.

Test Plan:
make rocksdbjava
make test
mvn -f rocksjni.pom package

@Adam please test mac osx compile

Reviewers: yhchiang, adamretter, ankgup87

Subscribers: dhruba, adam

Differential Revision: https://reviews.facebook.net/D31449
2015-01-17 23:28:06 +01:00
fyrz
5ff8aec4db [RocksJava] TTL Support 2015-01-17 23:28:06 +01:00
fyrz
ca47da9e63 [RocksJava] TTL-Support 2015-01-17 23:28:05 +01:00
Yueh-Hsuan Chiang
1190ebe5aa Merge pull request #461 from fyrz/RocksJava-DirectSlice-Fix
[RocksJava] DirectSlice String termination fix
2015-01-17 09:08:50 -08:00
fyrz
ea25ff7158 [RocksJava] Integrated proposed simplificiation 2015-01-17 01:22:29 +01:00
fyrz
d68e83c356 [RocksJava] DirectSlice String termination fix
DirectSlice fix for non terminated String copy. This lead sometimes
to problems with DirectSliceTest.
2015-01-17 01:15:49 +01:00
sdong
0ddf5f73ec memenv: normalize file path
Summary: Now using memenv, DB will not able to be reopened, since a "//" in the file name. Fix it by normalizing file path.

Test Plan: Add a unit test that used to fail and now pass.

Reviewers: rven, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D31767
2015-01-16 15:31:39 -08:00
Yueh-Hsuan Chiang
4d9d5955ac Merge pull request #464 from fyrz/RocksJava-Various-Fixes
[RocksJava] Fixes to latest WriteBatchWithIndex addition
2015-01-16 14:41:52 -08:00
Yoshinori Matsunobu
ceaea2b72d Adding prefix_extractor string config parameter
Summary:
This diff enables to configure prefix_extractor
string parameter as a CF option.

Test Plan: make all check, ./options_test

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31653
2015-01-16 13:44:42 -08:00
Igor Canadi
3d628f8f22 Update format_version comment
Summary: We added a new format version. Reflect that in the comments.

Test Plan: none

Reviewers: sdong, rven, yhchiang, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31629
2015-01-16 09:18:45 -08:00
Igor Canadi
155bec4cf2 fallocate also tests FALLOC_FL_KEEP_SIZE 2015-01-15 14:15:43 -08:00
fyrz
c75c02e7a2 [RocksJava] WriteBatchWithIndexTest fix
Previous to this commit identiy checks were performed. Now tests are
performed using equals - method as intended.
2015-01-15 21:56:46 +01:00
fyrz
c787fb50b8 [RocksJava] JavaDoc errors in Java8
Some of the latest commits included illegal JavaDoc formattings.
2015-01-15 21:56:46 +01:00
Yueh-Hsuan Chiang
b229f970df Remove Compaction::ReleaseInputs().
Summary:
This patch remove the unnecessary Compaction::ReleaseInputs().

Compaction::ReleaseInputs() tries to unref its input_version
and column_family.  However, such unref is always done in
~Compaction(), and all current ReleaseInputs() calls are
right before the destructor.

Test Plan: ./db_test

Reviewers: igor

Reviewed By: igor

Subscribers: igor, rven, dhruba, sdong

Differential Revision: https://reviews.facebook.net/D31605
2015-01-15 12:44:19 -08:00
Venkatesh Radhakrishnan
f2ddb8b452 Fix for bug where GeoDB accesses key after next modification of iterator
Summary:
While running cross-functional tests for weak iterators, I
encountered a bug in GeoDB. GeoDB reads a key from the database and
tries to use it after doing a Seek. Fixing it by storing the key locally
so that it is still visible after the Seek.

Test Plan: Run geodb_test

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31599
2015-01-15 11:47:41 -08:00
Thomas Dudziak
d10f1de2b4 Ported LevelDB's fault_injection_test
Summary:
This is a port of [[ https://github.com/google/leveldb/blob/master/db/fault_injection_test.cc | LevelDB's fault_injection_test ]] to RocksDB. Unfortunately it fails with:

```
==== Test FaultInjectionTest.FaultTest
db/fault_injection_test.cc:491: Corruption: no meta-nextfile entry in descriptor
#0   ./fault_injection_test() [0x41477a] rocksdb::FaultInjectionTest::PartialCompactTestReopenWithFault(rocksdb::FaultInjectionTest::ResetMethod, int, int) /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:491
#1   ./fault_injection_test() [0x40a38a] rocksdb::_Test_FaultTest::_Run() /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:517
#2   ./fault_injection_test() [0x415bea] rocksdb::_Test_FaultTest::_RunIt() /data/users/tomdzk/rocksdb/db/fault_injection_test.cc:507
#3   ./fault_injection_test() [0x584367] rocksdb::test::RunAllTests() /data/users/tomdzk/rocksdb/util/testharness.cc:70
#4   /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libc.so.6(__libc_start_main+0x10e) [0x7f7a40857efe] ??  ??:0
#5   ./fault_injection_test() [0x408bb8] _start ??:0
```

so I commented out the test invocation in the source code for now (lines 514-520) so it can be merged.

Test Plan: This is a new test.

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31587
2015-01-15 10:28:10 -08:00
Igor Canadi
2bb059007b Change db_stress to work with format_version == 2 2015-01-14 16:25:36 -08:00
Igor Canadi
9ab5adfc59 New BlockBasedTable version -- better compressed block format
Summary:
This diff adds BlockBasedTable format_version = 2. New format version brings better compressed block format for these compressions:
1) Zlib -- encode decompressed size in compressed block header
2) BZip2 -- encode decompressed size in compressed block header
3) LZ4 and LZ4HC -- instead of doing memcpy of size_t encode size as varint32. memcpy is very bad because the DB is not portable accross big/little endian machines or even platforms where size_t might be 8 or 4 bytes.

It does not affect format for snappy.

If you write a new database with format_version = 2, it will not be readable by RocksDB versions before 3.10. DB::Open() will return corruption in that case.

Test Plan:
Added a new test in db_test.
I will also run db_bench and verify VSIZE when block_cache == 1GB

Reviewers: yhchiang, rven, MarkCallaghan, dhruba, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D31461
2015-01-14 16:24:24 -08:00
Yueh-Hsuan Chiang
2355931c6f Merge pull request #450 from adamretter/writebatch-with-index
Add WriteBatchWithIndex to the Java API
2015-01-14 13:40:04 -08:00
Adam Retter
3d246c89cc Abstract duplicate code on key and value slice objects into generic methods 2015-01-14 21:16:06 +00:00
Adam Retter
2d0dd8db3b Implement WBWIRocksIterator for WriteBatchWithIndex in the Java API 2015-01-14 21:16:06 +00:00
Adam Retter
de678b288e Abstractions for common iterator behaviour 2015-01-14 21:16:06 +00:00
Adam Retter
e01acb3a04 Test for WriteBatchWithIndex#newIterator() 2015-01-14 21:16:06 +00:00
Adam Retter
56f24941ab Simplify the Java API by permitting WriteBatchWithIndex to be provided straight to RocksDB#write 2015-01-14 21:16:06 +00:00
Adam Retter
95d5f98487 Test for RocksDB#write(WriteBatchWithIndex) 2015-01-14 21:16:06 +00:00
Adam Retter
ef5b34dee0 Implement WriteBatchWithIndex in the Java API 2015-01-14 21:16:05 +00:00
Adam Retter
c6e5545612 Abstractions for common write batch behaviour 2015-01-14 21:16:05 +00:00
Adam Retter
be905491bf Test for WriteBatchWithIndex#newIteratorWithBase(org.rocksdb.RocksIterator) 2015-01-14 21:16:05 +00:00
Adam Retter
2241e3f4d5 Extract the interface for a RocksIterator 2015-01-14 21:16:05 +00:00