Commit Graph

2588 Commits

Author SHA1 Message Date
fyrz
39464a990f [RocksJava] Options Refactoring 3.6
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:

Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).

To align BuiltinComparator with every other Enum it was moved to
a separate file.

Test Plan:
make rocksdbjava
make jtest
2014-10-30 22:26:39 +01:00
Yueh-Hsuan Chiang
0f7f3b8605 Check InfoLogLevel earlier in Log functions.
Summary: Check InfoLogLevel earlier in Log functions.

Test Plan: auto_roll_logger_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27993
2014-10-30 13:36:18 -07:00
Yueh-Hsuan Chiang
73605d9179 Apply InfoLogLevel to the logs in util/db_info_dumper.cc
Summary:
* Rename util/db_info_dummper.cc to util/db_info_dumper.cc
* Apply InfoLogLevel to the logs in util/db_info_dumper.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27915
2014-10-30 13:34:44 -07:00
Yueh-Hsuan Chiang
fda592d902 Merge pull request #356 from fyrz/RocksJava-TableOptions-3.6
[RocksJava] BlockBasedTableConfig & PlainTableConfig enhancements
2014-10-30 13:17:53 -07:00
fyrz
c73d13bb81 [RocksJava] Integrate review comments from yhchiang 2014-10-30 20:54:35 +01:00
fyrz
b011e201fa Integrated review comments by ankgup87
- Added tests
- Minor code-style changes
2014-10-30 20:54:35 +01:00
fyrz
2c1bd8846f BlockBasedTableConfig & PlainTableConfig enhancements
Summary:
BlockBasedTableConfig
- ported Checksum
- ported IndexType

PlainTableConfig
- added missing options
- added EncodingType

Test Plan:
make rocksdbjava
make jtest

Differential Revision: https://reviews.facebook.net/D26595
2014-10-30 20:54:35 +01:00
Igor Canadi
e770d61751 Merge pull request #371 from dlezama/master
Fix build break because of unsigned/signed mismatch
2014-10-30 11:01:41 -07:00
Damian Lezama
41af0f56b0 Fix build break because of unsigned/signed mismatch 2014-10-30 10:36:13 -07:00
sdong
c5db7f2605 Fix CompactionPickerTest.Level1Trigger2
Summary: CompactionPickerTest.Level1Trigger2 now depends on the STL implementation to be correct. Fix it.

Test Plan: Run the test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27963
2014-10-30 10:00:09 -07:00
Yueh-Hsuan Chiang
37e9b63701 Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h
Summary: Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27885
2014-10-29 17:57:00 -07:00
Yueh-Hsuan Chiang
217cc217d7 Apply InfoLogLevel to the logs in table/meta_blocks.cc
Summary: Apply InfoLogLevel to the logs in table/meta_blocks.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27903
2014-10-29 17:55:19 -07:00
Igor Canadi
635905481d WalManager
Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety.

Test Plan: added test

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27873
2014-10-29 17:43:37 -07:00
Yueh-Hsuan Chiang
fd95745a59 Fix compile error in table/plain_table_index.cc
Summary:
Fix compile error in table/plain_table_index.cc

Test Plan:
make
2014-10-29 17:42:38 -07:00
sdong
c3dd0f75da comparator_db_test to cover more irregular comparators
Summary:
comparator_db_test now adds verification for three more comparators:
(1) one that store double as string
(2) one that cast uint64 to string
(3) one that concatenate two strings, prefixing their sizes.
(4) one that order by hash of the string

Test Plan:
Run ./comparator_db_test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27927
2014-10-29 17:34:24 -07:00
Yueh-Hsuan Chiang
6afafa3694 Apply InfoLogLevel to the logs in utilities/merge_operators/uint64add.cc
Summary:
Apply InfoLogLevel to the logs and add missing copy-right information
to  utilities/merge_operators/uint64add.cc.

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27897
2014-10-29 17:10:21 -07:00
Yueh-Hsuan Chiang
e7ad69b9fe Apply InfoLogLevel to the logs in table/plain_table_index.cc
Summary: Apply InfoLogLevel to the logs in table/plain_table_index.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27909
2014-10-29 17:08:40 -07:00
Yueh-Hsuan Chiang
bbd9c53457 Apply InfoLogLevel to the logs in table/block_based_table_builder.cc
Summary: Apply InfoLogLevel to the logs in table/block_based_table_builder.cc

Test Plan: make

Reviewers: igor, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27921
2014-10-29 17:08:20 -07:00
Lei Jin
065766b8d2 DynamicCompactionOptions: relax the check bound a little
Summary:
Increase the level size so that impact of a single file is smaller.
Also relax the bound

Test Plan: ran locally

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27939
2014-10-29 17:02:21 -07:00
sdong
5c82a8837e Add a test in compaction_picker_test to test the max score
Summary: Add a new unit test in compaction_picker_test to make sure level-based compaction to pick up the level with the largest score.

Test Plan: Run the new test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27933
2014-10-29 16:58:18 -07:00
sdong
86de2007b8 Add ComparatorDBTest to test non-default comparators
Summary:
Add some helper functions to make sure DB works well for non-default comparators.
Add a test for SimpleSuffixReverseComparator.

Test Plan: Run the new test

Reviewers: ljin, rven, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27831
2014-10-29 15:42:36 -07:00
Lei Jin
17be187ff9 dummy var to suppress compiler warning/error
Summary: Revmoed this in D25641, causing compiler complain. put it back

Test Plan: make release

Reviewers: igor, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27891
2014-10-29 15:38:34 -07:00
Igor Canadi
c2999f54bd Revert "tmp"
This reverts commit 9ab0132360.
2014-10-29 15:29:33 -07:00
sdong
76d1c28e82 Make CompactionPicker more easily tested
Summary:
Make compaction picker easier to test.
The basic idea is to separate a minimum subcomponent of Version to VersionStorageInfo, which just responsible to LSM tree. A stub VersionStorageInfo can then be easily created and passed into compaction picker so that we can check the outputs.

It now passes most tests. Still two things need to be done:
(1) deal with the FIFO compaction's file size.
(2) write an example test to make sure the interface can do the job.

Add a compaction_picker_test to make sure compaction picker codes can be easily unit tested.

Test Plan:
Pass all unit tests and compaction_picker_test

Reviewers: yhchiang, rven, igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27639
2014-10-29 15:16:53 -07:00
Yueh-Hsuan Chiang
01e6f85098 Apply InfoLogLevel to the logs in db/transaction_log_impl.h
Summary: Apply InfoLogLevel to the logs in db/transaction_log_impl.h

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27867
2014-10-29 15:13:31 -07:00
Yueh-Hsuan Chiang
082e49ba82 Apply InfoLogLevel to the logs in db/repair.cc
Summary: Apply InfoLogLevel to the logs in db/repair.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27855
2014-10-29 15:12:50 -07:00
Yueh-Hsuan Chiang
c4b468000b Apply InfoLogLevel to the logs in db/flush_job.cc
Summary: Apply InfoLogLevel to the logs in db/flush_job.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27849
2014-10-29 15:12:10 -07:00
Yueh-Hsuan Chiang
34d436b7db Apply InfoLogLevel to the logs in db/column_family.cc
Summary: Apply InfoLogLevel to the logs in db/column_family.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27843
2014-10-29 15:11:32 -07:00
Yueh-Hsuan Chiang
cda9943f9f Apply InfoLogLevel to the logs in db/compaction_picker.cc
Summary: Apply InfoLogLevel to the logs in db/compaction_picker.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27837
2014-10-29 15:11:12 -07:00
Yueh-Hsuan Chiang
7b3a618f94 Apply InfoLogLevel to the logs in db/db_filesnapshot.cc
Summary: Apply InfoLogLevel to the logs in db/db_filesnapshot.cc

Test Plan: make

Reviewers: ljin, sdong, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27813
2014-10-29 14:06:14 -07:00
Lei Jin
2d4fe048f4 remove dead code
Summary:
as title

Test Plan:
make db_test
2014-10-29 13:38:06 -07:00
Lei Jin
9ab0132360 tmp
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2014-10-29 13:36:47 -07:00
Lei Jin
76d54530d3 minor - remove default value for ChangeFilterOptions() and ChangeCompactionOptions()
Summary:
So now all open() in db_test should get options from callsite. And
destroy() always uses the last used options saved on open()
I will start to integrate env_mem in the next diff

Test Plan: make all check -j32

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27819
2014-10-29 13:36:18 -07:00
Lei Jin
44f0ff31c2 use fallocate(FALLOC_FL_PUNCH_HOLE) to release unused blocks at the end of file
Summary:
ftruncate does not always free preallocated unused space at the end of file.
In some cases, we pin too much disk space than it should

Test Plan: env_test

Reviewers: sdong, rven, yhchiang, igor

Reviewed By: igor

Subscribers: nkg-, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D25641
2014-10-29 12:24:49 -07:00
Lei Jin
97451f837e add an env var ROCKSDB_TESTS_FROM to control where to start from a list of tests
Summary:
Sometimes, I got a test failure. After fixing that, I want to resume
db_test from that test. ROCKSDB_TESTS_FROM is for this purpose.

Test Plan: as title

Reviewers: yhchiang, rven, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27807
2014-10-29 12:02:11 -07:00
Lei Jin
e130e88bc6 DBTest: options clean up - part 4
Summary: as title

Test Plan: as part 1

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27789
2014-10-29 12:00:42 -07:00
Lei Jin
34f3c5a20f DBTest: options clean up - part 3
Summary: as title

Test Plan: same as part 1

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27705
2014-10-29 12:00:01 -07:00
Lei Jin
cdc7230e4c DBTest: options clean up - part 2
Summary: as title

Test Plan: same as part 1

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27693
2014-10-29 11:59:18 -07:00
Lei Jin
5a921b8950 DBTest: options clean up - part 1
Summary:
DBTest has several functions (Reopen(), TryReopen(), ChangeOptins(), etc
that takes a pointer to options), depending on if it is nullptr, it uses
different options underneath. This makes it really hard to track what
options is used in different test case. We should just kill the default
value and make it being passed into explicitly. It is going to be very
hairy. I will start with simple ones.

Test Plan:
make db_test
stacked diffs, will run test with full stack

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27687
2014-10-29 11:58:09 -07:00
Yueh-Hsuan Chiang
c9c935923e Move the check to the beginning of the loop in VersionEdit::EncodeTo() 2014-10-29 11:21:51 -07:00
Yueh-Hsuan Chiang
2110e43a5a Remove an unnecessary include file in version_edit.cc
Summary:
Remove an unnecessary include file in version_edit.cc
2014-10-29 11:19:44 -07:00
Igor Canadi
412b7f85bb Include atomic in mock_table.h 2014-10-28 18:10:55 -07:00
Igor Canadi
c082853340 Include all the mocks 2014-10-28 17:55:08 -07:00
Igor Canadi
abac3d6476 TableMock + framework for mock classes
Summary:
This diff replaces BlockBasedTable in flush_job_test with TableMock, making it depend on less things and making it closer to an unit test than integration test.

It also introduces a framework to compile mock classes -- Any file named *mock.cc will not be compiled into the build. It will only get compiled into the tests. What way we can mock out most other classes, Version, VersionSet, DBImpl, etc.

Test Plan: flush_job_test

Reviewers: ljin, rven, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27681
2014-10-28 17:52:32 -07:00
Yueh-Hsuan Chiang
fb3f8ffe5e Improve the robustness of PartialCompactionFailure test again.
Summary:
Improve the robustness of PartialCompactionFailure test again.

Test Plan:
./db_test
2014-10-28 15:35:10 -07:00
Yueh-Hsuan Chiang
60fa7d1365 Improve the robustnesss of PartialCompactionFailure test.
Summary:
Improve the robustness of PartialCompactionFailure test.

Test Plan:
./db_test
2014-10-28 15:17:50 -07:00
Yueh-Hsuan Chiang
3772a3d09d Fix the bug where compaction does not fail when RocksDB can't create a new file.
Summary:
This diff has two fixes.

1. Fix the bug where compaction does not fail when RocksDB can't create a new file.
2. When NewWritableFiles() fails in OpenCompactionOutputFiles(), previously such fail-to-created file will be still be included as a compaction output.  This patch also fixes this bug.
3. Allow VersionEdit::EncodeTo() to return Status and add basic check.

Test Plan:
./version_edit_test
export ROCKSDB_TESTS=FileCreationRandomFailure
./db_test

Reviewers: ljin, sdong, nkg-, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D25581
2014-10-28 14:27:26 -07:00
Yueh-Hsuan Chiang
c49dedbe04 Merge pull request #366 from fyrz/RocksJava-Backup-Restore-Improvements
[RocksJava] - BackupInfos & Restore-/BackupableDB enhancements
2014-10-28 12:14:11 -07:00
Igor Canadi
a39e931e50 FlushProcess
Summary:
Abstract out FlushProcess and take it out of DBImpl.
This also includes taking DeletionState outside of DBImpl.

Currently this diff is only doing the refactoring. Future work includes:
1. Decoupling flush_process.cc, make it depend on less state
2. Write flush_process_test, which will mock out everything that FlushProcess depends on and test it in isolation

Test Plan: make check

Reviewers: rven, yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27561
2014-10-28 11:54:33 -07:00
Lei Jin
efa2fb33b0 make LevelFileNumIterator and LevelFileIteratorState anonymous
Summary: No need to expose them in .h

Test Plan: make release

Reviewers: igor, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27645
2014-10-28 11:42:22 -07:00