Commit Graph

2815 Commits

Author SHA1 Message Date
fyrz
27129c739f [RocksJava] KeyMayExist w/o ColumnFamilies 2014-11-01 01:02:12 +01:00
sdong
86905e3cbb Move VersionBuilder logic to a separate .cc file
Summary: Move all the logic of VersionBuilder to a separate .cc file

Test Plan: make all check

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28083
2014-10-31 16:34:38 -07:00
Igor Canadi
74eb4fbe93 CompactionJob
Summary:
Long awaited CompactionJob class! Move most compaction-related things from DBImpl to CompactionJob, making CompactionJob easier to test and understand.

Currently this is just replicating exactly the same functionality with as little as change as possible. As future work, we should:
1. Add CompactionJob tests (I think I'll do that tomorrow)
2. Reduce CompactionJob's state that it inherits from DBImpl
3. Figure out how to do yielding to flush better. Currently I implemented a callback as we agreed yesterday, but I don't think it's a good long term solution.

This reduces db_impl.cc from 5000+ LOC to 3400!

Test Plan: make check, will add CompactionJob-specific tests, probably also move some tests from db_test to compaction_job_test

Reviewers: rven, yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27957
2014-10-31 16:31:25 -07:00
Yueh-Hsuan Chiang
8ddddd62d0 Fix incorrect fixing of lint errors in ldb_cmd.cc
Summary: Fix incorrect fixing of lint errors in ldb_cmd.cc

Test Plan: reduce_levels_test

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28137
2014-10-31 16:15:15 -07:00
Igor Canadi
46c14c6661 Fix #258. benchmarkharness -- make bm_min_usec uint 2014-10-31 15:41:56 -07:00
Lei Jin
72cb7cf201 Add fsync / corrupt simulation to env_mem
Summary: as title

Test Plan: env_mem_test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28077
2014-10-31 15:16:31 -07:00
Lei Jin
0e526eb9d7 introduce TestMemEnv and use it in db_test
Summary:
TestMemEnv simulates all Env APIs using in-memory data structures.
We can use it to speed up db_test run, which is now reduced ~7mins when it is
enabled.
We can also add features to simulate power/disk failures in the next
step
TestMemEnv is derived from helper/mem_env
mem_env can not be used for rocksdb since some of its APIs do not give
the same results as env_posix. And its file read/write is not thread safe

Test Plan:
make all -j32
./db_test
./env_mem_test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28035
2014-10-31 15:08:10 -07:00
Lei Jin
8db24f4b35 exclude mock test file from MOCK_SOURCES
Summary: as title

Test Plan: build with mock_env_test.cc

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28107
2014-10-31 15:07:27 -07:00
Lei Jin
5594d446ff unfriend DBImpl and InternalStats from VersionStorageInfo
Summary: as title

Test Plan: make release

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28053
2014-10-31 15:04:39 -07:00
Lei Jin
82e3ae5403 fix c_test
Summary: as title

Test Plan: ./c_test

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28119
2014-10-31 15:04:01 -07:00
Yueh-Hsuan Chiang
bc9f36fd5e Fix lint errors and coding style of ldb related codes.
Summary: Fix lint errors and coding style of ldb related codes.

Test Plan: ./ldb

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28125
2014-10-31 15:01:39 -07:00
sdong
c645250ee0 CompactionStats to support larger value of RecordIn and RecordDrop
Summary: now we use %8d for RecordIn and %10d for RecordDrop, which is far too small for some use cases. Extend both of them to %12d.

Test Plan: run one test in db_test and see the LOG file.

Reviewers: igor, MarkCallaghan, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28041
2014-10-31 13:51:48 -07:00
sdong
f7e6c856ab Fix BaseReferencedVersionBuilder's destructor order
Summary: BaseReferencedVersionBuilder now unreference version before destructing VersionBuilder, which is wrong. Fix it.

Test Plan:
make all check
valgrind test to tests that used to fail

Reviewers: igor, yhchiang, rven, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28101
2014-10-31 13:42:18 -07:00
Lei Jin
c76dcb44d4 fix
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2014-10-31 13:30:08 -07:00
Lei Jin
b452dede5c fix
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2014-10-31 12:23:00 -07:00
Lei Jin
29d83cc33a temporarily remove -Wshadow 2014-10-31 12:20:39 -07:00
Jonah Cohen
c1a924b9f0 Move convenience.h to /include
Summary: Move header file so it can be referenced externally.

Test Plan: Rebuild.

Reviewers: ljin

Reviewed By: ljin

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28095
2014-10-31 12:08:43 -07:00
Venkatesh Radhakrishnan
7e01d12026 Add support for in place update for db_stress
Summary:
Added two flags which operate as follows:
in_place_update: enable in_place_update for default column family
set_in_place_one_in: toggles the value of the option inplace_update_support with a probability of 1/N

Test Plan:
Run db_stress with the two flags above set.
Specifically tried in_place_update set to true and set_in_place_one_in set to 10,000.

Reviewers: ljin, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28029
2014-10-31 12:02:14 -07:00
Igor Canadi
9f7fc3ac45 Turn on -Wshadow
Summary:
...and fix all the errors :)

Jim suggested turning on -Wshadow because it helped him fix number of critical bugs in fbcode. I think it's a good idea to be -Wshadow clean.

Test Plan: compiles

Reviewers: yhchiang, rven, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27711
2014-10-31 11:59:54 -07:00
Yueh-Hsuan Chiang
98849a35fa Apply InfoLogLevel to the logs in table/block_based_table_reader.cc
Summary:
Apply InfoLogLevel to the logs in table/block_based_table_reader.cc

Also, add missing checks for the returned status in BlockBasedTable::Open

Test Plan: make

Reviewers: sdong, ljin, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28005
2014-10-31 11:41:15 -07:00
sdong
4d2ba38b65 Make VersionBuilder unit testable
Summary:
Rename Version::Builder to VersionBuilder and expose its definition to a header.
Make VerisonBuilder not reference Version or ColumnFamilyData, only working with VersionStorageInfo.
Add version_builder_test which has a simple test.

Test Plan: make all check

Reviewers: rven, yhchiang, igor, ljin

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27969
2014-10-31 10:44:06 -07:00
Yueh-Hsuan Chiang
2b1f23dcae Apply InfoLogLevel to the logs in db/db_iter.cc
Summary: Apply InfoLogLevel to the logs in db/db_iter.cc

Test Plan: make

Reviewers: igor, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27861
2014-10-30 16:54:34 -07:00
Yueh-Hsuan Chiang
ccaf1aa7cb Merge pull request #372 from fyrz/RocksJava-CF-Merge-Hardening
[RocksJava] Merge with ColumnFamilies & Hardening CFHandle
2014-10-30 16:49:40 -07:00
Yueh-Hsuan Chiang
a29118ffc7 Merge pull request #355 from fyrz/RocksJava-Options-Refactoring-3.6
[RocksJava] Options Refactoring 3.6
2014-10-30 16:46:41 -07:00
fyrz
85b04ca765 [RocksJava] Review comments - reformatted MergeTest 2014-10-31 00:41:04 +01:00
fyrz
df7abb4e8d [RocksJava] Integrated code review comments
- Added TODO comments for disabled methods
2014-10-31 00:06:52 +01:00
fyrz
171be0ed55 Merge with ColumnFamilies & Hardening CFHandle
Summary:
ColumnFamilyHandles face the same problem as RocksIterator previously
so used methods were also applied for ColumnFamilyHandles.

Another problem with CF was that Options passed to CFs were
always filled with default values. To enable Merge, all parts
of the database must share the same merge functionality which
is not possible using default values. So from now on every
CF will inherit from db options.

Changes to RocksDB:
- merge can now take also a cfhandle

Changes to MergeTest:
- Corrected formatting
- Included also GC tests
- Extended tests to cover CF related parts
- Corrected paths to cleanup properly within the test process
- Reduced verbosity of the test

Test Plan:
make rocksdbjava
make jtest

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D27999
2014-10-30 22:32:04 +01:00
fyrz
39464a990f [RocksJava] Options Refactoring 3.6
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:

Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).

To align BuiltinComparator with every other Enum it was moved to
a separate file.

Test Plan:
make rocksdbjava
make jtest
2014-10-30 22:26:39 +01:00
Yueh-Hsuan Chiang
0f7f3b8605 Check InfoLogLevel earlier in Log functions.
Summary: Check InfoLogLevel earlier in Log functions.

Test Plan: auto_roll_logger_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27993
2014-10-30 13:36:18 -07:00
Yueh-Hsuan Chiang
73605d9179 Apply InfoLogLevel to the logs in util/db_info_dumper.cc
Summary:
* Rename util/db_info_dummper.cc to util/db_info_dumper.cc
* Apply InfoLogLevel to the logs in util/db_info_dumper.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27915
2014-10-30 13:34:44 -07:00
Yueh-Hsuan Chiang
fda592d902 Merge pull request #356 from fyrz/RocksJava-TableOptions-3.6
[RocksJava] BlockBasedTableConfig & PlainTableConfig enhancements
2014-10-30 13:17:53 -07:00
fyrz
c73d13bb81 [RocksJava] Integrate review comments from yhchiang 2014-10-30 20:54:35 +01:00
fyrz
b011e201fa Integrated review comments by ankgup87
- Added tests
- Minor code-style changes
2014-10-30 20:54:35 +01:00
fyrz
2c1bd8846f BlockBasedTableConfig & PlainTableConfig enhancements
Summary:
BlockBasedTableConfig
- ported Checksum
- ported IndexType

PlainTableConfig
- added missing options
- added EncodingType

Test Plan:
make rocksdbjava
make jtest

Differential Revision: https://reviews.facebook.net/D26595
2014-10-30 20:54:35 +01:00
Igor Canadi
e770d61751 Merge pull request #371 from dlezama/master
Fix build break because of unsigned/signed mismatch
2014-10-30 11:01:41 -07:00
Damian Lezama
41af0f56b0 Fix build break because of unsigned/signed mismatch 2014-10-30 10:36:13 -07:00
sdong
c5db7f2605 Fix CompactionPickerTest.Level1Trigger2
Summary: CompactionPickerTest.Level1Trigger2 now depends on the STL implementation to be correct. Fix it.

Test Plan: Run the test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27963
2014-10-30 10:00:09 -07:00
Yueh-Hsuan Chiang
37e9b63701 Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h
Summary: Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27885
2014-10-29 17:57:00 -07:00
Yueh-Hsuan Chiang
217cc217d7 Apply InfoLogLevel to the logs in table/meta_blocks.cc
Summary: Apply InfoLogLevel to the logs in table/meta_blocks.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27903
2014-10-29 17:55:19 -07:00
Igor Canadi
635905481d WalManager
Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety.

Test Plan: added test

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27873
2014-10-29 17:43:37 -07:00
Yueh-Hsuan Chiang
fd95745a59 Fix compile error in table/plain_table_index.cc
Summary:
Fix compile error in table/plain_table_index.cc

Test Plan:
make
2014-10-29 17:42:38 -07:00
sdong
c3dd0f75da comparator_db_test to cover more irregular comparators
Summary:
comparator_db_test now adds verification for three more comparators:
(1) one that store double as string
(2) one that cast uint64 to string
(3) one that concatenate two strings, prefixing their sizes.
(4) one that order by hash of the string

Test Plan:
Run ./comparator_db_test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27927
2014-10-29 17:34:24 -07:00
Yueh-Hsuan Chiang
6afafa3694 Apply InfoLogLevel to the logs in utilities/merge_operators/uint64add.cc
Summary:
Apply InfoLogLevel to the logs and add missing copy-right information
to  utilities/merge_operators/uint64add.cc.

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27897
2014-10-29 17:10:21 -07:00
Yueh-Hsuan Chiang
e7ad69b9fe Apply InfoLogLevel to the logs in table/plain_table_index.cc
Summary: Apply InfoLogLevel to the logs in table/plain_table_index.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27909
2014-10-29 17:08:40 -07:00
Yueh-Hsuan Chiang
bbd9c53457 Apply InfoLogLevel to the logs in table/block_based_table_builder.cc
Summary: Apply InfoLogLevel to the logs in table/block_based_table_builder.cc

Test Plan: make

Reviewers: igor, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27921
2014-10-29 17:08:20 -07:00
Lei Jin
065766b8d2 DynamicCompactionOptions: relax the check bound a little
Summary:
Increase the level size so that impact of a single file is smaller.
Also relax the bound

Test Plan: ran locally

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27939
2014-10-29 17:02:21 -07:00
sdong
5c82a8837e Add a test in compaction_picker_test to test the max score
Summary: Add a new unit test in compaction_picker_test to make sure level-based compaction to pick up the level with the largest score.

Test Plan: Run the new test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27933
2014-10-29 16:58:18 -07:00
sdong
86de2007b8 Add ComparatorDBTest to test non-default comparators
Summary:
Add some helper functions to make sure DB works well for non-default comparators.
Add a test for SimpleSuffixReverseComparator.

Test Plan: Run the new test

Reviewers: ljin, rven, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27831
2014-10-29 15:42:36 -07:00
Lei Jin
17be187ff9 dummy var to suppress compiler warning/error
Summary: Revmoed this in D25641, causing compiler complain. put it back

Test Plan: make release

Reviewers: igor, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27891
2014-10-29 15:38:34 -07:00
Igor Canadi
c2999f54bd Revert "tmp"
This reverts commit 9ab0132360.
2014-10-29 15:29:33 -07:00