Commit Graph

2933 Commits

Author SHA1 Message Date
sdong
b0cda4a116 DBTest.DynamicMemtableOptions to use single background compaction
Summary:
Now DBTest.DynamicMemtableOptions sets background compaction to be 4, without actually increasing thread pool size (even before the feature of automatic increasing it). To make sure the behavior stays the same after the automatic thread pool increasing, set it back to 1.

Hopefully it can fix the occasional failure of the test.

Test Plan: Run the test

Reviewers: igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28281
2014-11-04 11:14:12 -08:00
Yueh-Hsuan Chiang
8810850dd4 Apply InfoLogLevel to the logs in db/compaction_job.cc
Summary: Apply InfoLogLevel to the logs in db/compaction_job.cc

Test Plan: db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D28275
2014-11-04 11:07:11 -08:00
Yueh-Hsuan Chiang
71783f6524 Merge pull request #377 from fyrz/RocksJava-KeyMayExist
[RocksJava] KeyMayExist w/o ColumnFamilies
2014-11-04 11:05:19 -08:00
Yueh-Hsuan Chiang
614bbcbe2c Merge pull request #374 from fyrz/RocksJava-DBOptions-Extension-3.6
[RocksJava] DBOptions extension 3.6
2014-11-04 10:51:59 -08:00
Yueh-Hsuan Chiang
d8e1196635 Apply InfoLogLevel to the logs in db/version_set.cc
Summary: Apply InfoLogLevel to the logs in db/version_set.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27879
2014-11-04 10:34:33 -08:00
Yueh-Hsuan Chiang
2a019f1d03 Apply InfoLogLevel to the logs in db/wal_manager.cc
Summary: Apply InfoLogLevel to the logs in db/wal_manager.cc

Test Plan: db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28239
2014-11-04 10:34:18 -08:00
Yueh-Hsuan Chiang
469d474ba0 Apply InfoLogLevel to the logs in db/db_impl.cc
Summary: Apply InfoLogLevel to the logs in db/db_impl.cc

Test Plan:
db_test
db_bench

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D28233
2014-11-04 10:28:08 -08:00
sdong
ac6afaf9ef Enforce naming convention of getters in version_set.h
Summary: Enforce the accessier naming convention in functions in version_set.h

Test Plan: make all check

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28143
2014-11-04 09:59:05 -08:00
sdong
09899f0b51 DB::Open() to automatically increase thread pool size if it is smaller than max number of parallel compactions or flushes
Summary:
With the patch, thread pool size will be automatically increased if DB's options ask for more parallelism of compactions or flushes.

Too many users have been confused by the API. Change it to make it harder for users to make mistakes

Test Plan: Add two unit tests to cover the function.

Reviewers: yhchiang, rven, igor, MarkCallaghan, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27555
2014-11-03 17:22:34 -08:00
Igor Canadi
636e57b52d Fix coverage script 2014-11-03 14:53:00 -08:00
Igor Canadi
30ca3752ba Revamp our build tools
Summary:
This diff is revamping our build tools:
1) Use third-party2 instead of third-party
2) consolidate clang and gcc scripts together, lots of duplication there
3) remove hdfs libs, we never compile rocksdb with them

clang compilation doesn't work yet. It doesn't work in master either. I plan to fix it soon, but I just spent 2 hours trying to make it work and failed. I'll ask experts.

Test Plan: compiles with gcc

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28047
2014-11-03 14:00:45 -08:00
Igor Canadi
051c67ff53 Merge pull request #378 from baotiao/master
add make clean in examples makefile
2014-11-03 11:27:50 -08:00
Igor Canadi
d0e7e49ae6 Merge pull request #379 from fyrz/RocksJavaBuildFix
[RocksJava] Build fix after options refactoring
2014-11-03 11:27:13 -08:00
fyrz
94e31ac227 [RocksJava] Extend Options with DBOptions implementation [RocksJava] Included DBOptionsTest and refactored OptionsTest
Summary: Options refactoring - Split Part2

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28017
2014-11-03 00:13:57 +01:00
fyrz
b060d30065 [RocksJava] Build fix after options refactoring 2014-11-02 23:48:40 +01:00
baotiao
9fd65e5667 add make clean in examples makefile 2014-11-03 04:42:28 +08:00
Igor Canadi
8e79ce68ce Revert "Fix lint errors and coding style of ldb related codes."
This reverts commit bc9f36fd5e.
2014-10-31 19:22:49 -07:00
Igor Canadi
45a612f990 Revert "Fix incorrect fixing of lint errors in ldb_cmd.cc"
This reverts commit 8ddddd62d0.
2014-10-31 19:22:42 -07:00
fyrz
27129c739f [RocksJava] KeyMayExist w/o ColumnFamilies 2014-11-01 01:02:12 +01:00
sdong
86905e3cbb Move VersionBuilder logic to a separate .cc file
Summary: Move all the logic of VersionBuilder to a separate .cc file

Test Plan: make all check

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28083
2014-10-31 16:34:38 -07:00
Igor Canadi
74eb4fbe93 CompactionJob
Summary:
Long awaited CompactionJob class! Move most compaction-related things from DBImpl to CompactionJob, making CompactionJob easier to test and understand.

Currently this is just replicating exactly the same functionality with as little as change as possible. As future work, we should:
1. Add CompactionJob tests (I think I'll do that tomorrow)
2. Reduce CompactionJob's state that it inherits from DBImpl
3. Figure out how to do yielding to flush better. Currently I implemented a callback as we agreed yesterday, but I don't think it's a good long term solution.

This reduces db_impl.cc from 5000+ LOC to 3400!

Test Plan: make check, will add CompactionJob-specific tests, probably also move some tests from db_test to compaction_job_test

Reviewers: rven, yhchiang, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27957
2014-10-31 16:31:25 -07:00
Yueh-Hsuan Chiang
8ddddd62d0 Fix incorrect fixing of lint errors in ldb_cmd.cc
Summary: Fix incorrect fixing of lint errors in ldb_cmd.cc

Test Plan: reduce_levels_test

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28137
2014-10-31 16:15:15 -07:00
Igor Canadi
46c14c6661 Fix #258. benchmarkharness -- make bm_min_usec uint 2014-10-31 15:41:56 -07:00
Lei Jin
72cb7cf201 Add fsync / corrupt simulation to env_mem
Summary: as title

Test Plan: env_mem_test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28077
2014-10-31 15:16:31 -07:00
Lei Jin
0e526eb9d7 introduce TestMemEnv and use it in db_test
Summary:
TestMemEnv simulates all Env APIs using in-memory data structures.
We can use it to speed up db_test run, which is now reduced ~7mins when it is
enabled.
We can also add features to simulate power/disk failures in the next
step
TestMemEnv is derived from helper/mem_env
mem_env can not be used for rocksdb since some of its APIs do not give
the same results as env_posix. And its file read/write is not thread safe

Test Plan:
make all -j32
./db_test
./env_mem_test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28035
2014-10-31 15:08:10 -07:00
Lei Jin
8db24f4b35 exclude mock test file from MOCK_SOURCES
Summary: as title

Test Plan: build with mock_env_test.cc

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28107
2014-10-31 15:07:27 -07:00
Lei Jin
5594d446ff unfriend DBImpl and InternalStats from VersionStorageInfo
Summary: as title

Test Plan: make release

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28053
2014-10-31 15:04:39 -07:00
Lei Jin
82e3ae5403 fix c_test
Summary: as title

Test Plan: ./c_test

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28119
2014-10-31 15:04:01 -07:00
Yueh-Hsuan Chiang
bc9f36fd5e Fix lint errors and coding style of ldb related codes.
Summary: Fix lint errors and coding style of ldb related codes.

Test Plan: ./ldb

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28125
2014-10-31 15:01:39 -07:00
sdong
c645250ee0 CompactionStats to support larger value of RecordIn and RecordDrop
Summary: now we use %8d for RecordIn and %10d for RecordDrop, which is far too small for some use cases. Extend both of them to %12d.

Test Plan: run one test in db_test and see the LOG file.

Reviewers: igor, MarkCallaghan, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28041
2014-10-31 13:51:48 -07:00
sdong
f7e6c856ab Fix BaseReferencedVersionBuilder's destructor order
Summary: BaseReferencedVersionBuilder now unreference version before destructing VersionBuilder, which is wrong. Fix it.

Test Plan:
make all check
valgrind test to tests that used to fail

Reviewers: igor, yhchiang, rven, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28101
2014-10-31 13:42:18 -07:00
Lei Jin
c76dcb44d4 fix
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2014-10-31 13:30:08 -07:00
Lei Jin
b452dede5c fix
Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
2014-10-31 12:23:00 -07:00
Lei Jin
29d83cc33a temporarily remove -Wshadow 2014-10-31 12:20:39 -07:00
Jonah Cohen
c1a924b9f0 Move convenience.h to /include
Summary: Move header file so it can be referenced externally.

Test Plan: Rebuild.

Reviewers: ljin

Reviewed By: ljin

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28095
2014-10-31 12:08:43 -07:00
Venkatesh Radhakrishnan
7e01d12026 Add support for in place update for db_stress
Summary:
Added two flags which operate as follows:
in_place_update: enable in_place_update for default column family
set_in_place_one_in: toggles the value of the option inplace_update_support with a probability of 1/N

Test Plan:
Run db_stress with the two flags above set.
Specifically tried in_place_update set to true and set_in_place_one_in set to 10,000.

Reviewers: ljin, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28029
2014-10-31 12:02:14 -07:00
Igor Canadi
9f7fc3ac45 Turn on -Wshadow
Summary:
...and fix all the errors :)

Jim suggested turning on -Wshadow because it helped him fix number of critical bugs in fbcode. I think it's a good idea to be -Wshadow clean.

Test Plan: compiles

Reviewers: yhchiang, rven, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27711
2014-10-31 11:59:54 -07:00
Yueh-Hsuan Chiang
98849a35fa Apply InfoLogLevel to the logs in table/block_based_table_reader.cc
Summary:
Apply InfoLogLevel to the logs in table/block_based_table_reader.cc

Also, add missing checks for the returned status in BlockBasedTable::Open

Test Plan: make

Reviewers: sdong, ljin, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28005
2014-10-31 11:41:15 -07:00
sdong
4d2ba38b65 Make VersionBuilder unit testable
Summary:
Rename Version::Builder to VersionBuilder and expose its definition to a header.
Make VerisonBuilder not reference Version or ColumnFamilyData, only working with VersionStorageInfo.
Add version_builder_test which has a simple test.

Test Plan: make all check

Reviewers: rven, yhchiang, igor, ljin

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D27969
2014-10-31 10:44:06 -07:00
Yueh-Hsuan Chiang
2b1f23dcae Apply InfoLogLevel to the logs in db/db_iter.cc
Summary: Apply InfoLogLevel to the logs in db/db_iter.cc

Test Plan: make

Reviewers: igor, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27861
2014-10-30 16:54:34 -07:00
Yueh-Hsuan Chiang
ccaf1aa7cb Merge pull request #372 from fyrz/RocksJava-CF-Merge-Hardening
[RocksJava] Merge with ColumnFamilies & Hardening CFHandle
2014-10-30 16:49:40 -07:00
Yueh-Hsuan Chiang
a29118ffc7 Merge pull request #355 from fyrz/RocksJava-Options-Refactoring-3.6
[RocksJava] Options Refactoring 3.6
2014-10-30 16:46:41 -07:00
fyrz
85b04ca765 [RocksJava] Review comments - reformatted MergeTest 2014-10-31 00:41:04 +01:00
fyrz
df7abb4e8d [RocksJava] Integrated code review comments
- Added TODO comments for disabled methods
2014-10-31 00:06:52 +01:00
fyrz
171be0ed55 Merge with ColumnFamilies & Hardening CFHandle
Summary:
ColumnFamilyHandles face the same problem as RocksIterator previously
so used methods were also applied for ColumnFamilyHandles.

Another problem with CF was that Options passed to CFs were
always filled with default values. To enable Merge, all parts
of the database must share the same merge functionality which
is not possible using default values. So from now on every
CF will inherit from db options.

Changes to RocksDB:
- merge can now take also a cfhandle

Changes to MergeTest:
- Corrected formatting
- Included also GC tests
- Extended tests to cover CF related parts
- Corrected paths to cleanup properly within the test process
- Reduced verbosity of the test

Test Plan:
make rocksdbjava
make jtest

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D27999
2014-10-30 22:32:04 +01:00
fyrz
39464a990f [RocksJava] Options Refactoring 3.6
Summary:
Options extends now two interfaces DBOptionsInterface
and ColumnFamilyOptionsInterface. There are also further
improvements to the Options bindings:

Optimize methods were ported to Java. (OptimizeForPointLookup,
OptimizeLevelCompaction, OptimizeUniversalCompaction).

To align BuiltinComparator with every other Enum it was moved to
a separate file.

Test Plan:
make rocksdbjava
make jtest
2014-10-30 22:26:39 +01:00
Yueh-Hsuan Chiang
0f7f3b8605 Check InfoLogLevel earlier in Log functions.
Summary: Check InfoLogLevel earlier in Log functions.

Test Plan: auto_roll_logger_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27993
2014-10-30 13:36:18 -07:00
Yueh-Hsuan Chiang
73605d9179 Apply InfoLogLevel to the logs in util/db_info_dumper.cc
Summary:
* Rename util/db_info_dummper.cc to util/db_info_dumper.cc
* Apply InfoLogLevel to the logs in util/db_info_dumper.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27915
2014-10-30 13:34:44 -07:00
Yueh-Hsuan Chiang
fda592d902 Merge pull request #356 from fyrz/RocksJava-TableOptions-3.6
[RocksJava] BlockBasedTableConfig & PlainTableConfig enhancements
2014-10-30 13:17:53 -07:00
fyrz
c73d13bb81 [RocksJava] Integrate review comments from yhchiang 2014-10-30 20:54:35 +01:00