Commit Graph

2928 Commits

Author SHA1 Message Date
Adam Retter
d904fbbb0b Addresed comments from code review https://reviews.facebook.net/D27567 2014-11-09 16:10:11 +00:00
Adam Retter
eeb9cf6c42 Test for WriteBatchHandler 2014-11-09 16:10:11 +00:00
Adam Retter
8fb4751d50 Iterator support for Write Batches 2014-11-09 16:10:11 +00:00
Igor Canadi
00211f9c5b Fix SIGSEGV in db_stresS 2014-11-08 13:01:31 -08:00
Igor Canadi
a4a2bfd6bd Merge pull request #391 from Liuchang0812/fixmake
remove unused target
2014-11-08 15:52:30 -05:00
Igor Canadi
856059094e Merge pull request #389 from Liuchang0812/master
fix "make static_lib" error
2014-11-08 15:52:18 -05:00
liuchang
01a7706379 remove unused target 2014-11-08 11:51:40 -08:00
liuchang
dc34104637 Merge branch 'fix-example' 2014-11-08 11:35:36 -08:00
liuchang
e7620536cf fix make static_lib error 2014-11-08 11:35:10 -08:00
Federico Piccinini
543df158c5 Expose sst_dump functionality as library call.
Summary:
Refactor sst_dump to follow the same structure as ldb. Introduce a
SSTDump interface.

Test Plan: built sst_dump and tried it manually.

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28485
2014-11-07 17:23:58 -08:00
Igor Canadi
e3d3567b5b Get rid of mutex in CompactionJob's state
Summary: Based on @sdong's feedback in the diff, we shouldn't keep db_mutex in CompactionJob's state. This diff removes db_mutex from CompactionJob state, by making next_file_number_ atomic. That way we only need to pass the lock to InstallCompactionResults() because of LogAndApply()

Test Plan: make check

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: sdong, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28491
2014-11-07 15:44:12 -08:00
Yueh-Hsuan Chiang
344edbb044 Fixed the shadowing in db/compaction.cc and include/rocksdb/db.h
Summary:
Fixed the shadowing in db/compaction.cc and include/rocksdb/db.h

Test Plan:
make
2014-11-07 15:22:10 -08:00
Yueh-Hsuan Chiang
b8b3903429 Fixed compile error in db/db_impl.cc
Summary:
Fixed compile error in db/db_impl.cc

Test Plan:
make
2014-11-07 15:13:01 -08:00
Yueh-Hsuan Chiang
b622ba5d6b Fixed compile error in db/flush_job.cc
Summary:
Fixed compile error in db/flush_job.cc

Test Plan:
make
2014-11-07 15:11:36 -08:00
Yueh-Hsuan Chiang
642ac9d8ab Fixed compile error in db/compaction.cc and db/compaction_picker.cc
Summary:
Fixed compile error in db/compaction.cc and db/compaction_picker.cc

Test Plan:
make
2014-11-07 15:08:12 -08:00
Igor Canadi
68effa0348 Fix -Wshadow for tools
Summary: Previously I made `make check` work with -Wshadow, but there are some tools that are not compiled using `make check`.

Test Plan: make all

Reviewers: yhchiang, rven, ljin, sdong

Reviewed By: ljin, sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28497
2014-11-07 15:04:30 -08:00
Yueh-Hsuan Chiang
8447861896 Fixed -WShadow errors in db/db_test.cc and include/rocksdb/metadata.h
Summary:
Fixed -WShadow errors in db/db_test.cc and include/rocksdb/metadata.h

Test Plan:
make
2014-11-07 14:57:51 -08:00
Yueh-Hsuan Chiang
28c82ff1b3 CompactFiles, EventListener and GetDatabaseMetaData
Summary:
This diff adds three sets of APIs to RocksDB.

= GetColumnFamilyMetaData =
* This APIs allow users to obtain the current state of a RocksDB instance on one column family.
* See GetColumnFamilyMetaData in include/rocksdb/db.h

= EventListener =
* A virtual class that allows users to implement a set of
  call-back functions which will be called when specific
  events of a RocksDB instance happens.
* To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners

= CompactFiles =
* CompactFiles API inputs a set of file numbers and an output level, and RocksDB
  will try to compact those files into the specified level.

= Example =
* Example code can be found in example/compact_files_example.cc, which implements
  a simple external compactor using EventListener, GetColumnFamilyMetaData, and
  CompactFiles API.

Test Plan:
listener_test
compactor_test
example/compact_files_example
export ROCKSDB_TESTS=CompactFiles
db_test
export ROCKSDB_TESTS=MetaData
db_test

Reviewers: ljin, igor, rven, sdong

Reviewed By: sdong

Subscribers: MarkCallaghan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 14:45:18 -08:00
Igor Canadi
5c93090530 Turn on -Wshadow
Summary:
Only one more try, I promise.

I talked to Jim and he mentioned that if we include our system includes with -isystem rather than with -I, that signals to the compile that those are system includes and thus no warnings are issued. So I turned our glibc includes into system includes and now we no longer get the warning from there, making us shadow-warning-free!

Test Plan: compiles with both clang and gcc

Reviewers: sdong, yhchiang, rven, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28479
2014-11-07 12:57:08 -08:00
Igor Canadi
31342c4005 Fix implicit compare 2014-11-07 12:41:05 -08:00
Igor Canadi
a0f887c9e4 Fix compile 2014-11-07 12:07:43 -08:00
Igor Canadi
53af5d877d Redesign pending_outputs_
Summary:
Here's a prototype of redesigning pending_outputs_. This way, we don't have to expose pending_outputs_ to other classes (CompactionJob, FlushJob, MemtableList). DBImpl takes care of it.

Still have to write some comments, but should be good enough to start the discussion.

Test Plan: make check, will also run stress test

Reviewers: ljin, sdong, rven, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28353
2014-11-07 11:50:34 -08:00
Jonah Cohen
ec101cd49a Correctly test both compaction styles in CompactionDeletionTriggerReopen
Summary:
CompactionDeletionTriggerReopen wasn't actually testing universal
compaction.

Test Plan: db_test

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28443
2014-11-06 17:28:49 -08:00
Yueh-Hsuan Chiang
8d87467bb0 Make PartialCompactionFailure Test more robust again.
Summary:
Make PartialCompactionFailure Test more robust again by
blocking background compaction until we simulate the
file creation error.

Test Plan:
export ROCKSDB_TESTS=PartialCompactionFailure
./db_test

Reviewers: sdong, igor, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28431
2014-11-06 17:07:52 -08:00
Lei Jin
64d302d304 make DropWritesFlush deterministic
Summary:
TEST_WaitForFlush should wait until it sees error when parameter is set
to true so we don't need to loop and timeout

Test Plan: ROCKSDB_TESTS=DropWritesFlush ./db_test

Reviewers: sdong, igor

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28419
2014-11-06 16:07:07 -08:00
Igor Canadi
cd5c0925ac Merge pull request #387 from fyrz/RocksJava-WShadow-Fix
[RocksJava] -WShadow improvements
2014-11-06 14:42:04 -08:00
fyrz
c4bf07c245 [RocksJava] -WShadow improvements
Minor corrections to resolve -WShadow build problems with RocksJava code.
2014-11-06 23:14:48 +01:00
Yueh-Hsuan Chiang
e526b71402 Make PartialCompactionFailure Test more robust.
Summary: Make PartialCompactionFailure Test more robust.

Test Plan:
export ROCKSDB_TESTS=PartialCompactionFailure
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28425
2014-11-06 13:53:02 -08:00
Igor Canadi
0c2be0de38 Turn on -Wshadow for travis 2014-11-06 13:29:17 -08:00
Igor Canadi
5fd33d26f1 Turn off -Wshadow
Summary:
So glibc is not -Wshadow-safe, so we need to turn it off :(

      error: ‘int sigaction(int, const sigaction*, sigaction*)’ hides
      constructor for ‘struct sigaction’

The rest of the changes in this diff is that we include .h files under rocksdb namespace, which is a no-no.

Test Plan: compiles now

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28413
2014-11-06 12:01:02 -08:00
Igor Canadi
9f20395cd6 Turn -Wshadow back on
Summary: It turns out that -Wshadow has different rules for gcc than clang. Previous commit fixed clang. This commits fixes the rest of the warnings for gcc.

Test Plan: compiles

Reviewers: ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28131
2014-11-06 11:14:28 -08:00
Lei Jin
c02338a698 update HISOTRY.md for new release
Summary:
as title

Test Plan:
n/a
2014-11-06 11:02:30 -08:00
sdong
367a3f9cb4 Improve DBTest.GroupCommitTest: artificially slowdown log writing to trigger group commit
Summary: In order to avoid random failure of DBTest.GroupCommitTest, artificially sleep 100 microseconds in each log writing.

Test Plan: Run the test in a machine where valgrind version of the test always fails multiple times and see it always succeed.

Reviewers: igor, yhchiang, rven, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28401
2014-11-06 10:48:06 -08:00
Igor Canadi
b52b144c73 Merge pull request #386 from EugenePig/java8
suppress JDK8 errors for #385
2014-11-06 10:46:20 -08:00
sdong
ac95ae1b5d Make sure WAL is synced for DB::Write() if write batch is empty
Summary: This patch makes it a contract that if an empty write batch is passed to DB::Write() and WriteOptions.sync = true, fsync is called to WAL.

Test Plan: A new unit test

Reviewers: ljin, rven, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, MarkCallaghan, leveldb

Differential Revision: https://reviews.facebook.net/D28365
2014-11-06 09:48:19 -08:00
Eugene Su
59d5497989 suppress JDK8 errors for #385 2014-11-06 16:25:53 +08:00
Shi Feng
ea18b944a7 Add db_bench option --report_file_operations
Summary: Add db_bench option --report_file_operations

Test Plan:
./db_bench --report_file_operations
Observe outputs on # of file operations

Reviewers: ljin, MarkCallaghan, sdong

Reviewed By: sdong

Subscribers: yhchiang, rven, igor, dhruba

Differential Revision: https://reviews.facebook.net/D27945
2014-11-05 18:40:18 -08:00
sdong
2ea1219eb6 Fix RecordIn and RecordDrop stats
Summary:
1. fix possible overflow of the two stats by using uint64_t
2. use a similar source of data to calculate RecordDrop. Previous one is not correct.

Test Plan: See outputs of db_bench settings, and the results look reasonable

Reviewers: MarkCallaghan, ljin, igor

Reviewed By: igor

Subscribers: rven, leveldb, yhchiang, dhruba

Differential Revision: https://reviews.facebook.net/D28155
2014-11-05 11:03:34 -08:00
Yueh-Hsuan Chiang
e4211d10c1 Apply InfoLogLevel to the logs in util/env_hdfs.cc
Summary: Apply InfoLogLevel to the logs in util/env_hdfs.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28011
2014-11-05 00:12:20 -08:00
maurice barnum
76f6c7c7c4 CompactionFilterV2: eliminate an often unnecessary allocation.
If a compaction filter implementation is simply filtering values, then
allocating the "changed values" bitmap is an extra memory allocation
that adds no value. Additionally, the compaction implementation has to
do marginally more work to calculate the offset into the bitmap
(vector<bool> specialization) for each record the filter did not mark
for deletion.

Explicitly handle the case where compact_->value_changed_buf_ is empty.
2014-11-05 05:31:11 +00:00
Lei Jin
29a9161f34 Note dynamic options in options.h
Summary: as title

Test Plan: n/a

Reviewers: igor, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28287
2014-11-04 16:23:45 -08:00
Lei Jin
fd24ae9d05 SetOptions() to return status and also add it to StackableDB
Summary: as title

Test Plan: ./db_test

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28269
2014-11-04 16:23:05 -08:00
Lei Jin
b1267750fb fix the asan check
Summary: as title

Test Plan: ran it

Reviewers: yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28311
2014-11-04 15:58:14 -08:00
sdong
83bf09144b Bump verison number to 3.7
Summary: As tittle

Test Plan: N/A

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28299
2014-11-04 14:52:02 -08:00
sdong
da5daa061c Replace some ASSERT_TRUE() asserts in DBTest.DynamicMemtableOptions and DynamicCompactionOptions with more specific ones
Summary: Replace some ASSERT_TRUE() to ASSERT_GT() and ASSERT_LT() so that in case the assert is triggered, the value is printed out.

Test Plan: Run the two tests

Reviewers: ljin, rven, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28293
2014-11-04 11:45:03 -08:00
sdong
b0cda4a116 DBTest.DynamicMemtableOptions to use single background compaction
Summary:
Now DBTest.DynamicMemtableOptions sets background compaction to be 4, without actually increasing thread pool size (even before the feature of automatic increasing it). To make sure the behavior stays the same after the automatic thread pool increasing, set it back to 1.

Hopefully it can fix the occasional failure of the test.

Test Plan: Run the test

Reviewers: igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28281
2014-11-04 11:14:12 -08:00
Yueh-Hsuan Chiang
8810850dd4 Apply InfoLogLevel to the logs in db/compaction_job.cc
Summary: Apply InfoLogLevel to the logs in db/compaction_job.cc

Test Plan: db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb, MarkCallaghan, dhruba

Differential Revision: https://reviews.facebook.net/D28275
2014-11-04 11:07:11 -08:00
Yueh-Hsuan Chiang
71783f6524 Merge pull request #377 from fyrz/RocksJava-KeyMayExist
[RocksJava] KeyMayExist w/o ColumnFamilies
2014-11-04 11:05:19 -08:00
Yueh-Hsuan Chiang
614bbcbe2c Merge pull request #374 from fyrz/RocksJava-DBOptions-Extension-3.6
[RocksJava] DBOptions extension 3.6
2014-11-04 10:51:59 -08:00
Yueh-Hsuan Chiang
d8e1196635 Apply InfoLogLevel to the logs in db/version_set.cc
Summary: Apply InfoLogLevel to the logs in db/version_set.cc

Test Plan: make

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D27879
2014-11-04 10:34:33 -08:00