Commit Graph

4291 Commits

Author SHA1 Message Date
Yueh-Hsuan Chiang
3fdb6e5234 Fixed old lint errors in db/filename.cc
Summary: Fixed old lint errors in db/filename.cc

Test Plan: make

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47445
2015-09-23 12:39:16 -07:00
Yueh-Hsuan Chiang
b349d22786 Fixed old lint errors in db/filename.h
Summary: Fixed old lint errors in db/filename.h

Test Plan: make

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47439
2015-09-23 12:22:44 -07:00
sdong
df34aea331 PlainTableReader to support non-mmap mode
Summary:
PlainTableReader now only allows mmap-mode. Add the support to non-mmap mode for more flexibility.
Refactor the codes to move all logic of reading data to PlainTableKeyDecoder, and consolidate the calls to Read() call and ReadVarint32() call. Implement the calls for both of mmap and non-mmap case seperately. For non-mmap mode, make copy of keys in several places when we need to move the buffer after reading the keys.

Test Plan: Add the mode of non-mmap case in plain_table_db_test. Run it in valgrind mode too.

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D47187
2015-09-23 11:41:07 -07:00
sdong
d746eaad5e RandomAccessFileReader should not inherit RandomAccessFile
Summary: RandomAccessFileReader unnecessarily inherited RandomAccessFile, which can introduce unnecessarily extra costs. Remove it.

Test Plan: Run all existing tests

Reviewers: yhchiang, anthony, igor, kradhakrishnan, rven, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D47409
2015-09-23 11:00:41 -07:00
krad
03dd8f3ca7 Fixing punit job description
Summary: The job errors with log print step. Removing the step and re-enabling
the test case.

Test Plan: Manual test run

Reviewers:

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-23 07:35:29 -07:00
sdong
d0c31641d2 Internal stats WAL file synced to match meaning of the stats of the same name
Summary: https://reviews.facebook.net/D23343 changed WAL sync bytes to extra fsync. This change does the same for internal stats.

Test Plan: Run all existing unit tests and verify results in db_bench.

Reviewers: anthony, rven, igor, MarkCallaghan, kradhakrishnan, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D47349
2015-09-22 14:23:11 -07:00
Siying Dong
48b4497f75 Merge pull request #730 from yuslepukhin/fix_write_batch_win_const_expr
Fix Windows constexpr issue and '#ifdef' column_family_test in Release.
2015-09-22 11:08:10 -07:00
krad
a6c22e3e41 Disabling parallel test CI job
Summary: Parallel test CI is failing. Reducing noise by temporarily disabling
the job

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-22 10:37:56 -07:00
Dmitri Smirnov
489a3e95d4 Re-work to support size_t max constant for 32/64-bit. 2015-09-22 10:34:21 -07:00
Yueh-Hsuan Chiang
ff57c6511d [RocksJava] Fix test failure of InfoLogLevelTest
Summary:
In patch https://reviews.facebook.net/D47067, we change the
log level of the initial database information to header level.
As a result, even when the InfoLogLevel is set to Fatal, the
LOG file of a newly opened rocksdb instance will not be empty.
However, the current InfoLogLevelTest expect it should be empty.

This patch fixes this issue by enabling InfoLogLevelTest to
ignore the Log header.

Test Plan: make jtest

Reviewers: fyrz, anthony, IslamAbdelRahman, sdong, adamretter

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47229
2015-09-21 18:26:21 -07:00
sdong
f1b9f804e9 Add a mode to always pick the oldest file to compact for each level
Summary:
Add options.compaction_pri, which specifies the policy about which file to compact first.
kCompactionPriByLargestSeq will compact oldest files first.
Verified the behavior in db_bench but did not write unit tests yet. Also need to make it settable through option string and dynamically changeable.

Test Plan: Will write unit tests

Reviewers: igor, rven, anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, MarkCallaghan

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D45951
2015-09-21 17:21:59 -07:00
Dmitri Smirnov
5e8f0a66db Use port::constant for std::muneric_limtis<>::max() 2015-09-21 16:56:47 -07:00
Dmitri Smirnov
2754ec9994 Fix Windows constexpr issue and '#ifdef' column_family_test in Release. 2015-09-21 16:21:01 -07:00
krad
dd2e1eeb35 Disabling log running jobs
Summary: Long running jobs are blocking the queue due to lack of adequate
machines in the CI

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-21 12:58:33 -07:00
Igor Canadi
199744f4c4 Merge pull request #728 from jsteemann/fix-missing-include-header
add missing header required for std::function
2015-09-21 19:50:00 +02:00
Igor Canadi
0e65693f17 Merge pull request #727 from jsteemann/micro-optimization
Micro optimizations
2015-09-21 19:37:18 +02:00
krad
4d6eb52d1a Fix to CI jobs.
Summary: The approach to serialize long scripts as string is flawed. Instead
adding args to invoke the long scripts.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-19 17:33:06 -07:00
krad
3bcc072d24 Added more CI jobs
Summary: Migrating the long script jobs and the left overs from Jenkin.

Test Plan: Run command manually

Reviewers: sdon igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-09-18 13:24:09 -07:00
jsteemann
669b892f97 add missing header required for std::function
otherwise Visual Studio will have trouble compiling this file
2015-09-18 22:18:40 +02:00
jsteemann
624ef456dd fixed formatting. thanks @4tXJ7f for pointing me at make format 2015-09-18 22:03:47 +02:00
jsteemann
bbb18c8278 removed unused variable of type Status, fixed indentation 2015-09-18 20:23:50 +02:00
jsteemann
4704833357 pass input string to WriteBatch() by const reference
this may lead to copying less data (in case compilers don't
optimize away copying the string by themselves)
2015-09-18 20:20:32 +02:00
jsteemann
5ec129971b key_ cannot become nullptr, so no check is needed for that
(ignoring the unlikely case that some overrides
`operator new throw(std::bad_alloc)` with a function that returns a nullptr)
2015-09-18 20:15:20 +02:00
jsteemann
834b12a8d5 made Size() function const because it does not modify data 2015-09-18 20:10:00 +02:00
Andres Noetzli
1fc16cb2eb Fix clang-format on Travis
Summary:
Commit 1b598213aa does not quite work as
intended. This patch fixes Travis by adding clang-format to the packages list
and excluding clang-format from the OSX build. In addition the build does not
stop when bad formatting is detected but the build still fails.

Test Plan: See https://travis-ci.org/facebook/rocksdb/builds/81037125

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47223
2015-09-18 10:15:02 -07:00
Igor Canadi
e244bdf392 Merge pull request #725 from adamretter/fail-failed-java-test
Exit with non-zero status if one or more Java tests fail
2015-09-18 07:54:50 -07:00
Adam Retter
7d937a0901 Exit with non-zero status if one or more Java tests fail 2015-09-18 15:26:01 +01:00
Andres Noetzli
1b598213aa Check formatting in Travis
Summary:
Check code formatting before running tests. Goal is to make it easier to deal
with external pull requests.

Test Plan: See https://travis-ci.org/facebook/rocksdb/builds/80952963

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47211
2015-09-17 22:02:05 -07:00
agiardullo
c7fba80291 Fix non-deterministic failure in backupable_db_test
Summary: FailOverwritingBackups has unexpected results when auto-compaction runs.

Test Plan: ran test a bunch of times

Reviewers: IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47181
2015-09-17 20:14:51 -07:00
Andres Noetzli
014fd55adc Support for SingleDelete()
Summary:
This patch fixes #7460559. It introduces SingleDelete as a new database
operation. This operation can be used to delete keys that were never
overwritten (no put following another put of the same key). If an overwritten
key is single deleted the behavior is undefined. Single deletion of a
non-existent key has no effect but multiple consecutive single deletions are
not allowed (see limitations).

In contrast to the conventional Delete() operation, the deletion entry is
removed along with the value when the two are lined up in a compaction. Note:
The semantics are similar to @igor's prototype that allowed to have this
behavior on the granularity of a column family (
https://reviews.facebook.net/D42093 ). This new patch, however, is more
aggressive when it comes to removing tombstones: It removes the SingleDelete
together with the value whenever there is no snapshot between them while the
older patch only did this when the sequence number of the deletion was older
than the earliest snapshot.

Most of the complex additions are in the Compaction Iterator, all other changes
should be relatively straightforward. The patch also includes basic support for
single deletions in db_stress and db_bench.

Limitations:
- Not compatible with cuckoo hash tables
- Single deletions cannot be used in combination with merges and normal
  deletions on the same key (other keys are not affected by this)
- Consecutive single deletions are currently not allowed (and older version of
  this patch supported this so it could be resurrected if needed)

Test Plan: make all check

Reviewers: yhchiang, sdong, rven, anthony, yoshinorim, igor

Reviewed By: igor

Subscribers: maykov, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43179
2015-09-17 11:42:56 -07:00
Igor Canadi
f35560d00d Merge pull request #723 from jsteemann/fix-typos
fixed typos in comments of options.h
2015-09-17 11:26:11 -07:00
jsteemann
f8b770a942 fixed typos 2015-09-17 19:58:09 +02:00
Venkatesh Radhakrishnan
51e1c11254 Do not flag error if file to be deleted does not exist
Summary:
Some users have observed errors in the log file when
the log file or sst file is already deleted.

Test Plan:
Make sure that the errors do not appear for already deleted
files.

Reviewers: sdong

Reviewed By: sdong

Subscribers: anthony, kradhakrishnan, yhchiang, rven, igor, IslamAbdelRahman, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47115
2015-09-17 10:21:34 -07:00
Mayank Pundir
a5e312a7a4 Improving condition for bottommost level during compaction
Summary: The diff modifies the condition checked to determine the bottommost level during compaction. Previously, absence of files in higher levels alone was used as the condition. Now, the function additionally evaluates if the higher levels have files which have non-overlapping key ranges, then the level can be safely considered as the bottommost level.

Test Plan: Unit test cases added and passing. However, unit tests of universal compaction are failing as a result of the changes made in this diff. Need to understand why that is happening.

Reviewers: igor

Subscribers: dhruba, sdong, lgalanis, meyering

Differential Revision: https://reviews.facebook.net/D46473
2015-09-16 17:47:50 -07:00
sdong
9aca7cd6d8 DB::Open() to flush info log after printing DB pointer
Summary: Now DB::Open() flushes info log before printing DB pointer, so it may not show up if no activity after DB open. Move log flushing from after printing options to printing DB pointer.

Test Plan: make commit-prereq

Reviewers: igor, IslamAbdelRahman, yhchiang, kradhakrishnan, anthony, rven

Reviewed By: rven

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D47121
2015-09-16 16:33:39 -07:00
Andres Noetzli
cecd903ad9 Fix gflags build in Travis script
Summary:
There were two minor issues with 9566342d28:
- The install command was doing `cmake` instead of `cmake .`
- `LIBRARY_PATH` was not set
Fix both issues.

Test Plan: Run tests on Travis: https://travis-ci.org/facebook/rocksdb/jobs/80699627 (bloom_test works now).

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47103
2015-09-16 13:12:52 -07:00
Andres Noetzli
16934d4959 Fix wrong constants in db_test_util
Summary: Fix two constants in WaitFor() that multiply a value with 000 instead of 1000.

Test Plan: make clean all check

Reviewers: rven, anthony, yhchiang, aekmekji, sdong, MarkCallaghan, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47091
2015-09-16 13:04:05 -07:00
Yueh-Hsuan Chiang
2e8e7eb391 Fix the verbosity issue in Java makefile
Summary:
When running `make rocksdbjava V=1`, it currently does
not print out the original command but something odd instead:

    mkdir -p jl/db

This patch tries to fix this issue.

Test Plan: make rocksdbjava V=1

Reviewers: igor, sdong, IslamAbdelRahman, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47061
2015-09-16 12:38:17 -07:00
Yueh-Hsuan Chiang
f21c7415a7 Change the log level of DB start-up log from Warn to Header.
Summary: Change the log level of DB start-up log from Warn to Header.

Test Plan: db_bench and observe the LOG header

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47067
2015-09-16 11:31:45 -07:00
Andres Noetzli
9566342d28 Build gflags from source for Travis
Summary:
Unfortunately it is not easily possible to install gflags as a package on
Travis. Change the Travis script to download and build gflags. This patch
avoids "sudo make install" so that we can continue to use Travis'
container-based infrastructure.

Test Plan: Run tests on Travis.

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47037
2015-09-16 11:10:24 -07:00
Alexey Maykov
3ebf11ed16 Adding the increment for a counter for a number of WAL syncs
Summary: This will unblock the corresponding change in MyRocks

Test Plan: ran rocksdb.write_sync test

Reviewers: sdong, kolmike

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D46911
2015-09-16 11:00:49 -07:00
Igor Canadi
81a61d75dc Skipped tests shouldn't be failures [part 2]
Summary: Missed one file in the previous commit

Test Plan: compiles

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47055
2015-09-15 22:59:53 -07:00
Igor Canadi
1b7ea8ce81 Skipped tests shouldn't be failures
Summary: If we skip a test, we shouldn't mark `make check` as failure. This fixes travis CI test.

Test Plan: Travis CI

Reviewers: noetzli, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47031
2015-09-15 18:10:36 -07:00
Ari Ekmekji
5ba3297d0d Add compaction time to log output
Summary:
Although compaction time is recorded in the statistics,
it is helpful to include this value in the log output corresponding
to the end of compaction.

Test Plan: make all && make check

Reviewers: yhchiang, sdong, igor, noetzli, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D47007
2015-09-15 17:11:44 -07:00
Dmytro Okhonko
31a27a3606 Callback for informing backup downloading added
Summary:
In case of huge db backup infromation about progress of downloading would help.
New callback parameter in CreateNewBackup() function will trigger whenever a some amount of data downloaded.
Task: 8057631

Test Plan:
ProgressCallbackDuringBackup test that cover new functionality added to BackupableDBTest tests.
other test succeed as well.

Reviewers: Guenena, benj, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46575
2015-09-15 17:08:30 -07:00
Igor Canadi
d93a9f2aaf [travis CI] Run ulimit -n 2000
Summary: This should let us pass tests on OS X.

Test Plan: none yet

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46989
2015-09-15 12:12:24 -07:00
Ari Ekmekji
2b683d4972 Add DBOption.max_subcompaction to option dump
Summary:
RocksDB options can be dumped to the log file, and
up to this point the max_subcompactions option was not included
in this dump. This fixes that.

Test Plan: makek all && make check

Reviewers: MarkCallaghan, igor, noetzli, anthony, yhchiang, sdong

Reviewed By: yhchiang, sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D46971
2015-09-15 11:41:46 -07:00
Igor Canadi
0e50a3fcc0 Merge issue with D46773
Summary: There was a merge issue with SleepingBackgroundTask

Test Plan: compiles now

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46977
2015-09-15 11:35:23 -07:00
Igor Canadi
a7e80379b0 LogAndApply() should fail if the column family has been dropped
Summary:
This patch finally fixes the ColumnFamilyTest.ReadDroppedColumnFamily test. The test has been failing very sporadically and it was hard to repro. However, I managed to write a new tests that reproes the failure deterministically.

Here's what happens:
1. We start the flush for the column family
2. We check if the column family was dropped here: a3fc49bfdd/db/flush_job.cc (L149)
3. This check goes through, ends up in InstallMemtableFlushResults() and it goes into LogAndApply()
4. At about this time, we start dropping the column family. Dropping the column family process gets to LogAndApply() at about the same time as LogAndApply() from flush process
5. Drop column family goes through LogAndApply() first, marking the column family as dropped.
6. Flush process gets woken up and gets a chance to write to the MANIFEST. However, this is where it gets stuck: a3fc49bfdd/db/version_set.cc (L1975)
7. We see that the column family was dropped, so there is no need to write to the MANIFEST. We return OK.
8. Flush gets OK back from LogAndApply() and it deletes the memtable, thinking that the data is now safely persisted to sst file.

The fix is pretty simple. Instead of OK, we return ShutdownInProgress. This is not really true, but we have been using this status code to also mean "this operation was canceled because the column family has been dropped".

The fix is only one LOC. All other code is related to tests. I added a new test that reproes the failure. I also moved SleepingBackgroundTask to util/testutil.h (because I needed it in column_family_test for my new test). There's plenty of other places where we reimplement SleepingBackgroundTask, but I'll address that in a separate commit.

Test Plan:
1. new test
2. make check
3. Make sure the ColumnFamilyTest.ReadDroppedColumnFamily doesn't fail on Travis: https://travis-ci.org/facebook/rocksdb/jobs/79952386

Reviewers: yhchiang, anthony, IslamAbdelRahman, kradhakrishnan, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D46773
2015-09-15 11:28:44 -07:00
krad
2819a1db32 Minor fix to CI job definition
Summary: Timeout needs to be set for command and job for it to work.

Test Plan: Manual

Reviewers: sdong igor

CC: levledb@

Task ID: #6968635

Blame Rev:
2015-09-15 10:56:48 -07:00