Commit Graph

4875 Commits

Author SHA1 Message Date
Andrew Kryczka
d733dd5728 [build] Fix env_win.cc compiler errors
Summary: I broke it in D53781.

Test Plan: tried the same code in util/env_posix.cc and it compiled successfully

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54303
2016-02-17 11:57:04 -08:00
Jonathan Wiepert
cf38e56f2f Fix broken appveyor build caused by D53991
Test Plan:
pushed remote as D53991_appveyor_test branch. Appveyor run:
https://ci.appveyor.com/project/Facebook/rocksdb/build/1.0.651
shows the file is now found and the run is back to the state before the initial change:
https://ci.appveyor.com/project/Facebook/rocksdb/build/1.0.620

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: igor, dhruba

Differential Revision: https://reviews.facebook.net/D54309
2016-02-17 11:49:52 -08:00
Igor Canadi
351252b497 Merge pull request #998 from fengjian0106/master
fix ios build error
2016-02-17 09:06:07 -08:00
fengjian
1336052497 fix ios build error 2016-02-17 20:22:40 +08:00
agiardullo
d08d50295c Fix transaction locking
Summary: Broke transaction locking in 4.4 in D52197.  Will cherry-pick this change into 4.4 (which hasn't yet been fully released).  Repro'd using db_bench.

Test Plan: unit tests and db_Bench

Reviewers: sdong, yhchiang, kradhakrishnan, ngbronson

Reviewed By: ngbronson

Subscribers: ngbronson, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54021
2016-02-16 17:15:05 -08:00
Yueh-Hsuan Chiang
730a422c3a Improve the documentation of LoadLatestOptions
Summary: Improve the documentation of LoadLatestOptions

Test Plan: No code change

Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54087
2016-02-16 14:55:24 -08:00
reid horuff
a7b6f0748a Improve write_with_callback_test to sync WAL
Summary: Currently write_with_callback_test does not test with WAL syncing enabled. This addresses that.

Test Plan: write_with_callback_test

Reviewers: anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba, hermanlee4

Differential Revision: https://reviews.facebook.net/D54255
2016-02-16 14:04:14 -08:00
reid horuff
5bcf952a87 Fix WriteImpl empty batch hanging issue
Summary: There is an issue in DBImpl::WriteImpl where if an empty writebatch comes in and sync=true then the logs will be marked as being synced yet the sync never actually happens because there is no data in the writebatch. This causes the next incoming batch to hang while waiting for the logs to complete syncing. This fix syncs logs even if the writebatch is empty.

Test Plan: DoubleEmptyBatch unit test in transaction_test.

Reviewers: yoshinorim, hermanlee4, sdong, ngbronson, anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D54057
2016-02-16 12:21:33 -08:00
Jonathan Wiepert
871cc5f987 fix build without gflags
Test Plan:
Built and ran with gflags:
% ./db_bench
LevelDB:    version 4.5
Date:       Tue Feb 16 12:04:23 2016
CPU:        40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
...

And without gflags:
% ./db_bench
Please install gflags to run rocksdb tools
%

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: igor, dhruba

Differential Revision: https://reviews.facebook.net/D54243
2016-02-16 12:16:47 -08:00
Igor Canadi
c90d63a23d can_unlock set but not used
Test Plan: I couldn't repro, but I hope this fixes it. See the error here: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_ubuntu1404_rocksdb_compile_6e9fd902d5cb25aef992363efa128640affd5196_16_02_11_04_33_37/0?type=T

Reviewers: yhchiang, andrewkr, sdong, anthony

Reviewed By: anthony

Subscribers: meyering, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54123
2016-02-16 11:24:40 -08:00
Mike Kolupaev
44371501f0 Fixed a segfault when compaction fails
Summary: We've hit it today.

Test Plan: `make -j check`; didn't reproduce the issue

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D54219
2016-02-16 11:11:16 -08:00
Igor Canadi
2f084d39b9 Merge pull request #992 from jofusa/jdennison/options-typo-fix
fixes typo in options logging
2016-02-16 10:26:25 -08:00
Jonathan Wiepert
7bd284c374 Separeate main from bench functionality to allow cusomizations
Summary: Isolate db_bench functionality from main so custom benchmark code can be written and managed

Test Plan:
Tested commands
./build_tools/regression_build_test.sh
./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000
./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --reads=500 --writes=500
./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --merge_keys=100 --numdistinct=100 --num_column_families=3 --num_hot_column_families=1
./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5
./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5
./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5
./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5

Test Results - https://phabricator.fb.com/P56130387

Additional tests for:
./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 --key_size=8 --merge_operator=put
./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 --merge_operator=uint64add

Results: https://phabricator.fb.com/P56130607

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53991
2016-02-16 06:17:31 -08:00
John Dennison
bd5f842bba fixes typo in options logging 2016-02-15 23:55:29 +00:00
Andrew Kryczka
1c868d6848 Fix includes for env_test
Summary:
Previously compilation failed when ROCKSDB_NO_FBCODE=1 because fcntl.h
wasn't included for open().

Related issue: https://github.com/facebook/rocksdb/issues/977

Test Plan:
verified below command works now:

  $ make clean && ROCKSDB_NO_FBCODE=1 ROCKSDB_DISABLE_FALLOCATE=1 make -j32 env_test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54135
2016-02-12 14:17:14 -08:00
krad
545a193952 Add J to commit_prereq so comilation/execution happens in parallel
Summary:
Users are confused on how to get the parallel compilation going. This
can help wire the parallelism.

Test Plan: Run manually

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53931
2016-02-12 12:09:07 -08:00
Andrew Kryczka
5bb7371ca6 [build] Evaluate test names only when db_test exists
Summary:
as titled, this will prevent the error that was printed because
test_names was evaluated before db_test was built.

Test Plan:
verified below command works and no longer prints errors:

  $ make release -j32

verified below command still finds the right tests:

  $ make J=32 parallel_check

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54117
2016-02-12 10:49:35 -08:00
Andrew Kryczka
6a2b4fcb80 Add flag to forcibly disable fallocate
Summary:
see https://github.com/facebook/rocksdb/issues/977; there are issues
with fallocate() on certain filesystems/kernel versions that can lead it to pre-
allocating blocks but never freeing them, even if they're unused.

Test Plan:
verified build commands omit DROCKSDB_FALLOCATE_PRESENT when this env
variable is set.

without disabling it:

  $ ROCKSDB_NO_FBCODE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  0

with disabling it:

  $ ROCKSDB_NO_FBCODE=1 DISABLE_FALLOCATE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $?
  1

Reviewers: kradhakrishnan, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54069
2016-02-11 17:00:01 -08:00
sdong
92a9ccf1a6 Add a new compaction priority that picks file whose overlapping ratio is smallest
Summary:
Add a new compaction priority as following:
For every file, we calculate total size of files overalapping with the file in the next level, over the file's size itself. The file with smallest ratio will be picked first.
My "db_bench --fillrandom" shows about 5% less compaction than kOldestSmallestSeqFirst if --hard_pending_compaction_bytes_limit value to keep LSM tree in shape. If not limiting hard_pending_compaction_bytes_limit, improvement is only 1% or 2%.

Test Plan: Add a unit test

Reviewers: andrewkr, kradhakrishnan, anthony, IslamAbdelRahman, yhchiang

Reviewed By: yhchiang

Subscribers: MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D54075
2016-02-11 15:59:19 -08:00
Igor Canadi
3dc3d1c144 Merge pull request #984 from petermattis/pmattis/comparator-iterate-upper-bound
Use user_comparator when comparing against iterate_upper_bound.
2016-02-11 10:22:40 -08:00
Peter Mattis
239aaf2fc0 Use user_comparator when comparing against iterate_upper_bound.
Fixes #983.
2016-02-11 08:47:16 -05:00
Yueh-Hsuan Chiang
908100399c Fixed a dependency issue of ThreadLocalPtr
Summary:
When a child thread that uses ThreadLocalPtr, ThreadLocalPtr::OnThreadExit
will be called when that child thread is destroyed.  However,
OnThreadExit will try to access a static singleton of ThreadLocalPtr,
which will be destroyed when the main thread exit.  As a result,
when a child thread that uses ThreadLocalPtr exits AFTER the main thread
exits, illegal memory access will occur.

This diff includes a test that reproduce this legacy bug.

    ==2095206==ERROR: AddressSanitizer: heap-use-after-free on address
    0x608000007fa0 at pc 0x959b79 bp 0x7f5fa7426b60 sp 0x7f5fa7426b58
    READ of size 8 at 0x608000007fa0 thread T1

This patch fix this issue by having the thread local mutex never be deleted
(but will leak small piece of memory at the end.)   The patch also describe
a better solution (thread_local) in the comment that requires gcc 4.8.1 and
in latest clang as a future work once we agree to move toward gcc 4.8.

Test Plan:
COMPILE_WITH_ASAN=1 make thread_local_test -j32
./thread_local_test --gtest_filter="*MainThreadDiesFirst"

Reviewers: anthony, hermanlee4, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53013
2016-02-10 16:56:01 -08:00
Gunnar Kudrjavets
337671b688 Add universal compaction benchmarks to run_flash_bench.sh
Summary:
Implement a benchmark for universal compaction based on the feature description (see below), in-person discussions, and reading source code:

https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
https://github.com/facebook/rocksdb/wiki/Universal-Compaction
https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#universal-compaction

Universal compaction benchmark is based on `overwrite` benchmark, adding compaction specific options to it, and executing it for different values of subcompaction to understand the impact of scaling out subcompactions for a particular scenario.

Test Plan:
  - Execute the benchmark on various machines for multiple iterations to verify the reliability.
  - Observe the output to make sure that compaction is taking place.
  - Observe the execution to make sure that arguments passed to `db_bench` are correct.

Reviewers: sdong, MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54045
2016-02-10 15:30:47 -08:00
Yueh-Hsuan Chiang
3a67bffaa8 Fix an ASAN error in transaction_test.cc
Summary:
One test in transaction_test.cc forgets to call SyncPoint::DisableProcessing().
As a result, a program might to access the SyncPoint singleton after it
already goes out of scope.

This patch fix this error by calling SyncPoint::DisableProcessing().

Test Plan: transaction_test

Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54033
2016-02-10 12:06:59 -08:00
Adam Retter
c5af85ecad Fix a memory leak of Slice objects from org.rocksdb.WBWIRocksIterator#entry1 2016-02-10 14:44:39 +00:00
Adam Retter
e84137c8ae Remove unnessecary java.util.List expense in JNI 2016-02-10 14:44:39 +00:00
Adam Retter
76e8beeeb9 Pass by pointer from/to Java from JNI not by object 2016-02-10 14:44:39 +00:00
Adam Retter
0e7e6f6e4b Improve Javadoc 2016-02-10 14:44:39 +00:00
Adam Retter
18eb563058 Improve the speed and synchronization around the construction of Java/JNI objects 2016-02-10 14:44:39 +00:00
Yueh-Hsuan Chiang
2a04268be3 Temporarily disable unstable tests in memory_test.cc
Summary:
memory_test.cc has some tests that are not unstable but
hard to reproduce, and the cause is the test itself not
the code.  Temporarily disable the tests until
we have a good fix.

Test Plan: memory_test

Reviewers: sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D54009
2016-02-09 16:28:34 -08:00
Igor Canadi
08a78b6b44 Merge pull request #979 from facebook/update_licenses
Updated all copyright headers to the new format.
2016-02-09 15:46:18 -08:00
Baraa Hamodi
21e95811d1 Updated all copyright headers to the new format. 2016-02-09 15:12:00 -08:00
Andrew Kryczka
59b3ee658f Env function for bulk metadata retrieval
Summary:
Added this new function, which returns filename, size, and modified
timestamp for each file in the provided directory. The default implementation
retrieves the metadata sequentially using existing functions. In the next diff
I'll make HdfsEnv override this function to use libhdfs's bulk get function.

This won't work on windows due to the path separator.

Test Plan:
new unit test

  $ ./env_test --gtest_filter=EnvPosixTest.ConsistentChildrenMetadata

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: IslamAbdelRahman, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53781
2016-02-09 14:54:32 -08:00
Yueh-Hsuan Chiang
4a8cbf4e31 Allows Get and MultiGet to read directly from SST files.
Summary:
Add kSstFileTier to ReadTier, which allows Get and MultiGet to
read only directly from SST files and skip mem-tables.

    kSstFileTier = 0x2      // data in SST files.
                          // Note that this ReadTier currently only supports
                          // Get and MultiGet and does not support iterators.

Test Plan: add new test in db_test.

Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: igor, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53511
2016-02-09 11:20:22 -08:00
agiardullo
fe93bf9b5d Transaction::UndoGetForUpdate
Summary: MyRocks wants to be able to un-lock a key that was just locked by GetForUpdate().  To do this safely, I am now keeping track of the number of reads(for update) and writes for each key in a transaction.  UndoGetForUpdate() will only unlock a key if it hasn't been written and the read count reaches 0.

Test Plan: more unit tests

Reviewers: igor, rven, yhchiang, spetrunia, sdong

Reviewed By: spetrunia, sdong

Subscribers: spetrunia, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47043
2016-02-09 10:46:11 -08:00
sdong
2608219cc9 crash_test: cover concurrent memtable insert in default crash test
Summary: Default crash test uses prefix hash memtable, which is not compatible to concurrent memtable. Allow prefix test run with skip list and use skip list memtable when concurrent insert is used.

Test Plan: Run "python -u tools/db_crashtest.py whitebox" and watch sometimes skip list is used.

Reviewers: anthony, yhchiang, kradhakrishnan, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53907
2016-02-09 08:02:38 -08:00
sdong
a76e9093f0 Fix LITE db_test build broken by previous commit
Summary: Previous commit introduces a test that is not supported in LITE. Fix it.

Test Plan: Build the test with ROCKSDB_LITE.

Reviewers: kradhakrishnan, IslamAbdelRahman, anthony, yhchiang, andrewkr

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53901
2016-02-05 14:29:09 -08:00
sdong
b1887c5dd9 Explictly fail when memtable doesn't support concurrent insert
Summary: If users turn on concurrent insert but the memtable doesn't support it, they might see unexcepted crash. Fix it by explicitly fail.

Test Plan:
Run different setting of stress_test and make sure it fails correctly.
Will add a unit test too.

Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, andrewkr, ngbronson

Reviewed By: ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53895
2016-02-05 14:15:50 -08:00
Gunnar Kudrjavets
8ed3438778 Add option to run fillseq with WAL enabled in addition to WAL disabled
Summary: This set of changes is part of the work to introduce benchmark for universal style compaction in RocksDB. It's conceptually separate from the compaction work, so sending it out as a separate diff to get it out of the way.

Test Plan:
  - Run `./tools/run_flash_bench.sh`.
  - Look at the contents of `report.txt` and `report2.txt` to make sure that data is reported and attributed correctly.
  - During `db_bench` execution time make sure that the correct flags are passed to `--disable_wal` depending on the benchmark being executed.

Reviewers: MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53865
2016-02-05 13:20:56 -08:00
sdong
73a9b0f4ba Update version to 4.5
Summary: Time to cut branch for release 4.5. Change the versions.

Test Plan: Not needed

Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, andrewkr, anthony

Reviewed By: anthony

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53883
2016-02-05 13:05:54 -08:00
reid horuff
6f71d3b68b Improve perf of Pessimistic Transaction expirations (and optimistic transactions)
Summary:
copy from task 8196669:

1) Optimistic transactions do not support batching writes from different threads.
2) Pessimistic transactions do not support batching writes if an expiration time is set.

In these 2 cases, we currently do not do any write batching in DBImpl::WriteImpl() because there is a WriteCallback that could decide at the last minute to abort the write.  But we could support batching write operations with callbacks if we make sure to process the callbacks correctly.

To do this, we would first need to modify write_thread.cc to stop preventing writes with callbacks from being batched together.  Then we would need to change DBImpl::WriteImpl() to call all WriteCallback's in a batch, only write the batches that succeed, and correctly set the state of each batch's WriteThread::Writer.

Test Plan: Added test WriteWithCallbackTest to write_callback_test.cc which creates multiple client threads and verifies that writes are batched and executed properly.

Reviewers: hermanlee4, anthony, ngbronson

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D52863
2016-02-05 10:44:13 -08:00
Islam AbdelRahman
8e6172bc57 Add BlockBasedTableOptions::index_block_restart_interval
Summary: Add a new option to BlockBasedTableOptions that will allow us to change the restart interval for the index block

Test Plan: unit tests

Reviewers: yhchiang, anthony, andrewkr, sdong

Reviewed By: sdong

Subscribers: march, dhruba

Differential Revision: https://reviews.facebook.net/D53721
2016-02-05 10:22:37 -08:00
sdong
34a40bf911 Add --allow_concurrent_memtable_write in stress test and run it in crash_test
Summary: Add an option of --allow_concurrent_memtable_write in stress test and cover it in crash test

Test Plan: Run crash test and make sure three combinations of the two options show up randomly.

Reviewers: IslamAbdelRahman, yhchiang, andrewkr, anthony, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53811
2016-02-04 16:15:18 -08:00
Siying Dong
73bf330c79 Merge pull request #973 from yuslepukhin/moveout_testcode
Move out of release build some test code
2016-02-04 15:50:21 -08:00
sdong
f7c0f4e3ef perf_context.cc and iostats_context.cc use different output macro (fix unity build)
Summary: Unity build because perf_context.cc and iostats_context.cc define a different OUTPUT macro. Fix it.

Test Plan: Build unity

Reviewers: kradhakrishnan, anthony, yhchiang, andrewkr, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: divchenko, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D53769
2016-02-03 17:27:08 -08:00
Dmitri Smirnov
9656eab005 This partially addresses issue https://github.com/facebook/rocksdb/issues/935
testutil.cc and testharness.cc could not be moved out at this time
  as they are used by 4 benchmarks in release builds.
2016-02-03 17:06:10 -08:00
Jonathan Wiepert
14a322033f Remove references to files deleted in commit abb4052278
Summary:
Remove obolete references to files in src.mk
Fix incorrect path for reference in source.mk

Test Plan: Ran build to ensure changes do not break anything.

Reviewers: leveldb, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D53733
2016-02-03 16:47:45 -08:00
krad
8445e53807 Add a mechanism to run all tests in sandcastle
Summary:
When making environment specific changes, it is better to run all CI
tests. This diff provides a mechanism to do that

Format is:

ROCKSDB_CHECK_ALL=1 arc diff

Test Plan: Submit request for diff

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D53631
2016-02-03 16:34:08 -08:00
Igor Canadi
461cec4e8e Merge pull request #972 from adamretter/wb-threads
Threaded tests for WriteBatch
2016-02-03 12:32:46 -08:00
Adam Retter
9ab269ab39 Threaded tests for WriteBatch 2016-02-03 20:15:49 +00:00