Commit Graph

357 Commits

Author SHA1 Message Date
Andres Noetzli
6a82fba75f Add missing hashCode() implementation
Summary:
Whenever a Java class implements equals(), it has to implement hashCode(), otherwise
there might be weird behavior when inserting instances of the class in a hash map for
example. This adds two missing hashCode() implementations and extends tests to test
the hashCode() implementations.

Test Plan: make jtest

Reviewers: rven, igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: anthony, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43017
2015-07-24 17:07:19 -07:00
Siying Dong
3dbf4ba220 RangeSync not to sync last 1MB of the file
Summary:
From other ones' investigation:

"sync_file_range() behavior highly depends on kernel version and filesystem.

xfs does neighbor page flushing outside of the specified ranges. For example, sync_file_range(fd, 8192, 16384) does not only trigger flushing page #3 to #4, but also flushing many more dirty pages (i.e. up to page#16)... Ranges of the sync_file_range() should be far enough from write() offset (at least 1MB)."

Test Plan: make all check

Reviewers: igor, rven, kradhakrishnan, yhchiang, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: yoshinorim, MarkCallaghan, sumeet, domas, dhruba, leveldb, ljin

Differential Revision: https://reviews.facebook.net/D15807
2015-07-21 16:22:40 -07:00
Yueh-Hsuan Chiang
7219088cda Move general compaction tests from db_test.cc to db_compaction_test.cc
Summary: Move general compaction tests from db_test.cc to db_compaction_test.cc

Test Plan:
db_test
db_compaction_test

Reviewers: igor, sdong, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42651
2015-07-21 03:05:57 -07:00
Yueh-Hsuan Chiang
0adecd9f4f Add db_inplace_update_test back to Makefile
Summary: Add db_inplace_update_test back to Makefile

Test Plan: make all check

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42717
2015-07-20 18:17:11 -07:00
Yueh-Hsuan Chiang
9f1de95187 Revert Makefile
Summary: Revert Makefile

Test Plan: make clean

Reviewers: sdong, anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42711
2015-07-20 16:42:02 -07:00
Yueh-Hsuan Chiang
6867fb19c7 Revert "Add missing db_log_iter_test in the test list"
Summary:
This reverts commit 03467bdd4d.

It seems the previous git merge remove all the tabs in Makefile
and cause the Makefile not runnable.  This patch revert that incorrect
merge.

Test Plan: make

Reviewers: sdong, anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42705

Conflicts:
	CMakeLists.txt
	Makefile
2015-07-20 16:18:35 -07:00
Yueh-Hsuan Chiang
7462286d33 Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc
Summary: Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc

Test Plan:
db_test
db_inplace_update_test

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D42657
2015-07-20 16:05:28 -07:00
Yueh-Hsuan Chiang
03467bdd4d Add missing db_log_iter_test in the test list
Summary: Add missing db_log_iter_test in the test list

Test Plan: make all check

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D42663
2015-07-20 15:51:33 -07:00
Yueh-Hsuan Chiang
331954ab83 Fixed DBTestUniversalManualCompactionOutputPathId test
Summary:
Fixed DBTestUniversalManualCompactionOutputPathId test
by changing the expected number of files when setting up
the test as flushes no-longer preempt compactions
in patch https://reviews.facebook.net/D41931.

Also, include db_universal_copaction_test in make all check.

Test Plan: db_universal_copaction_test

Reviewers: igor, sdong, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42639
2015-07-20 13:53:41 -07:00
Igor Canadi
35ca59364c Don't let flushes preempt compactions
Summary:
When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.

We have a special case that when you set max_background_flushes to 0, we
schedule the flush to execute on the compaction thread.

Test Plan: make check (there might be some unit tests that depend on this behavior)

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D41931
2015-07-17 12:02:52 -07:00
Andres Notzli
1d20fa9d0f Fixed and simplified merge_helper
Summary:
MergeUntil was not reporting a success when merging an operand with
a Value/Deletion despite the comments in MergeHelper and CompactionJob
indicating otherwise. This lead to operands being written to the compaction
output unnecessarily:

M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M2 M3 M4 M5 (before the diff)
M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M4 M5 (after the diff)

In addition, the code handling Values/Deletion was basically identical.
This patch unifies the code. Finally, this patch also adds testing for
merge_helper.

Test Plan: make && make check

Reviewers: sdong, rven, yhchiang, tnovak, igor

Reviewed By: igor

Subscribers: tnovak, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42351
2015-07-17 09:27:24 -07:00
lovro
e1c99e10c1 Replace std::priority_queue in MergingIterator with custom heap, take 2
Summary: Repeat of b6655a679d (reverted in b7a2369fb2) with a proper fix for the issue that 57d216ea65 was trying to fix.

Test Plan:
make check

for i in $(seq 100); do ./db_stress --test_batches_snapshots=1 --threads=32 --write_buffer_size=4194304 --destroy_db_initially=0 --reopen=20 --readpercent=45 --prefixpercent=5 --writepercent=35 --delpercent=5 --iterpercent=10 --db=/tmp/rocksdb_crashtest_KdCI5F --max_key=100000000 --mmap_read=0 --block_size=16384 --cache_size=1048576 --open_files=500000 --verify_checksum=1 --sync=0 --progress_reports=0 --disable_wal=0 --disable_data_sync=1 --target_file_size_base=2097152 --target_file_size_multiplier=2 --max_write_buffer_number=3 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --filter_deletes=0 --memtablerep=prefix_hash --prefix_size=7 --ops_per_thread=200 || break; done

Reviewers: anthony, sdong, igor, yhchiang

Reviewed By: igor, yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D41391
2015-07-15 03:34:40 -07:00
Igor Canadi
1bc8eb8777 make coverage should execute sequentially
Summary:
For some reason, parallel tests are failing in Jenkins. We turned off parallel tests for all tests except `make coverage`. Let's turn it off here, too, so that we can have Jenkins test working again.

For the error log, see: https://ci-builds.fb.com/job/rocksdb_coverage/166/console

Test Plan: commit and wait for jenkins

Reviewers: meyering, sdong, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42057
2015-07-15 07:43:08 +02:00
Yueh-Hsuan Chiang
801df912af Move UniversalCompaction related db-tests to db_universal_compaction_test.cc
Summary: Move UniversalCompaction related db-tests to db_universal_compaction_test.cc

Test Plan:
db_test
db_universal_compaction_test

Reviewers: igor, sdong, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42225
2015-07-14 18:24:45 -07:00
Yueh-Hsuan Chiang
3ca6b2541e Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc
Summary: Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc

Test Plan:
db_test
db_test_tailing_iterator

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D42021
2015-07-14 16:41:08 -07:00
Yueh-Hsuan Chiang
ce829c77e3 Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc
Summary: Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc

Test Plan:
db_test
db_log_iter_test

Reviewers: sdong, IslamAbdelRahman, igor, anthony

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42045
2015-07-14 16:08:21 -07:00
Yueh-Hsuan Chiang
c3f98bb89b Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc
Summary: Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc

Test Plan:
db_test
db_compaction_filter_test

Reviewers: igor, sdong, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42207
2015-07-14 16:03:47 -07:00
Igor Canadi
1879d9370c Add ldb_test.py back to make check
Summary: This is revert of D40449. I'm not sure why our tests failed before, they seem to be working fine on my devbox.

Test Plan: ran `make check`

Reviewers: anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D41955
2015-07-13 21:46:41 -07:00
Yueh-Hsuan Chiang
b10cf4e2e0 Move DynamicLevel related db-tests to db_dynamic_level_test.cc
Summary: Move DynamicLevel related db-tests to db_dynamic_level_test.cc

Test Plan:
db_dynamic_level_test
db_test

Reviewers: igor, anthony, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42039
2015-07-13 19:00:30 -07:00
Yueh-Hsuan Chiang
625467a08a Move reusable part of db_test.cc to util/db_test_util.h
Summary:
Move reusable part of db_test.cc to util/db_test_util.h.
This makes it more possible to partition db_test.cc into
multiple smaller test files.

Also, fixed many old lint errors in db_test.

Test Plan: db_test

Reviewers: igor, anthony, IslamAbdelRahman, sdong, kradhakrishnan

Reviewed By: sdong, kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D41973
2015-07-13 16:53:38 -07:00
Yueh-Hsuan Chiang
0b1ffe2e1d Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS
Summary:
Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS as
those environment doe not have compile option --no-as-needed

  ld: unknown option: --no-as-needed
  clang: error: linker command failed with exit code 1 (use -v to see invocation)

Test Plan: make shared_lib

Reviewers: meyering, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40353
2015-06-23 16:32:59 -07:00
Michael Callahan
3bdec09cb7 Remove ldb_tests.py from make check until it is working again.
Summary: Recent checkin added ldb_test.py to the make check target but the test fails.  Remove it again for now and make task.

Test Plan: No more ldb_tests.py running

Reviewers: igor, anthony

Reviewed By: anthony

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D40449
2015-06-19 17:41:49 -07:00
Michael Callahan
15325bf55b First version of rocksdb_dump and rocksdb_undump.
Summary: Hack up rocksdb_dump and rocksdb_undump utilities to get this task rolling/promote discussion.

Test Plan: Dump/undump databases recursively to see if nothing is lost.

Reviewers: sdong, yhchiang, rven, anthony, kradhakrishnan, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37269
2015-06-19 16:24:36 -07:00
Venkatesh Radhakrishnan
04251e1e3a Add wal files to Checkpoint for multiple column families.
Summary:
When there are multiple column families, the flush in
GetLiveFiles is not atomic, so that there are entries in the wal files
which are needed to get a consisten RocksDB. We now add the log files to
the checkpoint.

Test Plan:
CheckpointCF - This test forces more data to be written to
the other column families after the flush of the first column family but
before the second.

Reviewers: igor, yhchiang, IslamAbdelRahman, anthony, kradhakrishnan, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40323
2015-06-19 16:08:31 -07:00
Yueh-Hsuan Chiang
71b438c4a6 Remove unused target --- compactor_test
Summary:
Remove compactor_test, which depends on a directory not exist
in our code base.
    make compactor_test
    GEN      util/build_version.cc
    GEN      util/build_version.cc
    make: *** No rule to make target `utilities/compaction/compactor_test.o', needed by `compactor_test'.  Stop.

Test Plan: verify the output message of make compactor_test

Reviewers: rven, anthony, kradhakrishnan, igor, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40341
2015-06-18 15:54:52 -07:00
Igor Canadi
d03f110904 Link all libraries when building shared libraries
Summary:
The motivation is here: http://stackoverflow.com/questions/30706348/rocksdb-make-error

Apparently there is a bug (?) in g++ where we don't link rt library properly when building RocksDB's shared library. It looks like this works when we add an option -Wl, --no-as-needed.

Test Plan: make shared_lib

Reviewers: meyering, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D39681
2015-06-10 12:04:21 -07:00
Yueh-Hsuan Chiang
fe5c6321cb Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
Summary:
Allow EventListener::OnCompactionCompleted to return CompactionJobStats,
which contains useful information about a compaction.

Example CompactionJobStats returned by OnCompactionCompleted():
    smallest_output_key_prefix 05000000
    largest_output_key_prefix 06990000
    elapsed_time 42419
    num_input_records 300
    num_input_files 3
    num_input_files_at_output_level 2
    num_output_records 200
    num_output_files 1
    actual_bytes_input 167200
    actual_bytes_output 110688
    total_input_raw_key_bytes 5400
    total_input_raw_value_bytes 300000
    num_records_replaced 100
    is_manual_compaction 1

Test Plan: Developed a mega test in db_test which covers 20 variables in CompactionJobStats.

Reviewers: rven, igor, anthony, sdong

Reviewed By: sdong

Subscribers: tnovak, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38463
2015-06-02 17:07:16 -07:00
agiardullo
dc9d70de65 Optimistic Transactions
Summary: Optimistic transactions supporting begin/commit/rollback semantics.  Currently relies on checking the memtable to determine if there are any collisions at commit time.  Not yet implemented would be a way of enuring the memtable has some minimum amount of history so that we won't fail to commit when the memtable is empty.  You should probably start with transaction.h to get an overview of what is currently supported.

Test Plan: Added a new test, but still need to look into stress testing.

Reviewers: yhchiang, igor, rven, sdong

Reviewed By: sdong

Subscribers: adamretter, MarkCallaghan, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D33435
2015-05-29 14:36:35 -07:00
Igor Canadi
a3da590226 Decrease number of jobs in make release
Summary: as title

Test Plan: make release

Reviewers: MarkCallaghan, sdong

Reviewed By: sdong

Subscribers: sdong, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38853
2015-05-29 13:39:33 -04:00
Igor Canadi
309a9d0760 Run tests sequentally if J=1
Summary: Sometimes we want to run tests sequentially. J=1 gives us that option

Test Plan:
make J=1 check -- sequential
make J=2 check -- parallel

Reviewers: sdong, yhchiang, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38805
2015-05-22 09:11:29 -07:00
DerekSchenk
3e08175418 Add LDFLAGS to Java static library
Includes the LDFLAGS so that the correct libraries will be linked.  This links rt to resolve the issue https://github.com/facebook/rocksdb/issues/606.
2015-05-19 23:04:02 -04:00
Igor Canadi
08b6b3796e FORCE_GIT_SHA
Summary: In third-party2 build we need to force git sha because we're compiling from a different git repositry.

Test Plan: `FORCE_GIT_SHA=igor make`

Reviewers: kradhakrishnan, sdong

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38679
2015-05-19 11:45:01 -07:00
sdong
8c52788f0c Use version defined in Makefile in rocksdb_build_git_sha
Summary: Now rocksdb_build_git_sha is determined from "git sha". It is hard if the release is not from the repository directly but from a source code copy. Change to use the versions given in Makefile.

Test Plan: Run "make util/build_version.cc"

Reviewers: kradhakrishnan, rven, meyering, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D38451
2015-05-15 13:51:57 -07:00
Yueh-Hsuan Chiang
2fe24e4b5f Don't treat warnings as error when building rocksdbjavastatic
Summary: Don't treat warnings as error when building rocksdbjavastatic

Test Plan: make rocksdbjavastatic -j32

Reviewers: rven, fyrz, adamretter, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38187
2015-05-08 11:02:47 -07:00
Igor Canadi
beda81c538 rm -rf in make clean
Summary: Without this I get bunch of questions when I run `make clean`

Test Plan: no more questions!

Reviewers: rven, yhchiang, meyering, anthony

Reviewed By: meyering, anthony

Subscribers: meyering, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38145
2015-05-07 17:47:19 -07:00
Igor Canadi
a2c4cc7562 Don't treat warnings as error when building release
Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't.

Test Plan:
make dbg:
g++  -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native   -isystem ./third-party/gtest-1.7.0/fused-src  -Woverloaded-virtual -Wn
on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o

make all:
g++  -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native   -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o

make static_lib:
g++  -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native   -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o

Reviewers: sdong, yhchiang, rven

Reviewed By: rven

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 11:54:13 -07:00
Igor Canadi
d85d08c7b3 One last fix to Makefile
Summary: Based on comment from D37455

Test Plan: make install after make static_lib

Reviewers: meyering

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D37461
2015-04-20 20:46:08 -07:00
Igor Canadi
2db96dca18 Fix make install when there is no shared lib
Summary: make install fails when there is no shared lib. We need to revert the conditions, which will have the same effect, but without the failure

Test Plan: make install after only compiling static library

Reviewers: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37455
2015-04-20 20:39:54 -07:00
Igor Canadi
79c1b021a7 Fix Makefile
Summary: The current version tries to install librocksdb.so even though it doesn't exist. This version will install librocksdb.so.3.10.0 and then create soft links in place

Test Plan:
`make static_lib; sudo make install` does not try to install librocksdb.so
`make shared_lib; sudo make install` installs one library and 3 symlinks. Before, four libraries were installed

Reviewers: sdong, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D37425
2015-04-20 19:39:51 -07:00
Venkatesh Radhakrishnan
0a0501c8d5 Add Xfunc to makefile
Summary: Make target for running all xfunc tests

Test Plan: make xfunc

Reviewers: igor, sdong, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36873
2015-04-13 14:21:32 -07:00
Igor Canadi
abb4052278 Kill benchharness
Summary:
1. it doesn't work
2. we're not using it

In the future, if we need general benchmark framework, we should probably use https://github.com/google/benchmark

Test Plan: make all

Reviewers: yhchiang, rven, anthony, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36777
2015-04-13 10:17:42 -07:00
agiardullo
fabc115690 MemTableList tests
Summary: Add tests for MemTableList

Test Plan: run test

Reviewers: yhchiang, kradhakrishnan, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36735
2015-04-09 18:01:11 -07:00
sdong
465b25ca93 "make commit-prereq" doesn't really build ROCKSDB_LITE
Summary: "make commit-prereq" uses "make release" which overrides OPT, so ROCKSDB_LITE is not covered. Fix it by using "make static_lib"

Test Plan: Run it and see it fail (which is expected)

Reviewers: yhchiang, meyering, rven, anthony, kradhakrishnan, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D36813
2015-04-09 17:04:00 -07:00
Igor Canadi
91df4e969d Remove use of whole-archive to include jemalloc
Summary: I don't think we need to use whole-archive to include jemalloc. This change only affects our development builds -- it does not affect our open source builds (which don't support jemalloc) or our fbcode third-party2 builds (which use open-source build codepaths).

Test Plan:
make
verify that jemalloc is running by running `MALLOC_CONF="prof:true" ./cache_test` and observing that file was created

Reviewers: MarkCallaghan

Reviewed By: MarkCallaghan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36783
2015-04-09 15:10:53 -07:00
Igor Canadi
edbb08b5df Fix Makefile
Summary: These two files are test binaries and are not included in TESTS in Makefile.

Test Plan: `make clean` now deletes those files, too

Reviewers: sdong, kradhakrishnan, meyering

Reviewed By: kradhakrishnan, meyering

Subscribers: kradhakrishnan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36705
2015-04-08 14:33:07 -07:00
Jim Meyering
199313dc38 build: create .o files specifically for java-related targets
Summary:
When building rocksdbjava and rocksdbjavastatic, create -fPIC-enabled
binaries in a temporary subdirectory, jl/.
* Makefile (java_libobjects): New variable.
(java_libobjects): New rule.
(CLEAN_FILES): Arrange for "make clean" to remove that temporary dir.
(rocksdbjavastatic): Depend on the new variable.
Remove useless OPT=... line.
(rocksdbjava): Likewise.

Test Plan:
  JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \
    make rocksdbjavastatic

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36645
2015-04-08 10:26:06 -07:00
Jim Meyering
cba5920011 build: don't use a glob for java/rocksjni/*
Summary:
* src.mk (JNI_NATIVE_SOURCES): New variable, so we don't have to use
a glob in Makefile
* Makefile (JNI_NATIVE_SOURCES): Remove glob-using definition, now
that the explicit list of sources is in src.mk.

Test Plan:
  Run this:
    JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \
      make rocksdbjava

Reviewers: yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D36633
2015-04-07 15:19:25 -07:00
Igor Canadi
c66483c132 Fix github issue #563
Summary:
As described in https://github.com/facebook/rocksdb/issues/563, we should add minor version to SONAME, since we break ABI with minor releases.

I also turned PLATFORM_SHARED_VERSIONED to true by default. This is true in LevelDB and it was switched to false by D15117 for no apparent reason. It should only be false for iOS.

Test Plan: `make shared_lib` produced librocksdb.dylib.3.10.0

Reviewers: sdong, yhchiang, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36573
2015-04-07 13:22:22 -07:00
Jim Meyering
e3ee98b38a run 'make check's rules (and even subtests) in parallel
Summary:
When GNU parallel is available, "make check" tests are now run in parallel.
When /dev/shm is usable, we tell those tests to create temporary files therein.
Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests)
is no longer run sequentially: instead, each of its sub-tests is run independently, and can
be parallelized along with all other tests. To make that process easier, this change
creates a temporary directory, "t/", in which it puts a small script for each of those
subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME.

When GNU parallel is not available, we run the tests in sequence, just as before.
If GNU parallel is available and you don't like the default of running one subtest
per core, you can invoke "make J=1 check" to run only one test at a time.
Beware: this will take a long time, and it starts with the two longest-running tests, so you
will wait for a long time before seeing any results. Instead, if you want to use fewer resources
but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of
the cores are occupied by test runs.

To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log"
in the same directory from another window.  That will start with something like this:

and when complete should show numbers/names like this:

  Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=...  Wed Apr  1 10:51:42 2015

  152.221 PASS t/DBTest.FileCreationRandomFailure
  109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest
   82.315 PASS reduce_levels_test
   77.812 PASS t/DBTest.CompactionFilterWithValueChange
   73.236 PASS backupable_db_test
   63.428 PASS deletefile_test
   57.248 PASS table_test
   55.665 PASS prefix_test
   49.816 PASS t/DBTest.RateLimitingTest
  ...

Test Plan:
Timings (measured so as to exclude compile and link times):
With this change, all tests complete in 2m40s on a system for which nproc prints 32.
Prior to this this change, "make check" would take 24.5 minutes on that same system.

Here are durations (in seconds) of the longest-running subtests:

152.435 PASS t/DBTest.FileCreationRandomFailure
107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest
 81.391 PASS ./reduce_levels_test
 71.587 PASS ./backupable_db_test
 61.746 PASS ./deletefile_test
 57.960 PASS ./table_test
 55.230 PASS ./prefix_test
 54.060 PASS t/DBTest.CompactionFilterWithValueChange
 48.873 PASS t/DBTest.RateLimitingTest
 47.569 PASS ./fault_injection_test
 46.593 PASS t/DBTest.Randomized
 42.662 PASS t/DBTest.CompactionFilter
 31.793 PASS t/DBTest.SparseMerge
 30.612 PASS t/DBTest.CompactionFilterV2
 25.891 PASS t/DBTest.GroupCommitTest
 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase
 22.976 PASS ./rate_limiter_test
 18.942 PASS t/DBTest.OptimizeFiltersForHits
 16.851 PASS ./env_test
 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange
 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix

Reviewers: igor, sdong, rven, yhchiang, igor.sugak

Reviewed By: igor.sugak

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 12:35:25 -07:00
Igor Canadi
2511b7d947 Makefile minor cleanup
Summary:
Just couple of small changes:
1. removed signal_test, since it doesn't seem useful and we don't even run it as part of `make check`
2. moved perf_context_test to TESTS instead of PROGRAMS
3. `make release` probably shouldn't compile benchmarks. We currently rely on `make release` building db_bench (via Jenkins), so I left db_bench there.

This is just a minor cleanup. We need to rethink our targets since they are a bit messy right now. We can do this during our tech debt week.

Test Plan: make release

Reviewers: anthony, rven, yhchiang, sdong, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D36171
2015-03-30 16:05:35 -04:00