rocksdb/Makefile

1762 lines
58 KiB
Makefile
Raw Normal View History

# Copyright (c) 2011 The LevelDB Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file. See the AUTHORS file for names of contributors.
# Inherit some settings from environment variables, if available
#-----------------------------------------------
BASH_EXISTS := $(shell which bash)
SHELL := $(shell which bash)
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
CLEAN_FILES = # deliberately empty, so we can append below.
2014-11-26 15:48:03 +01:00
CFLAGS += ${EXTRA_CFLAGS}
CXXFLAGS += ${EXTRA_CXXFLAGS}
LDFLAGS += $(EXTRA_LDFLAGS)
MACHINE ?= $(shell uname -m)
ARFLAGS = ${EXTRA_ARFLAGS} rs
STRIPFLAGS = -S -x
2014-11-26 15:48:03 +01:00
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# Transform parallel LOG output into something more readable.
perl_command = perl -n \
-e '@a=split("\t",$$_,-1); $$t=$$a[8];' \
-e '$$t =~ /.*if\s\[\[\s"(.*?\.[\w\/]+)/ and $$t=$$1;' \
-e '$$t =~ s,^\./,,;' \
-e '$$t =~ s, >.*,,; chomp $$t;' \
-e '$$t =~ /.*--gtest_filter=(.*?\.[\w\/]+)/ and $$t=$$1;' \
-e 'printf "%7.3f %s %s\n", $$a[3], $$a[6] == 0 ? "PASS" : "FAIL", $$t'
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
quoted_perl_command = $(subst ','\'',$(perl_command))
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
# DEBUG_LEVEL can have three values:
# * DEBUG_LEVEL=2; this is the ultimate debug mode. It will compile rocksdb
# without any optimizations. To compile with level 2, issue `make dbg`
# * DEBUG_LEVEL=1; debug level 1 enables all assertions and debug code, but
# compiles rocksdb with -O2 optimizations. this is the default debug level.
# `make all` or `make <binary_target>` compile RocksDB with debug level 1.
# We use this debug level when developing RocksDB.
# * DEBUG_LEVEL=0; this is the debug level we use for release. If you're
# running rocksdb in production you most definitely want to compile RocksDB
# with debug level 0. To compile with level 0, run `make shared_lib`,
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
# `make install-shared`, `make static_lib`, `make install-static` or
# `make install`
# Set the default DEBUG_LEVEL to 1
DEBUG_LEVEL?=1
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
ifeq ($(MAKECMDGOALS),dbg)
DEBUG_LEVEL=2
endif
ifeq ($(MAKECMDGOALS),clean)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),release)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),shared_lib)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),install-shared)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),static_lib)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),install-static)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),install)
DEBUG_LEVEL=0
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
endif
ifeq ($(MAKECMDGOALS),rocksdbjavastatic)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),rocksdbjavastaticrelease)
DEBUG_LEVEL=0
endif
ifeq ($(MAKECMDGOALS),rocksdbjavastaticpublish)
DEBUG_LEVEL=0
endif
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
# compile with -O2 if debug level is not 2
ifneq ($(DEBUG_LEVEL), 2)
OPT += -O2 -fno-omit-frame-pointer
# Skip for archs that don't support -momit-leaf-frame-pointer
ifeq (,$(shell $(CXX) -fsyntax-only -momit-leaf-frame-pointer -xc /dev/null 2>&1))
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
OPT += -momit-leaf-frame-pointer
endif
endif
# if we're compiling for release, compile without debug code (-DNDEBUG) and
# don't treat warnings as errors
ifeq ($(DEBUG_LEVEL),0)
OPT += -DNDEBUG
Don't treat warnings as error when building release Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
2015-05-05 20:54:13 +02:00
DISABLE_WARNING_AS_ERROR=1
ifneq ($(USE_RTTI), 1)
CXXFLAGS += -fno-rtti
else
CXXFLAGS += -DROCKSDB_USE_RTTI
endif
else
ifneq ($(USE_RTTI), 0)
CXXFLAGS += -DROCKSDB_USE_RTTI
else
CXXFLAGS += -fno-rtti
endif
$(warning Warning: Compiling in debug mode. Don't use the resulting binary in production)
endif
#-----------------------------------------------
build: fix missing dependency problems Summary: Any time one would modify a dependent of any *test*.cc file, "make" would fail to rebuild the affected test binaries, e.g., db_test. That was due to the fact that we deliberately excluded those test-related files from the definition of SOURCES and only $(SOURCES) was used to create the automatically-generated .d dependency files. The fix is to generate a .d file for every source file. * src.mk: New file. Defines LIB_SOURCES, MOCK_SOURCES and TEST_BENCH_SOURCES. * Makefile: Include src.mk. Reflect s/SOURCES/LIB_SOURCES/ renaming. * build_tools/build_detect_platform: Remove the code that was used to generate SOURCES= and MOCK_SOURCES= definitions in make_config.mk. Those lists of files are now hard-coded in src.mk. Hard-coding this list of sources is desirable, because without that, one risks including stray .cc files in a build. Not reproducible. Test Plan: Touch a file used by db_test's dependent .o files and ensure that they are all recompiled. Before, none would be: $ touch db/db_impl.h && make db_test CC db/db_test.o CC db/column_family.o CC db/db_filesnapshot.o CC db/db_impl.o CC db/db_impl_debug.o CC db/db_impl_readonly.o CC db/forward_iterator.o CC db/internal_stats.o CC db/managed_iterator.o CC db/repair.o CC db/write_batch.o CC utilities/compacted_db/compacted_db_impl.o CC utilities/ttl/db_ttl_impl.o CC util/ldb_cmd.o CC util/ldb_tool.o CC util/sst_dump_tool.o CC util/xfunc.o CCLD db_test Reviewers: ljin, igor.sugak, igor, rven, sdong Reviewed By: sdong Subscribers: yhchiang, adamretter, fyrz, dhruba Differential Revision: https://reviews.facebook.net/D33849
2015-02-21 02:42:16 +01:00
include src.mk
AM_DEFAULT_VERBOSITY = 0
AM_V_GEN = $(am__v_GEN_$(V))
am__v_GEN_ = $(am__v_GEN_$(AM_DEFAULT_VERBOSITY))
am__v_GEN_0 = @echo " GEN " $@;
am__v_GEN_1 =
AM_V_at = $(am__v_at_$(V))
am__v_at_ = $(am__v_at_$(AM_DEFAULT_VERBOSITY))
am__v_at_0 = @
am__v_at_1 =
AM_V_CC = $(am__v_CC_$(V))
am__v_CC_ = $(am__v_CC_$(AM_DEFAULT_VERBOSITY))
am__v_CC_0 = @echo " CC " $@;
am__v_CC_1 =
CCLD = $(CC)
LINK = $(CCLD) $(AM_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) $(LDFLAGS) -o $@
AM_V_CCLD = $(am__v_CCLD_$(V))
am__v_CCLD_ = $(am__v_CCLD_$(AM_DEFAULT_VERBOSITY))
am__v_CCLD_0 = @echo " CCLD " $@;
am__v_CCLD_1 =
AM_V_AR = $(am__v_AR_$(V))
am__v_AR_ = $(am__v_AR_$(AM_DEFAULT_VERBOSITY))
am__v_AR_0 = @echo " AR " $@;
am__v_AR_1 =
Add EnvLibrados - RocksDB Env of RADOS (#1222) EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/".
2016-07-21 20:16:34 +02:00
ifdef ROCKSDB_USE_LIBRADOS
LIB_SOURCES += utilities/env_librados.cc
LDFLAGS += -lrados
endif
Add EnvLibrados - RocksDB Env of RADOS (#1222) EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/".
2016-07-21 20:16:34 +02:00
AM_LINK = $(AM_V_CCLD)$(CXX) $^ $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
# detect what platform we're building on
dummy := $(shell (export ROCKSDB_ROOT="$(CURDIR)"; export PORTABLE="$(PORTABLE)"; "$(CURDIR)/build_tools/build_detect_platform" "$(CURDIR)/make_config.mk"))
# this file is generated by the previous line to set build flags and sources
include make_config.mk
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
CLEAN_FILES += make_config.mk
missing_make_config_paths := $(shell \
grep "\/\S*" -o $(CURDIR)/make_config.mk | \
while read path; \
do [ -e $$path ] || echo $$path; \
done | sort | uniq)
$(foreach path, $(missing_make_config_paths), \
$(warning Warning: $(path) dont exist))
ifeq ($(PLATFORM), OS_AIX)
# no debug info
else ifneq ($(PLATFORM), IOS)
CFLAGS += -g
CXXFLAGS += -g
else
# no debug info for IOS, that will make our library big
OPT += -DNDEBUG
endif
ifeq ($(PLATFORM), OS_AIX)
ARFLAGS = -X64 rs
STRIPFLAGS = -X64 -x
endif
ifeq ($(PLATFORM), OS_SOLARIS)
PLATFORM_CXXFLAGS += -D _GLIBCXX_USE_C99
endif
ifneq ($(filter -DROCKSDB_LITE,$(OPT)),)
# found
CFLAGS += -fno-exceptions
CXXFLAGS += -fno-exceptions
# LUA is not supported under ROCKSDB_LITE
LUA_PATH =
endif
# ASAN doesn't work well with jemalloc. If we're compiling with ASAN, we should use regular malloc.
ifdef COMPILE_WITH_ASAN
DISABLE_JEMALLOC=1
EXEC_LDFLAGS += -fsanitize=address
PLATFORM_CCFLAGS += -fsanitize=address
PLATFORM_CXXFLAGS += -fsanitize=address
endif
# TSAN doesn't work well with jemalloc. If we're compiling with TSAN, we should use regular malloc.
ifdef COMPILE_WITH_TSAN
DISABLE_JEMALLOC=1
EXEC_LDFLAGS += -fsanitize=thread
PLATFORM_CCFLAGS += -fsanitize=thread -fPIC
PLATFORM_CXXFLAGS += -fsanitize=thread -fPIC
# Turn off -pg when enabling TSAN testing, because that induces
# a link failure. TODO: find the root cause
PROFILING_FLAGS =
# LUA is not supported under TSAN
LUA_PATH =
endif
# AIX doesn't work with -pg
ifeq ($(PLATFORM), OS_AIX)
PROFILING_FLAGS =
endif
# USAN doesn't work well with jemalloc. If we're compiling with USAN, we should use regular malloc.
ifdef COMPILE_WITH_UBSAN
DISABLE_JEMALLOC=1
EXEC_LDFLAGS += -fsanitize=undefined
PLATFORM_CCFLAGS += -fsanitize=undefined -DROCKSDB_UBSAN_RUN
PLATFORM_CXXFLAGS += -fsanitize=undefined -DROCKSDB_UBSAN_RUN
endif
ifndef DISABLE_JEMALLOC
ifdef JEMALLOC
PLATFORM_CXXFLAGS += -DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE
PLATFORM_CCFLAGS += -DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE
endif
EXEC_LDFLAGS := $(JEMALLOC_LIB) $(EXEC_LDFLAGS)
PLATFORM_CXXFLAGS += $(JEMALLOC_INCLUDE)
PLATFORM_CCFLAGS += $(JEMALLOC_INCLUDE)
endif
export GTEST_THROW_ON_FAILURE=1
export GTEST_HAS_EXCEPTIONS=1
GTEST_DIR = ./third-party/gtest-1.7.0/fused-src
# AIX: pre-defined system headers are surrounded by an extern "C" block
ifeq ($(PLATFORM), OS_AIX)
PLATFORM_CCFLAGS += -I$(GTEST_DIR)
PLATFORM_CXXFLAGS += -I$(GTEST_DIR)
else
PLATFORM_CCFLAGS += -isystem $(GTEST_DIR)
PLATFORM_CXXFLAGS += -isystem $(GTEST_DIR)
endif
# This (the first rule) must depend on "all".
default: all
WARNING_FLAGS = -W -Wextra -Wall -Wsign-compare -Wshadow \
-Wno-unused-parameter
ifndef DISABLE_WARNING_AS_ERROR
WARNING_FLAGS += -Werror
endif
ifdef LUA_PATH
ifndef LUA_INCLUDE
LUA_INCLUDE=$(LUA_PATH)/include
endif
LUA_INCLUDE_FILE=$(LUA_INCLUDE)/lualib.h
ifeq ("$(wildcard $(LUA_INCLUDE_FILE))", "")
# LUA_INCLUDE_FILE does not exist
$(error Cannot find lualib.h under $(LUA_INCLUDE). Try to specify both LUA_PATH and LUA_INCLUDE manually)
endif
LUA_FLAGS = -I$(LUA_INCLUDE) -DLUA -DLUA_COMPAT_ALL
CFLAGS += $(LUA_FLAGS)
CXXFLAGS += $(LUA_FLAGS)
ifndef LUA_LIB
LUA_LIB = $(LUA_PATH)/lib/liblua.a
endif
ifeq ("$(wildcard $(LUA_LIB))", "") # LUA_LIB does not exist
$(error $(LUA_LIB) does not exist. Try to specify both LUA_PATH and LUA_LIB manually)
endif
LDFLAGS += $(LUA_LIB)
endif
CFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CCFLAGS) $(OPT)
CXXFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT) -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers
LDFLAGS += $(PLATFORM_LDFLAGS)
# If NO_UPDATE_BUILD_VERSION is set we don't update util/build_version.cc, but
# the file needs to already exist or else the build will fail
ifndef NO_UPDATE_BUILD_VERSION
build: do not relink every single binary just for a timestamp Summary: Prior to this change, "make check" would always waste a lot of time relinking 60+ binaries. With this change, it does that only when the generated file, util/build_version.cc, changes, and that happens only when the date changes or when the current git SHA changes. This change makes some other improvements: before, there was no rule to build a deleted util/build_version.cc. If it was somehow removed, any attempt to link a program would fail. There is no longer any need for the separate file, build_tools/build_detect_version. Its functionality is now in the Makefile. * Makefile (DEPFILES): Don't filter-out util/build_version.cc. No need, and besides, removing that dependency was wrong. (date, git_sha, gen_build_version): New helper variables. (util/build_version.cc): New rule, to create this file and update it only if it would contain new information. * build_tools/build_detect_platform: Remove file. * db/db_impl.cc: Now, print only date (not the time). * util/build_version.h (rocksdb_build_compile_time): Remove declaration. No longer used. Test Plan: - Run "make check" twice, and note that the second time no linking is performed. - Remove util/build_version.cc and ensure that any "make" command regenerates it before doing anything else. - Run this: strings librocksdb.a|grep _build_. That prints output including the following: rocksdb_build_git_date:2015-02-19 rocksdb_build_git_sha:2.8.fb-1792-g3cb6cc0 Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D33591
2015-02-19 22:11:10 +01:00
date := $(shell date +%F)
ifdef FORCE_GIT_SHA
git_sha := $(FORCE_GIT_SHA)
else
git_sha := $(shell git rev-parse HEAD 2>/dev/null)
endif
gen_build_version = sed -e s/@@GIT_SHA@@/$(git_sha)/ -e s/@@GIT_DATE_TIME@@/$(date)/ util/build_version.cc.in
build: do not relink every single binary just for a timestamp Summary: Prior to this change, "make check" would always waste a lot of time relinking 60+ binaries. With this change, it does that only when the generated file, util/build_version.cc, changes, and that happens only when the date changes or when the current git SHA changes. This change makes some other improvements: before, there was no rule to build a deleted util/build_version.cc. If it was somehow removed, any attempt to link a program would fail. There is no longer any need for the separate file, build_tools/build_detect_version. Its functionality is now in the Makefile. * Makefile (DEPFILES): Don't filter-out util/build_version.cc. No need, and besides, removing that dependency was wrong. (date, git_sha, gen_build_version): New helper variables. (util/build_version.cc): New rule, to create this file and update it only if it would contain new information. * build_tools/build_detect_platform: Remove file. * db/db_impl.cc: Now, print only date (not the time). * util/build_version.h (rocksdb_build_compile_time): Remove declaration. No longer used. Test Plan: - Run "make check" twice, and note that the second time no linking is performed. - Remove util/build_version.cc and ensure that any "make" command regenerates it before doing anything else. - Run this: strings librocksdb.a|grep _build_. That prints output including the following: rocksdb_build_git_date:2015-02-19 rocksdb_build_git_sha:2.8.fb-1792-g3cb6cc0 Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D33591
2015-02-19 22:11:10 +01:00
# Record the version of the source that we are compiling.
# We keep a record of the git revision in this file. It is then built
# as a regular source file as part of the compilation process.
# One can run "strings executable_filename | grep _build_" to find
# the version of the source that we used to build the executable file.
FORCE:
util/build_version.cc: FORCE
$(AM_V_GEN)rm -f $@-t
$(AM_V_at)$(gen_build_version) > $@-t
$(AM_V_at)if test -f $@; then \
cmp -s $@-t $@ && rm -f $@-t || mv -f $@-t $@; \
else mv -f $@-t $@; fi
endif
build: do not relink every single binary just for a timestamp Summary: Prior to this change, "make check" would always waste a lot of time relinking 60+ binaries. With this change, it does that only when the generated file, util/build_version.cc, changes, and that happens only when the date changes or when the current git SHA changes. This change makes some other improvements: before, there was no rule to build a deleted util/build_version.cc. If it was somehow removed, any attempt to link a program would fail. There is no longer any need for the separate file, build_tools/build_detect_version. Its functionality is now in the Makefile. * Makefile (DEPFILES): Don't filter-out util/build_version.cc. No need, and besides, removing that dependency was wrong. (date, git_sha, gen_build_version): New helper variables. (util/build_version.cc): New rule, to create this file and update it only if it would contain new information. * build_tools/build_detect_platform: Remove file. * db/db_impl.cc: Now, print only date (not the time). * util/build_version.h (rocksdb_build_compile_time): Remove declaration. No longer used. Test Plan: - Run "make check" twice, and note that the second time no linking is performed. - Remove util/build_version.cc and ensure that any "make" command regenerates it before doing anything else. - Run this: strings librocksdb.a|grep _build_. That prints output including the following: rocksdb_build_git_date:2015-02-19 rocksdb_build_git_sha:2.8.fb-1792-g3cb6cc0 Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D33591
2015-02-19 22:11:10 +01:00
LIBOBJECTS = $(LIB_SOURCES:.cc=.o)
LIBOBJECTS += $(TOOL_LIB_SOURCES:.cc=.o)
MOCKOBJECTS = $(MOCK_LIB_SOURCES:.cc=.o)
GTEST = $(GTEST_DIR)/gtest/gtest-all.o
TESTUTIL = ./util/testutil.o
TESTHARNESS = ./util/testharness.o $(TESTUTIL) $(MOCKOBJECTS) $(GTEST)
VALGRIND_ERROR = 2
VALGRIND_VER := $(join $(VALGRIND_VER),valgrind)
VALGRIND_OPTS = --error-exitcode=$(VALGRIND_ERROR) --leak-check=full
BENCHTOOLOBJECTS = $(BENCH_LIB_SOURCES:.cc=.o) $(LIBOBJECTS) $(TESTUTIL)
Separeate main from bench functionality to allow cusomizations Summary: Isolate db_bench functionality from main so custom benchmark code can be written and managed Test Plan: Tested commands ./build_tools/regression_build_test.sh ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --reads=500 --writes=500 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --merge_keys=100 --numdistinct=100 --num_column_families=3 --num_hot_column_families=1 ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 Test Results - https://phabricator.fb.com/P56130387 Additional tests for: ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 --key_size=8 --merge_operator=put ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 --merge_operator=uint64add Results: https://phabricator.fb.com/P56130607 Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53991
2016-02-16 15:17:31 +01:00
EXPOBJECTS = $(EXP_LIB_SOURCES:.cc=.o) $(LIBOBJECTS) $(TESTUTIL)
TESTS = \
db_basic_test \
Encryption at rest support Summary: This PR adds support for encrypting data stored by RocksDB when written to disk. It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files. The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done. Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!). The Counter operation mode uses an initial counter & random initialization vector (IV). Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize). The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there. To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests. Typically you would run it like this: ``` ENCRYPTED_ENV=1 make check_some ``` There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings. Closes https://github.com/facebook/rocksdb/pull/2424 Differential Revision: D5322178 Pulled By: sdwilsh fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
2017-06-27 01:52:06 +02:00
db_encryption_test \
db_test2 \
external_sst_file_basic_test \
auto_roll_logger_test \
bloom_test \
dynamic_bloom_test \
c_test \
checkpoint_test \
crc32c_test \
coding_test \
inlineskiplist_test \
env_basic_test \
env_test \
hash_test \
thread_local_test \
rate_limiter_test \
perf_context_test \
iostats_context_test \
db_wal_test \
db_block_cache_test \
db_test \
db_bloom_filter_test \
db_iter_test \
db_log_iter_test \
db_compaction_filter_test \
db_compaction_test \
db_dynamic_level_test \
db_flush_test \
db_inplace_update_test \
db_iterator_test \
db_memtable_test \
db_merge_operator_test \
db_options_test \
db_range_del_test \
db_sst_test \
db_tailing_iter_test \
db_universal_compaction_test \
db_io_failure_test \
db_properties_test \
db_table_properties_test \
db_statistics_test \
db_write_test \
autovector_test \
blob_db_test \
cleanable_test \
column_family_test \
table_properties_collector_test \
arena_test \
block_test \
cache_test \
corruption_test \
slice_transform_test \
dbformat_test \
fault_injection_test \
filelock_test \
filename_test \
file_reader_writer_test \
block_based_filter_block_test \
full_filter_block_test \
partitioned_filter_block_test \
hash_table_test \
histogram_test \
log_test \
manual_compaction_test \
mock_env_test \
memtable_list_test \
merge_helper_test \
memory_test \
merge_test \
merger_test \
util_merge_operators_test \
options_file_test \
redis_test \
reduce_levels_test \
plain_table_db_test \
comparator_db_test \
external_sst_file_test \
prefix_test \
skiplist_test \
write_buffer_manager_test \
stringappend_test \
cassandra_format_test \
cassandra_functional_test \
cassandra_row_merge_test \
cassandra_serialize_test \
ttl_test \
date_tiered_test \
backupable_db_test \
document_db_test \
json_document_test \
add simulator Cache as class SimCache/SimLRUCache(with test) Summary: add class SimCache(base class with instrumentation api) and SimLRUCache(derived class with detailed implementation) which is used as an instrumented block cache that can predict hit rate for different cache size Test Plan: Add a test case in `db_block_cache_test.cc` called `SimCacheTest` to test basic logic of SimCache. Also add option `-simcache_size` in db_bench. if set with a value other than -1, then the benchmark will use this value as the size of the simulator cache and finally output the simulation result. ``` [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 1000000 RocksDB: version 4.8 Date: Tue May 17 16:56:16 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 6.809 micros/op 146874 ops/sec; 16.2 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.343 micros/op 157665 ops/sec; 17.4 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 986559 SimCache HITs: 264760 SimCache HITRATE: 26.84% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 10000000 RocksDB: version 4.8 Date: Tue May 17 16:57:10 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.066 micros/op 197394 ops/sec; 21.8 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.457 micros/op 154870 ops/sec; 17.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1059764 SimCache HITs: 374501 SimCache HITRATE: 35.34% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 100000000 RocksDB: version 4.8 Date: Tue May 17 16:57:32 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.632 micros/op 177572 ops/sec; 19.6 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.892 micros/op 145094 ops/sec; 16.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1150767 SimCache HITs: 1034535 SimCache HITRATE: 89.90% ``` Reviewers: IslamAbdelRahman, andrewkr, sdong Reviewed By: sdong Subscribers: MarkCallaghan, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57999
2016-05-24 08:35:23 +02:00
sim_cache_test \
spatial_db_test \
version_edit_test \
version_set_test \
compaction_picker_test \
version_builder_test \
file_indexer_test \
write_batch_test \
write_batch_with_index_test \
write_controller_test\
deletefile_test \
table_test \
geodb_test \
delete_scheduler_test \
options_test \
options_settable_test \
options_util_test \
event_logger_test \
timer_queue_test \
cuckoo_table_builder_test \
cuckoo_table_reader_test \
cuckoo_table_db_test \
flush_job_test \
wal_manager_test \
listener_test \
Support for SingleDelete() Summary: This patch fixes #7460559. It introduces SingleDelete as a new database operation. This operation can be used to delete keys that were never overwritten (no put following another put of the same key). If an overwritten key is single deleted the behavior is undefined. Single deletion of a non-existent key has no effect but multiple consecutive single deletions are not allowed (see limitations). In contrast to the conventional Delete() operation, the deletion entry is removed along with the value when the two are lined up in a compaction. Note: The semantics are similar to @igor's prototype that allowed to have this behavior on the granularity of a column family ( https://reviews.facebook.net/D42093 ). This new patch, however, is more aggressive when it comes to removing tombstones: It removes the SingleDelete together with the value whenever there is no snapshot between them while the older patch only did this when the sequence number of the deletion was older than the earliest snapshot. Most of the complex additions are in the Compaction Iterator, all other changes should be relatively straightforward. The patch also includes basic support for single deletions in db_stress and db_bench. Limitations: - Not compatible with cuckoo hash tables - Single deletions cannot be used in combination with merges and normal deletions on the same key (other keys are not affected by this) - Consecutive single deletions are currently not allowed (and older version of this patch supported this so it could be resurrected if needed) Test Plan: make all check Reviewers: yhchiang, sdong, rven, anthony, yoshinorim, igor Reviewed By: igor Subscribers: maykov, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43179
2015-09-17 20:42:56 +02:00
compaction_iterator_test \
compaction_job_test \
thread_list_test \
sst_dump_test \
column_aware_encoding_test \
compact_files_test \
optimistic_transaction_test \
write_callback_test \
heap_test \
compact_on_deletion_collector_test \
compaction_job_stats_test \
option_change_migration_test \
transaction_test \
ldb_cmd_test \
persistent_cache_test \
statistics_test \
lua_test \
range_del_aggregator_test \
lru_cache_test \
object_registry_test \
repair_test \
env_timed_test \
PARALLEL_TEST = \
backupable_db_test \
db_compaction_filter_test \
db_compaction_test \
db_sst_test \
db_test \
db_universal_compaction_test \
db_wal_test \
external_sst_file_test \
fault_injection_test \
inlineskiplist_test \
manual_compaction_test \
persistent_cache_test \
table_test \
transaction_test
SUBSET := $(TESTS)
ifdef ROCKSDBTESTS_START
SUBSET := $(shell echo $(SUBSET) | sed 's/^.*$(ROCKSDBTESTS_START)/$(ROCKSDBTESTS_START)/')
endif
ifdef ROCKSDBTESTS_END
SUBSET := $(shell echo $(SUBSET) | sed 's/$(ROCKSDBTESTS_END).*//')
endif
Utility to dump manifest contents. Summary: ./manifest_dump --file=/tmp/dbbench/MANIFEST-000002 Output looks like manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28 prev_log_number 0 --- level 0 --- --- level 1 --- --- level 2 --- 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1] 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1] 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1] 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1] 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1] 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1] 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1] 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1] 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1] 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1] 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1] 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1] 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1] --- level 3 --- --- level 4 --- --- level 5 --- --- level 6 --- Test Plan: run on test directory created by dbbench Reviewers: heyongqiang Reviewed By: heyongqiang CC: hustliubo Differential Revision: https://reviews.facebook.net/D4743
2012-08-17 19:48:40 +02:00
TOOLS = \
sst_dump \
db_sanity_test \
db_stress \
Write stress test Summary: The goal of this diff is to create a simple stress test with focus on catching: * bugs in compaction/flush processes, especially the ones that cause assertion errors * bugs in the code that deletes obsolete files There are two parts of the test: * write_stress, a binary that writes to the database * write_stress_runner.py, a script that invokes and kills write_stress Here are some interesting parts of write_stress: * Runs with very high concurrency of compactions and flushes (32 threads total) and tries to create a huge amount of small files * The keys written to the database are not uniformly distributed -- there is a 3-character prefix that mutates occasionally (in prefix mutator thread), in such a way that the first character mutates slower than second, which mutates slower than third character. That way, the compaction stress tests some interesting compaction features like trivial moves and bottommost level calculation * There is a thread that creates an iterator, holds it for couple of seconds and then iterates over all keys. This is supposed to test RocksDB's abilities to keep the files alive when there are references to them. * Some writes trigger WAL sync. This is stress testing our WAL sync code. * At the end of the run, we make sure that we didn't leak any of the sst files write_stress_runner.py changes the mode in which we run write_stress and also kills and restarts it. There are some interesting characteristics: * At the beginning we divide the full test runtime into smaller parts -- shorter runtimes (couple of seconds) and longer runtimes (100, 1000) seconds * The first time we run write_stress, we destroy the old DB. Every next time during the test, we use the same DB. * We can run in kill mode or clean-restart mode. Kill mode kills the write_stress violently. * We can run in mode where delete_obsolete_files_with_fullscan is true or false * We can run with low_open_files mode turned on or off. When it's turned on, we configure table cache to only hold a couple of files -- that way we need to reopen files every time we access them. Another goal was to create a stress test without a lot of parameters. So tools/write_stress_runner.py should only take one parameter -- runtime_sec and it should figure out everything else on its own. In a separate diff, I'll add this new test to our nightly legocastle runs. Test Plan: The goal of this test was to retroactively catch the following bugs: D33045, D48201, D46899, D42399. I failed to reproduce D48201, but all others have been caught! When i reverted https://reviews.facebook.net/D33045: ./write_stress --runtime_sec=200 --low_open_files_mode=true Iterator statuts not OK: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/089166.sst: No such file or directory When i reverted https://reviews.facebook.net/D42399: python tools/write_stress_runner.py --runtime_sec=5000 Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 Running write_stress, will kill after 2 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 7 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 8 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --low_open_files_mode=true Write to DB failed: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/019250.sst: No such file or directory ERROR: write_stress died with exitcode=-6 When i reverted https://reviews.facebook.net/D46899: python tools/write_stress_runner.py --runtime_sec=1000 runtime: 1000 Going to execute write stress for [3, 3, 100, 3, 2, 100, 1, 788] Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --low_open_files_mode=true Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 100 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true write_stress: db/db_impl.cc:2070: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion `log.getting_synced' failed. ERROR: write_stress died with exitcode=-6 Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan, sdong, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49533
2015-10-29 00:15:07 +01:00
write_stress \
ldb \
db_repl_stress \
rocksdb_dump \
rocksdb_undump \
blob_dump \
TEST_LIBS = \
librocksdb_env_basic_test.a
# TODO: add back forward_iterator_bench, after making it build in all environemnts.
BENCHMARKS = db_bench table_reader_bench cache_bench memtablerep_bench column_aware_encoding_exp persistent_cache_bench
# if user didn't config LIBNAME, set the default
ifeq ($(LIBNAME),)
# we should only run rocksdb in production with DEBUG_LEVEL 0
ifeq ($(DEBUG_LEVEL),0)
LIBNAME=librocksdb
else
LIBNAME=librocksdb_debug
endif
endif
LIBRARY = ${LIBNAME}.a
TOOLS_LIBRARY = ${LIBNAME}_tools.a
ROCKSDB_MAJOR = $(shell egrep "ROCKSDB_MAJOR.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
ROCKSDB_MINOR = $(shell egrep "ROCKSDB_MINOR.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
ROCKSDB_PATCH = $(shell egrep "ROCKSDB_PATCH.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
default: all
#-----------------------------------------------
# Create platform independent shared libraries.
#-----------------------------------------------
ifneq ($(PLATFORM_SHARED_EXT),)
ifneq ($(PLATFORM_SHARED_VERSIONED),true)
SHARED1 = ${LIBNAME}.$(PLATFORM_SHARED_EXT)
SHARED2 = $(SHARED1)
SHARED3 = $(SHARED1)
SHARED4 = $(SHARED1)
SHARED = $(SHARED1)
else
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
SHARED_MAJOR = $(ROCKSDB_MAJOR)
SHARED_MINOR = $(ROCKSDB_MINOR)
SHARED_PATCH = $(ROCKSDB_PATCH)
SHARED1 = ${LIBNAME}.$(PLATFORM_SHARED_EXT)
ifeq ($(PLATFORM), OS_MACOSX)
SHARED_OSX = $(LIBNAME).$(SHARED_MAJOR)
SHARED2 = $(SHARED_OSX).$(PLATFORM_SHARED_EXT)
SHARED3 = $(SHARED_OSX).$(SHARED_MINOR).$(PLATFORM_SHARED_EXT)
SHARED4 = $(SHARED_OSX).$(SHARED_MINOR).$(SHARED_PATCH).$(PLATFORM_SHARED_EXT)
else
SHARED2 = $(SHARED1).$(SHARED_MAJOR)
SHARED3 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR)
SHARED4 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR).$(SHARED_PATCH)
endif
SHARED = $(SHARED1) $(SHARED2) $(SHARED3) $(SHARED4)
$(SHARED1): $(SHARED4)
ln -fs $(SHARED4) $(SHARED1)
$(SHARED2): $(SHARED4)
ln -fs $(SHARED4) $(SHARED2)
$(SHARED3): $(SHARED4)
ln -fs $(SHARED4) $(SHARED3)
endif
shared_libobjects = $(patsubst %,shared-objects/%,$(LIBOBJECTS))
CLEAN_FILES += shared-objects
$(shared_libobjects): shared-objects/%.o: %.cc
$(AM_V_CC)mkdir -p $(@D) && $(CXX) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) -c $< -o $@
$(SHARED4): $(shared_libobjects)
$(CXX) $(PLATFORM_SHARED_LDFLAGS)$(SHARED3) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) $(shared_libobjects) $(LDFLAGS) -o $@
endif # PLATFORM_SHARED_EXT
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
.PHONY: blackbox_crash_test check clean coverage crash_test ldb_tests package \
release tags valgrind_check whitebox_crash_test format static_lib shared_lib all \
dbg rocksdbjavastatic rocksdbjava install install-static install-shared uninstall \
analyze tools tools_lib
all: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(TESTS)
all_but_some_tests: $(LIBRARY) $(BENCHMARKS) tools tools_lib test_libs $(SUBSET)
static_lib: $(LIBRARY)
shared_lib: $(SHARED)
tools: $(TOOLS)
tools_lib: $(TOOLS_LIBRARY)
test_libs: $(TEST_LIBS)
dbg: $(LIBRARY) $(BENCHMARKS) tools $(TESTS)
# creates static library and programs
release:
$(MAKE) clean
DEBUG_LEVEL=0 $(MAKE) static_lib tools db_bench
coverage:
$(MAKE) clean
COVERAGEFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS+="-lgcov" $(MAKE) J=1 all check
cd coverage && ./coverage_test.sh
# Delete intermediate files
find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
ifneq (,$(filter check parallel_check,$(MAKECMDGOALS)),)
# Use /dev/shm if it has the sticky bit set (otherwise, /tmp),
# and create a randomly-named rocksdb.XXXX directory therein.
# We'll use that directory in the "make check" rules.
ifeq ($(TMPD),)
TMPDIR := $(shell echo $${TMPDIR:-/tmp})
TMPD := $(shell f=/dev/shm; test -k $$f || f=$(TMPDIR); \
perl -le 'use File::Temp "tempdir";' \
-e 'print tempdir("'$$f'/rocksdb.XXXX", CLEANUP => 0)')
endif
endif
# Run all tests in parallel, accumulating per-test logs in t/log-*.
#
# Each t/run-* file is a tiny generated bourne shell script that invokes one of
# sub-tests. Why use a file for this? Because that makes the invocation of
# parallel below simpler, which in turn makes the parsing of parallel's
# LOG simpler (the latter is for live monitoring as parallel
# tests run).
#
# Test names are extracted by running tests with --gtest_list_tests.
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# This filter removes the "#"-introduced comments, and expands to
# fully-qualified names by changing input like this:
#
# DBTest.
# Empty
# WriteEmptyBatch
# MultiThreaded/MultiThreadedDBTest.
# MultiThreaded/0 # GetParam() = 0
# MultiThreaded/1 # GetParam() = 1
#
# into this:
#
# DBTest.Empty
# DBTest.WriteEmptyBatch
# MultiThreaded/MultiThreadedDBTest.MultiThreaded/0
# MultiThreaded/MultiThreadedDBTest.MultiThreaded/1
#
parallel_tests = $(patsubst %,parallel_%,$(PARALLEL_TEST))
.PHONY: gen_parallel_tests $(parallel_tests)
$(parallel_tests): $(PARALLEL_TEST)
$(AM_V_at)TEST_BINARY=$(patsubst parallel_%,%,$@); \
TEST_NAMES=` \
./$$TEST_BINARY --gtest_list_tests \
| perl -n \
-e 's/ *\#.*//;' \
-e '/^(\s*)(\S+)/; !$$1 and do {$$p=$$2; break};' \
-e 'print qq! $$p$$2!'`; \
for TEST_NAME in $$TEST_NAMES; do \
TEST_SCRIPT=t/run-$$TEST_BINARY-$${TEST_NAME//\//-}; \
echo " GEN " $$TEST_SCRIPT; \
printf '%s\n' \
'#!/bin/sh' \
"d=\$(TMPD)$$TEST_SCRIPT" \
'mkdir -p $$d' \
"TEST_TMPDIR=\$$d $(DRIVER) ./$$TEST_BINARY --gtest_filter=$$TEST_NAME" \
> $$TEST_SCRIPT; \
chmod a=rx $$TEST_SCRIPT; \
done
gen_parallel_tests:
$(AM_V_at)mkdir -p t
$(AM_V_at)rm -f t/run-*
$(MAKE) $(parallel_tests)
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# Reorder input lines (which are one per test) so that the
# longest-running tests appear first in the output.
# Do this by prefixing each selected name with its duration,
# sort the resulting names, and remove the leading numbers.
# FIXME: the "100" we prepend is a fake time, for now.
# FIXME: squirrel away timings from each run and use them
# (when present) on subsequent runs to order these tests.
#
# Without this reordering, these two tests would happen to start only
# after almost all other tests had completed, thus adding 100 seconds
# to the duration of parallel "make check". That's the difference
# between 4 minutes (old) and 2m20s (new).
#
# 152.120 PASS t/DBTest.FileCreationRandomFailure
# 107.816 PASS t/DBTest.EncodeDecompressedBlockSizeTest
#
slow_test_regexp = \
^t/run-table_test-HarnessTest.Randomized$$|^t/run-db_test-.*(?:FileCreationRandomFailure|EncodeDecompressedBlockSizeTest)$$|^.*RecoverFromCorruptedWALWithoutFlush$$
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
prioritize_long_running_tests = \
perl -pe 's,($(slow_test_regexp)),100 $$1,' \
| sort -k1,1gr \
| sed 's/^[.0-9]* //'
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# "make check" uses
# Run with "make J=1 check" to disable parallelism in "make check".
# Run with "make J=200% check" to run two parallel jobs per core.
# The default is to run one job per core (J=100%).
# See "man parallel" for its "-j ..." option.
J ?= 100%
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# Use this regexp to select the subset of tests whose names match.
tests-regexp = .
t_run = $(wildcard t/run-*)
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
.PHONY: check_0
check_0:
$(AM_V_GEN)export TEST_TMPDIR=$(TMPD); \
printf '%s\n' '' \
'To monitor subtest <duration,pass/fail,name>,' \
' run "make watch-log" in a separate window' ''; \
test -t 1 && eta=--eta || eta=; \
{ \
printf './%s\n' $(filter-out $(PARALLEL_TEST),$(TESTS)); \
printf '%s\n' $(t_run); \
} \
| $(prioritize_long_running_tests) \
| grep -E '$(tests-regexp)' \
| build_tools/gnu_parallel -j$(J) --plain --joblog=LOG $$eta --gnu '{} >& t/log-{/}'
valgrind-blacklist-regexp = InlineSkipTest.ConcurrentInsert|TransactionTest.DeadlockStress|DBCompactionTest.SuggestCompactRangeNoTwoLevel0Compactions|BackupableDBTest.RateLimiting|DBTest.CloseSpeedup|DBTest.ThreadStatusFlush|DBTest.RateLimitingTest|DBTest.EncodeDecompressedBlockSizeTest|FaultInjectionTest.UninstalledCompaction|HarnessTest.Randomized|ExternalSSTFileTest.CompactDuringAddFileRandom|ExternalSSTFileTest.IngestFileWithGlobalSeqnoRandomized
.PHONY: valgrind_check_0
valgrind_check_0:
$(AM_V_GEN)export TEST_TMPDIR=$(TMPD); \
printf '%s\n' '' \
'To monitor subtest <duration,pass/fail,name>,' \
' run "make watch-log" in a separate window' ''; \
test -t 1 && eta=--eta || eta=; \
{ \
printf './%s\n' $(filter-out $(PARALLEL_TEST) %skiplist_test options_settable_test, $(TESTS)); \
printf '%s\n' $(t_run); \
} \
| $(prioritize_long_running_tests) \
| grep -E '$(tests-regexp)' \
| grep -E -v '$(valgrind-blacklist-regexp)' \
| build_tools/gnu_parallel -j$(J) --plain --joblog=LOG $$eta --gnu \
'(if [[ "{}" == "./"* ]] ; then $(DRIVER) {}; else {}; fi) ' \
'>& t/valgrind_log-{/}'
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
CLEAN_FILES += t LOG $(TMPD)
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# When running parallel "make check", you can monitor its progress
# from another window.
# Run "make watch_LOG" to show the duration,PASS/FAIL,name of parallel
# tests as they are being run. We sort them so that longer-running ones
# appear at the top of the list and any failing tests remain at the top
# regardless of their duration. As with any use of "watch", hit ^C to
# interrupt.
watch-log:
watch --interval=0 'sort -k7,7nr -k4,4gr LOG|$(quoted_perl_command)'
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# If J != 1 and GNU parallel is installed, run the tests in parallel,
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# via the check_0 rule above. Otherwise, run them sequentially.
check: all
$(MAKE) gen_parallel_tests
$(AM_V_GEN)if test "$(J)" != 1 \
&& (build_tools/gnu_parallel --gnu --help 2>/dev/null) | \
grep -q 'GNU Parallel'; \
then \
$(MAKE) T="$$t" TMPD=$(TMPD) check_0; \
else \
for t in $(TESTS); do \
echo "===== Running $$t"; ./$$t || exit 1; done; \
fi
rm -rf $(TMPD)
ifneq ($(PLATFORM), OS_AIX)
ifeq ($(filter -DROCKSDB_LITE,$(OPT)),)
python tools/ldb_test.py
sh tools/rocksdb_dump_test.sh
endif
endif
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
# TODO add ldb_tests
check_some: $(SUBSET)
for t in $(SUBSET); do echo "===== Running $$t"; ./$$t || exit 1; done
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
.PHONY: ldb_tests
ldb_tests: ldb
python tools/ldb_test.py
2014-04-24 18:28:11 +02:00
crash_test: whitebox_crash_test blackbox_crash_test
blackbox_crash_test: db_stress
python -u tools/db_crashtest.py --simple blackbox
python -u tools/db_crashtest.py blackbox
ifeq ($(CRASH_TEST_KILL_ODD),)
CRASH_TEST_KILL_ODD=888887
endif
whitebox_crash_test: db_stress
python -u tools/db_crashtest.py --simple whitebox --random_kill_odd \
$(CRASH_TEST_KILL_ODD)
python -u tools/db_crashtest.py whitebox --random_kill_odd \
$(CRASH_TEST_KILL_ODD)
asan_check:
$(MAKE) clean
COMPILE_WITH_ASAN=1 $(MAKE) check -j32
$(MAKE) clean
asan_crash_test:
$(MAKE) clean
COMPILE_WITH_ASAN=1 $(MAKE) crash_test
$(MAKE) clean
ubsan_check:
$(MAKE) clean
COMPILE_WITH_UBSAN=1 $(MAKE) check -j32
$(MAKE) clean
ubsan_crash_test:
$(MAKE) clean
COMPILE_WITH_UBSAN=1 $(MAKE) crash_test
$(MAKE) clean
valgrind_test:
DISABLE_JEMALLOC=1 $(MAKE) valgrind_check
valgrind_check: $(TESTS)
$(MAKE) DRIVER="$(VALGRIND_VER) $(VALGRIND_OPTS)" gen_parallel_tests
$(AM_V_GEN)if test "$(J)" != 1 \
&& (build_tools/gnu_parallel --gnu --help 2>/dev/null) | \
grep -q 'GNU Parallel'; \
then \
$(MAKE) TMPD=$(TMPD) \
DRIVER="$(VALGRIND_VER) $(VALGRIND_OPTS)" valgrind_check_0; \
else \
for t in $(filter-out %skiplist_test options_settable_test,$(TESTS)); do \
$(VALGRIND_VER) $(VALGRIND_OPTS) ./$$t; \
ret_code=$$?; \
if [ $$ret_code -ne 0 ]; then \
exit $$ret_code; \
fi; \
done; \
fi
ifneq ($(PAR_TEST),)
parloop:
ret_bad=0; \
for t in $(PAR_TEST); do \
echo "===== Running $$t in parallel $(NUM_PAR)";\
if [ $(db_test) -eq 1 ]; then \
seq $(J) | v="$$t" build_tools/gnu_parallel --gnu --plain 's=$(TMPD)/rdb-{}; export TEST_TMPDIR=$$s;' \
'timeout 2m ./db_test --gtest_filter=$$v >> $$s/log-{} 2>1'; \
else\
seq $(J) | v="./$$t" build_tools/gnu_parallel --gnu --plain 's=$(TMPD)/rdb-{};' \
'export TEST_TMPDIR=$$s; timeout 10m $$v >> $$s/log-{} 2>1'; \
fi; \
ret_code=$$?; \
if [ $$ret_code -ne 0 ]; then \
ret_bad=$$ret_code; \
echo $$t exited with $$ret_code; \
fi; \
done; \
exit $$ret_bad;
endif
test_names = \
./db_test --gtest_list_tests \
| perl -n \
-e 's/ *\#.*//;' \
-e '/^(\s*)(\S+)/; !$$1 and do {$$p=$$2; break};' \
-e 'print qq! $$p$$2!'
parallel_check: $(TESTS)
$(AM_V_GEN)if test "$(J)" > 1 \
&& (build_tools/gnu_parallel --gnu --help 2>/dev/null) | \
grep -q 'GNU Parallel'; \
then \
echo Running in parallel $(J); \
else \
echo "Need to have GNU Parallel and J > 1"; exit 1; \
fi; \
ret_bad=0; \
echo $(J);\
echo Test Dir: $(TMPD); \
seq $(J) | build_tools/gnu_parallel --gnu --plain 's=$(TMPD)/rdb-{}; rm -rf $$s; mkdir $$s'; \
$(MAKE) PAR_TEST="$(shell $(test_names))" TMPD=$(TMPD) \
J=$(J) db_test=1 parloop; \
$(MAKE) PAR_TEST="$(filter-out db_test, $(TESTS))" \
TMPD=$(TMPD) J=$(J) db_test=0 parloop;
analyze: clean
$(CLANG_SCAN_BUILD) --use-analyzer=$(CLANG_ANALYZER) \
--use-c++=$(CXX) --use-cc=$(CC) --status-bugs \
-o $(CURDIR)/scan_build_report \
$(MAKE) dbg
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
CLEAN_FILES += unity.cc
unity.cc: Makefile
rm -f $@ $@-t
for source_file in $(LIB_SOURCES); do \
echo "#include \"$$source_file\"" >> $@-t; \
done
chmod a=r $@-t
mv $@-t $@
unity.a: unity.o
$(AM_V_AR)rm -f $@
$(AM_V_at)$(AR) $(ARFLAGS) $@ unity.o
# try compiling db_test with unity
unity_test: db/db_test.o db/db_test_util.o $(TESTHARNESS) unity.a
$(AM_LINK)
./unity_test
rocksdb.h rocksdb.cc: build_tools/amalgamate.py Makefile $(LIB_SOURCES) unity.cc
build_tools/amalgamate.py -I. -i./include unity.cc -x include/rocksdb/c.h -H rocksdb.h -o rocksdb.cc
clean:
rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(LIBRARY) $(SHARED)
rm -rf $(CLEAN_FILES) ios-x86 ios-arm scan_build_report
find . -name "*.[oda]" -exec rm -f {} \;
find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
rm -rf bzip2* snappy* zlib* lz4* zstd*
cd java; $(MAKE) clean
tags:
ctags * -R
cscope -b `find . -name '*.cc'` `find . -name '*.h'` `find . -name '*.c'`
ctags -e -R -o etags *
format:
build_tools/format-diff.sh
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
package:
bash build_tools/make_package.sh $(SHARED_MAJOR).$(SHARED_MINOR)
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
# ---------------------------------------------------------------------------
# Unit tests and tools
# ---------------------------------------------------------------------------
$(LIBRARY): $(LIBOBJECTS)
$(AM_V_AR)rm -f $@
$(AM_V_at)$(AR) $(ARFLAGS) $@ $(LIBOBJECTS)
$(TOOLS_LIBRARY): $(BENCH_LIB_SOURCES:.cc=.o) $(TOOL_LIB_SOURCES:.cc=.o) $(LIB_SOURCES:.cc=.o) $(TESTUTIL)
$(AM_V_AR)rm -f $@
$(AM_V_at)$(AR) $(ARFLAGS) $@ $^
librocksdb_env_basic_test.a: env/env_basic_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_V_AR)rm -f $@
$(AM_V_at)$(AR) $(ARFLAGS) $@ $^
Separeate main from bench functionality to allow cusomizations Summary: Isolate db_bench functionality from main so custom benchmark code can be written and managed Test Plan: Tested commands ./build_tools/regression_build_test.sh ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --reads=500 --writes=500 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --merge_keys=100 --numdistinct=100 --num_column_families=3 --num_hot_column_families=1 ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 Test Results - https://phabricator.fb.com/P56130387 Additional tests for: ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 --key_size=8 --merge_operator=put ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 --merge_operator=uint64add Results: https://phabricator.fb.com/P56130607 Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53991
2016-02-16 15:17:31 +01:00
db_bench: tools/db_bench.o $(BENCHTOOLOBJECTS)
$(AM_LINK)
cache_bench: cache/cache_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
persistent_cache_bench: utilities/persistent_cache/persistent_cache_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
memtablerep_bench: memtable/memtablerep_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
db_stress: tools/db_stress.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
Write stress test Summary: The goal of this diff is to create a simple stress test with focus on catching: * bugs in compaction/flush processes, especially the ones that cause assertion errors * bugs in the code that deletes obsolete files There are two parts of the test: * write_stress, a binary that writes to the database * write_stress_runner.py, a script that invokes and kills write_stress Here are some interesting parts of write_stress: * Runs with very high concurrency of compactions and flushes (32 threads total) and tries to create a huge amount of small files * The keys written to the database are not uniformly distributed -- there is a 3-character prefix that mutates occasionally (in prefix mutator thread), in such a way that the first character mutates slower than second, which mutates slower than third character. That way, the compaction stress tests some interesting compaction features like trivial moves and bottommost level calculation * There is a thread that creates an iterator, holds it for couple of seconds and then iterates over all keys. This is supposed to test RocksDB's abilities to keep the files alive when there are references to them. * Some writes trigger WAL sync. This is stress testing our WAL sync code. * At the end of the run, we make sure that we didn't leak any of the sst files write_stress_runner.py changes the mode in which we run write_stress and also kills and restarts it. There are some interesting characteristics: * At the beginning we divide the full test runtime into smaller parts -- shorter runtimes (couple of seconds) and longer runtimes (100, 1000) seconds * The first time we run write_stress, we destroy the old DB. Every next time during the test, we use the same DB. * We can run in kill mode or clean-restart mode. Kill mode kills the write_stress violently. * We can run in mode where delete_obsolete_files_with_fullscan is true or false * We can run with low_open_files mode turned on or off. When it's turned on, we configure table cache to only hold a couple of files -- that way we need to reopen files every time we access them. Another goal was to create a stress test without a lot of parameters. So tools/write_stress_runner.py should only take one parameter -- runtime_sec and it should figure out everything else on its own. In a separate diff, I'll add this new test to our nightly legocastle runs. Test Plan: The goal of this test was to retroactively catch the following bugs: D33045, D48201, D46899, D42399. I failed to reproduce D48201, but all others have been caught! When i reverted https://reviews.facebook.net/D33045: ./write_stress --runtime_sec=200 --low_open_files_mode=true Iterator statuts not OK: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/089166.sst: No such file or directory When i reverted https://reviews.facebook.net/D42399: python tools/write_stress_runner.py --runtime_sec=5000 Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 Running write_stress, will kill after 2 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 7 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 8 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --low_open_files_mode=true Write to DB failed: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/019250.sst: No such file or directory ERROR: write_stress died with exitcode=-6 When i reverted https://reviews.facebook.net/D46899: python tools/write_stress_runner.py --runtime_sec=1000 runtime: 1000 Going to execute write stress for [3, 3, 100, 3, 2, 100, 1, 788] Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --low_open_files_mode=true Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 100 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true write_stress: db/db_impl.cc:2070: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion `log.getting_synced' failed. ERROR: write_stress died with exitcode=-6 Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan, sdong, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49533
2015-10-29 00:15:07 +01:00
write_stress: tools/write_stress.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
db_sanity_test: tools/db_sanity_test.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
db_repl_stress: tools/db_repl_stress.o $(LIBOBJECTS) $(TESTUTIL)
$(AM_LINK)
arena_test: util/arena_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Implement autovector Summary: A vector that leverages pre-allocated stack-based array to achieve better performance for array with small amount of items. Test Plan: Added tests for both correctness and performance Here is the performance benchmark between vector and autovector Please note that in the test "Creation and Insertion Test", the test case were designed with the motivation described below: * no element inserted: internal array of std::vector may not really get initialize. * one element inserted: internal array of std::vector must have initialized. * kSize elements inserted. This shows the most time we'll spend if we keep everything in stack. * 2 * kSize elements inserted. The internal vector of autovector must have been initialized. Note: kSize is the capacity of autovector ===================================================== Creation and Insertion Test ===================================================== created 100000 vectors: each was inserted with 0 elements total time elapsed: 128000 (ns) created 100000 autovectors: each was inserted with 0 elements total time elapsed: 3641000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 0 elements total time elapsed: 9896000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 1 elements total time elapsed: 11089000 (ns) created 100000 autovectors: each was inserted with 1 elements total time elapsed: 5008000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 1 elements total time elapsed: 24271000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 4 elements total time elapsed: 39369000 (ns) created 100000 autovectors: each was inserted with 4 elements total time elapsed: 10121000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 4 elements total time elapsed: 28473000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 8 elements total time elapsed: 75013000 (ns) created 100000 autovectors: each was inserted with 8 elements total time elapsed: 18237000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 8 elements total time elapsed: 42464000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 16 elements total time elapsed: 102319000 (ns) created 100000 autovectors: each was inserted with 16 elements total time elapsed: 76724000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 16 elements total time elapsed: 68285000 (ns) ----------------------------------- ===================================================== Sequence Access Test ===================================================== performed 100000 sequence access against vector size: 4 total time elapsed: 198000 (ns) performed 100000 sequence access against autovector size: 4 total time elapsed: 306000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 8 total time elapsed: 565000 (ns) performed 100000 sequence access against autovector size: 8 total time elapsed: 512000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 16 total time elapsed: 1076000 (ns) performed 100000 sequence access against autovector size: 16 total time elapsed: 1070000 (ns) ----------------------------------- Reviewers: dhruba, haobo, sdong, chip Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14655
2013-12-13 00:32:56 +01:00
autovector_test: util/autovector_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Implement autovector Summary: A vector that leverages pre-allocated stack-based array to achieve better performance for array with small amount of items. Test Plan: Added tests for both correctness and performance Here is the performance benchmark between vector and autovector Please note that in the test "Creation and Insertion Test", the test case were designed with the motivation described below: * no element inserted: internal array of std::vector may not really get initialize. * one element inserted: internal array of std::vector must have initialized. * kSize elements inserted. This shows the most time we'll spend if we keep everything in stack. * 2 * kSize elements inserted. The internal vector of autovector must have been initialized. Note: kSize is the capacity of autovector ===================================================== Creation and Insertion Test ===================================================== created 100000 vectors: each was inserted with 0 elements total time elapsed: 128000 (ns) created 100000 autovectors: each was inserted with 0 elements total time elapsed: 3641000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 0 elements total time elapsed: 9896000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 1 elements total time elapsed: 11089000 (ns) created 100000 autovectors: each was inserted with 1 elements total time elapsed: 5008000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 1 elements total time elapsed: 24271000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 4 elements total time elapsed: 39369000 (ns) created 100000 autovectors: each was inserted with 4 elements total time elapsed: 10121000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 4 elements total time elapsed: 28473000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 8 elements total time elapsed: 75013000 (ns) created 100000 autovectors: each was inserted with 8 elements total time elapsed: 18237000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 8 elements total time elapsed: 42464000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 16 elements total time elapsed: 102319000 (ns) created 100000 autovectors: each was inserted with 16 elements total time elapsed: 76724000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 16 elements total time elapsed: 68285000 (ns) ----------------------------------- ===================================================== Sequence Access Test ===================================================== performed 100000 sequence access against vector size: 4 total time elapsed: 198000 (ns) performed 100000 sequence access against autovector size: 4 total time elapsed: 306000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 8 total time elapsed: 565000 (ns) performed 100000 sequence access against autovector size: 8 total time elapsed: 512000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 16 total time elapsed: 1076000 (ns) performed 100000 sequence access against autovector size: 16 total time elapsed: 1070000 (ns) ----------------------------------- Reviewers: dhruba, haobo, sdong, chip Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14655
2013-12-13 00:32:56 +01:00
column_family_test: db/column_family_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
table_properties_collector_test: db/table_properties_collector_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
bloom_test: util/bloom_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
dynamic_bloom_test: util/dynamic_bloom_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
c_test: db/c_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cache_test: cache/cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
coding_test: util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
hash_test: util/hash_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
option_change_migration_test: utilities/option_change_migration/option_change_migration_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
stringappend_test: utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cassandra_format_test: utilities/cassandra/cassandra_format_test.o utilities/cassandra/test_utils.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cassandra_functional_test: utilities/cassandra/cassandra_functional_test.o utilities/cassandra/test_utils.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cassandra_row_merge_test: utilities/cassandra/cassandra_row_merge_test.o utilities/cassandra/test_utils.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cassandra_serialize_test: utilities/cassandra/cassandra_serialize_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
redis_test: utilities/redis/redis_lists_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
hash_table_test: utilities/persistent_cache/hash_table_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
histogram_test: monitoring/histogram_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
thread_local_test: util/thread_local_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
corruption_test: db/corruption_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
crc32c_test: util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
slice_transform_test: util/slice_transform_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_basic_test: db/db_basic_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
Encryption at rest support Summary: This PR adds support for encrypting data stored by RocksDB when written to disk. It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files. The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done. Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!). The Counter operation mode uses an initial counter & random initialization vector (IV). Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize). The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there. To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests. Typically you would run it like this: ``` ENCRYPTED_ENV=1 make check_some ``` There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings. Closes https://github.com/facebook/rocksdb/pull/2424 Differential Revision: D5322178 Pulled By: sdwilsh fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
2017-06-27 01:52:06 +02:00
$(AM_LINK)
db_encryption_test: db/db_encryption_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_test: db/db_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_test2: db/db_test2.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_block_cache_test: db/db_block_cache_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_bloom_filter_test: db/db_bloom_filter_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_log_iter_test: db/db_log_iter_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_compaction_filter_test: db/db_compaction_filter_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_compaction_test: db/db_compaction_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_dynamic_level_test: db/db_dynamic_level_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_flush_test: db/db_flush_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_inplace_update_test: db/db_inplace_update_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_iterator_test: db/db_iterator_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_memtable_test: db/db_memtable_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_merge_operator_test: db/db_merge_operator_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_options_test: db/db_options_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_range_del_test: db/db_range_del_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_sst_test: db/db_sst_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_statistics_test: db/db_statistics_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_write_test: db/db_write_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
external_sst_file_basic_test: db/external_sst_file_basic_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
external_sst_file_test: db/external_sst_file_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_tailing_iter_test: db/db_tailing_iter_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_iter_test: db/db_iter_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_universal_compaction_test: db/db_universal_compaction_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_wal_test: db/db_wal_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_io_failure_test: db/db_io_failure_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_properties_test: db/db_properties_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_table_properties_test: db/db_table_properties_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
log_write_bench: util/log_write_bench.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK) $(PROFILING_FLAGS)
plain_table_db_test: db/plain_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
comparator_db_test: db/comparator_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
table_reader_bench: table/table_reader_bench.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK) $(PROFILING_FLAGS)
perf_context_test: db/perf_context_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_V_CCLD)$(CXX) $^ $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
prefix_test: db/prefix_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_V_CCLD)$(CXX) $^ $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
[RocksDB] BackupableDB Summary: In this diff I present you BackupableDB v1. You can easily use it to backup your DB and it will do incremental snapshots for you. Let's first describe how you would use BackupableDB. It's inheriting StackableDB interface so you can easily construct it with your DB object -- it will add a method RollTheSnapshot() to the DB object. When you call RollTheSnapshot(), current snapshot of the DB will be stored in the backup dir. To restore, you can just call RestoreDBFromBackup() on a BackupableDB (which is a static method) and it will restore all files from the backup dir. In the next version, it will even support automatic backuping every X minutes. There are multiple things you can configure: 1. backup_env and db_env can be different, which is awesome because then you can easily backup to HDFS or wherever you feel like. 2. sync - if true, it *guarantees* backup consistency on machine reboot 3. number of snapshots to keep - this will keep last N snapshots around if you want, for some reason, be able to restore from an earlier snapshot. All the backuping is done in incremental fashion - if we already have 00010.sst, we will not copy it again. *IMPORTANT* -- This is based on assumption that 00010.sst never changes - two files named 00010.sst from the same DB will always be exactly the same. Is this true? I always copy manifest, current and log files. 4. You can decide if you want to flush the memtables before you backup, or you're fine with backing up the log files -- either way, you get a complete and consistent view of the database at a time of backup. 5. More things you can find in BackupableDBOptions Here is the directory structure I use: backup_dir/CURRENT_SNAPSHOT - just 4 bytes holding the latest snapshot 0, 1, 2, ... - files containing serialized version of each snapshot - containing a list of files files/*.sst - sst files shared between snapshots - if one snapshot references 00010.sst and another one needs to backup it from the DB, it will just reference the same file files/ 0/, 1/, 2/, ... - snapshot directories containing private snapshot files - current, manifest and log files All the files are ref counted and deleted immediatelly when they get out of scope. Some other stuff in this diff: 1. Added GetEnv() method to the DB. Discussed with @haobo and we agreed that it seems right thing to do. 2. Fixed StackableDB interface. The way it was set up before, I was not able to implement BackupableDB. Test Plan: I have a unittest, but please don't look at this yet. I just hacked it up to help me with debugging. I will write a lot of good tests and update the diff. Also, `make asan_check` Reviewers: dhruba, haobo, emayanke Reviewed By: dhruba CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14295
2013-12-09 23:06:52 +01:00
backupable_db_test: utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
[RocksDB] BackupableDB Summary: In this diff I present you BackupableDB v1. You can easily use it to backup your DB and it will do incremental snapshots for you. Let's first describe how you would use BackupableDB. It's inheriting StackableDB interface so you can easily construct it with your DB object -- it will add a method RollTheSnapshot() to the DB object. When you call RollTheSnapshot(), current snapshot of the DB will be stored in the backup dir. To restore, you can just call RestoreDBFromBackup() on a BackupableDB (which is a static method) and it will restore all files from the backup dir. In the next version, it will even support automatic backuping every X minutes. There are multiple things you can configure: 1. backup_env and db_env can be different, which is awesome because then you can easily backup to HDFS or wherever you feel like. 2. sync - if true, it *guarantees* backup consistency on machine reboot 3. number of snapshots to keep - this will keep last N snapshots around if you want, for some reason, be able to restore from an earlier snapshot. All the backuping is done in incremental fashion - if we already have 00010.sst, we will not copy it again. *IMPORTANT* -- This is based on assumption that 00010.sst never changes - two files named 00010.sst from the same DB will always be exactly the same. Is this true? I always copy manifest, current and log files. 4. You can decide if you want to flush the memtables before you backup, or you're fine with backing up the log files -- either way, you get a complete and consistent view of the database at a time of backup. 5. More things you can find in BackupableDBOptions Here is the directory structure I use: backup_dir/CURRENT_SNAPSHOT - just 4 bytes holding the latest snapshot 0, 1, 2, ... - files containing serialized version of each snapshot - containing a list of files files/*.sst - sst files shared between snapshots - if one snapshot references 00010.sst and another one needs to backup it from the DB, it will just reference the same file files/ 0/, 1/, 2/, ... - snapshot directories containing private snapshot files - current, manifest and log files All the files are ref counted and deleted immediatelly when they get out of scope. Some other stuff in this diff: 1. Added GetEnv() method to the DB. Discussed with @haobo and we agreed that it seems right thing to do. 2. Fixed StackableDB interface. The way it was set up before, I was not able to implement BackupableDB. Test Plan: I have a unittest, but please don't look at this yet. I just hacked it up to help me with debugging. I will write a lot of good tests and update the diff. Also, `make asan_check` Reviewers: dhruba, haobo, emayanke Reviewed By: dhruba CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14295
2013-12-09 23:06:52 +01:00
checkpoint_test: utilities/checkpoint/checkpoint_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
document_db_test: utilities/document/document_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
json_document_test: utilities/document/json_document_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
add simulator Cache as class SimCache/SimLRUCache(with test) Summary: add class SimCache(base class with instrumentation api) and SimLRUCache(derived class with detailed implementation) which is used as an instrumented block cache that can predict hit rate for different cache size Test Plan: Add a test case in `db_block_cache_test.cc` called `SimCacheTest` to test basic logic of SimCache. Also add option `-simcache_size` in db_bench. if set with a value other than -1, then the benchmark will use this value as the size of the simulator cache and finally output the simulation result. ``` [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 1000000 RocksDB: version 4.8 Date: Tue May 17 16:56:16 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 6.809 micros/op 146874 ops/sec; 16.2 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.343 micros/op 157665 ops/sec; 17.4 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 986559 SimCache HITs: 264760 SimCache HITRATE: 26.84% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 10000000 RocksDB: version 4.8 Date: Tue May 17 16:57:10 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.066 micros/op 197394 ops/sec; 21.8 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.457 micros/op 154870 ops/sec; 17.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1059764 SimCache HITs: 374501 SimCache HITRATE: 35.34% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 100000000 RocksDB: version 4.8 Date: Tue May 17 16:57:32 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.632 micros/op 177572 ops/sec; 19.6 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.892 micros/op 145094 ops/sec; 16.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1150767 SimCache HITs: 1034535 SimCache HITRATE: 89.90% ``` Reviewers: IslamAbdelRahman, andrewkr, sdong Reviewed By: sdong Subscribers: MarkCallaghan, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57999
2016-05-24 08:35:23 +02:00
sim_cache_test: utilities/simulator_cache/sim_cache_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
spatial_db_test: utilities/spatialdb/spatial_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
env_mirror_test: utilities/env_mirror_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
env_timed_test: utilities/env_timed_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Add EnvLibrados - RocksDB Env of RADOS (#1222) EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/".
2016-07-21 20:16:34 +02:00
ifdef ROCKSDB_USE_LIBRADOS
env_librados_test: utilities/env_librados_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_V_CCLD)$(CXX) $^ $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
endif
object_registry_test: utilities/object_registry_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
ttl_test: utilities/ttl/ttl_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
date_tiered_test: utilities/date_tiered/date_tiered_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
write_batch_with_index_test: utilities/write_batch_with_index/write_batch_with_index_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
flush_job_test: db/flush_job_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Support for SingleDelete() Summary: This patch fixes #7460559. It introduces SingleDelete as a new database operation. This operation can be used to delete keys that were never overwritten (no put following another put of the same key). If an overwritten key is single deleted the behavior is undefined. Single deletion of a non-existent key has no effect but multiple consecutive single deletions are not allowed (see limitations). In contrast to the conventional Delete() operation, the deletion entry is removed along with the value when the two are lined up in a compaction. Note: The semantics are similar to @igor's prototype that allowed to have this behavior on the granularity of a column family ( https://reviews.facebook.net/D42093 ). This new patch, however, is more aggressive when it comes to removing tombstones: It removes the SingleDelete together with the value whenever there is no snapshot between them while the older patch only did this when the sequence number of the deletion was older than the earliest snapshot. Most of the complex additions are in the Compaction Iterator, all other changes should be relatively straightforward. The patch also includes basic support for single deletions in db_stress and db_bench. Limitations: - Not compatible with cuckoo hash tables - Single deletions cannot be used in combination with merges and normal deletions on the same key (other keys are not affected by this) - Consecutive single deletions are currently not allowed (and older version of this patch supported this so it could be resurrected if needed) Test Plan: make all check Reviewers: yhchiang, sdong, rven, anthony, yoshinorim, igor Reviewed By: igor Subscribers: maykov, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43179
2015-09-17 20:42:56 +02:00
compaction_iterator_test: db/compaction_iterator_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
compaction_job_test: db/compaction_job_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
compaction_job_stats_test: db/compaction_job_stats_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
compact_on_deletion_collector_test: utilities/table_properties_collectors/compact_on_deletion_collector_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
wal_manager_test: db/wal_manager_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
dbformat_test: db/dbformat_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
env_basic_test: env/env_basic_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
env_test: env/env_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
fault_injection_test: db/fault_injection_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
rate_limiter_test: util/rate_limiter_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
delete_scheduler_test: util/delete_scheduler_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
filename_test: db/filename_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
file_reader_writer_test: util/file_reader_writer_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Implement full filter for block based table. Summary: 1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file. 2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter. 3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type. 4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h. 5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc Benchmark: base commit 1d23b5c470844c1208301311f0889eca750431c0 Command: db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1 Read QPS increase for about 30% from 2230002 to 2991411. Test Plan: make all check valgrind db_test db_stress --use_block_based_filter = 0 ./auto_sanity_test.sh Reviewers: igor, yhchiang, ljin, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20979
2014-09-08 19:37:05 +02:00
block_based_filter_block_test: table/block_based_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
Implement full filter for block based table. Summary: 1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file. 2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter. 3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type. 4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h. 5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc Benchmark: base commit 1d23b5c470844c1208301311f0889eca750431c0 Command: db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1 Read QPS increase for about 30% from 2230002 to 2991411. Test Plan: make all check valgrind db_test db_stress --use_block_based_filter = 0 ./auto_sanity_test.sh Reviewers: igor, yhchiang, ljin, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20979
2014-09-08 19:37:05 +02:00
full_filter_block_test: table/full_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
partitioned_filter_block_test: table/partitioned_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
log_test: db/log_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cleanable_test: table/cleanable_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
table_test: table/table_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
block_test: table/block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
inlineskiplist_test: memtable/inlineskiplist_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
skiplist_test: memtable/skiplist_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
write_buffer_manager_test: memtable/write_buffer_manager_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
version_edit_test: db/version_edit_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
version_set_test: db/version_set_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
compaction_picker_test: db/compaction_picker_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
version_builder_test: db/version_builder_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
file_indexer_test: db/file_indexer_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
reduce_levels_test: tools/reduce_levels_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
write_batch_test: db/write_batch_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
write_controller_test: db/write_controller_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
merge_helper_test: db/merge_helper_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
memory_test: utilities/memory/memory_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
merge_test: db/merge_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
merger_test: table/merger_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
util_merge_operators_test: utilities/util_merge_operators_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
options_file_test: db/options_file_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
deletefile_test: db/deletefile_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
geodb_test: utilities/geodb/geodb_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
rocksdb_dump: tools/dump/rocksdb_dump.o $(LIBOBJECTS)
$(AM_LINK)
rocksdb_undump: tools/dump/rocksdb_undump.o $(LIBOBJECTS)
$(AM_LINK)
cuckoo_table_builder_test: table/cuckoo_table_builder_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cuckoo_table_reader_test: table/cuckoo_table_reader_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
cuckoo_table_db_test: db/cuckoo_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
listener_test: db/listener_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
thread_list_test: util/thread_list_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
compact_files_test: db/compact_files_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
options_test: options/options_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
options_settable_test: options/options_settable_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
options_util_test: utilities/options/options_util_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
db_bench_tool_test: tools/db_bench_tool_test.o $(BENCHTOOLOBJECTS) $(TESTHARNESS)
$(AM_LINK)
event_logger_test: util/event_logger_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
timer_queue_test: util/timer_queue_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
sst_dump_test: tools/sst_dump_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
column_aware_encoding_test: utilities/column_aware_encoding_test.o $(TESTHARNESS) $(EXPOBJECTS)
$(AM_LINK)
optimistic_transaction_test: utilities/transactions/optimistic_transaction_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
mock_env_test : env/mock_env_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
manual_compaction_test: db/manual_compaction_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
filelock_test: util/filelock_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
auto_roll_logger_test: util/auto_roll_logger_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
memtable_list_test: db/memtable_list_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
write_callback_test: db/write_callback_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
heap_test: util/heap_test.o $(GTEST)
$(AM_LINK)
transaction_test: utilities/transactions/transaction_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
sst_dump: tools/sst_dump.o $(LIBOBJECTS)
$(AM_LINK)
Utility to dump manifest contents. Summary: ./manifest_dump --file=/tmp/dbbench/MANIFEST-000002 Output looks like manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28 prev_log_number 0 --- level 0 --- --- level 1 --- --- level 2 --- 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1] 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1] 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1] 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1] 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1] 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1] 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1] 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1] 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1] 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1] 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1] 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1] 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1] --- level 3 --- --- level 4 --- --- level 5 --- --- level 6 --- Test Plan: run on test directory created by dbbench Reviewers: heyongqiang Reviewed By: heyongqiang CC: hustliubo Differential Revision: https://reviews.facebook.net/D4743
2012-08-17 19:48:40 +02:00
blob_dump: tools/blob_dump.o $(LIBOBJECTS)
$(AM_LINK)
column_aware_encoding_exp: utilities/column_aware_encoding_exp.o $(EXPOBJECTS)
$(AM_LINK)
repair_test: db/repair_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
ldb_cmd_test: tools/ldb_cmd_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
ldb: tools/ldb.o $(LIBOBJECTS)
$(AM_LINK)
iostats_context_test: monitoring/iostats_context_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_V_CCLD)$(CXX) $^ $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
persistent_cache_test: utilities/persistent_cache/persistent_cache_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
statistics_test: monitoring/statistics_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
lru_cache_test: cache/lru_cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
lua_test: utilities/lua/rocks_lua_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
range_del_aggregator_test: db/range_del_aggregator_test.o db/db_test_util.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
blob_db_test: utilities/blob_db/blob_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(AM_LINK)
#-------------------------------------------------
# make install related stuff
INSTALL_PATH ?= /usr/local
uninstall:
rm -rf $(INSTALL_PATH)/include/rocksdb \
$(INSTALL_PATH)/lib/$(LIBRARY) \
$(INSTALL_PATH)/lib/$(SHARED4) \
$(INSTALL_PATH)/lib/$(SHARED3) \
$(INSTALL_PATH)/lib/$(SHARED2) \
$(INSTALL_PATH)/lib/$(SHARED1)
install-headers:
install -d $(INSTALL_PATH)/lib
for header_dir in `find "include/rocksdb" -type d`; do \
install -d $(INSTALL_PATH)/$$header_dir; \
done
for header in `find "include/rocksdb" -type f -name *.h`; do \
install -C -m 644 $$header $(INSTALL_PATH)/$$header; \
done
install-static: install-headers $(LIBRARY)
install -C -m 755 $(LIBRARY) $(INSTALL_PATH)/lib
install-shared: install-headers $(SHARED4)
install -C -m 755 $(SHARED4) $(INSTALL_PATH)/lib && \
ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED3) && \
ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED2) && \
ln -fs $(SHARED4) $(INSTALL_PATH)/lib/$(SHARED1)
# install static by default + install shared if it exists
install: install-static
[ -e $(SHARED4) ] && $(MAKE) install-shared || :
#-------------------------------------------------
# ---------------------------------------------------------------------------
# Jni stuff
# ---------------------------------------------------------------------------
JAVA_INCLUDE = -I$(JAVA_HOME)/include/ -I$(JAVA_HOME)/include/linux
ifeq ($(PLATFORM), OS_SOLARIS)
ARCH := $(shell isainfo -b)
else
ARCH := $(shell getconf LONG_BIT)
endif
ifeq (,$(findstring ppc,$(MACHINE)))
ROCKSDBJNILIB = librocksdbjni-linux$(ARCH).so
else
ROCKSDBJNILIB = librocksdbjni-linux-$(MACHINE).so
endif
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux$(ARCH).jar
ROCKSDB_JAR_ALL = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar
2014-10-02 23:13:09 +02:00
ROCKSDB_JAVADOCS_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-javadoc.jar
ROCKSDB_SOURCES_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar
SHA256_CMD = sha256sum
ZLIB_VER ?= 1.2.11
ZLIB_SHA256 ?= c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1
ZLIB_DOWNLOAD_BASE ?= http://zlib.net
BZIP2_VER ?= 1.0.6
BZIP2_SHA256 ?= a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd
BZIP2_DOWNLOAD_BASE ?= http://www.bzip.org
SNAPPY_VER ?= 1.1.4
SNAPPY_SHA256 ?= 134bfe122fd25599bb807bb8130e7ba6d9bdb851e0b16efcb83ac4f5d0b70057
SNAPPY_DOWNLOAD_BASE ?= https://github.com/google/snappy/releases/download
LZ4_VER ?= 1.7.5
LZ4_SHA256 ?= 0190cacd63022ccb86f44fa5041dc6c3804407ad61550ca21c382827319e7e7e
LZ4_DOWNLOAD_BASE ?= https://github.com/lz4/lz4/archive
ZSTD_VER ?= 1.2.0
ZSTD_SHA256 ?= 4a7e4593a3638276ca7f2a09dc4f38e674d8317bbea51626393ca73fc047cbfb
ZSTD_DOWNLOAD_BASE ?= https://github.com/facebook/zstd/archive
ifeq ($(PLATFORM), OS_MACOSX)
ROCKSDBJNILIB = librocksdbjni-osx.jnilib
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar
SHA256_CMD = openssl sha256 -r
ifneq ("$(wildcard $(JAVA_HOME)/include/darwin)","")
JAVA_INCLUDE = -I$(JAVA_HOME)/include -I $(JAVA_HOME)/include/darwin
else
JAVA_INCLUDE = -I/System/Library/Frameworks/JavaVM.framework/Headers/
endif
endif
ifeq ($(PLATFORM), OS_FREEBSD)
JAVA_INCLUDE += -I$(JAVA_HOME)/include/freebsd
ROCKSDBJNILIB = librocksdbjni-freebsd$(ARCH).so
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-freebsd$(ARCH).jar
endif
ifeq ($(PLATFORM), OS_SOLARIS)
ROCKSDBJNILIB = librocksdbjni-solaris$(ARCH).so
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-solaris$(ARCH).jar
JAVA_INCLUDE = -I$(JAVA_HOME)/include/ -I$(JAVA_HOME)/include/solaris
SHA256_CMD = digest -a sha256
endif
ifeq ($(PLATFORM), OS_AIX)
JAVA_INCLUDE = -I$(JAVA_HOME)/include/ -I$(JAVA_HOME)/include/aix
ROCKSDBJNILIB = librocksdbjni-aix.so
EXTRACT_SOURCES = gunzip < TAR_GZ | tar xvf -
SNAPPY_MAKE_TARGET = libsnappy.la
endif
2014-09-09 02:44:52 +02:00
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L ${ZLIB_DOWNLOAD_BASE}/zlib-$(ZLIB_VER).tar.gz
ZLIB_SHA256_ACTUAL=`$(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(ZLIB_SHA256)" != "$$ZLIB_SHA256_ACTUAL" ]; then \
echo zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected=\"$(ZLIB_SHA256)\" actual=\"$$ZLIB_SHA256_ACTUAL\"; \
exit 1; \
fi
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC ${EXTRA_CFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --static && make
cp zlib-$(ZLIB_VER)/libz.a .
2014-09-09 02:44:52 +02:00
libbz2.a:
-rm -rf bzip2-$(BZIP2_VER)
curl -O -L ${BZIP2_DOWNLOAD_BASE}/$(BZIP2_VER)/bzip2-$(BZIP2_VER).tar.gz
BZIP2_SHA256_ACTUAL=`$(SHA256_CMD) bzip2-$(BZIP2_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(BZIP2_SHA256)" != "$$BZIP2_SHA256_ACTUAL" ]; then \
echo bzip2-$(BZIP2_VER).tar.gz checksum mismatch, expected=\"$(BZIP2_SHA256)\" actual=\"$$BZIP2_SHA256_ACTUAL\"; \
exit 1; \
fi
tar xvzf bzip2-$(BZIP2_VER).tar.gz
cd bzip2-$(BZIP2_VER) && make CFLAGS='-fPIC -O2 -g -D_FILE_OFFSET_BITS=64 ${EXTRA_CFLAGS}' AR='ar ${EXTRA_ARFLAGS}'
cp bzip2-$(BZIP2_VER)/libbz2.a .
2014-09-09 02:44:52 +02:00
libsnappy.a:
-rm -rf snappy-$(SNAPPY_VER)
curl -O -L ${SNAPPY_DOWNLOAD_BASE}/$(SNAPPY_VER)/snappy-$(SNAPPY_VER).tar.gz
SNAPPY_SHA256_ACTUAL=`$(SHA256_CMD) snappy-$(SNAPPY_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(SNAPPY_SHA256)" != "$$SNAPPY_SHA256_ACTUAL" ]; then \
echo snappy-$(SNAPPY_VER).tar.gz checksum mismatch, expected=\"$(SNAPPY_SHA256)\" actual=\"$$SNAPPY_SHA256_ACTUAL\"; \
exit 1; \
fi
tar xvzf snappy-$(SNAPPY_VER).tar.gz
cd snappy-$(SNAPPY_VER) && CFLAGS='${EXTRA_CFLAGS}' CXXFLAGS='${EXTRA_CXXFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --with-pic --enable-static --disable-shared
cd snappy-$(SNAPPY_VER) && make ${SNAPPY_MAKE_TARGET}
cp snappy-$(SNAPPY_VER)/.libs/libsnappy.a .
liblz4.a:
-rm -rf lz4-$(LZ4_VER)
curl -O -L ${LZ4_DOWNLOAD_BASE}/v$(LZ4_VER).tar.gz
mv v$(LZ4_VER).tar.gz lz4-$(LZ4_VER).tar.gz
LZ4_SHA256_ACTUAL=`$(SHA256_CMD) lz4-$(LZ4_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(LZ4_SHA256)" != "$$LZ4_SHA256_ACTUAL" ]; then \
echo lz4-$(LZ4_VER).tar.gz checksum mismatch, expected=\"$(LZ4_SHA256)\" actual=\"$$LZ4_SHA256_ACTUAL\"; \
exit 1; \
fi
tar xvzf lz4-$(LZ4_VER).tar.gz
cd lz4-$(LZ4_VER)/lib && make CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' all
cp lz4-$(LZ4_VER)/lib/liblz4.a .
2014-09-09 02:44:52 +02:00
libzstd.a:
-rm -rf zstd-$(ZSTD_VER)
curl -O -L ${ZSTD_DOWNLOAD_BASE}/v$(ZSTD_VER).tar.gz
mv v$(ZSTD_VER).tar.gz zstd-$(ZSTD_VER).tar.gz
ZSTD_SHA256_ACTUAL=`$(SHA256_CMD) zstd-$(ZSTD_VER).tar.gz | cut -d ' ' -f 1`; \
if [ "$(ZSTD_SHA256)" != "$$ZSTD_SHA256_ACTUAL" ]; then \
echo zstd-$(ZSTD_VER).tar.gz checksum mismatch, expected=\"$(ZSTD_SHA256)\" actual=\"$$ZSTD_SHA256_ACTUAL\"; \
exit 1; \
fi
tar xvzf zstd-$(ZSTD_VER).tar.gz
cd zstd-$(ZSTD_VER)/lib && make CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' all
cp zstd-$(ZSTD_VER)/lib/libzstd.a .
# A version of each $(LIBOBJECTS) compiled with -fPIC and a fixed set of static compression libraries
java_static_libobjects = $(patsubst %,jls/%,$(LIBOBJECTS))
CLEAN_FILES += jls
ifneq ($(ROCKSDB_JAVA_NO_COMPRESSION), 1)
JAVA_COMPRESSIONS = libz.a libbz2.a libsnappy.a liblz4.a libzstd.a
endif
JAVA_STATIC_FLAGS = -DZLIB -DBZIP2 -DSNAPPY -DLZ4 -DZSTD
JAVA_STATIC_INCLUDES = -I./zlib-$(ZLIB_VER) -I./bzip2-$(BZIP2_VER) -I./snappy-$(SNAPPY_VER) -I./lz4-$(LZ4_VER)/lib -I./zstd-$(ZSTD_VER)/lib
$(java_static_libobjects): jls/%.o: %.cc $(JAVA_COMPRESSIONS)
$(AM_V_CC)mkdir -p $(@D) && $(CXX) $(CXXFLAGS) $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES) -fPIC -c $< -o $@ $(COVERAGEFLAGS)
rocksdbjavastatic: $(java_static_libobjects)
cd java;$(MAKE) javalib;
rm -f ./java/target/$(ROCKSDBJNILIB)
$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC \
-o ./java/target/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) \
$(java_static_libobjects) $(COVERAGEFLAGS) \
$(JAVA_COMPRESSIONS) $(JAVA_STATIC_LDFLAGS)
cd java/target;strip $(STRIPFLAGS) $(ROCKSDBJNILIB)
cd java;jar -cf target/$(ROCKSDB_JAR) HISTORY*.md
cd java/target;jar -uf $(ROCKSDB_JAR) $(ROCKSDBJNILIB)
cd java/target/classes;jar -uf ../$(ROCKSDB_JAR) org/rocksdb/*.class org/rocksdb/util/*.class
cd java/target/apidocs;jar -cf ../$(ROCKSDB_JAVADOCS_JAR) *
cd java/src/main/java;jar -cf ../../../target/$(ROCKSDB_SOURCES_JAR) org
rocksdbjavastaticrelease: rocksdbjavastatic
2014-09-30 21:03:32 +02:00
cd java/crossbuild && vagrant destroy -f && vagrant up linux32 && vagrant halt linux32 && vagrant up linux64 && vagrant halt linux64
cd java;jar -cf target/$(ROCKSDB_JAR_ALL) HISTORY*.md
cd java/target;jar -uf $(ROCKSDB_JAR_ALL) librocksdbjni-*.so librocksdbjni-*.jnilib
cd java/target/classes;jar -uf ../$(ROCKSDB_JAR_ALL) org/rocksdb/*.class org/rocksdb/util/*.class
rocksdbjavastaticreleasedocker: rocksdbjavastatic
DOCKER_LINUX_X64_CONTAINER=`docker ps -aqf name=rocksdb_linux_x64-be`; \
if [ -z "$$DOCKER_LINUX_X64_CONTAINER" ]; then \
docker container create --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host --name rocksdb_linux_x64-be evolvedbinary/rocksjava:centos6_x64-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh; \
fi
docker start -a rocksdb_linux_x64-be
DOCKER_LINUX_X86_CONTAINER=`docker ps -aqf name=rocksdb_linux_x86-be`; \
if [ -z "$$DOCKER_LINUX_X86_CONTAINER" ]; then \
docker container create --attach stdin --attach stdout --attach stderr --volume `pwd`:/rocksdb-host --name rocksdb_linux_x86-be evolvedbinary/rocksjava:centos6_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh; \
fi
docker start -a rocksdb_linux_x86-be
cd java;jar -cf target/$(ROCKSDB_JAR_ALL) HISTORY*.md
cd java/target;jar -uf $(ROCKSDB_JAR_ALL) librocksdbjni-*.so librocksdbjni-*.jnilib
cd java/target/classes;jar -uf ../$(ROCKSDB_JAR_ALL) org/rocksdb/*.class org/rocksdb/util/*.class
rocksdbjavastaticpublish: rocksdbjavastaticrelease rocksdbjavastaticpublishcentral
rocksdbjavastaticpublishdocker: rocksdbjavastaticreleasedocker rocksdbjavastaticpublishcentral
rocksdbjavastaticpublishcentral:
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-javadoc.jar -Dclassifier=javadoc
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar -Dclassifier=sources
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux64.jar -Dclassifier=linux64
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux32.jar -Dclassifier=linux32
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar -Dclassifier=osx
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-win64.jar -Dclassifier=win64
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/target/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar
# A version of each $(LIBOBJECTS) compiled with -fPIC
java_libobjects = $(patsubst %,jl/%,$(LIBOBJECTS))
CLEAN_FILES += jl
$(java_libobjects): jl/%.o: %.cc
$(AM_V_CC)mkdir -p $(@D) && $(CXX) $(CXXFLAGS) -fPIC -c $< -o $@ $(COVERAGEFLAGS)
rocksdbjava: $(java_libobjects)
$(AM_V_GEN)cd java;$(MAKE) javalib;
$(AM_V_at)rm -f ./java/target/$(ROCKSDBJNILIB)
$(AM_V_at)$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC -o ./java/target/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) $(java_libobjects) $(JAVA_LDFLAGS) $(COVERAGEFLAGS)
$(AM_V_at)cd java;jar -cf target/$(ROCKSDB_JAR) HISTORY*.md
$(AM_V_at)cd java/target;jar -uf $(ROCKSDB_JAR) $(ROCKSDBJNILIB)
$(AM_V_at)cd java/target/classes;jar -uf ../$(ROCKSDB_JAR) org/rocksdb/*.class org/rocksdb/util/*.class
jclean:
cd java;$(MAKE) clean;
jtest_compile: rocksdbjava
cd java;$(MAKE) java_test
jtest_run:
cd java;$(MAKE) run_test
jtest: rocksdbjava
cd java;$(MAKE) sample;$(MAKE) test;
[JNI] Add an initial benchmark for java binding for rocksdb. Summary: * Add a benchmark for java binding for rocksdb. The java benchmark is a complete rewrite based on the c++ db/db_bench.cc and the DbBenchmark in dain's java leveldb. * Support multithreading. * 'readseq' is currently not supported as it requires RocksDB Iterator. * usage: --benchmarks Comma-separated list of operations to run in the specified order Actual benchmarks: fillseq -- write N values in sequential key order in async mode fillrandom -- write N values in random key order in async mode fillbatch -- write N/1000 batch where each batch has 1000 values in random key order in sync mode fillsync -- write N/100 values in random key order in sync mode fill100K -- write N/1000 100K values in random order in async mode readseq -- read N times sequentially readrandom -- read N times in random order readhot -- read N times in random order from 1% section of DB Meta Operations: delete -- delete DB DEFAULT: [fillseq, readrandom, fillrandom] --compression_ratio Arrange to generate values that shrink to this fraction of their original size after compression DEFAULT: 0.5 --use_existing_db If true, do not destroy the existing database. If you set this flag and also specify a benchmark that wants a fresh database, that benchmark will fail. DEFAULT: false --num Number of key/values to place in database. DEFAULT: 1000000 --threads Number of concurrent threads to run. DEFAULT: 1 --reads Number of read operations to do. If negative, do --nums reads. --key_size The size of each key in bytes. DEFAULT: 16 --value_size The size of each value in bytes. DEFAULT: 100 --write_buffer_size Number of bytes to buffer in memtable before compacting (initialized to default value by 'main'.) DEFAULT: 4194304 --cache_size Number of bytes to use as a cache of uncompressed data. Negative means use default settings. DEFAULT: -1 --seed Seed base for random number generators. DEFAULT: 0 --db Use the db with the following name. DEFAULT: /tmp/rocksdbjni-bench * Add RocksDB.write(). Test Plan: make jbench Reviewers: haobo, sdong, dhruba, ankgup87 Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17433
2014-04-09 09:48:20 +02:00
jdb_bench:
cd java;$(MAKE) db_bench;
commit_prereq: build_tools/rocksdb-lego-determinator \
build_tools/precommit_checker.py
J=$(J) build_tools/precommit_checker.py unit unit_481 clang_unit release release_481 clang_release tsan asan ubsan lite unit_non_shm
$(MAKE) clean && $(MAKE) jclean && $(MAKE) rocksdbjava;
# ---------------------------------------------------------------------------
# Platform-specific compilation
# ---------------------------------------------------------------------------
ifeq ($(PLATFORM), IOS)
# For iOS, create universal object files to be used on both the simulator and
# a device.
PLATFORMSROOT=/Applications/Xcode.app/Contents/Developer/Platforms
SIMULATORROOT=$(PLATFORMSROOT)/iPhoneSimulator.platform/Developer
DEVICEROOT=$(PLATFORMSROOT)/iPhoneOS.platform/Developer
IOSVERSION=$(shell defaults read $(PLATFORMSROOT)/iPhoneOS.platform/version CFBundleShortVersionString)
.cc.o:
mkdir -p ios-x86/$(dir $@)
$(CXX) $(CXXFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -arch x86_64 -c $< -o ios-x86/$@
mkdir -p ios-arm/$(dir $@)
xcrun -sdk iphoneos $(CXX) $(CXXFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -arch armv7s -arch arm64 -c $< -o ios-arm/$@
lipo ios-x86/$@ ios-arm/$@ -create -output $@
.c.o:
mkdir -p ios-x86/$(dir $@)
$(CC) $(CFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -arch x86_64 -c $< -o ios-x86/$@
mkdir -p ios-arm/$(dir $@)
xcrun -sdk iphoneos $(CC) $(CFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -arch armv7s -arch arm64 -c $< -o ios-arm/$@
lipo ios-x86/$@ ios-arm/$@ -create -output $@
else
.cc.o:
$(AM_V_CC)$(CXX) $(CXXFLAGS) -c $< -o $@ $(COVERAGEFLAGS)
.c.o:
$(AM_V_CC)$(CC) $(CFLAGS) -c $< -o $@
endif
# ---------------------------------------------------------------------------
# Source files dependencies detection
# ---------------------------------------------------------------------------
all_sources = $(LIB_SOURCES) $(MAIN_SOURCES) $(MOCK_LIB_SOURCES) $(TOOL_LIB_SOURCES) $(BENCH_LIB_SOURCES) $(TEST_LIB_SOURCES) $(EXP_LIB_SOURCES)
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
DEPFILES = $(all_sources:.cc=.d)
# Add proper dependency support so changing a .h file forces a .cc file to
# rebuild.
# The .d file indicates .cc file's dependencies on .h files. We generate such
# dependency by g++'s -MM option, whose output is a make dependency rule.
run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379
2015-04-06 21:35:25 +02:00
$(DEPFILES): %.d: %.cc
@$(CXX) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) \
-MM -MT'$@' -MT'$(<:.cc=.o)' "$<" -o '$@'
depend: $(DEPFILES)
# if the make goal is either "clean" or "format", we shouldn't
# try to import the *.d files.
# TODO(kailiu) The unfamiliarity of Make's conditions leads to the ugly
# working solution.
ifneq ($(MAKECMDGOALS),clean)
ifneq ($(MAKECMDGOALS),format)
ifneq ($(MAKECMDGOALS),jclean)
ifneq ($(MAKECMDGOALS),jtest)
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
ifneq ($(MAKECMDGOALS),package)
ifneq ($(MAKECMDGOALS),analyze)
-include $(DEPFILES)
endif
endif
endif
endif
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
endif
endif