andreacavalli/rocksdb
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
7e4a06c699
rocksdb/Makefile

701 lines
30 KiB
Makefile
Raw Normal View History

Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
# Copyright (c) 2011 The LevelDB Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file. See the AUTHORS file for names of contributors.
build shared libraries; updated version to 1.3; add Status accessors
2012-03-30 22:15:49 +02:00
# Inherit some settings from environment variables, if available
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
sync with upstream @21627589 Minor changes: * Reformat the bodies of the iterator interface routines in IteratorWrapper to make them a bit easier to read * Switched the default in the leveldb makefile to be optimized mode, rather than debug mode * Fix build problem in chromium port git-svn-id: https://leveldb.googlecode.com/svn/trunk@30 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-02 02:00:37 +02:00
#-----------------------------------------------
Compile -O2 by default and add `make dbg` Summary: To speed up the compilation while allowing us to compile in debug mode. Test Plan: make: see -O2 enabled make dbg: didn't see -O2 Reviewers: igor Reviewed By: igor CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D15969
2014-02-07 01:11:35 +01:00
ifneq ($(MAKECMDGOALS),dbg)
OPT += -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer
else
Get rid of all optimization flags in debug mode
2014-03-04 06:05:58 +01:00
# intentionally left blank
Compile -O2 by default and add `make dbg` Summary: To speed up the compilation while allowing us to compile in debug mode. Test Plan: make: see -O2 enabled make dbg: didn't see -O2 Reviewers: igor Reviewed By: igor CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D15969
2014-02-07 01:11:35 +01:00
endif
Fix some compilation bugs in different platforms Summary: detect some problems when testing my 3rd party release tool.
2014-02-28 07:15:30 +01:00
ifeq ($(MAKECMDGOALS),shared_lib)
RocksDBLite Summary: Introducing RocksDBLite! Removes all the non-essential features and reduces the binary size. This effort should help our adoption on mobile. Binary size when compiling for IOS (`TARGET_OS=IOS m static_lib`) is down to 9MB from 15MB (without stripping) Test Plan: compiles :) Reviewers: dhruba, haobo, ljin, sdong, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D17835
2014-04-15 22:39:26 +02:00
OPT += -DNDEBUG
endif
Don't compile sync_point if NDEBUG Summary: We don't really need sync_point.o if we're compiling with NDEBUG. This diff depends on D17823 Test Plan: compiles Reviewers: haobo, ljin, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17829
2014-04-17 19:49:58 +02:00
RocksDBLite Summary: Introducing RocksDBLite! Removes all the non-essential features and reduces the binary size. This effort should help our adoption on mobile. Binary size when compiling for IOS (`TARGET_OS=IOS m static_lib`) is down to 9MB from 15MB (without stripping) Test Plan: compiles :) Reviewers: dhruba, haobo, ljin, sdong, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D17835
2014-04-15 22:39:26 +02:00
ifeq ($(MAKECMDGOALS),static_lib)
OPT += -DNDEBUG
Fix some compilation bugs in different platforms Summary: detect some problems when testing my 3rd party release tool.
2014-02-28 07:15:30 +01:00
endif
RocksDBLite Summary: Introducing RocksDBLite! Removes all the non-essential features and reduces the binary size. This effort should help our adoption on mobile. Binary size when compiling for IOS (`TARGET_OS=IOS m static_lib`) is down to 9MB from 15MB (without stripping) Test Plan: compiles :) Reviewers: dhruba, haobo, ljin, sdong, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D17835
2014-04-15 22:39:26 +02:00
sync with upstream @21627589 Minor changes: * Reformat the bodies of the iterator interface routines in IteratorWrapper to make them a bit easier to read * Switched the default in the leveldb makefile to be optimized mode, rather than debug mode * Fix build problem in chromium port git-svn-id: https://leveldb.googlecode.com/svn/trunk@30 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-02 02:00:37 +02:00
#-----------------------------------------------
Platform detection during build, plus compatibility patches for machines without <cstdatomic>. This revision adds two major changes: 1. build_detect_platform which generates build_config.mk with platform-dependent flags for the build process 2. /port/atomic_pointer.h with anAtomicPointerimplementation for platforms without <cstdatomic> Some of this code is loosely based on patches submitted to the LevelDB mailing list at https://groups.google.com/forum/#!forum/leveldb Tip of the hat to Dave Smith and Edouard A, who both sent patches. The presence of Snappy (http://code.google.com/p/snappy/) and cstdatomic are now both detected in the build_detect_platform script (1.) which gets executing during make. For (2.), instead of broadly importing atomicops_* from Chromium or the Google performance tools, we chose to just implement AtomicPointer and the limited atomic load and store operations it needs. This resulted in much less code and fewer files - everything is contained in atomic_pointer.h. git-svn-id: https://leveldb.googlecode.com/svn/trunk@34 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-29 02:30:50 +02:00
# detect what platform we're building on
Fix building RocksDB in paths containing spaces -- quote path names in Makefile and build_detect_platform.
2014-05-11 06:01:25 +02:00
$(shell (export ROCKSDB_ROOT="$(CURDIR)"; "$(CURDIR)/build_tools/build_detect_platform" "$(CURDIR)/build_config.mk"))
Added bloom filter support. In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
2012-04-17 17:36:46 +02:00
# this file is generated by the previous line to set build flags and sources
Platform detection during build, plus compatibility patches for machines without <cstdatomic>. This revision adds two major changes: 1. build_detect_platform which generates build_config.mk with platform-dependent flags for the build process 2. /port/atomic_pointer.h with anAtomicPointerimplementation for platforms without <cstdatomic> Some of this code is loosely based on patches submitted to the LevelDB mailing list at https://groups.google.com/forum/#!forum/leveldb Tip of the hat to Dave Smith and Edouard A, who both sent patches. The presence of Snappy (http://code.google.com/p/snappy/) and cstdatomic are now both detected in the build_detect_platform script (1.) which gets executing during make. For (2.), instead of broadly importing atomicops_* from Chromium or the Google performance tools, we chose to just implement AtomicPointer and the limited atomic load and store operations it needs. This resulted in much less code and fewer files - everything is contained in atomic_pointer.h. git-svn-id: https://leveldb.googlecode.com/svn/trunk@34 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-29 02:30:50 +02:00
include build_config.mk
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
ifneq ($(PLATFORM), IOS)
CFLAGS += -g
CXXFLAGS += -g
else
# no debug info for IOS, that will make our library big
OPT += -DNDEBUG
endif
make asan_check Summary: Add asan_check rule to Makefile. After we add this, we will create Jenkins run that will check for asan errors! Test Plan: make asan_check Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14205
2013-11-20 01:33:24 +01:00
# ASAN doesn't work well with jemalloc. If we're compiling with ASAN, we should use regular malloc.
ifdef COMPILE_WITH_ASAN
# ASAN compile flags
EXEC_LDFLAGS += -fsanitize=address
PLATFORM_CCFLAGS += -fsanitize=address
PLATFORM_CXXFLAGS += -fsanitize=address
else
# if we're not compiling with ASAN, use jemalloc
EXEC_LDFLAGS := $(JEMALLOC_LIB) $(EXEC_LDFLAGS)
PLATFORM_CXXFLAGS += $(JEMALLOC_INCLUDE) -DHAVE_JEMALLOC
PLATFORM_CCFLAGS += $(JEMALLOC_INCLUDE) -DHAVE_JEMALLOC
endif
Add make install Summary: Add make install. If INSTALL_PATH is not set, then rocksdb will be installed under "/usr/local" directory (/usr/local/include for headers and /usr/local/lib for library file(s).) Test Plan: Develop a simple rocksdb app, called test.cc, and do the followings. make clean make static_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test sudo make uninstall make clean make shared_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test make INSTALL_PATH=/tmp/path install make INSTALL_PATH=/tmp/path uninstall and make sure things are installed / uninstalled in the specified path. Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23211
2014-09-12 05:41:02 +02:00
#-------------------------------------------------
# make install related stuff
INSTALL_PATH ?= /usr/local
uninstall:
Fix make install Summary: See https://github.com/facebook/rocksdb/issues/283 Test Plan: make install/uninstall Reviewers: ljin, sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23373
2014-09-16 00:30:17 +02:00
@rm -rf $(INSTALL_PATH)/include/rocksdb
@rm -rf $(INSTALL_PATH)/lib/$(LIBRARY)
@rm -rf $(INSTALL_PATH)/lib/$(SHARED)
Add make install Summary: Add make install. If INSTALL_PATH is not set, then rocksdb will be installed under "/usr/local" directory (/usr/local/include for headers and /usr/local/lib for library file(s).) Test Plan: Develop a simple rocksdb app, called test.cc, and do the followings. make clean make static_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test sudo make uninstall make clean make shared_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test make INSTALL_PATH=/tmp/path install make INSTALL_PATH=/tmp/path uninstall and make sure things are installed / uninstalled in the specified path. Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23211
2014-09-12 05:41:02 +02:00
install:
Fix make install Summary: See https://github.com/facebook/rocksdb/issues/283 Test Plan: make install/uninstall Reviewers: ljin, sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23373
2014-09-16 00:30:17 +02:00
@install -d $(INSTALL_PATH)/lib
@for header_dir in `find "include/rocksdb" -type d`; do \
install -d $(INSTALL_PATH)/$$header_dir; \
Add make install Summary: Add make install. If INSTALL_PATH is not set, then rocksdb will be installed under "/usr/local" directory (/usr/local/include for headers and /usr/local/lib for library file(s).) Test Plan: Develop a simple rocksdb app, called test.cc, and do the followings. make clean make static_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test sudo make uninstall make clean make shared_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test make INSTALL_PATH=/tmp/path install make INSTALL_PATH=/tmp/path uninstall and make sure things are installed / uninstalled in the specified path. Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23211
2014-09-12 05:41:02 +02:00
done
Fix make install Summary: See https://github.com/facebook/rocksdb/issues/283 Test Plan: make install/uninstall Reviewers: ljin, sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23373
2014-09-16 00:30:17 +02:00
@for header in `find "include/rocksdb" -type f -name *.h`; do \
install -C -m 644 $$header $(INSTALL_PATH)/$$header; \
done
@[ ! -e $(LIBRARY) ] || install -C -m 644 $(LIBRARY) $(INSTALL_PATH)/lib
@[ ! -e $(SHARED) ] || install -C -m 644 $(SHARED) $(INSTALL_PATH)/lib
Add make install Summary: Add make install. If INSTALL_PATH is not set, then rocksdb will be installed under "/usr/local" directory (/usr/local/include for headers and /usr/local/lib for library file(s).) Test Plan: Develop a simple rocksdb app, called test.cc, and do the followings. make clean make static_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test sudo make uninstall make clean make shared_lib -j32 sudo make install g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test ./test make INSTALL_PATH=/tmp/path install make INSTALL_PATH=/tmp/path uninstall and make sure things are installed / uninstalled in the specified path. Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23211
2014-09-12 05:41:02 +02:00
#-------------------------------------------------
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
CFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CCFLAGS) $(OPT)
Move -Wnon-virtual-dtor to c++ flags
2014-11-10 23:41:51 +01:00
CXXFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT) -Woverloaded-virtual -Wnon-virtual-dtor
Platform detection during build, plus compatibility patches for machines without <cstdatomic>. This revision adds two major changes: 1. build_detect_platform which generates build_config.mk with platform-dependent flags for the build process 2. /port/atomic_pointer.h with anAtomicPointerimplementation for platforms without <cstdatomic> Some of this code is loosely based on patches submitted to the LevelDB mailing list at https://groups.google.com/forum/#!forum/leveldb Tip of the hat to Dave Smith and Edouard A, who both sent patches. The presence of Snappy (http://code.google.com/p/snappy/) and cstdatomic are now both detected in the build_detect_platform script (1.) which gets executing during make. For (2.), instead of broadly importing atomicops_* from Chromium or the Google performance tools, we chose to just implement AtomicPointer and the limited atomic load and store operations it needs. This resulted in much less code and fewer files - everything is contained in atomic_pointer.h. git-svn-id: https://leveldb.googlecode.com/svn/trunk@34 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-29 02:30:50 +02:00
Build fixes and cleanups: (1) Separate out C++ and CC flags (fixes c_test compilation) (2) Move snappy/perftools detection to script (3) Fix db_bench_sqlite3 and db_bench_tree_db build rules
2012-03-21 18:28:03 +01:00
LDFLAGS += $(PLATFORM_LDFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Build fixes and cleanups: (1) Separate out C++ and CC flags (fixes c_test compilation) (2) Move snappy/perftools detection to script (3) Fix db_bench_sqlite3 and db_bench_tree_db build rules
2012-03-21 18:28:03 +01:00
LIBOBJECTS = $(SOURCES:.cc=.o)
MEMENVOBJECTS = $(MEMENV_SOURCES:.cc=.o)
TableMock + framework for mock classes Summary: This diff replaces BlockBasedTable in flush_job_test with TableMock, making it depend on less things and making it closer to an unit test than integration test. It also introduces a framework to compile mock classes -- Any file named *mock.cc will not be compiled into the build. It will only get compiled into the tests. What way we can mock out most other classes, Version, VersionSet, DBImpl, etc. Test Plan: flush_job_test Reviewers: ljin, rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D27681
2014-10-29 01:52:32 +01:00
MOCKOBJECTS = $(MOCK_SOURCES:.cc=.o)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
TESTUTIL = ./util/testutil.o
TableMock + framework for mock classes Summary: This diff replaces BlockBasedTable in flush_job_test with TableMock, making it depend on less things and making it closer to an unit test than integration test. It also introduces a framework to compile mock classes -- Any file named *mock.cc will not be compiled into the build. It will only get compiled into the tests. What way we can mock out most other classes, Version, VersionSet, DBImpl, etc. Test Plan: flush_job_test Reviewers: ljin, rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D27681
2014-10-29 01:52:32 +01:00
TESTHARNESS = ./util/testharness.o $(TESTUTIL) $(MOCKOBJECTS)
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
BENCHHARNESS = ./util/benchharness.o
Automating valgrind to run with jenkins Summary: The script valgrind_test.sh runs Valgrind for all tests in the makefile including leak-checks and outputs the logs for every test in a separate file with the name "valgrind_log_<testname>". It prints the failed tests in the file "valgrind_failed_tests". All these files are created in the directory "VALGRIND_LOGS" which can be changed in the Makefile. Finally it checks the line-count for the file "valgrind_failed_tests" and returns 0 if no tests failed and 1 otherwise. Test Plan: ./valgrind_test.sh; Changed the tests to incorporte leaks and verified correctness Reviewers: dhruba, sheki, MarkCallaghan Reviewed By: sheki CC: zshao Differential Revision: https://reviews.facebook.net/D8877
2013-02-25 20:17:26 +01:00
VALGRIND_ERROR = 2
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
VALGRIND_DIR = build_tools/VALGRIND_LOGS
Moving VALGRIND_VER which takes the valgrind version from third party to fbcode.gcc471.sh file Summary: the valgrind version being used is in facebook specific path and should be moved to the fbcode.gcc471.sh file instead of the makefile. The execution takes the environment's default valgrind version if the fbcode.gcc471.sh's valgrind_version is not available. Test Plan: make valgrind_check Reviewers: dhruba, sheki, akushner Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D9213
2013-03-07 20:11:30 +01:00
VALGRIND_VER := $(join $(VALGRIND_VER),valgrind)
Automating valgrind to run with jenkins Summary: The script valgrind_test.sh runs Valgrind for all tests in the makefile including leak-checks and outputs the logs for every test in a separate file with the name "valgrind_log_<testname>". It prints the failed tests in the file "valgrind_failed_tests". All these files are created in the directory "VALGRIND_LOGS" which can be changed in the Makefile. Finally it checks the line-count for the file "valgrind_failed_tests" and returns 0 if no tests failed and 1 otherwise. Test Plan: ./valgrind_test.sh; Changed the tests to incorporte leaks and verified correctness Reviewers: dhruba, sheki, MarkCallaghan Reviewed By: sheki CC: zshao Differential Revision: https://reviews.facebook.net/D8877
2013-02-25 20:17:26 +01:00
VALGRIND_OPTS = --error-exitcode=$(VALGRIND_ERROR) --leak-check=full
Titile: a command line shell to read/write data from a leveldb thrift server Summary: implemented a commond line shell to talk with leveldb thrift server, which is based on state pattern and can be easily extended. Test Plan: build and run Reviewers: dhruba, zshao, heyongqiang Differential Revision: https://reviews.facebook.net/D4713
2012-08-14 23:52:48 +02:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
TESTS = \
Reorder tests Summary: db_test should be the first to execute because it finds the most bugs. Also, when third parties report issues, we don't want ldb error message, we prefer to have db_test error message. For example, see thread: https://github.com/facebook/rocksdb/issues/25 Test Plan: make check Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14715
2013-12-18 22:37:06 +01:00
db_test \
Fix db_test Summary: Added deletion of DBIterators in DBIterator's tests Test Plan: make valgrind_check Reviewers: igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20043
2014-07-16 23:51:43 +02:00
db_iter_test \
Add a hash-index component for block Summary: this is the key component extracted from diff: https://reviews.facebook.net/D14271 I separate it to a dedicated patch to make the review easier. Test Plan: added a unit test and passed it. Reviewers: haobo, sdong, dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D16245
2014-02-18 23:58:55 +01:00
block_hash_index_test \
moving autovector_test after db_test
2014-01-02 12:30:29 +01:00
autovector_test \
[column families] Support to create and drop column families Summary: This diff provides basic implementations of CreateColumnFamily(), DropColumnFamily() and ListColumnFamilies(). It builds on top of https://reviews.facebook.net/D14733 It also includes a bug fix for DBImplReadOnly, where Get implementation would be redirected to DBImpl instead of DBImplReadOnly. Test Plan: Added unit test Reviewers: dhruba, haobo, kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D15021
2014-01-02 18:08:12 +01:00
column_family_test \
Improve the "table stats" Summary: The primary motivation of the changes is to make it easier to figure out the inside of the tables. * rename "table stats" to "table properties" since now we have more than "integers" to store in the property block. * Add filter block size to the basic table properties. * Whenever a table is built, we'll log the table properties (the sample output is in Test Plan). * Make an api to expose deleted keys. Test Plan: Passed all existing test. and the sample output of table stats: ================================================================== Basic Properties ------------------------------------------------------------------ # data blocks: 1 # entries: 1 raw key size: 9 raw average key size: 9 raw value size: 9 raw average value size: 0 data block size: 25 index block size: 27 filter block size: 18 (estimated) table size: 70 filter policy: rocksdb.BuiltinBloomFilter ================================================================== User collected properties: InternalKeyPropertiesCollector ------------------------------------------------------------------ kDeletedKeys: 1 ================================================================== Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14187
2013-11-20 01:29:42 +01:00
table_properties_collector_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
arena_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
auto_roll_logger_test \
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
benchharness_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
block_test \
Added bloom filter support. In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
2012-04-17 17:36:46 +02:00
bloom_test \
[RocksDB] [Performance Branch] Added dynamic bloom, to be used for memable non-existing key filtering Summary: as title Test Plan: dynamic_bloom_test Reviewers: dhruba, sdong, kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D14385
2013-11-27 23:27:02 +01:00
dynamic_bloom_test \
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
c_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
cache_test \
coding_test \
corruption_test \
crc32c_test \
dbformat_test \
env_test \
External Value Store Summary: Developing a capability for storing values on external backing file(s). This is just a highly unoptimized first pass - supports: 1) Allocating some portion of external file to be used to store value 2) Freeing the range, enabling it to be reused by other values As next steps, I plan to: 1) Create some kind of stress testing. Once I can measure stuff, I can focus on optimizing. 2) Optimize locking. 3) Optimize freelist data structure. Currently we have O(n) for both freeing and allocation. 4) Figure out how to do recovery. Test Plan: Created a unit test. Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13389
2013-10-17 02:33:49 +02:00
blob_store_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
filelock_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
filename_test \
Implement full filter for block based table. Summary: 1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file. 2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter. 3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type. 4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h. 5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc Benchmark: base commit 1d23b5c470844c1208301311f0889eca750431c0 Command: db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1 Read QPS increase for about 30% from 2230002 to 2991411. Test Plan: make all check valgrind db_test db_stress --use_block_based_filter = 0 ./auto_sanity_test.sh Reviewers: igor, yhchiang, ljin, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20979
2014-09-08 19:37:05 +02:00
block_based_filter_block_test \
full_filter_block_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
histogram_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
log_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
manual_compaction_test \
Sync with upstream @23860137. Fix GCC -Wshadow warnings in LevelDB's public header files, reported by Dustin. Add in-memory Env implementation (helpers/memenv/*). This enables users to create LevelDB databases in-memory. Initialize ShardedLRUCache::last_id_ to zero. This fixes a Valgrind warning. (Also delete port/sha1_* which were removed upstream some time ago.)
2011-09-12 11:21:10 +02:00
memenv_test \
introduce TestMemEnv and use it in db_test Summary: TestMemEnv simulates all Env APIs using in-memory data structures. We can use it to speed up db_test run, which is now reduced ~7mins when it is enabled. We can also add features to simulate power/disk failures in the next step TestMemEnv is derived from helper/mem_env mem_env can not be used for rocksdb since some of its APIs do not give the same results as env_posix. And its file read/write is not thread safe Test Plan: make all -j32 ./db_test ./env_mem_test Reviewers: sdong, yhchiang, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28035
2014-10-31 23:08:10 +01:00
mock_env_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
merge_test \
Merger test Summary: I abandoned https://reviews.facebook.net/D18789, but I wrote a good unit test there, so let's check it in. :) Test Plan: this is test Reviewers: sdong, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22827
2014-09-09 07:24:40 +02:00
merger_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
redis_test \
reduce_levels_test \
A Simple Plain Table Summary: A Simple plain table format. No block structure. When creating the table reader, scanning the full table to create indexes. Test Plan:Add unit test Reviewers:haobo,dhruba,kailiu CC: Task ID: # Blame Rev:
2013-10-29 04:34:02 +01:00
plain_table_db_test \
Add ComparatorDBTest to test non-default comparators Summary: Add some helper functions to make sure DB works well for non-default comparators. Add a test for SimpleSuffixReverseComparator. Test Plan: Run the new test Reviewers: ljin, rven, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D27831
2014-10-29 21:49:45 +01:00
comparator_db_test \
fix db_sanity_test
2014-04-01 02:06:53 +02:00
prefix_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
skiplist_test \
Implement log blobs Summary: This patch adds the ability for the user to add sequences of arbitrary data (blobs) to write batches. These blobs are saved to the log along with everything else in the write batch. You can add multiple blobs per WriteBatch and the ordering of blobs, puts, merges, and deletes are preserved. Blobs are not saves to SST files. RocksDB ignores blobs in every way except for writing them to the log. Before committing this patch, I need to add some test code. But I'm submitting it now so people can comment on the API. Test Plan: make -j32 check Reviewers: dhruba, haobo, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12195
2013-08-15 01:32:46 +02:00
stringappend_test \
Timestamp and TTL Wrapper for rocksdb Summary: When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open. Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest Reviewed By: vamsi CC: zshao, xjin, vkrest, MarkCallaghan Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 22:33:13 +02:00
ttl_test \
[RocksDB] BackupableDB Summary: In this diff I present you BackupableDB v1. You can easily use it to backup your DB and it will do incremental snapshots for you. Let's first describe how you would use BackupableDB. It's inheriting StackableDB interface so you can easily construct it with your DB object -- it will add a method RollTheSnapshot() to the DB object. When you call RollTheSnapshot(), current snapshot of the DB will be stored in the backup dir. To restore, you can just call RestoreDBFromBackup() on a BackupableDB (which is a static method) and it will restore all files from the backup dir. In the next version, it will even support automatic backuping every X minutes. There are multiple things you can configure: 1. backup_env and db_env can be different, which is awesome because then you can easily backup to HDFS or wherever you feel like. 2. sync - if true, it *guarantees* backup consistency on machine reboot 3. number of snapshots to keep - this will keep last N snapshots around if you want, for some reason, be able to restore from an earlier snapshot. All the backuping is done in incremental fashion - if we already have 00010.sst, we will not copy it again. *IMPORTANT* -- This is based on assumption that 00010.sst never changes - two files named 00010.sst from the same DB will always be exactly the same. Is this true? I always copy manifest, current and log files. 4. You can decide if you want to flush the memtables before you backup, or you're fine with backing up the log files -- either way, you get a complete and consistent view of the database at a time of backup. 5. More things you can find in BackupableDBOptions Here is the directory structure I use: backup_dir/CURRENT_SNAPSHOT - just 4 bytes holding the latest snapshot 0, 1, 2, ... - files containing serialized version of each snapshot - containing a list of files files/*.sst - sst files shared between snapshots - if one snapshot references 00010.sst and another one needs to backup it from the DB, it will just reference the same file files/ 0/, 1/, 2/, ... - snapshot directories containing private snapshot files - current, manifest and log files All the files are ref counted and deleted immediatelly when they get out of scope. Some other stuff in this diff: 1. Added GetEnv() method to the DB. Discussed with @haobo and we agreed that it seems right thing to do. 2. Fixed StackableDB interface. The way it was set up before, I was not able to implement BackupableDB. Test Plan: I have a unittest, but please don't look at this yet. I just hacked it up to help me with debugging. I will write a lot of good tests and update the diff. Also, `make asan_check` Reviewers: dhruba, haobo, emayanke Reviewed By: dhruba CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14295
2013-12-09 23:06:52 +01:00
backupable_db_test \
JSON (Document) API sketch Summary: This is a rough sketch of our new document API. Would like to get some thoughts and comments about the high-level architecture and API. I didn't optimize for performance at all. Leaving some low-hanging fruit so that we can be happy when we fix them! :) Currently, bunch of features are not supported at all. Indexes can be only specified when creating database. There is no query planner whatsoever. This will all be added in due time. Test Plan: Added a simple unit test Reviewers: haobo, yhchiang, dhruba, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18747
2014-07-10 18:31:42 +02:00
document_db_test \
JSONDocument Summary: After evaluating options for JSON storage, I decided to implement our own. The reason is that we'll be able to optimize it better and we get to reduce unnecessary dependencies (which is what we'd get with folly). I also plan to write a serializer/deserializer for JSONDocument with our own binary format similar to BSON. That way we'll store binary JSON format in RocksDB instead of the plain-text JSON. This means less storage and faster deserialization. There are still some inefficiencies left here. I plan to optimize them after we develop a functioning DocumentDB. That way we can move and iterate faster. Test Plan: added a unit test Reviewers: dhruba, haobo, sdong, ljin, yhchiang Reviewed By: haobo Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18831
2014-06-20 11:14:14 +02:00
json_document_test \
SpatialDB Summary: This diff is adding spatial index support to RocksDB. When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels. Each element inserted into SpatialDB has: * a bounding box, which determines how will the element be indexed * string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text) * feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar) * a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level). Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements. Test Plan: Added bunch of unit tests in spatial_db_test Reviewers: dhruba, yinwang Reviewed By: yinwang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
spatial_db_test \
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
version_edit_test \
A number of smaller fixes and performance improvements: - Implemented Get() directly instead of building on top of a full merging iterator stack. This speeds up the "readrandom" benchmark by up to 15-30%. - Fixed an opensource compilation problem. Added --db=<name> flag to control where the database is placed. - Automatically compact a file when we have done enough overlapping seeks to that file. - Fixed a performance bug where we would read from at least one file in a level even if none of the files overlapped the key being read. - Makefile fix for Mac OSX installations that have XCode 4 without XCode 3. - Unified the two occurrences of binary search in a file-list into one routine. - Found and fixed a bug where we would unnecessarily search the last file when looking for a key larger than all data in the level. - A fix to avoid the need for trivial move compactions and therefore gets rid of two out of five syncs in "fillseq". - Removed the MANIFEST file write when switching to a new memtable/log-file for a 10-20% improvement on fill speed on ext4. - Adding a SNAPPY setting in the Makefile for folks who have Snappy installed. Snappy compresses values and speeds up writes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@32 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-22 04:36:45 +02:00
version_set_test \
fix c_test Summary: as title Test Plan: ./c_test Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28119
2014-10-31 23:04:01 +01:00
compaction_picker_test \
Make VersionBuilder unit testable Summary: Rename Version::Builder to VersionBuilder and expose its definition to a header. Make VerisonBuilder not reference Version or ColumnFamilyData, only working with VersionStorageInfo. Add version_builder_test which has a simple test. Test Plan: make all check Reviewers: rven, yhchiang, igor, ljin Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D27969
2014-10-31 16:48:19 +01:00
version_builder_test \
hints for narrowing down FindFile range and avoiding checking unrelevant L0 files Summary: The file tree structure in Version is prebuilt and the range of each file is known. On the Get() code path, we do binary search in FindFile() by comparing target key with each file's largest key and also check the range for each L0 file. With some pre-calculated knowledge, each key comparision that has been done can serve as a hint to narrow down further searches: (1) If a key falls within a L0 file's range, we can safely skip the next file if its range does not overlap with the current one. (2) If a key falls within a file's range in level L0 - Ln-1, we should only need to binary search in the next level for files that overlap with the current one. (1) will be able to skip some files depending one the key distribution. (2) can greatly reduce the range of binary search, especially for bottom levels, given that one file most likely only overlaps with N files from the level below (where N is max_bytes_for_level_multiplier). So on level L, we will only look at ~N files instead of N^L files. Some inital results: measured with 500M key DB, when write is light (10k/s = 1.2M/s), this improves QPS ~7% on top of blocked bloom. When write is heavier (80k/s = 9.6M/s), it gives us ~13% improvement. Test Plan: make all check Reviewers: haobo, igor, dhruba, sdong, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17205
2014-04-21 18:10:12 +02:00
file_indexer_test \
Push- instead of pull-model for managing Write stalls Summary: Introducing WriteController, which is a source of truth about per-DB write delays. Let's define an DB epoch as a period where there are no flushes and compactions (i.e. new epoch is started when flush or compaction finishes). Each epoch can either: * proceed with all writes without delay * delay all writes by fixed time * stop all writes The three modes are recomputed at each epoch change (flush, compaction), rather than on every write (which is currently the case). When we have a lot of column families, our current pull behavior adds a big overhead, since we need to loop over every column family for every write. With new push model, overhead on Write code-path is minimal. This is just the start. Next step is to also take care of stalls introduced by slow memtable flushes. The final goal is to eliminate function MakeRoomForWrite(), which currently needs to be called for every column family by every write. Test Plan: make check for now. I'll add some unit tests later. Also, perf test. Reviewers: dhruba, yhchiang, MarkCallaghan, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22791
2014-09-08 20:20:25 +02:00
write_batch_test \
write_controller_test\
Improve the "table stats" Summary: The primary motivation of the changes is to make it easier to figure out the inside of the tables. * rename "table stats" to "table properties" since now we have more than "integers" to store in the property block. * Add filter block size to the basic table properties. * Whenever a table is built, we'll log the table properties (the sample output is in Test Plan). * Make an api to expose deleted keys. Test Plan: Passed all existing test. and the sample output of table stats: ================================================================== Basic Properties ------------------------------------------------------------------ # data blocks: 1 # entries: 1 raw key size: 9 raw average key size: 9 raw value size: 9 raw average value size: 0 data block size: 25 index block size: 27 filter block size: 18 (estimated) table size: 70 filter policy: rocksdb.BuiltinBloomFilter ================================================================== User collected properties: InternalKeyPropertiesCollector ------------------------------------------------------------------ kDeletedKeys: 1 ================================================================== Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14187
2013-11-20 01:29:42 +01:00
deletefile_test \
thread local pointer storage Summary: This is not a generic thread local implementation in the sense that it only takes pointer. But it does support multiple instances per thread and lets user plugin function to perform cleanup when thread exits or an instance gets destroyed. Test Plan: unit test for now Reviewers: haobo, igor, sdong, dhruba Reviewed By: igor CC: leveldb, kailiu Differential Revision: https://reviews.facebook.net/D16131
2014-02-26 02:47:37 +01:00
table_test \
A GIS implementation for rocksdb. Summary: This patch stores gps locations in rocksdb. Each object is uniquely identified by an id. Each object has a gps (latitude, longitude) associated with it. The geodb supports looking up an object either by its gps location or by its id. There is a method to retrieve all objects within a circular radius centered at a specified gps location. Test Plan: Simple unit-test attached. Reviewers: leveldb, haobo Reviewed By: haobo CC: leveldb, tecbot, haobo Differential Revision: https://reviews.facebook.net/D15567
2013-12-21 02:17:00 +01:00
thread_local_test \
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
geodb_test \
remove tailing_iter.h/cc Summary: as title Test Plan: make all check ran db_bench and saw seek stats at the end Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21651
2014-08-13 02:13:15 +02:00
rate_limiter_test \
Implementation of CuckooTableReader Summary: Contains: - Implementation of TableReader based on Cuckoo Hashing - Unittests for CuckooTableReader - Performance test for TableReader Test Plan: make cuckoo_table_reader_test ./cuckoo_table_reader_test make valgrind_check make asan_check Reviewers: yhchiang, sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20511
2014-07-26 01:37:32 +02:00
options_test \
Integrating Cuckoo Hash SST Table format into RocksDB Summary: Contains the following changes: - Implementation of cuckoo_table_factory - Adding cuckoo table into AdaptiveTableFactory - Adding cuckoo_table_db_test, similar to lines of plain_table_db_test - Minor fixes to Reader: When a key is found in the table, return the key found instead of the search key. - Minor fixes to Builder: Add table properties that are required by Version::UpdateTemporaryStats() during Get operation. Don't define curr_node as a reference variable as the memory locations may get reassigned during tree.push_back operation, leading to invalid memory access. Test Plan: cuckoo_table_reader_test --enable_perf cuckoo_table_builder_test cuckoo_table_db_test make check all make valgrind_check make asan_check Reviewers: sdong, igor, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21219
2014-08-12 05:21:07 +02:00
cuckoo_table_builder_test \
cuckoo_table_reader_test \
WriteBatchWithIndex: a wrapper of WriteBatch, with a searchable index Summary: Add WriteBatchWithIndex so that a user can query data out of a WriteBatch, to support MongoDB's read-its-own-write. WriteBatchWithIndex uses a skiplist to store the binary index. The index stores the offset of the entry in the write batch. When searching for a key, the key for the entry is read by read the entry from the write batch from the offset. Define a new iterator class for querying data out of WriteBatchWithIndex. A user can create an iterator of the write batch for one column family, seek to a key and keep calling Next() to see next entries. I will add more unit tests if people are OK about this API. Test Plan: make all check Add unit tests. Reviewers: yhchiang, igor, MarkCallaghan, ljin Reviewed By: ljin Subscribers: dhruba, leveldb, xjin Differential Revision: https://reviews.facebook.net/D21381
2014-08-19 00:19:17 +02:00
cuckoo_table_db_test \
WalManager Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety. Test Plan: added test Reviewers: ljin, yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D27873
2014-10-30 01:43:37 +01:00
flush_job_test \
CompactFiles, EventListener and GetDatabaseMetaData Summary: This diff adds three sets of APIs to RocksDB. = GetColumnFamilyMetaData = * This APIs allow users to obtain the current state of a RocksDB instance on one column family. * See GetColumnFamilyMetaData in include/rocksdb/db.h = EventListener = * A virtual class that allows users to implement a set of call-back functions which will be called when specific events of a RocksDB instance happens. * To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners = CompactFiles = * CompactFiles API inputs a set of file numbers and an output level, and RocksDB will try to compact those files into the specified level. = Example = * Example code can be found in example/compact_files_example.cc, which implements a simple external compactor using EventListener, GetColumnFamilyMetaData, and CompactFiles API. Test Plan: listener_test compactor_test example/compact_files_example export ROCKSDB_TESTS=CompactFiles db_test export ROCKSDB_TESTS=MetaData db_test Reviewers: ljin, igor, rven, sdong Reviewed By: sdong Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
wal_manager_test \
listener_test \
write_batch_with_index_test
Fix the "IO error" in auto_roll_logger_test Summary: I missed InitTestDb() in one of my tess. InitTestDb() initializes the test directory, without which the test will throw IO error. This problem didn't occur before because I've already run the tests before so the test directory is already there. Test Plan: Reviewers: dhruba CC: Task ID: # Blame Rev:
2013-02-19 09:09:16 +01:00
Utility to dump manifest contents. Summary: ./manifest_dump --file=/tmp/dbbench/MANIFEST-000002 Output looks like manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28 prev_log_number 0 --- level 0 --- --- level 1 --- --- level 2 --- 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1] 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1] 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1] 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1] 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1] 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1] 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1] 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1] 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1] 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1] 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1] 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1] 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1] --- level 3 --- --- level 4 --- --- level 5 --- --- level 6 --- Test Plan: run on test directory created by dbbench Reviewers: heyongqiang Reviewed By: heyongqiang CC: hustliubo Differential Revision: https://reviews.facebook.net/D4743
2012-08-17 19:48:40 +02:00
TOOLS = \
Revert "Don't compile ldb tool into static library" This reverts commit e296577ef64deac036a275a1a4c0d6172cfa42df.
2014-04-15 20:29:02 +02:00
sst_dump \
fix db_sanity_test
2014-04-01 02:06:53 +02:00
db_sanity_test \
Revert "Don't compile ldb tool into static library" This reverts commit e296577ef64deac036a275a1a4c0d6172cfa42df.
2014-04-15 20:29:02 +02:00
db_stress \
ldb \
make asan_check Summary: Add asan_check rule to Makefile. After we add this, we will create Jenkins run that will check for asan errors! Test Plan: make asan_check Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14205
2013-11-20 01:33:24 +01:00
db_repl_stress \
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
options_test \
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
blob_store_bench
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
perf_context.get_from_output_files_time is set for MultiGet() and ReadOnly DB too. Summary: perf_context.get_from_output_files_time is now only set writable DB's DB::Get(). Extend it to MultiGet() and read only DB. Test Plan: make all check Fix perf_context_test and extend it to cover MultiGet(), as long as read-only DB. Run it and watch the results Reviewers: ljin, yhchiang, igor Reviewed By: igor Subscribers: rven, leveldb Differential Revision: https://reviews.facebook.net/D24207
2014-10-03 02:02:30 +02:00
PROGRAMS = db_bench signal_test table_reader_bench log_and_apply_bench cache_bench perf_context_test $(TOOLS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Improve the build files to simplify the 3rd party release process Summary: * Added LIBNAME to enable configurable library name. * remove/check fPIC in linux platform from build_detect_platform Test Plan: make Reviewers: emayanke Differential Revision: https://reviews.facebook.net/D12321
2013-08-16 05:53:21 +02:00
# The library name is configurable since we are maintaining libraries of both
# debug/release mode.
LIBNAME in Makefile is not really configurable Summary: In new third-party release tool, `LIBNAME=<customized_library> make` will not really change the LIBNAME. However it's very odd that the same approach works with old third-party release tools. I checked previous rocksdb version and both librocksdb.a and librocksdb_debug.a were correctly generated and copied to the right place. Test Plan: `LIBNAME=hello make -j32` generates hello.a `make -j32` generates librocksdb.a Reviewers: igor, sdong, haobo, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D15555
2014-01-29 20:35:05 +01:00
ifeq ($(LIBNAME),)
LIBNAME=librocksdb
endif
Improve the build files to simplify the 3rd party release process Summary: * Added LIBNAME to enable configurable library name. * remove/check fPIC in linux platform from build_detect_platform Test Plan: make Reviewers: emayanke Differential Revision: https://reviews.facebook.net/D12321
2013-08-16 05:53:21 +02:00
LIBRARY = ${LIBNAME}.a
Sync with upstream @23860137. Fix GCC -Wshadow warnings in LevelDB's public header files, reported by Dustin. Add in-memory Env implementation (helpers/memenv/*). This enables users to create LevelDB databases in-memory. Initialize ShardedLRUCache::last_id_ to zero. This fixes a Valgrind warning. (Also delete port/sha1_* which were removed upstream some time ago.)
2011-09-12 11:21:10 +02:00
MEMENVLIBRARY = libmemenv.a
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
add major minor micro version to java jars
2014-10-02 20:07:45 +02:00
ROCKSDB_MAJOR = $(shell egrep "ROCKSDB_MAJOR.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
ROCKSDB_MINOR = $(shell egrep "ROCKSDB_MINOR.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
ROCKSDB_PATCH = $(shell egrep "ROCKSDB_PATCH.[0-9]" include/rocksdb/version.h | cut -d ' ' -f 3)
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
build shared libraries; updated version to 1.3; add Status accessors
2012-03-30 22:15:49 +02:00
default: all
Improve the comment for the shared library in Make file
2013-10-23 07:38:49 +02:00
#-----------------------------------------------
# Create platform independent shared libraries.
#-----------------------------------------------
build shared libraries; updated version to 1.3; add Status accessors
2012-03-30 22:15:49 +02:00
ifneq ($(PLATFORM_SHARED_EXT),)
merge 1.5 Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
2012-08-27 08:45:35 +02:00
ifneq ($(PLATFORM_SHARED_VERSIONED),true)
Improve the build files to simplify the 3rd party release process Summary: * Added LIBNAME to enable configurable library name. * remove/check fPIC in linux platform from build_detect_platform Test Plan: make Reviewers: emayanke Differential Revision: https://reviews.facebook.net/D12321
2013-08-16 05:53:21 +02:00
SHARED1 = ${LIBNAME}.$(PLATFORM_SHARED_EXT)
merge 1.5 Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
2012-08-27 08:45:35 +02:00
SHARED2 = $(SHARED1)
SHARED3 = $(SHARED1)
Don't keep managing two rocksdb version Summary: Before this diff, there are two places with rocksdb versions. After the diff: 1. we only have one source of truth for rocksdb version 2. we have a script that we can use to get the version that we can use in other compilations (java, go, etc). Test Plan: make Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24333
2014-10-02 20:59:22 +02:00
SHARED4 = $(SHARED1)
merge 1.5 Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
2012-08-27 08:45:35 +02:00
SHARED = $(SHARED1)
else
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
SHARED_MAJOR = $(ROCKSDB_MAJOR)
SHARED_MINOR = $(ROCKSDB_MINOR)
Don't keep managing two rocksdb version Summary: Before this diff, there are two places with rocksdb versions. After the diff: 1. we only have one source of truth for rocksdb version 2. we have a script that we can use to get the version that we can use in other compilations (java, go, etc). Test Plan: make Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24333
2014-10-02 20:59:22 +02:00
SHARED_PATCH = $(ROCKSDB_PATCH)
Improve the build files to simplify the 3rd party release process Summary: * Added LIBNAME to enable configurable library name. * remove/check fPIC in linux platform from build_detect_platform Test Plan: make Reviewers: emayanke Differential Revision: https://reviews.facebook.net/D12321
2013-08-16 05:53:21 +02:00
SHARED1 = ${LIBNAME}.$(PLATFORM_SHARED_EXT)
build shared libraries; updated version to 1.3; add Status accessors
2012-03-30 22:15:49 +02:00
SHARED2 = $(SHARED1).$(SHARED_MAJOR)
SHARED3 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR)
Don't keep managing two rocksdb version Summary: Before this diff, there are two places with rocksdb versions. After the diff: 1. we only have one source of truth for rocksdb version 2. we have a script that we can use to get the version that we can use in other compilations (java, go, etc). Test Plan: make Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24333
2014-10-02 20:59:22 +02:00
SHARED4 = $(SHARED1).$(SHARED_MAJOR).$(SHARED_MINOR).$(SHARED_PATCH)
SHARED = $(SHARED1) $(SHARED2) $(SHARED3) $(SHARED4)
$(SHARED1): $(SHARED4)
ln -fs $(SHARED4) $(SHARED1)
$(SHARED2): $(SHARED4)
ln -fs $(SHARED4) $(SHARED2)
$(SHARED3): $(SHARED4)
ln -fs $(SHARED4) $(SHARED3)
build shared libraries; updated version to 1.3; add Status accessors
2012-03-30 22:15:49 +02:00
endif
Don't keep managing two rocksdb version Summary: Before this diff, there are two places with rocksdb versions. After the diff: 1. we only have one source of truth for rocksdb version 2. we have a script that we can use to get the version that we can use in other compilations (java, go, etc). Test Plan: make Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24333
2014-10-02 20:59:22 +02:00
$(SHARED4):
Update Makefile
2014-04-29 23:29:45 +02:00
$(CXX) $(PLATFORM_SHARED_LDFLAGS)$(SHARED2) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) $(SOURCES) $(LDFLAGS) -o $@
merge 1.5 Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
2012-08-27 08:45:35 +02:00
endif # PLATFORM_SHARED_EXT
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
.PHONY: blackbox_crash_test check clean coverage crash_test ldb_tests package \
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
release tags valgrind_check whitebox_crash_test format static_lib shared_lib all \
Resolving merge conflict
2014-09-24 01:00:54 +02:00
dbg rocksdbjavastatic rocksdbjava install uninstall
Compile -O2 by default and add `make dbg` Summary: To speed up the compilation while allowing us to compile in debug mode. Test Plan: make: see -O2 enabled make dbg: didn't see -O2 Reviewers: igor Reviewed By: igor CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D15969
2014-02-07 01:11:35 +01:00
Fix compile issues when doing make release
2014-04-16 00:59:34 +02:00
all: $(LIBRARY) $(PROGRAMS) $(TESTS)
Stop continually re-creating build_version.c Summary: We continually rebuilt build_version.c because we put the current date into it, but that's what __DATE__ already is. This makes builds faster. This also fixes an issue with 'make clean FOO' not working properly. Also tweak the build rules to be more consistent, always have warnings, and add a 'make release' rule to handle flags for release builds. Test Plan: make, make clean Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D8139
2013-01-24 20:45:11 +01:00
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
static_lib: $(LIBRARY)
shared_lib: $(SHARED)
Fix compile issues when doing make release
2014-04-16 00:59:34 +02:00
dbg: $(LIBRARY) $(PROGRAMS) $(TESTS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
Fix compile issues when doing make release
2014-04-16 00:59:34 +02:00
# creates static library and programs
Stop continually re-creating build_version.c Summary: We continually rebuilt build_version.c because we put the current date into it, but that's what __DATE__ already is. This makes builds faster. This also fixes an issue with 'make clean FOO' not working properly. Also tweak the build rules to be more consistent, always have warnings, and add a 'make release' rule to handle flags for release builds. Test Plan: make, make clean Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D8139
2013-01-24 20:45:11 +01:00
release:
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(MAKE) clean
Fix compile issues when doing make release
2014-04-16 00:59:34 +02:00
OPT="-DNDEBUG -O2" $(MAKE) static_lib $(PROGRAMS) -j32
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
coverage:
$(MAKE) clean
Disable the html-based coverage report by default
2014-02-06 09:11:18 +01:00
COVERAGEFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS+="-lgcov" $(MAKE) all check -j32
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
(cd coverage; ./coverage_test.sh)
# Delete intermediate files
Fixing the warning messages captured under mac os # Consider using `git commit -m 'One line title' && arc diff`. # You will save time by running lint and unit in the background. Summary: The work to make sure mac os compiles rocksdb is not completed yet. But at least we can start cleaning some warnings captured only by g++ from mac os.. Test Plan: ran make in mac os Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14049
2013-11-13 05:05:28 +01:00
find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Better INSTALL.md and Makefile rules Summary: We have a lot of problems with gflags. However, when compiling rocksdb static library, we don't need gflags dependency. Reorganize INSTALL.md such that first-time customers don't need any dependency installed to actually build rocksdb static library. Test Plan: none Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18501
2014-05-08 01:51:30 +02:00
check: $(TESTS) ldb
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
for t in $(TESTS); do echo "***** Running $$t"; ./$$t || exit 1; done
Reorder tests Summary: db_test should be the first to execute because it finds the most bugs. Also, when third parties report issues, we don't want ldb error message, we prefer to have db_test error message. For example, see thread: https://github.com/facebook/rocksdb/issues/25 Test Plan: make check Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14715
2013-12-18 22:37:06 +01:00
python tools/ldb_test.py
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Better INSTALL.md and Makefile rules Summary: We have a lot of problems with gflags. However, when compiling rocksdb static library, we don't need gflags dependency. Reorganize INSTALL.md such that first-time customers don't need any dependency installed to actually build rocksdb static library. Test Plan: none Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18501
2014-05-08 01:51:30 +02:00
ldb_tests: ldb
Enhanced ldb to support data access commands Summary: Added put/get/scan/batchput/delete/approxsize Test Plan: Added pyunit script to test the newly added commands Reviewers: chip, leveldb Reviewed By: chip CC: zshao, emayanke Differential Revision: https://reviews.facebook.net/D7947
2013-01-11 20:09:23 +01:00
python tools/ldb_test.py
Run whitebox test before black box
2014-04-24 18:28:11 +02:00
crash_test: whitebox_crash_test blackbox_crash_test
[Make most of the changes suggested by Aaron] Summary: $title Test Plan: Revert Plan: OK Task ID: # Reviewers: emayanke, akushner Reviewed By: akushner Differential Revision: https://reviews.facebook.net/D10923
2013-05-24 04:10:13 +02:00
blackbox_crash_test: db_stress
[Fix whilebox crash test failure] Summary: I think the check for "error" that I added had caused false alarm. Fixed that. Test Plan: Revert Plan: OK Task ID: # Reviewers: emayanke, dhruba Reviewed By: emayanke Differential Revision: https://reviews.facebook.net/D11139
2013-06-07 20:06:20 +02:00
python -u tools/db_crashtest.py
[Make most of the changes suggested by Aaron] Summary: $title Test Plan: Revert Plan: OK Task ID: # Reviewers: emayanke, akushner Reviewed By: akushner Differential Revision: https://reviews.facebook.net/D10923
2013-05-24 04:10:13 +02:00
whitebox_crash_test: db_stress
[Fix whilebox crash test failure] Summary: I think the check for "error" that I added had caused false alarm. Fixed that. Test Plan: Revert Plan: OK Task ID: # Reviewers: emayanke, dhruba Reviewed By: emayanke Differential Revision: https://reviews.facebook.net/D11139
2013-06-07 20:06:20 +02:00
python -u tools/db_crashtest2.py
Invoke crash test from the Makefile Summary: make crash_test will now invoke the crash_test. Also some cleanup in the db_crashtest.py file Test Plan: make crash_test Reviewers: akushner, vamsi, sheki, dhruba Reviewed By: vamsi Differential Revision: https://reviews.facebook.net/D9987
2013-04-05 22:44:59 +02:00
make asan_check Summary: Add asan_check rule to Makefile. After we add this, we will create Jenkins run that will check for asan errors! Test Plan: make asan_check Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14205
2013-11-20 01:33:24 +01:00
asan_check:
$(MAKE) clean
COMPILE_WITH_ASAN=1 $(MAKE) check -j32
Split asan_check into asan_check and asan_crash_test
2013-11-20 01:44:40 +01:00
$(MAKE) clean
asan_crash_test:
$(MAKE) clean
Single-threaded asan_crash_test
2014-04-22 00:42:04 +02:00
COMPILE_WITH_ASAN=1 $(MAKE) crash_test
make asan_check Summary: Add asan_check rule to Makefile. After we add this, we will create Jenkins run that will check for asan errors! Test Plan: make asan_check Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14205
2013-11-20 01:33:24 +01:00
$(MAKE) clean
Adding a rule in the Makefile to run valgrind on the rocksdb tests Summary: Added automated valgrind testing for rocksdb by adding valgrind_check in the Makefile Test Plan: make clean; make all check Reviewers: dhruba, sheki, MarkCallaghan, zshao Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D8787
2013-02-22 02:10:33 +01:00
valgrind_check: all $(PROGRAMS) $(TESTS)
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
mkdir -p $(VALGRIND_DIR)
Automating valgrind to run with jenkins Summary: The script valgrind_test.sh runs Valgrind for all tests in the makefile including leak-checks and outputs the logs for every test in a separate file with the name "valgrind_log_<testname>". It prints the failed tests in the file "valgrind_failed_tests". All these files are created in the directory "VALGRIND_LOGS" which can be changed in the Makefile. Finally it checks the line-count for the file "valgrind_failed_tests" and returns 0 if no tests failed and 1 otherwise. Test Plan: ./valgrind_test.sh; Changed the tests to incorporte leaks and verified correctness Reviewers: dhruba, sheki, MarkCallaghan Reviewed By: sheki CC: zshao Differential Revision: https://reviews.facebook.net/D8877
2013-02-25 20:17:26 +01:00
echo TESTS THAT HAVE VALGRIND ERRORS > $(VALGRIND_DIR)/valgrind_failed_tests; \
echo TIMES in seconds TAKEN BY TESTS ON VALGRIND > $(VALGRIND_DIR)/valgrind_tests_times; \
for t in $(filter-out skiplist_test,$(TESTS)); do \
stime=`date '+%s'`; \
Use version 3.8.1 for valgrind in third_party and do away with log files Summary: valgrind 3.7.0 used currently has a bug that needs LD_PRELOAD being set as a workaround. This caused problems when run on jenkins. 3.8.1 has fixed this issue and we should use it from third party Also, have done away with log files. The whole output will be there on the terminal and the failed tests will be listed at the end. This is done because jenkins only lets us download the different files and not view them in the browser which is undesirable. Test Plan: make valgrind_check Reviewers: akushner, dhruba, vamsi, sheki, heyongqiang Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D9171
2013-03-07 00:12:38 +01:00
$(VALGRIND_VER) $(VALGRIND_OPTS) ./$$t; \
Automating valgrind to run with jenkins Summary: The script valgrind_test.sh runs Valgrind for all tests in the makefile including leak-checks and outputs the logs for every test in a separate file with the name "valgrind_log_<testname>". It prints the failed tests in the file "valgrind_failed_tests". All these files are created in the directory "VALGRIND_LOGS" which can be changed in the Makefile. Finally it checks the line-count for the file "valgrind_failed_tests" and returns 0 if no tests failed and 1 otherwise. Test Plan: ./valgrind_test.sh; Changed the tests to incorporte leaks and verified correctness Reviewers: dhruba, sheki, MarkCallaghan Reviewed By: sheki CC: zshao Differential Revision: https://reviews.facebook.net/D8877
2013-02-25 20:17:26 +01:00
if [ $$? -eq $(VALGRIND_ERROR) ] ; then \
echo $$t >> $(VALGRIND_DIR)/valgrind_failed_tests; \
fi; \
etime=`date '+%s'`; \
echo $$t $$((etime - stime)) >> $(VALGRIND_DIR)/valgrind_tests_times; \
done
Adding a rule in the Makefile to run valgrind on the rocksdb tests Summary: Added automated valgrind testing for rocksdb by adding valgrind_check in the Makefile Test Plan: make clean; make all check Reviewers: dhruba, sheki, MarkCallaghan, zshao Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D8787
2013-02-22 02:10:33 +01:00
remove tailing_iter.h/cc Summary: as title Test Plan: make all check ran db_bench and saw seek stats at the end Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21651
2014-08-13 02:13:15 +02:00
unity.cc:
Changes to support unity build: * Script for building the unity.cc file via Makefile * Unity executable Makefile target for testing builds * Source code changes to fix compilation of unity build
2014-08-07 19:05:04 +02:00
$(shell (export ROCKSDB_ROOT="$(CURDIR)"; "$(CURDIR)/build_tools/unity" "$(CURDIR)/unity.cc"))
unity: unity.cc unity.o
$(CXX) unity.o $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
clean:
Changes to support unity build: * Script for building the unity.cc file via Makefile * Unity executable Makefile target for testing builds * Source code changes to fix compilation of unity build
2014-08-07 19:05:04 +02:00
-rm -f $(PROGRAMS) $(TESTS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) build_config.mk unity.cc
Platform detection during build, plus compatibility patches for machines without <cstdatomic>. This revision adds two major changes: 1. build_detect_platform which generates build_config.mk with platform-dependent flags for the build process 2. /port/atomic_pointer.h with anAtomicPointerimplementation for platforms without <cstdatomic> Some of this code is loosely based on patches submitted to the LevelDB mailing list at https://groups.google.com/forum/#!forum/leveldb Tip of the hat to Dave Smith and Edouard A, who both sent patches. The presence of Snappy (http://code.google.com/p/snappy/) and cstdatomic are now both detected in the build_detect_platform script (1.) which gets executing during make. For (2.), instead of broadly importing atomicops_* from Chromium or the Google performance tools, we chose to just implement AtomicPointer and the limited atomic load and store operations it needs. This resulted in much less code and fewer files - everything is contained in atomic_pointer.h. git-svn-id: https://leveldb.googlecode.com/svn/trunk@34 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-29 02:30:50 +02:00
-rm -rf ios-x86/* ios-arm/*
Addressing review comments
2014-09-09 02:44:52 +02:00
-find . -name "*.[oda]" -exec rm {} \;
Fixing the warning messages captured under mac os # Consider using `git commit -m 'One line title' && arc diff`. # You will save time by running lint and unit in the background. Summary: The work to make sure mac os compiles rocksdb is not completed yet. But at least we can start cleaning some warnings captured only by g++ from mac os.. Test Plan: ran make in mac os Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14049
2013-11-13 05:05:28 +01:00
-find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
2014-09-26 22:57:12 +02:00
-rm -rf bzip2* snappy* zlib*
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
tags:
ctags * -R
cscope -b `find . -name '*.cc'` `find . -name '*.h'`
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
A script that automatically reformat affected lines Summary: Added a script that reformat only the affected lines in a given diff. I planned to make that file as pre-commit hook but looks it's a little bit more difficult than I thought. Since I don't want to spend too much time on this task right now, I eventually added a "make command" to achieve this with a few additional key strokes. Also make the clang-format solely inherited from Google's style -- there are still debates on some of the style issues, but we can address them later once we reach a consensus. Test Plan: Did some ugly format change and ran "make format", all affected lines are formatted as expected. Reviewers: igor, sdong, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D15147
2014-01-14 09:39:42 +01:00
format:
build_tools/format-diff.sh
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
package:
bash build_tools/make_package.sh $(SHARED_MAJOR).$(SHARED_MINOR)
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
# ---------------------------------------------------------------------------
# Unit tests and tools
# ---------------------------------------------------------------------------
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
$(LIBRARY): $(LIBOBJECTS)
rm -f $@
$(AR) -rs $@ $(LIBOBJECTS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
db_bench: db/db_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) db/db_bench.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
add cache bench Summary: 1. A benchmark for cache Test Plan: ./cache_bench Reviewers: yhchiang, dhruba, sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22809
2014-09-06 00:55:43 +02:00
cache_bench: util/cache_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) util/cache_bench.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Add a hash-index component for block Summary: this is the key component extracted from diff: https://reviews.facebook.net/D14271 I separate it to a dedicated patch to make the review easier. Test Plan: added a unit test and passed it. Reviewers: haobo, sdong, dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D16245
2014-02-18 23:58:55 +01:00
block_hash_index_test: table/block_hash_index_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/block_hash_index_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[tools] Add a tool to stress test concurrent writing to levelDB Summary: Created a tool that runs multiple threads that concurrently read and write to levelDB. All writes to the DB are stored in an in-memory hashtable and verified at the end of the test. All writes for a given key are serialzied. Test Plan: - Verified by writing only a few keys and logging all writes and verifying that values read and written are correct. - Verified correctness of value generator. - Ran with various parameters of number of keys, locks, and threads. Reviewers: dhruba, MarkCallaghan, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5829
2012-10-03 18:58:45 +02:00
db_stress: tools/db_stress.o $(LIBOBJECTS) $(TESTUTIL)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) tools/db_stress.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[tools] Add a tool to stress test concurrent writing to levelDB Summary: Created a tool that runs multiple threads that concurrently read and write to levelDB. All writes to the DB are stored in an in-memory hashtable and verified at the end of the test. All writes for a given key are serialzied. Test Plan: - Verified by writing only a few keys and logging all writes and verifying that values read and written are correct. - Verified correctness of value generator. - Ran with various parameters of number of keys, locks, and threads. Reviewers: dhruba, MarkCallaghan, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5829
2012-10-03 18:58:45 +02:00
DB Sanity Test Summary: @kailiu mentioned on meeting yesterday that we sometimes have trouble opening DB created by old version with the new version. This will be very important to test for column families, since I'm changing disk format for the MANIFEST. I added a tool that can help us test that. Usage: ./db_sanity_test <path> create will create a bunch of DBs under <path> <change RocksDB version> ./db_sanity_test <path> verify will verify consistency of DBs created under <path> Test Plan: ran the db_sanity_test Reviewers: kailiu, dhruba, haobo Reviewed By: kailiu CC: leveldb, kailiu, xjin Differential Revision: https://reviews.facebook.net/D16605
2014-03-06 20:36:39 +01:00
db_sanity_test: tools/db_sanity_test.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) tools/db_sanity_test.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[tools] Add a tool to stress test concurrent writing to levelDB Summary: Created a tool that runs multiple threads that concurrently read and write to levelDB. All writes to the DB are stored in an in-memory hashtable and verified at the end of the test. All writes for a given key are serialzied. Test Plan: - Verified by writing only a few keys and logging all writes and verifying that values read and written are correct. - Verified correctness of value generator. - Ran with various parameters of number of keys, locks, and threads. Reviewers: dhruba, MarkCallaghan, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5829
2012-10-03 18:58:45 +02:00
Create a long running test to check GetUpdatesSince. Summary: Create an Executable with * A thread to do Put's * A thread to use GetUpdatesSince. Check if we miss any sequence Numbers. Test Plan: It runs. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7383
2012-12-11 20:57:35 +01:00
db_repl_stress: tools/db_repl_stress.o $(LIBOBJECTS) $(TESTUTIL)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) tools/db_repl_stress.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Create a long running test to check GetUpdatesSince. Summary: Create an Executable with * A thread to do Put's * A thread to use GetUpdatesSince. Check if we miss any sequence Numbers. Test Plan: It runs. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7383
2012-12-11 20:57:35 +01:00
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
blob_store_bench: tools/blob_store_bench.o $(LIBOBJECTS) $(TESTUTIL)
$(CXX) tools/blob_store_bench.o $(LIBOBJECTS) $(TESTUTIL) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Adding FreeBSD support, removing Chromium files, adding benchmark. - LevelDB patch for FreeBSD. This resolves Issue 22. Contributed by dforsythe (thanks!). - Removing Chromium-specific files. They are now going to live in the Chromium repository. - Adding a benchmark page comparing LevelDB performance to SQLite and Kyoto Cabinet's TreeDB, along with code to generate the benchmarks. Thanks to Kevin Tseng for compiling the benchmarks, and Scott Hess and Mikio Hirabayashi for their help and advice. git-svn-id: https://leveldb.googlecode.com/svn/trunk@40 62dab493-f737-651d-591e-8d6aee1b9529
2011-07-27 03:46:25 +02:00
[RocksDB] Add stacktrace signal handler Summary: This diff provides the ability to print out a stacktrace when the process receives certain signals. Currently, we enable this for the following signals (program error related): SIGILL SIGSEGV SIGBUS SIGABRT Application simply #include "util/stack_trace.h" and call leveldb::InstallStackTraceHandler() during initialization, if signal handler is needed. It's not done automatically when openning db, because it's the application(process)'s responsibility to install signal handler and some applications might already have their own (like fbcode). Sample output: Received signal 11 (Segmentation fault) #0 0x408ff0 ./signal_test() [0x408ff0] /home/haobo/rocksdb/util/signal_test.cc:4 #1 0x40827d ./signal_test() [0x40827d] /home/haobo/rocksdb/util/signal_test.cc:24 #2 0x7f8bb183172e /usr/local/fbcode/gcc-4.7.1-glibc-2.14.1/lib/libc.so.6(__libc_start_main+0x10e) [0x7f8bb183172e] ??:0 #3 0x408ebc ./signal_test() [0x408ebc] /home/engshare/third-party/src/glibc/glibc-2.14.1/glibc-2.14.1/csu/../sysdeps/x86_64/elf/start.S:113 Segmentation fault (core dumped) For each frame, we print the raw pointer, the symbol provided by backtrace_symbols (still not good enough), and the source file/line. Note that address translation is done by directly shell out to addr2line. ??:0 means addr2line fails to do the translation. Hacky, but I think it's good for now. Test Plan: signal_test.cc Reviewers: dhruba, MarkCallaghan Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D10173
2013-04-11 19:54:35 +02:00
signal_test: util/signal_test.o $(LIBOBJECTS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/signal_test.o $(LIBOBJECTS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[RocksDB] Add stacktrace signal handler Summary: This diff provides the ability to print out a stacktrace when the process receives certain signals. Currently, we enable this for the following signals (program error related): SIGILL SIGSEGV SIGBUS SIGABRT Application simply #include "util/stack_trace.h" and call leveldb::InstallStackTraceHandler() during initialization, if signal handler is needed. It's not done automatically when openning db, because it's the application(process)'s responsibility to install signal handler and some applications might already have their own (like fbcode). Sample output: Received signal 11 (Segmentation fault) #0 0x408ff0 ./signal_test() [0x408ff0] /home/haobo/rocksdb/util/signal_test.cc:4 #1 0x40827d ./signal_test() [0x40827d] /home/haobo/rocksdb/util/signal_test.cc:24 #2 0x7f8bb183172e /usr/local/fbcode/gcc-4.7.1-glibc-2.14.1/lib/libc.so.6(__libc_start_main+0x10e) [0x7f8bb183172e] ??:0 #3 0x408ebc ./signal_test() [0x408ebc] /home/engshare/third-party/src/glibc/glibc-2.14.1/glibc-2.14.1/csu/../sysdeps/x86_64/elf/start.S:113 Segmentation fault (core dumped) For each frame, we print the raw pointer, the symbol provided by backtrace_symbols (still not good enough), and the source file/line. Note that address translation is done by directly shell out to addr2line. ??:0 means addr2line fails to do the translation. Hacky, but I think it's good for now. Test Plan: signal_test.cc Reviewers: dhruba, MarkCallaghan Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D10173
2013-04-11 19:54:35 +02:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
arena_test: util/arena_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/arena_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Implement autovector Summary: A vector that leverages pre-allocated stack-based array to achieve better performance for array with small amount of items. Test Plan: Added tests for both correctness and performance Here is the performance benchmark between vector and autovector Please note that in the test "Creation and Insertion Test", the test case were designed with the motivation described below: * no element inserted: internal array of std::vector may not really get initialize. * one element inserted: internal array of std::vector must have initialized. * kSize elements inserted. This shows the most time we'll spend if we keep everything in stack. * 2 * kSize elements inserted. The internal vector of autovector must have been initialized. Note: kSize is the capacity of autovector ===================================================== Creation and Insertion Test ===================================================== created 100000 vectors: each was inserted with 0 elements total time elapsed: 128000 (ns) created 100000 autovectors: each was inserted with 0 elements total time elapsed: 3641000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 0 elements total time elapsed: 9896000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 1 elements total time elapsed: 11089000 (ns) created 100000 autovectors: each was inserted with 1 elements total time elapsed: 5008000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 1 elements total time elapsed: 24271000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 4 elements total time elapsed: 39369000 (ns) created 100000 autovectors: each was inserted with 4 elements total time elapsed: 10121000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 4 elements total time elapsed: 28473000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 8 elements total time elapsed: 75013000 (ns) created 100000 autovectors: each was inserted with 8 elements total time elapsed: 18237000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 8 elements total time elapsed: 42464000 (ns) ----------------------------------- created 100000 vectors: each was inserted with 16 elements total time elapsed: 102319000 (ns) created 100000 autovectors: each was inserted with 16 elements total time elapsed: 76724000 (ns) created 100000 VectorWithReserveSizes: each was inserted with 16 elements total time elapsed: 68285000 (ns) ----------------------------------- ===================================================== Sequence Access Test ===================================================== performed 100000 sequence access against vector size: 4 total time elapsed: 198000 (ns) performed 100000 sequence access against autovector size: 4 total time elapsed: 306000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 8 total time elapsed: 565000 (ns) performed 100000 sequence access against autovector size: 8 total time elapsed: 512000 (ns) ----------------------------------- performed 100000 sequence access against vector size: 16 total time elapsed: 1076000 (ns) performed 100000 sequence access against autovector size: 16 total time elapsed: 1070000 (ns) ----------------------------------- Reviewers: dhruba, haobo, sdong, chip Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14655
2013-12-13 00:32:56 +01:00
autovector_test: util/autovector_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/autovector_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[column families] Support to create and drop column families Summary: This diff provides basic implementations of CreateColumnFamily(), DropColumnFamily() and ListColumnFamilies(). It builds on top of https://reviews.facebook.net/D14733 It also includes a bug fix for DBImplReadOnly, where Get implementation would be redirected to DBImpl instead of DBImplReadOnly. Test Plan: Added unit test Reviewers: dhruba, haobo, kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D15021
2014-01-02 18:08:12 +01:00
column_family_test: db/column_family_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/column_family_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Improve the "table stats" Summary: The primary motivation of the changes is to make it easier to figure out the inside of the tables. * rename "table stats" to "table properties" since now we have more than "integers" to store in the property block. * Add filter block size to the basic table properties. * Whenever a table is built, we'll log the table properties (the sample output is in Test Plan). * Make an api to expose deleted keys. Test Plan: Passed all existing test. and the sample output of table stats: ================================================================== Basic Properties ------------------------------------------------------------------ # data blocks: 1 # entries: 1 raw key size: 9 raw average key size: 9 raw value size: 9 raw average value size: 0 data block size: 25 index block size: 27 filter block size: 18 (estimated) table size: 70 filter policy: rocksdb.BuiltinBloomFilter ================================================================== User collected properties: InternalKeyPropertiesCollector ------------------------------------------------------------------ kDeletedKeys: 1 ================================================================== Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14187
2013-11-20 01:29:42 +01:00
table_properties_collector_test: db/table_properties_collector_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/table_properties_collector_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Support user-defined table stats collector Summary: 1. Added a new option that support user-defined table stats collection. 2. Added a deleted key stats collector in `utilities` Test Plan: Added a unit test for newly added code. Also ran make check to make sure other tests are not broken. Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13491
2013-10-16 20:50:50 +02:00
Added bloom filter support. In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
2012-04-17 17:36:46 +02:00
bloom_test: util/bloom_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/bloom_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Added bloom filter support. In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
2012-04-17 17:36:46 +02:00
[RocksDB] [Performance Branch] Added dynamic bloom, to be used for memable non-existing key filtering Summary: as title Test Plan: dynamic_bloom_test Reviewers: dhruba, sdong, kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D14385
2013-11-27 23:27:02 +01:00
dynamic_bloom_test: util/dynamic_bloom_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/dynamic_bloom_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
c_test: db/c_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/c_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
cache_test: util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
coding_test: util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
External Value Store Summary: Developing a capability for storing values on external backing file(s). This is just a highly unoptimized first pass - supports: 1) Allocating some portion of external file to be used to store value 2) Freeing the range, enabling it to be reused by other values As next steps, I plan to: 1) Create some kind of stress testing. Once I can measure stuff, I can focus on optimizing. 2) Optimize locking. 3) Optimize freelist data structure. Currently we have O(n) for both freeing and allocation. 4) Figure out how to do recovery. Test Plan: Created a unit test. Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13389
2013-10-17 02:33:49 +02:00
blob_store_test: util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(TESTUTIL)
$(CXX) util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o$@ $(LDFLAGS) $(COVERAGEFLAGS)
Implemented StringAppendOperator and unit tests. Summary: Implemented the StringAppendOperator class (subclass of MergeOperator). Found in utilities/merge_operators/string_append/stringappend.{h,cc} It is a rocksdb Merge Operator that supports string/list concatenation with a configurable delimiter. The tests are found in .../stringappend_test.cc. It implements a map : key -> (list of strings), with core operations Append(list_key,val) and Get(list_key). Test Plan: 1. Navigate to your rocksdb repository 2. Execute: make stringappend_test (to compile) 3. Execute: ./stringappend_test (to run the tests) 4. Execute: make all check (to test the ENTIRE rocksdb codebase / regression) Reviewers: haobo, dhruba, zshao Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D10737
2013-05-10 19:40:10 +02:00
stringappend_test: utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Implemented StringAppendOperator and unit tests. Summary: Implemented the StringAppendOperator class (subclass of MergeOperator). Found in utilities/merge_operators/string_append/stringappend.{h,cc} It is a rocksdb Merge Operator that supports string/list concatenation with a configurable delimiter. The tests are found in .../stringappend_test.cc. It implements a map : key -> (list of strings), with core operations Append(list_key,val) and Get(list_key). Test Plan: 1. Navigate to your rocksdb repository 2. Execute: make stringappend_test (to compile) 3. Execute: ./stringappend_test (to run the tests) 4. Execute: make all check (to test the ENTIRE rocksdb codebase / regression) Reviewers: haobo, dhruba, zshao Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D10737
2013-05-10 19:40:10 +02:00
Completed the implementation and test cases for Redis API. Summary: Completed the implementation for the Redis API for Lists. The Redis API uses rocksdb as a backend to persistently store maps from key->list. It supports basic operations for appending, inserting, pushing, popping, and accessing a list, given its key. Test Plan: - Compile with: make redis_test - Test with: ./redis_test - Run all unit tests (for all rocksdb) with: make all check - To use an interactive REDIS client use: ./redis_test -m - To clean the database before use: ./redis_test -m -d Reviewers: haobo, dhruba, zshao Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D10833
2013-06-11 20:19:49 +02:00
redis_test: utilities/redis/redis_lists_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) utilities/redis/redis_lists_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Completed the implementation and test cases for Redis API. Summary: Completed the implementation for the Redis API for Lists. The Redis API uses rocksdb as a backend to persistently store maps from key->list. It supports basic operations for appending, inserting, pushing, popping, and accessing a list, given its key. Test Plan: - Compile with: make redis_test - Test with: ./redis_test - Run all unit tests (for all rocksdb) with: make all check - To use an interactive REDIS client use: ./redis_test -m - To clean the database before use: ./redis_test -m -d Reviewers: haobo, dhruba, zshao Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D10833
2013-06-11 20:19:49 +02:00
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
benchharness_test: util/benchharness_test.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS)
$(CXX) util/benchharness_test.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS) $(EXEC_LDFLAGS) -o$@ $(LDFLAGS) $(COVERAGEFLAGS)
Added benchmark functionality on the lines of folly/Benchmark.h Summary: Added benchmark functionality on the lines of folly/Benchmark.h Test Plan: Added unit tests Reviewers: igor, haobo, sdong, ljin, yhchiang, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D17973
2014-04-21 21:29:55 +02:00
Performant util/histogram. Summary: Earlier way to record in histogram=> Linear search BucketLimit array to find the bucket and increment the counter Current way to record in histogram=> Store a HistMap statically which points the buckets of each value in the range [kFirstValue, kLastValue); In the proccess use vectors instead of array's and refactor some code to HistogramHelper class. Test Plan: run db_bench with histogram=1 and see a histogram being printed. Reviewers: dhruba, chip, heyongqiang Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8265
2013-01-29 21:23:31 +01:00
histogram_test: util/histogram_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/histogram_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o$@ $(LDFLAGS) $(COVERAGEFLAGS)
Performant util/histogram. Summary: Earlier way to record in histogram=> Linear search BucketLimit array to find the bucket and increment the counter Current way to record in histogram=> Store a HistMap statically which points the buckets of each value in the range [kFirstValue, kLastValue); In the proccess use vectors instead of array's and refactor some code to HistogramHelper class. Test Plan: run db_bench with histogram=1 and see a histogram being printed. Reviewers: dhruba, chip, heyongqiang Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8265
2013-01-29 21:23:31 +01:00
thread local pointer storage Summary: This is not a generic thread local implementation in the sense that it only takes pointer. But it does support multiple instances per thread and lets user plugin function to perform cleanup when thread exits or an instance gets destroyed. Test Plan: unit test for now Reviewers: haobo, igor, sdong, dhruba Reviewed By: igor CC: leveldb, kailiu Differential Revision: https://reviews.facebook.net/D16131
2014-02-26 02:47:37 +01:00
thread_local_test: util/thread_local_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/thread_local_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
corruption_test: db/corruption_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/corruption_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
crc32c_test: util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/crc32c_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
db_test: db/db_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Fix db_test Summary: Added deletion of DBIterators in DBIterator's tests Test Plan: make valgrind_check Reviewers: igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20043
2014-07-16 23:51:43 +02:00
db_iter_test: db/db_iter_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/db_iter_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
log_write_bench: util/log_write_bench.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/log_write_bench.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS) -pg
A simple benchmark to measure WAL append latency Summary: A simple benchmark that simulates WAL append. It can be used to test different platform/file system's performance on WAL. Test Plan: run it. Reviewers: haobo, kailiu Reviewed By: haobo CC: igor, dhruba, i.am.jin.lei, yhchiang, leveldb, nkg- Differential Revision: https://reviews.facebook.net/D16239
2014-02-19 23:55:34 +01:00
A Simple Plain Table Summary: A Simple plain table format. No block structure. When creating the table reader, scanning the full table to create indexes. Test Plan:Add unit test Reviewers:haobo,dhruba,kailiu CC: Task ID: # Blame Rev:
2013-10-29 04:34:02 +01:00
plain_table_db_test: db/plain_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/plain_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Add ComparatorDBTest to test non-default comparators Summary: Add some helper functions to make sure DB works well for non-default comparators. Add a test for SimpleSuffixReverseComparator. Test Plan: Run the new test Reviewers: ljin, rven, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D27831
2014-10-29 21:49:45 +01:00
comparator_db_test: db/comparator_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/comparator_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library.
2014-04-21 22:01:50 +02:00
table_reader_bench: table/table_reader_bench.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/table_reader_bench.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS) -pg
A very simple benchmark to measure Table implemenation's Get() And Iterator performance Summary: It is a very simple benchmark to measure a Table implementation's Get() and iterator performance if all the data is in memory. Test Plan: N/A Reviewers: dhruba, haobo, kailiu Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13743
2013-10-31 21:38:54 +01:00
log_and_apply_bench on a new benchmark framework Summary: db_test includes Benchmark for LogAndApply. This diff removes it from db_test and puts it into a separate log_and_apply bench. I just wanted to play around with our new benchmark framework and figure out how it works. I would also like to show you how great it is! I believe right set of microbenchmarks can speed up our productivity a lot and help catch early regressions. Test Plan: no Reviewers: dhruba, haobo, sdong, ljin, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D18261
2014-05-05 20:11:48 +02:00
log_and_apply_bench: db/log_and_apply_bench.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS)
$(CXX) db/log_and_apply_bench.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS) -pg
[RocksDB] Expose thread local perf counter for low overhead, per call level performance statistics. Summary: As title. No locking/atomic is needed due to thread local. There is also no need to modify the existing client interface, in order to expose related counters. perf_context_test shows a simple example of retrieving the number of user key comparison done for each put and get call. More counters could be added later. Sample output ./perf_context_test 1000000 ==== Test PerfContextTest.KeyComparisonCount Inserting 1000000 key/value pairs ... total user key comparison get: 43446523 total user key comparison put: 8017877 max user key comparison get: 88939 avg user key comparison get:43 Basically, the current skiplist does well on average, but could perform poorly in extreme cases. Test Plan: run perf_context_test <total number of entries to put/get> Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D12225
2013-08-13 08:59:04 +02:00
perf_context_test: db/perf_context_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/perf_context_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
[RocksDB] prefixhash memtable test Summary: as title, half baked test for prefixhash memtable. Also contains deadlock test option Test Plan: run it Reviewers: igor, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D13887
2013-10-23 06:59:44 +02:00
prefix_test: db/prefix_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/prefix_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
[RocksDB] BackupableDB Summary: In this diff I present you BackupableDB v1. You can easily use it to backup your DB and it will do incremental snapshots for you. Let's first describe how you would use BackupableDB. It's inheriting StackableDB interface so you can easily construct it with your DB object -- it will add a method RollTheSnapshot() to the DB object. When you call RollTheSnapshot(), current snapshot of the DB will be stored in the backup dir. To restore, you can just call RestoreDBFromBackup() on a BackupableDB (which is a static method) and it will restore all files from the backup dir. In the next version, it will even support automatic backuping every X minutes. There are multiple things you can configure: 1. backup_env and db_env can be different, which is awesome because then you can easily backup to HDFS or wherever you feel like. 2. sync - if true, it *guarantees* backup consistency on machine reboot 3. number of snapshots to keep - this will keep last N snapshots around if you want, for some reason, be able to restore from an earlier snapshot. All the backuping is done in incremental fashion - if we already have 00010.sst, we will not copy it again. *IMPORTANT* -- This is based on assumption that 00010.sst never changes - two files named 00010.sst from the same DB will always be exactly the same. Is this true? I always copy manifest, current and log files. 4. You can decide if you want to flush the memtables before you backup, or you're fine with backing up the log files -- either way, you get a complete and consistent view of the database at a time of backup. 5. More things you can find in BackupableDBOptions Here is the directory structure I use: backup_dir/CURRENT_SNAPSHOT - just 4 bytes holding the latest snapshot 0, 1, 2, ... - files containing serialized version of each snapshot - containing a list of files files/*.sst - sst files shared between snapshots - if one snapshot references 00010.sst and another one needs to backup it from the DB, it will just reference the same file files/ 0/, 1/, 2/, ... - snapshot directories containing private snapshot files - current, manifest and log files All the files are ref counted and deleted immediatelly when they get out of scope. Some other stuff in this diff: 1. Added GetEnv() method to the DB. Discussed with @haobo and we agreed that it seems right thing to do. 2. Fixed StackableDB interface. The way it was set up before, I was not able to implement BackupableDB. Test Plan: I have a unittest, but please don't look at this yet. I just hacked it up to help me with debugging. I will write a lot of good tests and update the diff. Also, `make asan_check` Reviewers: dhruba, haobo, emayanke Reviewed By: dhruba CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14295
2013-12-09 23:06:52 +01:00
backupable_db_test: utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/backupable/backupable_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
JSON (Document) API sketch Summary: This is a rough sketch of our new document API. Would like to get some thoughts and comments about the high-level architecture and API. I didn't optimize for performance at all. Leaving some low-hanging fruit so that we can be happy when we fix them! :) Currently, bunch of features are not supported at all. Indexes can be only specified when creating database. There is no query planner whatsoever. This will all be added in due time. Test Plan: Added a simple unit test Reviewers: haobo, yhchiang, dhruba, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18747
2014-07-10 18:31:42 +02:00
document_db_test: utilities/document/document_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/document/document_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
JSONDocument Summary: After evaluating options for JSON storage, I decided to implement our own. The reason is that we'll be able to optimize it better and we get to reduce unnecessary dependencies (which is what we'd get with folly). I also plan to write a serializer/deserializer for JSONDocument with our own binary format similar to BSON. That way we'll store binary JSON format in RocksDB instead of the plain-text JSON. This means less storage and faster deserialization. There are still some inefficiencies left here. I plan to optimize them after we develop a functioning DocumentDB. That way we can move and iterate faster. Test Plan: added a unit test Reviewers: dhruba, haobo, sdong, ljin, yhchiang Reviewed By: haobo Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18831
2014-06-20 11:14:14 +02:00
json_document_test: utilities/document/json_document_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/document/json_document_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
SpatialDB Summary: This diff is adding spatial index support to RocksDB. When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels. Each element inserted into SpatialDB has: * a bounding box, which determines how will the element be indexed * string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text) * feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar) * a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level). Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements. Test Plan: Added bunch of unit tests in spatial_db_test Reviewers: dhruba, yinwang Reviewed By: yinwang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20361
2014-07-23 20:22:58 +02:00
spatial_db_test: utilities/spatialdb/spatial_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/spatialdb/spatial_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Timestamp and TTL Wrapper for rocksdb Summary: When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open. Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest Reviewed By: vamsi CC: zshao, xjin, vkrest, MarkCallaghan Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 22:33:13 +02:00
ttl_test: utilities/ttl/ttl_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) utilities/ttl/ttl_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Timestamp and TTL Wrapper for rocksdb Summary: When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open. Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest Reviewed By: vamsi CC: zshao, xjin, vkrest, MarkCallaghan Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 22:33:13 +02:00
WriteBatchWithIndex: a wrapper of WriteBatch, with a searchable index Summary: Add WriteBatchWithIndex so that a user can query data out of a WriteBatch, to support MongoDB's read-its-own-write. WriteBatchWithIndex uses a skiplist to store the binary index. The index stores the offset of the entry in the write batch. When searching for a key, the key for the entry is read by read the entry from the write batch from the offset. Define a new iterator class for querying data out of WriteBatchWithIndex. A user can create an iterator of the write batch for one column family, seek to a key and keep calling Next() to see next entries. I will add more unit tests if people are OK about this API. Test Plan: make all check Add unit tests. Reviewers: yhchiang, igor, MarkCallaghan, ljin Reviewed By: ljin Subscribers: dhruba, leveldb, xjin Differential Revision: https://reviews.facebook.net/D21381
2014-08-19 00:19:17 +02:00
write_batch_with_index_test: utilities/write_batch_with_index/write_batch_with_index_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/write_batch_with_index/write_batch_with_index_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Timestamp and TTL Wrapper for rocksdb Summary: When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open. Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest Reviewed By: vamsi CC: zshao, xjin, vkrest, MarkCallaghan Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 22:33:13 +02:00
FlushProcess Summary: Abstract out FlushProcess and take it out of DBImpl. This also includes taking DeletionState outside of DBImpl. Currently this diff is only doing the refactoring. Future work includes: 1. Decoupling flush_process.cc, make it depend on less state 2. Write flush_process_test, which will mock out everything that FlushProcess depends on and test it in isolation Test Plan: make check Reviewers: rven, yhchiang, sdong, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D27561
2014-10-28 19:54:33 +01:00
flush_job_test: db/flush_job_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/flush_job_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
WalManager Summary: Decoupling code that deals with archived log files outside of DBImpl. That will make this code easier to reason about and test. It will also make the code easier to improve, because an improver doesn't have to understand DBImpl code in entirety. Test Plan: added test Reviewers: ljin, yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D27873
2014-10-30 01:43:37 +01:00
wal_manager_test: db/wal_manager_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/wal_manager_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
dbformat_test: db/dbformat_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/dbformat_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
env_test: util/env_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/env_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
generic rate limiter Summary: A generic rate limiter that can be shared by threads and rocksdb instances. Will use this to smooth out write traffic generated by compaction and flush. This will help us get better p99 behavior on flash storage. Test Plan: unit test output ==== Test RateLimiterTest.Rate request size [1 - 1023], limit 10 KB/sec, actual rate: 10.374969 KB/sec, elapsed 2002265 request size [1 - 2047], limit 20 KB/sec, actual rate: 20.771242 KB/sec, elapsed 2002139 request size [1 - 4095], limit 40 KB/sec, actual rate: 41.285299 KB/sec, elapsed 2202424 request size [1 - 8191], limit 80 KB/sec, actual rate: 81.371605 KB/sec, elapsed 2402558 request size [1 - 16383], limit 160 KB/sec, actual rate: 162.541268 KB/sec, elapsed 3303500 Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19359
2014-07-08 20:41:57 +02:00
rate_limiter_test: util/rate_limiter_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/rate_limiter_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
filename_test: db/filename_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/filename_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Implement full filter for block based table. Summary: 1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file. 2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter. 3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type. 4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h. 5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc Benchmark: base commit 1d23b5c470844c1208301311f0889eca750431c0 Command: db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1 Read QPS increase for about 30% from 2230002 to 2991411. Test Plan: make all check valgrind db_test db_stress --use_block_based_filter = 0 ./auto_sanity_test.sh Reviewers: igor, yhchiang, ljin, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20979
2014-09-08 19:37:05 +02:00
block_based_filter_block_test: table/block_based_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/block_based_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
full_filter_block_test: table/full_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/full_filter_block_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Added bloom filter support. In particular, we add a new FilterPolicy class. An instance of this class can be supplied in Options when opening a database. If supplied, the instance is used to generate summaries of keys (e.g., a bloom filter) which are placed in sstables. These summaries are consulted by DB::Get() so we can avoid reading sstable blocks that are guaranteed to not contain the key we are looking for. This change provides one implementation of FilterPolicy based on bloom filters. Other changes: - Updated version number to 1.4. - Some build tweaks. - C binding for CompactRange. - A few more benchmarks: deleteseq, deleterandom, readmissing, seekrandom. - Minor .gitignore update.
2012-04-17 17:36:46 +02:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
log_test: db/log_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/log_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
table_test: table/table_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) table/table_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Unit test to test block format. Summary: This is a standalone unit test to test the format of a block. Test Plan: ./block_test Reviewers: sheki Reviewed By: sheki Differential Revision: https://reviews.facebook.net/D7533
2012-12-20 20:05:41 +01:00
block_test: table/block_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) table/block_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Unit test to test block format. Summary: This is a standalone unit test to test the format of a block. Test Plan: ./block_test Reviewers: sheki Reviewed By: sheki Differential Revision: https://reviews.facebook.net/D7533
2012-12-20 20:05:41 +01:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
skiplist_test: db/skiplist_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/skiplist_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
version_edit_test: db/version_edit_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/version_edit_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
A number of smaller fixes and performance improvements: - Implemented Get() directly instead of building on top of a full merging iterator stack. This speeds up the "readrandom" benchmark by up to 15-30%. - Fixed an opensource compilation problem. Added --db=<name> flag to control where the database is placed. - Automatically compact a file when we have done enough overlapping seeks to that file. - Fixed a performance bug where we would read from at least one file in a level even if none of the files overlapped the key being read. - Makefile fix for Mac OSX installations that have XCode 4 without XCode 3. - Unified the two occurrences of binary search in a file-list into one routine. - Found and fixed a bug where we would unnecessarily search the last file when looking for a key larger than all data in the level. - A fix to avoid the need for trivial move compactions and therefore gets rid of two out of five syncs in "fillseq". - Removed the MANIFEST file write when switching to a new memtable/log-file for a 10-20% improvement on fill speed on ext4. - Adding a SNAPPY setting in the Makefile for folks who have Snappy installed. Snappy compresses values and speeds up writes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@32 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-22 04:36:45 +02:00
version_set_test: db/version_set_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/version_set_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
A number of smaller fixes and performance improvements: - Implemented Get() directly instead of building on top of a full merging iterator stack. This speeds up the "readrandom" benchmark by up to 15-30%. - Fixed an opensource compilation problem. Added --db=<name> flag to control where the database is placed. - Automatically compact a file when we have done enough overlapping seeks to that file. - Fixed a performance bug where we would read from at least one file in a level even if none of the files overlapped the key being read. - Makefile fix for Mac OSX installations that have XCode 4 without XCode 3. - Unified the two occurrences of binary search in a file-list into one routine. - Found and fixed a bug where we would unnecessarily search the last file when looking for a key larger than all data in the level. - A fix to avoid the need for trivial move compactions and therefore gets rid of two out of five syncs in "fillseq". - Removed the MANIFEST file write when switching to a new memtable/log-file for a 10-20% improvement on fill speed on ext4. - Adding a SNAPPY setting in the Makefile for folks who have Snappy installed. Snappy compresses values and speeds up writes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@32 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-22 04:36:45 +02:00
Make CompactionPicker more easily tested Summary: Make compaction picker easier to test. The basic idea is to separate a minimum subcomponent of Version to VersionStorageInfo, which just responsible to LSM tree. A stub VersionStorageInfo can then be easily created and passed into compaction picker so that we can check the outputs. It now passes most tests. Still two things need to be done: (1) deal with the FIFO compaction's file size. (2) write an example test to make sure the interface can do the job. Add a compaction_picker_test to make sure compaction picker codes can be easily unit tested. Test Plan: Pass all unit tests and compaction_picker_test Reviewers: yhchiang, rven, igor, ljin Reviewed By: ljin Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D27639
2014-10-27 23:49:46 +01:00
compaction_picker_test: db/compaction_picker_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/compaction_picker_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Make VersionBuilder unit testable Summary: Rename Version::Builder to VersionBuilder and expose its definition to a header. Make VerisonBuilder not reference Version or ColumnFamilyData, only working with VersionStorageInfo. Add version_builder_test which has a simple test. Test Plan: make all check Reviewers: rven, yhchiang, igor, ljin Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D27969
2014-10-31 16:48:19 +01:00
version_builder_test: db/version_builder_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/version_builder_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
hints for narrowing down FindFile range and avoiding checking unrelevant L0 files Summary: The file tree structure in Version is prebuilt and the range of each file is known. On the Get() code path, we do binary search in FindFile() by comparing target key with each file's largest key and also check the range for each L0 file. With some pre-calculated knowledge, each key comparision that has been done can serve as a hint to narrow down further searches: (1) If a key falls within a L0 file's range, we can safely skip the next file if its range does not overlap with the current one. (2) If a key falls within a file's range in level L0 - Ln-1, we should only need to binary search in the next level for files that overlap with the current one. (1) will be able to skip some files depending one the key distribution. (2) can greatly reduce the range of binary search, especially for bottom levels, given that one file most likely only overlaps with N files from the level below (where N is max_bytes_for_level_multiplier). So on level L, we will only look at ~N files instead of N^L files. Some inital results: measured with 500M key DB, when write is light (10k/s = 1.2M/s), this improves QPS ~7% on top of blocked bloom. When write is heavier (80k/s = 9.6M/s), it gives us ~13% improvement. Test Plan: make all check Reviewers: haobo, igor, dhruba, sdong, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17205
2014-04-21 18:10:12 +02:00
file_indexer_test : db/file_indexer_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/file_indexer_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Revert "Fix sst_dump and reduce_levels_test compile errors" This reverts commit d8f00b4109e3df10be56141f3ff3ba9b0d10f585.
2014-04-15 20:28:52 +02:00
reduce_levels_test: tools/reduce_levels_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) tools/reduce_levels_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Add a tool to change number of levels Summary: as subject. Test Plan: manually test it, will add a testcase Reviewers: dhruba, MarkCallaghan Differential Revision: https://reviews.facebook.net/D6345
2012-10-31 19:47:18 +01:00
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
write_batch_test: db/write_batch_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/write_batch_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[Rocksdb] Support Merge operation in rocksdb Summary: This diff introduces a new Merge operation into rocksdb. The purpose of this review is mostly getting feedback from the team (everyone please) on the design. Please focus on the four files under include/leveldb/, as they spell the client visible interface change. include/leveldb/db.h include/leveldb/merge_operator.h include/leveldb/options.h include/leveldb/write_batch.h Please go over local/my_test.cc carefully, as it is a concerete use case. Please also review the impelmentation files to see if the straw man implementation makes sense. Note that, the diff does pass all make check and truly supports forward iterator over db and a version of Get that's based on iterator. Future work: - Integration with compaction - A raw Get implementation I am working on a wiki that explains the design and implementation choices, but coding comes just naturally and I think it might be a good idea to share the code earlier. The code is heavily commented. Test Plan: run all local tests Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: leveldb, zshao, sheki, emayanke, MarkCallaghan Differential Revision: https://reviews.facebook.net/D9651
2013-03-21 23:59:47 +01:00
Push- instead of pull-model for managing Write stalls Summary: Introducing WriteController, which is a source of truth about per-DB write delays. Let's define an DB epoch as a period where there are no flushes and compactions (i.e. new epoch is started when flush or compaction finishes). Each epoch can either: * proceed with all writes without delay * delay all writes by fixed time * stop all writes The three modes are recomputed at each epoch change (flush, compaction), rather than on every write (which is currently the case). When we have a lot of column families, our current pull behavior adds a big overhead, since we need to loop over every column family for every write. With new push model, overhead on Write code-path is minimal. This is just the start. Next step is to also take care of stalls introduced by slow memtable flushes. The final goal is to eliminate function MakeRoomForWrite(), which currently needs to be called for every column family by every write. Test Plan: make check for now. I'll add some unit tests later. Also, perf test. Reviewers: dhruba, yhchiang, MarkCallaghan, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22791
2014-09-08 20:20:25 +02:00
write_controller_test: db/write_controller_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/write_controller_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Use specific DB name in merge_test Summary: Currently, merge_test uses /tmp/testdb for the test database. It should really use something more specific to merge_test. Most of the other tests use test::TmpDir() + "/<test name>db". This patch implements such behavior for merge_test; it makes merge_test use test::TmpDir() + "/merge_testdb" Test Plan: make clean make -j32 merge_test ./merge_test Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D11877
2013-07-29 22:26:38 +02:00
merge_test: db/merge_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) db/merge_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
Merger test Summary: I abandoned https://reviews.facebook.net/D18789, but I wrote a good unit test there, so let's check it in. :) Test Plan: this is test Reviewers: sdong, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22827
2014-09-09 07:24:40 +02:00
merger_test: table/merger_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/merger_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Add APIs to query SST file metadata and to delete specific SST files Summary: An api to query the level, key ranges, size etc for each SST file and an api to delete a specific file from the db and all associated state in the bookkeeping datastructures. Notes: Editing the manifest version does not release the obsolete files right away. However deleting the file directly will mess up the iterator. We may need a more aggressive/timely file deletion api. I have used std::unique_ptr - will switch to boost:: since this is external. thoughts? Unit test is fragile right now as it expects the compaction at certain levels. Test Plan: unittest Reviewers: dhruba, vamsi, emayanke CC: zshao, leveldb, haobo Task ID: # Blame Rev:
2013-08-22 23:32:53 +02:00
deletefile_test: db/deletefile_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/deletefile_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS)
A GIS implementation for rocksdb. Summary: This patch stores gps locations in rocksdb. Each object is uniquely identified by an id. Each object has a gps (latitude, longitude) associated with it. The geodb supports looking up an object either by its gps location or by its id. There is a method to retrieve all objects within a circular radius centered at a specified gps location. Test Plan: Simple unit-test attached. Reviewers: leveldb, haobo Reviewed By: haobo CC: leveldb, tecbot, haobo Differential Revision: https://reviews.facebook.net/D15567
2013-12-21 02:17:00 +01:00
geodb_test: utilities/geodb/geodb_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/geodb/geodb_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Adding a new SST table builder based on Cuckoo Hashing Summary: Cuckoo Hashing based SST table builder. Contains: - Cuckoo Hashing logic and file storage logic. - Unit tests for logic Test Plan: make cuckoo_table_builder_test ./cuckoo_table_builder_test make check all Reviewers: yhchiang, igor, sdong, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19545
2014-07-21 22:26:09 +02:00
cuckoo_table_builder_test: table/cuckoo_table_builder_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) table/cuckoo_table_builder_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Implementation of CuckooTableReader Summary: Contains: - Implementation of TableReader based on Cuckoo Hashing - Unittests for CuckooTableReader - Performance test for TableReader Test Plan: make cuckoo_table_reader_test ./cuckoo_table_reader_test make valgrind_check make asan_check Reviewers: yhchiang, sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20511
2014-07-26 01:37:32 +02:00
cuckoo_table_reader_test: table/cuckoo_table_reader_test.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS)
$(CXX) table/cuckoo_table_reader_test.o $(LIBOBJECTS) $(TESTHARNESS) $(BENCHHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Integrating Cuckoo Hash SST Table format into RocksDB Summary: Contains the following changes: - Implementation of cuckoo_table_factory - Adding cuckoo table into AdaptiveTableFactory - Adding cuckoo_table_db_test, similar to lines of plain_table_db_test - Minor fixes to Reader: When a key is found in the table, return the key found instead of the search key. - Minor fixes to Builder: Add table properties that are required by Version::UpdateTemporaryStats() during Get operation. Don't define curr_node as a reference variable as the memory locations may get reassigned during tree.push_back operation, leading to invalid memory access. Test Plan: cuckoo_table_reader_test --enable_perf cuckoo_table_builder_test cuckoo_table_db_test make check all make valgrind_check make asan_check Reviewers: sdong, igor, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21219
2014-08-12 05:21:07 +02:00
cuckoo_table_db_test: db/cuckoo_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/cuckoo_table_db_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
CompactFiles, EventListener and GetDatabaseMetaData Summary: This diff adds three sets of APIs to RocksDB. = GetColumnFamilyMetaData = * This APIs allow users to obtain the current state of a RocksDB instance on one column family. * See GetColumnFamilyMetaData in include/rocksdb/db.h = EventListener = * A virtual class that allows users to implement a set of call-back functions which will be called when specific events of a RocksDB instance happens. * To register EventListener, simply insert an EventListener to ColumnFamilyOptions::listeners = CompactFiles = * CompactFiles API inputs a set of file numbers and an output level, and RocksDB will try to compact those files into the specified level. = Example = * Example code can be found in example/compact_files_example.cc, which implements a simple external compactor using EventListener, GetColumnFamilyMetaData, and CompactFiles API. Test Plan: listener_test compactor_test example/compact_files_example export ROCKSDB_TESTS=CompactFiles db_test export ROCKSDB_TESTS=MetaData db_test Reviewers: ljin, igor, rven, sdong Reviewed By: sdong Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D24705
2014-11-07 23:45:18 +01:00
listener_test: db/listener_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) db/listener_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
compactor_test: utilities/compaction/compactor_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) utilities/compaction/compactor_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741
2014-05-16 19:35:41 +02:00
options_test: util/options_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/options_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Build fixes and cleanups: (1) Separate out C++ and CC flags (fixes c_test compilation) (2) Move snappy/perftools detection to script (3) Fix db_bench_sqlite3 and db_bench_tree_db build rules
2012-03-21 18:28:03 +01:00
$(MEMENVLIBRARY) : $(MEMENVOBJECTS)
Sync with upstream @23860137. Fix GCC -Wshadow warnings in LevelDB's public header files, reported by Dustin. Add in-memory Env implementation (helpers/memenv/*). This enables users to create LevelDB databases in-memory. Initialize ShardedLRUCache::last_id_ to zero. This fixes a Valgrind warning. (Also delete port/sha1_* which were removed upstream some time ago.)
2011-09-12 11:21:10 +02:00
rm -f $@
Build fixes and cleanups: (1) Separate out C++ and CC flags (fixes c_test compilation) (2) Move snappy/perftools detection to script (3) Fix db_bench_sqlite3 and db_bench_tree_db build rules
2012-03-21 18:28:03 +01:00
$(AR) -rs $@ $(MEMENVOBJECTS)
Sync with upstream @23860137. Fix GCC -Wshadow warnings in LevelDB's public header files, reported by Dustin. Add in-memory Env implementation (helpers/memenv/*). This enables users to create LevelDB databases in-memory. Initialize ShardedLRUCache::last_id_ to zero. This fixes a Valgrind warning. (Also delete port/sha1_* which were removed upstream some time ago.)
2011-09-12 11:21:10 +02:00
Compile -O2 by default and add `make dbg` Summary: To speed up the compilation while allowing us to compile in debug mode. Test Plan: make: see -O2 enabled make dbg: didn't see -O2 Reviewers: igor Reviewed By: igor CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D15969
2014-02-07 01:11:35 +01:00
memenv_test : helpers/memenv/memenv_test.o $(MEMENVOBJECTS) $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) helpers/memenv/memenv_test.o $(MEMENVOBJECTS) $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Sync with upstream @23860137. Fix GCC -Wshadow warnings in LevelDB's public header files, reported by Dustin. Add in-memory Env implementation (helpers/memenv/*). This enables users to create LevelDB databases in-memory. Initialize ShardedLRUCache::last_id_ to zero. This fixes a Valgrind warning. (Also delete port/sha1_* which were removed upstream some time ago.)
2011-09-12 11:21:10 +02:00
introduce TestMemEnv and use it in db_test Summary: TestMemEnv simulates all Env APIs using in-memory data structures. We can use it to speed up db_test run, which is now reduced ~7mins when it is enabled. We can also add features to simulate power/disk failures in the next step TestMemEnv is derived from helper/mem_env mem_env can not be used for rocksdb since some of its APIs do not give the same results as env_posix. And its file read/write is not thread safe Test Plan: make all -j32 ./db_test ./env_mem_test Reviewers: sdong, yhchiang, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D28035
2014-10-31 23:08:10 +01:00
mock_env_test : util/mock_env_test.o $(LIBOBJECTS) $(TESTHARNESS)
$(CXX) util/mock_env_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[rocksdb] do not trim range for level0 in manual compaction Summary: https://code.google.com/p/leveldb/issues/detail?can=1&q=178&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary&id=178 Ported the solution as is to RocksDB. Test Plan: moved the unit test as manual_compaction_test Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D11331
2013-06-17 22:58:17 +02:00
manual_compaction_test: util/manual_compaction_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/manual_compaction_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
[rocksdb] do not trim range for level0 in manual compaction Summary: https://code.google.com/p/leveldb/issues/detail?can=1&q=178&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary&id=178 Ported the solution as is to RocksDB. Test Plan: moved the unit test as manual_compaction_test Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D11331
2013-06-17 22:58:17 +02:00
Changing Makefile to have rocksdb instead of leveldb in binary-names Summary: did a find-replace Test Plan: make Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D11979
2013-08-04 02:07:28 +02:00
rocksdb_shell: tools/shell/ShellContext.o tools/shell/ShellState.o tools/shell/LeveldbShell.o tools/shell/DBClientProxy.o tools/shell/ShellContext.h tools/shell/ShellState.h tools/shell/DBClientProxy.h $(LIBOBJECTS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) tools/shell/ShellContext.o tools/shell/ShellState.o tools/shell/LeveldbShell.o tools/shell/DBClientProxy.o $(LIBOBJECTS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Merge remote-tracking branch 'origin/master' into task1181723 Conflicts: Makefile
2012-08-23 18:50:43 +02:00
[RocksDB] Add stacktrace signal handler Summary: This diff provides the ability to print out a stacktrace when the process receives certain signals. Currently, we enable this for the following signals (program error related): SIGILL SIGSEGV SIGBUS SIGABRT Application simply #include "util/stack_trace.h" and call leveldb::InstallStackTraceHandler() during initialization, if signal handler is needed. It's not done automatically when openning db, because it's the application(process)'s responsibility to install signal handler and some applications might already have their own (like fbcode). Sample output: Received signal 11 (Segmentation fault) #0 0x408ff0 ./signal_test() [0x408ff0] /home/haobo/rocksdb/util/signal_test.cc:4 #1 0x40827d ./signal_test() [0x40827d] /home/haobo/rocksdb/util/signal_test.cc:24 #2 0x7f8bb183172e /usr/local/fbcode/gcc-4.7.1-glibc-2.14.1/lib/libc.so.6(__libc_start_main+0x10e) [0x7f8bb183172e] ??:0 #3 0x408ebc ./signal_test() [0x408ebc] /home/engshare/third-party/src/glibc/glibc-2.14.1/glibc-2.14.1/csu/../sysdeps/x86_64/elf/start.S:113 Segmentation fault (core dumped) For each frame, we print the raw pointer, the symbol provided by backtrace_symbols (still not good enough), and the source file/line. Note that address translation is done by directly shell out to addr2line. ??:0 means addr2line fails to do the translation. Hacky, but I think it's good for now. Test Plan: signal_test.cc Reviewers: dhruba, MarkCallaghan Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D10173
2013-04-11 19:54:35 +02:00
DBClientProxy_test: tools/shell/test/DBClientProxyTest.o tools/shell/DBClientProxy.o $(LIBRARY)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) tools/shell/test/DBClientProxyTest.o tools/shell/DBClientProxy.o $(LIBRARY) $(EXEC_LDFLAGS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Titile: a command line shell to read/write data from a leveldb thrift server Summary: implemented a commond line shell to talk with leveldb thrift server, which is based on state pattern and can be easily extended. Test Plan: build and run Reviewers: dhruba, zshao, heyongqiang Differential Revision: https://reviews.facebook.net/D4713
2012-08-14 23:52:48 +02:00
Prevent concurrent multiple opens of leveldb database. Summary: The fcntl call cannot detect lock conflicts when invoked multiple times from the same thread. Use a static lockedFile Set to record the paths that are locked. A lockfile request checks to see if htis filename already exists in lockedFiles, if so, then it triggers an error. Otherwise, it inserts the filename in the lockedFiles Set. A unlock file request verifies that the filename is in the lockedFiles set and removes it from lockedFiles set. Test Plan: unit test attached Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4755
2012-08-18 09:26:50 +02:00
filelock_test: util/filelock_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/filelock_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Allow the logs to be purged by TTL. Summary: * Add a SplitByTTLLogger to enable this feature. In this diff I implemented generalized AutoSplitLoggerBase class to simplify the development of such classes. * Refactor the existing AutoSplitLogger and fix several bugs. Test Plan: * Added a unit tests for different types of "auto splitable" loggers individually. * Tested the composited logger which allows the log files to be splitted by both TTL and log size. Reviewers: heyongqiang, dhruba Reviewed By: heyongqiang CC: zshao, leveldb Differential Revision: https://reviews.facebook.net/D8037
2013-02-05 04:42:40 +01:00
auto_roll_logger_test: util/auto_roll_logger_test.o $(LIBOBJECTS) $(TESTHARNESS)
Add automatic coverage report scripts Summary: Ultimate goals of the coverage report are: * Report the coverage for all files (done in this diff) * Report the coverage for recently updated files (not fully finished) * Report is available in html form (done in this diff, but need some extra work to integrate it in Jenkin) Task link: https://our.intern.facebook.com/intern/tasks/?s=1154818042&t=2604914 Test Plan: Ran: coverage/coverage_test.sh The sample output can be found here: https://phabricator.fb.com/P2433892 Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D11943
2013-08-01 22:59:01 +02:00
$(CXX) util/auto_roll_logger_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Prevent concurrent multiple opens of leveldb database. Summary: The fcntl call cannot detect lock conflicts when invoked multiple times from the same thread. Use a static lockedFile Set to record the paths that are locked. A lockfile request checks to see if htis filename already exists in lockedFiles, if so, then it triggers an error. Otherwise, it inserts the filename in the lockedFiles Set. A unlock file request verifies that the filename is in the lockedFiles set and removes it from lockedFiles set. Test Plan: unit test attached Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4755
2012-08-18 09:26:50 +02:00
Revert "Fix sst_dump and reduce_levels_test compile errors" This reverts commit d8f00b4109e3df10be56141f3ff3ba9b0d10f585.
2014-04-15 20:28:52 +02:00
sst_dump: tools/sst_dump.o $(LIBOBJECTS)
$(CXX) tools/sst_dump.o $(LIBOBJECTS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Utility to dump manifest contents. Summary: ./manifest_dump --file=/tmp/dbbench/MANIFEST-000002 Output looks like manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28 prev_log_number 0 --- level 0 --- --- level 1 --- --- level 2 --- 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1] 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1] 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1] 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1] 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1] 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1] 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1] 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1] 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1] 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1] 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1] 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1] 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1] --- level 3 --- --- level 4 --- --- level 5 --- --- level 6 --- Test Plan: run on test directory created by dbbench Reviewers: heyongqiang Reviewed By: heyongqiang CC: hustliubo Differential Revision: https://reviews.facebook.net/D4743
2012-08-17 19:48:40 +02:00
Revert "Don't compile ldb tool into static library" This reverts commit e296577ef64deac036a275a1a4c0d6172cfa42df.
2014-04-15 20:29:02 +02:00
ldb: tools/ldb.o $(LIBOBJECTS)
$(CXX) tools/ldb.o $(LIBOBJECTS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)
Prevent concurrent multiple opens of leveldb database. Summary: The fcntl call cannot detect lock conflicts when invoked multiple times from the same thread. Use a static lockedFile Set to record the paths that are locked. A lockfile request checks to see if htis filename already exists in lockedFiles, if so, then it triggers an error. Otherwise, it inserts the filename in the lockedFiles Set. A unlock file request verifies that the filename is in the lockedFiles set and removes it from lockedFiles set. Test Plan: unit test attached Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4755
2012-08-18 09:26:50 +02:00
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
# ---------------------------------------------------------------------------
# Jni stuff
# ---------------------------------------------------------------------------
Fix build
2014-04-18 19:47:03 +02:00
JNI_NATIVE_SOURCES = ./java/rocksjni/*.cc
[Java] Use environmental variable JAVA_HOME in Makefile for RocksJava.
2014-05-04 22:15:33 +02:00
JAVA_INCLUDE = -I$(JAVA_HOME)/include/ -I$(JAVA_HOME)/include/linux
Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
2014-09-26 22:57:12 +02:00
ARCH := $(shell getconf LONG_BIT)
ROCKSDBJNILIB = librocksdbjni-linux$(ARCH).so
add major minor micro version to java jars
2014-10-02 20:07:45 +02:00
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux$(ARCH).jar
make fat jar unclassified to satisfy sonatype
2014-10-02 22:46:43 +02:00
ROCKSDB_JAR_ALL = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar
use javadoc instead of javadocs
2014-10-02 23:13:09 +02:00
ROCKSDB_JAVADOCS_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-javadoc.jar
add javadoc and sources targets for sonatype
2014-10-02 22:57:54 +02:00
ROCKSDB_SOURCES_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
ifeq ($(PLATFORM), OS_MACOSX)
Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
2014-09-26 22:57:12 +02:00
ROCKSDBJNILIB = librocksdbjni-osx.jnilib
add major minor micro version to java jars
2014-10-02 20:07:45 +02:00
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar
Fix for building RocksDB Java on Mac OS X Yosemite
2014-10-23 16:39:48 +02:00
ifneq ("$(wildcard $(JAVA_HOME)/include/darwin)","")
JAVA_INCLUDE = -I$(JAVA_HOME)/include -I $(JAVA_HOME)/include/darwin
else
JAVA_INCLUDE = -I/System/Library/Frameworks/JavaVM.framework/Headers/
endif
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
endif
Addressing review comments
2014-09-09 02:44:52 +02:00
libz.a:
-rm -rf zlib-1.2.8
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
curl -O http://zlib.net/zlib-1.2.8.tar.gz
tar xvzf zlib-1.2.8.tar.gz
cd zlib-1.2.8 && CFLAGS='-fPIC' ./configure --static && make
use GetContext to replace callback function pointer Summary: Intead of passing callback function pointer and its arg on Table::Get() interface, passing GetContext. This makes the interface cleaner and possible better perf. Also adding a fast pass for SaveValue() Test Plan: make all check Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24057
2014-09-29 20:09:09 +02:00
cp zlib-1.2.8/libz.a .
Addressing review comments
2014-09-09 02:44:52 +02:00
libbz2.a:
-rm -rf bzip2-1.0.6
use GetContext to replace callback function pointer Summary: Intead of passing callback function pointer and its arg on Table::Get() interface, passing GetContext. This makes the interface cleaner and possible better perf. Also adding a fast pass for SaveValue() Test Plan: make all check Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24057
2014-09-29 20:09:09 +02:00
curl -O http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
tar xvzf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6 && make CFLAGS='-fPIC -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64'
cp bzip2-1.0.6/libbz2.a .
Addressing review comments
2014-09-09 02:44:52 +02:00
libsnappy.a:
-rm -rf snappy-1.1.1
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
curl -O https://snappy.googlecode.com/files/snappy-1.1.1.tar.gz
tar xvzf snappy-1.1.1.tar.gz
Addressing review comments
2014-09-09 02:44:52 +02:00
cd snappy-1.1.1 && ./configure --with-pic --enable-static
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
cd snappy-1.1.1 && make
cp snappy-1.1.1/.libs/libsnappy.a .
use GetContext to replace callback function pointer Summary: Intead of passing callback function pointer and its arg on Table::Get() interface, passing GetContext. This makes the interface cleaner and possible better perf. Also adding a fast pass for SaveValue() Test Plan: make all check Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D24057
2014-09-29 20:09:09 +02:00
Addressing review comments
2014-09-09 02:44:52 +02:00
rocksdbjavastatic: libz.a libbz2.a libsnappy.a
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
OPT="-fPIC -DNDEBUG -O2" $(MAKE) $(LIBRARY) -j
cd java;$(MAKE) java;
rm -f ./java/$(ROCKSDBJNILIB)
$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC -o ./java/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) $(LIBOBJECTS) $(COVERAGEFLAGS) libz.a libbz2.a libsnappy.a
cd java;jar -cf $(ROCKSDB_JAR) org/rocksdb/*.class org/rocksdb/util/*.class HISTORY*.md $(ROCKSDBJNILIB)
singular javadoc directory
2014-10-02 23:57:18 +02:00
cd java/javadoc;jar -cf ../$(ROCKSDB_JAVADOCS_JAR) *
add javadoc and sources targets for sonatype
2014-10-02 22:57:54 +02:00
cd java;jar -cf $(ROCKSDB_SOURCES_JAR) org
RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
2014-08-18 23:03:46 +02:00
Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
2014-09-26 22:57:12 +02:00
rocksdbjavastaticrelease: rocksdbjavastatic
shrink vagrant commands to single line
2014-09-30 21:03:32 +02:00
cd java/crossbuild && vagrant destroy -f && vagrant up linux32 && vagrant halt linux32 && vagrant up linux64 && vagrant halt linux64
Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
2014-09-26 22:57:12 +02:00
cd java;jar -cf $(ROCKSDB_JAR_ALL) org/rocksdb/*.class org/rocksdb/util/*.class HISTORY*.md librocksdbjni-*.so librocksdbjni-*.jnilib
make publish jni jars depend on release jni jars
2014-10-06 17:20:56 +02:00
rocksdbjavastaticpublish: rocksdbjavastaticrelease
add not about updating pom version and rename pom to be unversioned
2014-10-02 23:42:49 +02:00
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-javadoc.jar -Dclassifier=javadoc
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-sources.jar -Dclassifier=sources
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux64.jar -Dclassifier=linux64
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-linux32.jar -Dclassifier=linux32
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar -Dclassifier=osx
mvn gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=sonatype-nexus-staging -DpomFile=java/rocksjni.pom -Dfile=java/rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH).jar
add maven publication target and instructions
2014-10-02 22:29:47 +02:00
Add doc and end of line
2014-06-22 22:27:22 +02:00
rocksdbjava:
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
OPT="-fPIC -DNDEBUG -O2" $(MAKE) $(LIBRARY) -j32
cd java;$(MAKE) java;
Fixed some make and linking issues of RocksDBJava Summary: Fixed some make and linking issues of RocksDBJava. Specifically: * Add JAVA_LDFLAGS, which does not include gflags * rocksdbjava library now uses JAVA_LDFLAGS instead of LDFLAGS * java/Makefile now includes build_config.mk * rearrange make rocksdbjava workflow to ensure the library file is correctly included in the jar file. Test Plan: make rocksdbjava make jdb_bench java/jdb_bench.sh Reviewers: dhruba, swapnilghike, zzbennett, rsumbaly, ankgup87 Reviewed By: ankgup87 Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20289
2014-07-22 07:41:54 +02:00
rm -f ./java/$(ROCKSDBJNILIB)
$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC -o ./java/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) $(LIBOBJECTS) $(JAVA_LDFLAGS) $(COVERAGEFLAGS)
cd java;jar -cf $(ROCKSDB_JAR) org/rocksdb/*.class org/rocksdb/util/*.class HISTORY*.md $(ROCKSDBJNILIB)
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
jclean:
cd java;$(MAKE) clean;
rm -f $(ROCKSDBJNILIB)
jtest:
[JNI] Add java api and java tests for WriteBatch and WriteOptions, add put() and remove() to RocksDB. Summary: * Add java api for rocksdb::WriteBatch and rocksdb::WriteOptions, which are necessary components for running benchmark. * Add java test for org.rocksdb.WriteBatch and org.rocksdb.WriteOptions. * Add remove() to org.rocksdb.RocksDB, and add put() and remove() to RocksDB which take org.rocksdb.WriteOptions. Test Plan: make jtest Reviewers: haobo, sdong, dhruba Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D17373
2014-04-02 22:14:55 +02:00
cd java;$(MAKE) sample;$(MAKE) test;
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
[JNI] Add an initial benchmark for java binding for rocksdb. Summary: * Add a benchmark for java binding for rocksdb. The java benchmark is a complete rewrite based on the c++ db/db_bench.cc and the DbBenchmark in dain's java leveldb. * Support multithreading. * 'readseq' is currently not supported as it requires RocksDB Iterator. * usage: --benchmarks Comma-separated list of operations to run in the specified order Actual benchmarks: fillseq -- write N values in sequential key order in async mode fillrandom -- write N values in random key order in async mode fillbatch -- write N/1000 batch where each batch has 1000 values in random key order in sync mode fillsync -- write N/100 values in random key order in sync mode fill100K -- write N/1000 100K values in random order in async mode readseq -- read N times sequentially readrandom -- read N times in random order readhot -- read N times in random order from 1% section of DB Meta Operations: delete -- delete DB DEFAULT: [fillseq, readrandom, fillrandom] --compression_ratio Arrange to generate values that shrink to this fraction of their original size after compression DEFAULT: 0.5 --use_existing_db If true, do not destroy the existing database. If you set this flag and also specify a benchmark that wants a fresh database, that benchmark will fail. DEFAULT: false --num Number of key/values to place in database. DEFAULT: 1000000 --threads Number of concurrent threads to run. DEFAULT: 1 --reads Number of read operations to do. If negative, do --nums reads. --key_size The size of each key in bytes. DEFAULT: 16 --value_size The size of each value in bytes. DEFAULT: 100 --write_buffer_size Number of bytes to buffer in memtable before compacting (initialized to default value by 'main'.) DEFAULT: 4194304 --cache_size Number of bytes to use as a cache of uncompressed data. Negative means use default settings. DEFAULT: -1 --seed Seed base for random number generators. DEFAULT: 0 --db Use the db with the following name. DEFAULT: /tmp/rocksdbjni-bench * Add RocksDB.write(). Test Plan: make jbench Reviewers: haobo, sdong, dhruba, ankgup87 Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17433
2014-04-09 09:48:20 +02:00
jdb_bench:
cd java;$(MAKE) db_bench;
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
# ---------------------------------------------------------------------------
# Platform-specific compilation
# ---------------------------------------------------------------------------
ctags and cscope support to Makefile Summary: Added a target to Makefile called 'tags' that runs ctags and cscope on all *.cc and *.h file Test Plan: Run 'make tags'. Then start vim and do :set tags=./tags :cs add cscope.out These commands should give you no error messages. You should then be able to access cscope db and ctags as normal in vim. Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D11103
2013-06-05 19:37:38 +02:00
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
ifeq ($(PLATFORM), IOS)
# For iOS, create universal object files to be used on both the simulator and
# a device.
merge 1.5 Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
2012-08-27 08:45:35 +02:00
PLATFORMSROOT=/Applications/Xcode.app/Contents/Developer/Platforms
SIMULATORROOT=$(PLATFORMSROOT)/iPhoneSimulator.platform/Developer
DEVICEROOT=$(PLATFORMSROOT)/iPhoneOS.platform/Developer
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
IOSVERSION=$(shell defaults read $(PLATFORMSROOT)/iPhoneOS.platform/version CFBundleShortVersionString)
Fixing Makefile issue reported in Issue 15 (misspelled flag) git-svn-id: https://leveldb.googlecode.com/svn/trunk@35 62dab493-f737-651d-591e-8d6aee1b9529
2011-06-30 00:53:17 +02:00
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
.cc.o:
mkdir -p ios-x86/$(dir $@)
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
$(CXX) $(CXXFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -arch x86_64 -c $< -o ios-x86/$@
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
mkdir -p ios-arm/$(dir $@)
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
xcrun -sdk iphoneos $(CXX) $(CXXFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -arch armv7s -arch arm64 -c $< -o ios-arm/$@
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
lipo ios-x86/$@ ios-arm/$@ -create -output $@
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
.c.o:
mkdir -p ios-x86/$(dir $@)
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
$(CC) $(CFLAGS) -isysroot $(SIMULATORROOT)/SDKs/iPhoneSimulator$(IOSVERSION).sdk -arch i686 -arch x86_64 -c $< -o ios-x86/$@
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
mkdir -p ios-arm/$(dir $@)
Make RocksDB compile for iOS Summary: I had to make number of changes to the code and Makefile: * Add `make lib`, that will create static library without debug info. We need this to avoid growing binary too much. Currently it's 14MB. * Remove cpuinfo() function and use __SSE4_2__ macro. We actually used the macro as part of Fast_CRC32() function. As a result, I also accidentally fixed this issue: https://www.facebook.com/groups/rocksdb.dev/permalink/549700778461774/?stream_ref=2 * Remove __thread locals in OS_MACOSX Test Plan: `make lib PLATFORM=IOS` Reviewers: ljin, haobo, dhruba, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17475
2014-04-04 22:11:44 +02:00
xcrun -sdk iphoneos $(CC) $(CFLAGS) -isysroot $(DEVICEROOT)/SDKs/iPhoneOS$(IOSVERSION).sdk -arch armv6 -arch armv7 -arch armv7s -arch arm64 -c $< -o ios-arm/$@
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
lipo ios-x86/$@ ios-arm/$@ -create -output $@
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
else
Initial checkin. git-svn-id: https://leveldb.googlecode.com/svn/trunk@2 62dab493-f737-651d-591e-8d6aee1b9529
2011-03-18 23:37:00 +01:00
.cc.o:
Remove PLATFORM_SHARED_CFLAGS when compiling .o files Summary: The flags is accidentally introduced, and resulted in problems in 3rd party release. Test Plan: run make in all platforms (when doing the 3rd party release)
2013-08-17 01:39:23 +02:00
$(CXX) $(CXXFLAGS) -c $< -o $@ $(COVERAGEFLAGS)
C binding for leveldb, better readseq benchmark for SQLite. - Added a C binding for LevelDB. May be useful as a stable ABI that can be used by programs that keep leveldb in a shared library, or for JNI API. - Replaced SQLite's readseq benchmark to a more efficient version. SQLite readseq speeds increased by about a factor of 2x from the previous version. Also updated benchmark page to reflect readseq speed up. git-svn-id: https://leveldb.googlecode.com/svn/trunk@46 62dab493-f737-651d-591e-8d6aee1b9529
2011-08-05 22:40:49 +02:00
.c.o:
Remove PLATFORM_SHARED_CFLAGS when compiling .o files Summary: The flags is accidentally introduced, and resulted in problems in 3rd party release. Test Plan: run make in all platforms (when doing the 3rd party release)
2013-08-17 01:39:23 +02:00
$(CC) $(CFLAGS) -c $< -o $@
Update from upstream @21551990 * Patch LevelDB to build for OSX and iOS * Fix race condition in memtable iterator deletion. * Other small fixes. git-svn-id: https://leveldb.googlecode.com/svn/trunk@29 62dab493-f737-651d-591e-8d6aee1b9529
2011-05-28 02:53:58 +02:00
endif
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 21:39:24 +01:00
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
# ---------------------------------------------------------------------------
# Source files dependencies detection
# ---------------------------------------------------------------------------
# Add proper dependency support so changing a .h file forces a .cc file to
# rebuild.
# The .d file indicates .cc file's dependencies on .h files. We generate such
# dependency by g++'s -MM option, whose output is a make dependency rule.
# The sed command makes sure the "target" file in the generated .d file has
# the correct path prefix.
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 21:39:24 +01:00
%.d: %.cc
Clean up the Makefile and the build scripts Summary: As Aaron suggested, there are quite some problems with our Makefile and scripts. So in this diff I did some cleanup for them and revise some part of the scripts/makefile to help people better understand some mysterious parts. Test Plan: Ran make in several modes; Ran the updated scripts. Reviewers: dhruba, emayanke, akushner Differential Revision: https://reviews.facebook.net/D12285
2013-08-14 22:29:05 +02:00
$(CXX) $(CXXFLAGS) $(PLATFORM_SHARED_CFLAGS) -MM $< -o $@
make util/env_posix.cc work under mac Summary: This diff invoves some more complicated issues in the posix environment. Test Plan: works under mac os. will need to verify dev box. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14061
2013-11-17 08:44:39 +01:00
ifeq ($(PLATFORM), OS_MACOSX)
@sed -i '' -e 's,.*:,$*.o:,' $@
else
@sed -i -e 's,.*:,$*.o:,' $@
endif
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 21:39:24 +01:00
DEPFILES = $(filter-out util/build_version.d,$(SOURCES:.cc=.d))
depend: $(DEPFILES)
A script that automatically reformat affected lines Summary: Added a script that reformat only the affected lines in a given diff. I planned to make that file as pre-commit hook but looks it's a little bit more difficult than I thought. Since I don't want to spend too much time on this task right now, I eventually added a "make command" to achieve this with a few additional key strokes. Also make the clang-format solely inherited from Google's style -- there are still debates on some of the style issues, but we can address them later once we reach a consensus. Test Plan: Did some ugly format change and ran "make format", all affected lines are formatted as expected. Reviewers: igor, sdong, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D15147
2014-01-14 09:39:42 +01:00
# if the make goal is either "clean" or "format", we shouldn't
# try to import the *.d files.
# TODO(kailiu) The unfamiliarity of Make's conditions leads to the ugly
# working solution.
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 21:39:24 +01:00
ifneq ($(MAKECMDGOALS),clean)
A script that automatically reformat affected lines Summary: Added a script that reformat only the affected lines in a given diff. I planned to make that file as pre-commit hook but looks it's a little bit more difficult than I thought. Since I don't want to spend too much time on this task right now, I eventually added a "make command" to achieve this with a few additional key strokes. Also make the clang-format solely inherited from Google's style -- there are still debates on some of the style issues, but we can address them later once we reach a consensus. Test Plan: Did some ugly format change and ran "make format", all affected lines are formatted as expected. Reviewers: igor, sdong, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D15147
2014-01-14 09:39:42 +01:00
ifneq ($(MAKECMDGOALS),format)
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
ifneq ($(MAKECMDGOALS),jclean)
ifneq ($(MAKECMDGOALS),jtest)
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
ifneq ($(MAKECMDGOALS),package)
Various build cleanups/improvements Summary: Specific changes: 1) Turn on -Werror so all warnings are errors 2) Fix some warnings the above now complains about 3) Add proper dependency support so changing a .h file forces a .c file to rebuild 4) Automatically use fbcode gcc on any internal machine rather than whatever system compiler is laying around 5) Fix jemalloc to once again be used in the builds (seemed like it wasn't being?) 6) Fix issue where 'git' would fail in build_detect_version because of LD_LIBRARY_PATH being set in the third-party build system Test Plan: make, make check, make clean, touch a header file, make sure rebuild is expected Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7887
2013-01-14 21:39:24 +01:00
-include $(DEPFILES)
endif
A script that automatically reformat affected lines Summary: Added a script that reformat only the affected lines in a given diff. I planned to make that file as pre-commit hook but looks it's a little bit more difficult than I thought. Since I don't want to spend too much time on this task right now, I eventually added a "make command" to achieve this with a few additional key strokes. Also make the clang-format solely inherited from Google's style -- there are still debates on some of the style issues, but we can address them later once we reach a consensus. Test Plan: Did some ugly format change and ran "make format", all affected lines are formatted as expected. Reviewers: igor, sdong, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D15147
2014-01-14 09:39:42 +01:00
endif
Add a jni library for rocksdb which supports Open, Get, Put, and Close. Summary: This diff contains a simple jni library for rocksdb which supports open, get, put and closeusing default options (including Options, ReadOptions, and WriteOptions.) In the usual case, Java developers can use the c++ rocksdb library in the way similar to the following: RocksDB db = RocksDB.open(path_to_db); ... db.put("hello".getBytes(), "world".getBytes(); byte[] value = db.get("hello".getBytes()); ... db.close(); Specifically, this diff has the following major classes: * RocksDB: a Java wrapper class which forwards the operations from the java side to c++ rocksdb library. * RocksDBException: ncapsulates the error of an operation. This exception type is used to describe an internal error from the c++ rocksdb library. This diff also include a simple java sample code calling c++ rocksdb library. To build the rocksdb jni library, simply run make jni, and make jtest will try to build and run the sample code. Note that if the rocksdb is not built with the default glibc that Java uses, java will try to load the wrong glibc during the run time. As a result, the sample code might not work properly during the run time. Test Plan: * make jni * make jtest Reviewers: haobo, dhruba, sdong, igor, ljin Reviewed By: dhruba CC: leveldb, xjin Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 22:19:21 +01:00
endif
endif
Package generation for Ubuntu and CentOS Summary: I put together a script to assist in the generation of deb's and rpm's. I've tested that this works on ubuntu via vagrant. I've included the Vagrantfile here, but I can remove it if it's not useful. The package.sh script should work on any ubuntu or centos machine, I just added a bit of logic in there to allow a base Ubuntu or Centos machine to be able to build RocksDB from scratch. Example output on Ubuntu 14.04: ``` root@vagrant-ubuntu-trusty-64:/vagrant# ./tools/package.sh [+] g++-4.7 is already installed. skipping. [+] libgflags-dev is already installed. skipping. [+] ruby-all-dev is already installed. skipping. [+] fpm is already installed. skipping. Created package {:path=>"rocksdb_3.5_amd64.deb"} root@vagrant-ubuntu-trusty-64:/vagrant# dpkg --info rocksdb_3.5_amd64.deb new debian package, version 2.0. size 17392022 bytes: control archive=1518 bytes. 275 bytes, 11 lines control 2911 bytes, 38 lines md5sums Package: rocksdb Version: 3.5 License: BSD Vendor: Facebook Architecture: amd64 Maintainer: rocksdb@fb.com Installed-Size: 83358 Section: default Priority: extra Homepage: http://rocksdb.org/ Description: RocksDB is an embeddable persistent key-value store for fast storage. ``` Example output on CentOS 6.5: ``` [root@localhost vagrant]# rpm -qip rocksdb-3.5-1.x86_64.rpm Name : rocksdb Relocations: /usr Version : 3.5 Vendor: Facebook Release : 1 Build Date: Mon 29 Sep 2014 01:26:11 AM UTC Install Date: (not installed) Build Host: localhost Group : default Source RPM: rocksdb-3.5-1.src.rpm Size : 96231106 License: BSD Signature : (none) Packager : rocksdb@fb.com URL : http://rocksdb.org/ Summary : RocksDB is an embeddable persistent key-value store for fast storage. Description : RocksDB is an embeddable persistent key-value store for fast storage. ``` Test Plan: How this gets used is really up to the RocksDB core team. If you want to actually get this into mainline, you might have to change `make install` such that it install the RocksDB shared object file as well, which would require you to link against gflags (maybe?) and that would require some potential modifications to the script here (basically add a depends on that package). Currently, this will install the headers and a pre-compiled statically linked object file. If that's what you want out of life, than this requires no modifications. Reviewers: ljin, yhchiang, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D24141
2014-09-30 01:09:46 +02:00
endif
Reference in New Issue Copy Permalink