Commit Graph

35 Commits

Author SHA1 Message Date
Yanqin Jin
b4ebda7a39 Make buckifier python3 compatible (#5922)
Summary:
Make buckifier/buckify_rocksdb.py run on both Python 3 and 2
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5922

Test Plan:
```
$python3 buckifier/buckify_rocksdb.py
$python3 buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fakes/module:mock1"], "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"]}}'
$python2 buckifier/buckify_rocksdb.py
$python2 buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fakes/module:mock1"], "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"]}}'
```

Differential Revision: D17920611

Pulled By: riversand963

fbshipit-source-id: cc6e2f36013a88a710d96098f6ca18cbe85e3f62
2019-10-23 13:52:27 -07:00
Yanqin Jin
925250f42f Include db_stress_tool in rocksdb tools lib (#5950)
Summary:
include db_stress_tool in rocksdb tools lib

Test Plan (on devserver):
```
$make db_stress
$./db_stress
$make all && make check
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5950

Differential Revision: D18044399

Pulled By: riversand963

fbshipit-source-id: 895585abbbdfd8b954965921dba4b1400b7af1b1
2019-10-21 19:40:35 -07:00
Yanqin Jin
30edf1874c Change buckifier to support parameterized dependencies (#5648)
Summary:
Users may desire to specify extra dependencies via buck. This PR allows users to pass additional dependencies as a JSON object so that the buckifier script can generate TARGETS file with desired extra dependencies.

Test plan (on dev server)
```
$python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fakes/module:mock1"], "extra_compiler_flags": ["-DROCKSDB_LITE", "-Os"]}}'
Generating TARGETS
Extra dependencies:
{'': {'extra_compiler_flags': [], 'extra_deps': []}, 'test_dep1': {'extra_compiler_flags': ['-O2', '-DROCKSDB_LITE'], 'extra_deps': [':fake', '//dep1/mock']}}
Generated TARGETS Summary:
- 5 libs
- 0 binarys
- 296 tests
```
Verify the TARGETS file.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5648

Differential Revision: D16565043

Pulled By: riversand963

fbshipit-source-id: a6ef02274174fcf159692d7b846e828454d01e89
2019-08-02 10:55:17 -07:00
Yanqin Jin
55f4f5486d Update buckifier templates (#5647)
Summary:
Update buckifier templates in the scripts.

Test plan (on devserver)
```
$python buckifier/buckify_rocksdb.py
```
Then
```
$git diff
```
Verify that generated TARGETS file is the same (except for indentation).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5647

Differential Revision: D16555647

Pulled By: riversand963

fbshipit-source-id: 32574a4d0e820858eab2391304dd731141719bcd
2019-07-30 16:00:35 -07:00
Mark Rambacher
c8267120d8 Add support for loading dynamic libraries into the RocksDB environment (#5281)
Summary:
This change adds a Dynamic Library class to the RocksDB Env.  Dynamic libraries are populated via the  Env::LoadLibrary method.

The addition of dynamic library support allows for a few different features to be developed:
1.  The compression code can be changed to use dynamic library support.  This would allow RocksDB to determine at run-time what compression packages were installed.  This change would eliminate the need to make sure the build-time and run-time environment had the same library set.  It would also simplify some of the Java build issues (where it attempts to build and include various packages inside the RocksDB jars).

2.  Along with other features (to be provided in a subsequent PR), this change would allow code/configurations to be added to RocksDB at run-time.  For example, the build system includes code for building an "rados" environment and adding "Cassandra" features.  Instead of these extensions being built into the base RocksDB code, these extensions could be loaded at run-time as required/appropriate, either by configuration or explicitly.

We intend to push out other changes in support of the extending RocksDB at run-time via configurations.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5281

Differential Revision: D15447613

Pulled By: riversand963

fbshipit-source-id: 452cd4f54511c0bceee18f6d9d919aae9fd25fef
2019-06-03 23:02:56 -07:00
Siying Dong
e9e0101ca4 Move test related files under util/ to test_util/ (#5377)
Summary:
There are too many types of files under util/. Some test related files don't belong to there or just are just loosely related. Mo
ve them to a new directory test_util/, so that util/ is cleaner.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5377

Differential Revision: D15551366

Pulled By: siying

fbshipit-source-id: 0f5c8653832354ef8caa31749c0143815d719e2c
2019-05-30 11:25:51 -07:00
Fosco Marotto
6c2bf9e916 Add copyright headers per FB open-source checkup tool. (#5199)
Summary:
internal task: T35568575
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5199

Differential Revision: D14962794

Pulled By: gfosco

fbshipit-source-id: 93838ede6d0235eaecff90d200faed9a8515bbbe
2019-04-18 10:55:01 -07:00
Philip Jameson
51a90415e2 Add load statements to rocksdb TARGETS files
Reviewed By: siying

Differential Revision: D13993686

fbshipit-source-id: 0c55e8952307bcf457c1d78d527a0c86b59628e8
2019-02-13 14:08:21 -08:00
Philip Jameson
a3ab59d030 Fix skylark incompatible build files in rocksdb
Reviewed By: ttsugriy

Differential Revision: D13583867

fbshipit-source-id: 8f218a9ffd9807d386ba0adc966af2a9a48ac64c
2019-01-07 13:37:40 -08:00
Yi Wu
77a8d4d476 Detect if Jemalloc is linked with the binary (#4844)
Summary:
Declare Jemalloc non-standard APIs as weak symbols, so that if Jemalloc is linked with the binary, these symbols will be replaced by Jemalloc's, otherwise they will be nullptr. This is similar to how folly detect jemalloc, but we assume the main program use jemalloc as long as jemalloc is linked: https://github.com/facebook/folly/blob/master/folly/memory/Malloc.h#L147
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4844

Differential Revision: D13574934

Pulled By: yiwu-arbug

fbshipit-source-id: 7ea871beb1be7d5a1259cc38f9b78078793db2db
2019-01-03 16:30:12 -08:00
anand76
e58d76955a Fix buck dev mode fbcode builds (#4747)
Summary:
Don't enable ROCKSDB_JEMALLOC unless the build mode is opt and default
allocator is jemalloc. In dev mode, this is causing compile/link errors such as -
```
stderr: buck-out/dev/gen/rocksdb/src/rocksdb_lib#compile-pic-malloc_stats.cc.o4768b59e,gcc-5-glibc-2.23-clang/db/malloc_stats.cc.o:malloc_stats.cc:function rocksdb::DumpMallocStats(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*): error: undefined reference to 'malloc_stats_print'
clang-7.0: error: linker command failed with exit code 1
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4747

Differential Revision: D13324840

Pulled By: anand1976

fbshipit-source-id: 45ffbd4f63fe4d9e8a0473d8f066155e4ef64a14
2018-12-05 10:40:31 -08:00
Philip Jameson
6c6cb465b1 Change BUCK template files (#4624)
Summary:
Slightly changes the format of generated BUCK files for Facebook consumption. Generated targets end up looking like this:
```
cpp_library(
    name = "rocksdb_tools_lib",
    srcs = [
        "tools/db_bench_tool.cc",
        "tools/trace_analyzer_tool.cc",
        "util/testutil.cc",
    ],
    auto_headers = AutoHeaders.RECURSIVE_GLOB,
    arch_preprocessor_flags = rocksdb_arch_preprocessor_flags,
    compiler_flags = rocksdb_compiler_flags,
    preprocessor_flags = rocksdb_preprocessor_flags,
    deps = [":rocksdb_lib"],
    external_deps = rocksdb_external_deps,
)
```
Instead of
```
cpp_library(
    name = "rocksdb_tools_lib",
    srcs = [
        "tools/db_bench_tool.cc",
        "tools/trace_analyzer_tool.cc",
        "util/testutil.cc",
    ],
    headers = AutoHeaders.RECURSIVE_GLOB,
    arch_preprocessor_flags = rocksdb_arch_preprocessor_flags,
    compiler_flags = rocksdb_compiler_flags,
    preprocessor_flags = rocksdb_preprocessor_flags,
    deps = [":rocksdb_lib"],
    external_deps = rocksdb_external_deps,
)
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4624

Reviewed By: riversand963

Differential Revision: D12906711

Pulled By: philipjameson

fbshipit-source-id: 32ab64a3390cdcf2c4043ff77517ac1ad58a5e2b
2018-11-02 14:22:13 -07:00
Yi Wu
f60c4e5a58 Set -DROCKSDB_JEMALLOC for buck build if jemalloc presents (#4489)
Summary:
Set the macro if default allocator is jemalloc. It doesn't handle the case when allocator is specified, e.g.
```
cpp_binary(
    name="xxx"
    allocator="jemalloc", # or "malloc" or something else
    deps=["//rocksdb:rocksdb"],
)
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4489

Differential Revision: D10363683

Pulled By: yiwu-arbug

fbshipit-source-id: 5da490336a8e78e0feb0900c29e8036e7ec6f12b
2018-10-15 11:41:47 -07:00
Yi Wu
31d46993cc Update TARGETS file template (#4426)
Summary:
Update template of TARGETS file according to recent changes in #4371 , #4363 and dbf44c314b.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4426

Differential Revision: D10025053

Pulled By: yiwu-arbug

fbshipit-source-id: e6a0a702bfd401fc1af240ee446f5690f0bcd85d
2018-09-25 14:14:01 -07:00
Philip Jameson
a876995ed4 Grab straggler files to explicitly import AutoHeaders
Summary: There were a few files that were missed when AutoHeaders were moved to their own file. Add explicit loads

Reviewed By: yfeldblum

Differential Revision: D9499942

fbshipit-source-id: 942bf3a683b8961e1b6244136f6337477dcc45af
2018-08-28 21:28:55 -07:00
Siying Dong
a61ff876a1 Remove two CI tests (#4110)
Summary:
Two CI tests never pass because of the environment problem. Delete them.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4110

Differential Revision: D8805713

Pulled By: siying

fbshipit-source-id: 6eb4813dc2094ee2045ec8ede7fe8967d546d6e8
2018-07-12 11:43:25 -07:00
Peter (Stig) Edwards
2694b6dc26 Remove unused imports, from python scripts. (#4057)
Summary:
Also remove redefined variable.
As reported on https://lgtm.com/projects/g/facebook/rocksdb/
Closes https://github.com/facebook/rocksdb/pull/4057

Differential Revision: D8648342

Pulled By: ajkr

fbshipit-source-id: afd2ba84d1364d316010179edd44777e64ca9183
2018-06-26 12:43:04 -07:00
Yi Wu
58c221440c Update TARGETS file (#4028)
Summary:
-Wshorten-64-to-32 is invalid flag in fbcode. Changing it to -Warrowing.
Closes https://github.com/facebook/rocksdb/pull/4028

Differential Revision: D8553694

Pulled By: yiwu-arbug

fbshipit-source-id: 1523cbcb4c76cf1d2b10a4d28b5f58c78e6cb876
2018-06-21 14:42:39 -07:00
Fosco Marotto
d9bfb35d31 Update buckifier and TARGETS
Summary:
Some flags used via make were not applied in the buckifier/targets file, causing some failures to be missed by testing infra ( ie the one fixed by #3434 )
Closes https://github.com/facebook/rocksdb/pull/3452

Differential Revision: D7457419

Pulled By: gfosco

fbshipit-source-id: e4aed2915ca3038c1485bbdeebedfc33d5704a49
2018-03-30 14:26:53 -07:00
Siying Dong
8bc41f4f5d Update TARGETS
Summary:
Watch the build
Closes https://github.com/facebook/rocksdb/pull/3533

Differential Revision: D7063777

Pulled By: siying

fbshipit-source-id: db9cdfc362a8d281dada6513ab034a6d6f0d552e
2018-03-06 12:27:28 -08:00
Mark Isaacson
b8eb32f8cf Suppress lint in old files
Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues.

Reviewed By: yiwu-arbug

Differential Revision: D6821806

fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c
2018-01-29 12:56:42 -08:00
Andres Suarez
fad14050ae Remove import use from TARGETS
Summary:
We're moving away from `import`. The equivalent internal construct that
gets the directory from `fbcode/` is `package_name()`. This is a
Skylark friendly wrapper around [`get_base_path`].

The additional whitespace change is from running `python ./buckifier/buckify_rocksdb.py`.

[`get_base_path`]: https://buckbuild.com/function/get_base_path.html
Closes https://github.com/facebook/rocksdb/pull/3210

Reviewed By: yiwu-arbug

Differential Revision: D6451242

Pulled By: zertosh

fbshipit-source-id: 445757261de0ec89d5d332c1ba9af097086326dc
2017-11-30 15:27:34 -08:00
Yi Wu
54095d3389 TARGETS file not include tests in opt mode
Summary:
Do not build the tests in opt mode, since SyncPoint and other test code will not be included.
Closes https://github.com/facebook/rocksdb/pull/3204

Differential Revision: D6431154

Pulled By: yiwu-arbug

fbshipit-source-id: c404ef042c1a6f679e5c1dc57600b3d8cb52fc28
2017-11-30 10:56:58 -08:00
Yi Wu
dd49f89466 Fix TARGETS lint warnings.
Summary:
Fix buckifier script and regenerate TARGETS file with no lint warnings.
Closes https://github.com/facebook/rocksdb/pull/3170

Differential Revision: D6328993

Pulled By: yiwu-arbug

fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2
2017-11-15 14:28:34 -08:00
Andrew Gallagher
5449c0990b rocksdb: make buildable on aarch64
Summary:
- Remove default arch-specified flags.
- Move non-default arch-specific flags to arch-specific param.

Reviewed By: yiwu-arbug

Differential Revision: D5597499

fbshipit-source-id: c53108ac39c73ac36893d3fd9aaf3b5e3080f1ae
2017-08-13 17:13:54 -07:00
Alan Somers
5883a1ae24 Fix /bin/bash shebangs
Summary:
"/bin/bash" is a Linuxism.  "/usr/bin/env bash" is portable.
Closes https://github.com/facebook/rocksdb/pull/2646

Differential Revision: D5556259

Pulled By: ajkr

fbshipit-source-id: cbffd38ecdbfffb2438969ec007ab345ed893ccb
2017-08-03 15:56:46 -07:00
Yi Wu
3a3fb00b7a TARGETS file not setting sse explicitly
Summary:
We don't need to set them explicitly.
Closes https://github.com/facebook/rocksdb/pull/2660

Differential Revision: D5514141

Pulled By: yiwu-arbug

fbshipit-source-id: 10edebfc3cfe0afc00a34519f87fcea4d65069ae
2017-07-27 17:41:36 -07:00
Islam AbdelRahman
216644c61c enable UBSAN macro in TARGETS
Summary:
simply enable the macro in internal build, it wont hurt other sanitizers and will fix UBSAN issues
Closes https://github.com/facebook/rocksdb/pull/2625

Differential Revision: D5475897

Pulled By: IslamAbdelRahman

fbshipit-source-id: 262c6fd5de3c1906f4b29e55b39110f125f41057
2017-07-24 10:54:37 -07:00
Islam AbdelRahman
132013366d Make TARGETS file portable
Summary:
Instead of hard coding the path of the internal repo.
Make TARGETS file work anywhere in fbcode
Closes https://github.com/facebook/rocksdb/pull/2586

Differential Revision: D5428122

Pulled By: IslamAbdelRahman

fbshipit-source-id: 21adec82bfbff14ea93532bee789b5f5bbee5b01
2017-07-14 15:45:36 -07:00
Siying Dong
afbef65187 Bug fix: Fast CRC Support printing is not honest
Summary:
11c5d4741a introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts.
Closes https://github.com/facebook/rocksdb/pull/2513

Differential Revision: D5343802

Pulled By: yiwu-arbug

fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05
2017-06-28 21:41:42 -07:00
Yi Wu
982cec22af Fix TARGETS file tests list
Summary:
1. The buckifier script assume each test "foo" comes with a .cc file of the same name (i.e. foo.cc). Update cassandra tests to follow this pattern so that the buckifier script can recognize them.
2. add blob_db_test
Closes https://github.com/facebook/rocksdb/pull/2506

Differential Revision: D5331517

Pulled By: yiwu-arbug

fbshipit-source-id: 86f3eba471fc621186ab44cbd073b6162cde8e57
2017-06-27 14:12:02 -07:00
Yi Wu
b49b371092 allow numa >= 2.0.8
Summary:
Allow numa >= 2.0.8 in buck TARGET file.
Closes https://github.com/facebook/rocksdb/pull/2504

Differential Revision: D5330550

Pulled By: yiwu-arbug

fbshipit-source-id: 8ffb6167b4ad913877eac16a20a91023b31f8d41
2017-06-27 11:27:02 -07:00
Aaron Gao
e7612798b5 update buckifer/TARGETS
Summary:
update targets file for release
Closes https://github.com/facebook/rocksdb/pull/2358

Differential Revision: D5115705

Pulled By: lightmark

fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794
2017-05-24 11:56:57 -07:00
Siying Dong
a22ed4eab1 internal_repo_rocksdb to build Java and RocksDB LITE
Summary: Build Java and RocksDB LITE as a customized unit test under internal_repo_rocksdb. One thing I'm not sure is that whether these two tests are triggered in every flavor.

Reviewed By: IslamAbdelRahman

Differential Revision: D4855868

fbshipit-source-id: 82a1628b458744d7692bbd29ef7424cca1294031
2017-04-12 15:13:41 -07:00
Islam AbdelRahman
a30b75cdcf Add buckifier script to github repo
Summary:
Add buckifier script and TARGETS file to github repo
Closes https://github.com/facebook/rocksdb/pull/2083

Differential Revision: D4825822

Pulled By: IslamAbdelRahman

fbshipit-source-id: 205f4a7
2017-04-04 16:24:26 -07:00