Summary:
Since they operate at distinct abstraction layers, I thought it
was prudent to combine with EncryptedEnv CI test for each PR, for efficiency
in testing. Also added supported compressions to sst_dump --help output
so that CI job can verify no compiled-in compression support.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850
Test Plan: CI, some manual stuff
Reviewed By: riversand963
Differential Revision: D35682346
Pulled By: pdillinger
fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
Summary:
Especially after updating to C++17, I don't see a compelling case for
*requiring* any folly components in RocksDB. I was able to purge the existing
hard dependencies, and it can be quite difficult to strip out non-trivial components
from folly for use in RocksDB. (The prospect of doing that on F14 has changed
my mind on the best approach here.)
But this change creates an optional integration where we can plug in
components from folly at compile time, starting here with F14FastMap to replace
std::unordered_map when possible (probably no public APIs for example). I have
replaced the biggest CPU users of std::unordered_map with compile-time
pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set.
USE_FOLLY is always set in the Meta-internal buck build, and a simulation of
that is in the Makefile for public CI testing. A full folly build is not needed, but
checking out the full folly repo is much simpler for getting the dependency,
and anything else we might want to optionally integrate in the future.
Some picky details:
* I don't think the distributed mutex stuff is actually used, so it was easy to remove.
* I implemented an alternative to `folly::constexpr_log2` (which is much easier
in C++17 than C++11) so that I could pull out the hard dependencies on
`ConstexprMath.h`
* I had to add noexcept move constructors/operators to some types to make
F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a
macro to make that easier in some common cases.
* Updated Meta-internal buck build to use folly F14Map (always)
No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a
production integration for open source users.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546
Test Plan:
CircleCI tests updated so that a couple of them use folly.
Most internal unit & stress/crash tests updated to use Meta-internal latest folly.
(Note: they should probably use buck but they currently use Makefile.)
Example performance improvement: when filter partitions are pinned in cache,
they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build
a test that exercises that heavily. Build DB with
```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters
```
and test with (simultaneous runs with & without folly, ~20 times each to see
convergence)
```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache
```
Average ops/s no folly: 26229.2
Average ops/s with folly: 26853.3 (+2.4%)
Reviewed By: ajkr
Differential Revision: D34181736
Pulled By: pdillinger
fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
Summary:
* Fix LIB_MODE=shared for Meta-internal builds (use PIC libraries
appropriately)
* Fix gnu_parallel to recognize CircleCI and Travis builds as not
connected to a terminal (was previously relying on the
`| cat_ignore_eagain` stuff for Ubuntu 16). This problem could cause
timeouts that should be 10m to balloon to 5h.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9553
Test Plan: manual and CI
Reviewed By: jay-zhuang
Differential Revision: D34182886
Pulled By: pdillinger
fbshipit-source-id: e95fd8002d94c8dc414bae1975e4fd348589f2b5
Summary:
For internal build enviroment only. Developer could run the
microbenchmark without `ROCKSDB_NO_FBCODE=1`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8954
Test Plan: `$ make microbench` on dev server
Reviewed By: ajkr
Differential Revision: D31163717
Pulled By: jay-zhuang
fbshipit-source-id: 1ff59f660ca05afd0fd5c7c7dcdfd831ac365462
Summary:
Required for platform009, which is incompat with the centos
assembler.
author: pbrady@fb.com D29099768
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8733
Test Plan: CI
Reviewed By: pixelb
Differential Revision: D30687156
Pulled By: jay-zhuang
fbshipit-source-id: 796f90842cbf0ca11bad07e7d654bce1fafc4ba0
Summary:
DistributedMutex hasn't been used in the code base and enabling
`USE_FOLLY_DISTRIBUTED_MUTEX` only runs the mutex tests from third-party
lib. So disabling it for now.
The implementation may also out of date, should re-sync with folly before
using.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8584
Test Plan: CI
Reviewed By: ajkr
Differential Revision: D29888960
Pulled By: jay-zhuang
fbshipit-source-id: 3e75f73386c6ed03efb96a1400258d602a724f17
Summary:
Was seeing
./cache_test: error while loading shared libraries: libasan.so.5: cannot open shared object file: No such file or directory
etc. using COMPILE_WITH_ASAN=1 without USE_CLANG=1
Now including compiler libs in runtime ld path.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8402
Test Plan: reproduced with local builds
Reviewed By: akankshamahajan15
Differential Revision: D29107729
Pulled By: pdillinger
fbshipit-source-id: 13805b87b846b39522c9dd6a231ca245c58f1c71
Summary:
Internal builds failing
Pull Request resolved: https://github.com/facebook/rocksdb/pull/8399
Test Plan:
I can reproduce a failure by putting a bad version of `as` in
my PATH. This indicates that before this change, the custom compiler is
falsely relying on host `as`. This change fixes that, ignoring the bad
`as` on PATH.
Reviewed By: akankshamahajan15
Differential Revision: D29094159
Pulled By: pdillinger
fbshipit-source-id: c432e90404ea4d39d885a685eebbb08be9eda1c8
Summary:
Upgrade tool chain to the latest. It is done mostly manually as build_tools/build_detect_platform fails to update many of them.
Try to fix a new clang analyze warning with the new tool chain.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7251
Test Plan: "make all", "USE_CLANG=1 make all"
Reviewed By: riversand963
Differential Revision: D23091090
fbshipit-source-id: 732e5a30137837431438f85f36296406b641f975
