92731b6b4a
Summary: Primarily, this change adds a way to work around a bug limiting the effective output (and therefore debugability) of the Linux builds using parallel make. We would get make[1]: write error: stdout probably due to a kernel bug, apparently affecting both available ubuntu 16 machine images (maybe not affecting docker images, less horsepower). https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393 Now in the CircleCI config, make output on Ubuntu is piped through a custom 'cat' that ignores EAGAIN errors, which seems to fix the problem. Significant other changes: * Add another linux build that combines * LIB_MODE=shared, to ensure this works with compile and unit test execution * Alternative rocksdb namespace, to ensure this works (not rely on Travis) * ASSERT_STATUS_CHECKED=1, but with building all unit tests and running those expected to pass with it * Run release build with and without gflags. (Was running only without, ignore large swaths of code in a normal release build! Two regressions in this build, only with gflags, in the last week not caught by CI!) * Use gflags with unity and LITE build, as typical case. Debugability improvements: * Use V=1 to show commands being executed (thanks to EAGAIN work-around) * Print kernel version and compiler versions as part of V=1 output from Makefile Cosmetic other changes: * Put more commands on one line, for less clutter in CircleCI output pages * Remove redundant "all" in "make all check" and put make command options before targets * Change some recursive "make clean" into dependency on "clean," toward minimizing unnecessary overhead (detect platform, build version, etc.) of extra recursive makes Pull Request resolved: https://github.com/facebook/rocksdb/pull/7078 Reviewed By: siying Differential Revision: D22391647 Pulled By: pdillinger fbshipit-source-id: d446fccf5a8c568b37dc8748621c8a5c546fe135
202 lines
8.0 KiB
YAML
202 lines
8.0 KiB
YAML
version: 2.1
|
|
|
|
orbs:
|
|
win: circleci/windows@2.4.0
|
|
|
|
executors:
|
|
windows-2xlarge:
|
|
machine:
|
|
image: 'windows-server-2019-vs2019:201908-06'
|
|
resource_class: windows.2xlarge
|
|
shell: bash.exe
|
|
|
|
jobs:
|
|
build-linux:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: pyenv global 3.5.2
|
|
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
|
|
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
|
|
|
|
build-linux-shared_lib-alt_namespace-status_checked:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: pyenv global 3.5.2
|
|
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
|
|
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 ASSERT_STATUS_CHECKED=1 LIB_MODE=shared OPT="-DROCKSDB_NAMESPACE=alternative_rocksdb_ns" make V=1 -j32 all check_some | .circleci/cat_ignore_eagain
|
|
|
|
build-linux-release:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
|
|
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
|
|
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
|
|
- run: make V=1 -j32 release | .circleci/cat_ignore_eagain
|
|
- run: ./db_stress --version # ensure with gflags
|
|
|
|
build-linux-lite:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: pyenv global 3.5.2
|
|
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
|
|
- run: SKIP_FORMAT_BUCK_CHECKS=1 PRINT_PARALLEL_OUTPUTS=1 LITE=1 make V=1 J=32 -j32 check | .circleci/cat_ignore_eagain
|
|
|
|
build-linux-lite-release:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: large
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
|
|
- run: if ./db_stress --version; then false; else true; fi # ensure without gflags
|
|
- run: sudo apt-get update -y && sudo apt-get install -y libgflags-dev
|
|
- run: LITE=1 make V=1 -j32 release | .circleci/cat_ignore_eagain
|
|
- run: ./db_stress --version # ensure with gflags
|
|
|
|
build-linux-clang-no-test:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: sudo apt-get update -y && sudo apt-get install -y clang libgflags-dev
|
|
- run: CC=clang CXX=clang++ USE_CLANG=1 PORTABLE=1 make V=1 -j32 all | .circleci/cat_ignore_eagain
|
|
|
|
build-linux-clang10-no-test:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
|
|
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
|
|
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
|
|
- run: CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 all | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
|
|
|
|
build-linux-clang10-asan:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: pyenv global 3.5.2
|
|
- run: echo "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
|
|
- run: echo "deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" | sudo tee -a /etc/apt/sources.list
|
|
- run: sudo apt-get update -y && sudo apt-get install -y clang-10 libgflags-dev
|
|
- run: SKIP_FORMAT_BUCK_CHECKS=1 COMPILE_WITH_ASAN=1 CC=clang-10 CXX=clang++-10 ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 PRINT_PARALLEL_OUTPUTS=1 make V=1 -j32 check | .circleci/cat_ignore_eagain # aligned new doesn't work for reason we haven't figured out
|
|
|
|
build-linux-cmake:
|
|
machine:
|
|
image: ubuntu-1604:201903-01
|
|
resource_class: 2xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: (mkdir build && cd build && cmake -DWITH_GFLAGS=0 .. && make V=1 -j32) | .circleci/cat_ignore_eagain
|
|
|
|
build-linux-unity:
|
|
docker: # executor type
|
|
- image: gcc:latest
|
|
resource_class: xlarge
|
|
steps:
|
|
- checkout # check out the code in the project directory
|
|
- run: apt-get update -y && apt-get install -y libgflags-dev
|
|
- run: TEST_TMPDIR=/dev/shm && make V=1 -j16 unity_test | .circleci/cat_ignore_eagain
|
|
|
|
build-windows:
|
|
executor: windows-2xlarge
|
|
parameters:
|
|
extra_cmake_opt:
|
|
default: ""
|
|
type: string
|
|
environment:
|
|
THIRDPARTY_HOME: C:/Users/circleci/thirdparty
|
|
CMAKE_HOME: C:/Users/circleci/thirdparty/cmake-3.16.4-win64-x64
|
|
CMAKE_BIN: C:/Users/circleci/thirdparty/cmake-3.16.4-win64-x64/bin/cmake.exe
|
|
CMAKE_GENERATOR: Visual Studio 16 2019
|
|
SNAPPY_HOME: C:/Users/circleci/thirdparty/snappy-1.1.7
|
|
SNAPPY_INCLUDE: C:/Users/circleci/thirdparty/snappy-1.1.7;C:/Users/circleci/thirdparty/snappy-1.1.7/build
|
|
SNAPPY_LIB_DEBUG: C:/Users/circleci/thirdparty/snappy-1.1.7/build/Debug/snappy.lib
|
|
|
|
steps:
|
|
- checkout
|
|
- run:
|
|
name: "Install thirdparty dependencies"
|
|
command: |
|
|
mkdir ${THIRDPARTY_HOME}
|
|
cd ${THIRDPARTY_HOME}
|
|
echo "Installing CMake..."
|
|
curl --fail --silent --show-error --output cmake-3.16.4-win64-x64.zip --location https://github.com/Kitware/CMake/releases/download/v3.16.4/cmake-3.16.4-win64-x64.zip
|
|
unzip -q cmake-3.16.4-win64-x64.zip
|
|
echo "Building Snappy dependency..."
|
|
curl --fail --silent --show-error --output snappy-1.1.7.zip --location https://github.com/google/snappy/archive/1.1.7.zip
|
|
unzip -q snappy-1.1.7.zip
|
|
cd snappy-1.1.7
|
|
mkdir build
|
|
cd build
|
|
${CMAKE_BIN} -G "${CMAKE_GENERATOR}" ..
|
|
msbuild.exe Snappy.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
|
|
- run:
|
|
name: "Build RocksDB"
|
|
command: |
|
|
mkdir build
|
|
cd build
|
|
${CMAKE_BIN} -G "${CMAKE_GENERATOR}" -DCMAKE_BUILD_TYPE=Debug -DOPTDBG=1 -DPORTABLE=1 -DSNAPPY=1 -DJNI=1 << parameters.extra_cmake_opt >> ..
|
|
cd ..
|
|
msbuild.exe build/rocksdb.sln -maxCpuCount -property:Configuration=Debug -property:Platform=x64
|
|
- run:
|
|
name: "Test RocksDB"
|
|
shell: powershell.exe
|
|
command: |
|
|
build_tools\run_ci_db_test.ps1 -SuiteRun db_basic_test,db_test,db_test2,env_basic_test,env_test,db_merge_operand_test -Concurrency 16
|
|
|
|
workflows:
|
|
build-linux:
|
|
jobs:
|
|
- build-linux
|
|
build-linux-shared_lib-alt_namespace-status_checked:
|
|
jobs:
|
|
- build-linux-shared_lib-alt_namespace-status_checked
|
|
build-linux-lite:
|
|
jobs:
|
|
- build-linux-lite
|
|
build-linux-release:
|
|
jobs:
|
|
- build-linux-release
|
|
build-linux-lite-release:
|
|
jobs:
|
|
- build-linux-lite-release
|
|
build-linux-clang-no-test:
|
|
jobs:
|
|
- build-linux-clang-no-test
|
|
build-linux-clang10-no-test:
|
|
jobs:
|
|
- build-linux-clang10-no-test
|
|
build-linux-clang10-asan:
|
|
jobs:
|
|
- build-linux-clang10-asan
|
|
build-linux-cmake:
|
|
jobs:
|
|
- build-linux-cmake
|
|
build-linux-unity:
|
|
jobs:
|
|
- build-linux-unity
|
|
build-windows:
|
|
jobs:
|
|
- build-windows
|
|
build-windows-cxx20:
|
|
jobs:
|
|
- build-windows:
|
|
extra_cmake_opt: -DCMAKE_CXX_STANDARD=20
|