netty5/.github/workflows/ci-deploy.yml

143 lines
5.3 KiB
YAML
Raw Normal View History

# ----------------------------------------------------------------------------
# Copyright 2021 The Netty Project
#
# The Netty Project licenses this file to you under the Apache License,
# version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at:
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# ----------------------------------------------------------------------------
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
name: Deploy project
on:
push:
branches: [ "master" ]
schedule:
- cron: '30 1 * * 1' # At 01:30 on Monday, every Monday.
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
env:
MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryhandler.count=5 -Dmaven.wagon.httpconnectionManager.ttlSeconds=240
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
jobs:
stage-snapshot:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- setup: linux-x86_64-java11
docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml build"
docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml run stage-snapshot"
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
- setup: linux-aarch64
docker-compose-build: "-f docker/docker-compose.centos-7.yaml build"
docker-compose-run: "-f docker/docker-compose.centos-7.yaml run cross-compile-aarch64-stage-snapshot"
name: stage-snapshot-${{ matrix.setup }}
steps:
- uses: actions/checkout@v2
# Cache .m2/repository
- name: Cache local Maven repository
uses: actions/cache@v2
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ matrix.setup }}-${{ hashFiles('**/pom.xml') }}
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
restore-keys: |
${{ runner.os }}-maven-${{ matrix.setup }}-
${{ runner.os }}-maven-
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
# Enable caching of Docker layers
- uses: satackey/action-docker-layer-caching@v0.0.11
env:
docker-cache-name: staging-${{ matrix.setup }}-cache-docker
continue-on-error: true
with:
key: ${{ runner.os }}-staging-${{ env.docker-cache-name }}-{hash}
restore-keys: |
${{ runner.os }}-staging-${{ env.docker-cache-name }}-
- name: Create local staging directory
run: mkdir -p ~/local-staging
- name: Build docker image
run: docker-compose ${{ matrix.docker-compose-build }}
- name: Stage snapshots to local staging directory
run: docker-compose ${{ matrix.docker-compose-run }}
- name: Upload local staging directory
uses: actions/upload-artifact@v2
with:
name: ${{ matrix.setup }}-local-staging
path: ~/local-staging
if-no-files-found: error
deploy-staged-snapshots:
runs-on: ubuntu-18.04
# Wait until we have staged everything
needs: stage-snapshot
steps:
- uses: actions/checkout@v2
- name: Set up JDK 11
uses: actions/setup-java@v1
with:
java-version: '11'
# Cache .m2/repository
- name: Cache local Maven repository
uses: actions/cache@v2
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-deploy-staged-snapshots-${{ hashFiles('**/pom.xml') }}
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
restore-keys: |
${{ runner.os }}-maven-deploy-staged-snapshots-
${{ runner.os }}-maven-
Bring forward build automation changes (#11052) This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0. Details are as follows: * Use Github workflows for CI (#10879) Motivation: We should just use GitHub Actions for the CI Modifications: - Adjust docker / docker compose files - Add different workflows and jobs to deploy and build the project Result: Don't depend on external CI services * Fix non leak build condition * Only use build and deploy workflows for 4.1 for now * Add deploy job for cross compiled aarch64 (#10883) Motivation: We should also deploy snapshots for our cross compiled native jars. Modifications: - Add job and docker files for deploying cross compiled native jars - Ensure we map the maven cache into our docker containers Result: Deploy aarch64 jars and re-use cache * Use correct docker-compose file to deploy cross compiled artifacts * Use correct docker-compose task to deploy for cross compiled artifacts * Split pr and normal build (#10884) Motivation: We should better use seperate workflows for PR and normal builds Modifications: - Split workflows - Better cache reuse Result: Cleanup * Only deploy snapshots for one arch Motivation: We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems. See https://github.com/netty/netty/issues/10887 Modification: Skip all other deploys then x86_64 Result: Users are able to use SNAPSHOTS for x86_6 * Use maven cachen when running analyze job (#10888) Motivation: To prevent failures to problems while downloading dependencies we shoud cache these Modifications: Add maven cache Result: No more failures due problems while downloading dependencies * Also include one PR job that uses boringssl (#10886) Motivation: When validating PRs we should also at least run one job that uses boringssl Modifications: - Add job that uses boringssl - Cleanup docker compose files - Fix buffer leak in test Result: Also run with boringssl when PRs are validated * Use matrix for job configurations (#10889) Motivation: We can use the matrix feature to define our jobs. This reduces a lot of config Modification: Use job matrix Result: Easier to maintain * Correctly deploy artifacts that are build on different archs (#10893) Motivation: We need to take special care when deploying snapshots as we need to generate the jars in multiple steps Modifications: - Use the nexus staging pluging to stage jars locally in multiple steps - Add extra job that will merge these staged jars and deploy these Result: Fixes https://github.com/netty/netty/issues/10887 * Dont use cron for PRs Motivation: It doesnt make sense to use cron for PRs Modifications: Remove cron config Result: Cleanup * We run all combinations when validate the PR, let's just use one type for normal push Motivation: Let us just only use one build config when building the 4.1 branch. Modifications: As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches Result: Faster build times * Update action-docker-layer-caching (#10900) Motivation: We are three releases behind. Modifications: Update to latest version Result: Use up-to-date action-docker-layer-caching version * Verify we can load native modules and add job that verifies on aarch64 as well (#10933) Motivation: As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions. Modifications: - Add new maven module which tests loading of native modules - Add job that will also test loading on aarch64 Result: Less likely to introduce regressions related to loading native code in the future * Let script fail if one command fail (#10945) Motivation: We should use `set -e` to ensure we fail the script if one command fails. Modifications: Add set -e to script Result: Fail fast * Use action to report unit test errors (#10966) Motivation: To make it easier to understand why the build fails lets use an action that will report which unit test failed Modifications: - Replace custom script with action-surefire-report Result: Easier to understand test failures * Use custom script to check for build failures (#10968) Motivation: It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script. Modifications: - Replace action with custom script Result: Builds for PRs that are done via forks work again. * Publish test results after PR run (#11002) Motivation: To make it easier to understand why a build failed let us publish the rest results Modifications: Use a new workflow to be able to publish the test reports Result: Easier to understand why a PR did fail * Fix test reports name * Add workflow to cut releases (#11019) Motivation: Doing releases manually is error-prone, it would be better if we could do it via a workflow Modification: - Add workflow to cut releases - Add related scripts Result: Be able to easily cut a release via a workflow * Update build for master branch Motivation: The build changes were brought forward from 4.1, and contain many things specific to 4.1. Modification: Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master". Result: Builds should now work for the master branch. Co-authored-by: Norman Maurer <norman_maurer@apple.com>
2021-03-02 17:44:03 +01:00
# Setup some env to re-use later.
- name: Prepare enviroment variables
run: |
echo "LOCAL_STAGING_DIR=$HOME/local-staging" >> $GITHUB_ENV
# Hardcode the staging artifacts that need to be downloaded.
# These must match the matrix setups. There is currently no way to pull this out of the config.
- name: Download linux-aarch64 staging directory
uses: actions/download-artifact@v2
with:
name: linux-aarch64-local-staging
path: ~/linux-aarch64-local-staging
- name: Download linux-x86_64-java11 staging directory
uses: actions/download-artifact@v2
with:
name: linux-x86_64-java11-local-staging
path: ~/linux-x86_64-java11-local-staging
- name: Merge staging repositories
run: |
mkdir -p ~/local-staging/deferred
cat ~/linux-aarch64-local-staging/deferred/.index >> ~/local-staging/deferred/.index
cp -r ~/linux-aarch64-local-staging/deferred/* ~/local-staging/deferred/
cat ~/linux-x86_64-java11-local-staging/deferred/.index >> ~/local-staging/deferred/.index
cp -r ~/linux-x86_64-java11-local-staging/deferred/* ~/local-staging/deferred/
- uses: s4u/maven-settings-action@v2.2.0
with:
servers: |
[{
"id": "sonatype-nexus-snapshots",
"username": "${{ secrets.SONATYPE_USERNAME }}",
"password": "${{ secrets.SONATYPE_PASSWORD }}"
}]
- name: Deploy local staged artifacts
run: ./mvnw -B --file pom.xml org.sonatype.plugins:nexus-staging-maven-plugin:deploy-staged -DaltStagingDirectory=$LOCAL_STAGING_DIR