Bring forward build automation changes (#11052)

This brings forward the build and release automation changes from 4.1 (#10879, #10883, #10884, #10886, #10888, #10889, #10893, #10900, #10933, #10945, #10966, #10968, #11002, and #11019) to 5.0.

Details are as follows:

* Use Github workflows for CI (#10879)

Motivation:

We should just use GitHub Actions for the CI

Modifications:

- Adjust docker / docker compose files
- Add different workflows and jobs to deploy and build the project

Result:

Don't depend on external CI services

* Fix non leak build condition

* Only use build and deploy workflows for 4.1 for now

* Add deploy job for cross compiled aarch64 (#10883)

Motivation:

We should also deploy snapshots for our cross compiled native jars.

Modifications:

- Add job and docker files for deploying cross compiled native jars
- Ensure we map the maven cache into our docker containers

Result:

Deploy aarch64 jars and re-use cache

* Use correct docker-compose file to deploy cross compiled artifacts

* Use correct docker-compose task to deploy for cross compiled artifacts

* Split pr and normal build (#10884)

Motivation:

We should better use seperate workflows for PR and normal builds

Modifications:

- Split workflows
- Better cache reuse

Result:

Cleanup

* Only deploy snapshots for one arch

Motivation:

We need to find a way to deploy SNAPSHOTS for different arch with the same timestamp. Otherwise it will cause problems.

See https://github.com/netty/netty/issues/10887

Modification:

Skip all other deploys then x86_64

Result:

Users are able to use SNAPSHOTS for x86_6

* Use maven cachen when running analyze job (#10888)

Motivation:

To prevent failures to problems while downloading dependencies we shoud cache these

Modifications:

Add maven cache

Result:

No more failures due problems while downloading dependencies

* Also include one PR job that uses boringssl (#10886)

Motivation:

When validating PRs we should also at least run one job that uses boringssl

Modifications:

- Add job that uses boringssl
- Cleanup docker compose files
- Fix buffer leak in test

Result:

Also run with boringssl when PRs are validated

* Use matrix for job configurations (#10889)

Motivation:

We can use the matrix feature to define our jobs. This reduces a lot of config

Modification:

Use job matrix

Result:

Easier to maintain

* Correctly deploy artifacts that are build on different archs (#10893)

Motivation:

We need to take special care when deploying snapshots as we need to generate the jars in multiple steps

Modifications:

- Use the nexus staging pluging to stage jars locally in multiple steps
- Add extra job that will merge these staged jars and deploy these

Result:

Fixes https://github.com/netty/netty/issues/10887

* Dont use cron for PRs

Motivation:

It doesnt make sense to use cron for PRs

Modifications:

Remove cron config

Result:

Cleanup

* We run all combinations when validate the PR, let's just use one type for normal push

Motivation:

Let us just only use one build config when building the 4.1 branch.

Modifications:

As we already do a full validation when doing the PR builds we can just only use one build config for pushes to the "main" branches

Result:

Faster build times

* Update action-docker-layer-caching (#10900)

Motivation:

We are three releases behind.

Modifications:

Update to latest version

Result:

Use up-to-date action-docker-layer-caching version

* Verify we can load native modules and add job that verifies on aarch64 as well (#10933)

Motivation:

As shown in the past we need to verify we actually can load the native as otherwise we may introduce regressions.

Modifications:

- Add new maven module which tests loading of native modules
- Add job that will also test loading on aarch64

Result:

Less likely to introduce regressions related to loading native code in the future

* Let script fail if one command fail (#10945)

Motivation:

We should use `set -e` to ensure we fail the script if one command fails.

Modifications:

Add set -e to script

Result:

Fail fast

* Use action to report unit test errors (#10966)

Motivation:

To make it easier to understand why the build fails lets use an action that will report which unit test failed

Modifications:

- Replace custom script with action-surefire-report

Result:

Easier to understand test failures

* Use custom script to check for build failures (#10968)

Motivation:

It turns out we can't use the action to check for build failures as it can't be used when a PR is done from a fork. Let's just use our simple script.

Modifications:

- Replace action with custom script

Result:

Builds for PRs that are done via forks work again.

* Publish test results after PR run (#11002)

Motivation:

To make it easier to understand why a build failed let us publish the rest results

Modifications:

Use a new workflow to be able to publish the test reports

Result:

Easier to understand why a PR did fail

* Fix test reports name

* Add workflow to cut releases (#11019)

Motivation:

Doing releases manually is error-prone, it would be better if we could do it via a workflow

Modification:

- Add workflow to cut releases
- Add related scripts

Result:

Be able to easily cut a release via a workflow

* Update build for master branch

Motivation:
The build changes were brought forward from 4.1, and contain many things specific to 4.1.

Modification:
Changed baseline Java version from 8 to 11, and changed branch references from "4.1" to "master".

Result:
Builds should now work for the master branch.

Co-authored-by: Norman Maurer <norman_maurer@apple.com>

.github/scripts/check_build_result.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+set -e
+
+if [ "$#" -ne 1 ]; then
+    echo "Expected build log as argument"
+    exit 1
+fi
+
+if grep -q 'BUILD FAILURE' $1 ; then
+    echo "Build failure detected, please inspect build log"
+    exit 1
+else
+    echo "Build successful"
+    exit 0
+fi

.github/scripts/check_leak.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+set -e
+
+if [ "$#" -ne 1 ]; then
+    echo "Expected build log as argument"
+    exit 1
+fi
+
+if grep -q 'LEAK:' $1 ; then
+    echo "Leak detected, please inspect build log"
+    exit 1
+else
+    echo "No Leak detected"
+    exit 0
+fi
+

.github/scripts/merge_local_staging.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+set -e
+if [ "$#" -lt 2 ]; then
+    echo "Expected target directory and at least one local staging directory"
+    exit 1
+fi
+TARGET=$1
+
+for ((i=2; i<=$#; i++))
+do
+  DIR="${!i}"
+  SUB_DIR=$(ls -d "${DIR}"/* | awk -F / '{print $NF}')
+
+  if [ ! -d "${TARGET}/${SUB_DIR}" ]
+  then
+      mkdir -p "${TARGET}/${SUB_DIR}"
+  fi
+  cat "${DIR}"/"${SUB_DIR}"/.index >> "${TARGET}/${SUB_DIR}"/.index
+  cp -r "${DIR}"/"${SUB_DIR}"/* "${TARGET}/${SUB_DIR}"/
+done

.github/scripts/release_checkout_tag.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e
+
+if [ "$#" -ne 1 ]; then
+    echo "Expected release.properties file"
+    exit 1
+fi
+
+TAG=$(grep scm.tag= "$1" | cut -d'=' -f2)
+
+echo "Checkout tag $TAG"
+git checkout "$TAG"
+exit 0

.github/scripts/release_rollback.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+if [ "$#" -ne 3 ]; then
+    echo "Expected release.properties file, repository name and branch"
+    exit 1
+fi
+
+TAG=$(grep scm.tag= "$1" | cut -d'=' -f2)
+git remote set-url origin git@github.com:"$2".git
+git fetch
+git checkout "$3"
+mvn -B --file pom.xml release:rollback
+git push origin :"$TAG"

.github/workflows/ci-build.yml

@@ -0,0 +1,59 @@
+name: Build project
+
+on:
+  push:
+    branches: [ "master"]
+
+  schedule:
+    - cron: '30 1 * * 1'  # At 01:30 on Monday, every Monday.
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - setup: linux-x86_64-java11
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml run build"
+
+    name: ${{ matrix.setup }}
+    steps:
+      - uses: actions/checkout@v2
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: build-${{ matrix.setup }}-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-build-${{ env.cache-name }}-
+            ${{ runner.os }}-build-
+
+      # Enable caching of Docker layers
+      - uses: satackey/action-docker-layer-caching@v0.0.11
+        continue-on-error: true
+        with:
+          key: build-${{ matrix.setup }}-docker-cache-{hash}
+          restore-keys: |
+            build-${{ matrix.setup }}-docker-cache-
+
+      - name: Build docker image
+        run: docker-compose ${{ matrix.docker-compose-build }}
+
+      - name: Build project without leak detection
+        run: docker-compose ${{ matrix.docker-compose-run }} | tee build.output
+
+      - name: Checking for test failures
+        run: ./.github/scripts/check_build_result.sh build.output
+
+      - uses: actions/upload-artifact@v2
+        if: ${{ failure() }}
+        with:
+          name: target
+          path: "**/target/"

.github/workflows/ci-deploy.yml

@@ -0,0 +1,127 @@
+name: Deploy project
+
+on:
+  push:
+    branches: [ "master" ]
+
+  schedule:
+    - cron: '30 1 * * 1'  # At 01:30 on Monday, every Monday.
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  stage-snapshot:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - setup: linux-x86_64-java11
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.11.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.11.yaml run stage-snapshot"
+          - setup: linux-aarch64
+            docker-compose-build: "-f docker/docker-compose.centos-7.yaml build"
+            docker-compose-run: "-f docker/docker-compose.centos-7.yaml run cross-compile-aarch64-stage-snapshot"
+
+    name: stage-snapshot-${{ matrix.setup }}
+    steps:
+      - uses: actions/checkout@v2
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: staging-${{ matrix.setup }}-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-staging-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-staging-${{ env.cache-name }}-
+            ${{ runner.os }}-staging-
+
+      # Enable caching of Docker layers
+      - uses: satackey/action-docker-layer-caching@v0.0.11
+        env:
+          docker-cache-name: staging-${{ matrix.setup }}-cache-docker
+        continue-on-error: true
+        with:
+          key: ${{ runner.os }}-staging-${{ env.docker-cache-name }}-{hash}
+          restore-keys: |
+            ${{ runner.os }}-staging-${{ env.docker-cache-name }}-
+
+      - name: Create local staging directory
+        run: mkdir -p ~/local-staging
+
+      - name: Build docker image
+        run: docker-compose ${{ matrix.docker-compose-build }}
+
+      - name: Stage snapshots to local staging directory
+        run: docker-compose ${{ matrix.docker-compose-run }}
+
+      - name: Upload local staging directory
+        uses: actions/upload-artifact@v2
+        with:
+          name: ${{ matrix.setup }}-local-staging
+          path: ~/local-staging
+          if-no-files-found: error
+
+  deploy-staged-snapshots:
+    runs-on: ubuntu-18.04
+    # Wait until we have staged everything
+    needs: stage-snapshot
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: '11'
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: deploy-staging-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-deploy-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-deploy-${{ env.cache-name }}-
+            ${{ runner.os }}-deploy-
+
+      # Setup some env to re-use later.
+      - name: Prepare enviroment variables
+        run: |
+          echo "LOCAL_STAGING_DIR=$HOME/local-staging" >> $GITHUB_ENV
+
+      # Hardcode the staging artifacts that need to be downloaded.
+      # These must match the matrix setups. There is currently no way to pull this out of the config.
+      - name: Download linux-aarch64 staging directory
+        uses: actions/download-artifact@v2
+        with:
+          name: linux-aarch64-local-staging
+          path: ~/linux-aarch64-local-staging
+
+      - name: Download linux-x86_64-java11 staging directory
+        uses: actions/download-artifact@v2
+        with:
+          name: linux-x86_64-java11-local-staging
+          path: ~/linux-x86_64-java11-local-staging
+
+      - name: Merge staging repositories
+        run: |
+          mkdir -p ~/local-staging/deferred
+          cat ~/linux-aarch64-local-staging/deferred/.index >>  ~/local-staging/deferred/.index
+          cp -r ~/linux-aarch64-local-staging/deferred/* ~/local-staging/deferred/
+          cat ~/linux-x86_64-java11-local-staging/deferred/.index >>  ~/local-staging/deferred/.index
+          cp -r ~/linux-x86_64-java11-local-staging/deferred/* ~/local-staging/deferred/
+
+      - uses: s4u/maven-settings-action@v2.2.0
+        with:
+          servers: |
+            [{
+              "id": "sonatype-nexus-snapshots",
+              "username": "${{ secrets.SONATYPE_USERNAME }}",
+              "password": "${{ secrets.SONATYPE_PASSWORD }}"
+            }]
+
+      - name: Deploy local staged artifacts
+        run: mvn -B --file pom.xml org.sonatype.plugins:nexus-staging-maven-plugin:deploy-staged -DaltStagingDirectory=$LOCAL_STAGING_DIR

.github/workflows/ci-pr-reports.yml

@@ -0,0 +1,31 @@
+name: PR Reports
+on:
+  workflow_run:
+    workflows: [ "Build PR" ]
+    types:
+      - completed
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - setup: linux-x86_64-java11
+          - setup: linux-x86_64-java15
+          - setup: linux-x86_64-java11-boringssl
+    steps:
+      - name: Download Artifacts
+        uses: dawidd6/action-download-artifact@v2.11.0
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          workflow: ${{ github.event.workflow_run.workflow_id }}
+          commit: ${{ github.event.workflow_run.head_commit.id }}
+          # File location set in ci-pr.yml and must be coordinated.
+          name: test-results-${{ matrix.setup }}
+      - name: Publish Test Report
+        uses: scacap/action-surefire-report@v1.0.7
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          report_paths: '**/target/surefire-reports/TEST-*.xml'
+          commit: ${{ github.event.workflow_run.head_commit.id }}
+          check_name: ${{ matrix.setup }} test reports

.github/workflows/ci-pr.yml

@@ -0,0 +1,139 @@
+name: Build PR
+
+on:
+  pull_request:
+    branches: [ "master"]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  verify-pr:
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: '11'
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: verify-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-pr-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-pr-${{ env.cache-name }}-
+            ${{ runner.os }}-pr-
+      - name: Verify with Maven
+        run: mvn verify -B --file pom.xml -DskipTests=true
+
+  build-pr-aarch64:
+    name: linux-aarch64-verify-native
+    # The host should always be Linux
+    runs-on: ubuntu-20.04
+    needs: verify-pr
+    steps:
+      - uses: actions/checkout@v2
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: build-pr-aarch64-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-pr-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-pr-${{ env.cache-name }}-
+            ${{ runner.os }}-pr-
+
+      - uses: uraimo/run-on-arch-action@v2.0.5
+        name: Run commands
+        id: runcmd
+        with:
+          arch: aarch64
+          distro: ubuntu20.04
+
+          # Not required, but speeds up builds by storing container images in
+          # a GitHub package registry.
+          githubToken: ${{ github.token }}
+
+          # Mount the .m2/repository
+          dockerRunArgs: |
+            --volume "/home/runner/.m2/repository/:/root/.m2/repository"
+
+          # Install dependencies
+          install: |
+            apt-get update -q -y
+            apt-get install -q -y openjdk-11-jdk autoconf automake libtool make tar maven git
+
+          # Compile native code and the modules it depend on and run NativeLoadingTest. This is enough to ensure
+          # we can load the native module on aarch64
+          run: |
+            JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64 mvn -pl testsuite-native -am clean package -DskipTests=true -Dcheckstyle.skip=true -DskipNativeTestsuite=false
+
+  build-pr:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - setup: linux-x86_64-java11
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml run build-leak"
+          - setup: linux-x86_64-java15
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.115.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.115.yaml run build-leak"
+          - setup: linux-x86_64-java11-boringssl
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml run build-leak-boringssl-static"
+
+    name: ${{ matrix.setup }} build
+    needs: verify-pr
+    steps:
+      - uses: actions/checkout@v2
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: build-${{ matrix.setup }}-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-pr-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-pr-${{ env.cache-name }}-
+            ${{ runner.os }}-pr-
+
+      # Enable caching of Docker layers
+      - uses: satackey/action-docker-layer-caching@v0.0.11
+        continue-on-error: true
+        with:
+          key: build-${{ matrix.setup }}-docker-cache-{hash}
+          restore-keys: |
+            build-${{ matrix.setup }}-docker-cache-
+
+      - name: Build docker image
+        run: docker-compose ${{ matrix.docker-compose-build }}
+
+      - name: Build project with leak detection
+        run: docker-compose ${{ matrix.docker-compose-run }} | tee build-leak.output
+
+      - name: Checking for test failures
+        run: ./.github/scripts/check_build_result.sh build-leak.output
+
+      - name: Checking for detected leak
+        run: ./.github/scripts/check_leak.sh build-leak.output
+
+      - name: Upload Test Results
+        if: always()
+        uses: actions/upload-artifact@v2
+        with:
+          name: test-results-${{ matrix.setup }}
+          path: '**/target/surefire-reports/TEST-*.xml'
+
+      - uses: actions/upload-artifact@v2
+        if: ${{ failure() }}
+        with:
+          name: build-${{ matrix.setup }}-target
+          path: "**/target/"
+

.github/workflows/ci-release.yml

@@ -0,0 +1,210 @@
+name: Release
+
+on:
+
+  # Releases can only be triggered via the action tab
+  workflow_dispatch:
+
+jobs:
+  prepare-release:
+    runs-on: ubuntu-18.04
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          ref: master
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: '11'
+
+      - name: Setup git configuration
+        run: |
+          git config --global user.email "netty-project-bot@users.noreply.github.com"
+          git config --global user.name "Netty Project Bot"
+
+      - name: Install SSH key
+        uses: shimataro/ssh-key-action@v2
+        with:
+          key: ${{ secrets.SSH_PRIVATE_KEY_PEM }}
+          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
+
+      # Cache .m2/repository
+      - uses: actions/cache@v2
+        env:
+          cache-name: release-cache-m2-repository
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-pr-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-pr-${{ env.cache-name }}-
+            ${{ runner.os }}-pr-
+
+      - name: Prepare release with Maven
+        run: |
+          mvn -DpreparationGoals=clean release:prepare -B --file pom.xml -DskipTests=true
+          mvn clean
+
+      - name: Checkout tag
+        run: ./.github/scripts/release_checkout_tag.sh release.properties
+
+      - name: Upload workspace
+        uses: actions/upload-artifact@v2
+        with:
+          name: prepare-release-workspace
+          path: ${{ github.workspace }}/**
+
+  stage-release-linux:
+    runs-on: ubuntu-latest
+    needs: prepare-release
+    strategy:
+      matrix:
+        include:
+          - setup: linux-x86_64-java11
+            docker-compose-build: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.11.yaml build"
+            docker-compose-run: "-f docker/docker-compose.yaml -f docker/docker-compose.centos-6.11.yaml run stage-release"
+          - setup: linux-aarch64
+            docker-compose-build: "-f docker/docker-compose.centos-7.yaml build"
+            docker-compose-run: "-f docker/docker-compose.centos-7.yaml run cross-compile-aarch64-stage-release"
+
+    name: stage-release-${{ matrix.setup }}
+
+    steps:
+      - name: Download release-workspace
+        uses: actions/download-artifact@v2
+        with:
+          name: prepare-release-workspace
+          path: ./prepare-release-workspace/
+
+      - name: Adjust mvnw permissions
+        run: chmod 755 ./prepare-release-workspace/mvnw
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: '11'
+
+      - name: Setup git configuration
+        run: |
+          git config --global user.email "netty-project-bot@users.noreply.github.com"
+          git config --global user.name "Netty Project Bot"
+
+      - name: Install SSH key
+        uses: shimataro/ssh-key-action@v2
+        with:
+          key: ${{ secrets.SSH_PRIVATE_KEY_PEM }}
+          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
+
+      # Enable caching of Docker layers
+      - uses: satackey/action-docker-layer-caching@v0.0.11
+        env:
+          docker-cache-name: staging-${{ matrix.setup }}-cache-docker
+        continue-on-error: true
+        with:
+          key: ${{ runner.os }}-staging-${{ env.docker-cache-name }}-{hash}
+          restore-keys: |
+            ${{ runner.os }}-staging-${{ env.docker-cache-name }}-
+
+      - uses: s4u/maven-settings-action@v2.2.0
+        with:
+          servers: |
+            [{
+              "id": "sonatype-nexus-staging",
+              "username": "${{ secrets.SONATYPE_USERNAME }}",
+              "password": "${{ secrets.SONATYPE_PASSWORD }}"
+            }]
+
+      - name: Create local staging directory
+        run: mkdir -p ~/local-staging
+
+      - name: Build docker image
+        working-directory: ./prepare-release-workspace/
+        run: docker-compose ${{ matrix.docker-compose-build }}
+
+      - name: Stage release to local staging directory
+        working-directory: ./prepare-release-workspace/
+        env:
+          GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
+          GPG_KEYNAME: ${{ secrets.GPG_KEYNAME }}
+          GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
+        run: docker-compose ${{ matrix.docker-compose-run }}
+
+      - name: Upload local staging directory
+        uses: actions/upload-artifact@v2
+        with:
+          name: ${{ matrix.setup }}-local-staging
+          path: ~/local-staging
+          if-no-files-found: error
+
+      - name: Rollback release on failure
+        working-directory: ./prepare-release-workspace/
+        if: ${{ failure() }}
+        # Rollback the release in case of an failure
+        run: bash ./.github/scripts/release_rollback.sh release.properties netty/netty master
+
+  deploy-staged-release:
+    runs-on: ubuntu-18.04
+    # Wait until we have staged everything
+    needs: stage-release-linux
+    steps:
+      - name: Download release-workspace
+        uses: actions/download-artifact@v2
+        with:
+          name: prepare-release-workspace
+          path: ./prepare-release-workspace/
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: '11'
+
+      - name: Setup git configuration
+        run: |
+          git config --global user.email "netty-project-bot@users.noreply.github.com"
+          git config --global user.name "Netty Project Bot"
+
+      - name: Install SSH key
+        uses: shimataro/ssh-key-action@v2
+        with:
+          key: ${{ secrets.SSH_PRIVATE_KEY_PEM }}
+          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
+
+      # Hardcode the staging artifacts that need to be downloaded.
+      # These must match the matrix setups. There is currently no way to pull this out of the config.
+      - name: Download linux-aarch64 staging directory
+        uses: actions/download-artifact@v2
+        with:
+          name: linux-aarch64-local-staging
+          path: ~/linux-aarch64-local-staging
+
+      - name: Download linux-x86_64-java11 staging directory
+        uses: actions/download-artifact@v2
+        with:
+          name: linux-x86_64-java11-local-staging
+          path: ~/linux-x86_64-java11-local-staging
+
+      # This step takes care of merging all the previous staged repositories in a way that will allow us to deploy
+      # all together with one maven command.
+      - name: Merge staging repositories
+        working-directory: ./prepare-release-workspace/
+        run: bash ./.github/scripts/merge_local_staging.sh /home/runner/local-staging/staging ~/linux-aarch64-local-staging/staging ~/linux-x86_64-java11-local-staging/staging
+
+      - uses: s4u/maven-settings-action@v2.2.0
+        with:
+          servers: |
+            [{
+              "id": "sonatype-nexus-staging",
+              "username": "${{ secrets.SONATYPE_USERNAME }}",
+              "password": "${{ secrets.SONATYPE_PASSWORD }}"
+            }]
+
+      - name: Deploy local staged artifacts
+        working-directory: ./prepare-release-workspace/
+        # If we don't want to close the repository we can add -DskipStagingRepositoryClose=true
+        run: mvn -B --file pom.xml org.sonatype.plugins:nexus-staging-maven-plugin:deploy-staged -DnexusUrl=https://oss.sonatype.org -DserverId=sonatype-nexus-staging -DaltStagingDirectory=/home/runner/local-staging -DskipStagingRepositoryClose=true
+
+      - name: Rollback release on failure
+        working-directory: ./prepare-release-workspace/
+        if: ${{ failure() }}
+        # Rollback the release in case of an failure
+        run: bash ./.github/scripts/release_rollback.sh release.properties netty/netty master

.github/workflows/codeql-analysis.yml

@@ -7,10 +7,10 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [4.1, master]
+    branches: ["4.1", master]
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [4.1, master]
+    branches: ["4.1", master]
   schedule:
     - cron: '0 13 * * 3'
 
@@ -36,6 +36,17 @@ jobs:
         # a pull request then we can checkout the head.
         fetch-depth: 2
 
+    # Cache .m2/repository
+    - uses: actions/cache@v2
+      env:
+        cache-name: verify-cache-m2-repository
+      with:
+        path: ~/.m2/repository
+        key: ${{ runner.os }}-pr-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
+        restore-keys: |
+          ${{ runner.os }}-pr-${{ env.cache-name }}-
+          ${{ runner.os }}-pr-
+
     # If this run was triggered by a pull request event, then checkout
     # the head of the pull request instead of the merge commit.
     - run: git checkout HEAD^2

docker/Dockerfile.centos6

@@ -1,7 +1,7 @@
-ARG centos_version=6
+FROM centos:6.10
-FROM centos:$centos_version
+
-# needed to do again after FROM due to docker limitation
+# Update as we need to use the vault now.
-ARG centos_version
+RUN sed -i -e 's/^mirrorlist/#mirrorlist/g' -e 's/^#baseurl=http:\/\/mirror.centos.org\/centos\/$releasever\//baseurl=http:\/\/vault.centos.org\/6.10\//g' /etc/yum.repos.d/CentOS-Base.repo
 
 # install dependencies
 RUN yum install -y \

docker/README.md

@@ -10,12 +10,6 @@ cd /path/to/netty/
 docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.centos-6.111.yaml run test
 ```
 
-## centos 7 with java 11
-
-```
-docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.centos-7.111.yaml run test
-```
-
 ## aarch64 cross compile for transport-native-epoll on X86_64
 
 ```

docker/docker-compose.centos-6.111.yaml

@@ -6,16 +6,18 @@ services:
     image: netty:centos-6-1.11
     build:
       args:
-        centos_version : "6"
         java_version : "adopt@1.11.0-9"
 
-  test:
+  build:
     image: netty:centos-6-1.11
 
-  test-leak:
+  build-leak:
     image: netty:centos-6-1.11
 
-  test-boringssl-static:
+  build-boringssl-static:
+    image: netty:centos-6-1.11
+
+  build-leak-boringssl-static:
     image: netty:centos-6-1.11
 
   shell:

docker/docker-compose.centos-6.112.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-6-1.12
-    build:
-      args:
-        centos_version : "6"
-        java_version : "adopt@1.12.0-2"
-
-  test:
-    image: netty:centos-6-1.12
-
-  test-leak:
-    image: netty:centos-6-1.12
-
-  test-boringssl-static:
-    image: netty:centos-6-1.12
-
-  shell:
-    image: netty:centos-6-1.12

docker/docker-compose.centos-6.113.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-6-1.13
-    build:
-      args:
-        centos_version : "6"
-        java_version : "adopt@1.13.0-2"
-
-  test:
-    image: netty:centos-6-1.13
-
-  test-leak:
-    image: netty:centos-6-1.13
-
-  test-boringssl-static:
-    image: netty:centos-6-1.13
-
-  shell:
-    image: netty:centos-6-1.13

docker/docker-compose.centos-6.114.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-6-1.14
-    build:
-      args:
-        centos_version : "6"
-        java_version : "adopt@1.14.0-2"
-
-  test:
-    image: netty:centos-6-1.14
-
-  test-leak:
-    image: netty:centos-6-1.14
-
-  test-boringssl-static:
-    image: netty:centos-6-1.14
-
-  shell:
-    image: netty:centos-6-1.14

docker/docker-compose.centos-6.115.yaml

@@ -6,18 +6,20 @@ services:
     image: netty:centos-6-1.15
     build:
       args:
-        centos_version : "6"
         # use zulu and not adoptjdk for now as adoptjdk15 does not support centos6
         # https://github.com/AdoptOpenJDK/openjdk-build/issues/2097
         java_version : "zulu@1.15.0-1"
 
-  test:
+  build:
     image: netty:centos-6-1.15
 
-  test-leak:
+  build-leak:
     image: netty:centos-6-1.15
 
-  test-boringssl-static:
+  build-boringssl-static:
+    image: netty:centos-6-1.15
+
+  build-leak-boringssl-static:
     image: netty:centos-6-1.15
 
   shell:

docker/docker-compose.centos-6.graalvm111.yaml

@@ -6,16 +6,18 @@ services:
     image: netty:centos-6-1.11
     build:
       args:
-        centos_version : "6"
         java_version : "graalvm-ce-java11@20.1.0"
 
-  test:
+  build:
     image: netty:centos-6-1.11
 
-  test-leak:
+  build-leak:
     image: netty:centos-6-1.11
 
-  test-boringssl-static:
+  build-boringssl-static:
+    image: netty:centos-6-1.11
+
+  build-leak-boringssl-static:
     image: netty:centos-6-1.11
 
   shell:

docker/docker-compose.centos-6.openj9111.yaml

@@ -6,16 +6,18 @@ services:
     image: netty:centos-6-openj9-1.11
     build:
       args:
-        centos_version : "6"
         java_version : "adopt-openj9@1.11.0-9"
 
-  test:
+  build:
     image: netty:centos-6-openj9-1.11
 
-  test-leak:
+  build-leak:
     image: netty:centos-6-openj9-1.11
 
-  test-boringssl-static:
+  build-boringssl-static:
+    image: netty:centos-6-openj9-1.11
+
+  build-leak-boringssl-static:
     image: netty:centos-6-openj9-1.11
 
   shell:

docker/docker-compose.centos-7.111.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-7-1.11
-    build:
-      args:
-        centos_version : "7"
-        java_version : "adopt@1.11.0-9"
-
-  test:
-    image: netty:centos-7-1.11
-
-  test-leak:
-    image: netty:centos-7-1.11
-
-  test-boringssl-static:
-    image: netty:centos-7-1.11
-
-  shell:
-    image: netty:centos-7-1.11

docker/docker-compose.centos-7.112.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-7-1.12
-    build:
-      args:
-        centos_version : "7"
-        java_version : "adopt@1.12.0-2"
-
-  test:
-    image: netty:centos-7-1.12
-
-  test-leak:
-    image: netty:centos-7-1.12
-
-  test-boringssl-static:
-    image: netty:centos-7-1.12
-
-  shell:
-    image: netty:centos-7-1.12

docker/docker-compose.centos-7.113.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-7-1.13
-    build:
-      args:
-        centos_version : "7"
-        java_version : "adopt@1.13.0-2"
-
-  test:
-    image: netty:centos-7-1.13
-
-  test-leak:
-    image: netty:centos-7-1.13
-
-  test-boringssl-static:
-    image: netty:centos-7-1.13
-
-  shell:
-    image: netty:centos-7-1.13

docker/docker-compose.centos-7.114.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-7-1.14
-    build:
-      args:
-        centos_version : "7"
-        java_version : "adopt@1.14.0-2"
-
-  test:
-    image: netty:centos-7-1.14
-
-  test-leak:
-    image: netty:centos-7-1.14
-
-  test-boringssl-static:
-    image: netty:centos-7-1.14
-
-  shell:
-    image: netty:centos-7-1.14

docker/docker-compose.centos-7.115.yaml

@@ -1,22 +0,0 @@
-version: "3"
-
-services:
-
-  runtime-setup:
-    image: netty:centos-7-1.15
-    build:
-      args:
-        centos_version : "7"
-        java_version : "adopt@1.15.0-1"
-
-  test:
-    image: netty:centos-7-1.15
-
-  test-leak:
-    image: netty:centos-7-1.15
-
-  test-boringssl-static:
-    image: netty:centos-7-1.15
-
-  shell:
-    image: netty:centos-7-1.15

docker/docker-compose.centos-7.yaml

@@ -0,0 +1,58 @@
+version: "3"
+
+services:
+
+
+  cross-compile-aarch64-runtime-setup:
+    image: netty:cross_compile_aarch64
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile.cross_compile_aarch64
+      args:
+        gcc_version: "4.9-2016.02"
+        java_version: "adopt@1.11.0-9"
+
+  cross-compile-aarch64-common: &cross-compile-aarch64-common
+    depends_on: [ cross-compile-aarch64-runtime-setup ]
+    image: netty:cross_compile_aarch64
+    volumes:
+      - ~/.ssh:/root/.ssh
+      - ~/.gnupg:/root/.gnupg
+      - ~/.m2:/root/.m2
+      - ..:/code
+    working_dir: /code
+
+  cross-compile-aarch64-deploy:
+    <<: *cross-compile-aarch64-common
+    command: /bin/bash -cl "./mvnw -Plinux-aarch64 -pl transport-native-unix-common,transport-native-epoll clean deploy -DskipTests=true"
+
+  cross-compile-aarch64-stage-snapshot:
+    <<: *cross-compile-aarch64-common
+    volumes:
+      - ~/.ssh:/root/.ssh
+      - ~/.gnupg:/root/.gnupg
+      - ~/.m2:/root/.m2
+      - ~/local-staging:/root/local-staging
+      - ..:/code
+    command: /bin/bash -cl "./mvnw -Plinux-aarch64 -pl transport-native-unix-common,transport-native-epoll clean package org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DaltStagingDirectory=/root/local-staging -DskipRemoteStaging=true -DskipTests=true"
+
+  cross-compile-aarch64-stage-release:
+    <<: *cross-compile-aarch64-common
+    environment:
+      - GPG_KEYNAME
+      - GPG_PASSPHRASE
+      - GPG_PRIVATE_KEY
+    volumes:
+      - ~/.ssh:/root/.ssh
+      - ~/.m2:/root/.m2
+      - ~/local-staging:/root/local-staging
+      - ..:/code
+    command: /bin/bash -cl "cat <(echo -e \"${GPG_PRIVATE_KEY}\") | gpg --batch --import && ./mvnw -B -Plinux-aarch64 -pl transport-native-unix-common,transport-native-epoll -am clean javadoc:jar package gpg:sign org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DnexusUrl=https://oss.sonatype.org -DserverId=sonatype-nexus-staging -DaltStagingDirectory=/root/local-staging -DskipRemoteStaging=true -DskipTests=true -Dgpg.passphrase=${GPG_PASSPHRASE} -Dgpg.keyname=${GPG_KEYNAME}"
+
+  cross-compile-aarch64-shell:
+    <<: *cross-compile-aarch64-common
+    entrypoint: /bin/bash
+
+  cross-compile-aarch64-build:
+    <<: *cross-compile-aarch64-common
+    command: /bin/bash -cl "./mvnw -pl transport-native-unix-common,transport-native-epoll clean package -Plinux-aarch64 -DskipTests=true"

docker/docker-compose.yaml

@@ -6,68 +6,61 @@ services:
     image: netty:default
     build:
       context: .
-      dockerfile: Dockerfile.centos
+      dockerfile: Dockerfile.centos6
 
   common: &common
     image: netty:default
     depends_on: [runtime-setup]
     volumes:
-      - ~/.ssh:/root/.ssh:delegated
+      - ~/.ssh:/root/.ssh
-      - ~/.gnupg:/root/.gnupg:delegated
+      - ~/.gnupg:/root/.gnupg
-      - ..:/code:delegated
+      - ~/.m2:/root/.m2
+      - ..:/code
     working_dir: /code
 
-  test-leak:
+  build-leak:
     <<: *common
-    command: /bin/bash -cl "./mvnw -Pleak clean install -Dio.netty.testsuite.badHost=netty.io -Dmaven.wagon.http.pool=false"
+    command: /bin/bash -cl "./mvnw -Pleak clean install -Dio.netty.testsuite.badHost=netty.io"
 
-  test:
+  build:
     <<: *common
-    command: /bin/bash -cl "./mvnw clean install -Dio.netty.testsuite.badHost=netty.io -Dmaven.wagon.http.pool=false"
+    command: /bin/bash -cl "./mvnw clean install -Dio.netty.testsuite.badHost=netty.io"
 
-  test-boringssl-static:
+  deploy:
     <<: *common
-    command: /bin/bash -cl "./mvnw -P boringssl clean install -Dio.netty.testsuite.badHost=netty.io -Dxml.skip=true -Dmaven.wagon.http.pool=false"
+    command: /bin/bash -cl "./mvnw clean deploy -DskipTests=true"
+
+  stage-snapshot:
+    <<: *common
+    volumes:
+      - ~/.ssh:/root/.ssh
+      - ~/.gnupg:/root/.gnupg
+      - ~/.m2:/root/.m2
+      - ~/local-staging:/root/local-staging
+      - ..:/code
+    command: /bin/bash -cl "./mvnw clean package org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DaltStagingDirectory=/root/local-staging -DskipRemoteStaging=true -DskipTests=true"
+
+  stage-release:
+    <<: *common
+    environment:
+      - GPG_KEYNAME
+      - GPG_PASSPHRASE
+      - GPG_PRIVATE_KEY
+    volumes:
+      - ~/.ssh:/root/.ssh
+      - ~/.m2:/root/.m2
+      - ~/local-staging:/root/local-staging
+      - ..:/code
+    command: /bin/bash -cl "cat <(echo -e \"${GPG_PRIVATE_KEY}\") | gpg --batch --import && ./mvnw clean javadoc:jar package gpg:sign org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DnexusUrl=https://oss.sonatype.org -DserverId=sonatype-nexus-staging -DaltStagingDirectory=/root/local-staging -DskipRemoteStaging=true -DskipTests=true -Dgpg.passphrase=${GPG_PASSPHRASE} -Dgpg.keyname=${GPG_KEYNAME}"
+
+  build-boringssl-static:
+    <<: *common
+    command: /bin/bash -cl "./mvnw -P boringssl clean install -Dio.netty.testsuite.badHost=netty.io -Dxml.skip=true"
+
+  build-leak-boringssl-static:
+    <<: *common
+    command: /bin/bash -cl "./mvnw -Pboringssl,leak clean install -Dio.netty.testsuite.badHost=netty.io -Dxml.skip=true"
 
   shell:
     <<: *common
-    environment:
-      - SANOTYPE_USER
-      - SANOTYPE_PASSWORD
-    volumes:
-      - ~/.ssh:/root/.ssh:delegated
-      - ~/.gnupg:/root/.gnupg:delegated
-      - ..:/code:delegated
-      - ~/.m2:/root/.m2:delegated
     entrypoint: /bin/bash
-
-  cross-compile-aarch64-runtime-setup:
-    image: netty:cross_compile_aarch64
-    build:
-      context: .
-      dockerfile: Dockerfile.cross_compile_aarch64
-      args:
-        gcc_version : "4.9-2016.02"
-
-  cross-compile-aarch64-shell:
-    image: netty:cross_compile_aarch64
-    depends_on: [cross-compile-aarch64-runtime-setup]
-    volumes:
-      - ~/.ssh:/root/.ssh:delegated
-      - ~/.gnupg:/root/.gnupg:delegated
-      - ..:/code:delegated
-      - ~/.m2:/root/.m2:delegated
-    entrypoint: /bin/bash
-    working_dir: /code
-
-  cross-compile-aarch64-build:
-    image: netty:cross_compile_aarch64
-    depends_on: [cross-compile-aarch64-runtime-setup]
-    volumes:
-      - ~/.ssh:/root/.ssh:delegated
-      - ~/.gnupg:/root/.gnupg:delegated
-      - ..:/code:delegated
-      - ~/.m2:/root/.m2:delegated
-    # Since we are cross compiling netty-transport-native-epoll as aarch64 which cannot be loaded on x86_64, we add `skipTests` here to skip the test.
-    command: /bin/bash -cl "pushd ./transport-native-unix-common && ../mvnw clean install -Plinux-aarch64 && popd && pushd ./transport-native-epoll && ../mvnw clean install -Plinux-aarch64 -DskipTests && popd"
-    working_dir: /code

handler/src/test/java/io/netty/handler/ssl/SslHandlerTest.java

@@ -1198,6 +1198,7 @@ public class SslHandlerTest {
                     assertEquals(expected, buffer);
                 } finally {
                     expected.release();
+                    buffer.release();
                 }
             } else {
                 throw (Throwable) obj;

scripts/finish_release.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+set -e
+
+if [ "$#" -ne 2 ]; then
+    echo "Expected staging profile id and branch name, login into oss.sonatype.org to retrieve it"
+    exit 1
+fi
+
+OS=$(uname)
+
+if [ "$OS" != "Darwin" ]; then
+    echo "Needs to be executed on macOS"
+    exit 1
+fi
+
+BRANCH=$(git branch --show-current)
+
+git fetch
+git checkout "$2"
+
+export JAVA_HOME="$JAVA8_HOME"
+
+./mvnw -Psonatype-oss-release -pl resolver-dns-native-macos,transport-native-unix-common,transport-native-kqueue clean package gpg:sign org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DstagingRepositoryId="$1" -DnexusUrl=https://oss.sonatype.org -DserverId=sonatype-nexus-staging -DskipTests=true
+
+./mvnw -Psonatype-oss-release,full,uber-staging -pl all,tarball clean package gpg:sign org.sonatype.plugins:nexus-staging-maven-plugin:deploy -DstagingRepositoryId="$1" -DnexusUrl=https://oss.sonatype.org -DserverId=sonatype-nexus-staging -DskipTests=true
+
+git checkout "$BRANCH"