mirror of
https://github.com/mattermost/mattermost.git
synced 2026-08-29 02:28:16 +08:00
ci: move FIPS and binary params tests to weekly schedule (#36036)
* ci: move FIPS and binary params tests to weekly schedule Move low-regression-risk test suites out of the per-push/PR Server CI workflow into a new weekly scheduled workflow (Monday 1am EST / 5am UTC): - Postgres with binary parameters (1x 8-core runner) - Postgres FIPS sharded tests (4x 8-core runners + merge job) - mmctl FIPS tests (1x 8-core runner) This reduces the per-push 8-core runner demand from 14 concurrent jobs to 5 (4 Postgres shards + 1 ES), which should significantly reduce queue times that currently reach 90+ minutes during peak hours. The weekly workflow also supports workflow_dispatch for manual triggering when urgent FIPS or binary parameter verification is needed. #### Release Note ```release-note NONE ``` Co-authored-by: Claude <claude@anthropic.com> * ci: move coverage shards to 2-core runners Add a 'runner' input to server-test-template.yml (defaults to ubuntu-latest-8-cores for backward compatibility) and set coverage shards to ubuntu-22.04 (2-core). Coverage is non-blocking (allow-failure: true) so longer runtime doesn't impact PR feedback. Estimated ~20-30 min per shard on 2-core vs ~7-9 min on 8-core, but frees 4 more 8-core slots per push. Combined with the FIPS/binary-params weekly move, per-push 8-core demand drops from 14 → 4 (just the Postgres test shards + ES v8). Co-authored-by: Claude <claude@anthropic.com> * ci: decouple race detector from binary params, add nightly race job The race detector was accidentally bundled with binary params via the fullyparallel=false → RACE_MODE coupling in the test template. These test different things: - Binary params: Postgres driver binary encoding mode - Race detector: Go data race detection Changes: - Add explicit 'race-enabled' input to server-test-template.yml - Remove implicit fullyparallel→race coupling from template - Binary params now runs with fullyparallel: true (default) - New server-ci-nightly-race.yml runs -race nightly at 2am EST on ubuntu-22.04 (2-core) to avoid 8-core contention Co-authored-by: Claude <claude@anthropic.com> * ci: add push trigger for release-* branches to weekly workflow FIPS and binary params validation must run automatically on release branch pushes, not just on the weekly schedule. Without this trigger, release branches would lose FIPS/binary coverage entirely. Co-authored-by: Claude <claude@anthropic.com> * ci: use ET instead of EST in schedule comments Cron runs at fixed UTC times regardless of DST. Use ~ET to avoid implying exact EST/EDT correspondence. Co-authored-by: Claude <claude@anthropic.com> * ci: restore conditional FIPS on per-push, unshard weekly FIPS Per review feedback from @lieut-data: 1. Restore FIPS jobs in server-ci.yml with conditional execution: run on all pushes (master/release) and on PRs when go.mod changed or branch name contains 'fips'. This ensures Go upgrades and explicit FIPS work get immediate feedback. 2. Remove sharding from weekly FIPS — no speed pressure on a weekly schedule, so a single unsharded job is simpler (eliminates the 4-shard matrix + merge job). 3. Restore gomod-changed detection step in the go job. Both per-push (conditional, unsharded) and weekly (unconditional, unsharded) FIPS runs use single jobs now, reducing complexity. Co-authored-by: Claude <claude@anthropic.com> * ci: restore FIPS sharding for PR runs, remove from push events FIPS tests in server-ci.yml now only trigger on PRs where the branch name contains 'fips' or go.mod changed. Sharding (4 shards + merge) restored for fast iteration on FIPS-related PRs. Regular FIPS coverage provided by the weekly workflow (unsharded). This addresses lieut-data's review feedback to restore sharding where it matters most: during active PR iteration. Co-authored-by: Claude <claude@anthropic.com> * ci: add explicit permissions to weekly and nightly workflows Set minimum required permissions (contents: read) on both new workflow files per review feedback. Reusable workflows called via 'uses' inherit the caller's permissions. Co-authored-by: Claude <claude@anthropic.com> * ci: keep coverage shards on 8-core runners Comment out the 2-core runner override for coverage shards per Eva's feedback. Coverage stays on the default 8-core runners. Co-authored-by: Claude <claude@anthropic.com> --------- Co-authored-by: Claude <claude@anthropic.com>
This commit is contained in:
Pavel Zeman
@@ -0,0 +1,52 @@
|
||||
# Nightly race detector run.
|
||||
#
|
||||
# Runs all server tests with Go's -race detector enabled to catch
|
||||
# data races. Runs sequentially (fullyparallel: false) on a 2-core
|
||||
# runner to minimize resource usage — race detection adds significant
|
||||
# overhead that makes parallel execution impractical.
|
||||
#
|
||||
# Schedule: Nightly at ~2am ET (6am UTC)
|
||||
#
|
||||
# Previously, -race was implicitly bundled with the binary params job
|
||||
# via the fullyparallel=false → RACE_MODE coupling. This decouples
|
||||
# the two concerns: binary params tests the Postgres driver encoding;
|
||||
# this workflow tests for data races.
|
||||
name: Server CI Nightly Race
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 6 * * *" # Daily 6am UTC (~2am ET)
|
||||
workflow_dispatch: # Manual trigger for on-demand race detection
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
go:
|
||||
name: Compute Go Version
|
||||
runs-on: ubuntu-22.04
|
||||
outputs:
|
||||
version: ${{ steps.calculate.outputs.GO_VERSION }}
|
||||
steps:
|
||||
- name: Checkout mattermost project
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- name: Calculate version
|
||||
id: calculate
|
||||
working-directory: server/
|
||||
run: echo GO_VERSION=$(cat .go-version) >> "${GITHUB_OUTPUT}"
|
||||
|
||||
test-race:
|
||||
name: Race Detector
|
||||
needs: go
|
||||
uses: ./.github/workflows/server-test-template.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
name: Race Detector
|
||||
datasource: postgres://mmuser:mostest@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10
|
||||
drivername: postgres
|
||||
logsartifact: race-detector-server-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: false
|
||||
fullyparallel: false
|
||||
race-enabled: true
|
||||
runner: ubuntu-22.04
|
||||
allow-failure: true
|
||||
@@ -0,0 +1,75 @@
|
||||
# Weekly Server CI jobs that don't need to run on every push/PR.
|
||||
# These are important for compliance and compatibility but have low
|
||||
# regression rates, so running them weekly saves significant 8-core
|
||||
# runner capacity (9 fewer concurrent 8-core jobs per master push).
|
||||
#
|
||||
# Schedule: Monday ~1am ET (5am UTC)
|
||||
#
|
||||
# Jobs moved here from server-ci.yml:
|
||||
# - Postgres with binary parameters (1x 8-core)
|
||||
# - Postgres FIPS unsharded (1x 8-core) — sharded runs stay in server-ci.yml for PR iteration
|
||||
# - mmctl FIPS tests (1x 8-core)
|
||||
name: Server CI Weekly
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 5 * * 1" # Monday 5am UTC (~1am ET)
|
||||
push:
|
||||
branches:
|
||||
- 'release-*'
|
||||
workflow_dispatch: # Allow manual trigger for urgent FIPS/binary verification
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
go:
|
||||
name: Compute Go Version
|
||||
runs-on: ubuntu-22.04
|
||||
outputs:
|
||||
version: ${{ steps.calculate.outputs.GO_VERSION }}
|
||||
steps:
|
||||
- name: Checkout mattermost project
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- name: Calculate version
|
||||
id: calculate
|
||||
working-directory: server/
|
||||
run: echo GO_VERSION=$(cat .go-version) >> "${GITHUB_OUTPUT}"
|
||||
|
||||
test-postgres-binary:
|
||||
name: Postgres with binary parameters
|
||||
needs: go
|
||||
uses: ./.github/workflows/server-test-template.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
name: Postgres with binary parameters
|
||||
datasource: postgres://mmuser:mostest@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10&binary_parameters=yes
|
||||
drivername: postgres
|
||||
logsartifact: postgres-binary-server-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: false
|
||||
|
||||
test-postgres-normal-fips:
|
||||
name: Postgres FIPS
|
||||
needs: go
|
||||
uses: ./.github/workflows/server-test-template.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
name: Postgres FIPS
|
||||
datasource: postgres://mmuser:mostest-fips-test@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10
|
||||
drivername: postgres
|
||||
logsartifact: postgres-server-fips-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: true
|
||||
|
||||
test-mmctl-fips:
|
||||
name: Run mmctl tests (FIPS)
|
||||
needs: go
|
||||
uses: ./.github/workflows/mmctl-test-template.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
name: mmctl
|
||||
datasource: postgres://mmuser:mostest-fips-test@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10
|
||||
drivername: postgres
|
||||
logsartifact: mmctl-fips-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: true
|
||||
@@ -195,20 +195,10 @@ jobs:
|
||||
echo "Making sure docs are updated"
|
||||
make mmctl-docs
|
||||
if [[ -n $(git status --porcelain) ]]; then echo "Please update the mmctl docs using make mmctl-docs"; exit 1; fi
|
||||
test-postgres-binary:
|
||||
if: github.event_name == 'push' # Only run postgres binary tests on master/release pushes: odds are low this regresses, so save the cycles for pull requests.
|
||||
name: Postgres with binary parameters
|
||||
needs: go
|
||||
uses: ./.github/workflows/server-test-template.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
name: Postgres with binary parameters
|
||||
datasource: postgres://mmuser:mostest@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10&binary_parameters=yes
|
||||
drivername: postgres
|
||||
logsartifact: postgres-binary-server-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: false
|
||||
fullyparallel: false
|
||||
# NOTE: Postgres with binary parameters has been moved to server-ci-weekly.yml
|
||||
# (runs Monday 1am EST / 5am UTC). Low regression risk doesn't justify
|
||||
# consuming 8-core runners on every push.
|
||||
|
||||
# -- Sharded into 4 parallel runners for ~88% wall-time improvement --
|
||||
test-postgres-normal:
|
||||
name: Postgres (shard ${{ matrix.shard }})
|
||||
@@ -255,11 +245,11 @@ jobs:
|
||||
elasticsearch-version: "8.9.0"
|
||||
test-target: "test-server-elasticsearch"
|
||||
|
||||
# FIPS tests: run on PRs when go.mod changed or branch name contains "fips".
|
||||
# Sharded for fast iteration. Weekly workflow provides regular full coverage.
|
||||
test-postgres-normal-fips:
|
||||
# Always run on pushes to master/release branches.
|
||||
# For PRs, run when the branch name contains "fips" or any go.mod was changed.
|
||||
if: github.event_name == 'push' || contains(github.head_ref, 'fips') || needs.go.outputs.gomod-changed == 'true'
|
||||
name: Postgres FIPS (shard ${{ matrix.shard }})
|
||||
if: contains(github.head_ref, 'fips') || needs.go.outputs.gomod-changed == 'true'
|
||||
name: "Postgres FIPS (shard ${{ matrix.shard }})"
|
||||
needs: go
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -305,6 +295,7 @@ jobs:
|
||||
enablecoverage: true
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: false
|
||||
# runner: ubuntu-22.04 # Coverage is non-blocking (allow-failure), no need for 8-core
|
||||
shard-index: ${{ matrix.shard }}
|
||||
shard-total: 4
|
||||
test-mmctl:
|
||||
@@ -320,9 +311,8 @@ jobs:
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: false
|
||||
test-mmctl-fips:
|
||||
if: contains(github.head_ref, 'fips') || needs.go.outputs.gomod-changed == 'true'
|
||||
name: Run mmctl tests (FIPS)
|
||||
# Skip FIPS testing for forks, which won't have docker login credentials.
|
||||
if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
|
||||
needs: go
|
||||
uses: ./.github/workflows/mmctl-test-template.yml
|
||||
secrets: inherit
|
||||
@@ -330,9 +320,10 @@ jobs:
|
||||
name: mmctl
|
||||
datasource: postgres://mmuser:mostest-fips-test@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10
|
||||
drivername: postgres
|
||||
logsartifact: mmctl-test-logs
|
||||
logsartifact: mmctl-fips-test-logs
|
||||
go-version: ${{ needs.go.outputs.version }}
|
||||
fips-enabled: true
|
||||
|
||||
build-mattermost-server:
|
||||
name: Build mattermost server app
|
||||
needs: go
|
||||
|
||||
@@ -50,6 +50,16 @@ on:
|
||||
required: false
|
||||
type: number
|
||||
default: 1
|
||||
runner:
|
||||
description: "GitHub-hosted runner label (default: ubuntu-latest-8-cores)"
|
||||
required: false
|
||||
type: string
|
||||
default: "ubuntu-latest-8-cores"
|
||||
race-enabled:
|
||||
description: "Run tests with Go race detector (-race)"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
id-token: write
|
||||
@@ -58,7 +68,7 @@ permissions:
|
||||
jobs:
|
||||
test:
|
||||
name: ${{ inputs.name }}
|
||||
runs-on: ubuntu-latest-8-cores
|
||||
runs-on: ${{ inputs.runner }}
|
||||
continue-on-error: ${{ inputs.allow-failure }} # Used to avoid blocking PRs in case of flakiness
|
||||
env:
|
||||
COMPOSE_PROJECT_NAME: ghactions
|
||||
@@ -175,7 +185,7 @@ jobs:
|
||||
env:
|
||||
BUILD_IMAGE: ${{ steps.build.outputs.BUILD_IMAGE }}
|
||||
run: |
|
||||
if [[ ${{ github.ref_name }} == 'master' && ${{ inputs.fullyparallel }} != true && "${{ inputs.test-target }}" == "test-server" ]]; then
|
||||
if [[ "${{ inputs.race-enabled }}" == "true" ]]; then
|
||||
export RACE_MODE="-race"
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user