mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-19 10:03:13 +08:00
xiaomage
6eb200022e
Adds a fault-tolerance suite that verifies read/write behavior under drive and node loss against the erasure-coding contract and snapshots health-endpoint responses at every degradation tier. Scenarios are derived from product source (default_parity_count, erasure set sizing): - A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives - B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes - C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down lands exactly on the read-quorum boundary; 2 nodes down breaks it - C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads inside the reported divergence window (read quorum met while the lock majority is broken) By default a reads-refused-despite-met-read-quorum observation is reported as known-divergence without failing the suite; the strict input escalates it. Chain order becomes: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security -> replication -> fault-tolerance -> performance. Covers the findings of the 2026-09 external degradation report (reads at read quorum, health-endpoint readiness truthfulness, degradedReasons capture) as automated regression probes.
62 lines
2.4 KiB
YAML
62 lines
2.4 KiB
YAML
# Copyright 2024 RustFS Team
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
# Functional chain driver: runs the ten functional suites in a fixed order
|
|
# (upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
|
|
# replication -> fault-tolerance -> performance). Each suite attempts the next handoff even
|
|
# when its tests fail.
|
|
#
|
|
# Each suite workflow can still be dispatched standalone (workflow_dispatch);
|
|
# only chain-triggered runs forward to the next suite via repository_dispatch,
|
|
# so a standalone run never drags the rest of the chain behind it.
|
|
#
|
|
# Why not workflow_run chaining: GitHub does not guarantee delivery of
|
|
# workflow_run events (they are fire-and-forget), and the head-SHA filter made
|
|
# newly added suites (storage) unable to trigger at all. Explicit
|
|
# repository_dispatch handoffs are verifiable and re-drivable.
|
|
|
|
name: RustFS Functional Chain
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
workflow_run:
|
|
# Entry point: start the chain after the nightly build completes. The
|
|
# build's own conclusion does not gate the chain; each suite reports its
|
|
# own result to rustfs/backlog and the dashboard.
|
|
workflows: ["Nightly GNU Build"]
|
|
types: [completed]
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
start-chain:
|
|
name: Start functional chain (upgrade first)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
if: ${{ github.event_name == 'workflow_dispatch' || (github.event_name == 'workflow_run' && github.event.workflow_run.event == 'schedule') }}
|
|
steps:
|
|
- name: Dispatch first suite (upgrade)
|
|
env:
|
|
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
|
run: |
|
|
set -euo pipefail
|
|
if [ -z "${GH_TOKEN:-}" ]; then
|
|
echo "PF_TESTING_GH_TOKEN is not configured; cannot start the functional chain" >&2
|
|
exit 1
|
|
fi
|
|
gh api --method POST repos/rustfs/rustfs/dispatches \
|
|
-f event_type='rustfs-chain-upgrade' \
|
|
-F 'client_payload[from_suite]=nightly-build'
|