rustfs/scripts/test/rustfs_heal_test.md

RustFS Heal Test

Node-outage heal test driven by scripts/test/rustfs_heal_test.sh, based on the Obsidian note "RustFS Heal 测试步骤". Uses the same 3-node test environment as the pool expansion test (vm000 vm001 vm002).

All status checks talk to the RustFS admin API directly (SigV4-signed, jq assertions), no rc required.

What it does

1. Downloads the .deb package on all nodes (release tag or a direct URL such as the nightly/R2 package).
2. Installs it, writes the 3x4 config (http://rustfs-node{1...3}:9000/data/rustfs{1...4}/mnmd), starts all three nodes simultaneously, verifies the cluster is up.
3. Writes data with warp while monitoring disk usage on the surviving nodes (df -B1G | grep /data/rustfs):
   • when both surviving nodes reach STOP_NODE_AT_GB (default 15 GiB), stop the outage node (vm002, OUTAGE_NODE_INDEX=2);
   • keep writing until both surviving nodes reach WARP_STOP_AT_GB (default 40 GiB), then stop warp.
4. Restarts the outage node.
5. Starts cluster heal: POST /rustfs/admin/v3/heal/ with body {"recursive":true} (retried, returns a clientToken).
6. Monitors the heal task via POST /rustfs/admin/v3/heal/?clientToken=<token> until the server verdict is a terminal success (finished/completed) with objects_failed == 0.
7. Result analysis: heal stats (scanned/healed/failed), an S3 read-back verification of the written objects (list the test bucket and GET a sample — every read must succeed), per-node disk usage (observability), pass/fail verdict.

Success is the server's own scan/repair verdict (heal finished, 0 failed) plus an end-to-end data read-back; per-node disk usage is logged as observability, not a pass gate (EC distributes different shards per node, so a fixed per-node GB target is not a meaningful invariant).

Self-hosted runner prerequisites

• Register the admin host (e.g. heal) as a runner with the smoke-testing label.
• Install jq, openssl, curl and warp on the runner. rc is not required.
• The runner user must be able to SSH to vm000/vm001/vm002 without a password prompt; nodes need passwordless sudo for the SSH user and resolvable rustfs-node* hostnames.
• Admin API credentials need the admin:server-info, admin:heal and admin:rebalance actions.

Configuration

Same repository secrets/variables as the pool expansion workflow:

Kind	Name	Purpose
Secret	RUSTFS_ACCESS_KEY	RustFS access key (default rustfs@test)
Secret	RUSTFS_SECRET_KEY	RustFS secret key (default rustfs@test)
Var	RUSTFS_API_ENDPOINT	Admin API endpoint, e.g. http://127.0.0.1:9000 (RUSTFS_RC_ENDPOINT fallback)
Var	RUSTFS_NODES	vm000 vm001 vm002
Var	RUSTFS_SSH_USER	azureuser
Var	RUSTFS_NIGHTLY_PACKAGE_URL	Default nightly deb URL (defaults to the R2 latest alias)

Workflow inputs

Input	Default	Meaning
package_url	nightly	Direct .deb URL; empty = latest nightly
stop_node_gb	15	Stop outage node at N GiB on survivors
warp_stop_gb	40	Stop warp at N GiB on survivors
cleanup_before	true	Reset nodes before the test
cleanup_after	true	Reset nodes after the test

⚠️ --reset purges the rustfs package and deletes the data directories on all nodes. Only run against a dedicated test environment.

Manual usage

./scripts/test/rustfs_heal_test.sh --all -y \
  --package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
  --endpoint http://127.0.0.1:9000

./scripts/test/rustfs_heal_test.sh --steps 5,6,7
./scripts/test/rustfs_heal_test.sh --reset -y

Known issues

• Nightly builds gate pool/rebalance activation on a live fleet capability proof (rustfs/backlog#2031); the script retries heal/rebalance starts and prints a hint when the signature appears.
• The cluster-level GET /rustfs/admin/v3/background-heal/status aggregator returns 501 in the single-pool 3x4 topology (no notification system), so the script monitors the started heal task via its clientToken instead.
• The heal task may report progress: null while running; the script logs this as evidence (rustfs/backlog#2035) rather than coercing it to zero, and reads the canonical camelCase progress fields (objectsScanned/objectsHealed/objectsFailed/progressPercentage) with a snake_case fallback.
• The server-side per-task heal timeout defaults to 5 minutes; the script writes RUSTFS_HEAL_TASK_TIMEOUT_SECS=21600 (6h) into the node config so a multi-tens-of-GiB heal can finish. The background scanner is disabled (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the explicit heal is the only repair mechanism and the outage effect stays observable.