mirror of
https://github.com/coder/coder.git
synced 2026-09-21 20:51:01 +08:00
J. Scott Miller
d8b76831ff
## Problem
Tests that run real Terraform (e.g.
`enterprise/coderd.TestWorkspaceTemplateParamsChange`,
`provisioner/terraform.TestProvision`) intermittently fail when
populating the shared provider cache. On a cache miss,
`DownloadTFProviders` shells out to `terraform init` and `terraform
providers mirror` against the live registry, which periodically returns
transient 5xx errors from the registry/GitHub (504, 500). The
cache-population helper had no application-level retry, so a single
transient failure failed the whole test. Terraform's own registry client
only retries each request once ("the request failed after 2 attempts"),
which is insufficient for these bursts.
## Fix
`runCmd` now retries on any non-zero exit using
`github.com/coder/retry`, logging each failed attempt and preserving the
original failure message format. Retry-all is safe here because
`terraform init` and `terraform providers mirror` are idempotent: each
run reconciles the existing state in the working directory.
The backoff window is deliberately wide: 5 attempts with a 5s floor and
30s ceiling. `coder/retry` grows the delay by phi (~1.618) from the
floor and caps it at the ceiling, so attempts start at roughly t=0s, 8s,
21s, 42s, and 72s (waits of ~8.1s, ~13.1s, ~21.2s, and 30s capped, plus
command runtime). Registry/GitHub incidents typically last seconds to
minutes rather than a single unlucky request, so a narrow window would
only survive an isolated blip, while the early second attempt (~8s)
still recovers quickly from brief ones. This is affordable because the
network path runs only on a cache miss, not on every test: a populated
cache short-circuits via `os.Stat` and is reused within and across runs
(persisted by `.github/actions/test-cache`). The wait is therefore
rarely incurred and is negligible against the 20m per-package test
timeout. The only downside is a slower failure on a genuinely doomed
run.
This only affects the test provider-cache helper. Production provisioner
code, the Windows no-op path, and the CI cache strategy are unchanged.
Refs https://github.com/coder/internal/issues/1201
<details>
<summary>Investigation notes</summary>
The CI cache (`~/.cache/coderv2-test`, via `.github/actions/test-cache`)
is persisted across runs and keyed by a hash of a stable caller-supplied
label + template file contents, so cache hits avoid the network
entirely. The flake only surfaces on a cache miss (provider version
bump, monthly cache reset, or new label/template), where the populating
`terraform init` was the sole unprotected network call. This change
closes that gap without weakening the "use real Terraform" intent of the
tests.
</details>
---
🤖 Generated with Coder Agents on behalf of @jscottmiller.