fix: retry and cache e2e Coder release downloads to reduce test-e2e ssh flake (#27470)

closes DEVEX-651

## Summary

Fixes coder/internal#218 (`flake: e2e-test / test ssh`).

Despite the title, the `ssh with client v2.8.0` / `ssh with agent
v2.12.1` cases (`site/e2e/tests/outdatedCLI.spec.ts`,
`outdatedAgent.spec.ts`) are not failing because of a bug in SSH. They
fail during **setup**, in `downloadCoderVersion()`, which runs
`install.sh` to fetch an old Coder release from GitHub. Transient GitHub
errors (HTTP 403/503, surfacing as nonzero `curl` exit codes such as 22
or 1) make `install.sh` fail and take the whole ssh test down with it.

This is an external-download flake, confirmed by the recurring
`install.sh failed with code {22,1}` evidence in the issue thread and
Ethan's note ("Networking issues again").

## Changes

1. **Retry-with-backoff** (`site/e2e/helpers.ts`):
`downloadCoderVersion()` now retries `install.sh` up to 5 times with
exponential backoff and jitter (~1s, 2s, 4s, 8s). A single transient
download failure no longer fails the test. `install.sh` already reuses
completed binaries and resumes partial downloads (`curl -C -`), so
retries are cheap.
2. **Cross-run cache** (`.github/workflows/ci.yaml`): the `test-e2e` job
now persists `/tmp/coder-e2e-cache` with `actions/cache`, so most runs
skip the GitHub download entirely. The key is derived from the spec
files that pin the downloaded versions, so it invalidates when those
versions change. Saves are restricted to `main` (`restore` runs
everywhere), matching the existing cache-poisoning convention used for
the Vale and golangci-lint caches.

Before this change, neither retry, mirror, nor cross-run caching
protected this path; the only caching was within a single run.

## Testing

- `biome check e2e/helpers.ts` passes.
- `tsc --noEmit` introduces no new errors.
- CI `test-e2e` exercises the changed path.

<details>
<summary>Investigation notes</summary>

- The failure always originates in `downloadCoderVersion` ->
`install.sh` -> `fetch()` (`curl -#fL ...
https://github.com/coder/coder/releases/download/vX.Y.Z/...`).
- `curl` exit 22 = server returned an HTTP error (403 seen in logs);
exit 1 = other transient failure. GitHub also returned 503s across the
workflow in some occurrences.
- `/tmp/coder-e2e-cache` was not persisted by any `actions/cache` step
in `ci.yaml`, so every fresh job re-downloaded from GitHub and was
exposed to the flake.
- Retry addresses transient failures; the cache removes the dependency
on GitHub for most runs. Combined, they target the root cause at two
layers.

</details>

---

This PR was generated by Coder Agents on behalf of @aqandrew.
This commit is contained in:
Andrew Aquino
2026-07-27 09:40:43 -07:00
committed by GitHub
parent ce4ee923c2
commit 5699f1cdfb
2 changed files with 75 additions and 30 deletions
+23
View File
@@ -984,6 +984,19 @@ jobs:
- run: pnpm playwright:install
working-directory: site
# Cache the Coder release binaries downloaded by the outdatedCLI /
# outdatedAgent e2e tests so most runs skip the flaky GitHub release
# download entirely. The cache key is keyed off the test files that pin
# the downloaded versions, so it invalidates when those versions change.
- name: Restore e2e Coder release binary cache
id: coder-e2e-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/coder-e2e-cache
key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/tests/outdatedCLI.spec.ts', 'site/e2e/tests/outdatedAgent.spec.ts') }}
restore-keys: |
coder-e2e-cache-${{ runner.os }}-
# Run tests that don't require a premium license without a premium license
- run: pnpm playwright:test --forbid-only --workers 1
if: ${{ !matrix.variant.premium }}
@@ -1000,6 +1013,16 @@ jobs:
CODER_E2E_REQUIRE_PREMIUM_TESTS: "1"
working-directory: site
- name: Save e2e Coder release binary cache
# Only the default branch is trusted to write the cache, so PR runs
# cannot poison the cache that subsequent runs restore from. Skip when
# the cache already had an exact key hit (no new content).
if: always() && github.ref == 'refs/heads/main' && steps.coder-e2e-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/coder-e2e-cache
key: ${{ steps.coder-e2e-cache.outputs.cache-primary-key }}
- name: Upload Playwright failure artifacts
if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1