opensource/coder
3
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-09-21 20:51:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix: retry and cache e2e Coder release downloads to reduce test-e2e ssh flake (#27470)

Browse Source 
closes DEVEX-651

## Summary

Fixes coder/internal#218 (`flake: e2e-test / test ssh`).

Despite the title, the `ssh with client v2.8.0` / `ssh with agent
v2.12.1` cases (`site/e2e/tests/outdatedCLI.spec.ts`,
`outdatedAgent.spec.ts`) are not failing because of a bug in SSH. They
fail during **setup**, in `downloadCoderVersion()`, which runs
`install.sh` to fetch an old Coder release from GitHub. Transient GitHub
errors (HTTP 403/503, surfacing as nonzero `curl` exit codes such as 22
or 1) make `install.sh` fail and take the whole ssh test down with it.

This is an external-download flake, confirmed by the recurring
`install.sh failed with code {22,1}` evidence in the issue thread and
Ethan's note ("Networking issues again").

## Changes

1. **Retry-with-backoff** (`site/e2e/helpers.ts`):
`downloadCoderVersion()` now retries `install.sh` up to 5 times with
exponential backoff and jitter (~1s, 2s, 4s, 8s). A single transient
download failure no longer fails the test. `install.sh` already reuses
completed binaries and resumes partial downloads (`curl -C -`), so
retries are cheap.
2. **Cross-run cache** (`.github/workflows/ci.yaml`): the `test-e2e` job
now persists `/tmp/coder-e2e-cache` with `actions/cache`, so most runs
skip the GitHub download entirely. The key is derived from the spec
files that pin the downloaded versions, so it invalidates when those
versions change. Saves are restricted to `main` (`restore` runs
everywhere), matching the existing cache-poisoning convention used for
the Vale and golangci-lint caches.

Before this change, neither retry, mirror, nor cross-run caching
protected this path; the only caching was within a single run.

## Testing

- `biome check e2e/helpers.ts` passes.
- `tsc --noEmit` introduces no new errors.
- CI `test-e2e` exercises the changed path.

<details>
<summary>Investigation notes</summary>

- The failure always originates in `downloadCoderVersion` ->
`install.sh` -> `fetch()` (`curl -#fL ...
https://github.com/coder/coder/releases/download/vX.Y.Z/...`).
- `curl` exit 22 = server returned an HTTP error (403 seen in logs);
exit 1 = other transient failure. GitHub also returned 503s across the
workflow in some occurrences.
- `/tmp/coder-e2e-cache` was not persisted by any `actions/cache` step
in `ci.yaml`, so every fresh job re-downloaded from GitHub and was
exposed to the flake.
- Retry addresses transient failures; the cache removes the dependency
on GitHub for most runs. Combined, they target the root cause at two
layers.

</details>

---

This PR was generated by Coder Agents on behalf of @aqandrew.
This commit is contained in:
Andrew Aquino
2026-07-27 09:40:43 -07:00
committed by GitHub
parent ce4ee923c2
commit 5699f1cdfb
2 changed files with 75 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yaml
+23
View File
@@ -984,6 +984,19 @@ jobs:
- run: pnpm playwright:install
working-directory: site
# Cache the Coder release binaries downloaded by the outdatedCLI /
# outdatedAgent e2e tests so most runs skip the flaky GitHub release
# download entirely. The cache key is keyed off the test files that pin
# the downloaded versions, so it invalidates when those versions change.
- name: Restore e2e Coder release binary cache
id: coder-e2e-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/coder-e2e-cache
key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/tests/outdatedCLI.spec.ts', 'site/e2e/tests/outdatedAgent.spec.ts') }}
restore-keys: |
coder-e2e-cache-${{ runner.os }}-
# Run tests that don't require a premium license without a premium license
- run: pnpm playwright:test --forbid-only --workers 1
if: ${{ !matrix.variant.premium }}
@@ -1000,6 +1013,16 @@ jobs:
CODER_E2E_REQUIRE_PREMIUM_TESTS: "1"
working-directory: site
- name: Save e2e Coder release binary cache
# Only the default branch is trusted to write the cache, so PR runs
# cannot poison the cache that subsequent runs restore from. Skip when
# the cache already had an exact key hit (no new content).
if: always() && github.ref == 'refs/heads/main' && steps.coder-e2e-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/coder-e2e-cache
key: ${{ steps.coder-e2e-cache.outputs.cache-primary-key }}
- name: Upload Playwright failure artifacts
if: failure() && github.actor != 'dependabot[bot]' && runner.os == 'Linux' && !github.event.pull_request.head.repo.fork
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
site/e2e/helpers.ts
+52 -30
View File
@@ -487,38 +487,60 @@ export const downloadCoderVersion = async (
return binaryPath;
}
// Run our official install script to install the binary
await new Promise<void>((resolve, reject) => {
const cp = spawn(
path.join(__dirname, "../../install.sh"),
[
"--version",
versionNumber,
"--method",
"standalone",
"--prefix",
tempDir,
"--binary-name",
binaryName,
],
{
env: {
...process.env,
XDG_CACHE_HOME: "/tmp/coder-e2e-cache",
TRACE: "1", // tells install.sh to `set -x`, helpful if something goes wrong
// runInstallScript runs our official install script to install the binary,
// resolving with the script's exit code.
const runInstallScript = (): Promise<number> =>
new Promise<number>((resolve, reject) => {
const cp = spawn(
path.join(__dirname, "../../install.sh"),
[
"--version",
versionNumber,
"--method",
"standalone",
"--prefix",
tempDir,
"--binary-name",
binaryName,
],
{
env: {
...process.env,
XDG_CACHE_HOME: "/tmp/coder-e2e-cache",
TRACE: "1", // tells install.sh to `set -x`, helpful if something goes wrong
},
},
},
);
cp.stderr.on("data", (data) => console.error(data.toString()));
cp.stdout.on("data", (data) => console.info(data.toString()));
cp.on("close", (code) => {
if (code === 0) {
resolve();
} else {
reject(new Error(`install.sh failed with code ${code}`));
}
);
cp.stderr.on("data", (data) => console.error(data.toString()));
cp.stdout.on("data", (data) => console.info(data.toString()));
cp.on("error", (err) => reject(err));
cp.on("close", (code) => resolve(code ?? 1));
});
});
// The install script downloads the release asset from GitHub, which
// occasionally returns a transient error (e.g. HTTP 403/503, surfacing as a
// nonzero curl exit code). Retry with exponential backoff so a single hiccup
// does not fail the test. Partial downloads are resumed and completed
// binaries are reused across attempts by install.sh.
const maxAttempts = 5;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const code = await runInstallScript();
if (code === 0) {
return binaryPath;
}
if (attempt === maxAttempts) {
throw new Error(
`install.sh failed with code ${code} after ${maxAttempts} attempts`,
);
}
// Exponential backoff with jitter: ~1s, 2s, 4s, 8s between attempts.
const backoffMs =
2 ** (attempt - 1) * 1000 + Math.floor(Math.random() * 1000);
console.error(
`install.sh attempt ${attempt}/${maxAttempts} failed with code ${code}; retrying in ${backoffMs}ms`,
);
await new Promise((resolve) => setTimeout(resolve, backoffMs));
}
return binaryPath;
};