From 4e512f786f2a58cc088f7ca4aff5228353d961a4 Mon Sep 17 00:00:00 2001 From: Bobby Ho Date: Wed, 29 Jul 2026 16:26:13 -0700 Subject: [PATCH] fix: prefetch outdated Coder CLI in e2e setup instead of in the test (#27629) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary `e2e/tests/outdatedCLI.spec.ts` has a 30 second budget in which it must create a template and workspace, start an agent, download an 84 MiB release binary from GitHub, and then exercise the actual thing under test: whether a `v2.8.0` client can still SSH into a workspace served by HEAD. In the run that filed this ticket, `install.sh` spent **20.04 seconds** of that budget on an HTTP request the test does not need, leaving 5.4 seconds for the download. The SSH flow never executed. Worth being precise about the shape, because it changes the fix. The stall is not in the code under test and it is not an SSH problem. `install.sh` resolves the latest stable release version **unconditionally**, even when `--version 2.8.0` is passed explicitly, and on the pinned path that value feeds nothing but a cosmetic post-install advisory string. Two thirds of the test's budget went to producing one sentence of console output that the test discards. Refs: https://github.com/coder/internal/issues/1571 ## Problem ### What the test is for This is a backward-compatibility test, and `v2.8.0` is the compatibility floor it enforces rather than a "supported version" in the release-channel sense. The pin traces to one code comment, `we no longer support versions prior to Tailnet v2 API support`, citing 059e533544; that commit first shipped in v2.8.0, so the pin sits exactly on the boundary it names. Worth stating plainly: this is the oldest client expected to still interoperate, not a version that receives patches. Release support is mainline / stable / n-2 / ESR, all far newer. The test's value is that it runs the *real* historical binary, compiled in Feb 2024, against a current server: `codersdk` REST compatibility, tailnet coordination v2, DERP negotiation, `coder ssh --stdio` as an SSH transport, and the agent accepting a session. Nobody gets to assert what that old client sends over the wire, which is exactly why the binary has to be downloaded rather than faked. The structural defect is that the download shares a timeout with the assertion: ```text ┌─────────────────────────────────────────────────────────────────┐ │ ONE 30-second Playwright test budget │ ├──────────────────────────────┬──────────────────────────────────┤ │ What we want to measure │ Incidental setup │ │ (deterministic, local) │ (network, non-deterministic) │ │ │ │ │ • template + workspace │ • HTTP HEAD to github.com │ │ • agent connect │ • 84 MiB download from │ │ • coder ssh --stdio │ GitHub release CDN │ │ • SSH handshake + exec │ • tar extraction │ │ • workspace stop │ │ └──────────────────────────────┴──────────────────────────────────┘ ~7-14 s, stable 0 s (cached) .. ∞ (unbounded) ``` A test that asserts protocol compatibility should not be able to fail because `github.com` was slow. ### The evidence CI runs Playwright with `DEBUG: pw:api`, and `downloadCoderVersion` passes `TRACE=1` to `install.sh`, which makes it `set -x`. The job log therefore stamps every phase. Reconstructed from [job 80049661296](https://github.com/coder/coder/actions/runs/27124540074/job/80049661296), `t=` relative to test start: ```text t=+0.000 08:17:44.671 browserContext.newPage <- test starts t=+0.882 08:17:45.553 login complete t=+4.228 08:17:48.899 workspace create submitted t=+4.519 08:17:49.190 agent-status-ready visible <- startAgent returns t=+4.526 08:17:49.197 install.sh: parse_arg --version 2.8.0 ... <- downloadCoderVersion t=+4.531 08:17:49.202 curl -sSLI https://github.com/coder/coder/releases/latest : : 20.042 SECONDS OF NOTHING : (agent logs keepalives; the page sits idle) : t=+24.573 08:18:09.244 response= 200 .../releases/tag/v2.33.6 <- probe returns t=+24.575 08:18:09.246 STABLE_VERSION=2.33.6 <- feeds a log line t=+24.582 08:18:09.253 curl -#fL -o .../coder_2.8.0_linux_amd64.tar.gz.incomplete : 5.4 s of an 84 MiB download t=+30.000 08:18:14.671 Playwright kills the test ``` Three observations rule out the originally suspected cause (slow SSH readiness or general runner slowness): - **The SSH flow never started.** `sshIntoWorkspace` is called after `downloadCoderVersion` returns, and it never returned. There is no `coder ssh --stdio` process in the log. - **The agent was healthy.** `agent-status-ready` resolved in 88 ms, and through the entire 20 second stall the agent logs a live DERP connection, successful STUN, and a completed wireguard handshake. - **The runner was fast, not slow.** Login plus template plus workspace plus agent took 4.5 seconds. ### Where the 20 seconds goes ```text install.sh main() ... L431 STABLE_VERSION=$(echo_latest_stable_version) <- ALWAYS runs | +-- echo_latest_stable_version() (install.sh:94) curl -sSLI https://github.com/coder/coder/releases/latest # no --connect-timeout # no --max-time # non-200 => exit 1 (hard failure) L454-461 the only consumers when --version is pinned: if VERSION == STABLE_VERSION: STABLE=1 L148 advisory="To install our stable release (v${STABLE_VERSION}), ..." L159 "Coder ${channel}release v${VERSION} installed. ${advisory}" ``` That is the whole dependency chain. `-sSLI` also follows redirects and `/releases/latest` *is* a redirect, so this is at minimum two round-trips to `github.com` with no timeout ceiling on either. ### Why 30 seconds and not 60 `test.setTimeout(60_000)` used to be on this test. #16236 removed it, and that removal was deliberate: it was itself a flake fix (coder/internal#204, #279) whose thesis was that `go run` compiling inside a resource-constrained test run was the problem. Having pre-built the binary, it consistently stripped the allowances that existed to absorb compile time: | File | Change in #16236 | Was that allowance really compile time? | |---|---|---| | `app.spec.ts` | `setTimeout(75_000)` removed, click timeout `60_000` -> `10_000` | Yes | | `webTerminal.spec.ts` | `setTimeout(75_000)` removed | Yes | | `helpers.ts` | agent-ready wait `45_000` -> `15_000` | Yes | | `outdatedCLI.spec.ts` | `setTimeout(60_000)` removed | **No: also an 84 MiB download** | | `outdatedAgent.spec.ts` | timeout untouched, 60 s survives | n/a | The reasoning was sound and the sweep internally consistent. It had one blind spot: for `app.spec.ts` and `webTerminal.spec.ts` that budget genuinely was the compiler's, but here it covered compile time **plus** a release download, and only the compile half went away. With 60 seconds, the failing run above would have finished in roughly 31 to 43 seconds and passed. ### Budget arithmetic At `t=+24.58` the test still had to do: | Remaining work | Realistic cost | |---|---:| | Download 84 MiB tarball | 2 - 8 s | | `tar` extract | 0.3 - 1 s | | `coder ssh --stdio` cold start | 0.5 - 2 s | | Tailnet dial + SSH handshake | 1 - 3 s | | `stopWorkspace` | 2 - 4 s | | **Needed** | **~6 - 18 s** | | **Available** | **5.42 s** | ## Fix Move the download into the existing `testsSetup` Playwright project, where it gets a 300 second budget and where a failure is attributed to the download rather than to SSH. ```mermaid flowchart TB subgraph BEFORE["BEFORE: one budget, two concerns"] direction TB T1["tests project, timeout 30s"] T1A["outdatedCLI.spec.ts
login / template / workspace / agent
downloadCoderVersion <- NETWORK
sshIntoWorkspace / exec / stopWorkspace"] T1 --> T1A end subgraph AFTER["AFTER: network work has its own clock"] direction TB S2["testsSetup project, timeout 300s"] S2A["downloadCoderVersions.spec.ts
stable-version probe + 84 MiB + retries
all live HERE"] T2["tests project, timeout 60s"] T2A["outdatedCLI.spec.ts
downloadCoderVersion = cache hit, ~300ms
SSH path gets the whole budget"] S2 --> S2A S2A -- "dependencies" --> T2 T2 --> T2A end BEFORE ~~~ AFTER style T1A fill:#ffe5e5,stroke:#cc0000,stroke-width:2px style S2A fill:#e5ffe5,stroke:#007700,stroke-width:2px style T2A fill:#e5ffe5,stroke:#007700,stroke-width:2px ``` ### Why it works `downloadCoderVersion` was already idempotent and cache-checking: it spawns ` version` first and returns early on exit 0. So the test keeps its existing call and that call simply becomes a no-op costing a few hundred milliseconds. **No test logic changes.** ```mermaid sequenceDiagram autonumber participant S as testsSetup:
downloadCoderVersions participant IS as install.sh participant GH as github.com participant T as tests:
outdatedCLI participant CD as coderd + agent Note over S: budget 300s S->>IS: downloadCoderVersion(v2.8.0) IS->>GH: stable-version probe (unbounded) IS->>GH: fetch 84 MiB asset GH-->>IS: /tmp/coder-e2e-cache/bin/coder-e2e-2.8.0 IS-->>S: binaryPath Note over T: budget 60s, local only T->>T: downloadCoderVersion(v2.8.0) Note right of T: spawn " version" -> exit 0
returns early, ~300ms, no network T->>CD: coder ssh --stdio, handshake, exec "exit 0" CD-->>T: exit code 0 ``` ### Why the prefetch is non-fatal The obvious implementation raises on failure. That would be wrong here, and I verified why rather than assuming: `tests` declares `dependencies: ["testsSetup"]`, and a failing setup project stops dependent tests from **running at all**. Adding a deliberately-throwing setup spec produced: ```text ✓ 1 [testsSetup] › addUsersAndLicense.spec.ts › setup deployment (11.7s) ✓ 2 [testsSetup] › downloadCoderVersions.spec.ts › download outdated CLI (353ms) ✘ 3 [testsSetup] › zzTempFail.spec.ts › temporary blast radius probe (0ms) 1 failed 1 did not run <- outdatedCLI never ran 2 passed ``` So raising would convert a one-test flake into a whole-suite outage on any GitHub hiccup. Instead the prefetch logs a warning and returns, and the test's own `downloadCoderVersion` call fetches inline as it does today. The failure path is therefore no worse than the status quo, and the success path removes the network from the test entirely. Of the three policies available (fail hard, fall back inline, or skip the test), this is the only one that cannot regress anything: it never blocks the suite, and it never silently drops coverage the way an auto-skip would. ### Restoring the 60 second budget This is the second half of the change, and it exists for the fallback path above. It cannot reintroduce what #16236 fixed: the timeout value has no causal relationship to how the binary is produced, `coderBinary` stays pre-built, `go run` stays gone, and only `outdatedCLI.spec.ts` is touched. It does give back a bounded sliver of the CI-latency goal, and the bound is small. A passing run is unaffected. The cost lands only when this one test hangs, and then it is +30 s once: `--workers 1` so there is no fan-out, `CODER_E2E_TEST_RETRIES` is unset in CI so `retries` is 0 and nothing multiplies it, and the job budget is `timeout-minutes: 20`. ## Measurements Four scenarios, locally on darwin/arm64 against a freshly built `site/e2e/bin/coder`: | Scenario | setup spec | `outdatedCLI` | `install.sh` inside the test? | Result | |---|---:|---:|---|---| | Cold, empty cache | 5.7 s | 10.0 s | **no**, ran in setup | ✓ passed | | Warm cache | 340 ms | 11.9 s | **no**, 0 invocations | ✓ passed | | Prefetch fails, cache empty | 1 ms | 15.4 s | yes, inline fallback | ✓ passed | | Setup spec throws | n/a | did not run | n/a | blast radius above | The cold run is the load-bearing one: `install.sh` is invoked from the setup spec and the test runs local-only in 10.0 s, so the 84 MiB download and the 20 s probe are no longer on the assertion's clock. For context on what "local only" costs, eight consecutive `main` runs where the CI cache already made `install.sh` a no-op: | Job | duration | |---|---:| | 90382498172 | 11.7 s | | 90352398170 | 12.1 s | | 90335547858 | 8.1 s | | 90317232684 | 13.7 s | | 90297156949 | 8.4 s | | 90280065546 | 6.9 s | | 90265047082 | 6.9 s | | 90250803989 | 6.6 s | 6.6 to 13.7 seconds. This change makes that the only path rather than the lucky one. Also checked: the test name is byte identical (`ssh with client v2.8.0`) so flake tracking keeps matching it, `outdatedAgent` remains skipped, and `webTerminal`, `auditLogs`, and `updateTemplate` still pass, so the added setup dependency disturbs nothing. The full 60-test suite was not run locally because the premium tests need `CODER_E2E_LICENSE`. ## Also in this change The pinned versions move to `site/e2e/constants.ts` as `oldestSupportedCLIVersion` and `oldestSupportedAgentVersion`, so the setup spec and the tests share one source of truth, and the comments explaining *why* those particular versions travel with them. The CI cache key follows them there: it previously hashed the two spec files, and now hashes `constants.ts`, so it still invalidates exactly when a pinned version changes. ## Not addressed here The 20 second probe is relocated, not removed. `install.sh` still resolves the latest stable version on every pinned install, with no `--connect-timeout` or `--max-time`, and still treats a non-200 as fatal, so a GitHub hiccup can fail an install whose target tarball is already cached locally. That is a user-facing bug in its own right and wants its own PR, since fixing it means deciding what a pinned install should print when we no longer look up what "stable" currently is. Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/ci.yaml | 6 ++--- site/e2e/constants.ts | 10 ++++++++ site/e2e/setup/downloadCoderVersions.spec.ts | 27 ++++++++++++++++++++ site/e2e/tests/outdatedAgent.spec.ts | 8 +++--- site/e2e/tests/outdatedCLI.spec.ts | 13 ++++++---- 5 files changed, 51 insertions(+), 13 deletions(-) create mode 100644 site/e2e/setup/downloadCoderVersions.spec.ts diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 661564a42a..757f3276ad 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -987,14 +987,14 @@ jobs: # Cache the Coder release binaries downloaded by the outdatedCLI / # outdatedAgent e2e tests so most runs skip the flaky GitHub release - # download entirely. The cache key is keyed off the test files that pin - # the downloaded versions, so it invalidates when those versions change. + # download entirely. The cache key is keyed off the file that pins the + # downloaded versions, so it invalidates when those versions change. - name: Restore e2e Coder release binary cache id: coder-e2e-cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: /tmp/coder-e2e-cache - key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/tests/outdatedCLI.spec.ts', 'site/e2e/tests/outdatedAgent.spec.ts') }} + key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/constants.ts') }} restore-keys: | coder-e2e-cache-${{ runner.os }}- diff --git a/site/e2e/constants.ts b/site/e2e/constants.ts index 4e95d642ea..946f10c022 100644 --- a/site/e2e/constants.ts +++ b/site/e2e/constants.ts @@ -2,6 +2,16 @@ import * as path from "node:path"; export const coderBinary = path.join(__dirname, "./bin/coder"); +// The oldest client and agent versions that Coder still supports. The +// compatibility tests download these release binaries and run them against the +// current server. Changing either value changes which release asset the e2e +// suite fetches, and invalidates the CI cache that stores them. +// +// we no longer support versions prior to Tailnet v2 API support: https://github.com/coder/coder/commit/059e533544a0268acbc8831006b2858ead2f0d8e +export const oldestSupportedCLIVersion = "v2.8.0"; +// we no longer support versions w/o DRPC +export const oldestSupportedAgentVersion = "v2.12.1"; + // Default port from the server export const coderPort = process.env.CODER_E2E_PORT ? Number(process.env.CODER_E2E_PORT) diff --git a/site/e2e/setup/downloadCoderVersions.spec.ts b/site/e2e/setup/downloadCoderVersions.spec.ts new file mode 100644 index 0000000000..fb81c9bccb --- /dev/null +++ b/site/e2e/setup/downloadCoderVersions.spec.ts @@ -0,0 +1,27 @@ +import { test } from "@playwright/test"; +import { oldestSupportedCLIVersion } from "../constants"; +import { downloadCoderVersion } from "../helpers"; + +// Fetching a release binary is network-bound and effectively unbounded: the +// install script downloads an 84 MiB asset and retries with backoff. Doing it +// here rather than inside the compatibility test keeps that time off the test's +// timeout, so a slow GitHub response can no longer surface as an SSH failure. +test("download outdated Coder CLI", async () => { + // Generous because this covers a cold download plus install.sh's retries. + test.setTimeout(300_000); + + // A failure here is deliberately not fatal. Tests in the `tests` project + // depend on this project, and a failing dependency stops all of them from + // running, so a GitHub outage would block the entire suite rather than the + // one test that needs this binary. outdatedCLI.spec.ts calls + // downloadCoderVersion itself, so it retries inline and fails alone. + try { + await downloadCoderVersion(oldestSupportedCLIVersion); + } catch (error) { + console.error( + `Failed to prefetch the Coder ${oldestSupportedCLIVersion} CLI. ` + + "outdatedCLI.spec.ts will download it inline and may time out.", + error, + ); + } +}); diff --git a/site/e2e/tests/outdatedAgent.spec.ts b/site/e2e/tests/outdatedAgent.spec.ts index 9992a5476e..43f3a4685e 100644 --- a/site/e2e/tests/outdatedAgent.spec.ts +++ b/site/e2e/tests/outdatedAgent.spec.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; import { test } from "@playwright/test"; +import { oldestSupportedAgentVersion } from "../constants"; import { createTemplate, createWorkspace, @@ -12,15 +13,12 @@ import { } from "../helpers"; import { beforeCoderTest } from "../hooks"; -// we no longer support versions w/o DRPC -const agentVersion = "v2.12.1"; - test.beforeEach(async ({ page }) => { beforeCoderTest(page); await login(page); }); -test.skip(`ssh with agent ${agentVersion}`, async ({ page }) => { +test.skip(`ssh with agent ${oldestSupportedAgentVersion}`, async ({ page }) => { test.setTimeout(60_000); const token = randomUUID(); @@ -43,7 +41,7 @@ test.skip(`ssh with agent ${agentVersion}`, async ({ page }) => { ], }); const workspaceName = await createWorkspace(page, template); - const binaryPath = await downloadCoderVersion(agentVersion); + const binaryPath = await downloadCoderVersion(oldestSupportedAgentVersion); const agent = await startAgentWithCommand(page, token, binaryPath); const client = await sshIntoWorkspace(page, workspaceName); diff --git a/site/e2e/tests/outdatedCLI.spec.ts b/site/e2e/tests/outdatedCLI.spec.ts index cad37bb05a..a7d07970e3 100644 --- a/site/e2e/tests/outdatedCLI.spec.ts +++ b/site/e2e/tests/outdatedCLI.spec.ts @@ -1,5 +1,6 @@ import { randomUUID } from "node:crypto"; import { test } from "@playwright/test"; +import { oldestSupportedCLIVersion } from "../constants"; import { createTemplate, createWorkspace, @@ -12,15 +13,17 @@ import { } from "../helpers"; import { beforeCoderTest } from "../hooks"; -// we no longer support versions prior to Tailnet v2 API support: https://github.com/coder/coder/commit/059e533544a0268acbc8831006b2858ead2f0d8e -const clientVersion = "v2.8.0"; - test.beforeEach(async ({ page }) => { beforeCoderTest(page); await login(page); }); -test(`ssh with client ${clientVersion}`, async ({ page }) => { +test(`ssh with client ${oldestSupportedCLIVersion}`, async ({ page }) => { + // setup/downloadCoderVersions.spec.ts normally has the binary cached by now, + // leaving this a local-only test. The extra headroom covers the case where + // that prefetch failed and downloadCoderVersion has to fetch it inline. + test.setTimeout(60_000); + const token = randomUUID(); const template = await createTemplate(page, { graph: [ @@ -42,7 +45,7 @@ test(`ssh with client ${clientVersion}`, async ({ page }) => { }); const workspaceName = await createWorkspace(page, template); const agent = await startAgent(page, token); - const binaryPath = await downloadCoderVersion(clientVersion); + const binaryPath = await downloadCoderVersion(oldestSupportedCLIVersion); const client = await sshIntoWorkspace(page, workspaceName, binaryPath); await new Promise((resolve, reject) => {