## Summary
`e2e/tests/outdatedCLI.spec.ts` has a 30 second budget in which it must
create a template and workspace, start an agent, download an 84 MiB
release binary from GitHub, and then exercise the actual thing under
test: whether a `v2.8.0` client can still SSH into a workspace served by
HEAD. In the run that filed this ticket, `install.sh` spent **20.04
seconds** of that budget on an HTTP request the test does not need,
leaving 5.4 seconds for the download. The SSH flow never executed.
Worth being precise about the shape, because it changes the fix. The
stall is not in the code under test and it is not an SSH problem.
`install.sh` resolves the latest stable release version
**unconditionally**, even when `--version 2.8.0` is passed explicitly,
and on the pinned path that value feeds nothing but a cosmetic
post-install advisory string. Two thirds of the test's budget went to
producing one sentence of console output that the test discards.
Refs: https://github.com/coder/internal/issues/1571
## Problem
### What the test is for
This is a backward-compatibility test, and `v2.8.0` is the compatibility
floor it enforces rather than a "supported version" in the
release-channel sense. The pin traces to one code comment, `we no longer
support versions prior to Tailnet v2 API support`, citing 059e533544;
that commit first shipped in v2.8.0, so the pin sits exactly on the
boundary it names. Worth stating plainly: this is the oldest client
expected to still interoperate, not a version that receives patches.
Release support is mainline / stable / n-2 / ESR, all far newer.
The test's value is that it runs the *real* historical binary, compiled
in Feb 2024, against a current server: `codersdk` REST compatibility,
tailnet coordination v2, DERP negotiation, `coder ssh --stdio` as an SSH
transport, and the agent accepting a session. Nobody gets to assert what
that old client sends over the wire, which is exactly why the binary has
to be downloaded rather than faked.
The structural defect is that the download shares a timeout with the
assertion:
```text
┌─────────────────────────────────────────────────────────────────┐
│ ONE 30-second Playwright test budget │
├──────────────────────────────┬──────────────────────────────────┤
│ What we want to measure │ Incidental setup │
│ (deterministic, local) │ (network, non-deterministic) │
│ │ │
│ • template + workspace │ • HTTP HEAD to github.com │
│ • agent connect │ • 84 MiB download from │
│ • coder ssh --stdio │ GitHub release CDN │
│ • SSH handshake + exec │ • tar extraction │
│ • workspace stop │ │
└──────────────────────────────┴──────────────────────────────────┘
~7-14 s, stable 0 s (cached) .. ∞ (unbounded)
```
A test that asserts protocol compatibility should not be able to fail
because `github.com` was slow.
### The evidence
CI runs Playwright with `DEBUG: pw:api`, and `downloadCoderVersion`
passes `TRACE=1` to `install.sh`, which makes it `set -x`. The job log
therefore stamps every phase. Reconstructed from [job
80049661296](https://github.com/coder/coder/actions/runs/27124540074/job/80049661296),
`t=` relative to test start:
```text
t=+0.000 08:17:44.671 browserContext.newPage <- test starts
t=+0.882 08:17:45.553 login complete
t=+4.228 08:17:48.899 workspace create submitted
t=+4.519 08:17:49.190 agent-status-ready visible <- startAgent returns
t=+4.526 08:17:49.197 install.sh: parse_arg --version 2.8.0 ... <- downloadCoderVersion
t=+4.531 08:17:49.202 curl -sSLI https://github.com/coder/coder/releases/latest
:
: 20.042 SECONDS OF NOTHING
: (agent logs keepalives; the page sits idle)
:
t=+24.573 08:18:09.244 response= 200 .../releases/tag/v2.33.6 <- probe returns
t=+24.575 08:18:09.246 STABLE_VERSION=2.33.6 <- feeds a log line
t=+24.582 08:18:09.253 curl -#fL -o .../coder_2.8.0_linux_amd64.tar.gz.incomplete
: 5.4 s of an 84 MiB download
t=+30.000 08:18:14.671 Playwright kills the test
```
Three observations rule out the originally suspected cause (slow SSH
readiness or general runner slowness):
- **The SSH flow never started.** `sshIntoWorkspace` is called after
`downloadCoderVersion` returns, and it never returned. There is no
`coder ssh --stdio` process in the log.
- **The agent was healthy.** `agent-status-ready` resolved in 88 ms, and
through the entire 20 second stall the agent logs a live DERP
connection, successful STUN, and a completed wireguard handshake.
- **The runner was fast, not slow.** Login plus template plus workspace
plus agent took 4.5 seconds.
### Where the 20 seconds goes
```text
install.sh main()
...
L431 STABLE_VERSION=$(echo_latest_stable_version) <- ALWAYS runs
|
+-- echo_latest_stable_version() (install.sh:94)
curl -sSLI https://github.com/coder/coder/releases/latest
# no --connect-timeout
# no --max-time
# non-200 => exit 1 (hard failure)
L454-461 the only consumers when --version is pinned:
if VERSION == STABLE_VERSION: STABLE=1
L148 advisory="To install our stable release (v${STABLE_VERSION}), ..."
L159 "Coder ${channel}release v${VERSION} installed. ${advisory}"
```
That is the whole dependency chain. `-sSLI` also follows redirects and
`/releases/latest` *is* a redirect, so this is at minimum two
round-trips to `github.com` with no timeout ceiling on either.
### Why 30 seconds and not 60
`test.setTimeout(60_000)` used to be on this test. #16236 removed it,
and that removal was deliberate: it was itself a flake fix
(coder/internal#204, #279) whose thesis was that `go run` compiling
inside a resource-constrained test run was the problem. Having pre-built
the binary, it consistently stripped the allowances that existed to
absorb compile time:
| File | Change in #16236 | Was that allowance really compile time? |
|---|---|---|
| `app.spec.ts` | `setTimeout(75_000)` removed, click timeout `60_000`
-> `10_000` | Yes |
| `webTerminal.spec.ts` | `setTimeout(75_000)` removed | Yes |
| `helpers.ts` | agent-ready wait `45_000` -> `15_000` | Yes |
| `outdatedCLI.spec.ts` | `setTimeout(60_000)` removed | **No: also an
84 MiB download** |
| `outdatedAgent.spec.ts` | timeout untouched, 60 s survives | n/a |
The reasoning was sound and the sweep internally consistent. It had one
blind spot: for `app.spec.ts` and `webTerminal.spec.ts` that budget
genuinely was the compiler's, but here it covered compile time **plus**
a release download, and only the compile half went away. With 60
seconds, the failing run above would have finished in roughly 31 to 43
seconds and passed.
### Budget arithmetic
At `t=+24.58` the test still had to do:
| Remaining work | Realistic cost |
|---|---:|
| Download 84 MiB tarball | 2 - 8 s |
| `tar` extract | 0.3 - 1 s |
| `coder ssh --stdio` cold start | 0.5 - 2 s |
| Tailnet dial + SSH handshake | 1 - 3 s |
| `stopWorkspace` | 2 - 4 s |
| **Needed** | **~6 - 18 s** |
| **Available** | **5.42 s** |
## Fix
Move the download into the existing `testsSetup` Playwright project,
where it gets a 300 second budget and where a failure is attributed to
the download rather than to SSH.
```mermaid
flowchart TB
subgraph BEFORE["BEFORE: one budget, two concerns"]
direction TB
T1["tests project, timeout 30s"]
T1A["outdatedCLI.spec.ts<br/>login / template / workspace / agent<br/><b>downloadCoderVersion <- NETWORK</b><br/>sshIntoWorkspace / exec / stopWorkspace"]
T1 --> T1A
end
subgraph AFTER["AFTER: network work has its own clock"]
direction TB
S2["testsSetup project, timeout 300s"]
S2A["downloadCoderVersions.spec.ts<br/>stable-version probe + 84 MiB + retries<br/>all live HERE"]
T2["tests project, timeout 60s"]
T2A["outdatedCLI.spec.ts<br/>downloadCoderVersion = cache hit, ~300ms<br/>SSH path gets the whole budget"]
S2 --> S2A
S2A -- "dependencies" --> T2
T2 --> T2A
end
BEFORE ~~~ AFTER
style T1A fill:#ffe5e5,stroke:#cc0000,stroke-width:2px
style S2A fill:#e5ffe5,stroke:#007700,stroke-width:2px
style T2A fill:#e5ffe5,stroke:#007700,stroke-width:2px
```
### Why it works
`downloadCoderVersion` was already idempotent and cache-checking: it
spawns `<binaryPath> version` first and returns early on exit 0. So the
test keeps its existing call and that call simply becomes a no-op
costing a few hundred milliseconds. **No test logic changes.**
```mermaid
sequenceDiagram
autonumber
participant S as testsSetup:<br/>downloadCoderVersions
participant IS as install.sh
participant GH as github.com
participant T as tests:<br/>outdatedCLI
participant CD as coderd + agent
Note over S: budget 300s
S->>IS: downloadCoderVersion(v2.8.0)
IS->>GH: stable-version probe (unbounded)
IS->>GH: fetch 84 MiB asset
GH-->>IS: /tmp/coder-e2e-cache/bin/coder-e2e-2.8.0
IS-->>S: binaryPath
Note over T: budget 60s, local only
T->>T: downloadCoderVersion(v2.8.0)
Note right of T: spawn "<bin> version" -> exit 0<br/>returns early, ~300ms, no network
T->>CD: coder ssh --stdio, handshake, exec "exit 0"
CD-->>T: exit code 0
```
### Why the prefetch is non-fatal
The obvious implementation raises on failure. That would be wrong here,
and I verified why rather than assuming: `tests` declares `dependencies:
["testsSetup"]`, and a failing setup project stops dependent tests from
**running at all**. Adding a deliberately-throwing setup spec produced:
```text
✓ 1 [testsSetup] › addUsersAndLicense.spec.ts › setup deployment (11.7s)
✓ 2 [testsSetup] › downloadCoderVersions.spec.ts › download outdated CLI (353ms)
✘ 3 [testsSetup] › zzTempFail.spec.ts › temporary blast radius probe (0ms)
1 failed
1 did not run <- outdatedCLI never ran
2 passed
```
So raising would convert a one-test flake into a whole-suite outage on
any GitHub hiccup. Instead the prefetch logs a warning and returns, and
the test's own `downloadCoderVersion` call fetches inline as it does
today. The failure path is therefore no worse than the status quo, and
the success path removes the network from the test entirely.
Of the three policies available (fail hard, fall back inline, or skip
the test), this is the only one that cannot regress anything: it never
blocks the suite, and it never silently drops coverage the way an
auto-skip would.
### Restoring the 60 second budget
This is the second half of the change, and it exists for the fallback
path above. It cannot reintroduce what #16236 fixed: the timeout value
has no causal relationship to how the binary is produced, `coderBinary`
stays pre-built, `go run` stays gone, and only `outdatedCLI.spec.ts` is
touched.
It does give back a bounded sliver of the CI-latency goal, and the bound
is small. A passing run is unaffected. The cost lands only when this one
test hangs, and then it is +30 s once: `--workers 1` so there is no
fan-out, `CODER_E2E_TEST_RETRIES` is unset in CI so `retries` is 0 and
nothing multiplies it, and the job budget is `timeout-minutes: 20`.
## Measurements
Four scenarios, locally on darwin/arm64 against a freshly built
`site/e2e/bin/coder`:
| Scenario | setup spec | `outdatedCLI` | `install.sh` inside the test?
| Result |
|---|---:|---:|---|---|
| Cold, empty cache | 5.7 s | 10.0 s | **no**, ran in setup | ✓ passed |
| Warm cache | 340 ms | 11.9 s | **no**, 0 invocations | ✓ passed |
| Prefetch fails, cache empty | 1 ms | 15.4 s | yes, inline fallback | ✓
passed |
| Setup spec throws | n/a | did not run | n/a | blast radius above |
The cold run is the load-bearing one: `install.sh` is invoked from the
setup spec and the test runs local-only in 10.0 s, so the 84 MiB
download and the 20 s probe are no longer on the assertion's clock.
For context on what "local only" costs, eight consecutive `main` runs
where the CI cache already made `install.sh` a no-op:
| Job | duration |
|---|---:|
| 90382498172 | 11.7 s |
| 90352398170 | 12.1 s |
| 90335547858 | 8.1 s |
| 90317232684 | 13.7 s |
| 90297156949 | 8.4 s |
| 90280065546 | 6.9 s |
| 90265047082 | 6.9 s |
| 90250803989 | 6.6 s |
6.6 to 13.7 seconds. This change makes that the only path rather than
the lucky one.
Also checked: the test name is byte identical (`ssh with client v2.8.0`)
so flake tracking keeps matching it, `outdatedAgent` remains skipped,
and `webTerminal`, `auditLogs`, and `updateTemplate` still pass, so the
added setup dependency disturbs nothing. The full 60-test suite was not
run locally because the premium tests need `CODER_E2E_LICENSE`.
## Also in this change
The pinned versions move to `site/e2e/constants.ts` as
`oldestSupportedCLIVersion` and `oldestSupportedAgentVersion`, so the
setup spec and the tests share one source of truth, and the comments
explaining *why* those particular versions travel with them. The CI
cache key follows them there: it previously hashed the two spec files,
and now hashes `constants.ts`, so it still invalidates exactly when a
pinned version changes.
## Not addressed here
The 20 second probe is relocated, not removed. `install.sh` still
resolves the latest stable version on every pinned install, with no
`--connect-timeout` or `--max-time`, and still treats a non-200 as
fatal, so a GitHub hiccup can fail an install whose target tarball is
already cached locally. That is a user-facing bug in its own right and
wants its own PR, since fixing it means deciding what a pinned install
should print when we no longer look up what "stable" currently is.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Provisioner steps broken into smaller granular actions.
Changes:
- `ExtractArchive` moved to `init` request (was in `configure`)
- Writing `tfstate` moved to `plan` (was in `configure`)
- Moved most plan/apply outputs to `GraphComplete`
Using `go run` inside of a test is fragile, because it means we have to
wait for `go` to compile the binary while also constrained on resources
by the fact that Playwright and coderd are already running. We should
instead compile a coder binary for the current platform before the tests
and use it directly.
Closescoder/internal#168
Gets rid of the "global state" authentication, and adds a `login` helper
which should be called at the beginning of each test. This means that
not every test needs to authenticated as admin, and we can even have
tests that encompass multiple permission levels.
We also now create more than just the single admin user during setup, so
that we can have a set of users to pick from as appropriate.
Fixes#8218
Removes `wsconncache` and related "is legacy?" functions and API calls that were used by it.
The only leftover is that Agents still use the legacy IP, so that back level clients or workspace proxies can dial them correctly.
We should eventually remove this: #11819
