fix: prefetch outdated Coder CLI in e2e setup instead of in the test (#27629)

## Summary

`e2e/tests/outdatedCLI.spec.ts` has a 30 second budget in which it must
create a template and workspace, start an agent, download an 84 MiB
release binary from GitHub, and then exercise the actual thing under
test: whether a `v2.8.0` client can still SSH into a workspace served by
HEAD. In the run that filed this ticket, `install.sh` spent **20.04
seconds** of that budget on an HTTP request the test does not need,
leaving 5.4 seconds for the download. The SSH flow never executed.

Worth being precise about the shape, because it changes the fix. The
stall is not in the code under test and it is not an SSH problem.
`install.sh` resolves the latest stable release version
**unconditionally**, even when `--version 2.8.0` is passed explicitly,
and on the pinned path that value feeds nothing but a cosmetic
post-install advisory string. Two thirds of the test's budget went to
producing one sentence of console output that the test discards.

Refs: https://github.com/coder/internal/issues/1571

## Problem

### What the test is for

This is a backward-compatibility test, and `v2.8.0` is the compatibility
floor it enforces rather than a "supported version" in the
release-channel sense. The pin traces to one code comment, `we no longer
support versions prior to Tailnet v2 API support`, citing 059e533544;
that commit first shipped in v2.8.0, so the pin sits exactly on the
boundary it names. Worth stating plainly: this is the oldest client
expected to still interoperate, not a version that receives patches.
Release support is mainline / stable / n-2 / ESR, all far newer.

The test's value is that it runs the *real* historical binary, compiled
in Feb 2024, against a current server: `codersdk` REST compatibility,
tailnet coordination v2, DERP negotiation, `coder ssh --stdio` as an SSH
transport, and the agent accepting a session. Nobody gets to assert what
that old client sends over the wire, which is exactly why the binary has
to be downloaded rather than faked.

The structural defect is that the download shares a timeout with the
assertion:

```text
┌─────────────────────────────────────────────────────────────────┐
│  ONE 30-second Playwright test budget                           │
├──────────────────────────────┬──────────────────────────────────┤
│  What we want to measure     │  Incidental setup                │
│  (deterministic, local)      │  (network, non-deterministic)    │
│                              │                                  │
│  • template + workspace      │  • HTTP HEAD to github.com       │
│  • agent connect             │  • 84 MiB download from          │
│  • coder ssh --stdio         │    GitHub release CDN            │
│  • SSH handshake + exec      │  • tar extraction                │
│  • workspace stop            │                                  │
└──────────────────────────────┴──────────────────────────────────┘
      ~7-14 s, stable                   0 s (cached) .. ∞ (unbounded)
```

A test that asserts protocol compatibility should not be able to fail
because `github.com` was slow.

### The evidence

CI runs Playwright with `DEBUG: pw:api`, and `downloadCoderVersion`
passes `TRACE=1` to `install.sh`, which makes it `set -x`. The job log
therefore stamps every phase. Reconstructed from [job
80049661296](https://github.com/coder/coder/actions/runs/27124540074/job/80049661296),
`t=` relative to test start:

```text
t=+0.000  08:17:44.671  browserContext.newPage                      <- test starts
t=+0.882  08:17:45.553  login complete
t=+4.228  08:17:48.899  workspace create submitted
t=+4.519  08:17:49.190  agent-status-ready visible                  <- startAgent returns
t=+4.526  08:17:49.197  install.sh: parse_arg --version 2.8.0 ...   <- downloadCoderVersion
t=+4.531  08:17:49.202  curl -sSLI https://github.com/coder/coder/releases/latest
                        :
                        :   20.042 SECONDS OF NOTHING
                        :   (agent logs keepalives; the page sits idle)
                        :
t=+24.573 08:18:09.244  response= 200 .../releases/tag/v2.33.6      <- probe returns
t=+24.575 08:18:09.246  STABLE_VERSION=2.33.6                       <- feeds a log line
t=+24.582 08:18:09.253  curl -#fL -o .../coder_2.8.0_linux_amd64.tar.gz.incomplete
                        :   5.4 s of an 84 MiB download
t=+30.000 08:18:14.671  Playwright kills the test
```

Three observations rule out the originally suspected cause (slow SSH
readiness or general runner slowness):

- **The SSH flow never started.** `sshIntoWorkspace` is called after
`downloadCoderVersion` returns, and it never returned. There is no
`coder ssh --stdio` process in the log.
- **The agent was healthy.** `agent-status-ready` resolved in 88 ms, and
through the entire 20 second stall the agent logs a live DERP
connection, successful STUN, and a completed wireguard handshake.
- **The runner was fast, not slow.** Login plus template plus workspace
plus agent took 4.5 seconds.

### Where the 20 seconds goes

```text
install.sh main()
  ...
  L431   STABLE_VERSION=$(echo_latest_stable_version)   <- ALWAYS runs
                |
                +-- echo_latest_stable_version()  (install.sh:94)
                      curl -sSLI https://github.com/coder/coder/releases/latest
                      #  no --connect-timeout
                      #  no --max-time
                      #  non-200 => exit 1  (hard failure)

  L454-461  the only consumers when --version is pinned:
              if VERSION == STABLE_VERSION: STABLE=1

  L148      advisory="To install our stable release (v${STABLE_VERSION}), ..."
  L159      "Coder ${channel}release v${VERSION} installed. ${advisory}"
```

That is the whole dependency chain. `-sSLI` also follows redirects and
`/releases/latest` *is* a redirect, so this is at minimum two
round-trips to `github.com` with no timeout ceiling on either.

### Why 30 seconds and not 60

`test.setTimeout(60_000)` used to be on this test. #16236 removed it,
and that removal was deliberate: it was itself a flake fix
(coder/internal#204, #279) whose thesis was that `go run` compiling
inside a resource-constrained test run was the problem. Having pre-built
the binary, it consistently stripped the allowances that existed to
absorb compile time:

| File | Change in #16236 | Was that allowance really compile time? |
|---|---|---|
| `app.spec.ts` | `setTimeout(75_000)` removed, click timeout `60_000`
-> `10_000` | Yes |
| `webTerminal.spec.ts` | `setTimeout(75_000)` removed | Yes |
| `helpers.ts` | agent-ready wait `45_000` -> `15_000` | Yes |
| `outdatedCLI.spec.ts` | `setTimeout(60_000)` removed | **No: also an
84 MiB download** |
| `outdatedAgent.spec.ts` | timeout untouched, 60 s survives | n/a |

The reasoning was sound and the sweep internally consistent. It had one
blind spot: for `app.spec.ts` and `webTerminal.spec.ts` that budget
genuinely was the compiler's, but here it covered compile time **plus**
a release download, and only the compile half went away. With 60
seconds, the failing run above would have finished in roughly 31 to 43
seconds and passed.

### Budget arithmetic

At `t=+24.58` the test still had to do:

| Remaining work | Realistic cost |
|---|---:|
| Download 84 MiB tarball | 2 - 8 s |
| `tar` extract | 0.3 - 1 s |
| `coder ssh --stdio` cold start | 0.5 - 2 s |
| Tailnet dial + SSH handshake | 1 - 3 s |
| `stopWorkspace` | 2 - 4 s |
| **Needed** | **~6 - 18 s** |
| **Available** | **5.42 s** |

## Fix

Move the download into the existing `testsSetup` Playwright project,
where it gets a 300 second budget and where a failure is attributed to
the download rather than to SSH.

```mermaid
flowchart TB
    subgraph BEFORE["BEFORE: one budget, two concerns"]
        direction TB
        T1["tests project, timeout 30s"]
        T1A["outdatedCLI.spec.ts<br/>login / template / workspace / agent<br/><b>downloadCoderVersion &lt;- NETWORK</b><br/>sshIntoWorkspace / exec / stopWorkspace"]
        T1 --> T1A
    end

    subgraph AFTER["AFTER: network work has its own clock"]
        direction TB
        S2["testsSetup project, timeout 300s"]
        S2A["downloadCoderVersions.spec.ts<br/>stable-version probe + 84 MiB + retries<br/>all live HERE"]
        T2["tests project, timeout 60s"]
        T2A["outdatedCLI.spec.ts<br/>downloadCoderVersion = cache hit, ~300ms<br/>SSH path gets the whole budget"]
        S2 --> S2A
        S2A -- "dependencies" --> T2
        T2 --> T2A
    end

    BEFORE ~~~ AFTER

    style T1A fill:#ffe5e5,stroke:#cc0000,stroke-width:2px
    style S2A fill:#e5ffe5,stroke:#007700,stroke-width:2px
    style T2A fill:#e5ffe5,stroke:#007700,stroke-width:2px
```

### Why it works

`downloadCoderVersion` was already idempotent and cache-checking: it
spawns `<binaryPath> version` first and returns early on exit 0. So the
test keeps its existing call and that call simply becomes a no-op
costing a few hundred milliseconds. **No test logic changes.**

```mermaid
sequenceDiagram
    autonumber
    participant S as testsSetup:<br/>downloadCoderVersions
    participant IS as install.sh
    participant GH as github.com
    participant T as tests:<br/>outdatedCLI
    participant CD as coderd + agent

    Note over S: budget 300s
    S->>IS: downloadCoderVersion(v2.8.0)
    IS->>GH: stable-version probe (unbounded)
    IS->>GH: fetch 84 MiB asset
    GH-->>IS: /tmp/coder-e2e-cache/bin/coder-e2e-2.8.0
    IS-->>S: binaryPath

    Note over T: budget 60s, local only
    T->>T: downloadCoderVersion(v2.8.0)
    Note right of T: spawn "<bin> version" -> exit 0<br/>returns early, ~300ms, no network
    T->>CD: coder ssh --stdio, handshake, exec "exit 0"
    CD-->>T: exit code 0
```

### Why the prefetch is non-fatal

The obvious implementation raises on failure. That would be wrong here,
and I verified why rather than assuming: `tests` declares `dependencies:
["testsSetup"]`, and a failing setup project stops dependent tests from
**running at all**. Adding a deliberately-throwing setup spec produced:

```text
✓  1 [testsSetup] › addUsersAndLicense.spec.ts › setup deployment (11.7s)
✓  2 [testsSetup] › downloadCoderVersions.spec.ts › download outdated CLI (353ms)
✘  3 [testsSetup] › zzTempFail.spec.ts › temporary blast radius probe (0ms)
  1 failed
  1 did not run      <- outdatedCLI never ran
  2 passed
```

So raising would convert a one-test flake into a whole-suite outage on
any GitHub hiccup. Instead the prefetch logs a warning and returns, and
the test's own `downloadCoderVersion` call fetches inline as it does
today. The failure path is therefore no worse than the status quo, and
the success path removes the network from the test entirely.

Of the three policies available (fail hard, fall back inline, or skip
the test), this is the only one that cannot regress anything: it never
blocks the suite, and it never silently drops coverage the way an
auto-skip would.

### Restoring the 60 second budget

This is the second half of the change, and it exists for the fallback
path above. It cannot reintroduce what #16236 fixed: the timeout value
has no causal relationship to how the binary is produced, `coderBinary`
stays pre-built, `go run` stays gone, and only `outdatedCLI.spec.ts` is
touched.

It does give back a bounded sliver of the CI-latency goal, and the bound
is small. A passing run is unaffected. The cost lands only when this one
test hangs, and then it is +30 s once: `--workers 1` so there is no
fan-out, `CODER_E2E_TEST_RETRIES` is unset in CI so `retries` is 0 and
nothing multiplies it, and the job budget is `timeout-minutes: 20`.

## Measurements

Four scenarios, locally on darwin/arm64 against a freshly built
`site/e2e/bin/coder`:

| Scenario | setup spec | `outdatedCLI` | `install.sh` inside the test?
| Result |
|---|---:|---:|---|---|
| Cold, empty cache | 5.7 s | 10.0 s | **no**, ran in setup | ✓ passed |
| Warm cache | 340 ms | 11.9 s | **no**, 0 invocations | ✓ passed |
| Prefetch fails, cache empty | 1 ms | 15.4 s | yes, inline fallback | ✓
passed |
| Setup spec throws | n/a | did not run | n/a | blast radius above |

The cold run is the load-bearing one: `install.sh` is invoked from the
setup spec and the test runs local-only in 10.0 s, so the 84 MiB
download and the 20 s probe are no longer on the assertion's clock.

For context on what "local only" costs, eight consecutive `main` runs
where the CI cache already made `install.sh` a no-op:

| Job | duration |
|---|---:|
| 90382498172 | 11.7 s |
| 90352398170 | 12.1 s |
| 90335547858 | 8.1 s |
| 90317232684 | 13.7 s |
| 90297156949 | 8.4 s |
| 90280065546 | 6.9 s |
| 90265047082 | 6.9 s |
| 90250803989 | 6.6 s |

6.6 to 13.7 seconds. This change makes that the only path rather than
the lucky one.

Also checked: the test name is byte identical (`ssh with client v2.8.0`)
so flake tracking keeps matching it, `outdatedAgent` remains skipped,
and `webTerminal`, `auditLogs`, and `updateTemplate` still pass, so the
added setup dependency disturbs nothing. The full 60-test suite was not
run locally because the premium tests need `CODER_E2E_LICENSE`.

## Also in this change

The pinned versions move to `site/e2e/constants.ts` as
`oldestSupportedCLIVersion` and `oldestSupportedAgentVersion`, so the
setup spec and the tests share one source of truth, and the comments
explaining *why* those particular versions travel with them. The CI
cache key follows them there: it previously hashed the two spec files,
and now hashes `constants.ts`, so it still invalidates exactly when a
pinned version changes.

## Not addressed here

The 20 second probe is relocated, not removed. `install.sh` still
resolves the latest stable version on every pinned install, with no
`--connect-timeout` or `--max-time`, and still treats a non-200 as
fatal, so a GitHub hiccup can fail an install whose target tarball is
already cached locally. That is a user-facing bug in its own right and
wants its own PR, since fixing it means deciding what a pinned install
should print when we no longer look up what "stable" currently is.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

.github/workflows/ci.yaml

@@ -987,14 +987,14 @@ jobs:
 
       # Cache the Coder release binaries downloaded by the outdatedCLI /
       # outdatedAgent e2e tests so most runs skip the flaky GitHub release
-      # download entirely. The cache key is keyed off the test files that pin
-      # the downloaded versions, so it invalidates when those versions change.
+      # download entirely. The cache key is keyed off the file that pins the
+      # downloaded versions, so it invalidates when those versions change.
       - name: Restore e2e Coder release binary cache
         id: coder-e2e-cache
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: /tmp/coder-e2e-cache
-          key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/tests/outdatedCLI.spec.ts', 'site/e2e/tests/outdatedAgent.spec.ts') }}
+          key: coder-e2e-cache-${{ runner.os }}-${{ hashFiles('site/e2e/constants.ts') }}
           restore-keys: |
             coder-e2e-cache-${{ runner.os }}-