mirror of
https://github.com/coder/coder.git
synced 2026-09-22 13:10:21 +08:00
Bobby HoandClaude Opus 5
4e512f786f
## Summary `e2e/tests/outdatedCLI.spec.ts` has a 30 second budget in which it must create a template and workspace, start an agent, download an 84 MiB release binary from GitHub, and then exercise the actual thing under test: whether a `v2.8.0` client can still SSH into a workspace served by HEAD. In the run that filed this ticket, `install.sh` spent **20.04 seconds** of that budget on an HTTP request the test does not need, leaving 5.4 seconds for the download. The SSH flow never executed. Worth being precise about the shape, because it changes the fix. The stall is not in the code under test and it is not an SSH problem. `install.sh` resolves the latest stable release version **unconditionally**, even when `--version 2.8.0` is passed explicitly, and on the pinned path that value feeds nothing but a cosmetic post-install advisory string. Two thirds of the test's budget went to producing one sentence of console output that the test discards. Refs: https://github.com/coder/internal/issues/1571 ## Problem ### What the test is for This is a backward-compatibility test, and `v2.8.0` is the compatibility floor it enforces rather than a "supported version" in the release-channel sense. The pin traces to one code comment, `we no longer support versions prior to Tailnet v2 API support`, citing 059e533544; that commit first shipped in v2.8.0, so the pin sits exactly on the boundary it names. Worth stating plainly: this is the oldest client expected to still interoperate, not a version that receives patches. Release support is mainline / stable / n-2 / ESR, all far newer. The test's value is that it runs the *real* historical binary, compiled in Feb 2024, against a current server: `codersdk` REST compatibility, tailnet coordination v2, DERP negotiation, `coder ssh --stdio` as an SSH transport, and the agent accepting a session. Nobody gets to assert what that old client sends over the wire, which is exactly why the binary has to be downloaded rather than faked. The structural defect is that the download shares a timeout with the assertion: ```text ┌─────────────────────────────────────────────────────────────────┐ │ ONE 30-second Playwright test budget │ ├──────────────────────────────┬──────────────────────────────────┤ │ What we want to measure │ Incidental setup │ │ (deterministic, local) │ (network, non-deterministic) │ │ │ │ │ • template + workspace │ • HTTP HEAD to github.com │ │ • agent connect │ • 84 MiB download from │ │ • coder ssh --stdio │ GitHub release CDN │ │ • SSH handshake + exec │ • tar extraction │ │ • workspace stop │ │ └──────────────────────────────┴──────────────────────────────────┘ ~7-14 s, stable 0 s (cached) .. ∞ (unbounded) ``` A test that asserts protocol compatibility should not be able to fail because `github.com` was slow. ### The evidence CI runs Playwright with `DEBUG: pw:api`, and `downloadCoderVersion` passes `TRACE=1` to `install.sh`, which makes it `set -x`. The job log therefore stamps every phase. Reconstructed from [job 80049661296](https://github.com/coder/coder/actions/runs/27124540074/job/80049661296), `t=` relative to test start: ```text t=+0.000 08:17:44.671 browserContext.newPage <- test starts t=+0.882 08:17:45.553 login complete t=+4.228 08:17:48.899 workspace create submitted t=+4.519 08:17:49.190 agent-status-ready visible <- startAgent returns t=+4.526 08:17:49.197 install.sh: parse_arg --version 2.8.0 ... <- downloadCoderVersion t=+4.531 08:17:49.202 curl -sSLI https://github.com/coder/coder/releases/latest : : 20.042 SECONDS OF NOTHING : (agent logs keepalives; the page sits idle) : t=+24.573 08:18:09.244 response= 200 .../releases/tag/v2.33.6 <- probe returns t=+24.575 08:18:09.246 STABLE_VERSION=2.33.6 <- feeds a log line t=+24.582 08:18:09.253 curl -#fL -o .../coder_2.8.0_linux_amd64.tar.gz.incomplete : 5.4 s of an 84 MiB download t=+30.000 08:18:14.671 Playwright kills the test ``` Three observations rule out the originally suspected cause (slow SSH readiness or general runner slowness): - **The SSH flow never started.** `sshIntoWorkspace` is called after `downloadCoderVersion` returns, and it never returned. There is no `coder ssh --stdio` process in the log. - **The agent was healthy.** `agent-status-ready` resolved in 88 ms, and through the entire 20 second stall the agent logs a live DERP connection, successful STUN, and a completed wireguard handshake. - **The runner was fast, not slow.** Login plus template plus workspace plus agent took 4.5 seconds. ### Where the 20 seconds goes ```text install.sh main() ... L431 STABLE_VERSION=$(echo_latest_stable_version) <- ALWAYS runs | +-- echo_latest_stable_version() (install.sh:94) curl -sSLI https://github.com/coder/coder/releases/latest # no --connect-timeout # no --max-time # non-200 => exit 1 (hard failure) L454-461 the only consumers when --version is pinned: if VERSION == STABLE_VERSION: STABLE=1 L148 advisory="To install our stable release (v${STABLE_VERSION}), ..." L159 "Coder ${channel}release v${VERSION} installed. ${advisory}" ``` That is the whole dependency chain. `-sSLI` also follows redirects and `/releases/latest` *is* a redirect, so this is at minimum two round-trips to `github.com` with no timeout ceiling on either. ### Why 30 seconds and not 60 `test.setTimeout(60_000)` used to be on this test. #16236 removed it, and that removal was deliberate: it was itself a flake fix (coder/internal#204, #279) whose thesis was that `go run` compiling inside a resource-constrained test run was the problem. Having pre-built the binary, it consistently stripped the allowances that existed to absorb compile time: | File | Change in #16236 | Was that allowance really compile time? | |---|---|---| | `app.spec.ts` | `setTimeout(75_000)` removed, click timeout `60_000` -> `10_000` | Yes | | `webTerminal.spec.ts` | `setTimeout(75_000)` removed | Yes | | `helpers.ts` | agent-ready wait `45_000` -> `15_000` | Yes | | `outdatedCLI.spec.ts` | `setTimeout(60_000)` removed | **No: also an 84 MiB download** | | `outdatedAgent.spec.ts` | timeout untouched, 60 s survives | n/a | The reasoning was sound and the sweep internally consistent. It had one blind spot: for `app.spec.ts` and `webTerminal.spec.ts` that budget genuinely was the compiler's, but here it covered compile time **plus** a release download, and only the compile half went away. With 60 seconds, the failing run above would have finished in roughly 31 to 43 seconds and passed. ### Budget arithmetic At `t=+24.58` the test still had to do: | Remaining work | Realistic cost | |---|---:| | Download 84 MiB tarball | 2 - 8 s | | `tar` extract | 0.3 - 1 s | | `coder ssh --stdio` cold start | 0.5 - 2 s | | Tailnet dial + SSH handshake | 1 - 3 s | | `stopWorkspace` | 2 - 4 s | | **Needed** | **~6 - 18 s** | | **Available** | **5.42 s** | ## Fix Move the download into the existing `testsSetup` Playwright project, where it gets a 300 second budget and where a failure is attributed to the download rather than to SSH. ```mermaid flowchart TB subgraph BEFORE["BEFORE: one budget, two concerns"] direction TB T1["tests project, timeout 30s"] T1A["outdatedCLI.spec.ts<br/>login / template / workspace / agent<br/><b>downloadCoderVersion <- NETWORK</b><br/>sshIntoWorkspace / exec / stopWorkspace"] T1 --> T1A end subgraph AFTER["AFTER: network work has its own clock"] direction TB S2["testsSetup project, timeout 300s"] S2A["downloadCoderVersions.spec.ts<br/>stable-version probe + 84 MiB + retries<br/>all live HERE"] T2["tests project, timeout 60s"] T2A["outdatedCLI.spec.ts<br/>downloadCoderVersion = cache hit, ~300ms<br/>SSH path gets the whole budget"] S2 --> S2A S2A -- "dependencies" --> T2 T2 --> T2A end BEFORE ~~~ AFTER style T1A fill:#ffe5e5,stroke:#cc0000,stroke-width:2px style S2A fill:#e5ffe5,stroke:#007700,stroke-width:2px style T2A fill:#e5ffe5,stroke:#007700,stroke-width:2px ``` ### Why it works `downloadCoderVersion` was already idempotent and cache-checking: it spawns `<binaryPath> version` first and returns early on exit 0. So the test keeps its existing call and that call simply becomes a no-op costing a few hundred milliseconds. **No test logic changes.** ```mermaid sequenceDiagram autonumber participant S as testsSetup:<br/>downloadCoderVersions participant IS as install.sh participant GH as github.com participant T as tests:<br/>outdatedCLI participant CD as coderd + agent Note over S: budget 300s S->>IS: downloadCoderVersion(v2.8.0) IS->>GH: stable-version probe (unbounded) IS->>GH: fetch 84 MiB asset GH-->>IS: /tmp/coder-e2e-cache/bin/coder-e2e-2.8.0 IS-->>S: binaryPath Note over T: budget 60s, local only T->>T: downloadCoderVersion(v2.8.0) Note right of T: spawn "<bin> version" -> exit 0<br/>returns early, ~300ms, no network T->>CD: coder ssh --stdio, handshake, exec "exit 0" CD-->>T: exit code 0 ``` ### Why the prefetch is non-fatal The obvious implementation raises on failure. That would be wrong here, and I verified why rather than assuming: `tests` declares `dependencies: ["testsSetup"]`, and a failing setup project stops dependent tests from **running at all**. Adding a deliberately-throwing setup spec produced: ```text ✓ 1 [testsSetup] › addUsersAndLicense.spec.ts › setup deployment (11.7s) ✓ 2 [testsSetup] › downloadCoderVersions.spec.ts › download outdated CLI (353ms) ✘ 3 [testsSetup] › zzTempFail.spec.ts › temporary blast radius probe (0ms) 1 failed 1 did not run <- outdatedCLI never ran 2 passed ``` So raising would convert a one-test flake into a whole-suite outage on any GitHub hiccup. Instead the prefetch logs a warning and returns, and the test's own `downloadCoderVersion` call fetches inline as it does today. The failure path is therefore no worse than the status quo, and the success path removes the network from the test entirely. Of the three policies available (fail hard, fall back inline, or skip the test), this is the only one that cannot regress anything: it never blocks the suite, and it never silently drops coverage the way an auto-skip would. ### Restoring the 60 second budget This is the second half of the change, and it exists for the fallback path above. It cannot reintroduce what #16236 fixed: the timeout value has no causal relationship to how the binary is produced, `coderBinary` stays pre-built, `go run` stays gone, and only `outdatedCLI.spec.ts` is touched. It does give back a bounded sliver of the CI-latency goal, and the bound is small. A passing run is unaffected. The cost lands only when this one test hangs, and then it is +30 s once: `--workers 1` so there is no fan-out, `CODER_E2E_TEST_RETRIES` is unset in CI so `retries` is 0 and nothing multiplies it, and the job budget is `timeout-minutes: 20`. ## Measurements Four scenarios, locally on darwin/arm64 against a freshly built `site/e2e/bin/coder`: | Scenario | setup spec | `outdatedCLI` | `install.sh` inside the test? | Result | |---|---:|---:|---|---| | Cold, empty cache | 5.7 s | 10.0 s | **no**, ran in setup | ✓ passed | | Warm cache | 340 ms | 11.9 s | **no**, 0 invocations | ✓ passed | | Prefetch fails, cache empty | 1 ms | 15.4 s | yes, inline fallback | ✓ passed | | Setup spec throws | n/a | did not run | n/a | blast radius above | The cold run is the load-bearing one: `install.sh` is invoked from the setup spec and the test runs local-only in 10.0 s, so the 84 MiB download and the 20 s probe are no longer on the assertion's clock. For context on what "local only" costs, eight consecutive `main` runs where the CI cache already made `install.sh` a no-op: | Job | duration | |---|---:| | 90382498172 | 11.7 s | | 90352398170 | 12.1 s | | 90335547858 | 8.1 s | | 90317232684 | 13.7 s | | 90297156949 | 8.4 s | | 90280065546 | 6.9 s | | 90265047082 | 6.9 s | | 90250803989 | 6.6 s | 6.6 to 13.7 seconds. This change makes that the only path rather than the lucky one. Also checked: the test name is byte identical (`ssh with client v2.8.0`) so flake tracking keeps matching it, `outdatedAgent` remains skipped, and `webTerminal`, `auditLogs`, and `updateTemplate` still pass, so the added setup dependency disturbs nothing. The full 60-test suite was not run locally because the premium tests need `CODER_E2E_LICENSE`. ## Also in this change The pinned versions move to `site/e2e/constants.ts` as `oldestSupportedCLIVersion` and `oldestSupportedAgentVersion`, so the setup spec and the tests share one source of truth, and the comments explaining *why* those particular versions travel with them. The CI cache key follows them there: it previously hashed the two spec files, and now hashes `constants.ts`, so it still invalidates exactly when a pinned version changes. ## Not addressed here The 20 second probe is relocated, not removed. `install.sh` still resolves the latest stable version on every pinned install, with no `--connect-timeout` or `--max-time`, and still treats a non-200 as fatal, so a GitHub hiccup can fail an install whose target tarball is already cached locally. That is a user-facing bug in its own right and wants its own PR, since fixing it means deciding what a pinned install should print when we no longer look up what "stable" currently is. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
e2e
The structure of the end-to-end tests is optimized for speed and reliability. Not all tests require setting up a new PostgreSQL instance or using the Terraform provisioner. Deciding when to trade time for robustness rests with the developers; the framework's role is to facilitate this process.
Take a look at prior art in tests/ for inspiration. To run a test:
cd site
# Build the frontend assets. If you are actively changing
# the site to debug an issue, add `--watch`.
pnpm build
# Alternatively, build with debug info and source maps:
NODE_ENV=development pnpm vite build --mode=development
# Install the browsers to `~/.cache/ms-playwright`.
pnpm playwright:install
# Run E2E tests. You can see the configuration of the server
# in `playwright.config.ts`. This builds and runs `site/e2e/bin/coder`.
pnpm playwright:test
# Run a specific test (`-g` stands for grep. It accepts regex).
pnpm playwright:test -g '<your test here>'
Using nix
If this breaks, it is likely because the flake chromium version and playwright are no longer compatible. To fix this, update the flake to get the latest chromium version, and adjust the playwright version in the package.json.
You can see the playwright version here: https://search.nixos.org/packages?channel=unstable&show=playwright-driver&from=0&size=50&sort=relevance&type=packages&query=playwright-driver
# Optionally add '--command zsh' to choose your shell.
nix develop
cd site
pnpm install
pnpm build
pnpm playwright:test
To run the playwright debugger from VSCode, just launch VSCode from the nix environment and have the extension installed.
# Optionally add '--command zsh' to choose your shell.
nix develop
code .
Enterprise tests
Enterprise tests require a license key to run.
export CODER_E2E_LICENSE=<license key>
Debugging tests
To debug a test, it is more helpful to run it in ui mode.
pnpm playwright:test-ui