mirror of
https://github.com/coder/coder.git
synced 2026-09-22 05:05:20 +08:00
Bobby HoandClaude Opus 5
4e512f786f
## Summary `e2e/tests/outdatedCLI.spec.ts` has a 30 second budget in which it must create a template and workspace, start an agent, download an 84 MiB release binary from GitHub, and then exercise the actual thing under test: whether a `v2.8.0` client can still SSH into a workspace served by HEAD. In the run that filed this ticket, `install.sh` spent **20.04 seconds** of that budget on an HTTP request the test does not need, leaving 5.4 seconds for the download. The SSH flow never executed. Worth being precise about the shape, because it changes the fix. The stall is not in the code under test and it is not an SSH problem. `install.sh` resolves the latest stable release version **unconditionally**, even when `--version 2.8.0` is passed explicitly, and on the pinned path that value feeds nothing but a cosmetic post-install advisory string. Two thirds of the test's budget went to producing one sentence of console output that the test discards. Refs: https://github.com/coder/internal/issues/1571 ## Problem ### What the test is for This is a backward-compatibility test, and `v2.8.0` is the compatibility floor it enforces rather than a "supported version" in the release-channel sense. The pin traces to one code comment, `we no longer support versions prior to Tailnet v2 API support`, citing 059e533544; that commit first shipped in v2.8.0, so the pin sits exactly on the boundary it names. Worth stating plainly: this is the oldest client expected to still interoperate, not a version that receives patches. Release support is mainline / stable / n-2 / ESR, all far newer. The test's value is that it runs the *real* historical binary, compiled in Feb 2024, against a current server: `codersdk` REST compatibility, tailnet coordination v2, DERP negotiation, `coder ssh --stdio` as an SSH transport, and the agent accepting a session. Nobody gets to assert what that old client sends over the wire, which is exactly why the binary has to be downloaded rather than faked. The structural defect is that the download shares a timeout with the assertion: ```text ┌─────────────────────────────────────────────────────────────────┐ │ ONE 30-second Playwright test budget │ ├──────────────────────────────┬──────────────────────────────────┤ │ What we want to measure │ Incidental setup │ │ (deterministic, local) │ (network, non-deterministic) │ │ │ │ │ • template + workspace │ • HTTP HEAD to github.com │ │ • agent connect │ • 84 MiB download from │ │ • coder ssh --stdio │ GitHub release CDN │ │ • SSH handshake + exec │ • tar extraction │ │ • workspace stop │ │ └──────────────────────────────┴──────────────────────────────────┘ ~7-14 s, stable 0 s (cached) .. ∞ (unbounded) ``` A test that asserts protocol compatibility should not be able to fail because `github.com` was slow. ### The evidence CI runs Playwright with `DEBUG: pw:api`, and `downloadCoderVersion` passes `TRACE=1` to `install.sh`, which makes it `set -x`. The job log therefore stamps every phase. Reconstructed from [job 80049661296](https://github.com/coder/coder/actions/runs/27124540074/job/80049661296), `t=` relative to test start: ```text t=+0.000 08:17:44.671 browserContext.newPage <- test starts t=+0.882 08:17:45.553 login complete t=+4.228 08:17:48.899 workspace create submitted t=+4.519 08:17:49.190 agent-status-ready visible <- startAgent returns t=+4.526 08:17:49.197 install.sh: parse_arg --version 2.8.0 ... <- downloadCoderVersion t=+4.531 08:17:49.202 curl -sSLI https://github.com/coder/coder/releases/latest : : 20.042 SECONDS OF NOTHING : (agent logs keepalives; the page sits idle) : t=+24.573 08:18:09.244 response= 200 .../releases/tag/v2.33.6 <- probe returns t=+24.575 08:18:09.246 STABLE_VERSION=2.33.6 <- feeds a log line t=+24.582 08:18:09.253 curl -#fL -o .../coder_2.8.0_linux_amd64.tar.gz.incomplete : 5.4 s of an 84 MiB download t=+30.000 08:18:14.671 Playwright kills the test ``` Three observations rule out the originally suspected cause (slow SSH readiness or general runner slowness): - **The SSH flow never started.** `sshIntoWorkspace` is called after `downloadCoderVersion` returns, and it never returned. There is no `coder ssh --stdio` process in the log. - **The agent was healthy.** `agent-status-ready` resolved in 88 ms, and through the entire 20 second stall the agent logs a live DERP connection, successful STUN, and a completed wireguard handshake. - **The runner was fast, not slow.** Login plus template plus workspace plus agent took 4.5 seconds. ### Where the 20 seconds goes ```text install.sh main() ... L431 STABLE_VERSION=$(echo_latest_stable_version) <- ALWAYS runs | +-- echo_latest_stable_version() (install.sh:94) curl -sSLI https://github.com/coder/coder/releases/latest # no --connect-timeout # no --max-time # non-200 => exit 1 (hard failure) L454-461 the only consumers when --version is pinned: if VERSION == STABLE_VERSION: STABLE=1 L148 advisory="To install our stable release (v${STABLE_VERSION}), ..." L159 "Coder ${channel}release v${VERSION} installed. ${advisory}" ``` That is the whole dependency chain. `-sSLI` also follows redirects and `/releases/latest` *is* a redirect, so this is at minimum two round-trips to `github.com` with no timeout ceiling on either. ### Why 30 seconds and not 60 `test.setTimeout(60_000)` used to be on this test. #16236 removed it, and that removal was deliberate: it was itself a flake fix (coder/internal#204, #279) whose thesis was that `go run` compiling inside a resource-constrained test run was the problem. Having pre-built the binary, it consistently stripped the allowances that existed to absorb compile time: | File | Change in #16236 | Was that allowance really compile time? | |---|---|---| | `app.spec.ts` | `setTimeout(75_000)` removed, click timeout `60_000` -> `10_000` | Yes | | `webTerminal.spec.ts` | `setTimeout(75_000)` removed | Yes | | `helpers.ts` | agent-ready wait `45_000` -> `15_000` | Yes | | `outdatedCLI.spec.ts` | `setTimeout(60_000)` removed | **No: also an 84 MiB download** | | `outdatedAgent.spec.ts` | timeout untouched, 60 s survives | n/a | The reasoning was sound and the sweep internally consistent. It had one blind spot: for `app.spec.ts` and `webTerminal.spec.ts` that budget genuinely was the compiler's, but here it covered compile time **plus** a release download, and only the compile half went away. With 60 seconds, the failing run above would have finished in roughly 31 to 43 seconds and passed. ### Budget arithmetic At `t=+24.58` the test still had to do: | Remaining work | Realistic cost | |---|---:| | Download 84 MiB tarball | 2 - 8 s | | `tar` extract | 0.3 - 1 s | | `coder ssh --stdio` cold start | 0.5 - 2 s | | Tailnet dial + SSH handshake | 1 - 3 s | | `stopWorkspace` | 2 - 4 s | | **Needed** | **~6 - 18 s** | | **Available** | **5.42 s** | ## Fix Move the download into the existing `testsSetup` Playwright project, where it gets a 300 second budget and where a failure is attributed to the download rather than to SSH. ```mermaid flowchart TB subgraph BEFORE["BEFORE: one budget, two concerns"] direction TB T1["tests project, timeout 30s"] T1A["outdatedCLI.spec.ts<br/>login / template / workspace / agent<br/><b>downloadCoderVersion <- NETWORK</b><br/>sshIntoWorkspace / exec / stopWorkspace"] T1 --> T1A end subgraph AFTER["AFTER: network work has its own clock"] direction TB S2["testsSetup project, timeout 300s"] S2A["downloadCoderVersions.spec.ts<br/>stable-version probe + 84 MiB + retries<br/>all live HERE"] T2["tests project, timeout 60s"] T2A["outdatedCLI.spec.ts<br/>downloadCoderVersion = cache hit, ~300ms<br/>SSH path gets the whole budget"] S2 --> S2A S2A -- "dependencies" --> T2 T2 --> T2A end BEFORE ~~~ AFTER style T1A fill:#ffe5e5,stroke:#cc0000,stroke-width:2px style S2A fill:#e5ffe5,stroke:#007700,stroke-width:2px style T2A fill:#e5ffe5,stroke:#007700,stroke-width:2px ``` ### Why it works `downloadCoderVersion` was already idempotent and cache-checking: it spawns `<binaryPath> version` first and returns early on exit 0. So the test keeps its existing call and that call simply becomes a no-op costing a few hundred milliseconds. **No test logic changes.** ```mermaid sequenceDiagram autonumber participant S as testsSetup:<br/>downloadCoderVersions participant IS as install.sh participant GH as github.com participant T as tests:<br/>outdatedCLI participant CD as coderd + agent Note over S: budget 300s S->>IS: downloadCoderVersion(v2.8.0) IS->>GH: stable-version probe (unbounded) IS->>GH: fetch 84 MiB asset GH-->>IS: /tmp/coder-e2e-cache/bin/coder-e2e-2.8.0 IS-->>S: binaryPath Note over T: budget 60s, local only T->>T: downloadCoderVersion(v2.8.0) Note right of T: spawn "<bin> version" -> exit 0<br/>returns early, ~300ms, no network T->>CD: coder ssh --stdio, handshake, exec "exit 0" CD-->>T: exit code 0 ``` ### Why the prefetch is non-fatal The obvious implementation raises on failure. That would be wrong here, and I verified why rather than assuming: `tests` declares `dependencies: ["testsSetup"]`, and a failing setup project stops dependent tests from **running at all**. Adding a deliberately-throwing setup spec produced: ```text ✓ 1 [testsSetup] › addUsersAndLicense.spec.ts › setup deployment (11.7s) ✓ 2 [testsSetup] › downloadCoderVersions.spec.ts › download outdated CLI (353ms) ✘ 3 [testsSetup] › zzTempFail.spec.ts › temporary blast radius probe (0ms) 1 failed 1 did not run <- outdatedCLI never ran 2 passed ``` So raising would convert a one-test flake into a whole-suite outage on any GitHub hiccup. Instead the prefetch logs a warning and returns, and the test's own `downloadCoderVersion` call fetches inline as it does today. The failure path is therefore no worse than the status quo, and the success path removes the network from the test entirely. Of the three policies available (fail hard, fall back inline, or skip the test), this is the only one that cannot regress anything: it never blocks the suite, and it never silently drops coverage the way an auto-skip would. ### Restoring the 60 second budget This is the second half of the change, and it exists for the fallback path above. It cannot reintroduce what #16236 fixed: the timeout value has no causal relationship to how the binary is produced, `coderBinary` stays pre-built, `go run` stays gone, and only `outdatedCLI.spec.ts` is touched. It does give back a bounded sliver of the CI-latency goal, and the bound is small. A passing run is unaffected. The cost lands only when this one test hangs, and then it is +30 s once: `--workers 1` so there is no fan-out, `CODER_E2E_TEST_RETRIES` is unset in CI so `retries` is 0 and nothing multiplies it, and the job budget is `timeout-minutes: 20`. ## Measurements Four scenarios, locally on darwin/arm64 against a freshly built `site/e2e/bin/coder`: | Scenario | setup spec | `outdatedCLI` | `install.sh` inside the test? | Result | |---|---:|---:|---|---| | Cold, empty cache | 5.7 s | 10.0 s | **no**, ran in setup | ✓ passed | | Warm cache | 340 ms | 11.9 s | **no**, 0 invocations | ✓ passed | | Prefetch fails, cache empty | 1 ms | 15.4 s | yes, inline fallback | ✓ passed | | Setup spec throws | n/a | did not run | n/a | blast radius above | The cold run is the load-bearing one: `install.sh` is invoked from the setup spec and the test runs local-only in 10.0 s, so the 84 MiB download and the 20 s probe are no longer on the assertion's clock. For context on what "local only" costs, eight consecutive `main` runs where the CI cache already made `install.sh` a no-op: | Job | duration | |---|---:| | 90382498172 | 11.7 s | | 90352398170 | 12.1 s | | 90335547858 | 8.1 s | | 90317232684 | 13.7 s | | 90297156949 | 8.4 s | | 90280065546 | 6.9 s | | 90265047082 | 6.9 s | | 90250803989 | 6.6 s | 6.6 to 13.7 seconds. This change makes that the only path rather than the lucky one. Also checked: the test name is byte identical (`ssh with client v2.8.0`) so flake tracking keeps matching it, `outdatedAgent` remains skipped, and `webTerminal`, `auditLogs`, and `updateTemplate` still pass, so the added setup dependency disturbs nothing. The full 60-test suite was not run locally because the premium tests need `CODER_E2E_LICENSE`. ## Also in this change The pinned versions move to `site/e2e/constants.ts` as `oldestSupportedCLIVersion` and `oldestSupportedAgentVersion`, so the setup spec and the tests share one source of truth, and the comments explaining *why* those particular versions travel with them. The CI cache key follows them there: it previously hashed the two spec files, and now hashes `constants.ts`, so it still invalidates exactly when a pinned version changes. ## Not addressed here The 20 second probe is relocated, not removed. `install.sh` still resolves the latest stable version on every pinned install, with no `--connect-timeout` or `--max-time`, and still treats a non-200 as fatal, so a GitHub hiccup can fail an install whose target tarball is already cached locally. That is a user-facing bug in its own right and wants its own PR, since fixing it means deciding what a pinned install should print when we no longer look up what "stable" currently is. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coder is a self-hosted platform for cloud development environments and AI coding agents. Workspaces are defined with Terraform, connected through a secure Wireguard® tunnel, and automatically shut down when not used. Coder Agents runs a native AI coding agent whose loop executes in the control plane on your infrastructure, with no API keys in workspaces.
- Define cloud development environments in Terraform
- EC2 VMs, Kubernetes Pods, Docker Containers, etc.
- Automatically shutdown idle resources to save on costs
- Onboard developers in seconds instead of days
- Delegate coding work to AI agents on your infrastructure
- Bring any model (Anthropic, OpenAI, Google, Bedrock, self-hosted)
- No LLM credentials in workspaces, user identity on every action
- Centralized model governance, cost tracking, and audit logging
Quickstart
The most convenient way to try Coder is to install it on your local machine and experiment with provisioning cloud development environments using Docker (works on Linux, macOS, and Windows).
# First, install Coder
curl -L https://coder.com/install.sh | sh
# Start the Coder server (caches data in ~/.cache/coder)
coder server
# Navigate to http://localhost:3000 to create your initial user,
# create a Docker template and provision a workspace
Install
The easiest way to install Coder is to use the
install script for Linux
and macOS. For Windows, use the latest ..._installer.exe file from GitHub
Releases.
curl -L https://coder.com/install.sh | sh
You can run the install script with --dry-run to see the commands that will be used to install without executing them. Run the install script with --help for additional flags.
See install for additional methods.
Once installed, you can start a production deployment with a single command:
# Automatically sets up an external access URL on *.try.coder.app
coder server
# Requires a PostgreSQL instance (version 13 or higher) and external access URL
coder server --postgres-url <url> --access-url <url>
Use coder --help to get a list of flags and environment variables. See the install guides for a complete tutorial.
Documentation
Browse the documentation or visit a specific section below:
- Workspaces: Workspaces contain the IDEs, dependencies, and configuration information needed for software development
- Templates: Templates are written in Terraform and describe the infrastructure for workspaces
- Coder Agents: Delegate coding work to AI agents running on your self-hosted infrastructure
- Administration: Learn how to operate Coder
- Premium: Learn about paid features built for large teams
- IDEs: Connect your existing editor to a workspace
Support
Feel free to open an issue if you have questions, run into bugs, or have a feature request.
Join our Discord to provide feedback on in-progress features and chat with the community using Coder!
Integrations
New integrations are always in progress. Open an issue to request one. Contributions are welcome in any official or community repository.
Official
- Coder Registry: Templates, modules, and integrations for common development environments
- VS Code Extension: Open any Coder workspace in VS Code with a single click
- JetBrains Toolbox Plugin: Open any Coder workspace from JetBrains Toolbox with a single click
- JetBrains Gateway Plugin: Open any Coder workspace in JetBrains Gateway with a single click
- Dev Containers: Build development environments using
devcontainer.jsonon Docker, Kubernetes, and OpenShift - Kubernetes Log Stream: Stream Kubernetes Pod events to the Coder startup logs
- Self-Hosted VS Code Extension Marketplace: A private extension marketplace that works in restricted or airgapped networks integrating with code-server.
- GitHub Actions: An action to set up the Coder CLI in GitHub workflows
Community
- Community Templates: Community-contributed workspace templates in the Coder Registry
- Community Modules: Community-contributed modules to extend Coder templates
- Provision Coder with Terraform: Provision Coder on Google GKE, Azure AKS, AWS EKS, DigitalOcean DOKS, IBMCloud K8s, OVHCloud K8s, and Scaleway K8s Kapsule with Terraform
- Coder Template GitHub Action: A GitHub Action that updates Coder templates
- Discord: Chat with the community and provide feedback on in-progress features
Contributing
New contributors are always welcome. If you are new to the Coder codebase, see the contribution guide to get started.
Hiring
Apply on the careers page if you are interested in joining the team.