mirror of
https://github.com/coder/coder.git
synced 2026-11-03 05:55:24 +08:00
Jake Howell
0c3c65d85b
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Closes [DEVEX-381](https://linear.app/codercom/issue/DEVEX-381/flake-test-tasksendwaitsforworkingappstate). Follow-up to #25648 and #25858, which addressed a different symptom of the same test. ## Symptom ``` task_send_test.go:348: context expired while waiting for trap: context deadline exceeded --- FAIL: Test_TaskSend/WaitsForWorkingAppState (26.02s) ``` Windows-only, on `test-go-pg (windows-2022)`. Reported four times since #25648 landed (2026-06-02, 2026-06-10, 2026-07-01). ## Root cause The test: 1. `setupCLITaskTest` inserts `workspace_app_status(state=idle)` at the end of setup. 2. `WaitsForWorkingAppState` then inserts `workspace_app_status(state=working)` before starting the CLI. 3. Both are persisted via `dbtime.Now()`, which rounds to microseconds. Windows `time.Now()` resolution is coarser than that (often ~1 ms or worse), so back-to-back calls frequently round to the same microsecond. 4. `GetLatestWorkspaceAppStatusesByWorkspaceIDs` has no tiebreaker: ```sql ORDER BY workspace_id, created_at DESC ``` Its sibling `GetLatestWorkspaceAppStatusByAppID` already uses `ORDER BY created_at DESC, id DESC` for exactly this reason. When the two rows collide, Postgres picks either. 5. On the failing runs, the query returned the `idle` row. `waitForTaskIdle` saw idle on the first poll, returned nil, `TaskSend` proceeded, and the CLI completed successfully in ~5 s. 6. But the test was blocked at `resetTrap.MustWait(ctx)` waiting for a **second** `ticker.Reset` that never happened. `WaitLong = 25s` elapsed, line 348 failed. CI log confirms the sequence: only one `Ticker.Reset(5s)` is caught, then `Ticker.Stop([]) call, matched 0 traps` (from `defer ticker.Stop()`), then the trap wait times out. This is the same class of flake Spike documented in #15923 and #21332 ("Windows in particular doesn't have high-resolution timers"), just hidden behind a SQL `ORDER BY`. ## Fix Two changes: 1. **`coderd/database/queries/workspaceapps.sql`**: add an `id DESC` tiebreaker to `GetLatestWorkspaceAppStatusesByWorkspaceIDs`, matching `GetLatestWorkspaceAppStatusByAppID`. Makes the query deterministic when `created_at` collides. 2. **`cli/task_test.go` / `cli/task_send_test.go`**: add a `withoutInitialAppStatus()` option to `setupCLITaskTest` and use it from `WaitsForWorkingAppState`. The test now inserts a single `working` row, so the collision cannot happen in the first place. Belt-and-braces with change 1. Comments in both places reference DEVEX-381 and #21332 so the next agent doesn't have to re-derive this. ## Verification - `go test ./cli -run 'Test_TaskSend' -count=1`: all 12 subtests pass, `WaitsForWorkingAppState` completes in ~5.6 s (was ~16 s previously due to a longer poll loop). - Stress: 20 sequential runs of `WaitsForWorkingAppState` on Linux, race-enabled binary, all pass in ~5.5 s each. - `go test ./coderd -run 'AppStatus|Task' -count=1` passes. - `go vet ./coderd/database/... ./cli/...` clean. - `make lint/emdash` clean. - `gofmt` clean. Not reproducible on Linux (real time between the two patches is orders of magnitude larger than microsecond); the Windows path is fixed by making the ordering deterministic and by not creating the collision in the first place. <details> <summary>Implementation plan & decision log</summary> ### Investigation 1. Pulled the failing job log for run `28483879823/job/84428355669`. 2. Traced the mock-clock trap sequence: one `NewTicker` and exactly one `Ticker.Reset(5s)` were caught, then `Ticker.Stop([]) call, matched 0 traps` fires (the `defer ticker.Stop()` on `waitForTaskIdle` return). This proves `waitForTaskIdle` returned after a single poll, not that the trap machinery hung. 3. The command exited with `<nil>` (`clitest.go:299: command "coder task send" exited with error: <nil>`) and a `POST /send` completed in 5.4 s. So the CLI succeeded; the test's own trap wait is what timed out. 4. The only `waitForTaskIdle` return-nil paths are `Active + CurrentState.State in {Idle, Complete, Failed}` and `Active + CurrentState == nil past 30s grace`. First observation of nil cannot be past 30s. So `TaskByID` must have returned `State == Idle`. 5. Traced `TaskByID` → `taskGet` → `workspaceData` → `GetLatestWorkspaceAppStatusesByWorkspaceIDs`. Found the missing tiebreaker; the sibling query one line above (`GetLatestWorkspaceAppStatusByAppID`) already had it. 6. Confirmed the two `PATCH /app-status` calls in the Windows log happened at `00:26:13.077` and `00:26:13.093`, well within Windows timer resolution. 7. Confirmed `dbtime.Now()` rounds to microseconds; Windows `time.Now()` doesn't have that precision, so `Round(time.Microsecond)` on two calls close together frequently produces equal values. ### Prior art from Spike - #15923: loosened `HeartbeatPeriod * 9/10` to `3/4` for Windows. - #21332: switched `assert.After` to `assert.NotBefore` because timestamps can equal on Windows. Both explicitly cite "Windows doesn't always have high-resolution timers available." ### Considered alternatives - **Only fix the test.** Works today but leaves the SQL query non-deterministic; another test that relies on `GetLatestWorkspaceAppStatusesByWorkspaceIDs` could hit the same collision. - **Only fix the SQL query.** Would give a stable answer but not necessarily the *right* one. If both patches share a `created_at`, `id DESC` picks whichever UUID sorted higher, still random with respect to insertion order. - **Make `dbtime.Now()` monotonic per process.** Cleanest at the source, but affects every timestamp in the database and has broader implications than a targeted flake fix. Going with both the query fix (defense in depth, matches existing pattern) and the test fix (eliminates the collision at the source) is the smallest change that closes the flake and hardens the query. ### Rejected commit-message scopes Changes touch both `cli/` and `coderd/database/`, so per AGENTS.md the scope is omitted for the cross-cutting commit and PR title. </details>
Coder is a self-hosted platform for cloud development environments and AI coding agents. Workspaces are defined with Terraform, connected through a secure Wireguard® tunnel, and automatically shut down when not used. Coder Agents runs a native AI coding agent whose loop executes in the control plane on your infrastructure, with no API keys in workspaces.
- Define cloud development environments in Terraform
- EC2 VMs, Kubernetes Pods, Docker Containers, etc.
- Automatically shutdown idle resources to save on costs
- Onboard developers in seconds instead of days
- Delegate coding work to AI agents on your infrastructure
- Bring any model (Anthropic, OpenAI, Google, Bedrock, self-hosted)
- No LLM credentials in workspaces, user identity on every action
- Centralized model governance, cost tracking, and audit logging
Quickstart
The most convenient way to try Coder is to install it on your local machine and experiment with provisioning cloud development environments using Docker (works on Linux, macOS, and Windows).
# First, install Coder
curl -L https://coder.com/install.sh | sh
# Start the Coder server (caches data in ~/.cache/coder)
coder server
# Navigate to http://localhost:3000 to create your initial user,
# create a Docker template and provision a workspace
Install
The easiest way to install Coder is to use the
install script for Linux
and macOS. For Windows, use the latest ..._installer.exe file from GitHub
Releases.
curl -L https://coder.com/install.sh | sh
You can run the install script with --dry-run to see the commands that will be used to install without executing them. Run the install script with --help for additional flags.
See install for additional methods.
Once installed, you can start a production deployment with a single command:
# Automatically sets up an external access URL on *.try.coder.app
coder server
# Requires a PostgreSQL instance (version 13 or higher) and external access URL
coder server --postgres-url <url> --access-url <url>
Use coder --help to get a list of flags and environment variables. See the install guides for a complete tutorial.
Documentation
Browse the documentation or visit a specific section below:
- Workspaces: Workspaces contain the IDEs, dependencies, and configuration information needed for software development
- Templates: Templates are written in Terraform and describe the infrastructure for workspaces
- Coder Agents: Delegate coding work to AI agents running on your self-hosted infrastructure
- Administration: Learn how to operate Coder
- Premium: Learn about paid features built for large teams
- IDEs: Connect your existing editor to a workspace
Support
Feel free to open an issue if you have questions, run into bugs, or have a feature request.
Join our Discord to provide feedback on in-progress features and chat with the community using Coder!
Integrations
New integrations are always in progress. Open an issue to request one. Contributions are welcome in any official or community repository.
Official
- Coder Registry: Templates, modules, and integrations for common development environments
- VS Code Extension: Open any Coder workspace in VS Code with a single click
- JetBrains Toolbox Plugin: Open any Coder workspace from JetBrains Toolbox with a single click
- JetBrains Gateway Plugin: Open any Coder workspace in JetBrains Gateway with a single click
- Dev Containers: Build development environments using
devcontainer.jsonon Docker, Kubernetes, and OpenShift - Kubernetes Log Stream: Stream Kubernetes Pod events to the Coder startup logs
- Self-Hosted VS Code Extension Marketplace: A private extension marketplace that works in restricted or airgapped networks integrating with code-server.
- GitHub Actions: An action to set up the Coder CLI in GitHub workflows
Community
- Community Templates: Community-contributed workspace templates in the Coder Registry
- Community Modules: Community-contributed modules to extend Coder templates
- Provision Coder with Terraform: Provision Coder on Google GKE, Azure AKS, AWS EKS, DigitalOcean DOKS, IBMCloud K8s, OVHCloud K8s, and Scaleway K8s Kapsule with Terraform
- Coder Template GitHub Action: A GitHub Action that updates Coder templates
- Discord: Chat with the community and provide feedback on in-progress features
Contributing
New contributors are always welcome. If you are new to the Coder codebase, see the contribution guide to get started.
Hiring
Apply on the careers page if you are interested in joining the team.