mirror of
https://github.com/coder/coder.git
synced 2026-09-22 13:10:21 +08:00
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Closes [DEVEX-381](https://linear.app/codercom/issue/DEVEX-381/flake-test-tasksendwaitsforworkingappstate). Follow-up to #25648 and #25858, which addressed a different symptom of the same test. ## Symptom ``` task_send_test.go:348: context expired while waiting for trap: context deadline exceeded --- FAIL: Test_TaskSend/WaitsForWorkingAppState (26.02s) ``` Windows-only, on `test-go-pg (windows-2022)`. Reported four times since #25648 landed (2026-06-02, 2026-06-10, 2026-07-01). ## Root cause The test: 1. `setupCLITaskTest` inserts `workspace_app_status(state=idle)` at the end of setup. 2. `WaitsForWorkingAppState` then inserts `workspace_app_status(state=working)` before starting the CLI. 3. Both are persisted via `dbtime.Now()`, which rounds to microseconds. Windows `time.Now()` resolution is coarser than that (often ~1 ms or worse), so back-to-back calls frequently round to the same microsecond. 4. `GetLatestWorkspaceAppStatusesByWorkspaceIDs` has no tiebreaker: ```sql ORDER BY workspace_id, created_at DESC ``` Its sibling `GetLatestWorkspaceAppStatusByAppID` already uses `ORDER BY created_at DESC, id DESC` for exactly this reason. When the two rows collide, Postgres picks either. 5. On the failing runs, the query returned the `idle` row. `waitForTaskIdle` saw idle on the first poll, returned nil, `TaskSend` proceeded, and the CLI completed successfully in ~5 s. 6. But the test was blocked at `resetTrap.MustWait(ctx)` waiting for a **second** `ticker.Reset` that never happened. `WaitLong = 25s` elapsed, line 348 failed. CI log confirms the sequence: only one `Ticker.Reset(5s)` is caught, then `Ticker.Stop([]) call, matched 0 traps` (from `defer ticker.Stop()`), then the trap wait times out. This is the same class of flake Spike documented in #15923 and #21332 ("Windows in particular doesn't have high-resolution timers"), just hidden behind a SQL `ORDER BY`. ## Fix Two changes: 1. **`coderd/database/queries/workspaceapps.sql`**: add an `id DESC` tiebreaker to `GetLatestWorkspaceAppStatusesByWorkspaceIDs`, matching `GetLatestWorkspaceAppStatusByAppID`. Makes the query deterministic when `created_at` collides. 2. **`cli/task_test.go` / `cli/task_send_test.go`**: add a `withoutInitialAppStatus()` option to `setupCLITaskTest` and use it from `WaitsForWorkingAppState`. The test now inserts a single `working` row, so the collision cannot happen in the first place. Belt-and-braces with change 1. Comments in both places reference DEVEX-381 and #21332 so the next agent doesn't have to re-derive this. ## Verification - `go test ./cli -run 'Test_TaskSend' -count=1`: all 12 subtests pass, `WaitsForWorkingAppState` completes in ~5.6 s (was ~16 s previously due to a longer poll loop). - Stress: 20 sequential runs of `WaitsForWorkingAppState` on Linux, race-enabled binary, all pass in ~5.5 s each. - `go test ./coderd -run 'AppStatus|Task' -count=1` passes. - `go vet ./coderd/database/... ./cli/...` clean. - `make lint/emdash` clean. - `gofmt` clean. Not reproducible on Linux (real time between the two patches is orders of magnitude larger than microsecond); the Windows path is fixed by making the ordering deterministic and by not creating the collision in the first place. <details> <summary>Implementation plan & decision log</summary> ### Investigation 1. Pulled the failing job log for run `28483879823/job/84428355669`. 2. Traced the mock-clock trap sequence: one `NewTicker` and exactly one `Ticker.Reset(5s)` were caught, then `Ticker.Stop([]) call, matched 0 traps` fires (the `defer ticker.Stop()` on `waitForTaskIdle` return). This proves `waitForTaskIdle` returned after a single poll, not that the trap machinery hung. 3. The command exited with `<nil>` (`clitest.go:299: command "coder task send" exited with error: <nil>`) and a `POST /send` completed in 5.4 s. So the CLI succeeded; the test's own trap wait is what timed out. 4. The only `waitForTaskIdle` return-nil paths are `Active + CurrentState.State in {Idle, Complete, Failed}` and `Active + CurrentState == nil past 30s grace`. First observation of nil cannot be past 30s. So `TaskByID` must have returned `State == Idle`. 5. Traced `TaskByID` → `taskGet` → `workspaceData` → `GetLatestWorkspaceAppStatusesByWorkspaceIDs`. Found the missing tiebreaker; the sibling query one line above (`GetLatestWorkspaceAppStatusByAppID`) already had it. 6. Confirmed the two `PATCH /app-status` calls in the Windows log happened at `00:26:13.077` and `00:26:13.093`, well within Windows timer resolution. 7. Confirmed `dbtime.Now()` rounds to microseconds; Windows `time.Now()` doesn't have that precision, so `Round(time.Microsecond)` on two calls close together frequently produces equal values. ### Prior art from Spike - #15923: loosened `HeartbeatPeriod * 9/10` to `3/4` for Windows. - #21332: switched `assert.After` to `assert.NotBefore` because timestamps can equal on Windows. Both explicitly cite "Windows doesn't always have high-resolution timers available." ### Considered alternatives - **Only fix the test.** Works today but leaves the SQL query non-deterministic; another test that relies on `GetLatestWorkspaceAppStatusesByWorkspaceIDs` could hit the same collision. - **Only fix the SQL query.** Would give a stable answer but not necessarily the *right* one. If both patches share a `created_at`, `id DESC` picks whichever UUID sorted higher, still random with respect to insertion order. - **Make `dbtime.Now()` monotonic per process.** Cleanest at the source, but affects every timestamp in the database and has broader implications than a targeted flake fix. Going with both the query fix (defense in depth, matches existing pattern) and the test fix (eliminates the collision at the source) is the smallest change that closes the flake and hardens the query. ### Rejected commit-message scopes Changes touch both `cli/` and `coderd/database/`, so per AGENTS.md the scope is omitted for the cross-cutting commit and PR title. </details>
131 lines
4.6 KiB
SQL
131 lines
4.6 KiB
SQL
-- name: GetWorkspaceAppsByAgentID :many
|
|
SELECT * FROM workspace_apps WHERE agent_id = $1 ORDER BY slug ASC;
|
|
|
|
-- name: GetWorkspaceAppsByAgentIDs :many
|
|
SELECT * FROM workspace_apps WHERE agent_id = ANY(@ids :: uuid [ ]) ORDER BY slug ASC;
|
|
|
|
-- name: GetWorkspaceAppByAgentIDAndSlug :one
|
|
SELECT * FROM workspace_apps WHERE agent_id = $1 AND slug = $2;
|
|
|
|
-- name: GetWorkspaceAppsCreatedAfter :many
|
|
SELECT * FROM workspace_apps WHERE created_at > $1 ORDER BY slug ASC;
|
|
|
|
-- name: UpsertWorkspaceApp :one
|
|
INSERT INTO
|
|
workspace_apps (
|
|
id,
|
|
created_at,
|
|
agent_id,
|
|
slug,
|
|
display_name,
|
|
icon,
|
|
command,
|
|
url,
|
|
external,
|
|
subdomain,
|
|
sharing_level,
|
|
healthcheck_url,
|
|
healthcheck_interval,
|
|
healthcheck_threshold,
|
|
health,
|
|
display_order,
|
|
hidden,
|
|
open_in,
|
|
display_group,
|
|
tooltip
|
|
)
|
|
VALUES
|
|
($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20)
|
|
ON CONFLICT (id) DO UPDATE SET
|
|
display_name = EXCLUDED.display_name,
|
|
icon = EXCLUDED.icon,
|
|
command = EXCLUDED.command,
|
|
url = EXCLUDED.url,
|
|
external = EXCLUDED.external,
|
|
subdomain = EXCLUDED.subdomain,
|
|
sharing_level = EXCLUDED.sharing_level,
|
|
healthcheck_url = EXCLUDED.healthcheck_url,
|
|
healthcheck_interval = EXCLUDED.healthcheck_interval,
|
|
healthcheck_threshold = EXCLUDED.healthcheck_threshold,
|
|
health = EXCLUDED.health,
|
|
display_order = EXCLUDED.display_order,
|
|
hidden = EXCLUDED.hidden,
|
|
open_in = EXCLUDED.open_in,
|
|
display_group = EXCLUDED.display_group,
|
|
agent_id = EXCLUDED.agent_id,
|
|
slug = EXCLUDED.slug,
|
|
tooltip = EXCLUDED.tooltip
|
|
WHERE
|
|
-- Prevent cross-tenant/cross-workspace agent rebinding (SEC-91).
|
|
-- App IDs persist across builds of the same workspace, but agent IDs are
|
|
-- regenerated every build, so compare by the workspace that owns the agent
|
|
-- rather than by agent_id. Permit unowned apps to be claimed and permit
|
|
-- same-workspace rebuilds. If an existing app belongs to a workspace, block
|
|
-- moves to both different workspaces and template import or dry-run agents
|
|
-- that resolve to no workspace. The conflicting row is then left untouched,
|
|
-- and the :one query returns no row, which the caller treats as a
|
|
-- rejection.
|
|
NOT EXISTS (
|
|
SELECT 1
|
|
FROM workspace_agents AS existing_agent
|
|
INNER JOIN workspace_resources AS existing_resource
|
|
ON existing_agent.resource_id = existing_resource.id
|
|
INNER JOIN workspace_builds AS existing_build
|
|
ON existing_resource.job_id = existing_build.job_id
|
|
WHERE existing_agent.id = workspace_apps.agent_id
|
|
)
|
|
OR EXISTS (
|
|
SELECT 1
|
|
FROM workspace_agents AS existing_agent
|
|
INNER JOIN workspace_resources AS existing_resource
|
|
ON existing_agent.resource_id = existing_resource.id
|
|
INNER JOIN workspace_builds AS existing_build
|
|
ON existing_resource.job_id = existing_build.job_id
|
|
INNER JOIN workspace_agents AS incoming_agent
|
|
ON incoming_agent.id = EXCLUDED.agent_id
|
|
INNER JOIN workspace_resources AS incoming_resource
|
|
ON incoming_agent.resource_id = incoming_resource.id
|
|
INNER JOIN workspace_builds AS incoming_build
|
|
ON incoming_resource.job_id = incoming_build.job_id
|
|
WHERE
|
|
existing_agent.id = workspace_apps.agent_id
|
|
AND existing_build.workspace_id = incoming_build.workspace_id
|
|
)
|
|
RETURNING *;
|
|
|
|
-- name: UpdateWorkspaceAppHealthByID :exec
|
|
UPDATE
|
|
workspace_apps
|
|
SET
|
|
health = $2
|
|
WHERE
|
|
id = $1;
|
|
|
|
-- name: InsertWorkspaceAppStatus :one
|
|
INSERT INTO workspace_app_statuses (id, created_at, workspace_id, agent_id, app_id, state, message, uri)
|
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
|
RETURNING *;
|
|
|
|
-- name: GetWorkspaceAppStatusesByAppIDs :many
|
|
SELECT * FROM workspace_app_statuses WHERE app_id = ANY(@ids :: uuid [ ])
|
|
ORDER BY created_at DESC, id DESC;
|
|
|
|
-- name: GetLatestWorkspaceAppStatusByAppID :one
|
|
SELECT *
|
|
FROM workspace_app_statuses
|
|
WHERE app_id = @app_id::uuid
|
|
ORDER BY created_at DESC, id DESC
|
|
LIMIT 1;
|
|
|
|
-- name: GetLatestWorkspaceAppStatusesByWorkspaceIDs :many
|
|
-- id DESC is a stability tiebreaker, not an insertion-order signal: back-to-back
|
|
-- inserts can share a created_at on platforms with coarse time.Now() resolution,
|
|
-- and id is a random UUID, so this only guarantees a deterministic pick, not the
|
|
-- later row. Callers must not depend on sub-microsecond recency here.
|
|
SELECT DISTINCT ON (workspace_id)
|
|
*
|
|
FROM workspace_app_statuses
|
|
WHERE workspace_id = ANY(@ids :: uuid[])
|
|
ORDER BY workspace_id, created_at DESC, id DESC;
|
|
|