## Summary
Each of the five `TestPrebuildsAutobuild` subtests spent about 30
seconds of its 60 second context budget waiting for a prebuilt workspace
whose build job had already been created and queued. On a quiet machine
the remaining budget is enough and the test passes; under
`test-go-race-pg` it is not, and the subtest fails at `found 0 running
prebuilds so far, want 1`.
Worth being precise about the shape, because it changes the fix: this is
not a data race. The 30 second stall is deterministic and every run pays
it in full. Only the *failure* is intermittent, because it depends on
whether the leftover budget covers the rest of the test.
Refs: https://github.com/coder/internal/issues/1578
## Problem
`StoreReconciler` publishes `provisioner_job_posted` to pubsub so that
provisionerd wakes up and acquires a newly created job. That publish
does not happen inline. `publishProvisionerJob` performs a non-blocking
send onto an internal buffered channel, and the goroutine that drains
that channel and calls `provisionerjobs.PostJob` is created inside
`StoreReconciler.Run`:
```go
// enterprise/coderd/prebuilds/reconcile.go, inside Run()
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-ctx.Done():
return
case job := <-c.provisionNotifyCh:
err := provisionerjobs.PostJob(c.pubsub, job)
...
}
}
}()
```
These tests drive the reconciler directly through `SnapshotState` /
`CalculateActions` / `ReconcilePreset` and never start `Run`. The
notification therefore lands in a cap-10 channel with no reader, the
non-blocking send succeeds silently, and provisionerd does not learn
about the job until the Acquirer's 30 second backup poll fires.
Laid out as a relay across goroutines, the hand-off is severed at the
first hop:
```mermaid
flowchart LR
subgraph G1["goroutine: test body"]
T1["ReconcilePreset()"]
T2["testutil.Eventually<br/>1s poll of the DB"]
end
subgraph G2["goroutine: Run() drain worker"]
D["case job := <-provisionNotifyCh:<br/>PostJob(pubsub, job)"]
end
subgraph G3["goroutine: pubsub listener"]
H["Acquirer.jobPosted<br/>-> clearOrPend(domain)"]
end
subgraph G4["goroutine: domain.poll"]
P["ticker 30s, REAL clock<br/>-> clearOrPend(domain)"]
end
subgraph G5["goroutine: provisionerd AcquireJob"]
A["select { <-ctx.Done() ; <-clearance }"]
end
CH1[["provisionNotifyCh<br/>chan ProvisionerJob, cap 10"]]
CH2[["clearance<br/>chan struct{}, cap 1"]]
DB[("Postgres")]
T1 -- "non-blocking send" --> CH1
CH1 -. "NO READER:<br/>Run() never started" .-> D
D -. "never reached" .-> H
H -. "never fires" .-> CH2
P -- "every 30s:<br/>the only live writer" --> CH2
CH2 --> A
A -- "AcquireProvisionerJob" --> DB
T2 -- "GetRunningPrebuiltWorkspaces" --> DB
style G2 fill:#f2f2f2,stroke-dasharray: 5 5
style CH1 fill:#ffe5e5,stroke:#cc0000,stroke-width:2px
```
Two properties turn this into a quiet latency bug rather than an obvious
failure:
- The channel is **buffered**, so a send with no reader succeeds instead
of blocking or panicking. The writer never learns that nobody is
listening.
- `domain.poll` ticks on the **real** clock, so the test's mock clock
cannot skip it. That is the entire 30 seconds.
From the CI job that filed the ticket, every job created through the
HTTP API is picked up in about a millisecond, and only the
reconciler-created prebuild job is not:
```text
19:11:40.760 pubsub: publish event=provisioner_job_posted <- template import job
19:11:40.761 acquirer: got job posting <- picked up in 1ms
...
19:11:40.903 prebuild job scheduled job_id=ac196e6e-...
(no "pubsub: publish", no "acquirer: got job posting")
19:11:41 .. 19:12:10 30 x "found 0 running prebuilds so far, want 1"
19:12:10.803 acquirer: successfully acquired job ac196e6e-... <- 29.899s later, via backup poll
```
Corroboration from the existing test suite: `FailureTTLOnlyAfterClaimed`
had already run into this. It builds its Acquirer on a mock clock and
calls `acquirerClock.Advance(30 * time.Second)` right after reconciling,
with a comment about the backup-poll ticker. A previous author found the
same dependency and worked around it by making the poll fire instantly
rather than by restoring the notification.
### Where that lands in the test
Each subtest is three helpers called in order. They never call each
other; they communicate through Postgres plus one returned value.
`runReconciliationLoop` performs no writes itself, they all happen
inside `ReconcilePreset`, whose transaction has committed by the time it
returns.
```mermaid
sequenceDiagram
autonumber
participant T as test body
participant H1 as runReconciliationLoop
participant H2 as getRunningPrebuilds
participant H3 as claimPrebuild
participant R as StoreReconciler
participant DB as Postgres
participant PD as provisionerd
T->>H1: (t, ctx, db, reconciler, presets)
H1->>R: ReconcilePreset
R->>DB: InsertWorkspace(owner=prebuilds)
R->>DB: builder.Build -> build(start) + job(pending)
R->>DB: COMMIT
R->>R: publishProvisionerJob -> provisionNotifyCh<br/>non-blocking send, no reader, DROPPED
R-->>H1: nil
Note over H1,PD: nothing publishes provisioner_job_posted
H1-->>T: void
T->>H2: (t, ctx, db, want=1)
loop 30 polls, 1s apart
H2->>DB: GetRunningPrebuiltWorkspaces
DB-->>H2: 0 rows (job still pending)
end
PD->>DB: acquire, via the 30s backup poll
PD->>DB: CompleteJob, IsPrebuild so deadline stays zero
H2->>DB: GetRunningPrebuiltWorkspaces
DB-->>H2: 1 row (succeeded)
H2->>DB: UPDATE agents SET lifecycle_state='ready'
H2-->>T: rows, test captures prebuild.ID
T->>H3: (client, userClient, user, version, presetID)
H3->>DB: CreateUserWorkspace(presetID)<br/>-> ClaimPrebuiltWorkspace, requires ready
DB-->>H3: same workspace, new owner
H3-->>T: workspace
Note over T: require.Equal(prebuild.ID, workspace.ID)<br/>~30s of the 60s budget already gone
```
The defect is in `runReconciliationLoop`, but the waiting, and therefore
the failing log line, is in `getRunningPrebuilds`. Note also that
`claimPrebuild` was never affected: it builds through the HTTP API,
which publishes on the normal `wsbuilder` path, so its job was always
acquired in about a millisecond. The bug was never "prebuild jobs are
slow", it was "jobs created by the reconciler, driven directly, are
never announced".
## Fix
Publish the pending provisioner jobs on the reconciler's behalf, in the
test helper, immediately after reconciling. No production code changes.
```mermaid
flowchart LR
subgraph G1["goroutine: test body"]
T1["ReconcilePreset()"]
T3["NEW: post pending jobs<br/>provisionerjobs.PostJob(pb, job)"]
end
subgraph G3["goroutine: pubsub listener"]
H["Acquirer.jobPosted<br/>-> clearOrPend(domain)"]
end
subgraph G5["goroutine: provisionerd AcquireJob"]
A["unblocks on <-clearance"]
end
CH2[["clearance<br/>chan struct{}, cap 1"]]
DB[("Postgres")]
T1 --> T3
T3 -- "publish provisioner_job_posted" --> H
H -- "send" --> CH2
CH2 --> A
A -- "AcquireProvisionerJob, ~1ms" --> DB
style T3 fill:#e5ffe5,stroke:#007700,stroke-width:2px
```
This works because provisionerd is already subscribed and already parked
in `select { <-ctx.Done(); <-clearance }`. It needs exactly one write to
`clearance`, and today the only live writer is the 30 second poll
ticker. Publishing to pubsub gives `jobPosted` a reason to fire, and
`clearOrPendLocked` performs that write immediately.
Posting every still-`pending` job, rather than trying to identify the
one just created, keeps the helper idempotent and avoids coupling to
whichever clock stamped `created_at`. Re-posting a job that was already
acquired is harmless: the Acquirer re-queries and finds nothing.
The same three helpers after the change. `getRunningPrebuilds` collapses
to a single poll, and nothing else about the test moves:
```mermaid
sequenceDiagram
autonumber
participant T as test body
participant H1 as runReconciliationLoop
participant H2 as getRunningPrebuilds
participant H3 as claimPrebuild
participant R as StoreReconciler
participant PS as Pubsub
participant DB as Postgres
participant PD as provisionerd
T->>H1: (t, ctx, db, pb, reconciler, presets)
H1->>R: ReconcilePreset
R->>DB: InsertWorkspace + build(start) + job(pending), COMMIT
R->>R: publishProvisionerJob still dropped<br/>(production path, unchanged)
R-->>H1: nil
Note over H1,DB: job row is committed and visible,<br/>which is why the query below finds it
H1->>DB: GetProvisionerJobsCreatedAfter(zero time)
DB-->>H1: all jobs, filtered in Go to status=pending
H1->>PS: PostJob -> provisioner_job_posted
PS->>PD: acquirer wakes, clearance write
H1-->>T: void
PD->>DB: acquire in ~1ms, then CompleteJob
T->>H2: (t, ctx, db, want=1)
H2->>DB: GetRunningPrebuiltWorkspaces
DB-->>H2: 1 row (succeeded), queued_for ~3ms
H2->>DB: UPDATE agents SET lifecycle_state='ready'
H2-->>T: rows, test captures prebuild.ID
T->>H3: (client, userClient, user, version, presetID)
H3->>DB: CreateUserWorkspace(presetID) -> claim
DB-->>H3: same workspace, new owner
H3-->>T: workspace
Note over T: same assertions, ~55s of budget still unspent
```
`getRunningPrebuilds` still polls, still forces agents ready,
`claimPrebuild` still claims, and every assertion is unchanged. Its
floor is now one `testutil.IntervalSlow` tick, about a second, because
`testutil.Eventually` fires on a ticker rather than checking
immediately.
Note that `publishProvisionerJob` at `reconcile.go:940` is still
dropped. That call site is correct; it simply has no drain worker behind
it when `Run` is not started. The `PostJob` added here is a manual redo
of what it already intended.
Starting `reconciler.Run(ctx)` instead would be closer to production,
but `Run` also starts a reconciliation ticker on the **mock** clock, and
these tests jump that clock by hours. Each jump would fire an
unscheduled `ReconcileAll` that rebuilds a replacement prebuild
mid-assertion, which is the opposite of what a deflake should introduce.
## Measurements
Single subtest with `-race` against Postgres, the closest local
approximation of `test-go-race-pg`, three iterations:
| Run | before | after |
|-----|--------|--------|
| 1 | 58.85s | 30.68s |
| 2 | 55.42s | 35.64s |
| 3 | 57.47s | 31.30s |
The baseline passed all three, at 1.1s to 4.6s of margin against the 60
second context. That is the flake caught in the act: locally green, one
scheduling hiccup from red. After the change the margin is 24s to 29s.
All five subtests against real Postgres go from roughly 35s each to
6.76s each, and `queued_for` on the prebuild job drops from 29.975s to
single-digit milliseconds.
## Also in this change
Two smaller items in the same helper, both aimed at the next person to
see this symptom.
Diagnostics while waiting for prebuilds: poll count, elapsed time, and
`queued_for` (`started_at - created_at` on the provisioner job), plus a
warning naming this defect if the wait exceeds 10 seconds. `queued_for`
is the field that discriminates: about 0 means the notification arrived
and any slowness is elsewhere, about 30 seconds means it was lost and
the backup poll took over.
There is deliberately no duration computed against `completed_at`. A
single `provisioner_jobs` row mixes time bases in these tests:
`created_at` and `started_at` come from the real clock, while
`completed_at` is stamped by `CompleteJob` from the injected mock clock.
My first version of the logging did subtract them and printed
`ran_for=-22543h3m21s`.
`getRunningPrebuilds` also now resets its accumulator each poll. It
appended rows on every iteration without clearing, so an iteration that
appended and then returned early on a transient error would double count
and leave the expected count permanently unreachable, producing this
same `found N running prebuilds` symptom for an unrelated reason.
- Wires quartz.Clock into provisionerdserver.Acquirer
- Allows overriding Acquirer in coderd.Options
- Updates existing tests to use an Acquirer driven by a quartz.Mock
Before this change `enterprise/coderd/prebuilds` package tests would
take ~60-70s to run.
After this change, it's down to ~10s.
> Generated by Coder agents, massaged by this human.
`TestWorkspaceTagsTerraform` runs a real terraform provisioner but
waited on builds with `coderdtest` helpers whose deadlines are sized for
the echo provisioner used by most tests, which replays canned responses
and completes in well under a second. On Windows runners, where
terraform providers are not cached and every `terraform init` downloads
from the registry, template imports exceeded the 25s budget in
`AwaitTemplateVersionJobCompleted` and workspace builds exceeded the 10s
context in `AwaitWorkspaceBuildJobCompleted`, even though the test
intends a 120s budget.
Add `AwaitTemplateVersionJobCompletedWithTimeout` and
`AwaitWorkspaceBuildJobCompletedWithTimeout`, which take a
caller-provided wait bound, and use them in the test with
`2*testutil.WaitSuperLong` (120s). Also fix
`AwaitWorkspaceBuildJobCompleted` creating a `WaitShort` (10s) context
while polling for `WaitMedium` (15s), which guaranteed `context deadline
exceeded` errors for the final five seconds of polling.
`TestWorkspaceTemplateParamsChange` has the same shape (real terraform
provisioner, 120s test context, plain await helpers) and the same latent
bug, so it gets the same fix.
Closes https://github.com/coder/internal/issues/1470 (Linear: PLAT-176)
<details>
<summary>Root cause analysis</summary>
Two CI failures, same mechanism:
- 2026-04-16 (run 24493089585, windows-2022):
`overrides_with_dynamic_option_from_var/dynamic` failed at
`coderdtest.AwaitTemplateVersionJobCompleted` with `Condition never
satisfied ... make sure you set IncludeProvisionerDaemon!`. The template
import job (real terraform init/plan, with network provider download)
did not complete within `WaitLong` (25s).
- 2026-05-27 (run 26492817796, windows-2022): `tag_param/dynamic` failed
at `coderdtest.AwaitWorkspaceBuildJobCompleted` with `failed to get
workspace build ...: context deadline exceeded`. The helper's internal
context was `WaitShort` (10s) while its polling window was `WaitMedium`
(15s), so after 10s every poll could only fail. The logged `terraform
apply: exit status 1` and the `TempDir RemoveAll ... Access is denied`
cleanup error are consequences of test teardown canceling the in-flight
job while the provider exe was still file-locked.
The test declares a 120s budget (`2*testutil.WaitSuperLong`, commented
"This can take a while"), but the await helpers ignored it and applied
their own 10-25s budgets. `testutil.CacheTFProviders` is a no-op on
Windows, so real builds are much slower there.
This change raises the ceiling for the tests rather than making
terraform faster; both observed failure signatures are eliminated. The
default helper budgets are unchanged for the ~880 existing call sites.
One small behavior change: `AwaitTemplateVersionJobCompleted` previously
marked the test failed on any transient poll error via `assert.NoError`;
it now logs and keeps polling, matching the workspace build helper, and
still fails on timeout.
`TestWorkspaceTemplateParamsChange` is the sibling real-terraform test
in the same file (also covered by the original provider-caching work in
#20603). It runs three sequential real builds with the plain await
helpers under a 120s context, so it is exposed to the same Windows
slowness even though it has not produced its own issue yet. Its context
is raised to `6*testutil.WaitSuperLong` to outlast three sequential
await budgets.
API note: a context-taking variant was considered first, but a
`time.Duration` parameter avoids an implicit "context must have a
deadline" contract and matches how the existing helpers manage their own
wait budgets.
</details>
---
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.
`TestExecutorAutostartOK` and its sibling positive autostart tests
compute the autobuild tick from
`sched.Next(workspace.LatestBuild.CreatedAt)`, but the server persists
`next_start_at` from the build's completion time. When build creation
and completion straddle the schedule's next fire time, the persisted
value advances past the test's tick, the executor's eligibility query
(`next_start_at <= tick`) drops the workspace, and the test fails with
an empty transitions map. This surfaced in flaky test runs.
Add `coderdtest.NextAutostartTick(t, workspace)` which returns
`*workspace.NextStartAt`, and use it across the affected positive
autostart paths in `coderd/autobuild`, `coderd`, and
`enterprise/coderd`.
Generated with assistance from Coder Agents.
Closes https://github.com/coder/coder/issues/13112
**Breaking Change**: Removed status code `StatusNotModified` when no
diffs occur in a patch. Now the patch is always applied and a template
is always returned.
`time.Now()` has nanosecond precision while Postgres timestamps are
microsecond precision. When tests compare `time.Now()` against
DB-sourced timestamps using `Before`/`After`/`WithinRange`/etc., there
is a non-zero flake risk from the precision mismatch.
This replaces `time.Now()` with `dbtime.Now()` (which rounds to
microsecond precision) in all test assertions that compare against
database timestamps.
Follows from #22684.
## Changes (11 files)
| File | Changes |
|---|---|
| `coderd/apikey_test.go` | 11 comparisons with `ExpiresAt` |
| `coderd/users_test.go` | 2 comparisons with `ExpiresAt` |
| `coderd/oauth2_test.go` | 1 comparison with `token.Expiry` |
| `coderd/workspaces_test.go` | 2 comparisons with `DormantAt` |
| `coderd/workspaceagents_test.go` | 3 comparisons with
`ConnectedAt`/`DisconnectedAt` |
| `coderd/workspaceapps/db_test.go` | 1 comparison with `token.Expiry` |
| `coderd/provisionerdserver/provisionerdserver_test.go` | 1 comparison
with `key.ExpiresAt` |
| `enterprise/coderd/workspaces_test.go` | 1 comparison with `DormantAt`
|
| `enterprise/coderd/license/license_test.go` | 3 `NotBefore` values |
| `enterprise/coderd/licenses_test.go` | 2 `NotBefore` values |
| `enterprise/coderd/users_test.go` | 3 `Next()` comparisons |
## Not changed (intentionally)
- `scaletest/placebo/run_test.go` — compares wall-clock elapsed time,
not DB timestamps
- `cli/server_test.go`, `coderd/jwtutils/jwt_test.go`,
`enterprise/aibridgeproxyd/aibridgeproxyd_test.go` — TLS cert fields,
not DB-stored
- `coderd/azureidentity/azureidentity_test.go` — Azure cert expiry, not
DB
🤖 Generated by Claude Opus 4.6 but reviewed manually.
## Problem
`TestWorkspaceProvisionerdServerMetrics` flakes because metric
assertions run immediately after
`AwaitWorkspaceBuildJobCompleted` returns, but metrics are updated
**asynchronously after the
DB transaction commits** in `completeWorkspaceBuildJob`.
The timeline in the provisioner server:
1. DB transaction commits (`provisionerdserver.go:~2362`) — job marked
completed
2. Audit logging, notifications, DB queries (`~2370-2427`)
3. **Metric `.Observe()`** (`~2463`) — happens ~100 lines later
The test synchronization (`AwaitWorkspaceBuildJobCompleted`) polls for
`CompletedAt != nil`,
which fires at step 1. The metric assertion then executes before step 3,
causing the flake.
## Fix
Wrap all three metric assertions (prebuild creation, prebuild claim,
regular workspace
creation) in `require.Eventually` to poll until the metric appears, then
assert on the value.
## Test
- `go test -run TestWorkspaceProvisionerdServerMetrics -count=5` — all
pass
- `go test -race -run TestWorkspaceProvisionerdServerMetrics -count=1` —
clean
In relation to
[`internal#1281`](https://github.com/coder/internal/issues/1281)
Remove the `soft_limit` field from the `Feature` type and simplify
license limit handling. This change:
- Removes the `soft_limit` field from the API and SDK
- Uses the soft limit value as the single `limit` value in the UI and
API
- Simplifies warning logic to only show warnings when the limit is
exceeded
- Updates tests to reflect the new behavior
- Updates the UI to use the single limit value for display
Since Go 1.22, the loop variable capture issue is resolved. Variables
declared by for loops are now per-iteration rather than per-loop, making
the 'v := v' pattern unnecessary.
This PR adds some metrics to help identify job enqueue rates and
latencies. This work was initiated as a way to help reduce the cost of
the observation/measurement itself for autostart scaletests, which
impacts our ability to identify/reason about the load caused by
autostart. See: https://github.com/coder/internal/issues/1209
I've extended the metrics here to account for regular user initiated
builds, prebuilds, autostarts, etc. IMO there is still the question here
of whether we want to include or need the `transition` label, which is
only present on workspace builds. Including it does lead to an increase
in cardinality, and in the case of the histogram (when not using native
histograms) that's at least a few extra series for every bucket. We
could remove the transition label there but keep it on the counter.
Additionally, the histogram is currently observing latencies for other
jobs, such as template builds/version imports, those do not have a
transition type associated with them.
Tested briefly in a workspace, can see metric values like the following:
-
`coderd_workspace_builds_enqueued_total{build_reason="autostart",provisioner_type="terraform",status="success",transition="start"}
1`
-
`coderd_provisioner_job_queue_wait_seconds_bucket{build_reason="autostart",job_type="workspace_build",provisioner_type="terraform",transition="start",le="0.025"}
1`
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The test was creating two template versions without explicit names,
relying on `namesgenerator.NameDigitWith()` which can produce
collisions. When both versions got the same random name, the test failed
with a 409 Conflict error.
Fix by giving each version an explicit name (`v1`, `v2`).
Closes https://github.com/coder/internal/issues/1309
---
*Generated by [mux](https://mux.coder.com)*
Only task workspaces have the checks in wsbuilder for violating the
managed agent caps in the license.
Stopped tasks that are resumed with a regular workspace start **still
count as usage**.
The removal of that permission from the role broke valid use cases (e.g.
a site owner user creating a workspace owned by a system account and
then trying to share it with another user).
The bulk of the PR is made up of the rollbacks of the previously
introduced test updates necessitated by the removal.
Related to: https://github.com/coder/internal/issues/1285
Agents were losing authentication during workspace shutdown, causing
shutdown scripts to fail. The auth query required agents to belong to
the latest build, but during shutdown a `stop` build becomes latest while
the `start` build's agents are still running.
Modified the auth query to allow `start` build agents to authenticate
temporarily during `stop` execution. The query allows auth when:
- Agent's `start` build job succeeded
- Latest build is `stop` with `pending`/`running` job status
- Builds are adjacent (`stop` is `build_number + 1`)
- Template versions match
Auth closes once `stop` completes.
Renamed `GetWorkspaceAgentAndLatestBuildByAuthToken` to
`GetAuthenticatedWorkspaceAgentAndBuildByAuthToken` since it returns the
agent's build (not always latest) during shutdown.
Closes coder/internal#1249
Fixes#19467
## Description
This PR addresses database connection pool exhaustion during prebuilds
reconciliation by introducing two changes:
* `CanSkipReconciliation`: Filters out presets that don't need
reconciliation before spawning goroutines. This ensures we only create
goroutines for presets that will (_most likely_) perform database
operations, avoiding unnecessary connection pool usage.
* Dynamic `eg.SetLimit`: Limits concurrent goroutines based on the
configured database connection pool size (`CODER_PG_CONN_MAX_OPEN / 2`).
This replaces the previous hardcoded limit of 5, ensuring the
reconciliation loop scales appropriately with the configured pool size
while leaving capacity for other database operations.
## Changes
* Add `CanSkipReconciliation()` method to `PresetSnapshot` that returns
true for inactive presets with no running workspaces, no pending jobs,
or expired prebuilds.
* Add `maxDBConnections` parameter to `NewStoreReconciler` and compute
`reconciliationConcurrency` as half the pool size (minimum 1).
* Add `ReconciliationConcurrency()` getter method to `StoreReconciler`.
* Add `eg.SetLimit(c.reconciliationConcurrency)` to bound concurrent
reconciliation goroutines.
* Add `PresetsTotal` and `PresetsReconciled` to `ReconcileStats` for
observability.
* Add `TestCanSkipReconciliation` unit tests.
* Add `TestReconciliationConcurrency` unit tests.
* Add benchmark tests for reconciliation performance.
## Benchmarks
* `BenchmarkReconcileAll_NoOps`: Tests presets with no reconciliation
actions. All presets are filtered by `CanSkipReconciliation`, resulting
in no goroutines spawned and no database connections used.
* `BenchmarkReconcileAll_ConnectionContention`: Tests presets where all
require reconciliation actions. All presets spawn goroutines, but
concurrency is limited by `eg.SetLimit(reconciliationConcurrency)`.
* `BenchmarkReconcileAll_Mix`: Simulates a realistic scenario with a
large subset of inactive presets (filtered by `CanSkipReconciliation`)
and a smaller subset requiring reconciliation (limited by
`eg.SetLimit`).
Closes: https://github.com/coder/coder/issues/20606
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
The implementation for prebuilt workspaces is complex and conversations
regarding edge cases and bugs frequently get bogged down by minutiae,
because it's hard to reason about the behaviour of the system.
To alleviate this, I've introduced otel tracing to the StoreReconciler
(see attached). We can now directly observe the behaviour of the
prebuilds system under load in order to inform our decisions.
Traces are terminated at the boundary between prebuilds and workspace
builder, because of prebuilt workspaces' "fire and forget" philosophy
and to prevent span explosion.
<img width="3024" height="1718" alt="image"
src="https://github.com/user-attachments/assets/f9b207be-8f2c-475e-98a8-46ef70bda446"
/>
Delete builds were not deleting resources as the tf state being sent in the apply request was empty.
State removed from apply request and read from the session instead.
Provisioner steps broken into smaller granular actions.
Changes:
- `ExtractArchive` moved to `init` request (was in `configure`)
- Writing `tfstate` moved to `plan` (was in `configure`)
- Moved most plan/apply outputs to `GraphComplete`
closes: https://github.com/coder/internal/issues/858
Similar to https://github.com/coder/coder/pull/19375, this one uses
system permissions for fetching actual user and group data.
Modifies the `workspaces_expanded` view to fetch the required data; this way it's made available to all code paths that make use of it.
Also fixes a bug in a test helper function that can result in `null` being saved to the DB for `user_acl` or `group_acl` and break tests; a defensive check constraint that prevents this is worth a PR, e.g:
`ALTER TABLE workspaces
ADD CONSTRAINT group_acl_is_object CHECK (jsonb_typeof(group_acl) = 'object');`
Also adds missing `OwnerName` in `ConvertWorkspaceRows`.
Closes https://github.com/coder/coder/issues/20711
We now allow agents to be created on dormant workspaces.
I've ran the test with and without the change. I've confirmed that -
without the fix - it triggers the "rbac: unauthorized" error.
Fixes https://github.com/coder/internal/issues/1119
## Description
The `CacheTFProviders` function in `testutil/terraform_cache.go` was
only available on Linux and macOS due to the `//go:build linux ||
darwin` build tag. This caused a compile error on Windows when
`enterprise/coderd/workspaces_test.go` tried to call it:
```
enterprise\coderd\workspaces_test.go:3403:28: undefined: testutil.CacheTFProviders
```
## Changes
1. Added `testutil/terraform_cache_windows.go` with a Windows-specific
stub implementation that returns an empty string
2. Updated `downloadProviders` helper in
`enterprise/coderd/workspaces_test.go` to handle empty paths gracefully
## Behavior
- On Linux/macOS: Terraform providers are cached as before
- On Windows: Provider caching is skipped, tests download providers
normally during `terraform init`
## Testing
This should fix the Windows nightly gauntlet failure. The test will
still run on Windows, just without provider caching optimization.
Co-authored-by: blink-so[bot] <211532188+blink-so[bot]@users.noreply.github.com>
Fixes flaky `TestWorkspaceTagsTerraform` and
`TestWorkspaceTemplateParamsChange` tests that were failing with
`connection reset by peer` errors when downloading the coder/coder
provider.
This applies the same caching solution which was done in
https://github.com/coder/coder/pull/17373
1. Extracts provider caching logic into `testutil/terraform_cache.go`
2. Updates TestProvision to use the shared caching helpers
3. Updates enterprise workspace tests to use the shared caching helpers
The cache is persisted at `~/.cache/coderv2-test/` and automatically
cached between CI runs via existing GitHub Actions cache setup.
Closes https://github.com/coder/internal/issues/607
Adds shared_with_user and shared_with_group filters to the /workspaces
endpoint.
- `shared_with_user`: filters workspaces shared with a specific user.
Accepts a user UUID or username.
- `shared_with_group`: filters workspaces shared with a specific group.
Accepts:
- a group UUID, or
- `<organization name>/<group name>`, or
- `<group name>` (resolved in the default organization).
Closes
[coder/internal#1004](https://github.com/coder/internal/issues/1004)
## Description
When creating a prebuilt workspace, both `flags.IsPrebuild` and
`flags.IsFirstBuild` are true. Previously, the logic rejected cases with
multiple flags, so `coderd_workspace_creation_duration_seconds` wasn’t
updated for prebuilt creations. This is the only valid scenario where
two flags can be true.
## Changes
* Fix logic to update `coderd_workspace_creation_duration_seconds`
metric for prebuilt workspaces.
* Add prebuild helper functions to coderdenttest (other prebuild tests
can reuse this).
* Update workspace's provisionerdmetric tests to include this metric.
Follow-up: https://github.com/coder/coder/pull/19503
Related to: https://github.com/coder/coder/issues/19528
The flake here had two causes:
1. related to usage of time.Now() in MustWaitForProvisionersAvailable
and
2. the fact that UpdateProvisionerLastSeenAt can not use a time that is
further in the past than the current LastSeenAt time
Previously the test here was calling
`coderdtest.MustWaitForProvisionersAvailable` which was using `time.Now`
rather than the next tick time like the real `hasProvisionersAvailable`
function does. Additionally, when using `UpdateProvisionerLastSeenAt`
the underlying db query enforces that the time we're trying to set
`LastSeenAt` to cannot be older than the current value.
I was able to reliably reproduce the flake by executing both the
`UpdateProvisionerLastSeenAt` call and `tickCh <- next` in their own
goroutines, the former with a small sleep to reliably ensure we'd
trigger the autobuild before we set the `LastSeenAt` time. That's when I
also noticed that `coderdtest.MustWaitForProvisionersAvailable` was
using `time.Now` instead of the tick time. When I updated that function
to take in a tick time + added a 2nd call to
`UpdateProvisionerLastSeenAt` to set an original non-stale time, we
could then never get the test to pass because the later call to set the
stale time would not actually modify `LastSeenAt`. On top of that,
calling the provisioner daemons closer in the middle of the function
doesn't really do anything of value in this test.
**The fix for the flake is to keep the go routines, ensuring there would
be a flake if there was not a relevant fix, but to include the fix which
is to ensure that we explicitly wait for the provisioner to be stale
before passing the time to `tickCh`.**
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
## Description
This PR introduces one counter and two histograms related to workspace
creation and claiming. The goal is to provide clearer observability into
how workspaces are created (regular vs prebuild) and the time cost of
those operations.
### `coderd_workspace_creation_total`
* Metric type: Counter
* Name: `coderd_workspace_creation_total`
* Labels: `organization_name`, `template_name`, `preset_name`
This counter tracks whether a regular workspace (not created from a
prebuild pool) was created using a preset or not.
Currently, we already expose `coderd_prebuilt_workspaces_claimed_total`
for claimed prebuilt workspaces, but we lack a comparable metric for
regular workspace creations. This metric fills that gap, making it
possible to compare regular creations against claims.
Implementation notes:
* Exposed as a `coderd_` metric, consistent with other workspace-related
metrics (e.g. `coderd_api_workspace_latest_build`:
https://github.com/coder/coder/blob/main/coderd/prometheusmetrics/prometheusmetrics.go#L149).
* Every `defaultRefreshRate` (1 minute ), DB query
`GetRegularWorkspaceCreateMetrics` is executed to fetch all regular
workspaces (not created from a prebuild pool).
* The counter is updated with the total from all time (not just since
metric introduction). This differs from the histograms below, which only
accumulate from their introduction forward.
### `coderd_workspace_creation_duration_seconds` &
`coderd_prebuilt_workspace_claim_duration_seconds`
* Metric types: Histogram
* Names:
* `coderd_workspace_creation_duration_seconds`
* Labels: `organization_name`, `template_name`, `preset_name`, `type`
(`regular`, `prebuild`)
* `coderd_prebuilt_workspace_claim_duration_seconds`
* Labels: `organization_name`, `template_name`, `preset_name`
We already have `coderd_provisionerd_workspace_build_timings_seconds`,
which tracks build run times for all workspace builds handled by the
provisioner daemon.
However, in the context of this issue, we are only interested in
creation and claim build times, not all transitions; additionally, this
metric does not include `preset_name`, and adding it there would
significantly increase cardinality. Therefore, separate more focused
metrics are introduced here:
* `coderd_workspace_creation_duration_seconds`: Build time to create a
workspace (either a regular workspace or the build into a prebuild pool,
for prebuild initial provisioning build).
* `coderd_prebuilt_workspace_claim_duration_seconds`: Time to claim a
prebuilt workspace from the pool.
The reason for two separate histograms is that:
* Creation (regular or prebuild): provisioning builds with similar time
magnitude, generally expected to take longer than a claim operation.
* Claim: expected to be a much faster provisioning build.
#### Native histogram usage
Provisioning times vary widely between projects. Using static buckets
risks unbalanced or poorly informative histograms.
To address this, these metrics use [Prometheus native
histograms](https://prometheus.io/docs/specs/native_histograms/):
* First introduced in Prometheus v2.40.0
* Recommended stable usage from v2.45+
* Requires Go client `prometheus/client_golang` v1.15.0+
* Experimental and must be explicitly enabled on the server
(`--enable-feature=native-histograms`)
For compatibility, we also retain a classic bucket definition (aligned
with the existing provisioner metric:
https://github.com/coder/coder/blob/main/provisionerd/provisionerd.go#L182-L189).
* If native histograms are enabled, Prometheus ingests the
high-resolution histogram.
* If not, it falls back to the predefined buckets.
Implementation notes:
* Unlike the counter, these histograms are updated in real-time at
workspace build job completion.
* They reflect data only from the point of introduction forward (no
historical backfill).
## Relates to
Closes: https://github.com/coder/coder/issues/19528
Native histograms tested in observability stack:
https://github.com/coder/observability/pull/50
## Description
This PR ensures that activity-based deadline extensions ("activity
bumping") are not applied to prebuilt workspaces. Prebuilds are managed
by the reconciliation loop and must not have `deadline` or
`max_deadline` values set or extended, as they are not part of the
regular lifecycle executor path.
## Changes
- Update `ActivityBumpWorkspace` SQL query to discard prebuilt
workspaces
- Update application layer to avoid calling activity bump logic on
prebuilt workspaces
Related with:
* Issue: https://github.com/coder/coder/issues/18898
* PR: https://github.com/coder/coder/pull/19252
we missed these in the previous PR, we find `tickTime2`
and pass it to the `tickCh`, but we were incorrectly passing `tickTime`
to `UpdateProvisionerLastSeenAt` in some places
Signed-off-by: Callum Styan <callumstyan@gmail.com>
## Description
This PR updates the API to prevent lifecycle configuration endpoints
from being used on prebuilt workspaces. Since prebuilds are managed by
the reconciliation loop and do not participate in the regular workspace
lifecycle, they must not support per-workspace overrides for fields like
deadline, TTL, autostart, or dormancy.
Attempting to use these endpoints on a prebuilt workspace will now
return a clear validation error (`409 Conflict`) with an appropriate
explanation. This prevents accidental misconfiguration and preserves the
lifecycle separation between prebuilds and regular workspaces.
## Changes
The following endpoints now return an error if the target workspace is a
prebuild:
* `PUT /workspaces/{workspace}/extend`
* `PUT /workspaces/{workspace}/ttl`
* `PUT /workspaces/{workspace}/autostart`
* `PUT /workspaces/{workspace}/dormant`
Update endpoints logic to use the API clock in order to allow time
mocking in tests.
Related with:
* Issue: https://github.com/coder/coder/issues/18898
* PR: https://github.com/coder/coder/pull/19252
## Description
This PR ensures that prebuilt workspaces are properly excluded from the
lifecycle executor and treated as a separate class of workspaces, fully
managed by the prebuild reconciliation loop.
It introduces two lifecycle guarantees:
* When a prebuilt workspace is created (i.e., when the workspace build
completes), all lifecycle-related fields are unset, ensuring the
workspace does not participate in TTL, autostop, autostart, dormancy, or
auto-deletion logic.
* When a prebuilt workspace is claimed, it transitions into a regular
user workspace. At this point, all lifecycle fields are correctly
populated according to template-level configurations, allowing the
workspace to be managed by the lifecycle executor as expected.
## Changes
* Prebuilt workspaces now have all lifecycle-relevant fields unset
during creation
* When a prebuild is claimed:
* Lifecycle fields are set based on template and workspace level
configurations. This ensures a clean transition into the standard
workspace lifecycle flow.
* Updated lifecycle-related SQL update queries to explicitly exclude
prebuilt workspaces.
## Relates
Related issue: https://github.com/coder/coder/issues/18898
To reduce the scope of this PR and make the review process more
manageable, the original implementation has been split into the
following focused PRs:
* https://github.com/coder/coder/pull/19259
* https://github.com/coder/coder/pull/19263
* https://github.com/coder/coder/pull/19264
* https://github.com/coder/coder/pull/19265
These PRs should be considered in conjunction with this one to
understand the complete set of lifecycle separation changes for prebuilt
workspaces.
Bobby Ho