## TL;DR
**Problem.** A template can declare which external auth provider it
wants via `data "coder_external_auth" { id = "..." }`, and that
declaration is honored at every stage of the build. It was ignored at
runtime. Any git operation going through `GIT_ASKPASS` supplies only a
hostname, never a provider ID, and the handler scanned *every* provider
configured on the deployment and returned whichever matched the hostname
**last in config order**, with no reference to what the requesting
workspace's own template declared. Reordering
`CODER_EXTERNAL_AUTH_<N>_*` silently redirected a plain `git clone` from
one OAuth client's token to a completely different one.
**Fix.** For hostname-only requests, resolve the calling agent's
workspace and build *before* selecting a provider, then narrow
candidates to the providers declared by that build's template version.
Exactly one match wins regardless of config order. No matching declared
provider falls back to today's deployment-wide scan, so a template that
declares only a GitHub provider can still clone an unrelated host. Two
or more matching declared providers return `409` naming them, rather
than picking one arbitrarily: `external_auth_providers` is stored sorted
by ID, so HCL declaration order is already unavailable and no principled
tie-break exists.
Requests supplying an explicit provider ID are untouched. Server-side
only: no wire protocol, proto, manifest, or database schema change, so
already-running agents get the corrected behavior on their next askpass
call with no restart.
Refs #23718
<details>
<summary><b>Call flow</b></summary>
```mermaid
flowchart TD
subgraph Push["1. Template import: coder templates push"]
A1["Terraform extracts coder_external_auth id/optional attrs"]
A2["CompleteJob(TemplateImport) validates each id<br/>against deployment config"]
A4["template_versions.external_auth_providers persisted"]
A1 --> A2 --> A4
end
subgraph PreBuild["2. Pre-build and workspace build (unaffected)"]
B1["User authenticates declared provider(s), exact-ID lookup"]
B2["Build resolves token by exact ID<br/>(provisionerdserver.go)"]
A4 --> B1 --> B2
end
subgraph Runtime["3. Workspace running: a credential is needed"]
B2 --> C0{"Caller supplies id or match?"}
C0 -->|"id (explicit)"| D1["Exact-ID match<br/>UNCHANGED, already deterministic<br/>(coder external-auth access-token)"]
C0 -->|"match only (GIT_ASKPASS)"| C1["git needs credentials for a hostname<br/>GIT_ASKPASS invoked, unchanged"]
C1 --> C2["coder gitaskpass sends ExternalAuthRequest{Match: host}<br/>unchanged (cli/gitaskpass.go)"]
C2 --> C3["workspaceAgentsExternalAuth<br/>(coderd/workspaceagents.go)"]
C3 --> C4["CHANGED:<br/>1. resolve workspace/build BEFORE matching<br/>2. read that build's declared provider IDs<br/>3. filter: declared AND regex matches host"]
C4 --> C5{"how many candidates?"}
C5 -->|"exactly 1"| C6["use it, regardless of config order"]
C5 -->|"0"| C7["fall back to deployment-wide scan<br/>(unchanged legacy behavior)"]
C5 -->|"2 or more"| C8["409 naming every matching ID"]
end
D1 --> E1["Token returned"]
C6 --> E1
C7 --> E1
style C4 fill:#1f4d2e,stroke:#4caf50,color:#fff
style C6 fill:#1f4d2e,stroke:#4caf50,color:#fff
style C8 fill:#1f4d2e,stroke:#4caf50,color:#fff
style D1 fill:#333,stroke:#888,color:#fff
```
</details>
## Verification
Two test functions were added in `coderd/workspaceagents_test.go`, and
the behavior no unit test can reach was verified against a local dev
cluster with two real GitHub OAuth Apps whose regexes both match
`github.com`.
| Behavior | Unit | Manual |
|---|---|---|
| Declared provider wins over a colliding one | yes | yes |
| Outcome independent of deployment config order | yes | yes |
| No declared match falls back to the full scan | yes | yes |
| Host the template never declared still resolves | yes | via fallback |
| Two declared providers matching one host return `409` | yes | not run
|
| Declared but unauthenticated provider returns its auth URL | yes | not
run |
| Two templates resolve independently and concurrently | yes | no |
| Explicit-ID path unaffected | no | yes |
| Running agent corrected with no restart | **no** | **yes** |
| Declared ID since removed from config falls back | **no** | **yes** |
| Recomputed per build after a template update | **no** | **yes** |
The last three are properties a unit test cannot express: they involve
swapping the server binary underneath a live agent, removing deployment
configuration, and rebuilding a workspace against a new template
version.
<details>
<summary><b>Unit test detail</b></summary>
`TestWorkspaceAgentsExternalAuthTemplateScoped` builds a deployment with
two providers sharing a regex, a template declaring one of them, and a
seeded token for **every** provider, so a mis-selection returns a valid
token with the wrong identity rather than an error. Subtests:
- `DeclaredProviderLast` / `DeclaredProviderFirst`: the declared
provider wins in both config orders. Only the `First` arm is
discriminating, since the pre-change loop had no `break` and returned
the last regex match, which the `Last` arm happens to agree with.
- `NoDeclaredProvidersFallsBackToFullScan`: a template declaring nothing
keeps today's behavior exactly, pinning the legacy last-match rule.
- `UnrelatedHostStillResolvesViaFallback`: a template declaring only a
GitHub provider still resolves a GitLab host.
- `AmbiguousDeclaredSetReturnsError`: `409` whose message names both
colliding provider IDs.
- `OptionalUnauthenticatedDeclaredProviderReturnsAuthURL`: returns the
auth URL for the *declared* provider, not for an unrelated one the user
happens to hold a token for.
`TestWorkspaceAgentsExternalAuthMultipleTemplates` runs two workspaces
from two templates, each declaring a different provider, issuing
requests concurrently. Each resolves to its own template's provider.
</details>
<details>
<summary><b>Manual verification detail</b></summary>
Local dev cluster, two GitHub OAuth Apps both defaulting to
`^(https?://)?github\.com(/.*)?$`, both authorized by the workspace
owner so a wrong selection yields a usable token rather than an error.
Workspace built from a template declaring only `github-dotfiles`. Tokens
redacted.
**Order independence.** Same workspace, never rebuilt, config order
reversed between runs:
| Deployment config order | Token returned |
|---|---|
| `[github-broad, github-dotfiles]` | `gho_<dotfiles>` |
| `[github-dotfiles, github-broad]` | `gho_<dotfiles>` |
**A/B against the pre-fix binary.** Everything held constant except the
coderd build, with `/api/v2/buildinfo` checked on both sides so the
comparison rests on verified binary identity. The workspace was never
stopped, rebuilt, or re-authorized:
| coderd | buildinfo | Token | Honors declaration |
|---|---|---|---|
| pre-fix | `v2.35.3-devel+11e03cfb3a` | `gho_<broad>` | no |
| this branch | `v2.35.3-devel+e8b87d0333` | `gho_<dotfiles>` | yes |
This doubles as the demonstration that a coderd-only upgrade corrects
behavior on a live agent's next askpass call.
**Declared provider removed from config.** `github-dotfiles` deleted
from deployment configuration while the workspace's template still
declared it. Result: `HTTP/2 200` with `gho_<broad>` via the fallback.
No `500`, no fail-closed `404`. The orphaned `external_auth_link` row
remained in the database throughout and correctly had no effect.
**Recomputation after a template update.**
| Workspace state | Build's declared provider | Token returned |
|---|---|---|
| new version pushed, workspace not updated | `github-dotfiles` |
`gho_<dotfiles>` |
| after `coder update` | `github-broad` | `gho_<broad>` |
The pair is what makes it conclusive: the first rules out following the
template's newest version, the second rules out a cached value.
**Explicit-ID path.** `coder external-auth access-token github-broad`
returned that provider's result even though the template declared only
`github-dotfiles`, and did not substitute the declared provider's
already-valid token.
Raw traces were captured with `GIT_CURL_VERBOSE=1 git -c
credential.helper="" ls-remote <private repo>`, reading the unredacted
`== Info: Server auth using Basic with user '<token>'` line. A private
repo is required, since a public one never triggers a `401` and
therefore never invokes `GIT_ASKPASS`.
</details>
Previously, \`ExternalAuthResponse\` contained no expiry information, so
workspace agents and git credential helpers had no way to know when a
cached token would stop being valid. Every git operation had to call
back to coderd via \`GIT_ASKPASS\` to get a fresh token, adding 1-2
seconds of latency.
This PR surfaces \`OAuthExpiry\` from the database as \`ExpiresAt\` in
\`ExternalAuthResponse\`, allowing agents to cache tokens with correct
eviction timing (compatible with \`git-credential-cache --timeout\` and
\`password_expiry_utc\` introduced in git 2.34).
\`ExpiresAt\` is normalized to UTC before JSON encoding to avoid
sub-minute precision loss that occurs when the PostgreSQL driver applies
historical Local Mean Time (LMT) timezone offsets to year-1 AD
timestamps.
The \`coder external-auth access-token\` CLI command gains \`--output
json\` to print the full response including \`ExpiresAt\`, enabling
scripts to consume the expiry without parsing heuristics.
Closes https://github.com/coder/coder/issues/26036
## Manual Test
<details>
<summary>Setup</summary>
1. Create a GitHub OAuth app at https://github.com/settings/developers
with:
- Homepage URL: `http://127.0.0.1:3000`
- Authorization callback URL:
`http://127.0.0.1:3000/external-auth/github/callback`
2. Start the dev server with the GitHub provider configured:
```sh
CODER_EXTERNAL_AUTH_0_ID=github CODER_EXTERNAL_AUTH_0_TYPE=github
CODER_EXTERNAL_AUTH_0_CLIENT_ID=<client-id>
CODER_EXTERNAL_AUTH_0_CLIENT_SECRET=<client-secret> ./scripts/develop.sh
```
3. Log in at `http://127.0.0.1:3000` (use `127.0.0.1`, not `localhost`,
so the OAuth state cookie domain matches the callback URL).
4. Go to Account > External Authentication and click **Connect** next to
GitHub. Complete the OAuth flow.
5. Create a workspace and SSH into it:
```sh
coder create test-workspace
coder ssh test-workspace
```
</details>
<details>
<summary>Flow 1: Token is valid — JSON output includes
<code>expires_at</code></summary>
Inside the workspace, run:
```sh
coder external-auth access-token github --output json
echo "Exit code: $?"
```
Expected output (GitHub tokens have no expiry, so \`expires_at\` is the
zero value):
```json
{
"access_token": "<redacted>",
"token_extra": null,
"url": "",
"type": "github",
"expires_at": "0001-01-01T00:00:00Z",
"username": "<redacted>",
"password": ""
}
```
```
Exit code: 0
```
</details>
<details>
<summary>Flow 2: Token missing — JSON output includes auth URL, exit
code 1</summary>
Disconnect GitHub in the Coder UI (Account > External Authentication >
Disconnect), then inside the workspace run:
```sh
coder external-auth access-token github --output json
echo "Exit code: $?"
```
Expected output:
```json
{
"access_token": "",
"token_extra": null,
"url": "http://127.0.0.1:3000/external-auth/github",
"type": "",
"expires_at": "0001-01-01T00:00:00Z",
"username": "",
"password": ""
}
```
```
Exit code: 1
```
</details>
## Problem
#23108 made prebuild claim delivery durable: when an agent connects to
`/api/v2/workspaceagents/me/reinit?wait=true`, the handler checks
whether the workspace's first build was created by the prebuilds system
user and whether its latest build succeeded, and if so pre-seeds a
`prebuild_claimed` reinitialization event in case the original pubsub
event was missed.
The check does not verify that the latest build is the claim build, so
it keeps firing for the rest of the workspace's life. Any workspace that
was claimed from a prebuild receives a spurious "prebuild claimed"
reinit every time its agent (re)opens the `/reinit` connection: after
every agent restart, every coderd deploy or replica restart, and every
dropped SSE connection. Each one shuts the agent down and reinitializes
it, killing SSH/IDE sessions and re-running startup scripts. In our
deployment, where most workspaces are claimed from prebuilds, this
caused fleet-wide "agent disconnected" blips whenever a coderd replica
restarted, and a few workspaces whose container exits when the agent
restarts went into a restart loop every 15-60 minutes. The agent-side
dedup (`lastOwnerID` in `cli/agent.go`) only suppresses the second event
within one agent process, so every new agent process takes at least one
spurious restart.
## Fix
Only seed the reinitialization event while the latest build is the claim
build itself, determined from the build job's input
(`prebuilt_workspace_stage`), the same signal `provisionerdserver` uses
when publishing the claim event:
- Latest build is the claim build: behavior unchanged (seed when the job
succeeded, 409 when it failed permanently, wait on pubsub while it is in
progress).
- Latest build is still a prebuilds-initiated build (claim build not
created yet): fall through to the pubsub subscription, which delivers
the claim event when the claim build completes.
- Latest build is any later user-initiated build: the claim was already
handled, so return 409 and the agent stops polling, the same as a
regular workspace.
`dbfake` gains a `MarkPrebuiltWorkspaceClaim()` builder option so tests
can model claim builds' job input, and the existing `TestReinit` claim
subtests now use it. A new subtest covers the long-claimed workspace
case.
One deliberate behavior change worth calling out: if a claim build fails
and the owner retries with another start build, the handler now returns
409 for that retry build rather than seeding a reinit. This matches the
existing treatment of failed claim builds as terminal for the reinit
poller.
## Verification
- `go test ./coderd/ -run TestReinit` against Postgres 17: all subtests
pass, including the new `workspace claimed in the past gets 409` case.
- `gofmt`, `go vet`, and `golangci-lint` (v1.64.8) are clean on the
touched packages.
- The fix mirrors behavior validated by hand against an affected
deployment: for a long-claimed workspace, `/reinit?wait=true` returned
the seeded `prebuild_claimed` event on every connection before the
change and a 409 afterwards.
Note: this branch was prepared in an environment without the full local
toolchain, so the repo's pre-commit hook (`make pre-commit`) was not run
locally; relying on CI for the full gen/fmt/lint suite. Opening as a
draft mainly to report the issue and propose a fix; happy to rework it
to the maintainers' preferred approach.
---------
Co-authored-by: Sas Swart <sas.swart.cdk@gmail.com>
This PR makes the agent-pushed pinned snapshot
(`chat_context_resources`) the sole source of workspace context for
chats, completing the "Release 5" cleanup. It removes legacy mechanisms
now superseded by the snapshot that agents push over dRPC
(`PushContextState`) and refresh via `chat-context/refresh`.
Removed:
- **Live-read at turn time.** MCP tool discovery, skill live-body reads,
and the instruction/skill history fallback that dialed the workspace on
every turn.
- **Context injected as message history.** The
`persist_workspace_context` generation action and its decision-loop
guard.
- **The legacy write path.** `POST`/`DELETE
/api/v2/workspaceagents/me/experimental/chat-context`, the agentsdk
`AddChatContext`/`ClearChatContext` methods, and the CLI one-shot
writer.
- **The `chats.last_injected_context` column** and all of its plumbing
(migration `000529`, queries, `db2sdk`, `dbauthz`, audit table, and the
frontend `ContextUsageIndicator` fallback).
Subagent context inheritance no longer copies parent context messages;
children now hydrate the parent's pinned `chat_context_resources` on
create, which yields an identical pin for the same workspace and agent.
What stays (still served by the live agent connection, not the
snapshot): `read_skill_file` supporting-file reads, `read_skill`
supporting-file listing, and MCP tool execution.
> [!NOTE]
> Migration `000529` drops `chats.last_injected_context` and recreates
the `chats_expanded` view without it. The down migration restores both.
<details>
<summary>Decision log (D1-D5)</summary>
- **D1 (subagent inheritance):** Re-point inheritance from the legacy
message copy to a pinned hydrate. Children call
`hydrateChatContextOnCreate` instead of copying parent context messages.
- **D2 (`persist_workspace_context`):** Remove the generation action
entirely along with the decision-loop guard it existed to satisfy, since
context is never injected into history anymore.
- **D3 (legacy HTTP + CLI):** Remove the experimental `chat-context`
POST/DELETE endpoints, the agentsdk methods, and the CLI one-shot. The
dRPC push + `chat-context/refresh` replace them.
- **D4 (frontend fallback):** Remove the `last_injected_context`
fallback in `ContextUsageIndicator`; pinned `resources` are the sole
source.
- **D5 (sequencing):** Ship as a single PR rather than a stacked pair.
</details>
---
Coder Agents generated on behalf of @kylecarbs.
Adds the `coder exp chat context` CLI for managing workspace context
sources, plus the agent-token refresh endpoint the in-workspace refresh
relies on. Part of breaking the "Workspace Context Sources for Coder
Agents" RFC (#26466) into small, reviewable PRs.
## What this adds
**CLI (`coder exp chat context`)**, talking to the agent's local IPC
socket from inside the workspace:
- `list` lists the registered scan roots (built-in defaults are not
shown).
- `show <path>` shows a source and the resources the agent resolves from
it, including failures.
- `add <path>` registers a path as an additional context source. With
`--chat`, it keeps the legacy one-shot behavior (read context from the
path once and inject it into a single chat).
- `remove <path>` unregisters a source.
- `refresh [<chat>]` re-pins chat context to the agent's latest
snapshot.
**Agent-token refresh path** for the no-argument `refresh`:
- `refresh <chat>` uses the existing user-facing
`ExperimentalClient.RefreshChatContext` (already on main) and works from
anywhere.
- `refresh` with no argument runs inside the workspace: it re-resolves
the agent's sources over the context socket (catching freshly-cloned
repos and startup-script writes), then asks the agent, authenticating
with its own token, to re-pin every drifted chat. No `coder login`
required.
- This adds `agentsdk.RefreshChatContext` and `POST
/api/v2/workspaceagents/me/experimental/chat-context/refresh`
(`workspaceAgentRefreshChatContext`), mirroring the existing clear
endpoint's agent-token auth model.
## Testing
- `go test ./cli` (`TestExpChatContextAdd`, `TestParseChatID`,
`TestResolveContextSourcePath`)
- `go test ./coderd/x/chatd -run TestChatContextRefreshFromAgentToken`
(end-to-end: echo-provisioned agent pushes a snapshot, drifts a bound
chat, the agent-token refresh re-pins it, and an agent-less chat stays
untouched)
- `go build ./...`, `go vet`, `golangci-lint`, `make gen` (no generated
changes; experimental commands are excluded from CLI golden/doc
generation)
<details>
<summary>Design notes</summary>
This is **Split 4** of #26466. Split sequence:
1. #26558 - prompt pin consumption (merged)
2. #26570 - `codersdk` context resource types (merged)
3. #26573 - the context indicator UI (merged)
4. **This PR** - the CLI + agent-token refresh.
5. The context diff (`changes`, `ChatContextResourceChange`, the changes
dialog, `buildContentPatch`) - last.
Key points:
- The agent-local context subsystem (`agent/agentsocket` IPC for source
CRUD, snapshot, resync), the user-facing
`ExperimentalClient.RefreshChatContext`, and the per-chat
`chatd.RefreshChatContext` all already exist on main, so this split is
the CLI surface plus the small agent-token refresh endpoint that fans
out per-chat refresh across an agent's drifted chats.
- `add <path>` resolves relative paths to absolute before handing them
to the agent (which requires canonical paths) but preserves a leading
`~` for the agent to expand against its own home.
`TestResolveContextSourcePath` covers this.
- The agent endpoint is annotated `@x-apidocgen {"skip": true}`,
matching the other agent-token chat-context endpoints.
- No diff/changes rendering is involved; that lands in the final split.
</details>
*This PR was created by Coder Agents on behalf of @kylecarbs.*
Relates to CODAGT-115
Adds metric `coderd_api_websocket_probes_total`. Every successful
heartbeat for a given path will increment the metric.
Comparing this with `coderd_api_concurrent_websockets` will give an
indication of how many websocket connections are open but in a 'wedged'
state (when heartbeats stopped versus when we closed the connection).
- The httpmw upstream from this endpoint only checks for read perms to the
workspace agent. Recreating a dev container should require `update`
perms since it mutates state. This also matches the behavior of the
`DELETE` endpoint
- Empty string is valid for `apiKeyID` in paths that genuinely lack a
caller key (e.g. agent-initiated context injection in
`workspaceAgentAddChatContext`). AI Gateway fail-closed check remains
the runtime safety net.
- Context injection paths (`persistInstructionFiles`, compaction) read
the key from `aibridge.DelegatedAPIKeyIDFromContext(ctx)`, set upstream
by `contextWithActiveTurnAPIKeyID`.
- Subagent context copy branches on `copiedRole ==
database.ChatMessageRoleUser` to choose the right append function.
> Generated by Coder Agents
`TestWatchAgentContainers/CoderdWebSocketCanHandleClientClosing` spent
about 15 seconds waiting for the real websocket heartbeat ticker to
detect that the client closed.
Add a clock-aware `HeartbeatClose` wrapper and pass `api.Clock` through
the containers watch handler so the test can drive the heartbeat
deterministically with `quartz.Mock`. The test still verifies the same
client-close teardown path, but it advances the heartbeat tick instead
of waiting for wall-clock time.
Refs #25557
Discovered as part of the work on CODAGT-381.
GetChats now returns only root chats (parent_chat_id IS NULL).
A new GetChildChatsByParentIDs query fetches children for visible
roots and embeds them in each parent's Children field. The
singular getChat endpoint does the same.
Archive invariant is one-way: parent archived implies child
archived. Parent archive/unarchive cascades via root_chat_id.
Individual child archive is permitted; child unarchive while the
parent is archived is rejected atomically (row lock on child,
re-read parent inside the transaction). Embedded children are
filtered by the caller's archive state so individually-archived
children stay hidden from active-parent views.
Gitsync MarkStale uses GetChatsByWorkspaceIDs directly;
MarkStaleParams.OwnerID removed (dead after the switch).
Frontend: buildChatTree reads from the embedded children field,
WebSocket handlers route child events into the parent's children
array, and archiving a child strips it from the parent cache.
Adds `coder exp chat context add` and `coder exp chat context clear`
commands that run inside a workspace to manage chat context files via
the agent token.
`add` reads instruction and skill files from a directory (defaulting to
cwd) and inserts them as context-file messages into an active chat.
Multiple calls are additive — `instructionFromContextFiles` already
accumulates all context-file parts across messages.
`clear` soft-deletes all context-file messages, causing
`contextFileAgentID()` to return `!found` on the next turn, which
triggers `needsInstructionPersist=true` and re-fetches defaults from the
agent.
Both commands auto-detect the target chat via `CODER_CHAT_ID` (already
set by `agentproc` on chat-spawned processes), or fall back to
single-active-chat resolution for the agent. The `--chat` flag overrides
both.
Also adds sub-agent context inheritance: `createChildSubagentChat` now
copies parent context-file messages to child chats at spawn time, so
delegated sub-agents share the same instruction context without
independently re-fetching from the workspace agent.
<details><summary>Implementation details</summary>
**New files:**
- `cli/exp_chat.go` — CLI command tree under `coder exp chat context`
**Modified files:**
- `agent/agentcontextconfig/api.go` — `ConfigFromDir()` reads context
from an arbitrary directory without env vars
- `codersdk/agentsdk/agentsdk.go` — `AddChatContext`/`ClearChatContext`
SDK methods
- `coderd/workspaceagents.go` — POST/DELETE handlers on
`/workspaceagents/me/chat-context`
- `coderd/coderd.go` — Route registration
- `coderd/database/queries/chats.sql` — `GetActiveChatsByAgentID`,
`SoftDeleteContextFileMessages`
- `coderd/database/dbauthz/dbauthz.go` — RBAC implementations for new
queries
- `coderd/x/chatd/subagent.go` — `copyParentContextFiles` for sub-agent
inheritance
- `cli/root.go` — Register `chatCommand()` in `AGPLExperimental()`
**Auth pattern:** Uses `AgentAuth` (same as `coder external-auth`) —
agent token via `CODER_AGENT_TOKEN` + `CODER_AGENT_URL` env vars.
</details>
> 🤖 Generated by Coder Agents
---------
Co-authored-by: Michael Suchacz <203725896+ibetitsmike@users.noreply.github.com>
Workspace agent logs could still fail after the earlier invalid UTF-8
fix because NUL bytes are valid Go/protobuf strings but are rejected by
Postgres text columns. The legacy HTTP log upload path also bypassed the
old sanitization entirely, and both server insert paths computed
logs_length from the unsanitized input.
Add a shared log-output sanitizer in agentsdk, use it in the protobuf
conversion path and both server-side insert paths, and compute
OutputLength from the sanitized string so overflow accounting matches
what is actually stored. This keeps the old invalid UTF-8 behavior while
also handling embedded NUL bytes consistently across DRPC and HTTP log
ingestion.
Refs [#23292 ](https://github.com/coder/coder/issues/23292)
Refs [#13433 ](https://github.com/coder/coder/issues/13433)
## Problem
When a prebuilt workspace is claimed, the agent reinitializes via a
single fire-and-forget pubsub event over SSE. If the agent's SSE
connection is interrupted at claim time, the event is permanently lost —
the workspace is stuck with no self-healing path.
Additionally, regular (non-prebuild) workspaces had no way to opt out of
the `/reinit` polling loop — agents would reconnect indefinitely to an
endpoint that would never send them anything useful.
## Root Cause
`workspaceAgentReinit` fetches the workspace (with its current
`owner_id`) via `GetWorkspaceByAgentID`, but never checked whether a
claim already happened. It only subscribed to pubsub for future events.
The database already has durable claim state (`owner_id` changes from
`PrebuildsSystemUserID` to the real user), but no layer ever consulted
it on reconnection.
## Solution
### Server-side durable check with first-build-initiator gating
**TOCTOU-safe ordering**: Subscribe to pubsub claim events *before* any
durable checks, so a claim that fires during the check is buffered in
the channel rather than lost.
**First-build-initiator gating**: When `!workspace.IsPrebuild()` (owner
is no longer the system user), look up the first build's `InitiatorID`.
The prebuild reconciler always uses `PrebuildsSystemUserID` as the
initiator. This distinguishes claimed prebuilds from regular workspaces
without any SQL schema changes.
- **Regular workspace** (first build initiator ≠ system user) → **409
Conflict**, agent stops reconnecting
- **Claimed prebuild, build completed** → pre-seed channel with reinit
event and close it, transmitter delivers one-shot then exits
- **Claimed prebuild, build in-progress** → fall through to pubsub
subscription, agent waits for completion event
- **Unclaimed prebuild** → pubsub subscription (existing happy path)
### Declarative reinit events (defense-in-depth)
- Added `UserID` field to `ReinitializationEvent` with JSON tags
- Switched pubsub serialization from raw string to JSON (with
backward-compat fallback for rolling upgrades)
- Populated `UserID` at both the publish site and the durable check
### Agent SDK: 409 handling
`WaitForReinitLoop` detects 409 Conflict from the server and closes the
`reinitEvents` channel, cleanly exiting the retry goroutine.
### Agent CLI: fixed two bugs + added reinitCtx
- **Closed channel (`!ok`)**: now blocks on `<-ctx.Done()` instead of
`continue`, keeping the current agent running. Previously this would
leak agents by skipping `agnt.Close()` and re-entering the loop.
- **Duplicate owner reinit**: cancels `reinitCtx` (stops the reinit
goroutine), then blocks on `<-ctx.Done()`. Previously `continue` would
skip cleanup and create a new agent on the next loop iteration.
- **`reinitCtx`**: a cancellable child of `ctx` passed to
`WaitForReinitLoop`, allowing the agent to stop the reinit HTTP polling
after reinit completes.
### Agent-side idempotency
Tracks `lastOwnerID` in the agent reinit loop — duplicate events for the
same owner are skipped.
## Testing
- **"unclaimed prebuild receives reinit via pubsub"**: prebuild owned by
system user, pubsub event triggers reinit
- **"claimed prebuild receives one-shot reinit on reconnect"**: first
build by system user, owner changed, build completed → immediate reinit
(no pubsub needed)
- **"claimed prebuild waits during in-progress claim build"**: claimed
but build still running → no reinit until build completes
- **"regular workspace gets 409"**: first build by real user → 409
Conflict, agent stops polling
- Updated claim publisher/listener tests: verify `UserID` survives JSON
round-trip + backward compat with raw string payloads
- Updated SSE round-trip test: verify `UserID` survives transmit →
receive cycle
Fixes#22359
## Rolling upgrade note
During a rolling deploy where old coderd instances coexist with new
ones, the pubsub `ReinitializationEvent` has a new `workspace_id` field
(JSON key `workspace_id`). Old publishers send a raw reason string
instead of JSON; the new listener gracefully falls back by treating the
entire payload as the reason and filling in `WorkspaceID` from context.
The only visible effect during the upgrade window is that `WorkspaceID`
may be the zero UUID in agent-side logs — this is cosmetic and resolves
once all instances are updated.
## Problem
Subagent chats were receiving git context (branch, remote origin, PR
status) from their parent or sibling chats' git operations. When a git
operation triggers external auth, the workspace agent sends `chat_id`
identifying which chat initiated it — but this was broken at two levels:
1. **Agent side:** `CODER_CHAT_ID` was never injected into process
environments. `chatd` sets `Coder-Chat-Id` HTTP headers and the
agent extracts them for process isolation, but never propagated
`CODER_CHAT_ID` to `cmd.Env`. So `gitaskpass` always sent an empty
`chat_id`.
2. **Server side:** `workspaceAgentsExternalAuth` ignored the `chat_id`
query param. `MarkStale` broadcast git context to **all** chats on
the workspace via `filterChatsByWorkspaceID`.
## Fix
- Inject `CODER_CHAT_ID` into `cmd.Env` in `agentproc` when the chat
ID is known, so `gitaskpass` can read and forward it.
- Read `chat_id` from query params in `workspaceAgentsExternalAuth`
and thread it through `chatGitRef`.
- Refactor `MarkStale` to accept a `MarkStaleParams` struct. When
`ChatID` is provided, target only that specific chat. When empty
(legacy agents, non-chat git operations), fall back to the existing
workspace-wide broadcast.
- Extract `markStaleSingle` helper to deduplicate the upsert+publish
logic.
<details><summary>Investigation notes</summary>
### Data flow before fix
```
chatd → sets Coder-Chat-Id header on agent conn
agent → extracts chatID, stores on process struct
agent → does NOT set CODER_CHAT_ID in cmd.Env ← gap 1
gitaskpass → reads CODER_CHAT_ID (always empty), sends chat_id=""
server handler → ignores chat_id query param ← gap 2
MarkStale → broadcasts to ALL workspace chats
```
### Data flow after fix
```
chatd → sets Coder-Chat-Id header on agent conn
agent → extracts chatID, stores on process struct
agent → sets CODER_CHAT_ID in cmd.Env
gitaskpass → reads CODER_CHAT_ID, sends chat_id=<uuid>
server handler → reads chat_id, passes to MarkStale
MarkStale → targets only that specific chat
```
</details>
We currently call GetWorkspaceAgentByID millions of times at scale
unnecessarily. This PR embeds immutable fields into the relevant
services instead of fetching for them every time.
resolves https://github.com/coder/scaletest/issues/84
Confirmed with a 10k scaletest that this changeset takes the query from
10M+ queries down to 39k
- Adds `_API_BASE_URL` to `CODER_EXTERNAL_AUTH_CONFIG_`
- Extracts and refactors existing GitHub PR sync logic to new packages
`coderd/gitsync` and `coderd/externalauth/gitprovider`
- Associated wiring and tests
Created using Opus 4.6
Fixes a flaky test (`TestUserTailnetTelemetry/invalid_header`) caused by
sub-microsecond precision mismatch between `time.Now()` calls on
Windows.
The server used `time.Now()` (nanosecond precision) for `ConnectedAt`
and `DisconnectedAt`, while the test compared against its own
`time.Now()`. On Windows, wall-clock jitter can cause the server
timestamp to appear slightly before the test's `predialTime`.
Switch to `dbtime.Now()` which rounds to microsecond precision (matching
Postgres), consistent with all other timestamps in `workspaceagents.go`.
Relates to: https://github.com/coder/internal/issues/1390
## Problem
When the git askpass flow triggered diff status refreshes, it updated
**every chat** connected to the workspace. This was wasteful and could
cause confusing status updates on unrelated chats.
## Solution
Thread the chat ID through the entire git askpass flow so only the chat
that initiated the git operation gets updated:
1. **`coderd/chatd/chattool/execute.go`** — Sets `CODER_CHAT_ID` env var
on spawned processes (alongside the existing `CODER_CHAT_AGENT`)
2. **`cli/gitaskpass.go`** — Reads `CODER_CHAT_ID` from the environment
and sends it as a `chat_id` query parameter in the `ExternalAuthRequest`
3. **`codersdk/agentsdk/agentsdk.go`** — Adds `ChatID` field to
`ExternalAuthRequest` and encodes it as a query param
4. **`coderd/workspaceagents.go`** — Parses `chat_id` query param and
passes it through to `storeChatGitRef` and
`triggerWorkspaceChatDiffStatusRefresh`
5. **`coderd/chats.go`** — `storeChatGitRef` and
`refreshWorkspaceChatDiffStatuses` now scope updates to just the
initiating chat when a chat ID is provided, falling back to
all-workspace-chats behavior for backwards compatibility (non-chat git
operations)
The listen loop in workspaceAgentsExternalAuthListen compared
OAuthExpiry using == which compares `time.Time` internal struct fields
including the `*time.Location` pointer.
`time.LoadLocation` does not cache the returned `*Location` pointer, so
each lib/pq connection gets a distinct pointer for the same timezone.
When `pq.ParseTimestamp()` applies the connection's location to a parsed
timestamp, the resulting time.Time embeds that connection-specific
pointer. If the `sql.DB` pool hands out different connections for the
two GetExternalAuthLink reads, the identical timestamp produces
`time.Time` values where == returns false despite representing the same
instant. This is intermittent because the pool _usually_ reuses the same
connection for sequential queries.
This change uses `.Equal()` to compare instants regardless of location.
Also makes the test's validation call counter atomic to fix a possible
data race between the HTTP server and test goroutines.
<!--
If you have used AI to produce some or all of this PR, please ensure you
have read our [AI Contribution
guidelines](https://coder.com/docs/about/contributing/AI_CONTRIBUTING)
before submitting.
-->
part of https://github.com/coder/coder/issues/21335
This moves updating app status (used by Tasks) into the workspace agent
API over dRPC. This will allow us to update the status without having to
re-authenticate each time, like we would with an HTTP PATCH request.
Further PRs in this stack will pipe these requests thru from the CLI MCP
server to the agentsock and finally to this dRPC call to coderd.
* Adds support for parameter `format=text` in the following API routes:
* `/api/v2/workspaceagents/:id/logs`
* `/api/v2/workspacebuilds/:id/logs`
* `/api/v2/templateversions/:id/logs`
* `/api/v2/templateversions/:id/dry-run/:id/logs`
* Adds links to view raw logs on the following pages:
* Workspace build page
* Template editor page
* Template version page
* Refactors existing log formatting in `cli/logs.go` to live in `codersdk`.
🤖 Generated with Claude Opus 4.5, reviewed by me.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Relates to https://github.com/coder/coder/pull/21676
* Replaces all existing usages of `httpapi.Heartbeat` with `httpapi.HeartbeatClose`
* Removes `httpapi.HeartbeatClose`
AI agents report status via patchWorkspaceAgentAppStatus, but this wasn't
extending workspace deadlines. This prevented proper task auto-pause behavior,
causing tasks to pause mid-execution when there were no human connections.
Now we call ActivityBumpWorkspace when agents report status, using the same
logic as SSH/IDE connections. We bump when transitioning to or from the working
state.
Closescoder/internal#1251
Relates to https://github.com/coder/internal/issues/1214
The `ExtractWorkspaceAgentParam` middleware ends up making 4 database
queries to follow the chain of `WorkspaceAgent` -> `WorkspaceResource`
-> `ProvisionerJob` -> `WorkspaceBuild` -- but then dropping all that
hard work on the floor. The `api.workspaceAgent` handler that references
this middleware then has to do all of that work again, plus one more
query to get the related `User` so we can get the username. This pattern
is also mirrored in `getDatabaseTerminal` but without the middleware.
This PR:
* Adds a new query `GetWorkspaceAgentAndWorkspaceByID` to fetch all
this information at once to avoid the multiple round-trips,
* Updates the existing usage of `GetWorkspaceAgentByID` to this new
query instead,
* Updates `ExtractWorkspaceAgentParam` to also store the workspace in
the request context
Dalibo: [0.63ms](https://explain.dalibo.com/plan/40bb597f3539gc6c)
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
## Description
This PR fixes an issue where `GetLatestWorkspaceAppStatusesByAppID`
returned an unbounded number of rows for a given app ID, which could
cause performance issues for noisy or long-running AI tasks.
## Impact
This change reduces database query overhead for workspace app status
updates, particularly for busy AI tasks that update their status
frequently. Previously, fetching the latest status would return all
historical statuses, now it returns only the most recent one.
Fixes#20862
---
🤖 This change was written by Claude Sonnet 4.5 Thinking using [mux](https://github.com/coder/mux) and reviewed by a human 🏄🏻♂️
Relates to https://github.com/coder/internal/issues/1098
Currently AgentAPI waits for only 2 seconds worth of identical terminal
screen snapshots before deciding a task has entered a "stable" state. We
interpret this as becoming "idle", resulting in a notification being
triggered. This behavior is not ideal and is ultimately the root cause
of our spammy notifications.
Unfortunately, until we move AgentAPI to either use the Claude Code SDK
(or ACP wrapper around it), we are unable to easily fix the root cause.
This PR instead waits until the agent is ready before it will send state
change notifications. This will at least resolve _some_ of the
complaints about task state notifications being too spammy.
---
🤖 PR was written by Claude Sonnet 4.5 using [Coder
Mux](https://github.com/coder/cmux) and reviewed by a human 👩
Disclaimer: Claude did all of this, reviewed and committed by me.
I find the "task is working" notification straight after creation to be
unnecessary.
Added logic to skip the notification if the first app status is
"working".
## Description
Send a notification to the workspace owner when an AI task’s app state
becomes `Working` or `Idle`.
An AI task is identified by a workspace build with `HasAITask = true`
and `AITaskSidebarAppID` matching the agent app’s ID.
## Changes
* Add `TemplateTaskWorking` notification template.
* Add `TemplateTaskIdle` notification template.
* Add `GetLatestWorkspaceAppStatusesByAppID` SQL query to get the
workspace app statuses ordered by latest first.
* Update `PATCH /workspaceagents/me/app-status` to enqueue:
* `TemplateTaskWorking` when state transitions to `working`
* `TemplateTaskIdle` when state transitions to `idle`
* Notification labels include:
* `task`: task initial prompt
* `workspace`: workspace name
* Notification dedupe: include a minute-bucketed timestamp (UTC
truncated to the minute) in the enqueue data to allow identical content
to resend within the same day (but not more than once per minute).
Closes: https://github.com/coder/coder/issues/19776
When clients disconnected from the /containers/watch endpoint, the WebSocket
connection between coderd and the agent stayed open. This caused heartbeat
traffic every 15s that was incorrectly counted as workspace activity,
extending workspace lifetimes indefinitely.
Now properly cancels the agent connection context when the client disconnects.
Fixes https://github.com/coder/coder/issues/19372
We increase the read limit to 4MiB (we use this limit elsewhere). We
also make sure to stop sending messages when `containersCh` becomes
closed.
This PR replaces the use of the **container** ID with the
**devcontainer** ID. This is a breaking change. This allows rebuilding a
devcontainer when there is no valid container ID.
"Idle" is more accurate than "complete" since:
1. AgentAPI only knows if the screen is active; it has no way of knowing
if the task is complete.
2. The LLM might be done with its current prompt, but that does not mean
the task is complete either (it likely needs refinement).
The "complete" state will be reserved for future definition.
Additionally, in the case where the screen goes idle but the LLM never
reported a status update, we can get an idle icon without a message, and
it looks kinda janky in the UI so if there is no message I display the
state text.
Closes https://github.com/coder/internal/issues/699
This change introduces a refactor of the devcontainers recreation logic
which is now handled asynchronously rather than being request scoped.
The response was consequently changed from "No Content" to "Accepted" to
reflect this.
A new `Status` field was introduced to the devcontainer struct which
replaces `Running` (bool). This reflects that the devcontainer can now
be in various states (starting, running, stopped or errored).
The status field also protects against multiple concurrent recrations,
as long as they are initiated via the API.
Updates #16424
Bobby Ho