## TL;DR
**Problem.** A template can declare which external auth provider it
wants via `data "coder_external_auth" { id = "..." }`, and that
declaration is honored at every stage of the build. It was ignored at
runtime. Any git operation going through `GIT_ASKPASS` supplies only a
hostname, never a provider ID, and the handler scanned *every* provider
configured on the deployment and returned whichever matched the hostname
**last in config order**, with no reference to what the requesting
workspace's own template declared. Reordering
`CODER_EXTERNAL_AUTH_<N>_*` silently redirected a plain `git clone` from
one OAuth client's token to a completely different one.
**Fix.** For hostname-only requests, resolve the calling agent's
workspace and build *before* selecting a provider, then narrow
candidates to the providers declared by that build's template version.
Exactly one match wins regardless of config order. No matching declared
provider falls back to today's deployment-wide scan, so a template that
declares only a GitHub provider can still clone an unrelated host. Two
or more matching declared providers return `409` naming them, rather
than picking one arbitrarily: `external_auth_providers` is stored sorted
by ID, so HCL declaration order is already unavailable and no principled
tie-break exists.
Requests supplying an explicit provider ID are untouched. Server-side
only: no wire protocol, proto, manifest, or database schema change, so
already-running agents get the corrected behavior on their next askpass
call with no restart.
Refs #23718
<details>
<summary><b>Call flow</b></summary>
```mermaid
flowchart TD
subgraph Push["1. Template import: coder templates push"]
A1["Terraform extracts coder_external_auth id/optional attrs"]
A2["CompleteJob(TemplateImport) validates each id<br/>against deployment config"]
A4["template_versions.external_auth_providers persisted"]
A1 --> A2 --> A4
end
subgraph PreBuild["2. Pre-build and workspace build (unaffected)"]
B1["User authenticates declared provider(s), exact-ID lookup"]
B2["Build resolves token by exact ID<br/>(provisionerdserver.go)"]
A4 --> B1 --> B2
end
subgraph Runtime["3. Workspace running: a credential is needed"]
B2 --> C0{"Caller supplies id or match?"}
C0 -->|"id (explicit)"| D1["Exact-ID match<br/>UNCHANGED, already deterministic<br/>(coder external-auth access-token)"]
C0 -->|"match only (GIT_ASKPASS)"| C1["git needs credentials for a hostname<br/>GIT_ASKPASS invoked, unchanged"]
C1 --> C2["coder gitaskpass sends ExternalAuthRequest{Match: host}<br/>unchanged (cli/gitaskpass.go)"]
C2 --> C3["workspaceAgentsExternalAuth<br/>(coderd/workspaceagents.go)"]
C3 --> C4["CHANGED:<br/>1. resolve workspace/build BEFORE matching<br/>2. read that build's declared provider IDs<br/>3. filter: declared AND regex matches host"]
C4 --> C5{"how many candidates?"}
C5 -->|"exactly 1"| C6["use it, regardless of config order"]
C5 -->|"0"| C7["fall back to deployment-wide scan<br/>(unchanged legacy behavior)"]
C5 -->|"2 or more"| C8["409 naming every matching ID"]
end
D1 --> E1["Token returned"]
C6 --> E1
C7 --> E1
style C4 fill:#1f4d2e,stroke:#4caf50,color:#fff
style C6 fill:#1f4d2e,stroke:#4caf50,color:#fff
style C8 fill:#1f4d2e,stroke:#4caf50,color:#fff
style D1 fill:#333,stroke:#888,color:#fff
```
</details>
## Verification
Two test functions were added in `coderd/workspaceagents_test.go`, and
the behavior no unit test can reach was verified against a local dev
cluster with two real GitHub OAuth Apps whose regexes both match
`github.com`.
| Behavior | Unit | Manual |
|---|---|---|
| Declared provider wins over a colliding one | yes | yes |
| Outcome independent of deployment config order | yes | yes |
| No declared match falls back to the full scan | yes | yes |
| Host the template never declared still resolves | yes | via fallback |
| Two declared providers matching one host return `409` | yes | not run
|
| Declared but unauthenticated provider returns its auth URL | yes | not
run |
| Two templates resolve independently and concurrently | yes | no |
| Explicit-ID path unaffected | no | yes |
| Running agent corrected with no restart | **no** | **yes** |
| Declared ID since removed from config falls back | **no** | **yes** |
| Recomputed per build after a template update | **no** | **yes** |
The last three are properties a unit test cannot express: they involve
swapping the server binary underneath a live agent, removing deployment
configuration, and rebuilding a workspace against a new template
version.
<details>
<summary><b>Unit test detail</b></summary>
`TestWorkspaceAgentsExternalAuthTemplateScoped` builds a deployment with
two providers sharing a regex, a template declaring one of them, and a
seeded token for **every** provider, so a mis-selection returns a valid
token with the wrong identity rather than an error. Subtests:
- `DeclaredProviderLast` / `DeclaredProviderFirst`: the declared
provider wins in both config orders. Only the `First` arm is
discriminating, since the pre-change loop had no `break` and returned
the last regex match, which the `Last` arm happens to agree with.
- `NoDeclaredProvidersFallsBackToFullScan`: a template declaring nothing
keeps today's behavior exactly, pinning the legacy last-match rule.
- `UnrelatedHostStillResolvesViaFallback`: a template declaring only a
GitHub provider still resolves a GitLab host.
- `AmbiguousDeclaredSetReturnsError`: `409` whose message names both
colliding provider IDs.
- `OptionalUnauthenticatedDeclaredProviderReturnsAuthURL`: returns the
auth URL for the *declared* provider, not for an unrelated one the user
happens to hold a token for.
`TestWorkspaceAgentsExternalAuthMultipleTemplates` runs two workspaces
from two templates, each declaring a different provider, issuing
requests concurrently. Each resolves to its own template's provider.
</details>
<details>
<summary><b>Manual verification detail</b></summary>
Local dev cluster, two GitHub OAuth Apps both defaulting to
`^(https?://)?github\.com(/.*)?$`, both authorized by the workspace
owner so a wrong selection yields a usable token rather than an error.
Workspace built from a template declaring only `github-dotfiles`. Tokens
redacted.
**Order independence.** Same workspace, never rebuilt, config order
reversed between runs:
| Deployment config order | Token returned |
|---|---|
| `[github-broad, github-dotfiles]` | `gho_<dotfiles>` |
| `[github-dotfiles, github-broad]` | `gho_<dotfiles>` |
**A/B against the pre-fix binary.** Everything held constant except the
coderd build, with `/api/v2/buildinfo` checked on both sides so the
comparison rests on verified binary identity. The workspace was never
stopped, rebuilt, or re-authorized:
| coderd | buildinfo | Token | Honors declaration |
|---|---|---|---|
| pre-fix | `v2.35.3-devel+11e03cfb3a` | `gho_<broad>` | no |
| this branch | `v2.35.3-devel+e8b87d0333` | `gho_<dotfiles>` | yes |
This doubles as the demonstration that a coderd-only upgrade corrects
behavior on a live agent's next askpass call.
**Declared provider removed from config.** `github-dotfiles` deleted
from deployment configuration while the workspace's template still
declared it. Result: `HTTP/2 200` with `gho_<broad>` via the fallback.
No `500`, no fail-closed `404`. The orphaned `external_auth_link` row
remained in the database throughout and correctly had no effect.
**Recomputation after a template update.**
| Workspace state | Build's declared provider | Token returned |
|---|---|---|
| new version pushed, workspace not updated | `github-dotfiles` |
`gho_<dotfiles>` |
| after `coder update` | `github-broad` | `gho_<broad>` |
The pair is what makes it conclusive: the first rules out following the
template's newest version, the second rules out a cached value.
**Explicit-ID path.** `coder external-auth access-token github-broad`
returned that provider's result even though the template declared only
`github-dotfiles`, and did not substitute the declared provider's
already-valid token.
Raw traces were captured with `GIT_CURL_VERBOSE=1 git -c
credential.helper="" ls-remote <private repo>`, reading the unredacted
`== Info: Server auth using Basic with user '<token>'` line. A private
repo is required, since a public one never triggers a `401` and
therefore never invokes `GIT_ASKPASS`.
</details>
This can cause bad refresh token errors, since it can only be used once.
Looks like there was an attempt to fix this by checking the database
after a failed refresh, but of course this depends on the first request
having updated the database in time, so both that and this fix are
required to fully solve.
Previously, \`ExternalAuthResponse\` contained no expiry information, so
workspace agents and git credential helpers had no way to know when a
cached token would stop being valid. Every git operation had to call
back to coderd via \`GIT_ASKPASS\` to get a fresh token, adding 1-2
seconds of latency.
This PR surfaces \`OAuthExpiry\` from the database as \`ExpiresAt\` in
\`ExternalAuthResponse\`, allowing agents to cache tokens with correct
eviction timing (compatible with \`git-credential-cache --timeout\` and
\`password_expiry_utc\` introduced in git 2.34).
\`ExpiresAt\` is normalized to UTC before JSON encoding to avoid
sub-minute precision loss that occurs when the PostgreSQL driver applies
historical Local Mean Time (LMT) timezone offsets to year-1 AD
timestamps.
The \`coder external-auth access-token\` CLI command gains \`--output
json\` to print the full response including \`ExpiresAt\`, enabling
scripts to consume the expiry without parsing heuristics.
Closes https://github.com/coder/coder/issues/26036
## Manual Test
<details>
<summary>Setup</summary>
1. Create a GitHub OAuth app at https://github.com/settings/developers
with:
- Homepage URL: `http://127.0.0.1:3000`
- Authorization callback URL:
`http://127.0.0.1:3000/external-auth/github/callback`
2. Start the dev server with the GitHub provider configured:
```sh
CODER_EXTERNAL_AUTH_0_ID=github CODER_EXTERNAL_AUTH_0_TYPE=github
CODER_EXTERNAL_AUTH_0_CLIENT_ID=<client-id>
CODER_EXTERNAL_AUTH_0_CLIENT_SECRET=<client-secret> ./scripts/develop.sh
```
3. Log in at `http://127.0.0.1:3000` (use `127.0.0.1`, not `localhost`,
so the OAuth state cookie domain matches the callback URL).
4. Go to Account > External Authentication and click **Connect** next to
GitHub. Complete the OAuth flow.
5. Create a workspace and SSH into it:
```sh
coder create test-workspace
coder ssh test-workspace
```
</details>
<details>
<summary>Flow 1: Token is valid — JSON output includes
<code>expires_at</code></summary>
Inside the workspace, run:
```sh
coder external-auth access-token github --output json
echo "Exit code: $?"
```
Expected output (GitHub tokens have no expiry, so \`expires_at\` is the
zero value):
```json
{
"access_token": "<redacted>",
"token_extra": null,
"url": "",
"type": "github",
"expires_at": "0001-01-01T00:00:00Z",
"username": "<redacted>",
"password": ""
}
```
```
Exit code: 0
```
</details>
<details>
<summary>Flow 2: Token missing — JSON output includes auth URL, exit
code 1</summary>
Disconnect GitHub in the Coder UI (Account > External Authentication >
Disconnect), then inside the workspace run:
```sh
coder external-auth access-token github --output json
echo "Exit code: $?"
```
Expected output:
```json
{
"access_token": "",
"token_extra": null,
"url": "http://127.0.0.1:3000/external-auth/github",
"type": "",
"expires_at": "0001-01-01T00:00:00Z",
"username": "",
"password": ""
}
```
```
Exit code: 1
```
</details>
## Problem
#23108 made prebuild claim delivery durable: when an agent connects to
`/api/v2/workspaceagents/me/reinit?wait=true`, the handler checks
whether the workspace's first build was created by the prebuilds system
user and whether its latest build succeeded, and if so pre-seeds a
`prebuild_claimed` reinitialization event in case the original pubsub
event was missed.
The check does not verify that the latest build is the claim build, so
it keeps firing for the rest of the workspace's life. Any workspace that
was claimed from a prebuild receives a spurious "prebuild claimed"
reinit every time its agent (re)opens the `/reinit` connection: after
every agent restart, every coderd deploy or replica restart, and every
dropped SSE connection. Each one shuts the agent down and reinitializes
it, killing SSH/IDE sessions and re-running startup scripts. In our
deployment, where most workspaces are claimed from prebuilds, this
caused fleet-wide "agent disconnected" blips whenever a coderd replica
restarted, and a few workspaces whose container exits when the agent
restarts went into a restart loop every 15-60 minutes. The agent-side
dedup (`lastOwnerID` in `cli/agent.go`) only suppresses the second event
within one agent process, so every new agent process takes at least one
spurious restart.
## Fix
Only seed the reinitialization event while the latest build is the claim
build itself, determined from the build job's input
(`prebuilt_workspace_stage`), the same signal `provisionerdserver` uses
when publishing the claim event:
- Latest build is the claim build: behavior unchanged (seed when the job
succeeded, 409 when it failed permanently, wait on pubsub while it is in
progress).
- Latest build is still a prebuilds-initiated build (claim build not
created yet): fall through to the pubsub subscription, which delivers
the claim event when the claim build completes.
- Latest build is any later user-initiated build: the claim was already
handled, so return 409 and the agent stops polling, the same as a
regular workspace.
`dbfake` gains a `MarkPrebuiltWorkspaceClaim()` builder option so tests
can model claim builds' job input, and the existing `TestReinit` claim
subtests now use it. A new subtest covers the long-claimed workspace
case.
One deliberate behavior change worth calling out: if a claim build fails
and the owner retries with another start build, the handler now returns
409 for that retry build rather than seeding a reinit. This matches the
existing treatment of failed claim builds as terminal for the reinit
poller.
## Verification
- `go test ./coderd/ -run TestReinit` against Postgres 17: all subtests
pass, including the new `workspace claimed in the past gets 409` case.
- `gofmt`, `go vet`, and `golangci-lint` (v1.64.8) are clean on the
touched packages.
- The fix mirrors behavior validated by hand against an affected
deployment: for a long-claimed workspace, `/reinit?wait=true` returned
the seeded `prebuild_claimed` event on every connection before the
change and a 409 afterwards.
Note: this branch was prepared in an environment without the full local
toolchain, so the repo's pre-commit hook (`make pre-commit`) was not run
locally; relying on CI for the full gen/fmt/lint suite. Opening as a
draft mainly to report the issue and propose a fix; happy to rework it
to the maintainers' preferred approach.
---------
Co-authored-by: Sas Swart <sas.swart.cdk@gmail.com>
Replaces the v2.10 `PushContextState` stub with a real coderd write
path. Phase 1 of the chat-side persistence story; nothing reads these
rows yet.
Follows [#25983](https://github.com/coder/coder/pull/25983) and unblocks
[CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd).
## What ships
### Schema (`000517_workspace_agent_context.{up,down}.sql`)
Two new tables plus `api_key_scope` enum extensions:
- `workspace_agent_context_snapshots` (PK `workspace_agent_id` to
`workspace_agents(id) ON DELETE CASCADE`): one row per agent,
overwritten per push. Holds `version`, `schema_version`,
`aggregate_hash`, `snapshot_error`, `received_at`.
- `workspace_agent_context_resources` (PK `(workspace_agent_id,
source)`): per-resource state. `body_kind` and `status` are `TEXT` +
`CHECK` so adding new wire kinds (the RFC's reserved
PLUGIN/HOOK/SUBAGENT/COMMAND) is a one-line CHECK update plus a Go
switch case.
### SQLC queries (`coderd/database/queries/workspaceagentcontext.sql`)
- `UpsertWorkspaceAgentContextSnapshot`
- `UpsertWorkspaceAgentContextResource`
- `DeleteStaleWorkspaceAgentContextResources`
(delete-where-source-not-in)
- `GetLatestWorkspaceAgentContextSnapshot`
- `ListWorkspaceAgentContextResources`
### Handler (`coderd/agentapi/context.go`)
`ContextAPI` is a new sub-API. `PushContextState`:
1. Rejects `schema_version > 1` with a non-`Unimplemented` error so a
forward-incompatible agent fails loudly during rollout instead of
slipping into the permanent fallback path the `Unimplemented`
translation reserves for old coderd deployments.
2. Validates resources: no empty/duplicate sources, every variant maps
to a known body kind, every status maps to a known enum value, the
`Body` oneof is set (even when status is non-OK, mirroring the wire
guarantee so coderd can attribute failures to a known kind).
3. Inside `Database.InTx`, reads the existing snapshot. If the push is
not `initial` and `version` is not strictly greater, returns `accepted =
false` and leaves stored state untouched. Otherwise upserts the snapshot
row, upserts each resource, then runs the stale-source prune so the
snapshot and resource rows always agree.
4. Returns `accepted = true` on success.
Resource bodies are stored as `protojson(body oneof variant)` in `body
JSONB` with `body_kind` as the discriminator. Adding a new field to an
existing variant is zero work since `protojson` tolerates new fields;
adding a new variant is a CHECK + switch case.
### RBAC + dbauthz
- New `ResourceWorkspaceAgentContext` (Create/Read/Update/Delete).
- New `SubjectTypeAgentContext` plus `subjectAgentContext` system role
and `dbauthz.AsAgentContext` helper. The push handler elevates to this
subject; the agent's own role does not get direct write access to the
table.
- New `workspace_agent_context:*` API key scopes registered in the enum
migration; internal-only (not added to `externalLowLevel`).
### Audit
These rows are agent-pushed state, not user-authored. They are
intentionally not added to `AuditActionMap` and not enumerated in
`enterprise/audit/table.go`, matching `boundary_logs`,
`workspace_agent_memory_resource_monitor`, etc. `enterprise/audit` tests
pass unchanged.
## Tests
- `coderd/agentapi/context_test.go`: 12 subtests covering
accepts/rejects (schema version, empty/duplicate source, unknown status,
missing body), version semantics (stale dropped, same-version replay
dropped, `initial=true` overwrites lower version), variant coverage,
non-OK status persistence, and the empty-active-set prune case.
- `coderd/database/dbauthz/dbauthz_test.go`: 5 `MethodTestSuite` cases
covering the new queries.
- `coderd/rbac/roles_test.go`: `WorkspaceAgentContext` permission row
asserting no human role currently has access.
-
`coderd/database/migrations/testdata/fixtures/000517_workspace_agent_context.up.sql`:
one snapshot + one resource per known body kind plus a non-OK status, so
the migration test suite never lands with these tables empty.
## Out of scope (later phases)
- Chat hydration (`chats.context_aggregate_hash`,
`last_injected_context`).
- Dirty-bit fan-out and `PUT /chats/{id}/context`.
- Agent-side `POST /api/v0/context/resync` barrier and the `coder exp
chat context` CLI.
- `codersdk` chat-context wire types and the dashboard Sources drawer.
- Removal of the chatd per-turn pull fallback.
## Compat property
This is a pure write path. If anything here returns errors the agent's
`RunPush` loop backs off, no chat behavior changes, and the workspace
keeps behaving exactly like it did before v2.10.
<details>
<summary>Implementation plan and decision log</summary>
Key design calls:
1. **Concurrency**: Accept iff `req.Initial || req.Version >
existing.Version`. The strict RFC reading ("version comparison is
authoritative") locks restarted agents out because their per-process
counter resets to 1; honoring `initial=true` reflects the real reboot
reality while still rejecting steady-state replays/out-of-order pushes.
2. **Body encoding**: `protojson` over the oneof variant body proto,
stored in JSONB with `body_kind` discriminator. Structured at the API/Go
layer, schema-tolerant at the storage layer, and Phase 2 readers
round-trip back via `protojson.Unmarshal`.
3. **Schema version rejection**: returns a normal error, not
`Unimplemented`. The agent's `RunPush` loop only short-circuits on
`Unimplemented`; that escape hatch is reserved for old coderd
deployments. A forward-incompatible agent should retry-and-back-off, not
flip the connection into permanent fallback.
4. **Validation strictness**: empty sources, duplicate sources,
`STATUS_UNSPECIFIED`, and missing `Body` oneof variants are rejected
before any write so a misbehaving agent cannot poison the snapshot
table. Phase 2 readers can trust every row maps to a known proto
variant.
</details>
_This PR was authored by Coder Agents on Kyle Carberry's behalf._
Adds the agent half of the workspace context sources RFC. The agent now
resolves instruction files, skills, and MCP configs into a typed
`Snapshot`, watches the relevant paths recursively, exposes the source
list over a workspace-agent HTTP API, and pushes each `Snapshot` to
coderd over a new `PushContextState` RPC on Agent API v2.10.
The coderd-side handler is a stub returning `Unimplemented` for now.
Real persistence to `workspace_agent_context`, chatd hydration on dirty
events, and the `KindMCPServer` MCP provider are tracked by
[CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd).
This matches the pattern used for v2.7 `ReportBoundaryLogs` in
[#21293](https://github.com/coder/coder/pull/21293), which bumped the
version and shipped a stub server so the wire and client could iterate
before the persistence layer landed.
## What ships
### agent/agentcontext (new package)
- `Source`, `Resource` (kinds `instruction_file`, `skill`, `mcp_config`,
`mcp_server` plus reserved `plugin`/`hook`/`subagent`/`command`),
`ResourceStatus`, `Snapshot`, `ComputeAggregateHash`.
- `Manager` owns the in-memory source list, performs the initial resolve
synchronously in `NewManager`, runs a re-resolve/watcher loop in `Run`,
exposes
`AddSource`/`RemoveSource`/`Sources`/`HasSource`/`Snapshot`/`SubscribeChanges`/`Resync`/`SeedSources`/`Close`.
- `Resolver` walks scan roots, classifies recognized files, enforces 64
KiB per-resource, 2 MiB aggregate, and 500-resource caps with
`StatusOversize`/`StatusExcluded`/`StatusUnreadable`/`StatusInvalid`
outcomes, skips `node_modules`/`vendor`/etc., validates symlink targets
stay inside the scan root, stamps `SourcePath` on user-derived
resources, and optionally pulls MCP server tool lists via an
`MCPProvider` interface. MCP config resources ship metadata only (size,
hash) so secrets in env blocks never leave the agent.
- `Watcher` is a recursive `fsnotify` wrapper with a 250 ms debounce,
dynamic arming of newly created directories, and an ENOSPC-tolerant
degraded mode that no-ops further syncs until the manager resyncs
explicitly.
- HTTP API for `GET/POST /sources`, `GET/DELETE /sources/{path}`, `POST
/resync` mounted at `/api/v0/context`.
- `Pusher` interface plus `RunPush` goroutine with exponential backoff
capped at 30 s. `DRPCPusher` adapts the generated `DRPCAgentClient210`
to `Pusher` and translates `drpcerr.Unimplemented` to
`ErrPushUnimplemented` so the push loop exits cleanly when talking to
coderd deployments that have not enabled the real handler.
### agent/proto (v2.10)
- New messages `ContextResource`, `PushContextStateRequest`,
`PushContextStateResponse` and the `PushContextState` RPC on `service
Agent`.
- Generated `DRPCAgentClient210` interface and
`codersdk/agentsdk.Client.ConnectRPC210` / `ConnectRPC210WithRole`.
- `tailnet/proto.CurrentMinor` bumped from `9` to `10`.
### Agent wiring
- `agent.Options.Client` declares both v2.9 and v2.10 connectors;
`run()` dials with `ConnectRPC210WithRole`.
- `apiConnRoutineManager` holds a `DRPCAgentClient210`. Existing v2.8
routines keep their narrower `DRPCAgentClient28` signature thanks to
interface embedding.
- `startAgentAPI210` is the v2.10 counterpart to `startAgentAPI` for
routines that need the new client. The push context state routine uses
it.
- A `contextManager` is constructed in `agent.init()`, seeded from the
existing `CODER_AGENT_EXP_*_DIRS` env vars, started in its own goroutine
under `gracefulCtx`, and closed in `agent.Close`.
- `handleManifest` calls `Manager.SeedSources` for sources rooted at the
manifest directory, then `Resync` after `manifest.Swap`, so the snapshot
reflects the workspace working directory immediately instead of waiting
for the next filesystem event.
- HTTP routes mounted at `/api/v0/context` when the manager is up.
### Coderd stub
`coderd/agentapi/context.go` returns `drpcerr.Unimplemented` for
`PushContextState`. The real handler that persists
`workspace_agent_context` rows, hydrates chats, and emits dirty events
lives in CODAGT-569.
## Tests
24 tests across `agent/agentcontext` cover types, paths, resolver
behavior with file caps, skill containers, MCP secret omission, symlink
target validation, the recursive watcher firing on real fsnotify events,
manager source CRUD / `Resync` / `SeedSources` / `Run` lifetime, the
HTTP API, the DRPC adapter, and the push retry / initial-flag /
unimplemented paths. Passes `go test -race -count=2`.
`TestAgent_ContextStatePushed` boots a full agent against
`agenttest.FakeAgentAPI` (which now records `PushContextState` traffic)
and asserts the seeded `AGENTS.md` appears in a snapshot push with
`schema_version = 1`.
<details>
<summary>Notes for reviewers</summary>
- Source CRUD is workspace-agent-token only; coderd is not in the path
for source mutation.
- Per-resource cap 64 KiB, aggregate 2 MiB, count cap 500; resources
past the cap ship with `StatusExcluded` and an empty payload so the
aggregate hash still detects content edits. MCP-emitted resources
enforce both a per-provider count cap and the aggregate byte cap.
- Symlinks inside the scan root are followed; symlinks pointing outside
(or broken) are rejected with `StatusExcluded` so credentials reachable
via a stray symlink stay off the wire.
- The initial push gates `lifecycle = ready` in the eventual full
design. For this PR the `SeedSources` plus `handleManifest`-driven
`Resync` keeps the snapshot fresh; the live push loop ships now and
DRPCPusher translates the coderd `Unimplemented` stub into a clean exit.
- The `PLUGIN`/`HOOK`/`SUBAGENT`/`COMMAND` kinds are reserved in proto
and Go enums but unused; the Claude Code plugin resolver ships in a
follow-up that does not need a schema migration.
- Two follow-ups remain, both tracked by
[CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd):
(1) the chatd-side handler that persists snapshots and dirties chats;
(2) the `coder exp chat context` CLI command set for
`list`/`show`/`add`/`remove`/`refresh`.
</details>
_This PR was authored by Coder Agents on Kyle Carberry's behalf._
- The httpmw upstream from this endpoint only checks for read perms to the
workspace agent. Recreating a dev container should require `update`
perms since it mutates state. This also matches the behavior of the
`DELETE` endpoint
Closes https://github.com/coder/coder/issues/13112
**Breaking Change**: Removed status code `StatusNotModified` when no
diffs occur in a patch. Now the patch is always applied and a template
is always returned.
1 of 9 [next >>](https://github.com/coder/coder/pull/24811)
RFC: [Bridge ↔ Boundaries Correlation
RFC](https://www.notion.so/Bridge-Boundaries-Correlation-313d579be59281f3b4efdbfd6896775a)
Adds three new proto fields for boundary session correlation.
**`ReportBoundaryLogsRequest`**
- `session_id` (string, field 2) — UUID generated by boundary at
startup,
shared across all batches from a single run.
- `confined_process` (string, field 3) — name of the confined process
(e.g. `claude-code`, `codex`, `copilot`).
**`BoundaryLog`**
- `sequence_number` (uint64, field 4) — monotonically increasing counter
per session, primary ordering key when boundary is in use.
`BoundaryLog.time` already existed at field 2; no change needed there.
API version bumped to v2.9.
No behaviour change in coderd or the agent. This is a pure schema bump
that the boundary repo will consume in its own stack.
> Generated by Coder Agents
## Problem
Workspaces showed as "Healthy" immediately after creation while the
agent was still downloading, starting, or connecting. If the agent never
connected, the workspace stayed "Healthy" for the entire connection
timeout (~120s), then abruptly flipped to "Unhealthy".
## Root cause
In `db2sdk.WorkspaceAgent`, the health switch had no case for
`WorkspaceAgentConnecting`. Agents in `connecting` status with a
non-`off` lifecycle (e.g. `created` after a fresh build) fell through to
the `default` case and were marked `Healthy = true`.
## Fix
Add an explicit case for `WorkspaceAgentConnecting` that sets `Healthy =
false` with reason `"agent has not yet connected"`. The case is placed
after the existing `!connected + off` case (which correctly catches
stopped agents as "not running") and before the `timeout`/`disconnected`
cases.
```
Status + Lifecycle → Health reason
──────────────────────────────────────────────────────
any !connected + off → "agent is not running"
connecting + created/starting → "agent has not yet connected" ← NEW
timeout + any → "agent is taking too long to connect"
disconnected + any → "agent has lost connection"
connected + start_error → "agent startup script exited with an error"
connected + shutting_down → "agent is shutting down"
connected + ready/starting → healthy
```
The frontend already handles this case — `getAgentHealthIssue()` returns
"Workspace agent is still connecting" with `severity: "info"` for
unhealthy workspaces with connecting agents.
## Test changes
- **Healthy test**: now actually connects the agent via `agenttest.New`
before asserting health (previously passed due to the bug).
- **New Connecting test**: verifies a never-connected agent is correctly
marked unhealthy.
- **Mixed health test**: connects a1 and waits for the mixed state
(`a1.Healthy && !workspace.Healthy`) to avoid a race where both agents
are initially connecting.
- **Sub-agent excluded test**: connects the parent agent and waits for
it to be healthy before creating the sub-agent.
- **TestWorkspaceAgent/Connect**: flipped assertion to `Health.Healthy
== false` for a `dbfake` agent that never connects.
<details>
<summary>Review notes</summary>
### Known follow-up
The `healthy:false` workspace search filter maps to `[disconnected,
timeout]` and does not include `connecting`. This is a pre-existing gap
that is now more consequential — a workspace unhealthy solely due to a
connecting agent won't appear in `healthy:false` results. Worth a
follow-up issue.
### Deep review findings addressed
| Finding | Severity | Status |
|---------|----------|--------|
| Mixed health test race (all 3 reviewers) | P2 | Fixed — tightened
`Eventually` condition |
| `TestWorkspaceAgent/Connect` assertion break | P1 | Fixed — flipped
assertion |
| CLI renders red for connecting agents | Obs | Acknowledged — design
trade-off, accurate but visually strong for transient state |
| Switch case ordering overlap | Obs | Documented with inline comment |
</details>
> 🤖 This PR was created with the help of Coder Agents, and needs a human
review. 🧑💻
Workspace agent logs could still fail after the earlier invalid UTF-8
fix because NUL bytes are valid Go/protobuf strings but are rejected by
Postgres text columns. The legacy HTTP log upload path also bypassed the
old sanitization entirely, and both server insert paths computed
logs_length from the unsanitized input.
Add a shared log-output sanitizer in agentsdk, use it in the protobuf
conversion path and both server-side insert paths, and compute
OutputLength from the sanitized string so overflow accounting matches
what is actually stored. This keeps the old invalid UTF-8 behavior while
also handling embedded NUL bytes consistently across DRPC and HTTP log
ingestion.
Refs [#23292 ](https://github.com/coder/coder/issues/23292)
Refs [#13433 ](https://github.com/coder/coder/issues/13433)
## Problem
When a prebuilt workspace is claimed, the agent reinitializes via a
single fire-and-forget pubsub event over SSE. If the agent's SSE
connection is interrupted at claim time, the event is permanently lost —
the workspace is stuck with no self-healing path.
Additionally, regular (non-prebuild) workspaces had no way to opt out of
the `/reinit` polling loop — agents would reconnect indefinitely to an
endpoint that would never send them anything useful.
## Root Cause
`workspaceAgentReinit` fetches the workspace (with its current
`owner_id`) via `GetWorkspaceByAgentID`, but never checked whether a
claim already happened. It only subscribed to pubsub for future events.
The database already has durable claim state (`owner_id` changes from
`PrebuildsSystemUserID` to the real user), but no layer ever consulted
it on reconnection.
## Solution
### Server-side durable check with first-build-initiator gating
**TOCTOU-safe ordering**: Subscribe to pubsub claim events *before* any
durable checks, so a claim that fires during the check is buffered in
the channel rather than lost.
**First-build-initiator gating**: When `!workspace.IsPrebuild()` (owner
is no longer the system user), look up the first build's `InitiatorID`.
The prebuild reconciler always uses `PrebuildsSystemUserID` as the
initiator. This distinguishes claimed prebuilds from regular workspaces
without any SQL schema changes.
- **Regular workspace** (first build initiator ≠ system user) → **409
Conflict**, agent stops reconnecting
- **Claimed prebuild, build completed** → pre-seed channel with reinit
event and close it, transmitter delivers one-shot then exits
- **Claimed prebuild, build in-progress** → fall through to pubsub
subscription, agent waits for completion event
- **Unclaimed prebuild** → pubsub subscription (existing happy path)
### Declarative reinit events (defense-in-depth)
- Added `UserID` field to `ReinitializationEvent` with JSON tags
- Switched pubsub serialization from raw string to JSON (with
backward-compat fallback for rolling upgrades)
- Populated `UserID` at both the publish site and the durable check
### Agent SDK: 409 handling
`WaitForReinitLoop` detects 409 Conflict from the server and closes the
`reinitEvents` channel, cleanly exiting the retry goroutine.
### Agent CLI: fixed two bugs + added reinitCtx
- **Closed channel (`!ok`)**: now blocks on `<-ctx.Done()` instead of
`continue`, keeping the current agent running. Previously this would
leak agents by skipping `agnt.Close()` and re-entering the loop.
- **Duplicate owner reinit**: cancels `reinitCtx` (stops the reinit
goroutine), then blocks on `<-ctx.Done()`. Previously `continue` would
skip cleanup and create a new agent on the next loop iteration.
- **`reinitCtx`**: a cancellable child of `ctx` passed to
`WaitForReinitLoop`, allowing the agent to stop the reinit HTTP polling
after reinit completes.
### Agent-side idempotency
Tracks `lastOwnerID` in the agent reinit loop — duplicate events for the
same owner are skipped.
## Testing
- **"unclaimed prebuild receives reinit via pubsub"**: prebuild owned by
system user, pubsub event triggers reinit
- **"claimed prebuild receives one-shot reinit on reconnect"**: first
build by system user, owner changed, build completed → immediate reinit
(no pubsub needed)
- **"claimed prebuild waits during in-progress claim build"**: claimed
but build still running → no reinit until build completes
- **"regular workspace gets 409"**: first build by real user → 409
Conflict, agent stops polling
- Updated claim publisher/listener tests: verify `UserID` survives JSON
round-trip + backward compat with raw string payloads
- Updated SSE round-trip test: verify `UserID` survives transmit →
receive cycle
Fixes#22359
## Rolling upgrade note
During a rolling deploy where old coderd instances coexist with new
ones, the pubsub `ReinitializationEvent` has a new `workspace_id` field
(JSON key `workspace_id`). Old publishers send a raw reason string
instead of JSON; the new listener gracefully falls back by treating the
entire payload as the reason and filling in `WorkspaceID` from context.
The only visible effect during the upgrade window is that `WorkspaceID`
may be the zero UUID in agent-side logs — this is cosmetic and resolves
once all instances are updated.
Fixes a flaky test (`TestUserTailnetTelemetry/invalid_header`) caused by
sub-microsecond precision mismatch between `time.Now()` calls on
Windows.
The server used `time.Now()` (nanosecond precision) for `ConnectedAt`
and `DisconnectedAt`, while the test compared against its own
`time.Now()`. On Windows, wall-clock jitter can cause the server
timestamp to appear slightly before the test's `predialTime`.
Switch to `dbtime.Now()` which rounds to microsecond precision (matching
Postgres), consistent with all other timestamps in `workspaceagents.go`.
Relates to: https://github.com/coder/internal/issues/1390
`time.Now()` has nanosecond precision while Postgres timestamps are
microsecond precision. When tests compare `time.Now()` against
DB-sourced timestamps using `Before`/`After`/`WithinRange`/etc., there
is a non-zero flake risk from the precision mismatch.
This replaces `time.Now()` with `dbtime.Now()` (which rounds to
microsecond precision) in all test assertions that compare against
database timestamps.
Follows from #22684.
## Changes (11 files)
| File | Changes |
|---|---|
| `coderd/apikey_test.go` | 11 comparisons with `ExpiresAt` |
| `coderd/users_test.go` | 2 comparisons with `ExpiresAt` |
| `coderd/oauth2_test.go` | 1 comparison with `token.Expiry` |
| `coderd/workspaces_test.go` | 2 comparisons with `DormantAt` |
| `coderd/workspaceagents_test.go` | 3 comparisons with
`ConnectedAt`/`DisconnectedAt` |
| `coderd/workspaceapps/db_test.go` | 1 comparison with `token.Expiry` |
| `coderd/provisionerdserver/provisionerdserver_test.go` | 1 comparison
with `key.ExpiresAt` |
| `enterprise/coderd/workspaces_test.go` | 1 comparison with `DormantAt`
|
| `enterprise/coderd/license/license_test.go` | 3 `NotBefore` values |
| `enterprise/coderd/licenses_test.go` | 2 `NotBefore` values |
| `enterprise/coderd/users_test.go` | 3 `Next()` comparisons |
## Not changed (intentionally)
- `scaletest/placebo/run_test.go` — compares wall-clock elapsed time,
not DB timestamps
- `cli/server_test.go`, `coderd/jwtutils/jwt_test.go`,
`enterprise/aibridgeproxyd/aibridgeproxyd_test.go` — TLS cert fields,
not DB-stored
- `coderd/azureidentity/azureidentity_test.go` — Azure cert expiry, not
DB
🤖 Generated by Claude Opus 4.6 but reviewed manually.
The listen loop in workspaceAgentsExternalAuthListen compared
OAuthExpiry using == which compares `time.Time` internal struct fields
including the `*time.Location` pointer.
`time.LoadLocation` does not cache the returned `*Location` pointer, so
each lib/pq connection gets a distinct pointer for the same timezone.
When `pq.ParseTimestamp()` applies the connection's location to a parsed
timestamp, the resulting time.Time embeds that connection-specific
pointer. If the `sql.DB` pool hands out different connections for the
two GetExternalAuthLink reads, the identical timestamp produces
`time.Time` values where == returns false despite representing the same
instant. This is intermittent because the pool _usually_ reuses the same
connection for sequential queries.
This change uses `.Equal()` to compare instants regardless of location.
Also makes the test's validation call counter atomic to fix a possible
data race between the HTTP server and test goroutines.
Since Go 1.22, the loop variable capture issue is resolved. Variables
declared by for loops are now per-iteration rather than per-loop, making
the 'v := v' pattern unnecessary.
Update the agent protobuf schema (agent/proto/agent.proto) to include:
- subagent_id field in WorkspaceAgentDevcontainer message
- id field in CreateSubAgentRequest message
Bump the Agent API version from v2.7 to v2.8 and update all client
references throughout the codebase (ConnectRPC27 -> ConnectRPC28,
DRPCAgentClient27 -> DRPCAgentClient28).
* Adds support for parameter `format=text` in the following API routes:
* `/api/v2/workspaceagents/:id/logs`
* `/api/v2/workspacebuilds/:id/logs`
* `/api/v2/templateversions/:id/logs`
* `/api/v2/templateversions/:id/dry-run/:id/logs`
* Adds links to view raw logs on the following pages:
* Workspace build page
* Template editor page
* Template version page
* Refactors existing log formatting in `cli/logs.go` to live in `codersdk`.
🤖 Generated with Claude Opus 4.5, reviewed by me.
---------
Co-authored-by: Claude <noreply@anthropic.com>
AI agents report status via patchWorkspaceAgentAppStatus, but this wasn't
extending workspace deadlines. This prevented proper task auto-pause behavior,
causing tasks to pause mid-execution when there were no human connections.
Now we call ActivityBumpWorkspace when agents report status, using the same
logic as SSH/IDE connections. We bump when transitioning to or from the working
state.
Closescoder/internal#1251
Agents were losing authentication during workspace shutdown, causing
shutdown scripts to fail. The auth query required agents to belong to
the latest build, but during shutdown a `stop` build becomes latest while
the `start` build's agents are still running.
Modified the auth query to allow `start` build agents to authenticate
temporarily during `stop` execution. The query allows auth when:
- Agent's `start` build job succeeded
- Latest build is `stop` with `pending`/`running` job status
- Builds are adjacent (`stop` is `build_number + 1`)
- Template versions match
Auth closes once `stop` completes.
Renamed `GetWorkspaceAgentAndLatestBuildByAuthToken` to
`GetAuthenticatedWorkspaceAgentAndBuildByAuthToken` since it returns the
agent's build (not always latest) during shutdown.
Closes coder/internal#1249
Fixes#19467
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
Add the AgentAPI changes to support the feature that transmits boundary
logs from workspaces to coderd via the agent API for eventual re-emission to
stderr. The API handlers are stubs for now because I'm trying to land
this feature from multiple smaller PRs.
High level architecture:
- Boundary records resource access in batches and sends proto message to
agent
- Agent proxies messages to coderd **(captured by the API changes in
this PR)**
- coderd re-emits logs to stderr
RFC:
https://www.notion.so/coderhq/Agent-Boundary-Logs-2afd579be59280f29629fc9823ac41ba
Provisioner steps broken into smaller granular actions.
Changes:
- `ExtractArchive` moved to `init` request (was in `configure`)
- Writing `tfstate` moved to `plan` (was in `configure`)
- Moved most plan/apply outputs to `GraphComplete`
fixes https://github.com/coder/internal/issues/1123
We want to tests that ports are not included after they are no longer used, but this isn't safe on the real OS networking stack because there is no way to guarantee a port _won't_ be used. Instead, we introduce an interface and fake implementation for testing.
On order to leave the filtering logic in the test path, this PR also does some refactoring.
Caching logic is left in the real OS querying implementation and a new test case is added for it in this PR.
Refactors Agent instance identity to be a SessionTokenProvider.
Refactors the CLI to create Agent clients via a centralized function, rather than add-hoc via individual command handlers and their flags.
This allows commands besides `coder agent`, but which still use the agent identity, to support instance identity authentication.
Fixes#19111 by unifying all API requests to go thru the SessionTokenProvider for auth credentials.
Fix https://github.com/coder/internal/issues/826
I wasn't able to recreate the flake, but my underlying assumption (from
reading the logs we have) is that there is a race condition where the
test will begin cleanup before the dev container recreation goroutine
has a chance to call `devcontainer up`.
I've refactored the test slightly and made it so that the test will not
finish until either the context has timed out, or `Up` has been called.
Fixes https://github.com/coder/internal/issues/907
We convert `workspacesdk.AgentConn` to an interface and generate a mock
for it. This allows writing `coderd` tests that rely on the agent's HTTP
api to not have to set up an entire tailnet networking stack.
Relates to https://github.com/coder/internal/issues/907
The test can take around 10s when it is the only one running, so in a
constrained environment like CI it makes sense that it still hits the 25
second timeout. For now we up the limit to 60 seconds until the test is
rewritten to greatly reduce the time taken.
Fixes https://github.com/coder/coder/issues/19372
We increase the read limit to 4MiB (we use this limit elsewhere). We
also make sure to stop sending messages when `containersCh` becomes
closed.
This PR replaces the use of the **container** ID with the
**devcontainer** ID. This is a breaking change. This allows rebuilding a
devcontainer when there is no valid container ID.
This commit consolidates two container endpoints on the backend and improves the
frontend devcontainer support by showing names and displaying apps as
appropriate.
With this change, the frontend now has knowledge of the subagent and we can also
display things like port forwards.
The frontend was updated to show dev container labels on the border as well as
subagent connection status. The recreation flow was also adjusted a bit to show
placeholder app icons when relevant.
Support for apps was also added, although these are still WIP on the backend.
And the port forwarding utility was added in since the sub agents now provide
the necessary info.
Fixescoder/internal#666
Closes https://github.com/coder/internal/issues/619
Implement the `coderd` side of the AgentAPI for the upcoming
dev-container agents work.
`agent/agenttest/client.go` is left unimplemented for a future PR
working to implement the agent side of this feature.
Bobby Ho