4 Commits

Include renames

Author	SHA1	Message	Date
Kyle Carberry	bca0ce04ca	feat: integrate agent context snapshots into chats (#26389) ... Makes the chat context foundation from #26385 live. That PR added the storage columns, writer queries, and a dormant `agentapi.ContextDirtyMarker` trigger with no production callers; this PR wires them together end to end. When a workspace agent pushes a context snapshot, bound chats now hydrate to that snapshot's hash, and a later push with a different hash flips already-pinned chats to dirty (emitting a `context_dirty` watch event after the transaction commits). Chat creation pins the agent's latest snapshot when one already exists. The experimental chat API exposes this as `Chat.Context` (`*ChatContext` with `dirty`, `dirty_since`, `error`), and a new `PUT /api/experimental/chats/{chat}/context` endpoint re-pins the agent's latest snapshot and clears the dirty marker. `context_dirty_resources` stays NULL (the resource-level diff is deferred to the UI phase) and the live per-turn context pull is unchanged. The end-to-end test provisions a workspace agent via the echo provisioner, connects it over the Agent API v2.10, and exercises the full path: an initial push hydrates a bound chat (clean), a second push with a different hash marks it dirty, the API reports the dirty state, and the refresh endpoint clears it. <details> <summary>Decision log</summary> - **API shape — sub-struct.** Dirty state is surfaced as `codersdk.Chat.Context *ChatContext { Dirty bool; DirtySince *time.Time; Error string }` rather than flat fields, matching the RFC's named `ChatContext` type and leaving room for future fields (resource diff, sources). `db2sdk.Chat` populates it when the chat is context-tracked (`len(ContextAggregateHash) > 0`), dirty, or carries a snapshot error, and leaves it nil (`omitempty`) otherwise. `Dirty` mirrors `context_dirty_since` being set. - **Marker wiring.** The chat daemon is injected directly as the `agentapi.ContextDirtyMarker`. It is unconditionally constructed (only its background worker is gated), so the marker is always non-nil and the wiring matches every other `api.chatDaemon` call site. `agentapi` still treats a nil marker as "chatd absent", so `PushContextState` stays a pure write path for any future caller that does not wire chatd in. - **Refresh is atomic.** `RefreshChatContext` reads the agent's latest snapshot and re-pins the chat in one repeatable-read transaction, so a concurrent push cannot land between the read and the write and leave the chat pinned to a stale hash with the dirty marker cleared. - **Hydrate + dirty run inside the push transaction.** The fan-out shares the push's transaction so a concurrent refresh cannot interleave with the version gate; `context_dirty` watch events publish only after commit. The pinned hash on dirtied chats is intentionally left unchanged — the refresh endpoint re-pins it. - **Dirtied chats keep their pinned hash.** Drift is advisory: a dirty chat stays usable, and refreshing is the only path that advances the pinned hash. - **Test binds `chats.agent_id` directly.** In production the binding is set lazily during a chat turn (`chatd.persistBuildAgentBinding`); the test sets it via `dbgen` so it exercises the context flow rather than turn resolution. Plan: `coderd/x/chatd` context integration + E2E (sub-struct API, create-time + push-time hydration, refresh endpoint; `context_dirty_resources` and the per-turn pull untouched). </details> 🤖 Generated by Coder Agents on behalf of @kylecarbs	2026-06-16 17:46:47 +00:00
Kyle Carberry	210261b143	feat: add chat context pinning storage and push trigger (#26385) ... Foundation for the Workspace Context Sources RFC (phase 3). The agent push (#25983) and coderd snapshot storage (#26145) already persist per-agent context snapshots; this PR lands the **chat-side storage** plus the **`agentapi` push trigger** that a follow-up will use to read them. It does **not** touch `chatd` and changes no behavior — nothing wires an implementation yet. ## What changed - Adds four nullable columns to `chats` — `context_aggregate_hash`, `context_dirty_since`, `context_dirty_resources`, and `context_error` — and rebuilds the `chats_expanded` view. - Adds three queries — `SetChatContextSnapshot`, `HydrateAgentChatsContext`, `MarkChatsContextDirtyByAgent` — with `dbauthz` wrappers and `audit` entries. They are store-interface methods covered by a Postgres test (`TestChatContextHydration`). - Adds the `agentapi.ContextDirtyMarker` interface and invokes it inside the `PushContextState` transaction, publishing collected events only after commit. ## Intentionally inert There are **no production callers** of the three queries and **no implementation** wired for `ContextDirtyMarker`, so the push trigger is dormant. This is deliberate: the PR is the durable storage/query foundation only. The actual integration — the `chatd` implementation that hydrates/dirties chats and backs a refresh endpoint, consuming the pinned context in prompt building, the rich SDK types + UI, and retiring the live per-turn pull — lands as a single follow-up PR. Splitting this way keeps the schema/query layer reviewable on its own and keeps the integration whole in one place. Refs #25983, #26145. <details> <summary>Decision log</summary> - **Columns over a side table.** The four `chats` columns are the durable model (accepting the one-time `chats_expanded` view/CTE churn). `last_injected_context` is deliberately left untouched — it is load-bearing for the live per-turn context pull. - **Keep `agentapi`, drop `chatd`.** The earlier revision wired the hydrate/dirty implementation through `chatd` and added a `PUT /chats/{chat}/context` refresh endpoint. Those were removed so this PR is pure foundation; `agentapi` defines the trigger + interface (it does not import `chatd`), and the `chatd` implementation arrives with the full integration. - **No new experiment flag.** The columns are dark and unread by prompt building. - **Authz.** The new query wrappers authorize chat updates under the chat RBAC object / `ResourceChat`, consistent with the existing system chat mutators. </details> --- 🤖 Generated by Coder Agents on behalf of @kylecarbs.	2026-06-15 14:41:00 -07:00
Kyle Carberry	b439b06ee6	feat: persist agent-pushed workspace context snapshots in coderd (#26145) ... Replaces the v2.10 `PushContextState` stub with a real coderd write path. Phase 1 of the chat-side persistence story; nothing reads these rows yet. Follows [#25983](https://github.com/coder/coder/pull/25983) and unblocks [CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd). ## What ships ### Schema (`000517_workspace_agent_context.{up,down}.sql`) Two new tables plus `api_key_scope` enum extensions: - `workspace_agent_context_snapshots` (PK `workspace_agent_id` to `workspace_agents(id) ON DELETE CASCADE`): one row per agent, overwritten per push. Holds `version`, `schema_version`, `aggregate_hash`, `snapshot_error`, `received_at`. - `workspace_agent_context_resources` (PK `(workspace_agent_id, source)`): per-resource state. `body_kind` and `status` are `TEXT` + `CHECK` so adding new wire kinds (the RFC's reserved PLUGIN/HOOK/SUBAGENT/COMMAND) is a one-line CHECK update plus a Go switch case. ### SQLC queries (`coderd/database/queries/workspaceagentcontext.sql`) - `UpsertWorkspaceAgentContextSnapshot` - `UpsertWorkspaceAgentContextResource` - `DeleteStaleWorkspaceAgentContextResources` (delete-where-source-not-in) - `GetLatestWorkspaceAgentContextSnapshot` - `ListWorkspaceAgentContextResources` ### Handler (`coderd/agentapi/context.go`) `ContextAPI` is a new sub-API. `PushContextState`: 1. Rejects `schema_version > 1` with a non-`Unimplemented` error so a forward-incompatible agent fails loudly during rollout instead of slipping into the permanent fallback path the `Unimplemented` translation reserves for old coderd deployments. 2. Validates resources: no empty/duplicate sources, every variant maps to a known body kind, every status maps to a known enum value, the `Body` oneof is set (even when status is non-OK, mirroring the wire guarantee so coderd can attribute failures to a known kind). 3. Inside `Database.InTx`, reads the existing snapshot. If the push is not `initial` and `version` is not strictly greater, returns `accepted = false` and leaves stored state untouched. Otherwise upserts the snapshot row, upserts each resource, then runs the stale-source prune so the snapshot and resource rows always agree. 4. Returns `accepted = true` on success. Resource bodies are stored as `protojson(body oneof variant)` in `body JSONB` with `body_kind` as the discriminator. Adding a new field to an existing variant is zero work since `protojson` tolerates new fields; adding a new variant is a CHECK + switch case. ### RBAC + dbauthz - New `ResourceWorkspaceAgentContext` (Create/Read/Update/Delete). - New `SubjectTypeAgentContext` plus `subjectAgentContext` system role and `dbauthz.AsAgentContext` helper. The push handler elevates to this subject; the agent's own role does not get direct write access to the table. - New `workspace_agent_context:*` API key scopes registered in the enum migration; internal-only (not added to `externalLowLevel`). ### Audit These rows are agent-pushed state, not user-authored. They are intentionally not added to `AuditActionMap` and not enumerated in `enterprise/audit/table.go`, matching `boundary_logs`, `workspace_agent_memory_resource_monitor`, etc. `enterprise/audit` tests pass unchanged. ## Tests - `coderd/agentapi/context_test.go`: 12 subtests covering accepts/rejects (schema version, empty/duplicate source, unknown status, missing body), version semantics (stale dropped, same-version replay dropped, `initial=true` overwrites lower version), variant coverage, non-OK status persistence, and the empty-active-set prune case. - `coderd/database/dbauthz/dbauthz_test.go`: 5 `MethodTestSuite` cases covering the new queries. - `coderd/rbac/roles_test.go`: `WorkspaceAgentContext` permission row asserting no human role currently has access. - `coderd/database/migrations/testdata/fixtures/000517_workspace_agent_context.up.sql`: one snapshot + one resource per known body kind plus a non-OK status, so the migration test suite never lands with these tables empty. ## Out of scope (later phases) - Chat hydration (`chats.context_aggregate_hash`, `last_injected_context`). - Dirty-bit fan-out and `PUT /chats/{id}/context`. - Agent-side `POST /api/v0/context/resync` barrier and the `coder exp chat context` CLI. - `codersdk` chat-context wire types and the dashboard Sources drawer. - Removal of the chatd per-turn pull fallback. ## Compat property This is a pure write path. If anything here returns errors the agent's `RunPush` loop backs off, no chat behavior changes, and the workspace keeps behaving exactly like it did before v2.10. <details> <summary>Implementation plan and decision log</summary> Key design calls: 1. **Concurrency**: Accept iff `req.Initial || req.Version > existing.Version`. The strict RFC reading ("version comparison is authoritative") locks restarted agents out because their per-process counter resets to 1; honoring `initial=true` reflects the real reboot reality while still rejecting steady-state replays/out-of-order pushes. 2. **Body encoding**: `protojson` over the oneof variant body proto, stored in JSONB with `body_kind` discriminator. Structured at the API/Go layer, schema-tolerant at the storage layer, and Phase 2 readers round-trip back via `protojson.Unmarshal`. 3. **Schema version rejection**: returns a normal error, not `Unimplemented`. The agent's `RunPush` loop only short-circuits on `Unimplemented`; that escape hatch is reserved for old coderd deployments. A forward-incompatible agent should retry-and-back-off, not flip the connection into permanent fallback. 4. **Validation strictness**: empty sources, duplicate sources, `STATUS_UNSPECIFIED`, and missing `Body` oneof variants are rejected before any write so a misbehaving agent cannot poison the snapshot table. Phase 2 readers can trust every row maps to a known proto variant. </details> _This PR was authored by Coder Agents on Kyle Carberry's behalf._	2026-06-15 09:38:52 -07:00
Kyle Carberry	cd3692c0c2	feat: add agent-side workspace context sources and Agent API v2.10 PushContextState (#25983) ... Adds the agent half of the workspace context sources RFC. The agent now resolves instruction files, skills, and MCP configs into a typed `Snapshot`, watches the relevant paths recursively, exposes the source list over a workspace-agent HTTP API, and pushes each `Snapshot` to coderd over a new `PushContextState` RPC on Agent API v2.10. The coderd-side handler is a stub returning `Unimplemented` for now. Real persistence to `workspace_agent_context`, chatd hydration on dirty events, and the `KindMCPServer` MCP provider are tracked by [CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd). This matches the pattern used for v2.7 `ReportBoundaryLogs` in [#21293](https://github.com/coder/coder/pull/21293), which bumped the version and shipped a stub server so the wire and client could iterate before the persistence layer landed. ## What ships ### agent/agentcontext (new package) - `Source`, `Resource` (kinds `instruction_file`, `skill`, `mcp_config`, `mcp_server` plus reserved `plugin`/`hook`/`subagent`/`command`), `ResourceStatus`, `Snapshot`, `ComputeAggregateHash`. - `Manager` owns the in-memory source list, performs the initial resolve synchronously in `NewManager`, runs a re-resolve/watcher loop in `Run`, exposes `AddSource`/`RemoveSource`/`Sources`/`HasSource`/`Snapshot`/`SubscribeChanges`/`Resync`/`SeedSources`/`Close`. - `Resolver` walks scan roots, classifies recognized files, enforces 64 KiB per-resource, 2 MiB aggregate, and 500-resource caps with `StatusOversize`/`StatusExcluded`/`StatusUnreadable`/`StatusInvalid` outcomes, skips `node_modules`/`vendor`/etc., validates symlink targets stay inside the scan root, stamps `SourcePath` on user-derived resources, and optionally pulls MCP server tool lists via an `MCPProvider` interface. MCP config resources ship metadata only (size, hash) so secrets in env blocks never leave the agent. - `Watcher` is a recursive `fsnotify` wrapper with a 250 ms debounce, dynamic arming of newly created directories, and an ENOSPC-tolerant degraded mode that no-ops further syncs until the manager resyncs explicitly. - HTTP API for `GET/POST /sources`, `GET/DELETE /sources/{path}`, `POST /resync` mounted at `/api/v0/context`. - `Pusher` interface plus `RunPush` goroutine with exponential backoff capped at 30 s. `DRPCPusher` adapts the generated `DRPCAgentClient210` to `Pusher` and translates `drpcerr.Unimplemented` to `ErrPushUnimplemented` so the push loop exits cleanly when talking to coderd deployments that have not enabled the real handler. ### agent/proto (v2.10) - New messages `ContextResource`, `PushContextStateRequest`, `PushContextStateResponse` and the `PushContextState` RPC on `service Agent`. - Generated `DRPCAgentClient210` interface and `codersdk/agentsdk.Client.ConnectRPC210` / `ConnectRPC210WithRole`. - `tailnet/proto.CurrentMinor` bumped from `9` to `10`. ### Agent wiring - `agent.Options.Client` declares both v2.9 and v2.10 connectors; `run()` dials with `ConnectRPC210WithRole`. - `apiConnRoutineManager` holds a `DRPCAgentClient210`. Existing v2.8 routines keep their narrower `DRPCAgentClient28` signature thanks to interface embedding. - `startAgentAPI210` is the v2.10 counterpart to `startAgentAPI` for routines that need the new client. The push context state routine uses it. - A `contextManager` is constructed in `agent.init()`, seeded from the existing `CODER_AGENT_EXP_*_DIRS` env vars, started in its own goroutine under `gracefulCtx`, and closed in `agent.Close`. - `handleManifest` calls `Manager.SeedSources` for sources rooted at the manifest directory, then `Resync` after `manifest.Swap`, so the snapshot reflects the workspace working directory immediately instead of waiting for the next filesystem event. - HTTP routes mounted at `/api/v0/context` when the manager is up. ### Coderd stub `coderd/agentapi/context.go` returns `drpcerr.Unimplemented` for `PushContextState`. The real handler that persists `workspace_agent_context` rows, hydrates chats, and emits dirty events lives in CODAGT-569. ## Tests 24 tests across `agent/agentcontext` cover types, paths, resolver behavior with file caps, skill containers, MCP secret omission, symlink target validation, the recursive watcher firing on real fsnotify events, manager source CRUD / `Resync` / `SeedSources` / `Run` lifetime, the HTTP API, the DRPC adapter, and the push retry / initial-flag / unimplemented paths. Passes `go test -race -count=2`. `TestAgent_ContextStatePushed` boots a full agent against `agenttest.FakeAgentAPI` (which now records `PushContextState` traffic) and asserts the seeded `AGENTS.md` appears in a snapshot push with `schema_version = 1`. <details> <summary>Notes for reviewers</summary> - Source CRUD is workspace-agent-token only; coderd is not in the path for source mutation. - Per-resource cap 64 KiB, aggregate 2 MiB, count cap 500; resources past the cap ship with `StatusExcluded` and an empty payload so the aggregate hash still detects content edits. MCP-emitted resources enforce both a per-provider count cap and the aggregate byte cap. - Symlinks inside the scan root are followed; symlinks pointing outside (or broken) are rejected with `StatusExcluded` so credentials reachable via a stray symlink stay off the wire. - The initial push gates `lifecycle = ready` in the eventual full design. For this PR the `SeedSources` plus `handleManifest`-driven `Resync` keeps the snapshot fresh; the live push loop ships now and DRPCPusher translates the coderd `Unimplemented` stub into a clean exit. - The `PLUGIN`/`HOOK`/`SUBAGENT`/`COMMAND` kinds are reserved in proto and Go enums but unused; the Claude Code plugin resolver ships in a follow-up that does not need a schema migration. - Two follow-ups remain, both tracked by [CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd): (1) the chatd-side handler that persists snapshots and dirties chats; (2) the `coder exp chat context` CLI command set for `list`/`show`/`add`/`remove`/`refresh`. </details> _This PR was authored by Coder Agents on Kyle Carberry's behalf._	2026-06-08 12:08:40 -07:00