opensource/coder
3
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-09-24 15:04:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
15,280 Commits 479 Branches 388 Tags
27ed052d86b5c8a630ccaff45ca39632da2c419c
Commit Graph
4 Commits
Author SHA1 Message Date
Ethan a1ec9df345 test(agent/agentcontext): use resolved temp dirs in symlink tests (#26602) 
These symlink resolver tests build expected paths from `t.TempDir()`,
but the resolver canonicalizes symlink targets with
`filepath.EvalSymlinks`.

On macOS, `/var` resolves to `/private/var`, so the raw temp dir path
can disagree with the resolver output and flake. Use
`testutil.TempDirResolved` in the symlink-sensitive tests so the
expectations are canonicalized the same way.

Closes https://github.com/coder/internal/issues/1574
Closes ENG-2849
 2026-06-23 06:06:17 +00:00
Kyle Carberry 0f37522e6f refactor(agent/agentcontext): fixed-location shallow context discovery (#26596) 
## Problem

`agent/agentcontext` resolved workspace context by walking the working
directory recursively (depth 8) and matching files by basename. This
over-injected context:

- Instruction-file matching was case-insensitive, so the generated
`docs/reference/api/agents.md` was treated as an instruction file.
- Symlinked instruction files (`CLAUDE.md`, `.cursorrules` ->
`AGENTS.md`) shipped as duplicate resources.
- Nested `AGENTS.md` (e.g. `site/AGENTS.md`) were collected from
anywhere in the tree.
- Skills were discovered from *any* `skills/` directory anywhere in the
tree, and `.mcp.json` from any depth.

Resolving the repo root produced six instruction sources for what was
effectively one file of guidance, plus skills/MCP found by an open-ended
walk.

## What changed

Replace the recursive scan with **fixed-location, shallow** discovery.
Each scan root is inspected at its top level only: the resolver never
descends into subdirectories and never climbs to a parent. Additional
directories are added explicitly as sources (HTTP API) or via the
`CODER_AGENT_EXP_*_DIRS` seeding env vars.

- **Single working-dir scan root.** The working directory is one scan
root. Instruction files and `.mcp.json` are read only at its top level.
- **Fixed-location skills.** Skills are discovered only from `skills`,
`.agents/skills`, `.claude/skills`, `.codex/skills` (one skill per
immediate subdir with a `SKILL.md`), not from arbitrary `skills/`
directories.
- **Case-sensitive instruction names.** Exact
`AGENTS.md`/`CLAUDE.md`/`.cursorrules`; a lower-case `agents.md` is
ignored.
- **Symlink dedup.** Resources are attributed to their resolved target,
so symlinked `CLAUDE.md`/`.cursorrules` collapse into the single
`AGENTS.md`.
- **Watcher** mirrors the same fixed-location set instead of recursively
watching every scan root (no more walking `node_modules`).
- The recursive `walkDir`, `skipDirNames`, `MaxScanDepth`, and
`isSkillsContainer` are removed.

Resolving the repo root now yields `AGENTS.md`, `.mcp.json`, and the
`.agents/skills`/`.claude/skills` skills, with no nested
instruction-file noise.

## Behavior change

`site/AGENTS.md` is no longer auto-injected when the working dir is the
repo root. It loads when the working dir **is** `site/` (its top level),
or when `site/` is added as an explicit source. There is intentionally
**no walk-up** to a `.git` project root: an agent started in a
subdirectory does not auto-inherit ancestor `AGENTS.md`; those
directories are added explicitly.

<details>
<summary>Verification &amp; decision log</summary>

**codex research (confirmed via source).** Instruction files: codex
walks up to the first `.git` ancestor and reads root-&gt;cwd, one file
per directory, exact-cased names (`codex-rs/core/src/agents_md.rs`).
Skills: fixed roots (`.agents/skills`, `.codex/skills`,
`$CODEX_HOME/skills`, ...) with bounded in-root recursion
(`core-skills/src/loader.rs`). MCP: `.codex/config.toml` via walk-up; a
project `.mcp.json` is not a runtime source in codex. No resource type
triggers an unbounded downward walk.

**Decisions.**
- Adopt codex's fixed-location, shallow discovery (case-sensitive names,
symlink dedup, top-level-only files, container-only skills).
- **Deliberately omit codex's walk-up to the `.git` project root.** In
Coder the working dir is the scan root and extra directories are added
explicitly (HTTP API / `CODER_AGENT_EXP_*_DIRS`), so the implicit
ancestor climb added surprise without benefit (e.g. `context add ./site`
should scan `./site`, not the repo root).
- Keep `.mcp.json` (codex uses `config.toml`, intentionally not added).
- Include `.claude/skills` and `.codex/skills` in the container list so
the repo's existing `.claude/skills` skills are not regressed; skills
recurse one level inside a container.

**Tests.** `TestManager_WorkingDirScannedShallow` (working dir read at
top level; ancestor root and nested subdir both excluded);
`TestResolver_SkillsOnlyFromFixedContainers`,
`TestResolver_MCPConfigOnlyAtScanRoot`,
`TestResolver_SymlinkedInstructionFilesDeduplicated`,
`TestResolver_InstructionFilesOnlyAtScanRoot`,
`TestResolver_InstructionNamesAreCaseSensitive`. Cap tests use multiple
scan roots.

**Local checks.** `gofmt`, `go vet`, `golangci-lint`, `go test -race`,
and `make lint/emdash` pass for the package.
</details>

---
🤖 Generated by Coder Agents on behalf of @kylecarbs.
 2026-06-23 03:42:40 +00:00
Kyle Carberry 2f8bba792a feat: push MCP server context and tools from agentcontext (#26533) 
## What

Live MCP servers and their tools now flow into the `agentcontext`
snapshot and are pushed to coderd via `PushContextState`, stored
alongside instruction files and skills. Previously the resolver's MCP
seam was unimplemented, so live MCP tool lists never reached the pushed
snapshot.

`agentcontext` is now **fully self-contained** for MCP: it connects to
the MCP servers declared in the `.mcp.json` files its own watcher
already discovers, lists their tools, and emits `KindMCPServer`
resources. It does **not** depend on or modify `agent/x/agentmcp` — that
package is left pristine and keeps serving the agent's MCP HTTP API. The
two MCP paths run independently, which means the legacy package can be
deleted later without touching this code.

## How

- **Self-contained runner** (`agentcontext/mcprunner.go`): a one-shot
MCP client (connect → initialize → list tools → close) with its own
`.mcp.json` parser. A Manager goroutine (`runMCPSync`) reloads it
whenever the discovered `KindMCPConfig` `path:contenthash` set changes,
then re-resolves so the new tools are published. Per-server connects run
in parallel (bounded) with a per-server timeout; a server that fails to
connect is recorded as a failure rather than aborting the batch. Each
connect also force-kills its subprocess on close, because mcp-go's stdio
`Close()` closes stdin and then blocks on `cmd.Wait()` with no kill — a
server that ignores stdin-close would otherwise stall the whole reload
loop.
- **Resource production** (`agentcontext/mcp.go`):
`buildMCPServerResources` turns the runner's non-blocking per-server
snapshot into `KindMCPServer` resources. Connected servers carry their
sorted tools (`StatusOK`); failed servers surface as `StatusUnreadable`
issues instead of vanishing; connected-but-no-tools-yet are skipped
until a later reload. The content hash is tool-set sensitive. The
resolver consumes this through a plain `MCPResources func() []Resource`
field (no `MCPProvider` interface).
- **Tool names**: emitted exactly as the server reports them. Flattening
into a single namespace (e.g. `server__tool`) is left to the control
plane in the next step, since each resource already carries the server
name.
- **Drift**: MCP resources are excluded from the snapshot
aggregate/drift hash (`driftResources`). MCP servers connect
asynchronously after boot; without this, a server finishing its connect
would dirty every hydrated chat even though nothing the user pinned
changed.
- **Wiring** (`agent.go`): the manager is given
`ManagerOptions.MCPExecer`/`MCPUpdateEnv`; `agent/x/agentmcp` is
untouched.
- **Config validation**: a structurally broken `.mcp.json` surfaces as
`StatusInvalid` rather than silently dropping all its servers.

coderd already persists `mcp_server`/`mcp_config` resource bodies
(including tools), so no coderd or proto changes were required.

## Testing

- **Unit**: `buildMCPServerResources` (grouping/sort/skip/failed/hash
sensitivity), MCP resources applied via the resolver seam, MCP exclusion
from the aggregate hash, `.mcp.json` parsing (transport inference, env
expansion), `toolInputSchema`, and `mcpConfigSet` change detection.
- **Proto serialization**
(`TestDRPCPusher_HappyPathSerializesAllFields`): a `KindMCPServer`
resource (tools + input schema) round-trips through `PushContextState`
into the `MCPServerBody` wire form, asserting the server name, tool
name/description, and the decoded `input_schema`.
- **Manager-level, real subprocess**
(`TestManager_MCPServerToolsInSnapshot`): a `.mcp.json` points at a
re-exec'd fake stdio MCP server; the runner connects it and its `echo`
tool surfaces as a `KindMCPServer` resource in the Manager snapshot —
the same snapshot pushed to coderd — exercising `runMCPSync` and the
resolver wiring end to end.
- **Regression** (`TestManager_MCPServerHangingCloseDoesNotStall`): the
fake server ignores stdin-close; the test asserts its tool still
surfaces, proving the runner force-kills the subprocess instead of
stalling the reload. Verified to fail without the fix.
- All pass under `-race`; `go build ./...`, `go vet`, and
`golangci-lint` are clean on the touched packages.

## Scope / follow-ups

This is the agent-side production+push half. The chatd consumer (reading
the pinned MCP resources for prompt/tool injection, including any
server-prefix flattening of tool names) and removing the legacy
`workspaceMCPToolsCache` pull path remain follow-ups, per the RFC
rollout. While both `agent/x/agentmcp` and `agentcontext` exist, stdio
MCP servers are spawned by both; this is intentional and temporary until
`agentmcp` is removed.

<details>
<summary>Implementation plan and decisions</summary>

**Goal:** produce live MCP server resources (with tools) from
`agentcontext` and push them to coderd.

**Starting state (main):** proto (`PushContextState`, `MCPServerBody`,
`MCPTool`), the drpc adapter, coderd storage
(`workspace_agent_context_resources`, body kind `mcp_server`), and the
resolver's MCP seam already existed; nothing implemented the seam or fed
live tools into the snapshot.

**Decision (agentcontext fully separate from agentmcp):** `agentcontext`
starts and lists its own MCP servers using only the connect-and-list
half of an mcp-go client, driven by the `.mcp.json` files its existing
watcher discovers. It shares no state with `agent/x/agentmcp` and does
not import it. Two earlier revisions of this branch were discarded: (1)
relocating `agentmcp` into `agentcontext` (rejected — it duplicates
config parsing and file watching `agentcontext` already does); (2)
reading `agentmcp`'s cached server snapshot via new accessors (rejected
— unnecessary coupling between two packages that should simply run
independently while one is being retired). The temporary double-spawn of
stdio servers is the accepted cost of keeping the two paths cleanly
separated until `agentmcp` is removed.

**Decision (no tool-name prefixing, no MCPProvider interface):** the
agent pushes raw, unflattened data — server name plus verbatim tool
names — and lets the control plane own any `server__tool` flattening.
With a single self-contained producer, the `MCPProvider` interface was
collapsed into a `func() []Resource` field on the resolver.

**Invariants held:** no secrets (env/headers) in pushed resources, only
server/tool metadata; MCP excluded from the drift hash; the seam is
non-blocking so the resolver never stalls on MCP I/O.

</details>

---

*This PR was created by Coder Agents on behalf of @kylecarbs.*
 2026-06-21 16:31:05 -06:00
Kyle Carberry cd3692c0c2 feat: add agent-side workspace context sources and Agent API v2.10 PushContextState (#25983) 
Adds the agent half of the workspace context sources RFC. The agent now
resolves instruction files, skills, and MCP configs into a typed
`Snapshot`, watches the relevant paths recursively, exposes the source
list over a workspace-agent HTTP API, and pushes each `Snapshot` to
coderd over a new `PushContextState` RPC on Agent API v2.10.

The coderd-side handler is a stub returning `Unimplemented` for now.
Real persistence to `workspace_agent_context`, chatd hydration on dirty
events, and the `KindMCPServer` MCP provider are tracked by
[CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd).
This matches the pattern used for v2.7 `ReportBoundaryLogs` in
[#21293](https://github.com/coder/coder/pull/21293), which bumped the
version and shipped a stub server so the wire and client could iterate
before the persistence layer landed.

## What ships

### agent/agentcontext (new package)

- `Source`, `Resource` (kinds `instruction_file`, `skill`, `mcp_config`,
`mcp_server` plus reserved `plugin`/`hook`/`subagent`/`command`),
`ResourceStatus`, `Snapshot`, `ComputeAggregateHash`.
- `Manager` owns the in-memory source list, performs the initial resolve
synchronously in `NewManager`, runs a re-resolve/watcher loop in `Run`,
exposes
`AddSource`/`RemoveSource`/`Sources`/`HasSource`/`Snapshot`/`SubscribeChanges`/`Resync`/`SeedSources`/`Close`.
- `Resolver` walks scan roots, classifies recognized files, enforces 64
KiB per-resource, 2 MiB aggregate, and 500-resource caps with
`StatusOversize`/`StatusExcluded`/`StatusUnreadable`/`StatusInvalid`
outcomes, skips `node_modules`/`vendor`/etc., validates symlink targets
stay inside the scan root, stamps `SourcePath` on user-derived
resources, and optionally pulls MCP server tool lists via an
`MCPProvider` interface. MCP config resources ship metadata only (size,
hash) so secrets in env blocks never leave the agent.
- `Watcher` is a recursive `fsnotify` wrapper with a 250 ms debounce,
dynamic arming of newly created directories, and an ENOSPC-tolerant
degraded mode that no-ops further syncs until the manager resyncs
explicitly.
- HTTP API for `GET/POST /sources`, `GET/DELETE /sources/{path}`, `POST
/resync` mounted at `/api/v0/context`.
- `Pusher` interface plus `RunPush` goroutine with exponential backoff
capped at 30 s. `DRPCPusher` adapts the generated `DRPCAgentClient210`
to `Pusher` and translates `drpcerr.Unimplemented` to
`ErrPushUnimplemented` so the push loop exits cleanly when talking to
coderd deployments that have not enabled the real handler.

### agent/proto (v2.10)

- New messages `ContextResource`, `PushContextStateRequest`,
`PushContextStateResponse` and the `PushContextState` RPC on `service
Agent`.
- Generated `DRPCAgentClient210` interface and
`codersdk/agentsdk.Client.ConnectRPC210` / `ConnectRPC210WithRole`.
- `tailnet/proto.CurrentMinor` bumped from `9` to `10`.

### Agent wiring

- `agent.Options.Client` declares both v2.9 and v2.10 connectors;
`run()` dials with `ConnectRPC210WithRole`.
- `apiConnRoutineManager` holds a `DRPCAgentClient210`. Existing v2.8
routines keep their narrower `DRPCAgentClient28` signature thanks to
interface embedding.
- `startAgentAPI210` is the v2.10 counterpart to `startAgentAPI` for
routines that need the new client. The push context state routine uses
it.
- A `contextManager` is constructed in `agent.init()`, seeded from the
existing `CODER_AGENT_EXP_*_DIRS` env vars, started in its own goroutine
under `gracefulCtx`, and closed in `agent.Close`.
- `handleManifest` calls `Manager.SeedSources` for sources rooted at the
manifest directory, then `Resync` after `manifest.Swap`, so the snapshot
reflects the workspace working directory immediately instead of waiting
for the next filesystem event.
- HTTP routes mounted at `/api/v0/context` when the manager is up.

### Coderd stub

`coderd/agentapi/context.go` returns `drpcerr.Unimplemented` for
`PushContextState`. The real handler that persists
`workspace_agent_context` rows, hydrates chats, and emits dirty events
lives in CODAGT-569.

## Tests

24 tests across `agent/agentcontext` cover types, paths, resolver
behavior with file caps, skill containers, MCP secret omission, symlink
target validation, the recursive watcher firing on real fsnotify events,
manager source CRUD / `Resync` / `SeedSources` / `Run` lifetime, the
HTTP API, the DRPC adapter, and the push retry / initial-flag /
unimplemented paths. Passes `go test -race -count=2`.

`TestAgent_ContextStatePushed` boots a full agent against
`agenttest.FakeAgentAPI` (which now records `PushContextState` traffic)
and asserts the seeded `AGENTS.md` appears in a snapshot push with
`schema_version = 1`.

<details>
<summary>Notes for reviewers</summary>

- Source CRUD is workspace-agent-token only; coderd is not in the path
for source mutation.
- Per-resource cap 64 KiB, aggregate 2 MiB, count cap 500; resources
past the cap ship with `StatusExcluded` and an empty payload so the
aggregate hash still detects content edits. MCP-emitted resources
enforce both a per-provider count cap and the aggregate byte cap.
- Symlinks inside the scan root are followed; symlinks pointing outside
(or broken) are rejected with `StatusExcluded` so credentials reachable
via a stray symlink stay off the wire.
- The initial push gates `lifecycle = ready` in the eventual full
design. For this PR the `SeedSources` plus `handleManifest`-driven
`Resync` keeps the snapshot fresh; the live push loop ships now and
DRPCPusher translates the coderd `Unimplemented` stub into a clean exit.
- The `PLUGIN`/`HOOK`/`SUBAGENT`/`COMMAND` kinds are reserved in proto
and Go enums but unused; the Claude Code plugin resolver ships in a
follow-up that does not need a schema migration.
- Two follow-ups remain, both tracked by
[CODAGT-569](https://linear.app/codercom/issue/CODAGT-569/enable-agent-api-v210-pushcontextstate-bump-currentminor-wire-coderd):
(1) the chatd-side handler that persists snapshots and dirties chats;
(2) the `coder exp chat context` CLI command set for
`list`/`show`/`add`/`remove`/`refresh`.

</details>

_This PR was authored by Coder Agents on Kyle Carberry's behalf._
 2026-06-08 12:08:40 -07:00