feat: defer MCP tool schemas behind a find_tools search (#28225)

## Summary

When the `mcp-tool-search` experiment is enabled, chatd stops inlining
connected MCP tool schemas into every generation. It instead exposes a
built-in `find_tools` tool whose description carries a compact catalog
of the deferred tools, and only ships full JSON schemas for tools the
model has activated by searching or by calling them directly.

Closes [CODAGT-760](https://linear.app/coder/issue/CODAGT-760).

## Problem

Tool-heavy agent configurations (GitHub, Linear, Notion, and dev-tooling
MCP servers) inline over 100k tokens of tool schema definitions into
every generation. Initial uncached requests reached ~216k tokens with
time-to-first-token close to nine minutes, while the model typically
invokes only a handful of tools per turn.

## How it works

- `decideMCPToolSearch` defers external and workspace `.mcp.json` MCP
tools whenever the experiment is enabled. Native, dynamic, provider,
skill, and transport tools are never deferred.
- `find_tools` embeds a server-grouped catalog in its tool description
(degrading to names-only, then counts-only, then a constant-size summary
past a context-scaled size cap) and scores keyword matches across tool
names, descriptions, parameter schemas, and server metadata. Queries can
scope to one server with a `server:` prefix, and exact `names` arguments
always activate.
- Activation state is ephemeral: it is re-derived each generation from
surviving chat history (`find_tools` results and direct calls to
deferred tools), so activations naturally lapse when compaction
summarizes them away. Aggregate activated schema weight is capped at 10%
of the context window, shedding the least recently activated schemas
first; `find_tools` shares that budget across parallel calls in one
step. No new persistence.
- Deferred tools stay registered for execution, so the model can call a
cataloged tool directly without searching first; the schema is activated
for subsequent steps.
- Fail-open: the experiment being disabled, an empty candidate set, or
an MCP tool named `find_tools` all disable deferral, leaving today's
behavior byte-identical on the wire.
- Prometheus counters/histograms track `find_tools` calls, matches,
activations, and deferred token weight.
- The conversation timeline renders `find_tools` calls with a collapsed
search summary and expandable match list, falling back to the generic
renderer on malformed payloads.

## Validation

- Unit tests for the catalog, matcher, experiment-gated decision, and
activation derivation; end-to-end chatd generation tests covering
search-then-call, direct-call activation, experiment-off wire parity,
compaction lapse, and subagent tool gating.
- Storybook interaction tests for the timeline rendering and
malformed-payload fallback.
- Remote dogfood UAT on dev.coder.com passed: deferral with a real MCP
server and Anthropic model, direct calls without prior search,
activation persistence across turns, experiment-off parity, and clean
UI/console.

> Disclosure: Mux (AI agent) authored this PR on Mike's behalf.
This commit is contained in:
Michael Suchacz
2026-08-18 19:12:47 +02:00
committed by GitHub
parent 62f4afbb60
commit 7724ee281a
26 changed files with 3762 additions and 59 deletions
+2
View File
@@ -5398,6 +5398,7 @@ const (
ExperimentWorkspaceUsage Experiment = "workspace-usage" // Enables the new workspace usage tracking.
ExperimentOAuth2 Experiment = "oauth2" // Enables OAuth2 provider functionality.
ExperimentMCPServerHTTP Experiment = "mcp-server-http" // Enables the MCP HTTP server functionality.
ExperimentMCPToolSearch Experiment = "mcp-tool-search" // Defers MCP tool schemas behind a searchable catalog in agent chats.
ExperimentWorkspaceBuildUpdates Experiment = "workspace-build-updates" // Enables publishing workspace build updates to the all builds pubsub channel.
ExperimentNATSPubsub Experiment = "nats_pubsub" // Enables embedded NATS pubsub.
ExperimentWorkspaceCapableLicensing Experiment = "workspace-capable-licensing" // Counts only users holding the workspace-create permission toward the license seat limit.
@@ -5451,6 +5452,7 @@ var ExperimentsKnown = Experiments{
ExperimentWorkspaceUsage,
ExperimentOAuth2,
ExperimentMCPServerHTTP,
ExperimentMCPToolSearch,
ExperimentNATSPubsub,
ExperimentWorkspaceBuildUpdates,
ExperimentWorkspaceCapableLicensing,