Michael Suchacz
7724ee281a
## Summary When the `mcp-tool-search` experiment is enabled, chatd stops inlining connected MCP tool schemas into every generation. It instead exposes a built-in `find_tools` tool whose description carries a compact catalog of the deferred tools, and only ships full JSON schemas for tools the model has activated by searching or by calling them directly. Closes [CODAGT-760](https://linear.app/coder/issue/CODAGT-760). ## Problem Tool-heavy agent configurations (GitHub, Linear, Notion, and dev-tooling MCP servers) inline over 100k tokens of tool schema definitions into every generation. Initial uncached requests reached ~216k tokens with time-to-first-token close to nine minutes, while the model typically invokes only a handful of tools per turn. ## How it works - `decideMCPToolSearch` defers external and workspace `.mcp.json` MCP tools whenever the experiment is enabled. Native, dynamic, provider, skill, and transport tools are never deferred. - `find_tools` embeds a server-grouped catalog in its tool description (degrading to names-only, then counts-only, then a constant-size summary past a context-scaled size cap) and scores keyword matches across tool names, descriptions, parameter schemas, and server metadata. Queries can scope to one server with a `server:` prefix, and exact `names` arguments always activate. - Activation state is ephemeral: it is re-derived each generation from surviving chat history (`find_tools` results and direct calls to deferred tools), so activations naturally lapse when compaction summarizes them away. Aggregate activated schema weight is capped at 10% of the context window, shedding the least recently activated schemas first; `find_tools` shares that budget across parallel calls in one step. No new persistence. - Deferred tools stay registered for execution, so the model can call a cataloged tool directly without searching first; the schema is activated for subsequent steps. - Fail-open: the experiment being disabled, an empty candidate set, or an MCP tool named `find_tools` all disable deferral, leaving today's behavior byte-identical on the wire. - Prometheus counters/histograms track `find_tools` calls, matches, activations, and deferred token weight. - The conversation timeline renders `find_tools` calls with a collapsed search summary and expandable match list, falling back to the generic renderer on malformed payloads. ## Validation - Unit tests for the catalog, matcher, experiment-gated decision, and activation derivation; end-to-end chatd generation tests covering search-then-call, direct-call activation, experiment-off wire parity, compaction lapse, and subagent tool gating. - Storybook interaction tests for the timeline rendering and malformed-payload fallback. - Remote dogfood UAT on dev.coder.com passed: deferral with a real MCP server and Anthropic model, direct calls without prior search, activation persistence across turns, experiment-off parity, and clean UI/console. > Disclosure: Mux (AI agent) authored this PR on Mike's behalf.
Metrics Documentation Generator
This tool generates the Prometheus metrics documentation at docs/admin/integrations/prometheus.md.
How It Works
The documentation is generated from two metrics files:
metrics(static, manually maintained)generated_metrics(auto-generated, do not edit)
These files are merged and used to produce the final documentation.
metrics (static)
Contains metrics that are not directly defined in the coder source code:
go_*: Go runtime metricsprocess_*: Process metrics from prometheus/client_golangpromhttp_*: Prometheus HTTP handler metricscoder_ai_gateway_*: AI Gateway metrics are registered through prefixed registerer that the scanner does not resolve.
Note
This file also contains edge cases where metric metadata cannot be accurately extracted by the scanner (e.g., labels determined by runtime logic). Static metrics take priority over generated metrics when both files contain the same metric name.
Edit this file to add metrics that should appear in the documentation but are not scanned from the coder codebase,
or to manually override metrics where the scanner generates incorrect metadata (e.g., missing runtime-determined labels like in agent_scripts_executed_total).
generated_metrics (auto-generated)
Contains metrics extracted from the coder source code by the AST scanner (scanner/scanner.go).
Do not edit this file directly. It is regenerated by running:
make scripts/metricsdocgen/generated_metrics
Updating Metrics Documentation
To regenerate the documentation after code changes:
make docs/admin/integrations/prometheus.md
This will:
- Run the scanner to update
generated_metrics - Merge
metricsandgenerated_metricsmetric files - Update the documentation file