feat: cap tool output to fit the model context window (#26637)

## Problem

Local tool results were persisted and replayed to the model verbatim,
with no size cap. A single oversized result, most often a multi-megabyte
response from an MCP tool, overflows the prompt on the next request.
Every retry rebuilds the same history and fails the same way, leaving
the chat wedged in `error`. Auto-compaction is reactive (token usage is
only known after a response), so it can't catch a single result that
blows the very next request.

## Fix

Cap every locally-executed tool result at its single choke point,
`executeSingleTool` in `chatloop`, so the cap covers built-in tools,
**global (deployment-pinned) MCP**, **workspace MCP**, and provider
runners uniformly. Because this runs before the result is published to
the live stream and before it is committed, the SSE preview, the
persisted message, and the model replay all see the same bounded output.

The budget is token-aware: a single tool result may use at most half the
model's context window (`~4 bytes/token`), with a `16KB` floor and a
`64KB` default when the window is unknown. Truncation keeps the head and
tail of the output and replaces the middle with a marker telling the
model how much was removed and to narrow its query; it is UTF-8 safe and
never exceeds the budget. Binary media `Data` is passed through
untouched (only the text payload is bounded).

A `coderd_chatd_tool_result_truncated_total{provider,model,tool_name}`
counter and a warning log record each truncation.

## Out of scope

- Provider-executed results (e.g. web search) arrive via the stream, not
`executeSingleTool`.
- Dynamic/external tool results submitted through the `/tool-results`
API are validated as JSON elsewhere.
- Cumulative growth across many results is still handled by context
compaction; this change only bounds any single result.

<details>
<summary>Implementation notes</summary>

- New `coderd/x/chatd/chatloop/tooltruncate.go`:
`toolResultByteBudget(contextLimitTokens)` and
`truncateToolResultText(text, maxBytes)` (pure, unit-tested).
- `chatloop.go`: added `ContextLimit` to `ExecuteLocalToolsOptions`;
threaded a computed byte budget through `executeTools` into
`executeSingleTool`, where `resp.Content` is capped for the text,
media-text, and error branches.
- `generation.go`: passes `ContextLimit: prepared.ContextLimitFallback`
(the model's configured context limit).
- `metrics.go`: new `ToolResultTruncatedTotal` counter +
`RecordToolResultTruncated`.
- Tunable knobs live as constants in `tooltruncate.go`
(`toolResultContextDivisor = 2`, `bytesPerTokenEstimate`,
`minToolResultBytes`, `defaultToolResultBytes`).

Verified: `go build ./coderd/x/chatd/...`, `go test
./coderd/x/chatd/chatloop/...`, and the `chatd` test binary compiles.

</details>

---

Resolves CODAGT-678

Generated by Coder Agents on behalf of @kylecarbs.
This commit is contained in:
Kyle Carberry
2026-06-24 09:16:38 -06:00
committed by GitHub
parent 68808c015e
commit 32217259b7
8 changed files with 266 additions and 4 deletions
+3
View File
@@ -298,6 +298,9 @@ coderd_chatd_tool_errors_total{provider="",model="",tool_name=""} 0
# HELP coderd_chatd_tool_result_size_bytes Size in bytes of each tool execution result.
# TYPE coderd_chatd_tool_result_size_bytes histogram
coderd_chatd_tool_result_size_bytes{provider="",model="",tool_name=""} 0
# HELP coderd_chatd_tool_result_truncated_total Total tool results truncated to fit the model context window.
# TYPE coderd_chatd_tool_result_truncated_total counter
coderd_chatd_tool_result_truncated_total{provider="",model="",tool_name=""} 0
# HELP coderd_chatd_ttft_seconds Time-to-first-token: wall time from LLM request to first streamed chunk.
# TYPE coderd_chatd_ttft_seconds histogram
coderd_chatd_ttft_seconds{provider="",model=""} 0