mirror of
https://github.com/coder/coder.git
synced 2026-09-22 05:05:20 +08:00
fix: tighten single tool result byte budget (#26763)
## Problem #26637 caps each locally-executed tool result (built-in, global/deployment MCP, workspace MCP) at a per-result byte budget derived from the model's context window. The budget was `ContextLimit/2 * 4 bytes` — i.e. **half the window at an optimistic 4 bytes/token**. On large-context models that is far too generous. With a `1,000,000`-token `ContextLimit` the per-result cap is **~2 MB**. A user hit exactly this with a chatd (deployment-pinned) MCP tool: the result was truncated to **1,998,709 characters** and still overflowed the prompt. 2 MB of dense text (JSON/logs/code) is ~650k–1M tokens — most or all of the window for a *single* result — so the cap fired but didn't actually prevent the overflow. ## Fix Tighten the two budget constants in `tooltruncate.go`: | constant | before | after | | --- | --- | --- | | `toolResultContextDivisor` | `2` (½ window) | `3` (⅓ window) | | `bytesPerTokenEstimate` | `4` | `3` (conservative) | The budget becomes `ContextLimit/3 * 3 ≈ ContextLimit` bytes: | ContextLimit | before | after | | --- | --- | --- | | 1,000,000 | ~2 MB | ~1 MB | | 200,000 | ~400 KB | ~200 KB | | unknown (≤0) | 64 KB | 64 KB (unchanged) | The 16 KB floor and 64 KB unknown-window default are unchanged. A conservative bytes-per-token estimate is intentional: dense payloads run well under 4 B/tok, so a lower estimate yields a smaller byte budget that is less likely to underestimate the true token cost. No behavioral code paths change — only the two constants and their doc comments. The existing `tooltruncate_internal_test.go` cases derive their expectations from the constants (`LargeWindow`) or exercise the floor/default (`BelowFloor`, `Unknown`), so they remain green. <details> <summary>Investigation notes</summary> Global/deployment MCP tools (`mcpclient.ConnectAll`) are appended to `prepared.Tools` and execute locally via `ExecuteLocalTools → executeTools → executeSingleTool`, so the #26637 cap *does* apply to them for text results (`convertCallResult` joins text content into `resp.Content`). The cap was simply too large: `toolResultByteBudget(ContextLimit)` = `ContextLimit/2*4` ≈ 2 MB for a 1M-token window. Reverse-engineering the reported `1,998,709` truncated characters confirms a `ContextLimit` of ~1,000,000 tokens. Known gaps left for follow-ups (out of scope here): - **Per-step aggregate is unbounded.** MCP tools advertise `Parallel: true` and `executeSingleTool` caps each result independently, so N parallel calls in one step can sum to N × the per-result cap. - **Binary/media `Data` bypasses the cap.** Only the text payload is bounded; `image`/`media`/blob embedded-resource results are base64-encoded untouched in `executeSingleTool`. - **Compaction is reactive.** It is gated on the prior step's reported usage (`latestPromptUsage`), so it can't pre-empt a single large result appended on the current step. </details> --- Generated by Coder Agents on behalf of @kylecarbs.
This commit is contained in:
@@ -7,19 +7,25 @@ import (
|
||||
|
||||
const (
|
||||
// toolResultContextDivisor bounds how much of a model's context
|
||||
// window a single tool result may occupy: at most 1/N of the
|
||||
// window a single tool result may occupy: at most 1/3 of the
|
||||
// window. This caps a single oversized result (most often a large
|
||||
// MCP response) so it cannot overflow the prompt on its own, while
|
||||
// still letting a generous amount of output through. Cumulative
|
||||
// still letting a useful amount of output through. The divisor is
|
||||
// deliberately larger than 2 so the absolute cap stays sane on
|
||||
// large-context (e.g. 1M-token) models, where a more generous
|
||||
// fraction would still admit multi-megabyte results. Cumulative
|
||||
// growth across many results is handled separately by context
|
||||
// compaction.
|
||||
toolResultContextDivisor = 2
|
||||
toolResultContextDivisor = 3
|
||||
|
||||
// bytesPerTokenEstimate converts a token budget into a byte budget.
|
||||
// Tool output is capped before tokenization, so this is a coarse,
|
||||
// provider-agnostic estimate. Roughly 4 bytes per token for typical
|
||||
// text.
|
||||
bytesPerTokenEstimate = 4
|
||||
// provider-agnostic estimate. It is deliberately conservative at ~3
|
||||
// bytes per token: dense payloads (JSON, logs, code, non-ASCII) run
|
||||
// well below 4 bytes per token, so the lower estimate yields a
|
||||
// smaller byte budget that is less likely to underestimate the true
|
||||
// token cost.
|
||||
bytesPerTokenEstimate = 3
|
||||
|
||||
// minToolResultBytes is the floor for the per-result byte budget so
|
||||
// small or unknown context windows still let useful output through.
|
||||
|
||||
Reference in New Issue
Block a user