Compare commits

...
Author SHA1 Message Date
John Choi 0650613122 fix(sdk): count structured read bytes in batch threshold estimate
Third and final instance of the structured-result gap. After the merge
with the batching PR, estimateOutdatedReclaimBytes still ignored
structured ToolOperationResult entries (only text/image/file), so its
pending-bytes counter stayed ~0 for real read_files results and the
128KB batch threshold never committed — batching was inert on production
transcripts (3-way sim showed fix+batch identical to no-fix).

Adds an else-if branch attributing the serialized bytes of an outdated
structured entry, mirroring replaceOutdatedReadContent.

3-way measurement (structured read-heavy, DeepSeek 10x cache pricing):
  A no rewrite (today):  40t $0.1723   120t $1.2395
  B fix + eager:         40t +28%       120t -27%
  C fix + batching:      40t -34%       120t -66%
The fix only pays off batched (B regresses short sessions by breaking
the cache every turn); combined it wins on both token volume and cache
stability.

Adds a regression test (structured stale read crosses a 2KB threshold
and commits); fails before this change.
2026-06-15 14:36:29 -07:00
John Choi f6f7174734 Merge remote-tracking branch 'origin/fix/message-builder-batch-outdated-rewrites' into fix/structured-read-outdated-rewrite 2026-06-15 14:32:10 -07:00
John Choi f120f07f3e fix(sdk): rewrite outdated reads in structured ToolOperationResult results
The outdated-read rewrite only fired for JSON-string or {type:text} tool
results. The runtime stores read_files/search/run_commands/fetch output
as ToolOperationResult[] ({query,result,success}, no type field), which
both array walkers skipped: extractReadLocatorsFromToolResultContent
never parsed locators from them, and replaceOutdatedReadContent never
rewrote them (no-op). The per-entry helpers already understood
{query,result} objects; this routes structured entries through them via
the existing isStructuredToolResultEntry guard.

Impact (structured read-heavy sim, before -> after): tokens -79 to -89%,
long-session cost -27%. Affects ~40% of real read results.

Adds a regression test that fails before this change.
2026-06-15 13:03:42 -07:00
John Choi 31d07cc6f8 perf(sdk): retune outdated-rewrite threshold to 128KB for executor caps
The 64KB default was calibrated before executor-layer output caps landed
(#11480/#11504: read_files/run_commands/search now cap at 48K chars).
With reads bounded at ~48K, 64KB sat awkwardly — one stale read can't
cross it, two overshoot — making it the worst non-extreme threshold in a
post-cap cost sweep.

Re-measured eager vs batched on 48K-capped transcripts (DeepSeek 10x
cache pricing): batching still beats eager 44-61%, confirming the
mechanism remains valuable after the caps (never-rewrite is now +35%
worse in long sessions). 128KB (~2-3 capped reads) is cheapest in both
short and long shapes, ~5-12 points better than the old 64KB.

Bumps LARGE_CONTENT test fixture to ~140KB so single-large-read commit
tests still exceed the raised threshold.
2026-06-15 10:35:27 -07:00
John Choi 8d9f370348 Merge remote-tracking branch 'origin/main' into fix/message-builder-batch-outdated-rewrites
# Conflicts:
#	sdk/packages/core/src/session/services/message-builder.ts
2026-06-15 10:13:01 -07:00
John Choi 3d54e4cff8 fix(sdk): batch orphaned read results and count stale image bytes
Addresses robinnewhouse review (two pre-approval follow-ups):

1. Tool-name lookups went through toolNameByIdCache only, so a
   tool_result orphaned by compaction/rollback (paired tool_use gone)
   was invisible to the batching scan and pruned from committed state —
   reverting its rewrite mid-transcript in exactly the history-shrinking
   case the batching needs to survive. resolveToolName now falls back to
   tool_result.name at all three lookup sites (transform, reindex,
   commit scan).

2. estimateOutdatedReclaimBytes attributed only text/file entries, but
   replaceOutdatedReadContent also replaces stale image siblings
   (flagged by codex too). Image-heavy sessions accrued ~0 pending bytes
   and never crossed the threshold. The estimator now counts stale image
   payload bytes using the same positional marker counting as the
   rewriter (countOutdatedImageEntries).

Both regression tests fail before this change: orphaned result keeps
its committed rewrite through a codec round-trip, and a 4KB stale image
crosses a 2KB threshold that its ~70-byte text marker alone would not.
2026-06-12 09:35:48 -07:00
John Choi edd525d8ba fix(sdk): keep outdated-rewrite batching state across fresh message rebuilds
Addresses review feedback: the runtime provider path rebuilds Message
objects every request (agentMessagesToMessages constructs new literals),
so the identity-based reindex check fails each build and resetIndexes
fires. Clearing committedOutdatedRewrites there (added for the rollback
P1) recounted already-committed bytes as pending on every request — once
the first 64KB committed, every newly-stale small read rewrote
immediately, degenerating to eager behavior in steady state.

committedOutdatedRewrites now survives resetIndexes. Rollback
correctness is preserved without it: the apply-time re-validation is
identity-free, and commitOutdatedRewrites now prunes committed locators
that are no longer outdated in the current index plus entries whose
tool_use_id left the transcript. Both prunes are no-ops in append-only
growth since outdatedness is monotonic.

Adds two regression tests that route messages through the real
agent-message codec round-trip (fresh objects per build, as production):
steady-state deferral of a small newly-stale read after a committed
large one, and rollback restoring full content.
2026-06-11 17:54:26 -07:00
John Choi 4d97c154cb test(sdk): trim redundant comments in rollback regression test 2026-06-11 17:15:57 -07:00
John Choi 6d2d82d57d fix(sdk): drop committed outdated rewrites when history is rolled back
Addresses review P1: committedOutdatedRewrites survived checkpoint
restore/clearHistory (the orchestrator reuses one MessageBuilder), so a
read that became the latest again after rollback stayed rewritten to
'[outdated...]' forever, hiding live file content from the provider.

Two guards: re-validate committed locators against the current index at
apply time, and clear the committed set in resetIndexes — that path only
fires on non-append-only history changes, where the provider prefix is
already broken, so stickiness loses nothing.

Adds a rollback regression test (commit rewrite, restore to before the
re-read, assert full content returns).
2026-06-11 17:08:05 -07:00
John Choi 7022ce4813 Merge branch 'main' into fix/message-builder-batch-outdated-rewrites 2026-06-11 16:07:15 -07:00
John Choi e37066e63f fix(sdk): count only reclaimable locator bytes when batching outdated-read rewrites
Addresses review feedback (greptile P1, codex P2): pendingBytes was
incremented with the whole tool-result block size once per outdated
locator, so multi-file read_files results were overcounted (N stale
locators = N x block bytes), crossing the batch threshold far earlier
than intended and partially defeating the cache-stability guarantee.

Now estimateOutdatedReclaimBytes attributes bytes per stale entry in
the parsed read result (falling back to full text size only when the
whole block is outdated, matching replaceOutdatedReadContent), counted
once per block. Adds a multi-locator regression test where a 3-file
read result is invalidated file-by-file and must only commit when the
actual reclaimable bytes cross the threshold.

Real-session replay improved from 12.6% to 19.7% net-token reduction
with the accurate counting (commits defer longer, breaks amortize
better).
2026-06-11 11:13:26 -07:00
John Choi d42b9aa48e fix(sdk): batch outdated-read rewrites in MessageBuilder to preserve provider prefix caches
MessageBuilder previously rewrote stale read_files results to
'[outdated - see the latest file content]' eagerly on every re-read.
Each rewrite mutates bytes in the middle of the provider-facing
transcript, invalidating provider prefix caches (DeepSeek/Anthropic/
MiniMax-style) from that message to the end of the conversation. Agents
re-read files constantly (read -> edit -> verify), so long sessions paid
full uncached input price on most requests.

Now pending outdated rewrites accumulate and only commit once the total
reclaimable bytes cross a 64KB threshold, then apply as a single batch
(one cache break amortized over a large context saving). Committed
rewrites are sticky so subsequent requests stay byte-stable.
2026-06-11 10:01:44 -07:00
3 changed files with 759 additions and 16 deletions
@@ -0,0 +1,471 @@
import type { Message } from "@cline/shared";
import { describe, expect, it } from "vitest";
import {
agentMessagesToMessages,
messagesToAgentMessages,
} from "../../runtime/config/agent-message-codec";
import { MessageBuilder } from "./message-builder";
/** Mimics the runtime provider-request path, which rebuilds fresh Message
* objects every request via the agent-message codec. */
function codecRoundTrip(messages: Message[]): Message[] {
return agentMessagesToMessages(messagesToAgentMessages(messages));
}
const SMALL_CONTENT = (v: number) => `export const x = ${v};\n`.repeat(40); // ~1KB
const LARGE_CONTENT = (v: number) => `export const x = ${v};\n`.repeat(7_000); // ~140KB (> 128KB threshold)
function readToolUse(id: string, path = "src/a.ts"): Message {
return {
role: "assistant",
content: [
{ type: "text", text: `reading ${path} (${id})` },
{
type: "tool_use",
id,
name: "read_files",
input: { files: [{ path }] },
},
],
};
}
function readToolResult(
id: string,
content: string,
path = "src/a.ts",
): Message {
return {
role: "user",
content: [
{
type: "tool_result",
tool_use_id: id,
name: "read_files",
content: JSON.stringify([{ path, result: content }]),
},
],
};
}
function serializedBlockAt(result: Message[], index: number): string {
return JSON.stringify(result[index]);
}
describe("MessageBuilder outdated-read rewrite batching (prefix-cache stability)", () => {
it("defers small outdated rewrites so request N stays a byte-stable prefix of request N+1", () => {
const builder = new MessageBuilder();
const base: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", SMALL_CONTENT(1)),
];
const reqA = builder.buildForApi(base);
const firstResultA = serializedBlockAt(reqA, 2);
expect(firstResultA).toContain("export const x = 1;");
// Re-read the same file: ~1KB reclaimable, below the batch threshold.
const withReread: Message[] = [
...base,
readToolUse("t2"),
readToolResult("t2", SMALL_CONTENT(2)),
];
const reqB = builder.buildForApi(withReread);
// The earlier read result must be byte-identical: no mid-transcript
// mutation, so the provider prefix cache stays valid.
expect(serializedBlockAt(reqB, 2)).toEqual(firstResultA);
expect(serializedBlockAt(reqB, 2)).not.toContain("outdated");
});
it("commits batched rewrites once reclaimable bytes cross the threshold", () => {
const builder = new MessageBuilder();
const base: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
// First build: t1 is outdated (~140KB reclaimable > 128KB threshold),
// so the rewrite commits immediately.
const reqA = builder.buildForApi(base);
expect(serializedBlockAt(reqA, 2)).toContain(
"outdated - see the latest file content",
);
expect(serializedBlockAt(reqA, 2)).not.toContain("export const x = 1;");
// Latest read is untouched.
expect(serializedBlockAt(reqA, 4)).toContain("export const x = 2;");
});
it("keeps committed rewrites sticky across subsequent builds", () => {
const builder = new MessageBuilder();
const base: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
const reqA = builder.buildForApi(base);
const rewrittenA = serializedBlockAt(reqA, 2);
expect(rewrittenA).toContain("outdated");
// Append unrelated activity; the committed rewrite must reproduce
// byte-identically so the prefix remains stable.
const extended: Message[] = [
...base,
{
role: "assistant",
content: [
{ type: "text", text: "running tests" },
{
type: "tool_use",
id: "t3",
name: "bash",
input: { command: "npm test" },
},
],
},
{
role: "user",
content: [
{
type: "tool_result",
tool_use_id: "t3",
name: "bash",
content: "tests passed",
},
],
},
];
const reqB = builder.buildForApi(extended);
expect(serializedBlockAt(reqB, 2)).toEqual(rewrittenA);
});
it("accumulates multiple small outdated reads and commits them together", () => {
// Each stale read result is ~850 bytes. With a 1.5KB threshold, one
// stale read stays pending; the second stale read pushes the batch
// over and both rewrite at once.
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
1_500,
);
const messages: Message[] = [{ role: "user", content: "task" }];
for (let i = 1; i <= 3; i++) {
messages.push(readToolUse(`t${i}`));
messages.push(readToolResult(`t${i}`, SMALL_CONTENT(i)));
const result = builder.buildForApi([...messages]);
const staleCount = result.filter((m) =>
JSON.stringify(m).includes("outdated - see the latest file content"),
).length;
if (i < 3) {
expect(staleCount).toBe(0); // pending, below threshold
} else {
expect(staleCount).toBe(2); // t1 + t2 committed together
}
}
});
it("counts multi-file read results per outdated locator, not per whole block", () => {
// One read_files call returns files A, B, C (~850 bytes each entry).
// Only A is later re-read, so the reclaimable amount is ~850 bytes —
// NOT the ~2.5KB whole-block size. With a 2KB threshold, a whole-block
// (over)count would commit immediately; correct per-locator attribution
// must defer.
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
2_000,
);
const multiReadUse: Message = {
role: "assistant",
content: [
{ type: "text", text: "reading three files" },
{
type: "tool_use",
id: "t1",
name: "read_files",
input: {
files: [
{ path: "src/a.ts" },
{ path: "src/b.ts" },
{ path: "src/c.ts" },
],
},
},
],
};
const multiReadResult: Message = {
role: "user",
content: [
{
type: "tool_result",
tool_use_id: "t1",
name: "read_files",
content: JSON.stringify([
{ path: "src/a.ts", result: SMALL_CONTENT(1) },
{ path: "src/b.ts", result: SMALL_CONTENT(2) },
{ path: "src/c.ts", result: SMALL_CONTENT(3) },
]),
},
],
};
const messages: Message[] = [
{ role: "user", content: "task" },
multiReadUse,
multiReadResult,
readToolUse("t2", "src/a.ts"),
readToolResult("t2", SMALL_CONTENT(4), "src/a.ts"),
];
const deferred = builder.buildForApi(messages);
// Only ~850 bytes (entry A) is reclaimable: must stay below the 2KB
// threshold and remain pending. Whole-block counting (~2.5KB) would
// wrongly commit here.
expect(JSON.stringify(deferred)).not.toContain("outdated");
// Re-read B as well: reclaimable is now A+B (~1.7KB)... still below.
const withB: Message[] = [
...messages,
readToolUse("t3", "src/b.ts"),
readToolResult("t3", SMALL_CONTENT(5), "src/b.ts"),
];
expect(JSON.stringify(builder.buildForApi(withB))).not.toContain(
"outdated",
);
// Re-read C too: A+B+C (~2.5KB) crosses the 2KB threshold; all three
// stale entries in the multi-read result commit together.
const withC: Message[] = [
...withB,
readToolUse("t4", "src/c.ts"),
readToolResult("t4", SMALL_CONTENT(6), "src/c.ts"),
];
const committed = builder.buildForApi(withC);
const multiBlock = JSON.stringify(committed[2]);
expect(multiBlock).toContain("outdated - see the latest file content");
expect(multiBlock).not.toContain("export const x = 1;");
expect(multiBlock).not.toContain("export const x = 2;");
expect(multiBlock).not.toContain("export const x = 3;");
});
it("never rewrites when the threshold is disabled via a huge value", () => {
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
Number.POSITIVE_INFINITY,
);
const messages: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
const result = builder.buildForApi(messages);
expect(JSON.stringify(result)).not.toContain("outdated");
});
it("restores full content after history is rolled back past the re-read", () => {
const builder = new MessageBuilder();
const t1Only: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
];
const withReread: Message[] = [
...t1Only,
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
const reqA = builder.buildForApi(withReread);
expect(serializedBlockAt(reqA, 2)).toContain("outdated");
// Same builder instance, history rolled back past the re-read.
const reqB = builder.buildForApi(t1Only);
expect(serializedBlockAt(reqB, 2)).not.toContain("outdated");
expect(serializedBlockAt(reqB, 2)).toContain("export const x = 1;");
});
it("keeps batching state when the runtime rebuilds fresh message objects per request", () => {
const builder = new MessageBuilder();
const history: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1", "src/big.ts"),
readToolResult("t1", LARGE_CONTENT(1), "src/big.ts"),
readToolUse("t2", "src/big.ts"),
readToolResult("t2", LARGE_CONTENT(2), "src/big.ts"),
];
// Request A: t1 (~80KB stale) crosses the threshold and commits.
const reqA = builder.buildForApi(codecRoundTrip(history));
expect(JSON.stringify(reqA[2])).toContain("outdated");
// Request B: a ~1KB re-read makes t3 newly stale. The committed 80KB
// must NOT be recounted as pending, so t3 stays deferred.
history.push(
readToolUse("t3", "src/small.ts"),
readToolResult("t3", SMALL_CONTENT(1), "src/small.ts"),
readToolUse("t4", "src/small.ts"),
readToolResult("t4", SMALL_CONTENT(2), "src/small.ts"),
);
const reqB = builder.buildForApi(codecRoundTrip(history));
expect(JSON.stringify(reqB[2])).toContain("outdated"); // t1 sticky
expect(JSON.stringify(reqB[6])).not.toContain("outdated"); // t3 deferred
expect(JSON.stringify(reqB[6])).toContain("export const x = 1;");
});
it("restores full content after rollback even with fresh message objects", () => {
const builder = new MessageBuilder();
const t1Only: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
];
const withReread: Message[] = [
...t1Only,
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
const reqA = builder.buildForApi(codecRoundTrip(withReread));
expect(JSON.stringify(reqA[2])).toContain("outdated");
const reqB = builder.buildForApi(codecRoundTrip(t1Only));
expect(JSON.stringify(reqB[2])).not.toContain("outdated");
expect(JSON.stringify(reqB[2])).toContain("export const x = 1;");
});
it("keeps committed rewrites applied when compaction drops the paired tool_use", () => {
const builder = new MessageBuilder();
const withReread: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", LARGE_CONTENT(1)),
readToolUse("t2"),
readToolResult("t2", LARGE_CONTENT(2)),
];
const reqA = builder.buildForApi(codecRoundTrip(withReread));
const rewrittenA = JSON.stringify(reqA[2]);
expect(rewrittenA).toContain("outdated");
// Compaction drops t1's tool_use but keeps the stale result, and the
// runtime delivers fresh objects so the name index rebuilds without
// t1. The rewrite must stay applied (via tool_result.name) so the
// suffix after the orphaned block is not mutated back to full content.
const compacted: Message[] = [
withReread[0],
{
role: "assistant",
content: [{ type: "text", text: "reading src/a.ts (t1)" }],
},
...withReread.slice(2),
];
const reqB = builder.buildForApi(codecRoundTrip(compacted));
expect(JSON.stringify(reqB[2])).toContain("outdated");
expect(JSON.stringify(reqB[2])).not.toContain("export const x = 1;");
});
it("counts stale image payload bytes toward the batch threshold", () => {
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
2_000,
);
const imageReadResult: Message = {
role: "user",
content: [
{
type: "tool_result",
tool_use_id: "t1",
name: "read_files",
content: [
{
type: "text",
text: JSON.stringify([
{ path: "img/shot.png", result: "Successfully read image" },
]),
},
{ type: "image", data: "A".repeat(4_000), mediaType: "image/png" },
],
},
],
};
const messages: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1", "img/shot.png"),
imageReadResult,
readToolUse("t2", "img/shot.png"),
readToolResult("t2", SMALL_CONTENT(1), "img/shot.png"),
];
// Text marker alone is ~70 bytes — below the 2KB threshold. The 4KB
// base64 payload must count, committing the batch.
const result = builder.buildForApi(messages);
const block = JSON.stringify(result[2]);
expect(block).toContain("outdated");
expect(block).not.toContain("AAAA");
});
it("counts structured ToolOperationResult bytes toward the batch threshold", () => {
// 2KB threshold; the stale structured read is ~4KB so it must cross it
// and commit. Regression for estimateOutdatedReclaimBytes ignoring
// structured {query, result} entries (which made batching never fire).
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
2_000,
);
const structuredResult = (id: string, body: string): Message => ({
role: "user",
content: [
{
type: "tool_result",
tool_use_id: id,
name: "read_files",
content: [
{ query: "src/a.ts", result: body, success: true },
] as unknown as never,
},
],
});
const messages: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1", "src/a.ts"),
structuredResult("t1", "x".repeat(4_000)),
readToolUse("t2", "src/a.ts"),
structuredResult("t2", "fresh"),
];
const result = builder.buildForApi(messages);
const block = JSON.stringify(result[2]);
expect(block).toContain("outdated");
expect(block).not.toContain("xxxx");
});
it("rewrites eagerly when threshold is 0 (legacy behavior)", () => {
const builder = new MessageBuilder(
undefined,
undefined,
undefined,
undefined,
0,
);
const messages: Message[] = [
{ role: "user", content: "task" },
readToolUse("t1"),
readToolResult("t1", SMALL_CONTENT(1)),
readToolUse("t2"),
readToolResult("t2", SMALL_CONTENT(2)),
];
const result = builder.buildForApi(messages);
expect(serializedBlockAt(result, 2)).toContain("outdated");
});
});
@@ -916,4 +916,36 @@ describe("MessageBuilder with structured ToolOperationResult content", () => {
expect(serialized).toContain(HEAD_MARKER);
expect(serialized).toContain(TAIL_MARKER);
});
it("rewrites an outdated structured read_files result when the file is re-read", () => {
// minOutdatedRewriteBytes = 0 commits immediately; this test targets the
// structured-entry locator/rewrite path, not the batching threshold.
const builder = new MessageBuilder(undefined, undefined, undefined, undefined, 0);
const messages: Message[] = [
toolUseMessage("call_1", "read_files", {
files: [{ path: "/tmp/a.ts" }],
}),
structuredToolResultMessage("call_1", "read_files", [
{
query: "/tmp/a.ts",
result: `${HEAD_MARKER}old-content${TAIL_MARKER}`,
success: true,
},
]),
toolUseMessage("call_2", "read_files", {
files: [{ path: "/tmp/a.ts" }],
}),
structuredToolResultMessage("call_2", "read_files", [
{ query: "/tmp/a.ts", result: "fresh-content", success: true },
]),
];
const result = builder.buildForApi(messages);
// The earlier read (call_1) is now outdated and must be rewritten.
const firstResult = JSON.stringify(result[1]);
expect(firstResult).toContain("[outdated");
expect(firstResult).not.toContain("old-content");
// The latest read (call_2) keeps its content.
expect(JSON.stringify(result[3])).toContain("fresh-content");
});
});
@@ -28,6 +28,17 @@ import {
const DEFAULT_MAX_TOOL_RESULT_CHARS = 50_000;
const DEFAULT_MAX_TOTAL_TEXT_BYTES = 6_000_000;
const MIN_TOTAL_BUDGET_TOOL_RESULT_BYTES = 8_000;
/**
* Outdated-read rewrites mutate tool results in the middle of the transcript,
* which invalidates provider prefix caches from that point onward. Instead of
* rewriting eagerly on every re-read, batch rewrites: defer them until the
* total reclaimable bytes across pending outdated reads crosses this
* threshold, then apply them all at once (one cache break amortized over a
* large context saving). 128KB ≈ 32K tokens, ~2-3 executor-capped reads
* (read_files caps at 48K chars), so a single re-read never breaks the
* cache alone. Set to 0 to rewrite eagerly.
*/
const DEFAULT_MIN_OUTDATED_REWRITE_BYTES = 131_072;
const TARGET_TOOL_NAMES = new Set([
"read",
"read_files",
@@ -75,16 +86,28 @@ export class MessageBuilder {
string
>();
private readResultLocatorCache = new WeakMap<object, ReadLocator[]>();
/**
* Committed outdated-read rewrites (locator keys per tool_use_id),
* applied on every build so the transcript stays byte-stable between
* batch commits. Survives resetIndexes: the runtime rebuilds fresh
* Message objects per request, so identity-based reindexing cannot be
* trusted to detect real history changes. Stale entries are instead
* re-validated against the current index at apply time and pruned in
* commitOutdatedRewrites.
*/
private readonly committedOutdatedRewrites = new Map<string, Set<string>>();
constructor(
private readonly maxToolResultChars = DEFAULT_MAX_TOOL_RESULT_CHARS,
private readonly targetToolNames = TARGET_TOOL_NAMES,
private readonly maxTotalTextBytes = DEFAULT_MAX_TOTAL_TEXT_BYTES,
private readonly mediaBudget: MediaBudgetOptions = {},
private readonly minOutdatedRewriteBytes = DEFAULT_MIN_OUTDATED_REWRITE_BYTES,
) {}
buildForApi(messages: Message[]): Message[] {
this.reindex(messages);
this.commitOutdatedRewrites(messages);
const repairedMessages = this.addMissingToolResults(messages);
const prepared = repairedMessages.map((message) => {
@@ -141,14 +164,17 @@ export class MessageBuilder {
return block;
}
const toolName = this.toolNameByIdCache.get(block.tool_use_id);
const toolName = this.resolveToolName(block);
let nextContent = block.content;
if (this.isReadTool(toolName) && block.is_error !== true) {
const locators = this.getReadLocators(block);
if (locators.length > 0) {
const outdated = locators.filter((locator) =>
this.isOutdatedReadLocator(locator, block.tool_use_id),
const committed = this.committedOutdatedRewrites.get(block.tool_use_id);
if (committed && committed.size > 0) {
const locators = this.getReadLocators(block);
const outdated = locators.filter(
(locator) =>
committed.has(this.toReadLocatorKey(locator)) &&
this.isOutdatedReadLocator(locator, block.tool_use_id),
);
if (outdated.length > 0) {
nextContent = this.replaceOutdatedReadContent(nextContent, outdated);
@@ -197,7 +223,7 @@ export class MessageBuilder {
}
}
} else if (block.type === "tool_result") {
const toolName = this.toolNameByIdCache.get(block.tool_use_id);
const toolName = this.resolveToolName(block);
if (!this.isReadTool(toolName) || block.is_error === true) {
continue;
}
@@ -222,6 +248,189 @@ export class MessageBuilder {
messages.length > 0 ? messages[messages.length - 1] : undefined;
}
/**
* Decide which outdated read results to rewrite for this build.
*
* Eagerly rewriting on every re-read mutates mid-transcript bytes and
* invalidates the provider prefix cache from that message to the end of
* the conversation. Instead, accumulate pending outdated locators and
* only commit them (all at once) when the total reclaimable bytes cross
* `minOutdatedRewriteBytes`. Committed rewrites are sticky so the
* serialized transcript stays stable on subsequent requests.
*/
private commitOutdatedRewrites(messages: Message[]): void {
const pending = new Map<string, Set<string>>();
const seenToolUseIds = new Set<string>();
let pendingBytes = 0;
for (const message of messages) {
if (!Array.isArray(message.content)) {
continue;
}
for (const block of message.content) {
if (block.type !== "tool_result" || block.is_error === true) {
continue;
}
const toolName = this.resolveToolName(block);
if (!this.isReadTool(toolName)) {
continue;
}
seenToolUseIds.add(block.tool_use_id);
const committed = this.committedOutdatedRewrites.get(block.tool_use_id);
const newKeys = new Set<string>();
const validKeys = new Set<string>();
for (const locator of this.getReadLocators(block)) {
const key = this.toReadLocatorKey(locator);
if (!this.isOutdatedReadLocator(locator, block.tool_use_id)) {
continue;
}
validKeys.add(key);
if (!committed?.has(key)) {
newKeys.add(key);
}
}
// Prune committed keys no longer outdated (history rollback);
// a no-op in append-only growth.
if (committed) {
for (const key of committed) {
if (!validKeys.has(key)) {
committed.delete(key);
}
}
if (committed.size === 0) {
this.committedOutdatedRewrites.delete(block.tool_use_id);
}
}
if (newKeys.size === 0) {
continue;
}
let keys = pending.get(block.tool_use_id);
if (!keys) {
keys = new Set<string>();
pending.set(block.tool_use_id, keys);
}
for (const key of newKeys) {
keys.add(key);
}
// Count only the bytes the rewrite will actually reclaim for the
// newly-outdated locators (not the whole block), once per block.
pendingBytes += this.estimateOutdatedReclaimBytes(
block.content,
newKeys,
);
}
}
for (const toolUseId of this.committedOutdatedRewrites.keys()) {
if (!seenToolUseIds.has(toolUseId)) {
this.committedOutdatedRewrites.delete(toolUseId);
}
}
if (pending.size === 0 || pendingBytes < this.minOutdatedRewriteBytes) {
return;
}
for (const [toolUseId, keys] of pending) {
let committed = this.committedOutdatedRewrites.get(toolUseId);
if (!committed) {
committed = new Set<string>();
this.committedOutdatedRewrites.set(toolUseId, committed);
}
for (const key of keys) {
committed.add(key);
}
}
}
/**
* Estimate the bytes the outdated rewrite would reclaim from a block for
* the given locator keys. Attribution is per-entry where the content is
* structured (parsed read results / file entries); unattributable text
* falls back to its full size only when every locator in the block is
* outdated (matching replaceOutdatedReadContent, which replaces whole
* unparseable text entries).
*/
private estimateOutdatedReclaimBytes(
content: ToolResultContent["content"],
outdatedKeys: ReadonlySet<string>,
): number {
const allLocators = this.extractReadLocatorsFromToolResultContent(content);
const blockFullyOutdated =
allLocators.length > 0 &&
allLocators.every((locator) =>
outdatedKeys.has(this.toReadLocatorKey(locator)),
);
const attributeText = (text: string): number => {
let parsed: unknown;
try {
parsed = JSON.parse(text);
} catch {
return blockFullyOutdated || allLocators.length === 0
? utf8ByteLength(text)
: 0;
}
const entries = Array.isArray(parsed) ? parsed : [parsed];
let total = 0;
for (const entry of entries) {
const locator = this.extractLocatorFromResultEntry(entry);
if (locator && outdatedKeys.has(this.toReadLocatorKey(locator))) {
total += utf8ByteLength(JSON.stringify(entry));
}
}
return total;
};
if (typeof content === "string") {
return attributeText(content);
}
// Stale image reads are replaced positionally (see
// replaceOutdatedReadContent), so count image sibling bytes the same way.
const outdatedKeySet = new Set(outdatedKeys);
let outdatedImageCount = 0;
for (const entry of content) {
if (entry.type === "text") {
outdatedImageCount += this.countOutdatedImageEntries(
entry.text,
outdatedKeySet,
);
}
}
let total = 0;
for (const entry of content) {
if (entry.type === "text") {
total += attributeText(entry.text);
} else if (entry.type === "image") {
if (outdatedImageCount > 0) {
outdatedImageCount -= 1;
total += utf8ByteLength(entry.data);
}
} else if (isStructuredToolResultEntry(entry)) {
// Structured ToolOperationResult[] entry ({query, result, ...}):
// count its bytes when its locator is outdated, matching how
// replaceOutdatedReadContent rewrites it.
const locator = this.extractLocatorFromResultEntry(entry);
if (locator && outdatedKeys.has(this.toReadLocatorKey(locator))) {
total += utf8ByteLength(JSON.stringify(entry));
}
} else if (entry.type === "file") {
if (
outdatedKeys.has(
this.toReadLocatorKey({
path: entry.path,
startLine: null,
endLine: null,
}),
)
) {
total += utf8ByteLength(entry.content);
}
}
}
return total;
}
private addMissingToolResults(messages: Message[]): Message[] {
const existingToolResultIds = this.collectToolResultIds(messages);
const repaired: Message[] = [];
@@ -495,12 +704,20 @@ export class MessageBuilder {
return this.tryParseReadLocators(content);
}
for (const entry of content) {
if (entry.type !== "text") {
if (entry.type === "text") {
const locators = this.tryParseReadLocators(entry.text);
if (locators.length > 0) {
return locators;
}
continue;
}
const locators = this.tryParseReadLocators(entry.text);
if (locators.length > 0) {
return locators;
// Structured ToolOperationResult[] entries (read_files/search/etc.)
// arrive as plain {query, result, ...} objects with no type field.
if (isStructuredToolResultEntry(entry)) {
const locator = this.extractLocatorFromResultEntry(entry);
if (locator) {
return [locator];
}
}
}
return [];
@@ -672,14 +889,22 @@ export class MessageBuilder {
text: OUTDATED_FILE_CONTENT,
} satisfies TextContent;
}
if (entry.type !== "text") {
return entry;
if (entry.type === "text") {
const replaced = this.replaceOutdatedInString(entry.text, outdatedKeys);
if (replaced === null) {
return { ...entry, text: OUTDATED_FILE_CONTENT };
}
return replaced === entry.text ? entry : { ...entry, text: replaced };
}
const replaced = this.replaceOutdatedInString(entry.text, outdatedKeys);
if (replaced === null) {
return { ...entry, text: OUTDATED_FILE_CONTENT };
// Structured ToolOperationResult[] entry: rewrite its result/content
// field in place when its locator is outdated.
if (isStructuredToolResultEntry(entry)) {
return this.replaceOutdatedReadEntry(
entry,
outdatedKeys,
) as typeof entry;
}
return replaced === entry.text ? entry : { ...entry, text: replaced };
return entry;
});
}
@@ -761,6 +986,21 @@ export class MessageBuilder {
return !!toolName && READ_TOOL_NAMES.has(toolName);
}
/**
* Tool results can outlive their paired tool_use (compaction/rollback),
* so fall back to the name on the result itself when the id lookup
* misses.
*/
private resolveToolName(block: ToolResultContent): string | undefined {
const cached = this.toolNameByIdCache.get(block.tool_use_id);
if (cached !== undefined) {
return cached;
}
return typeof block.name === "string" && block.name.length > 0
? block.name.toLowerCase()
: undefined;
}
private shouldTruncateTool(toolName: string | undefined): boolean {
return !!toolName && this.targetToolNames.has(toolName);
}