mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
fix(copilot): stop the special-tag parser discarding text it cannot resolve (#5952)
* fix(copilot): render unclosed special tags as text on completed messages
The special-tag parser suppressed everything after an opening tag with no
close, even after streaming ended — so an assistant message that merely
MENTIONED `<workspace_resource>` in prose lost its entire remainder in the
UI. The stream itself completed fine; only the render truncated.
Suppression now applies only while streaming. A completed message can never
finish an unclosed tag, so the marker was literal text and the remainder is
rendered as-is.
Originally written alongside the docs/ VFS work and reverted at review time
to keep that PR to one concern; this restores it on its own.
* improvement(copilot): show text as soon as an unclosed tag cannot resolve
Restoring the text only at end of stream left a real gap: once the model
mentions a tag in prose, everything after it stays invisible for the rest of
the stream and reappears in one jump when streaming stops. On a long reply
that is most of the message.
Two properties let us decide much earlier, both conservative — they only fire
on content that could not have parsed:
- Tags never nest, so any special-tag marker inside the body (opening OR
closing, not just a mismatched close) proves the opener was literal text.
- JSON-bodied tags must start with `{` or `[`, so the first non-space
character settles it — prose after the marker is caught on the very next
chunk rather than at the end.
The second is per-tag, not global: `thinking` bodies are prose by design
(parseTextTagBody), so the JSON rule cannot apply there and only the nesting
rule can rescue it. A test documents that remaining gap rather than leaving
it implicit.
A false positive is cheap by construction: text shows early and the
end-of-stream parse still produces the correct final render.
* fix(copilot): ignore tag syntax quoted inside a JSON tag body
The nesting rule treated any tag marker in the body as proof the opener was
literal text. But a JSON body can legitimately quote tag syntax — a
`<question>` asking which tag to use, a `<workspace_resource>` whose title
mentions one — and that is exactly the "model explains its own tags"
situation this whole fix exists for.
Bailing there is the one expensive false positive in the design: the raw JSON
renders and STAYS for the rest of the stream, then snaps into a card when the
real close arrives. More jarring than the bug being fixed.
For JSON-bodied tags the nesting scan now runs over a copy with string
literals blanked (escape-aware, and tolerant of the unterminated trailing
string that is normal mid-stream). Markers in real body position still count.
`thinking` is unaffected — its body is prose, so there are no strings to
confuse it.
Tests cover both halves: the streaming case does not bail, and the same body
resolves to a question card once it closes.
* fix(copilot): keep prose a mispaired tag would swallow, and bail on dead JSON
Two more real cases from traces, both of which the earlier heuristics missed.
1. A MATCHED pair whose body fails to parse was silently dropped, cursor and
all. When the model explains tag syntax and ends with a backticked example
containing a real closing tag, that example closes an EARLIER opener and
three paragraphs become the "body" — which is not valid JSON, so the whole
span vanished and the render resumed mid-sentence (trace b095e080).
Now emitted verbatim, but only when the body contains a tag-shaped marker,
which is what shows the pairing was wrong. A marker-free body that merely
fails validation is a genuinely malformed agent payload and keeps being
dropped — an existing test asserts that deliberately, and showing the user
raw JSON there would be a regression.
2. The JSON heuristic only checked the FIRST character, so
`{"type":"file"}</workspac and then prose...` looked viable forever: it
opens with `{`, and the truncated close is not a marker any rule can see
(trace afbeefd0). Track depth instead — once the top-level value closes,
any non-whitespace after it is fatal, so the stray character decides it
immediately. String contents are blanked first so braces inside strings do
not skew the count.
* fix(copilot): keep prose a tag wrapped instead of a JSON payload
Third real case from a trace (1206fd8a): `<workspace_resource>the gmail-agent
workflow</workspace_resource>` — a matched pair whose body is plain prose. The
sentence rendered as "...once I wired up to handle the welcome sequence" with
its subject silently removed.
The marker test could not fire (prose contains no tag markers), so it fell to
the deliberate drop-malformed-payload path. But that path exists for an agent
emitting BROKEN JSON, not for a tag wrapping prose.
The distinction is whether the body was ever an attempted payload, which
isViableJsonPrefix already answers: `{"type":"single_select"}` is a well-formed
JSON value failing its shape guard and keeps being dropped; `the gmail-agent
workflow` was never a payload and is emitted.
* refactor(copilot): resolve each special tag through four named outcomes
parseSpecialTags had grown five inline branches, each added for a specific
malformation found in a trace. The shape made "drop it" the implicit
fallback, which is how spans that were never malformed payloads ended up
silently swallowed.
Extracts resolveTagAt, returning one of four named outcomes — segment,
literal, discard, pending — so each decision is explicit and the main loop
just dispatches on it.
Fixes a latent bug the old shape hid: rejecting an unclosed tag ran `break`,
abandoning the rest of the message, so a genuinely valid tag after a literal
mention was never parsed. Resolution now resumes just past the rejected
opener and scanning continues. Test added.
Two behavior notes:
- Rejected spans are emitted in smaller pieces. The renderer concatenates
adjacent text segments into one markdown string, so this is display-neutral;
the two tests that asserted exact segment arrays now assert joined text.
- Each opener is judged on its own evidence. Previously one verdict ended the
whole parse, so a nested opener released everything; now the outer is
released immediately and the inner is a fresh candidate that can still hold
mid-stream. It resolves at end of stream either way.
* test(copilot): pin the no-closing-tag case as lossless
Fourth malformation from a trace (220cc02d): the model wrote an opening tag
with valid JSON and no closing tag of any kind. No marker rule can fire —
there is no marker — but the JSON value completes and prose follows, which
the depth rule settles at the first space after the `}`.
Already handled by that rule; this pins it. Asserted as lossless rather than
merely visible: mid-stream and complete, every character of the message
survives, so nothing waits for the stream to end.
* fix(copilot): stop the literal path flashing payloads and rescanning the buffer
Review round on the four-outcome rewrite. Three defects in how an opener is
judged, plus the cost of judging it.
A valid tag showed its raw payload as text while its closing marker streamed
in. The JSON value closes at the `}`, so a half-arrived `</opt` read as stray
trailing content and settled the tag as unresolvable. Every JSON-bodied tag hit
this, and most replies carry a trailing <options> block. dropArrivingClose now
ignores a trailing fragment that could still grow into this tag's own close.
Evidence that a close is genuinely wrong still lands immediately: a misspelled
`</workflow_resource>` is not a prefix of `</workspace_resource>`, and a
truncated `</workspac` stops being one the moment prose follows it.
bodyIsLiteralText scanned the raw body for tag markers while its streaming
counterpart blanked JSON string literals first. A payload that failed its shape
guard and legitimately quoted tag syntax was therefore called literal text and
rendered as raw JSON, which is what discard exists to prevent. Both paths now
judge the same blanked body.
An opener whose own close was misspelled reached forward and matched the NEXT
tag's close, swallowing a valid resource into one literal span and destroying
its chip. When markers in the body prove the matched close belongs to a
different opener, resolution resumes past the opener so the interior is
rescanned and the inner tag still renders.
Cost: resuming past a rejected opener instead of abandoning the message made
each parse O(openers x length), and the parse re-runs for every streamed chunk.
A 40KB reply repeatedly mentioning a tag name cost 7.3s of blocked main thread.
The unclosed-body inspection is now bounded, since both rules decide on their
first piece of evidence, and the opener and close lookups are memoized per
parse rather than rescanning to the end of the buffer for every opener. Same
message: 1.2s. A realistic 40KB reply with 8 mentions: 0.35ms per parse, 28ms
across the entire stream.
Also derives JSON_BODY_TAG_NAMES from SPECIAL_TAG_NAMES so a new tag cannot
silently fall back to the weaker prose heuristics; deletes a docstring the
rewrite orphaned above TAG_SHAPED_MARKER; corrects one still describing the
first-character check that depth tracking replaced; drops a test duplicating
one already in the file; and fixes a test that named the no-nesting rule but
passed with that rule deleted, because its JSON closed before the marker.
* refactor(copilot): drop the JSON-prefix wrapper the split orphaned
Splitting isViableJsonPrefixOf out so callers that had already blanked the
body would not pay for a second pass left isViableJsonPrefix behind with no
callers. Both call sites blank the body for their own marker scan first, so
the wrapper has nothing left to do.
Biome does not flag it, so it would have sat there indefinitely.
The wrapper's docstring carried the actual argument for depth tracking over a
first-character check, which is the non-obvious part of the function; it moves
onto isViableJsonPrefixOf rather than being deleted with it, along with the
already-blanked precondition its callers rely on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(copilot): drop the nested-marker scan the JSON rule already covers
The unclosed-tag check ran two rules over every JSON body: a scan for nested
special-tag markers, then a JSON-viability test. The scan earned nothing there.
A marker outside a string literal is stray content the viability rule already
rejects, and a marker inside one is legitimate quoted syntax the scan is
explicitly blanked to ignore. It cost 14 substring scans per opener per streamed
chunk to catch nothing.
Verified by deletion rather than by argument: with the scan disabled for
JSON-bodied tags, the only failures were the two tests written for the scan
itself, and neither is trace-derived. One documents its own contrivance — the
array is left unclosed on purpose because a closed value would let the JSON rule
decide it. Every test pinned from a real message still passes.
The rule stays for the prose-bodied tag, where a body that is not JSON leaves
nesting as the only available evidence. The foreign-close test moves to
<thinking> to cover it there; the marker-outside-strings test goes, since its
premise was the scan.
One narrow case regresses: stray non-JSON content inside a still-open structure
(`<question>[{...} </options>`) now stays hidden until the stream ends rather
than settling mid-stream. Depth tracking counts brackets, it does not validate
syntax. Self-healing at end of stream, and buying it back means a real JSON
prefix validator — more machinery than the case is worth.
Also pins two behaviors that had no coverage and would have been "simplified"
away. The two literal reasons resume at different offsets: never-a-payload must
resume past the CLOSE, because resuming past the opener rescans an interior
whose quoted markers the blanked scan never saw, re-parsing a tag inside a JSON
string and dropping it. Collapsing the reasons passes all 54 existing tests and
silently deletes text. Escape handling in the string blanker is likewise only
observable through depth skew.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): stop the rescan deleting quoted text, and bound its cost
Review found the foreign-markers rescan reintroducing the failure this branch
exists to remove. Three fixes, each with a mutation-checked regression test.
The rescan decides on the BLANKED body but resumes at the opener and re-scans
the RAW one, so a tag quoted inside a JSON string — invisible to the decision —
is re-parsed as a real tag on the second pass and dropped. A question whose
prompt quotes a <credential> example rendered with the quoted payload deleted.
Resolution now resumes at the offset of the marker that actually survived
blanking, so the quoted region is skipped and emitted verbatim. Promotion of a
quoted tag into a live control is not reachable: quotes inside the outer JSON
string must be escaped, so the inner body never survives JSON.parse.
That offset is taken from the blanked copy and applied to the raw body, which
makes index preservation load-bearing rather than decorative. A blanked astral
character emitted one space for two UTF-16 units, shifting every later offset
left; it now emits char.length. No test pins this — the drift only moves a text
segment boundary, and adjacent text segments concatenate — so the invariant is
stated in the docstring instead of guarded by a test that cannot fail.
The matched-pair body had no size cap, unlike the unclosed path. A borrowed
close stretches one body across most of the message, and the scan reruns for
every opener inside it on every streamed chunk. A 58KB reply of that shape —
organic, it is trace b095e080's own mechanism — cost 242ms per parse on the main
thread, and the parse reruns per chunk. Bounding the scan to MAX_UNCLOSED_BODY_SCAN
brings it to 35ms. The bound needed a guard the reviewers' version omitted: when
only a prefix was inspected, finding no reason is not evidence the body was a
payload, so it resolves to literal rather than discard. Without that, capping
converts unexamined bodies into silent deletions — the naive fix would have
caused the bug it was meant to prevent.
Finally, pushText dropped whitespace-only spans. Harmless when failed tags were
dropped whole; the literal path emits a rejected span in pieces, so a blank line
between two of them vanished and two markdown paragraphs merged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(copilot): assert what a reader sees, and pin the unclosed-thinking trade
Review found several tests asserting only `hasPendingTag`. That boolean can be
correct while a wrong resumeAt drops the trailing prose — the exact defect twice
fixed on this branch — so those tests could not have caught it. They now assert
the rendered text alongside the flag.
Adds renderedText(), since a dozen tests hand-rolled the same map/join, and
switches two discard tests off exact segment-array equality onto it. How a span
splits across adjacent text segments is not observable: the renderer
concatenates them. Pinning the array shape only breaks on refactors that change
nothing a reader sees, which is what those two did.
Adds a frame-replay test. Everything else asserts one end state, so nothing
covered behavior BETWEEN streamed frames, where a card could render and then
revert. replayFrames() parses every growing prefix and asserts card count never
decreases — appending can only add closes after the ones already matched, so no
earlier opener's resolution can change.
Pins the unclosed-<thinking> behavior as a deliberate trade rather than leaving
it incidental: hidden while streaming, shown once the stream ends. Hiding is the
right mid-stream default because a close is still plausible and `thinking`
renders as nothing anyway. Showing it at the end does leak reasoning when the
close never arrives, which is accepted — that is rare, and keeping it hidden
would swallow the answer whenever the model opened the tag and then wrote the
reply without closing it.
Not taken: the reviewer suggestion to swap the prose path's tag-name scan for
TAG_SHAPED_MARKER. It matches any `<foo>`, and thinking bodies are prose that
legitimately discuss markup, so it would release those bodies early and make the
leak above more frequent — the opposite of the default just chosen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(copilot): name the scan window's blind spot instead of glossing it
Bugbot is right that the window can hide a streaming tail, and the docstring
said the bound "reaches the same verdict as the full remainder for any real
payload" — true for every payload a tag carries, but it read as unconditional
and hid the exception.
The exception, now stated and pinned by a test: a JSON body whose top-level
value closes BEYOND the window, followed by prose and no closing tag, still
reads as a viable prefix, so the remainder waits for the stream to end instead of
settling mid-stream. Lossless — the completed parse renders every character —
and it needs a payload several times larger than any tag emits. A mention in
prose settles at its first character at any length, because prose does not open
with a brace; the test pins that half too, since it is the case that actually
occurs.
Not widening the window: the bound is what took a 58KB borrowed-close reply from
242ms to 35ms per parse, on the main thread, re-run every chunk. Trading a
measured freeze for a hypothetical one is the wrong direction. The docstring also
now covers the matched-pair path, which the same constant bounds since the
previous commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): make the indexOf cache correct for any call order
The cache was safe only under a precondition the code could not enforce: that
`from` never decreases within a parse. Violating it does not throw — a stale
index is returned and the parser silently mis-parses, which is the exact failure
class this branch exists to remove.
Each entry now records the offset it was searched at, and is reused only when the
new `from` is at or beyond it. A cached -1 still answers any later `from`, since
absence from an earlier offset implies absence from a later one; a cached hit
still answers when it lies ahead of `from`. Anything else rescans.
The precondition held and still holds — every non-pending outcome resumes
strictly past its opener. But it is a property of resolveTagAt's resume points,
not of this function, and one of those points deliberately resumes back inside a
span it already examined. Someone adjusting a resume point should pay a redundant
scan, not corrupt a parse.
Verified by mutation: reverting the guard fails the new backward-cursor test and
nothing else, so the guard is load-bearing and the monotonic path is unchanged.
Same-content benchmark on a 79KB reply: 3.13 ms/frame before, 2.96 after — the
entry object is allocated only on a real scan, which is already bounded.
memoizedIndexOf is exported for the test. The invariant cannot be provoked
through parseSpecialTags, because the cursor does not walk backward today; that
is the whole point, so it has to be exercised directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): resume at the window edge, not past the close, on a truncated body
Regression from the scan bound added two commits ago. When a borrowed close
stretches a body past MAX_UNCLOSED_BODY_SCAN and the inspected prefix is
marker-free prose, resolution resumed past the borrowed close — skipping the
uninspected remainder entirely. A valid tag sitting in that remainder was
flattened to plain text purely because of which side of the window it landed on.
Measured: with ~1KB of prose before it the chip rendered, with ~6KB it did not.
A verdict drawn from a prefix — "prose", or no verdict at all — says nothing
about the rest of the body, so resumption now lands on the first character that
was NOT inspected. Everything inspected is still emitted as text by the caller,
and scanning continues into the remainder, so the tag renders. Ordering matters:
foreign-markers keeps priority, since a marker found inside the window is a real
offset and resuming there is strictly earlier and safer.
No text was ever lost in this case — it stayed lossless throughout — but a
destroyed chip is a real regression, and it was introduced by the fix for the
previous one.
Cost is bounded: each truncated step advances a full window, so a long body costs
body-length/window re-entries rather than one scan per character. A 117KB
borrowed body parses in 0.1ms.
Verified by mutation: restoring the old resume point fails the new test at 6KB
and 60KB and nothing else. The test asserts three lengths so the boundary itself
is covered rather than one side of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): only discard a body that actually parsed as JSON
`discard` fired on any body parseSpecialTagData rejected, which conflated "is not
valid JSON" with "is valid JSON of the wrong shape" — even though its own comment
has always said "a well-formed value that failed its shape guard." The viability
rule cannot separate them, because bracket depth reads `{the Q4 report}` as a
viable JSON value.
So prose someone wrapped in braces was deleted:
I saved <workspace_resource>{the Q4 report}</workspace_resource> for you.
-> I saved for you.
along with the two commonest JSON slips a model makes, unquoted keys
(`{type: "file"}`) and single quotes (`{'type':'file'}`). All three are text loss
of exactly the kind this branch exists to remove.
Dropping text is only defensible for a payload the agent actually formed, so
discard now requires the body to parse. Anything that will not parse was never
demonstrably a payload and is shown instead. A valid payload with the wrong shape
still discards, which is the case the outcome was written for, and a valid tag
still renders.
Costs a second JSON.parse of a body that already failed one. That is the rare
path: a valid payload returns before it, and prose is rejected by the cheaper
viability rule before it.
Verified by mutation: removing the gate fails the new test and nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): apply the nesting rule to a prose body on the matched-pair path too
A prose body has no shape to fail — parseTextTagBody accepts any non-empty text —
so <thinking> was the one tag whose close was accepted unconditionally. The
unclosed path already refuses an opener whose body carries a tag marker, on the
grounds that tags never nest. The matched-pair path did not, and the two
disagreeing is the bug.
Mid-stream a nested marker disproves the outer opener, so its text is released
and the inner tag renders. When </thinking> finally arrived it was accepted as a
segment, and everything already on screen was swallowed into it and suppressed:
a <thinking>b <options>[...]</options> c -> "a <thinking>b [CARD] c"
a <thinking>b <options>[...]</options> c</thinking> d -> "a d"
A rendered card and its surrounding text disappearing from a message the reader
was already looking at. Both paths now apply the same rule, and resolution
resumes at the opener so the inner tag survives the rescan — safe here because
prose bodies are never blanked, so no marker is hidden from the scan the way one
can be inside a JSON string.
The frame-replay helper counted a thinking body as visible text, which is why its
card-monotonicity assertion could not see this. It now models the renderer:
thinking contributes nothing. Verified by mutation — removing the rule fails the
new test and nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(copilot): assert parser invariants over generated messages
Every regression review found on this branch was a combination nobody had written
an example for. The example tests cannot cover the space: a body is judged on
body kind, close state, JSON state, marker placement, size against the scan
window, and streaming mode — a product of roughly six hundred cells, each needing
both an outcome and a resume point.
Four invariants over messages composed from fragments, so a new combination is
covered without a new test:
- no character is lost from a message containing nothing droppable
- every valid tag renders as a card whatever surrounds it
- across streamed frames a card never un-renders and text never retracts
- the settled parse never shows less than the last streaming frame
Fragments are the shapes the parser must reject — prose mentions, misspelled and
truncated closes, brace-wrapped prose, unquoted keys, a nested marker in a prose
body, filler long enough to cross the scan window. None is a valid tag or a
well-formed payload, so nothing is eligible for discard, which is what makes
"output equals input" a legal assertion. Seeded, so a failure reproduces.
Validated against the three defects review found, with the fix for each reverted
and only these tests running: the flattened tag past the window fails the card
invariant, the deleted brace-wrapped prose fails the loss invariant, and the
retracted thinking close fails both loss and retraction. They pass on current
code, so nothing further is reachable through the shapes they generate.
The generator composes several fragments between tags rather than one. With a
single fragment it cannot place an unclosed opener and then enough prose to push
a valid tag past the scan window, which is exactly the shape of one of the three —
the first draft missed it for that reason alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): strip a stray null byte from the test file
A space in the visibleView card placeholder was written as U+0000, so the file
contained one null byte. Git classifies a file with a null byte as binary and
shows "Bin 34936 -> 41628 bytes" instead of a diff, which made the test file
unreviewable in the two commits since 2f9b60aa9d.
Nothing caught it: vitest, tsc and biome all read the file fine, and the byte sat
inside a string literal used only as a placeholder in test-only output, so no
assertion depended on it. Only `git diff` noticed.
The PR diff renders as text again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(copilot): split resolveTagAt into budget, classification, and resume
resolveTagAt answered three orthogonal questions in one body of branches — is
this a payload, how much of it may I read, and where do I resume — and every
regression review found on this parser was one of those answers changing without
another. Two of the five were regressions from the fix for the previous one, and
the last two were not missed inputs at all: one was the code contradicting its
own docstring, the other two branches applying different rules to the same
question. That is what coupling looks like, not bad luck.
Now three pieces:
- inspectWithin — the read budget, and nothing else. Both paths spend it through
the same helper; they previously applied the same constant in two shapes, and
the difference between those shapes was a bug.
- classifyBody — what the body IS. Pure: no positions, no outcome, no resume.
Returns one of a closed set of five classes.
- resumeForClass — where scanning continues, given the class. Nothing else.
The closed set is the point. resolveMatchedPair and resumeForClass each switch
over it exhaustively, so adding a class fails to compile until BOTH questions are
answered for it. Verified by adding a sixth case: tsc reports exactly two errors,
one per switch. The failure mode that produced five review rounds is no longer
representable.
Behaviour is identical, not merely believed to be. Diffed against the previous
implementation over 7,500 comparisons — 1,500 generated messages, each parsed
streaming, settled, and at three mid-stream cut points — asserting deep equality
of the full result, not just rendered text. The one divergence the differential
found was mine: the prose nested-marker path resumed at the marker rather than
the opener, emitting one text segment where the old code emitted two. Display-
identical, since adjacent text segments concatenate, and arguably better — but a
behaviour change does not belong in a refactor, so prose nesting is now its own
class that resumes exactly where it used to. The improvement is noted in the type
for whoever wants it.
The 70 tests, including the four property invariants added on the parent branch,
pass unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): stop backtick stripping from unbalancing a whole message
The display sanitizer removes a backtick sitting next to a workspace-resource
tag, so a stray one cannot stop a chip rendering. It could not tell a real tag
from prose MENTIONING the tag name, and a mention is the common case:
The `<workspace_resource>` tag needs a real `path` to render.
-> The <workspace_resource>` tag needs a real `path` to render.
The opening backtick goes, the closing one is left unpaired, and it opens a code
span that runs to the next backtick — inverting every code span for the rest of
the message. A three-paragraph explanation renders with most of its prose in
monospace and stray backticks visible in the text.
Both unpaired-strip patterns now require the complete opener-payload-closer to
be present with no backtick inside it. That is what separates the two cases: a
payload is JSON and contains no backticks, while a mention has no closer at all.
The one-sided stray backtick around a real tag — the case these patterns exist
for — still strips, and the balanced-code-span unwrap above them is untouched.
Pre-existing, and not in this branch's two files, but only reachable because of
them: until this branch, a prose mention of the tag blanked the rest of the
message, so the mangled formatting was never on screen to see. Verified end to
end through sanitizer -> parser -> Streamdown: the reported message keeps all 14
backticks and renders 7 balanced code spans with no spurious chip, and a real
backticked tag still renders its chip with the wrapping removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): apply the same backtick guard to the balanced unwrap
The previous commit fixed the two unpaired-backtick strips but left the balanced
unwrap above them, which has the identical flaw. It allows anything between
opener and closer, so a message writing the two markers as separately backticked
spans reads as ONE wrapped tag and loses its outer pair:
Use `<workspace_resource>` then close with `</workspace_resource>` here.
-> Use <workspace_resource>` then close with `</workspace_resource> here.
Two backticks gone, the remaining two mispaired, same message-wide code-span
inversion. This is the shape a message explaining the tag syntax naturally
takes, which is how the original report was produced.
All three patterns now forbid a backtick between opener and closer. Verified
across the matrix: a real tag still unwraps whether the stray backtick is on
both sides or one, and every mention shape keeps its backticks balanced.
Known and accepted: a resource whose title or path itself contains a backtick
will not have wrapping backticks stripped, so it renders as text instead of a
chip. That failure costs one chip and is rare; the one it replaces corrupts the
formatting of an entire message and is common.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): pair backticks the way markdown does, and bound the scan
Review found my previous two commits traded one bug for a worse one. Three
independent reviewers measured it.
The trailing-backtick pattern anchored on the bare `<workspace_resource>`
literal, so every opener started a lazy scan hunting a closer that never
arrives. A message repeating the tag name in prose went quadratic: 168KB cost
154ms per call, and this runs on the main thread for every streamed chunk, so a
long reply is seconds of frozen tab. The pattern it replaced was linear. Now
0.36ms on the same input.
Two correctness bugs came with it, both from matching `\s*` around the tag: the
pattern could start at one code span's delimiter and finish at another's, so
`Open `config.json` <tag> then run `bun test`` lost a backtick; and the
whitespace crossed newlines, eating one of the three backticks closing a fenced
block that contained a tag, leaving the rest of the message inside the fence.
The root problem was three global regexes each guessing at which backticks
belong together. They now model what markdown actually does: find code spans by
pairing a backtick with the next one on the same line, and unwrap a span only
when it genuinely contains a complete tag. Pairing is what makes the neighbour
and fence cases correct rather than separately patched. A leftover backtick
pressed directly against a tag, with no partner, is still stripped.
A negative lookahead stops any scan crossing another opener — the cost bound.
Adds tests for the neighbour and fence shapes, and one that pins the complexity
by asserting 168KB of repeated-opener prose finishes well inside 50ms; every
fixture until now was under 1KB, which is why both quadratics were invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): address the confirmed review findings
Five findings from the review, each reproduced before being fixed and
mutation-checked after.
The nesting rule on the matched-pair path tested for anything tag-shaped while
the streaming path tested for the tag NAMES. Reasoning that mentioned `<div>` or
a generic was therefore released as visible prose — the model's thinking on
screen because of an incidental angle bracket. Both paths now share one
predicate, hasSpecialTagMarker, so they cannot disagree about whether a body was
ever a tag. The broad regex keeps its other job: on a JSON body an invented name
like `</workflow_resource>` really is stray content.
Not changed, and worth saying why: a `<thinking>` body containing a REAL nested
tag still releases and renders it. Suppressing it instead would mean the
streaming path shows a card and the close then retracts it, which is the defect
the previous commit fixed. Security rated the containment loss P2 while noting it
grants no capability the top-level path lacks — a stream that can nest a tag can
emit one at top level, which already rendered.
unclosedTagCannotResolve blanked a full window before viability rejected the body
on its first character, which is the common case of a tag name in prose. Testing
that first: 43ms per streaming parse at 84KB, now 2ms.
The unclosed path sliced the whole remaining buffer and then bounded it, copying
the rest of the message per opener per chunk. inspectFrom slices once, bounded.
A message that is ONLY a discarded payload rendered the raw JSON: discard emits
no segment by design, so it reached the empty-segments fallback, whose job is to
never blank a plain-text message. It now knows a discard happened. Pre-existing,
but discard only became a first-class outcome on this branch.
Three docstrings still pointed at resolveTagAt for decisions the refactor moved
into classifyBody and resumeForClass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(copilot): quality pass over the parser and its tests
Four changes from a reuse/simplification/efficiency/altitude review. No behaviour
change; 162 tests unchanged.
blankJsonStringLiterals returns the body untouched when it contains no quote. No
quote means no string literal, so the loop was copying the body to itself
character by character. unclosedTagCannotResolve had learned to short-circuit
before calling it; literalTextReason had not, and blanked unconditionally on
every already-rejected tag on every later chunk. Putting the guard inside the
helper fixes both callers at once.
inspectWithin and inspectFrom were near-duplicates added at different times. One
function with an optional start expresses both, and keeps the property the second
one existed for: slice once, already bounded, never `content.slice(bodyStart)`
first and bound after.
The frame-replay property was 1727ms — 95% of the file's test time — because it
parses every prefix, so message length multiplies into parse count, and the
fragment pool included a 6KB filler. Retraction happens AT a frame boundary, not
as a function of message size, so that property now draws from the short
fragments; the window-crossing filler still gets coverage from the properties
that parse each message once. 1727ms to 205ms, same seeds, same assertions. It
also now calls replayFrames instead of hand-rolling the stepping loop it already
had a helper for.
Deferred deliberately, each with a reason:
- Collapsing `prose-nested-marker` into `nested-marker`. Its own docstring names
the simplification and defers it; it is a behaviour change and wants its own
commit and test, which is the pattern the rest of this branch follows.
- Bounding hasSpecialTagMarker on the prose path. Costs tens of microseconds on
a long reasoning body, but a marker past the bound would flip that body from
released to suppressed — a retraction, which is the bug two commits back.
- Splitting the parser out of this `'use client'` file. The strongest structural
finding: the file cannot be imported by server code, which is why the inbox
executor carries its own thinking-strip regex. It is a mechanical move with no
logic change and deserves its own PR, not commit 25 of this one.
- Generalising the backtick sanitizer past `workspace_resource`, and replacing it
with parser-owned backtick consumption. Same reasoning: real, and not here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(copilot): reattach an orphaned docstring and cut comment noise
One real defect. Adding hasSpecialTagMarker put it BETWEEN
unclosedTagCannotResolve and the docstring written for it, so two doc blocks sat
stacked and the function they described ended up with none. A reader scanning the
file would have read the first block as preamble for the wrong function. Moved
back, and while there: "14 substring scans" is SPECIAL_TAG_NAMES.length * 2, so
adding an eighth tag would have quietly made it wrong — it now says a pass per
tag name.
The rest is noise removal:
- Three comments narrating what the parser used to do. A comment is read by
someone looking at the current code, not the diff, and this branch already
writes that history at length in its commit messages.
- Three millisecond measurements. Each one already stated the durable claim —
quadratic, or a copy thrown away per opener per chunk — and then appended a
number that will rot. The one measurement kept is the blind-spot paragraph on
MAX_UNCLOSED_BODY_SCAN, where the number is what justifies the constant.
- Two of the four restatements of "the renderer concatenates adjacent text
segments". Kept where it is load-bearing, on pushText and on the test helper.
- One claim gone stale in the last commit: the read-budget doc still described
two helpers agreeing with each other, after they became one function.
Left alone deliberately: the rules that look arbitrary and are not — marker
blanking, the differing resume offsets, the accepted trades. Those are why this
file reads as heavily commented, and they are the ones worth having.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): sanitize backticks in one pass so a flush code span survives
Review caught a case the previous fix missed. Two passes each decided
independently which backticks belonged together, and with no space between a
code span and a tag they disagreed:
Open `config.json`<tag> ok -> lost the span's closing backtick
Open <tag>`config.json` ok -> lost the span's opening backtick
My test for this used a space between them, which is exactly why it passed.
Both passes are now one left-to-right scan alternating between a code span and a
tag with a stray backtick against it. A span consumes its own delimiters as the
scan reaches them, so a flush neighbour keeps its pair with no special case —
where a second pass had no way to know the backtick was already spoken for. This
is the third arrangement of this file; the first two each handled the cases they
were written for and broke a different one, which is what two passes guessing at
the same question produces.
A trailing backtick is only taken as a stray when no further backtick follows on
the line. Otherwise it is the opener of the next span. Mutation-checked: removing
that lookahead fails the new test and nothing else.
Swapping the alternation order changes nothing any fixture covers, so the comment
no longer claims the order is load-bearing — it distinguishes only a span that
opens flush against a tag and closes elsewhere, which nothing pins.
Thirteen shapes verified balanced, including all three flush variants, the
fenced block, the neighbour with a space, both one-sided strays, and the mention
shapes. 168KB of repeated openers: 0.18ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): rewind the scan-window resume so a straddling opener still parses
Applied from an automated code review of this branch. Two findings, both
verified by reverting the fix and confirming only the new test fails.
`resumeForClass`'s `unexamined` case resumed at exactly `bodyStart + 4096`.
That edge is an arbitrary cut, so an opener can begin just before it and
finish just after — leaving its `<` behind the cursor. The opener scan only
looks forward, so the tag was never found and its payload rendered as raw
JSON text on a COMPLETED message, not just mid-stream. Reproducible for
every filler length in 4077..4095; the existing borrowed-body test steps in
11-character units and never lands in that band.
Backing the resume off by the longest marker guarantees a straddling opener
is re-scanned from its `<`. The step is still ~4076 characters, so a long
body still costs a bounded number of re-entries. The new test sweeps every
offset across the band.
The two complexity tests asserted absolute wall-clock ceilings (<50ms,
<20ms), which measure the machine as much as the algorithm: they fail on a
loaded CI box, and set generously enough not to, they let a genuine
quadratic through at the single size they sample. Both now assert the
scaling ratio across a 4x input instead (quadratic ~16x, linear ~4x).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(copilot): trust the raw body for markers once it is known not to be JSON
`literalTextReason` blanks a body's quoted regions before scanning it for tag
markers, so that syntax quoted inside a JSON string is not mistaken for a real
nested tag. That blanking assumes quotes delimit JSON strings. One unbalanced
`"` breaks the assumption: everything after it is treated as string content,
which can hide a genuine marker.
The verdict then degrades from `foreign-markers` to `never-a-payload`, and the
resume changes with it — from the marker offset to past the close — flattening
a real tag inside the span. A card already on screen un-renders into raw JSON
when the closing tag finally arrives, and a valid tag after it never renders.
Blanking is only meaningful while the body might BE JSON. Once viability or a
failed parse has proved it never was, that premise is void and the raw text is
the honest evidence, so rescan it and resume at the marker.
Both routes to "never JSON" now funnel through one branch. The rescan applies
to the failed-parse route too, not only the viability one — patching just the
latter leaves the same defect reachable through the former.
Behaviour for a body that IS valid JSON is untouched: a well-formed payload
that fails its shape guard is still discarded, and tag syntax quoted inside a
valid payload is still invisible to the scan.
Adds the repro as two tests — the settled parse, and frame-by-frame so the
un-render is pinned directly — plus two unbalanced-quote fragments to the
property corpus. Reverting the rescan fails exactly those two tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(copilot): share the scaling-ratio harness between the two perf tests
Both complexity tests hand-rolled the same 15-line harness — build a repeated
tag mention, take the fastest of five runs at two input sizes, assert the
ratio — differing only in which function they timed. Changing the run count,
the sample sizes, or the threshold meant editing both in lockstep.
Extracted to `scalingRatioOver4x`, following the existing `*-test-helpers.ts`
convention in this tree. The rationale for asserting a ratio rather than a
wall-clock ceiling now lives in one place instead of being paraphrased twice.
Also drops a redundant disjunct in the opener scan: `nearestStart` and
`nearestTagName` are only ever assigned together, so `nearestStart === -1`
holds exactly when the name is empty. Testing the name alone is the same check
and is the one that narrows the union for `resolveTagAt`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(copilot): correct a resume comment the rewind fix left inaccurate
The `unexamined` case claimed "everything read is emitted as text by the
caller". That was true when the resume was exactly the window edge, but the
straddling-opener rewind holds the last marker's worth of the window back so it
can be re-scanned rather than flattened — so the sentence contradicted the
paragraph directly beneath it. Nothing is lost either way; the caller emits up
to wherever this resumes.
Also drops two "Round-N class:" prefixes from test comments. They point at
review rounds of this PR, which mean nothing to a reader after it merges; the
sentences that follow already say what the bug was.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(copilot): pin the blanking rule with a body staging cannot recover
CI runs the merge with staging, and staging's desktop PR (#5998) taught
`parseSpecialTagData` to recover a failed `<question>` body's prompt and render
it as text instead of returning null. That recovery lands before `classifyBody`
is ever consulted, so this fixture — whose quoted `</options>` sat inside its
`prompt` — stopped exercising the blanking rule and started asserting the
recovery. Merged, it rendered "A use </options> here? B" instead of "A B".
Moving the quoted marker to a non-`prompt` field restores what the test is for:
a marker inside a JSON string must be blanked before the scan, or a broken
payload gets classified as literal text and its raw JSON is shown. A body with
no recoverable prompt reaches `discard` on both sides of the merge, matching the
prompt-less fixture the sibling test above already uses.
Behaviour is unchanged on either branch alone; only the fixture moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(copilot): cover every tag in the property invariants, and pin that it stays that way
`VALID_TAGS` and `NEEDLES` hand-picked three or four of the seven tags the
parser resolves. `credential`, `usage_upgrade`, and `mothership-error` were
exercised by no invariant at all — not the card-count property, not
retraction-across-frames, not settled-vs-last-frame — and nothing failed to say
so. The property block's whole argument is that it covers combinations no fixed
example set would, which was only true for four sevenths of the tag space.
Fixtures are now keyed by tag name and checked against SPECIAL_TAG_NAMES, so
adding a tag without a fixture fails a test instead of quietly falling outside
every property here. `thinking` is listed separately: it renders nothing, so it
cannot carry a card invariant.
SPECIAL_TAG_NAMES is exported for that check, matching the five sibling
`*_TYPES` unions in the same module that are already exported. NEEDLES derives
from it too, so the memoization test searches every opener the parser does.
All three newly covered tags pass every invariant — this closes a coverage gap,
it does not fix a bug. Removing any one fixture fails the new guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(copilot): state the actual bound on the thinking nesting rule
The docstring implied generics are excluded from the marker check. They are not:
the match is a substring test, so `Promise<void>` is safe only because `void` is
not a tag name, while `Promise<options>` does match and would release a thinking
body as text.
Says so plainly now, with why it is left as-is: the boundary check that would
narrow it wants a lookbehind, which is Safari 16.4+ and would be a parse-time
SyntaxError on the versions this app still supports — a dead client chunk is a
worse outcome than the bug. Reaching it also needs an inline `<thinking>` body,
which the agent no longer emits, discussing a type named exactly after a tag.
Comment only; no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Justin Blumencranz
committed by
GitHub
GitHub
parent
985f9e6172
commit
ee7c061681
+93
@@ -2,6 +2,7 @@
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { scalingRatioOver4x } from '@/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers'
|
||||
import { sanitizeChatDisplayContent } from './chat-sanitize'
|
||||
|
||||
describe('sanitizeChatDisplayContent', () => {
|
||||
@@ -19,4 +20,96 @@ describe('sanitizeChatDisplayContent', () => {
|
||||
|
||||
expect(sanitizeChatDisplayContent(content)).toBe('Read and found the issue.')
|
||||
})
|
||||
|
||||
it('leaves a backticked mention of the tag name alone', () => {
|
||||
// Unwrapping exists so stray backticks cannot stop a real chip rendering. A
|
||||
// MENTION is not a tag: there is no payload and no closing marker, just the
|
||||
// name written in prose. Stripping its opening backtick leaves the closing
|
||||
// one unpaired, which opens a code span that swallows the rest of the
|
||||
// message — every later `code` toggles to the wrong state.
|
||||
const content = 'The `<workspace_resource>` tag needs a real `path` to render.'
|
||||
|
||||
expect(sanitizeChatDisplayContent(content)).toBe(content)
|
||||
})
|
||||
|
||||
it('keeps backticks balanced across a message that mentions the tag repeatedly', () => {
|
||||
const content =
|
||||
'Use `<workspace_resource>` for files.\n\nThe `<workspace_resource>` chip needs an `id`.'
|
||||
const backticks = (text: string) => (text.match(/`/g) || []).length
|
||||
|
||||
expect(backticks(sanitizeChatDisplayContent(content))).toBe(backticks(content))
|
||||
})
|
||||
|
||||
it('leaves prose that mentions the opener and the closer separately alone', () => {
|
||||
// The balanced unwrap reads a backticked opener, prose, and a backticked
|
||||
// closer as ONE wrapped tag, and strips the outer pair. This is the shape a
|
||||
// message explaining the tag syntax naturally takes.
|
||||
const content = 'Use `<workspace_resource>` then close with `</workspace_resource>` at the end.'
|
||||
|
||||
expect(sanitizeChatDisplayContent(content)).toBe(content)
|
||||
})
|
||||
|
||||
it('leaves a neighbouring code span alone', () => {
|
||||
// Only a backtick DIRECTLY against the tag is the tag's wrapping. Allowing
|
||||
// whitespace between let the pattern reach past the tag and take the
|
||||
// delimiter off an unrelated span, breaking its pair.
|
||||
const content =
|
||||
'Open `config.json` <workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource> then run `bun test`.'
|
||||
|
||||
expect(sanitizeChatDisplayContent(content)).toBe(content)
|
||||
})
|
||||
|
||||
it('leaves a code span sitting flush against the tag alone', () => {
|
||||
// No space between them, so the span's delimiter is directly against the
|
||||
// tag and a pattern looking for "a backtick beside a tag" cannot tell the
|
||||
// two apart. A single pass settles it: a span consumes its own delimiters as
|
||||
// the scan reaches them, and a trailing backtick is only taken as a stray
|
||||
// when no further backtick follows on the line.
|
||||
const before =
|
||||
'Open `config.json`<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource> ok'
|
||||
const after =
|
||||
'Open <workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource>`config.json` ok'
|
||||
const both =
|
||||
'Open `a`<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource>`b` ok'
|
||||
|
||||
expect(sanitizeChatDisplayContent(before)).toBe(before)
|
||||
expect(sanitizeChatDisplayContent(after)).toBe(after)
|
||||
expect(sanitizeChatDisplayContent(both)).toBe(both)
|
||||
})
|
||||
|
||||
it('does not break the closing fence of a code block containing a tag', () => {
|
||||
// Whitespace matching used to cross the newline and consume one of the three
|
||||
// closing backticks, so the block never closed and the rest of the message
|
||||
// rendered as code.
|
||||
const content =
|
||||
'Example:\n```md\n<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource>\n```\nDone.'
|
||||
|
||||
expect(sanitizeChatDisplayContent(content)).toBe(content)
|
||||
})
|
||||
|
||||
it('stays linear on a message that repeats the tag name without ever closing it', () => {
|
||||
// A lazy scan allowed to cross an opener restarts from every opener, which
|
||||
// is quadratic — 154ms for this input before the bound, on the main thread,
|
||||
// for every streamed chunk.
|
||||
//
|
||||
// Asserted as a scaling ratio, not a wall-clock ceiling — see
|
||||
// {@link scalingRatioOver4x} for why.
|
||||
expect(scalingRatioOver4x((content) => sanitizeChatDisplayContent(content))).toBeLessThan(8)
|
||||
})
|
||||
|
||||
it('still unwraps a real tag that carries a stray backtick on one side only', () => {
|
||||
// The case the unpaired strip is actually for: the model backticked the
|
||||
// opener but not the closer (or vice versa), which would block the chip.
|
||||
const leading =
|
||||
'`<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource> done'
|
||||
const trailing =
|
||||
'<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource>` done'
|
||||
|
||||
expect(sanitizeChatDisplayContent(leading)).toBe(
|
||||
'<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource> done'
|
||||
)
|
||||
expect(sanitizeChatDisplayContent(trailing)).toBe(
|
||||
'<workspace_resource>{"type":"file","path":"a.md","title":"a"}</workspace_resource> done'
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
+56
-5
@@ -1,12 +1,63 @@
|
||||
const HIDDEN_INLINE_REFERENCE_PATTERN =
|
||||
/`[^`\n]*(?:internal\/tool-results\/|internal\/blocktips\/|components\/integrations\/[^`\n]*README)[^`\n]*`/g
|
||||
const WORKSPACE_RESOURCE_CODE_SPAN_PATTERN =
|
||||
/`([^`\n]*<workspace_resource>[\s\S]*?<\/workspace_resource>[^`\n]*)`/g
|
||||
|
||||
/**
|
||||
* A complete workspace-resource tag: opener, payload, closer.
|
||||
*
|
||||
* Two constraints on the payload, both load-bearing:
|
||||
*
|
||||
* - **No backtick.** A payload is JSON and carries none, so this is what tells a
|
||||
* real tag from prose MENTIONING the tag name — a message explaining the
|
||||
* syntax writes the opener and the closer as two separately backticked spans.
|
||||
* - **No nested opener**, via the negative lookahead. A cost bound rather than a
|
||||
* correctness rule: a lazy scan allowed to cross an opener restarts from every
|
||||
* opener, so a message repeating the tag name is quadratic — on the main
|
||||
* thread, for every streamed chunk.
|
||||
*
|
||||
* Accepted trade: a resource whose title or path itself contains a backtick is
|
||||
* not matched, so it renders as text rather than a chip. That costs one chip and
|
||||
* is rare; the failure it replaces corrupts a whole message and is common.
|
||||
*/
|
||||
const COMPLETE_TAG_SOURCE =
|
||||
'<workspace_resource>(?:(?!<workspace_resource>)[^`])*?<\\/workspace_resource>'
|
||||
|
||||
/** Non-global so {@link RegExp.test} has no `lastIndex` to carry between calls. */
|
||||
const COMPLETE_WORKSPACE_RESOURCE_TAG = new RegExp(COMPLETE_TAG_SOURCE)
|
||||
|
||||
/**
|
||||
* One left-to-right pass over the two things that can own a backtick: an inline
|
||||
* code span, and a tag with a stray backtick pressed against it.
|
||||
*
|
||||
* ONE pass is the design. Two separate passes each have to guess which backticks
|
||||
* belong together, and every previous arrangement of this file got a different
|
||||
* case wrong — a span two words away, a code fence, then a span sitting flush
|
||||
* against the tag. Here a span consumes its own delimiters as the scan reaches
|
||||
* them, so `` `config.json`<tag> `` keeps its pair without a special case.
|
||||
*
|
||||
* The trailing backtick is only taken when no further backtick follows on the
|
||||
* line; otherwise it is not a stray at all but the opener of the next span, and
|
||||
* `` <tag>`config.json` `` would lose that span's delimiter. A LEADING backtick
|
||||
* needs no such guard, because a backtick that closes a span is consumed as part
|
||||
* of that span. Of the two, only the trailing lookahead is pinned by a test —
|
||||
* swapping the alternatives changes behaviour only for a span that both opens
|
||||
* flush against a tag and closes elsewhere, which no fixture covers.
|
||||
*/
|
||||
const CODE_SPAN_OR_FLANKED_TAG = new RegExp(
|
||||
`\`[^\`\\n]*\`|\`?(${COMPLETE_TAG_SOURCE})(?:\`(?![^\`\\n]*\`))?`,
|
||||
'g'
|
||||
)
|
||||
|
||||
export function sanitizeChatDisplayContent(content: string): string {
|
||||
return content
|
||||
.replace(WORKSPACE_RESOURCE_CODE_SPAN_PATTERN, '$1')
|
||||
.replace(CODE_SPAN_OR_FLANKED_TAG, (match, tag?: string) => {
|
||||
// A tag with stray backticks against it: keep the tag, drop the strays.
|
||||
if (tag !== undefined) return tag
|
||||
|
||||
// A code span. Unwrap it only when it genuinely holds a tag — the parser
|
||||
// lifts the tag out either way, so leaving the delimiters would strand a
|
||||
// pair of backticks around a hole. Anything else is someone else's span.
|
||||
const inner = match.slice(1, -1)
|
||||
return COMPLETE_WORKSPACE_RESOURCE_TAG.test(inner) ? inner : match
|
||||
})
|
||||
.replace(HIDDEN_INLINE_REFERENCE_PATTERN, '')
|
||||
.replace(/`(\s*<workspace_resource>)/g, '$1')
|
||||
.replace(/(<\/workspace_resource>\s*)`/g, '$1')
|
||||
}
|
||||
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
/**
|
||||
* Repeated mention of a tag name that never closes — the shape both the parser
|
||||
* and the display sanitizer used to be quadratic on, because a scan allowed to
|
||||
* cross an opener restarts from every opener.
|
||||
*/
|
||||
function buildRepeatedTagMentions(times: number): string {
|
||||
return 'The <workspace_resource> tag is used here. '.repeat(times)
|
||||
}
|
||||
|
||||
/** Fastest of five runs, so a single scheduling hiccup cannot skew the sample. */
|
||||
function fastest(run: (content: string) => void, content: string): number {
|
||||
let best = Number.POSITIVE_INFINITY
|
||||
for (let attempt = 0; attempt < 5; attempt++) {
|
||||
const startedAt = performance.now()
|
||||
run(content)
|
||||
best = Math.min(best, performance.now() - startedAt)
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
/**
|
||||
* How much slower `run` gets when its input grows 4x.
|
||||
*
|
||||
* Complexity is asserted as a RATIO rather than a wall-clock ceiling. A fixed
|
||||
* millisecond bound measures the machine as much as the algorithm: it fails on a
|
||||
* loaded CI box, and set generously enough not to, it lets a genuine quadratic
|
||||
* through at the single size it happens to sample. Quadratic costs ~16x for 4x
|
||||
* the input; linear costs ~4x.
|
||||
*/
|
||||
export function scalingRatioOver4x(run: (content: string) => void): number {
|
||||
// Warm up first — the JIT would otherwise charge the whole compile to the
|
||||
// small sample and flatter the ratio.
|
||||
fastest(run, buildRepeatedTagMentions(2_000))
|
||||
|
||||
const small = fastest(run, buildRepeatedTagMentions(2_000))
|
||||
const large = fastest(run, buildRepeatedTagMentions(8_000))
|
||||
|
||||
return large / small
|
||||
}
|
||||
+780
-9
@@ -14,11 +14,61 @@ vi.mock('@/lib/auth/auth-client', () => ({
|
||||
useSession: vi.fn(() => ({ data: null, isPending: false })),
|
||||
}))
|
||||
|
||||
import { scalingRatioOver4x } from '@/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers'
|
||||
import type {
|
||||
ContentSegment,
|
||||
IndexOfCache,
|
||||
} from '@/app/workspace/[workspaceId]/home/components/message-content/components/special-tags/special-tags'
|
||||
import {
|
||||
memoizedIndexOf,
|
||||
parseQuestionTagBody,
|
||||
parseSpecialTags,
|
||||
SPECIAL_TAG_NAMES,
|
||||
} from '@/app/workspace/[workspaceId]/home/components/message-content/components/special-tags/special-tags'
|
||||
|
||||
/**
|
||||
* What a reader actually sees: the renderer concatenates adjacent text segments
|
||||
* into one markdown string, so how a span is split across segments is not
|
||||
* observable. Assert on this rather than on segment-array shape.
|
||||
*/
|
||||
function renderedText(segments: ContentSegment[]): string {
|
||||
return segments.map((segment) => ('content' in segment ? segment.content : '')).join('')
|
||||
}
|
||||
|
||||
/**
|
||||
* What the reader can actually see. Mirrors chat-content.tsx: adjacent text
|
||||
* segments concatenate, a `thinking` segment renders NOTHING, and every other
|
||||
* segment is a card. Distinct from {@link renderedText}, which counts a thinking
|
||||
* body as text — using that here would hide a tag whose close swallows content
|
||||
* the stream had already put on screen.
|
||||
*/
|
||||
function visibleView(segments: ContentSegment[]) {
|
||||
return {
|
||||
text: segments
|
||||
.map((segment) =>
|
||||
segment.type === 'thinking' ? '' : 'content' in segment ? segment.content : ' CARD'
|
||||
)
|
||||
.join(''),
|
||||
cardCount: segments.filter((segment) => segment.type !== 'text' && segment.type !== 'thinking')
|
||||
.length,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Replays `content` the way it streams — one growing prefix per frame — and
|
||||
* returns what the reader would see on each. A parser bug that only shows up
|
||||
* between frames (a card that renders and then un-renders, text that appears and
|
||||
* then vanishes) is invisible to a single end-state assertion.
|
||||
*/
|
||||
function replayFrames(content: string, step = 1) {
|
||||
const frames: ReturnType<typeof visibleView>[] = []
|
||||
for (let end = 1; end <= content.length; end += step) {
|
||||
frames.push(visibleView(parseSpecialTags(content.slice(0, end), true).segments))
|
||||
}
|
||||
frames.push(visibleView(parseSpecialTags(content, false).segments))
|
||||
return frames
|
||||
}
|
||||
|
||||
const SINGLE_SELECT = {
|
||||
type: 'single_select',
|
||||
prompt: 'How should I handle the duplicate emails?',
|
||||
@@ -149,22 +199,514 @@ describe('parseSpecialTags with <question>', () => {
|
||||
expect(segments).toEqual([{ type: 'text', content: 'Thinking about it. ' }])
|
||||
})
|
||||
|
||||
it('strips a trailing partial opening tag while streaming', () => {
|
||||
const { segments, hasPendingTag } = parseSpecialTags('Let me ask. <ques', true)
|
||||
expect(hasPendingTag).toBe(true)
|
||||
expect(segments).toEqual([{ type: 'text', content: 'Let me ask. ' }])
|
||||
it('keeps the text when a matched pair fails to parse', () => {
|
||||
// Verbatim from a real message (trace b095e080). The model explained the
|
||||
// tag and ended with a backticked example containing a REAL closing tag,
|
||||
// which closed the earlier opener and made everything between it the body.
|
||||
// That body is not valid JSON, so the segment was dropped and the render
|
||||
// resumed mid-sentence at ") is what actually produces the interactive
|
||||
// chip." — three paragraphs silently gone.
|
||||
const raw =
|
||||
'Here you go — with the ending tag intentionally malformed as `</workflow_resource>`:\n\n' +
|
||||
'<workspace_resource>{"type": "file", "path": "files/notes.md", "title": "notes.md"}</workflow_resource>\n\n' +
|
||||
"Since the closing tag doesn't match the opening `<workspace_resource>`, the chat won't " +
|
||||
'recognize it as a valid resource chip. A properly matched pair ' +
|
||||
'(`<workspace_resource>...</workspace_resource>`) is what actually produces the interactive chip.'
|
||||
|
||||
const rendered = renderedText(parseSpecialTags(raw, false).segments)
|
||||
|
||||
expect(rendered).toContain("Since the closing tag doesn't match")
|
||||
expect(rendered).toContain('A properly matched pair')
|
||||
expect(rendered).toContain('"path": "files/notes.md"')
|
||||
// No segment renders as a resource chip — the body was never valid.
|
||||
expect(parseSpecialTags(raw, false).segments.some((s) => s.type === 'workspace_resource')).toBe(
|
||||
false
|
||||
)
|
||||
})
|
||||
|
||||
it('drops a question tag with an invalid body but keeps surrounding text', () => {
|
||||
it('still parses a valid tag that follows a rejected one', () => {
|
||||
const { segments } = parseSpecialTags(
|
||||
'I use <thinking> loosely here. Anyway: <options>[{"title":"A","description":"d"}]</options> done.',
|
||||
false
|
||||
)
|
||||
expect(segments.map((segment) => segment.type)).toContain('options')
|
||||
})
|
||||
|
||||
it('loses nothing when the model writes no closing tag at all', () => {
|
||||
// Verbatim from a real message (trace 220cc02d). No close tag exists, so no
|
||||
// marker rule can fire — but the JSON value completes and prose follows,
|
||||
// which settles it at the first space. Asserted as LOSSLESS: mid-stream and
|
||||
// complete, every character survives.
|
||||
const raw =
|
||||
'The dataset lives in <workspace_resource>{"type": "file", "path": "files/notes.md"} and I keep coming back to it whenever I need a quick reference. It never quite has everything.'
|
||||
const streaming = parseSpecialTags(raw, true)
|
||||
expect(streaming.hasPendingTag).toBe(false)
|
||||
expect(renderedText(streaming.segments)).toBe(raw)
|
||||
expect(renderedText(parseSpecialTags(raw, false).segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('does not rescan the interior of a body that carried no markers', () => {
|
||||
// Pins WHY the two literal reasons resume at different offsets. A
|
||||
// never-a-payload body resumes past the CLOSE; resuming past the opener
|
||||
// instead would rescan the interior, and since the marker scan runs on the
|
||||
// blanked body, a tag quoted inside a JSON string is invisible to it and
|
||||
// would be re-parsed as a real tag on the second pass — then dropped,
|
||||
// deleting the very text this parser exists to preserve.
|
||||
const raw =
|
||||
'A <question>{"a":"<options>{\\"k\\":{\\"title\\":\\"x\\",\\"description\\":\\"y\\"}}</options>"} junk</question> B'
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('keeps prose a tag wrapped instead of a payload', () => {
|
||||
// Verbatim from a real message (trace 1206fd8a): a matched pair whose body
|
||||
// is plain prose, never an attempted JSON payload. The sentence read
|
||||
// "...once I wired up to handle the welcome sequence" with the subject gone.
|
||||
const raw =
|
||||
'once I wired up <workspace_resource>the gmail-agent workflow</workspace_resource> to handle the welcome sequence.'
|
||||
const rendered = renderedText(parseSpecialTags(raw, false).segments)
|
||||
expect(rendered).toContain('the gmail-agent workflow')
|
||||
expect(rendered).toContain('to handle the welcome sequence')
|
||||
})
|
||||
|
||||
it('shows a body that will not parse at all, rather than dropping it', () => {
|
||||
// `discard` is only defensible for a payload the agent actually FORMED —
|
||||
// valid JSON that failed its shape guard. Bracket depth cannot tell prose
|
||||
// wrapped in braces from a real payload, so without an actual parse these
|
||||
// were deleted: the first is a resource name someone wrote in braces, the
|
||||
// other two are the commonest JSON slips a model makes.
|
||||
const cases = [
|
||||
'I saved <workspace_resource>{the Q4 report}</workspace_resource> for you.',
|
||||
'See <workspace_resource>{type: "file", path: "a.md"}</workspace_resource> ok',
|
||||
"See <workspace_resource>{'type':'file'}</workspace_resource> ok",
|
||||
]
|
||||
for (const raw of cases) {
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
}
|
||||
})
|
||||
|
||||
it('still drops a marker-free malformed payload rather than showing raw JSON', () => {
|
||||
// The complement of the case above: no tag markers in the body, so this is
|
||||
// a genuinely broken emission from the agent, not swallowed prose.
|
||||
const { segments, hasPendingTag } = parseSpecialTags(
|
||||
'Before. <question>{"type":"single_select"}</question> After.',
|
||||
false
|
||||
)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
expect(segments).toEqual([
|
||||
{ type: 'text', content: 'Before. ' },
|
||||
{ type: 'text', content: ' After.' },
|
||||
])
|
||||
// Asserted on the rendered text, not the segment array: how the surviving
|
||||
// prose is split across text segments is display-neutral, so pinning the
|
||||
// array shape would break on a behavior-preserving change to the split.
|
||||
expect(renderedText(segments)).toBe('Before. After.')
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
})
|
||||
|
||||
it('drops that same payload even when its JSON quotes tag syntax', () => {
|
||||
// The marker scan must blank JSON strings the way the streaming path does.
|
||||
// Scanning the raw body sees `</options>` inside the payload, calls the span
|
||||
// literal text, and renders the raw JSON — the outcome `discard` exists to
|
||||
// prevent.
|
||||
//
|
||||
// The quoted marker deliberately sits in a field OTHER than `prompt`: a
|
||||
// recoverable prompt is surfaced as text before this path is reached, so a
|
||||
// fixture carrying one would assert the recovery rather than the blanking
|
||||
// this test exists for. Matches the prompt-less body used above.
|
||||
const { segments } = parseSpecialTags(
|
||||
'A <question>[{"type":"single_select","title":"use </options> here?"}]</question> B',
|
||||
false
|
||||
)
|
||||
expect(renderedText(segments)).toBe('A B')
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
})
|
||||
|
||||
it('does not flash the payload while the closing tag is still arriving', () => {
|
||||
// Each frame below is a real mid-stream state: the JSON value has closed, so
|
||||
// without tolerating an arriving close the trailing `</opt` reads as stray
|
||||
// content and the whole payload is released as text until the final `>`.
|
||||
for (const fragment of ['<', '</', '</o', '</opt', '</options']) {
|
||||
const { segments, hasPendingTag } = parseSpecialTags(
|
||||
`see <options>[{"title":"a","description":"b"}]${fragment}`,
|
||||
true
|
||||
)
|
||||
expect(hasPendingTag).toBe(true)
|
||||
expect(renderedText(segments)).toBe('see ')
|
||||
}
|
||||
})
|
||||
|
||||
it('still rejects a close whose name is wrong rather than merely unfinished', () => {
|
||||
// The counterpart to the case above: `</workflow_resource>` can never grow
|
||||
// into `</workspace_resource>`, so it settles immediately instead of hiding
|
||||
// the rest of the message for the remainder of the stream.
|
||||
const raw =
|
||||
'see <workspace_resource>{"type":"file","path":"a.md"}</workflow_resource> and then prose.'
|
||||
const { hasPendingTag, segments } = parseSpecialTags(raw, true)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
// Asserted on the text too, not just the flag: a wrong resumeAt keeps the
|
||||
// flag correct while dropping the prose, which is the defect class this
|
||||
// whole change exists to prevent.
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('keeps a valid tag whose close an earlier broken tag would borrow', () => {
|
||||
// The first opener misspells its close, so it reaches forward and matches
|
||||
// the SECOND tag's close, swallowing a perfectly good resource into one
|
||||
// literal span. Resuming past the opener re-scans the interior instead.
|
||||
const raw =
|
||||
'See <workspace_resource>{"type":"file","path":"a.md"}</workflow_resource>\n' +
|
||||
'and a real one: <workspace_resource>{"type":"file","path":"b.md","title":"b"}</workspace_resource>'
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(segments.some((s) => s.type === 'workspace_resource')).toBe(true)
|
||||
expect(renderedText(segments)).toContain('</workflow_resource>')
|
||||
})
|
||||
|
||||
it('finds a nested tag an unbalanced quote hid from the blanked scan', () => {
|
||||
// One stray `"` is enough to make blankJsonStringLiterals treat the REST of
|
||||
// the body as a string literal, hiding the real `<options>` marker from the
|
||||
// scan. The verdict then degrades from `foreign-markers` to `never-a-payload`
|
||||
// and resumes past the close, flattening both nested tags into one literal
|
||||
// span — so a card already on screen un-renders when the close arrives.
|
||||
//
|
||||
// Blanking is only meaningful while the body might BE json. Once viability
|
||||
// has proved it never was, the raw text is the honest evidence.
|
||||
const raw =
|
||||
'Saved <workspace_resource>the notes file "notes.md and here is what to do next: ' +
|
||||
'<options>[{"title":"Ship it","description":"Open the PR"}]</options>\n' +
|
||||
'Full path: <workspace_resource>{"type":"file","path":"files/a.md","title":"a.md"}</workspace_resource>'
|
||||
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(segments.filter((segment) => segment.type === 'options')).toHaveLength(1)
|
||||
expect(segments.filter((segment) => segment.type === 'workspace_resource')).toHaveLength(1)
|
||||
// Balancing the quote must reach the same two cards — the quote is the only
|
||||
// difference, so this pins that it was never load-bearing for the outcome.
|
||||
const balanced = raw.replace('the notes file "notes.md', 'the notes file notes.md')
|
||||
const control = parseSpecialTags(balanced, false).segments
|
||||
expect(control.filter((segment) => segment.type === 'options')).toHaveLength(1)
|
||||
expect(control.filter((segment) => segment.type === 'workspace_resource')).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('never un-renders that card as the closing tag arrives', () => {
|
||||
// The frame-level face of the case above, and the invariant it broke: the
|
||||
// options card is on screen for many frames before the final `>` lands. A
|
||||
// card that renders must never revert to raw text.
|
||||
const raw =
|
||||
'Saved <workspace_resource>the notes file "notes.md and here is what to do next: ' +
|
||||
'<options>[{"title":"Ship it","description":"Open the PR"}]</options>\n' +
|
||||
'Full path: <workspace_resource>{"type":"file","path":"files/a.md","title":"a.md"}</workspace_resource>'
|
||||
|
||||
let sawCard = false
|
||||
for (let end = 1; end <= raw.length; end++) {
|
||||
const { segments } = parseSpecialTags(raw.slice(0, end), true)
|
||||
const hasCard = segments.some((segment) => segment.type === 'options')
|
||||
if (hasCard) sawCard = true
|
||||
expect(!sawCard || hasCard, `options card retracted at frame ${end}`).toBe(true)
|
||||
}
|
||||
expect(sawCard).toBe(true)
|
||||
// ...and the settled parse still has it.
|
||||
expect(parseSpecialTags(raw, false).segments.some((s) => s.type === 'options')).toBe(true)
|
||||
})
|
||||
|
||||
it('does not delete tag syntax quoted inside the body it rescans', () => {
|
||||
// The rescan decides on the BLANKED body, so a tag quoted inside a JSON
|
||||
// string is invisible to it. Resuming at the opener would re-scan that
|
||||
// quoted text raw, re-parse it as a real tag, and drop it — deleting text.
|
||||
// Resuming at the MARKER skips the quoted region, so it survives verbatim.
|
||||
const inner =
|
||||
'<credential>{\\"type\\":\\"link\\",\\"value\\":\\"https://x.example/p\\"}</credential>'
|
||||
const raw = `A <question>{"prompt":"${inner}"} </options></question> B`
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps the blank line between two rejected spans', () => {
|
||||
// The renderer concatenates adjacent text segments into one markdown string,
|
||||
// so a dropped whitespace-only span silently merges two paragraphs.
|
||||
const raw =
|
||||
'<workspace_resource>prose one</workspace_resource>\n\n<workspace_resource>prose two</workspace_resource>'
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('shows an oversized body it only partly inspected rather than discarding it', () => {
|
||||
// Only the first MAX_UNCLOSED_BODY_SCAN characters are scanned. Finding no
|
||||
// reason within that window is not evidence the body was a real payload, so
|
||||
// the span must be shown — discarding would delete text never examined.
|
||||
const body = `{"type":"file","path":"a.md","note":"${'x'.repeat(5000)}`
|
||||
const raw = `see <workspace_resource>${body}</workspace_resource> end`
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('settles a prose mention at any length, but defers a payload that closes past the window', () => {
|
||||
// The scan window's accepted blind spot, pinned so it stays a decision.
|
||||
//
|
||||
// A mention in prose settles at its FIRST character however long the message
|
||||
// runs — prose does not open with `{`, so viability fails immediately.
|
||||
const mention = `see <workspace_resource> ${'long prose. '.repeat(600)}`
|
||||
expect(parseSpecialTags(mention, true).hasPendingTag).toBe(false)
|
||||
|
||||
// But a JSON body whose top-level value closes BEYOND the window still reads
|
||||
// as a viable prefix, so the tail stays hidden until the stream ends. Needs a
|
||||
// payload several times larger than any tag emits, and it is lossless once
|
||||
// complete — the cost of bounding a scan that otherwise stalls the main
|
||||
// thread.
|
||||
const oversized = `see <workspace_resource>{"type":"file","note":"${'x'.repeat(5000)}"} and then prose.`
|
||||
expect(parseSpecialTags(oversized, true).hasPendingTag).toBe(true)
|
||||
expect(renderedText(parseSpecialTags(oversized, false).segments)).toBe(oversized)
|
||||
})
|
||||
|
||||
it('still finds a valid tag sitting past the scan window inside a borrowed body', () => {
|
||||
// The first opener has no close of its own, so it borrows the inner tag's.
|
||||
// Its body is marker-free prose for far longer than the scan window, so the
|
||||
// truncated inspection sees only prose and can say nothing about the rest.
|
||||
//
|
||||
// Resuming past the borrowed close would flatten the inner tag to text purely
|
||||
// because of where it fell relative to the window. Resuming at the first
|
||||
// uninspected character finds it. Asserted at three lengths so the boundary
|
||||
// itself is covered, not just one side of it.
|
||||
const inner =
|
||||
'<workspace_resource>{"type":"file","path":"files/b.md","title":"b.md"}</workspace_resource>'
|
||||
const build = (proseChars: number) =>
|
||||
`See <workspace_resource>${'prose word '.repeat(Math.ceil(proseChars / 11))}${inner} end`
|
||||
|
||||
for (const proseChars of [1_000, 6_000, 60_000]) {
|
||||
const raw = build(proseChars)
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(segments.filter((segment) => segment.type === 'workspace_resource')).toHaveLength(1)
|
||||
// The prose around it survives too — the span is emitted, not skipped.
|
||||
expect(renderedText(segments)).toContain('See <workspace_resource>prose word')
|
||||
expect(renderedText(segments)).toContain(' end')
|
||||
}
|
||||
})
|
||||
|
||||
it('still finds a valid tag whose opener STRADDLES the scan-window edge', () => {
|
||||
// The window edge is an arbitrary cut, so an opener can begin just before it
|
||||
// and finish just after. The test above steps in 11-character units and so
|
||||
// lands on only a few offsets; the straddle needs every offset in the band.
|
||||
//
|
||||
// Resuming exactly at the edge left the opener's `<` behind the cursor, and
|
||||
// the opener scan only looks FORWARD — so the tag was never found and its
|
||||
// payload rendered as raw JSON text, on a COMPLETED message. Every offset
|
||||
// across the band must still produce the card.
|
||||
const inner =
|
||||
'<workspace_resource>{"type":"file","path":"files/b.md","title":"b.md"}</workspace_resource>'
|
||||
|
||||
for (let filler = 4_060; filler <= 4_110; filler++) {
|
||||
const raw = `See <workspace_resource>${'z'.repeat(filler)}${inner} end`
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(
|
||||
segments.filter((segment) => segment.type === 'workspace_resource'),
|
||||
`filler ${filler}`
|
||||
).toHaveLength(1)
|
||||
// Nothing is duplicated or dropped by the rewind either.
|
||||
expect(renderedText(segments), `filler ${filler}`).toContain(' end')
|
||||
}
|
||||
})
|
||||
|
||||
it('still renders a matched pair whose body IS valid', () => {
|
||||
const raw =
|
||||
'see <workspace_resource>{"type":"file","path":"files/a.md","title":"a.md"}</workspace_resource> ok'
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(segments.some((s) => s.type === 'workspace_resource')).toBe(true)
|
||||
})
|
||||
|
||||
it('shows prose immediately mid-stream instead of blanking the rest', () => {
|
||||
const content = 'The `<workspace_resource>` chip only renders for a real file.'
|
||||
const { segments, hasPendingTag } = parseSpecialTags(content, true)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
expect(segments.map((s) => ('content' in s ? s.content : s.type)).join('')).toContain(
|
||||
'chip only renders for a real file.'
|
||||
)
|
||||
})
|
||||
|
||||
it('shows text once the JSON value has closed and stray content follows', () => {
|
||||
// Verbatim shape from a real message (trace afbeefd0): the close tag was
|
||||
// TRUNCATED to `</workspac`, so no marker rule can see it — but the JSON
|
||||
// value completes at the `}`, which makes everything after it fatal.
|
||||
const raw =
|
||||
'kicks off in <workspace_resource>{"type":"file","path":"files/notes.md"}</workspac and after that I brew a cup of coffee.'
|
||||
const { segments, hasPendingTag } = parseSpecialTags(raw, true)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
expect(renderedText(segments)).toContain('I brew a cup of coffee')
|
||||
})
|
||||
|
||||
it('tolerates braces inside JSON strings when tracking depth', () => {
|
||||
const raw = 'x <workspace_resource>{"title":"a } b","path":"files/a.md"'
|
||||
expect(parseSpecialTags(raw, true).hasPendingTag).toBe(true)
|
||||
})
|
||||
|
||||
it('does not let an escaped quote end a string early and skew the depth', () => {
|
||||
// If `\"` were read as the closing quote, the following `}` would count as a
|
||||
// real close, the top-level value would look finished, and the trailing text
|
||||
// would settle the tag as unresolvable mid-payload.
|
||||
const raw = 'x <workspace_resource>{"title":"a \\" } b","path":"files/a.md"'
|
||||
expect(parseSpecialTags(raw, true).hasPendingTag).toBe(true)
|
||||
})
|
||||
|
||||
it('still suppresses a JSON-bodied tag that is genuinely mid-stream', () => {
|
||||
const { segments, hasPendingTag } = parseSpecialTags(
|
||||
'Here you go <workspace_resource>{"type":"file","id":"abc"',
|
||||
true
|
||||
)
|
||||
expect(hasPendingTag).toBe(true)
|
||||
expect(segments).toEqual([{ type: 'text', content: 'Here you go ' }])
|
||||
})
|
||||
|
||||
it('bails when a foreign closing tag appears inside a prose body', () => {
|
||||
// Tags never nest, so a close for a different tag proves the opener was text.
|
||||
// Asserted on `thinking` because that is the only tag the nesting rule still
|
||||
// serves: a JSON body has no need of it, since a marker outside a string
|
||||
// literal is content the viability rule already rejects, and one inside is
|
||||
// legitimate quoted syntax that must not count as evidence.
|
||||
const raw = 'see <thinking>weighing it </question> more'
|
||||
const { hasPendingTag, segments } = parseSpecialTags(raw, true)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
expect(renderedText(segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('does not bail on tag syntax quoted inside a JSON string', () => {
|
||||
// The false positive this guards: a question whose text legitimately quotes
|
||||
// another tag. Bailing would show raw JSON that later snaps into a card.
|
||||
const streaming = 'ok <question>[{"type":"single_select","prompt":"Use the </options> tag?"'
|
||||
expect(parseSpecialTags(streaming, true).hasPendingTag).toBe(true)
|
||||
})
|
||||
|
||||
it('resolves that same question correctly once it closes', () => {
|
||||
// The other half of the guarantee: the body the streaming case refused to
|
||||
// bail on does render as a question card, so nothing flickered for nothing.
|
||||
const complete =
|
||||
'ok <question>[{"type":"single_select","prompt":"Use the </options> tag?","options":[{"id":"y","label":"Yes"},{"id":"n","label":"No"}]}]</question>'
|
||||
const { segments } = parseSpecialTags(complete, false)
|
||||
expect(segments.some((s) => s.type === 'question')).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects an opener a nested one disproves, then judges the inner on its own', () => {
|
||||
// Each opener is evaluated independently. The first is disproved by the
|
||||
// nested opener and its text is released immediately; the second is a fresh
|
||||
// candidate that nothing has ruled out yet, so it holds mid-stream.
|
||||
const streaming = parseSpecialTags('a <thinking>b <thinking> c', true)
|
||||
expect(streaming.hasPendingTag).toBe(true)
|
||||
expect(renderedText(streaming.segments)).toBe('a <thinking>b ')
|
||||
|
||||
// Once the stream ends nothing can close it, so the whole line is shown.
|
||||
const done = parseSpecialTags('a <thinking>b <thinking> c', false)
|
||||
expect(done.hasPendingTag).toBe(false)
|
||||
expect(renderedText(done.segments)).toBe('a <thinking>b <thinking> c')
|
||||
})
|
||||
|
||||
it('keeps reasoning suppressed when the body merely contains angle brackets', () => {
|
||||
// The nesting rule keys on tag NAMES, not on anything tag-shaped. Reasoning
|
||||
// that mentions `<div>` or a generic is still reasoning; releasing it would
|
||||
// put the model's thinking on screen for an incidental angle bracket.
|
||||
const { segments } = parseSpecialTags('a <thinking>weighing a <div> here</thinking> b', false)
|
||||
|
||||
expect(segments.some((segment) => segment.type === 'thinking')).toBe(true)
|
||||
expect(visibleView(segments).text).toBe('a b')
|
||||
})
|
||||
|
||||
it('renders nothing for a message that is only a discarded payload', () => {
|
||||
// `discard` emits no segment, so this is the one case that can end the parse
|
||||
// with an empty segment list. The fallback for an empty list is to emit the
|
||||
// raw content — which would put back the exact raw JSON the discard removed.
|
||||
const { segments } = parseSpecialTags('<question>{"type":"single_select"}</question>', false)
|
||||
|
||||
expect(visibleView(segments).text).toBe('')
|
||||
})
|
||||
|
||||
it('settles a long prose mention without scanning the whole window', () => {
|
||||
// Viability rejects on the first non-whitespace character when it is not `{`
|
||||
// or `[` — the common case. Testing that before blanking avoids copying a
|
||||
// full window per opener per chunk: 43ms to 2ms on this input.
|
||||
//
|
||||
// Asserted as a scaling ratio, not a wall-clock ceiling — see
|
||||
// {@link scalingRatioOver4x} for why.
|
||||
expect(scalingRatioOver4x((content) => parseSpecialTags(content, true))).toBeLessThan(8)
|
||||
})
|
||||
|
||||
it('does not let a late thinking close swallow content already on screen', () => {
|
||||
// A nested marker disproves the outer <thinking> mid-stream, so its text is
|
||||
// released and the inner tag renders as a card. A prose body has no shape to
|
||||
// fail — any non-empty text qualifies — so when </thinking> finally arrives it
|
||||
// would be accepted as a segment, and everything already on screen would be
|
||||
// swallowed into it and suppressed. The nesting rule has to apply on the
|
||||
// matched-pair path too, not just while streaming.
|
||||
const raw = 'a <thinking>b <options>[{"title":"x","description":"y"}]</options> c</thinking> d'
|
||||
|
||||
const settled = parseSpecialTags(raw, false)
|
||||
expect(settled.segments.some((segment) => segment.type === 'options')).toBe(true)
|
||||
expect(settled.segments.some((segment) => segment.type === 'thinking')).toBe(false)
|
||||
|
||||
// And nothing retracts across the stream: no rendered card un-renders.
|
||||
const frames = replayFrames(raw)
|
||||
let previous = 0
|
||||
for (const frame of frames) {
|
||||
expect(frame.cardCount).toBeGreaterThanOrEqual(previous)
|
||||
previous = frame.cardCount
|
||||
}
|
||||
})
|
||||
|
||||
it('hides an unclosed thinking body while streaming, then shows it once complete', () => {
|
||||
// A DELIBERATE trade, not an oversight. `thinking` bodies are prose, so the
|
||||
// JSON viability rule cannot apply and only the nesting rule can disprove the
|
||||
// opener — mid-stream the default is therefore to HIDE, since a close is
|
||||
// still plausible and releasing early would flash reasoning that is about to
|
||||
// become a suppressed segment.
|
||||
//
|
||||
// Once the stream ends the body is shown as text, which does leak the model's
|
||||
// reasoning for a message whose close never arrived. Accepted: forgetting the
|
||||
// close is rare, and the alternative — keeping it hidden — would swallow the
|
||||
// answer whenever the model opened `<thinking>` and then wrote the reply
|
||||
// without closing, which is the text-loss bug this whole change removes.
|
||||
const raw = 'a <thinking>still reasoning about'
|
||||
const streaming = parseSpecialTags(raw, true)
|
||||
expect(streaming.hasPendingTag).toBe(true)
|
||||
expect(renderedText(streaming.segments)).toBe('a ')
|
||||
|
||||
const complete = parseSpecialTags(raw, false)
|
||||
expect(complete.hasPendingTag).toBe(false)
|
||||
expect(renderedText(complete.segments)).toBe(raw)
|
||||
})
|
||||
|
||||
it('never retracts rendered text or a card across streamed frames', () => {
|
||||
// Frame-to-frame stability, which no end-state assertion can see. Replays a
|
||||
// real message one character at a time: text already shown must never
|
||||
// disappear, and a card once rendered must never revert to raw text.
|
||||
const content =
|
||||
'Updated <workspace_resource>{"type":"file","path":"files/a.md","title":"a.md"}</workspace_resource> ' +
|
||||
'and left `<question>` alone. ' +
|
||||
'<options>[{"title":"Ship it","description":"open the PR"}]</options>'
|
||||
|
||||
const frames = replayFrames(content)
|
||||
|
||||
// Card count is monotonically non-decreasing. Appending to the buffer can
|
||||
// only add closes AFTER the ones already matched, so no earlier opener's
|
||||
// resolution can change — a card that renders must never un-render.
|
||||
let previous = 0
|
||||
for (const frame of frames) {
|
||||
expect(frame.cardCount).toBeGreaterThanOrEqual(previous)
|
||||
previous = frame.cardCount
|
||||
}
|
||||
|
||||
// The settled parse is the richest: both tags resolved, prose intact.
|
||||
const settled = frames[frames.length - 1]
|
||||
expect(settled.cardCount).toBe(2)
|
||||
expect(settled.text).toContain('and left `<question>` alone.')
|
||||
})
|
||||
|
||||
it('renders an unclosed tag as text once the message is complete', () => {
|
||||
const content =
|
||||
'The `<workspace_resource>` file chip only renders when its path points to a real file.'
|
||||
const { segments, hasPendingTag } = parseSpecialTags(content, false)
|
||||
expect(hasPendingTag).toBe(false)
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
expect(renderedText(segments)).toBe(content)
|
||||
})
|
||||
|
||||
it('strips a trailing partial opening tag while streaming', () => {
|
||||
const { segments, hasPendingTag } = parseSpecialTags('Let me ask. <ques', true)
|
||||
expect(hasPendingTag).toBe(true)
|
||||
expect(segments).toEqual([{ type: 'text', content: 'Let me ask. ' }])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -251,3 +793,232 @@ describe('service_account tag validation', () => {
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
describe('memoizedIndexOf', () => {
|
||||
const CONTENT =
|
||||
'Use <workspace_resource> for files. Use <question> for cards. ' +
|
||||
'<options>[{"title":"Ship","description":"go"}]</options> and <question> again.'
|
||||
// Every opener the parser actually searches for, not a hand-picked few: the
|
||||
// cache is keyed per needle, so a needle absent from CONTENT exercises the
|
||||
// cached -1 path and a repeated one exercises reuse as the cursor advances.
|
||||
const NEEDLES = SPECIAL_TAG_NAMES.map((name) => `<${name}>`)
|
||||
|
||||
it('matches plain indexOf as the cursor advances', () => {
|
||||
const cache: IndexOfCache = new Map()
|
||||
for (let from = 0; from <= CONTENT.length; from++) {
|
||||
for (const needle of NEEDLES) {
|
||||
expect(memoizedIndexOf(cache, CONTENT, needle, from)).toBe(CONTENT.indexOf(needle, from))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('stays correct when the cursor moves BACKWARD', () => {
|
||||
// The cache is only reused when the new `from` is at or beyond the offset the
|
||||
// entry was searched at. Without that guard a cached hit — or a cached -1 —
|
||||
// is returned for a region it never examined, and the parser silently
|
||||
// mis-parses rather than failing loudly.
|
||||
//
|
||||
// parseSpecialTags never walks backward today, so this cannot be provoked
|
||||
// through the public API; the point is that a future change to a resume point
|
||||
// costs a redundant scan instead of a wrong answer.
|
||||
const cache: IndexOfCache = new Map()
|
||||
const offsets = [70, 5, 100, 0, 45, 62, 12, CONTENT.length, 40, 3]
|
||||
for (const from of offsets) {
|
||||
for (const needle of NEEDLES) {
|
||||
expect(memoizedIndexOf(cache, CONTENT, needle, from)).toBe(CONTENT.indexOf(needle, from))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('caches an absent needle instead of rescanning', () => {
|
||||
const cache: IndexOfCache = new Map()
|
||||
expect(memoizedIndexOf(cache, CONTENT, '<thinking>', 0)).toBe(-1)
|
||||
// Same offset or later: answerable from the entry, since absence from 0
|
||||
// implies absence from anywhere after it.
|
||||
expect(memoizedIndexOf(cache, CONTENT, '<thinking>', 30)).toBe(-1)
|
||||
expect(cache.get('<thinking>')).toEqual({ idx: -1, from: 0 })
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* Property tests over generated messages.
|
||||
*
|
||||
* The example tests above each pin one shape that was once broken. They cannot
|
||||
* cover the space: a body is judged on body kind, close state, JSON state, marker
|
||||
* placement, size against the scan window, and streaming mode — a product of
|
||||
* roughly six hundred combinations, each needing both an outcome and a resume
|
||||
* point. Every regression found in review so far was a cell nobody had written an
|
||||
* example for.
|
||||
*
|
||||
* These assert invariants instead, over messages composed from fragments, so a
|
||||
* new combination is covered without a new test. Seeded so a failure reproduces.
|
||||
*/
|
||||
describe('parser properties', () => {
|
||||
/** mulberry32 — deterministic, so a failing case is reproducible from its seed. */
|
||||
function makeRng(seed: number): () => number {
|
||||
let a = seed >>> 0
|
||||
return () => {
|
||||
a = (a + 0x6d2b79f5) >>> 0
|
||||
let t = Math.imul(a ^ (a >>> 15), 1 | a)
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One valid payload per card-rendering tag, keyed by tag name so
|
||||
* {@link SPECIAL_TAG_NAMES} can be checked for full coverage below. Hand-picking
|
||||
* a subset is how three of these went unexercised by every invariant without
|
||||
* anything failing to say so.
|
||||
*/
|
||||
const VALID_TAG_BY_NAME: Record<string, string> = {
|
||||
workspace_resource:
|
||||
'<workspace_resource>{"type":"file","path":"files/a.md","title":"a.md"}</workspace_resource>',
|
||||
options: '<options>[{"title":"Ship it","description":"Open the PR"}]</options>',
|
||||
question: `<question>${JSON.stringify(SINGLE_SELECT)}</question>`,
|
||||
credential:
|
||||
'<credential>{"type":"link","provider":"slack","value":"https://x.example/p"}</credential>',
|
||||
usage_upgrade:
|
||||
'<usage_upgrade>{"reason":"monthly cap","action":"upgrade_plan","message":"You hit your limit."}</usage_upgrade>',
|
||||
'mothership-error':
|
||||
'<mothership-error>{"message":"The tool call failed.","code":"E_TOOL"}</mothership-error>',
|
||||
}
|
||||
|
||||
/** Renders nothing rather than a card, so it cannot carry a card invariant. */
|
||||
const TAGS_WITHOUT_CARDS = ['thinking']
|
||||
|
||||
const VALID_TAGS = Object.values(VALID_TAG_BY_NAME)
|
||||
|
||||
it('covers every tag the parser knows', () => {
|
||||
// The invariants below are only as good as this list. Deriving the check from
|
||||
// SPECIAL_TAG_NAMES makes adding a tag without a fixture fail loudly here,
|
||||
// instead of quietly leaving it outside every property in this file.
|
||||
expect(new Set([...Object.keys(VALID_TAG_BY_NAME), ...TAGS_WITHOUT_CARDS])).toEqual(
|
||||
new Set(SPECIAL_TAG_NAMES)
|
||||
)
|
||||
})
|
||||
|
||||
/**
|
||||
* Fragments that must survive verbatim. Every one is a shape the parser has to
|
||||
* reject: prose mentions, malformed closes, bodies that never were payloads.
|
||||
* None is a valid tag and none is a well-formed payload, so nothing here is
|
||||
* eligible for `discard` — which makes "output equals input" a legal assertion.
|
||||
*/
|
||||
const LOSSLESS_FRAGMENTS = [
|
||||
'Plain prose with no markup at all. ',
|
||||
'A `<workspace_resource>` mention in prose. ',
|
||||
'Talking about `<question>` and `<options>` together. ',
|
||||
'<workspace_resource>{"type":"file","path":"a.md"}</workflow_resource> misspelled close. ',
|
||||
'<workspace_resource>{"type":"file","path":"a.md"}</workspac truncated close. ',
|
||||
'<workspace_resource>{"type":"file","path":"a.md"} and then prose, no close. ',
|
||||
'<workspace_resource>{the Q4 report}</workspace_resource> braces round prose. ',
|
||||
'<workspace_resource>{type: "file", path: "a.md"}</workspace_resource> unquoted keys. ',
|
||||
"<workspace_resource>{'type':'file'}</workspace_resource> single quotes. ",
|
||||
'<workspace_resource>the gmail-agent workflow</workspace_resource> prose body. ',
|
||||
'<thinking>reasoning <options> with a nested marker</thinking> after. ',
|
||||
'<workspace_resource>the notes file "notes.md unbalanced quote</workspace_resource> after. ',
|
||||
'<workspace_resource>notes "unbalanced then <options> marker</workspace_resource> tail. ',
|
||||
'\n\nA paragraph break above. ',
|
||||
`${'long filler prose. '.repeat(300)}crossing the scan window. `,
|
||||
]
|
||||
|
||||
const pick = <T>(rng: () => number, xs: T[]): T => xs[Math.floor(rng() * xs.length)]
|
||||
|
||||
function buildLossless(rng: () => number, pool = LOSSLESS_FRAGMENTS): string {
|
||||
const n = 1 + Math.floor(rng() * 5)
|
||||
return Array.from({ length: n }, () => pick(rng, pool)).join('')
|
||||
}
|
||||
|
||||
/**
|
||||
* The same shapes without the window-crossing filler.
|
||||
*
|
||||
* Frame replay parses every prefix, so message length multiplies into parse
|
||||
* count — the filler fragment alone took that property to ~1M parses and 1.7s,
|
||||
* 95% of this file's runtime. Retraction is a property of what happens AT a
|
||||
* frame boundary, so it is exercised by the boundaries, not by message size.
|
||||
* The scan window still gets its coverage from the other properties, which
|
||||
* parse each message once.
|
||||
*/
|
||||
const SHORT_FRAGMENTS = LOSSLESS_FRAGMENTS.filter((fragment) => fragment.length < 200)
|
||||
|
||||
it('never loses a character of a message with nothing droppable in it', () => {
|
||||
// The headline guarantee. Only a well-formed payload that failed its shape
|
||||
// guard may be removed, and no fragment here is one.
|
||||
for (let seed = 1; seed <= 400; seed++) {
|
||||
const raw = buildLossless(makeRng(seed))
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
expect(renderedText(segments), `seed ${seed}`).toBe(raw)
|
||||
}
|
||||
})
|
||||
|
||||
it('renders every valid tag as a card whatever surrounds it', () => {
|
||||
// A valid tag was flattened to text purely because of how much
|
||||
// prose preceded it, which no fixed example set would have found.
|
||||
for (let seed = 1; seed <= 400; seed++) {
|
||||
const rng = makeRng(seed)
|
||||
const tags = Array.from({ length: 1 + Math.floor(rng() * 3) }, () => pick(rng, VALID_TAGS))
|
||||
const parts: string[] = []
|
||||
for (const tag of tags) {
|
||||
// Several fragments, not one: the interesting shapes need an unclosed
|
||||
// opener AND enough prose after it to push the valid tag past the scan
|
||||
// window, so the opener borrows that tag's close from beyond what the
|
||||
// parser inspected. One fragment between tags can never build that.
|
||||
const run = 1 + Math.floor(rng() * 3)
|
||||
for (let i = 0; i < run; i++) parts.push(pick(rng, LOSSLESS_FRAGMENTS))
|
||||
parts.push(tag)
|
||||
}
|
||||
parts.push(pick(rng, LOSSLESS_FRAGMENTS))
|
||||
const raw = parts.join('')
|
||||
|
||||
const { segments } = parseSpecialTags(raw, false)
|
||||
const cards = segments.filter(
|
||||
(segment) => segment.type !== 'text' && segment.type !== 'thinking'
|
||||
)
|
||||
expect(cards, `seed ${seed}`).toHaveLength(tags.length)
|
||||
}
|
||||
})
|
||||
|
||||
it('never un-renders a card or retracts text across streamed frames', () => {
|
||||
// Content already on screen disappeared when a later close
|
||||
// arrived. Only visible across frames, never in an end-state assertion.
|
||||
//
|
||||
// Text may shrink slightly at a frame edge: a half-arrived opening marker is
|
||||
// deliberately hidden so it does not flash. That is bounded by the longest
|
||||
// opener, so anything beyond it is a real retraction.
|
||||
const LONGEST_OPENER = '<workspace_resource>'.length + 1
|
||||
|
||||
for (let seed = 1; seed <= 120; seed++) {
|
||||
const rng = makeRng(seed)
|
||||
const raw = `${buildLossless(rng, SHORT_FRAGMENTS)}${pick(rng, VALID_TAGS)}${buildLossless(rng, SHORT_FRAGMENTS)}`
|
||||
|
||||
let previousCards = 0
|
||||
let previousText = ''
|
||||
for (const frame of replayFrames(raw, 7)) {
|
||||
expect(frame.cardCount, `seed ${seed}: card un-rendered`).toBeGreaterThanOrEqual(
|
||||
previousCards
|
||||
)
|
||||
|
||||
const stable = previousText.slice(0, Math.max(0, previousText.length - LONGEST_OPENER))
|
||||
expect(frame.text.startsWith(stable), `seed ${seed}: text retracted`).toBe(true)
|
||||
|
||||
previousCards = frame.cardCount
|
||||
previousText = frame.text
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('settles to at least what the last streaming frame showed', () => {
|
||||
// The stream ending must only ever reveal more. A settled parse that renders
|
||||
// fewer cards than the frame before it is a retraction the user watches happen.
|
||||
for (let seed = 1; seed <= 200; seed++) {
|
||||
const rng = makeRng(seed)
|
||||
const raw = `${buildLossless(rng)}${pick(rng, VALID_TAGS)}${buildLossless(rng)}`
|
||||
|
||||
const lastFrame = visibleView(parseSpecialTags(raw, true).segments)
|
||||
const settled = visibleView(parseSpecialTags(raw, false).segments)
|
||||
|
||||
expect(settled.cardCount, `seed ${seed}`).toBeGreaterThanOrEqual(lastFrame.cardCount)
|
||||
expect(settled.text.length, `seed ${seed}`).toBeGreaterThanOrEqual(lastFrame.text.length)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
+598
-34
@@ -204,7 +204,12 @@ const RUNTIME_SPECIAL_TAG_NAMES = [
|
||||
'question',
|
||||
] as const
|
||||
|
||||
const SPECIAL_TAG_NAMES = [
|
||||
/**
|
||||
* Every tag the parser resolves. Exported so tests can assert their fixtures
|
||||
* cover all of them rather than hand-picking a subset that silently drifts —
|
||||
* the same treatment the sibling `*_TYPES` unions above already get.
|
||||
*/
|
||||
export const SPECIAL_TAG_NAMES = [
|
||||
'thinking',
|
||||
'options',
|
||||
'usage_upgrade',
|
||||
@@ -432,6 +437,25 @@ export function parseTextTagBody(body: string): string | null {
|
||||
return body.trim() ? body : null
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether `body` is syntactically valid JSON, regardless of its shape.
|
||||
*
|
||||
* Separates "the agent formed a payload that failed its shape guard" from "this
|
||||
* was never JSON" — the line that decides whether a failed body may be dropped
|
||||
* or must be shown (see {@link classifyBody}). Costs a second parse of a body
|
||||
* that already failed one, which is the rare path; the common cases never reach
|
||||
* it, since a valid payload returns earlier and prose is rejected by the cheaper
|
||||
* viability rule before this runs.
|
||||
*/
|
||||
function isParseableJson(body: string): boolean {
|
||||
try {
|
||||
JSON.parse(body)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
export function parseTagAttributes(openTag: string): Record<string, string> {
|
||||
const attributes: Record<string, string> = {}
|
||||
const attributePattern = /([A-Za-z_:][A-Za-z0-9_:-]*)="([^"]*)"/g
|
||||
@@ -508,35 +532,582 @@ function parseSpecialTagData(
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses inline special tags (`<options>`, `<usage_upgrade>`, `<workspace_resource>`) from streamed
|
||||
* text content. Complete tags are extracted into typed segments; incomplete
|
||||
* tags (still streaming) are suppressed from display and flagged via
|
||||
* `hasPendingTag` so the caller can show a loading indicator.
|
||||
* Any tag-shaped marker, including names that are not special tags at all — the
|
||||
* model inventing `</workflow_resource>` is exactly the case that matters.
|
||||
*/
|
||||
const TAG_SHAPED_MARKER = /<\/?[a-zA-Z][\w-]*>/
|
||||
|
||||
/**
|
||||
* The one tag whose body is prose rather than JSON (see {@link parseTextTagBody}),
|
||||
* so a non-JSON body there says nothing about whether a close is still coming.
|
||||
*/
|
||||
const PROSE_BODY_TAG_NAME: (typeof SPECIAL_TAG_NAMES)[number] = 'thinking'
|
||||
|
||||
/**
|
||||
* Tags whose body must be JSON.
|
||||
*
|
||||
* Trailing partial opening tags (e.g. `<opt`, `<usage_`) are also stripped
|
||||
* during streaming to prevent flashing raw markup.
|
||||
* Derived from {@link SPECIAL_TAG_NAMES} rather than hand-listed: a new tag is
|
||||
* JSON-bodied by default, so forgetting to update this set cannot silently
|
||||
* downgrade it to the weaker prose heuristics. Opting a tag out is an explicit
|
||||
* edit to {@link PROSE_BODY_TAG_NAME}.
|
||||
*/
|
||||
const JSON_BODY_TAG_NAMES: ReadonlySet<(typeof SPECIAL_TAG_NAMES)[number]> = new Set(
|
||||
SPECIAL_TAG_NAMES.filter((name) => name !== PROSE_BODY_TAG_NAME)
|
||||
)
|
||||
|
||||
/**
|
||||
* How much of a body to inspect per parse, on both the unclosed and matched-pair
|
||||
* paths.
|
||||
*
|
||||
* The rules in {@link unclosedTagCannotResolve} and {@link literalTextReason}
|
||||
* decide on their FIRST piece of evidence — the first foreign marker, or the
|
||||
* first character that breaks JSON viability — so a bounded window reaches the
|
||||
* same verdict as the full remainder for any payload a tag actually carries.
|
||||
* Unbounded, the check is O(body length) and runs once per opener inside a parse
|
||||
* that re-runs for every streamed chunk. A long reply repeatedly mentioning a
|
||||
* tag name, or one whose misspelled early close stretches a single body across
|
||||
* most of the message, is then quadratic in the length of the reply.
|
||||
*
|
||||
* The window's one blind spot, and why it is accepted: a JSON body whose
|
||||
* top-level value closes BEYOND the window, followed by prose and no closing tag,
|
||||
* still reads as a viable prefix, so the remainder stays hidden until the stream
|
||||
* ends rather than settling mid-stream. It is lossless — the completed parse
|
||||
* renders every character — and it needs a payload several times larger than any
|
||||
* tag emits (a `<workspace_resource>` runs ~100 characters, a `<question>` card
|
||||
* under ~1500). A mention in prose settles at its first character at any length,
|
||||
* because prose does not open with `{`. Widening or removing the window to close
|
||||
* that gap would trade a measured, reachable main-thread freeze for a
|
||||
* hypothetical one.
|
||||
*/
|
||||
const MAX_UNCLOSED_BODY_SCAN = 4096
|
||||
|
||||
/**
|
||||
* Length of the longest marker the scans can match.
|
||||
*
|
||||
* Derived from {@link SPECIAL_TAG_NAMES} rather than hand-counted, so adding a
|
||||
* longer tag name cannot silently shrink the rewind in {@link resumeForClass}.
|
||||
* Closing markers are the longer of the two forms, so they set the bound.
|
||||
*/
|
||||
const LONGEST_TAG_MARKER = Math.max(...SPECIAL_TAG_NAMES.map((name) => `</${name}>`.length))
|
||||
|
||||
/**
|
||||
* Strip the contents of JSON string literals from `body`, replacing them with
|
||||
* spaces so every other index is preserved.
|
||||
*
|
||||
* A JSON tag body can legitimately quote tag syntax — a `<question>` asking
|
||||
* which tag to use, or a `<workspace_resource>` whose title mentions one. Those
|
||||
* markers live inside a string and say nothing about whether the tag will
|
||||
* close, so the nesting rule must not see them. Tracks escapes so a `\"` inside
|
||||
* a string does not end it early. Handles an unterminated trailing string, which
|
||||
* is the normal state mid-stream.
|
||||
*
|
||||
* Index preservation is load-bearing, not decorative: {@link resumeForClass} takes an
|
||||
* offset found in the blanked copy and applies it to the RAW body. Iteration is by
|
||||
* code point, so a blanked astral character must emit `char.length` spaces —
|
||||
* emitting one would shrink the output and shift every later offset left.
|
||||
*/
|
||||
function blankJsonStringLiterals(body: string): string {
|
||||
// With no quote there is no string literal, so the loop below would copy the
|
||||
// body to itself character by character. Both callers reach here on bodies
|
||||
// that are usually plain prose, and this runs per opener per streamed chunk.
|
||||
if (!body.includes('"')) return body
|
||||
|
||||
let out = ''
|
||||
let inString = false
|
||||
let escaped = false
|
||||
|
||||
for (const char of body) {
|
||||
if (escaped) {
|
||||
escaped = false
|
||||
out += ' '.repeat(char.length)
|
||||
continue
|
||||
}
|
||||
if (char === '\\' && inString) {
|
||||
escaped = true
|
||||
out += ' '
|
||||
continue
|
||||
}
|
||||
if (char === '"') {
|
||||
inString = !inString
|
||||
out += '"'
|
||||
continue
|
||||
}
|
||||
out += inString ? ' '.repeat(char.length) : char
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* True while `scannable` could still grow into a single valid JSON value.
|
||||
*
|
||||
* Checking only the first character is not enough: a body like
|
||||
* `{"type":"file"}</workspac and then prose...` opens with `{` and looks fine,
|
||||
* but the value CLOSES at the `}` and everything after it is fatal. Tracking
|
||||
* depth catches that the moment the stray character arrives, instead of waiting
|
||||
* for a close tag that is never coming.
|
||||
*
|
||||
* Takes a body whose string literals are ALREADY blanked by
|
||||
* {@link blankJsonStringLiterals}, so braces and brackets inside JSON strings do
|
||||
* not affect the depth count. Both callers blank the body for their own marker
|
||||
* scan first, so taking the blanked form avoids a second pass over the same text.
|
||||
*/
|
||||
function isViableJsonPrefixOf(scannable: string): boolean {
|
||||
if (scannable.trim() === '') return true
|
||||
|
||||
const firstChar = scannable.trimStart().charAt(0)
|
||||
if (firstChar !== '{' && firstChar !== '[') return false
|
||||
|
||||
let depth = 0
|
||||
for (let i = 0; i < scannable.length; i++) {
|
||||
const char = scannable[i]
|
||||
if (char === '{' || char === '[') {
|
||||
depth++
|
||||
} else if (char === '}' || char === ']') {
|
||||
depth--
|
||||
if (depth < 0) return false
|
||||
// The top-level value just closed: only trailing whitespace may follow.
|
||||
if (depth === 0) return scannable.slice(i + 1).trim() === ''
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether `text` contains a marker for one of the tags this parser knows.
|
||||
*
|
||||
* Deliberately the tag NAMES rather than anything tag-shaped. A prose body may
|
||||
* legitimately contain `<div>` or `Promise<void>`; only a marker the parser
|
||||
* would itself act on proves the enclosing opener was text. Shared so the
|
||||
* streaming and matched-pair paths cannot answer the same question differently
|
||||
* — them disagreeing is what let a late close swallow content already on screen.
|
||||
*
|
||||
* The match is by substring, so a generic is safe only when its parameter is not
|
||||
* itself a tag name: `Promise<void>` does not match, `Promise<options>` does. The
|
||||
* narrowing is not worth its cost — it needs a `<thinking>` body, which the agent
|
||||
* no longer emits (reasoning arrives as structured thinking blocks), discussing a
|
||||
* type named exactly after a tag; and the boundary check that would fix it wants a
|
||||
* lookbehind, unavailable on the Safari versions this app still supports.
|
||||
*/
|
||||
function hasSpecialTagMarker(text: string): boolean {
|
||||
return SPECIAL_TAG_NAMES.some((name) => text.includes(`</${name}>`) || text.includes(`<${name}>`))
|
||||
}
|
||||
|
||||
/**
|
||||
* True when an opening tag with no close yet can NEVER resolve, so the text
|
||||
* after it should be shown immediately instead of held back until the stream
|
||||
* ends. Without it, a message that merely mentions a tag in prose goes blank
|
||||
* from that point on until streaming stops.
|
||||
*
|
||||
* One rule decides it, chosen by body kind:
|
||||
*
|
||||
* - **JSON-bodied tags** must stay a viable JSON prefix. Depth is tracked rather
|
||||
* than testing the first character alone, so a body whose top-level value has
|
||||
* already closed is caught the moment stray content follows it — a mention in
|
||||
* prose (no `{` at all), a misspelled close like `</workflow_resource>`, a
|
||||
* truncated `</workspac`, or no close whatsoever.
|
||||
* - **The prose-bodied tag** has no JSON to test, so the only evidence available
|
||||
* is that tags never nest: a marker for another special tag in the body means
|
||||
* this opener was literal text.
|
||||
*
|
||||
* Nested markers are NOT scanned for on a JSON body. A marker outside a string
|
||||
* literal is content the viability rule already rejects, and one inside is
|
||||
* legitimate quoted syntax that must not count as evidence — so the scan cost a
|
||||
* pass per tag name, open and close, to catch nothing.
|
||||
*
|
||||
* Both rules are conservative: they fire only on content that could not have
|
||||
* parsed. A false positive merely shows text early that a later chunk resolves
|
||||
* into a tag, and the end-of-stream parse still renders correctly.
|
||||
*/
|
||||
function unclosedTagCannotResolve(
|
||||
tagName: (typeof SPECIAL_TAG_NAMES)[number],
|
||||
body: string
|
||||
): boolean {
|
||||
const pending = dropArrivingClose(body, `</${tagName}>`)
|
||||
|
||||
if (!JSON_BODY_TAG_NAMES.has(tagName)) return hasSpecialTagMarker(pending)
|
||||
|
||||
// Cheap rejection before the expensive one. isViableJsonPrefixOf decides on
|
||||
// the first non-whitespace character when it is not `{` or `[` — which is the
|
||||
// common case, a tag name mentioned in prose — so testing it here avoids
|
||||
// blanking up to a full window of text only to throw the copy away.
|
||||
const firstChar = pending.trimStart().charAt(0)
|
||||
if (firstChar !== '' && firstChar !== '{' && firstChar !== '[') return true
|
||||
|
||||
// Blank string literals first so braces and brackets inside JSON strings do
|
||||
// not throw off the depth count.
|
||||
return !isViableJsonPrefixOf(blankJsonStringLiterals(pending))
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a trailing fragment that could still grow into `closeTag`.
|
||||
*
|
||||
* Mid-stream the closing marker arrives a character at a time, so a body sits at
|
||||
* `]</opt` for several frames before `</options>` completes. That fragment is an
|
||||
* arriving close, not stray content — counting it as fatal is what made a
|
||||
* perfectly valid tag show its raw payload as text until the final `>` landed.
|
||||
*
|
||||
* Only a fragment at the very END is dropped, so evidence that the close is
|
||||
* genuinely wrong still lands immediately: a misspelled `</workflow_resource>`
|
||||
* is not a prefix of `</workspace_resource>`, and a truncated `</workspac`
|
||||
* followed by prose stops being one the moment the prose arrives.
|
||||
*/
|
||||
function dropArrivingClose(body: string, closeTag: string): string {
|
||||
for (let n = Math.min(closeTag.length - 1, body.length); n > 0; n--) {
|
||||
if (body.endsWith(closeTag.slice(0, n))) return body.slice(0, -n)
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
/**
|
||||
* How one opening tag resolved. Naming the four outcomes is the point: the
|
||||
* parser previously decided each case inline, which is how "drop it" quietly
|
||||
* became the fallback for situations that were never malformed payloads.
|
||||
*/
|
||||
type TagResolution =
|
||||
/** Body parsed; emit the typed segment and resume after the closing tag. */
|
||||
| { outcome: 'segment'; segment: ContentSegment; resumeAt: number }
|
||||
/** Provably not a tag; render the span verbatim and resume after it. */
|
||||
| { outcome: 'literal'; resumeAt: number }
|
||||
/** A well-formed payload that failed its shape guard — dropped deliberately. */
|
||||
| { outcome: 'discard'; resumeAt: number }
|
||||
/** Still streaming and a close remains plausible; suppress the remainder. */
|
||||
| { outcome: 'pending' }
|
||||
|
||||
/**
|
||||
* Why a failed body was never an attempted payload — so the markers were literal
|
||||
* text and the span must be shown rather than swallowed. `null` means the body
|
||||
* really was a payload that failed its shape guard.
|
||||
*
|
||||
* The two reasons resume differently, which is why they are distinguished
|
||||
* rather than collapsed into a boolean (see {@link resumeForClass}).
|
||||
*/
|
||||
type LiteralTextVerdict =
|
||||
/**
|
||||
* The body carries a tag marker at `markerOffset` (an index into the body), so
|
||||
* the close we matched belongs to a different opener.
|
||||
*/
|
||||
| { reason: 'foreign-markers'; markerOffset: number }
|
||||
/** The tag wrapped prose that was never JSON to begin with. */
|
||||
| { reason: 'never-a-payload' }
|
||||
|
||||
function literalTextReason(
|
||||
tagName: (typeof SPECIAL_TAG_NAMES)[number],
|
||||
body: string
|
||||
): LiteralTextVerdict | null {
|
||||
const isJsonBodied = JSON_BODY_TAG_NAMES.has(tagName)
|
||||
// Markers inside a JSON string are content, not evidence — a `<question>` may
|
||||
// legitimately quote tag syntax in its prompt. Scanning the raw body here
|
||||
// would classify a broken payload as literal text and render it as raw JSON,
|
||||
// which is exactly what `discard` exists to prevent. Mirrors the same blanking
|
||||
// in unclosedTagCannotResolve, which judges the same body mid-stream.
|
||||
const scannable = isJsonBodied ? blankJsonStringLiterals(body) : body
|
||||
const marker = TAG_SHAPED_MARKER.exec(scannable)
|
||||
if (marker) return { reason: 'foreign-markers', markerOffset: marker.index }
|
||||
if (isJsonBodied && !isViableJsonPrefixOf(scannable)) return { reason: 'never-a-payload' }
|
||||
return null
|
||||
}
|
||||
|
||||
/** One memoized `indexOf` result, with the `from` it was computed at. */
|
||||
interface IndexOfCacheEntry {
|
||||
/** Result of `content.indexOf(needle, from)`, or -1 when absent from that point on. */
|
||||
idx: number
|
||||
/** The offset the search started at. The entry says nothing about content before it. */
|
||||
from: number
|
||||
}
|
||||
|
||||
export type IndexOfCache = Map<string, IndexOfCacheEntry>
|
||||
|
||||
/**
|
||||
* `content.indexOf(needle, from)` memoized per needle.
|
||||
*
|
||||
* The opener scan and the close lookup search the same handful of markers over
|
||||
* and over as the cursor advances. A needle absent from the message resolves to
|
||||
* -1 once and is never searched again; a present one is re-searched only when
|
||||
* the cursor passes its last hit. Unmemoized, each lookup rescans to the end of
|
||||
* the buffer for every opener, which is quadratic on a message that mentions a
|
||||
* tag name many times — and this parse re-runs for every streamed chunk.
|
||||
*
|
||||
* A cached result is only valid from the offset it was searched at, so the entry
|
||||
* carries that offset and is reused only when the new `from` is at or beyond it:
|
||||
*
|
||||
* - `idx === -1` means no hit at or after `entry.from`, so there is none at or
|
||||
* after any later `from` either.
|
||||
* - `idx >= from` means the first hit at or after `entry.from` is still ahead of
|
||||
* `from`, so nothing lies between them and it is still the first hit.
|
||||
*
|
||||
* Storing `from` is what makes this correct for ANY call order rather than only
|
||||
* for a monotonically advancing cursor. The cursor is monotonic today — every
|
||||
* non-pending outcome resumes strictly past its opener — but that is a property
|
||||
* of {@link resolveTagAt}'s resume points, and one of them deliberately resumes
|
||||
* back inside a span it already examined. A future adjustment that let the cursor
|
||||
* regress would, without this check, return a stale index and silently mis-parse
|
||||
* rather than fail loudly. With it, the worst case is a redundant scan.
|
||||
*/
|
||||
export function memoizedIndexOf(
|
||||
cache: IndexOfCache,
|
||||
content: string,
|
||||
needle: string,
|
||||
from: number
|
||||
): number {
|
||||
const entry = cache.get(needle)
|
||||
if (entry && from >= entry.from && (entry.idx === -1 || entry.idx >= from)) return entry.idx
|
||||
const idx = content.indexOf(needle, from)
|
||||
cache.set(needle, { idx, from })
|
||||
return idx
|
||||
}
|
||||
|
||||
/**
|
||||
* How much of a body may be inspected, and whether that is all of it.
|
||||
*
|
||||
* The read budget, isolated from what the body turns out to BE. Both the
|
||||
* unclosed and matched-pair paths spend it through this one function, so they
|
||||
* cannot drift out of agreement about how much of a body may be read.
|
||||
*/
|
||||
interface InspectedBody {
|
||||
/** The prefix actually examined. */
|
||||
text: string
|
||||
/** True when `text` is only a prefix, so no verdict drawn from it covers the rest. */
|
||||
truncated: boolean
|
||||
}
|
||||
|
||||
function inspectWithin(source: string, start = 0): InspectedBody {
|
||||
const end = start + MAX_UNCLOSED_BODY_SCAN
|
||||
return end < source.length
|
||||
? { text: source.slice(start, end), truncated: true }
|
||||
: { text: start === 0 ? source : source.slice(start), truncated: false }
|
||||
}
|
||||
|
||||
/**
|
||||
* What a matched body turned out to BE — independent of what the parser does
|
||||
* about it, and of where it resumes.
|
||||
*
|
||||
* A closed set, and that is the whole point: {@link resolveMatchedPair} and
|
||||
* {@link resumeForClass} each switch over it exhaustively, so adding a case
|
||||
* fails to compile until BOTH questions are answered for it. Every regression
|
||||
* review found on this parser was one of those two answers changing without the
|
||||
* other, which is a mistake this shape makes unrepresentable.
|
||||
*/
|
||||
type BodyClass =
|
||||
/** Parsed, and matched its shape guard. */
|
||||
| { kind: 'payload'; segment: ContentSegment }
|
||||
/** A tag marker at `offsetInBody` proves the close we matched belongs elsewhere. */
|
||||
| { kind: 'nested-marker'; offsetInBody: number }
|
||||
/**
|
||||
* The same proof, in a PROSE body. Separate because it resumes differently: a
|
||||
* prose body is never blanked, so nothing is hidden from the scan and rescanning
|
||||
* from the opener is safe, and these bodies are small enough that the extra pass
|
||||
* is free. Resuming at the marker instead would also be correct and would emit
|
||||
* one text segment rather than two — display-identical, since the renderer
|
||||
* concatenates them — but it is a behaviour change and does not belong in a
|
||||
* refactor.
|
||||
*/
|
||||
| { kind: 'prose-nested-marker' }
|
||||
/** Only a prefix was read, and it settled nothing. Says nothing about the rest. */
|
||||
| { kind: 'unexamined' }
|
||||
/** Not a payload at all — never JSON, or JSON that will not parse. */
|
||||
| { kind: 'never-json' }
|
||||
/** Parsed as JSON, then failed its shape guard. The only droppable class. */
|
||||
| { kind: 'broken-payload' }
|
||||
|
||||
/**
|
||||
* Classify a complete body. Pure: no positions, no outcome, no resume.
|
||||
*
|
||||
* Order is behavioural, not stylistic. The prose-nesting rule precedes the parse
|
||||
* because a prose body has no shape to fail — any non-empty text qualifies — so a
|
||||
* late close would otherwise swallow whatever the streaming path already showed.
|
||||
* The budget precedes the remaining rules so an unread remainder is never
|
||||
* mistaken for evidence.
|
||||
*/
|
||||
function classifyBody(tagName: (typeof SPECIAL_TAG_NAMES)[number], body: string): BodyClass {
|
||||
const isJsonBodied = JSON_BODY_TAG_NAMES.has(tagName)
|
||||
|
||||
if (!isJsonBodied) {
|
||||
// The same predicate the streaming path uses, so the two cannot disagree
|
||||
// about whether this body was ever a tag. Tag NAMES, not anything
|
||||
// tag-shaped: reasoning that mentions `<div>` or `Promise<void>` is still
|
||||
// reasoning, and releasing it as prose would put the model's thinking on
|
||||
// screen for an incidental angle bracket.
|
||||
if (hasSpecialTagMarker(body)) return { kind: 'prose-nested-marker' }
|
||||
}
|
||||
|
||||
const parsed = parseSpecialTagData(tagName, body)
|
||||
if (parsed) return { kind: 'payload', segment: parsed }
|
||||
|
||||
const inspected = inspectWithin(body)
|
||||
const verdict = literalTextReason(tagName, inspected.text)
|
||||
|
||||
if (verdict?.reason === 'foreign-markers') {
|
||||
return { kind: 'nested-marker', offsetInBody: verdict.markerOffset }
|
||||
}
|
||||
if (inspected.truncated) return { kind: 'unexamined' }
|
||||
|
||||
// Dropping text is only defensible for a payload the agent actually FORMED.
|
||||
// `{the Q4 report}` is prose in braces and `{type: "file"}` is an ordinary
|
||||
// model slip; bracket depth cannot tell either from a real payload, only a
|
||||
// parse can. Both routes to that answer are funnelled through one place so the
|
||||
// rescan below cannot be added to one and forgotten on the other.
|
||||
const neverJson =
|
||||
verdict?.reason === 'never-a-payload' || (isJsonBodied && !isParseableJson(body))
|
||||
|
||||
if (neverJson) {
|
||||
// literalTextReason blanked this body's quoted regions on the assumption it
|
||||
// was JSON. It never was, so that assumption is void — and a body with an
|
||||
// odd number of `"` blanks the WRONG regions, which can hide a real marker
|
||||
// and turn what should be `nested-marker` into `never-json`. The difference
|
||||
// is not academic: `never-json` resumes past the close, flattening a genuine
|
||||
// tag inside the span, so a card already on screen un-renders when the close
|
||||
// finally arrives. With the JSON premise gone, the raw text is the honest
|
||||
// evidence, and a marker in it means the close we matched belongs elsewhere.
|
||||
const rawMarker = TAG_SHAPED_MARKER.exec(inspected.text)
|
||||
if (rawMarker) return { kind: 'nested-marker', offsetInBody: rawMarker.index }
|
||||
return { kind: 'never-json' }
|
||||
}
|
||||
|
||||
return { kind: 'broken-payload' }
|
||||
}
|
||||
|
||||
/**
|
||||
* Where scanning continues, given what the body was. The third concern, kept
|
||||
* apart from the other two so a change to one cannot silently alter another.
|
||||
*
|
||||
* Every branch is strictly greater than the opener, which is what guarantees the
|
||||
* cursor advances and {@link memoizedIndexOf}'s cache stays coherent.
|
||||
*/
|
||||
function resumeForClass(cls: BodyClass, bodyStart: number, pastClose: number): number {
|
||||
switch (cls.kind) {
|
||||
case 'payload':
|
||||
case 'broken-payload':
|
||||
case 'never-json':
|
||||
// The whole span was read and accounted for; continue after it.
|
||||
return pastClose
|
||||
case 'nested-marker':
|
||||
// Resume AT the marker, not past the borrowed close and not at the opener.
|
||||
// Past the close would skip a genuine tag after it; the opener would rescan
|
||||
// a region the blanked scan could not see into, re-parsing tag syntax
|
||||
// quoted inside a JSON string and dropping it.
|
||||
return bodyStart + cls.offsetInBody
|
||||
case 'prose-nested-marker':
|
||||
// Rescan the whole body: nothing was blanked, so no marker is hidden.
|
||||
return bodyStart
|
||||
case 'unexamined':
|
||||
// Resume just short of the first character NOT read: the last marker's
|
||||
// worth of the window is held back rather than emitted as text, so it is
|
||||
// re-scanned on the next pass instead of being flattened. Nothing is lost
|
||||
// — the caller emits up to wherever this resumes. It still advances nearly
|
||||
// a full window per step, so a long body costs a bounded number of
|
||||
// re-entries.
|
||||
//
|
||||
// The rewind is load-bearing: the window edge is an arbitrary cut, so an
|
||||
// opener can straddle it. Resuming exactly at the edge leaves that opener's
|
||||
// `<` behind the cursor, and the opener scan only looks FORWARD — so the tag
|
||||
// is never found and renders as raw payload text, on a completed message.
|
||||
// Backing off by the longest marker guarantees any straddling opener is
|
||||
// re-scanned from its `<`.
|
||||
return bodyStart + MAX_UNCLOSED_BODY_SCAN - (LONGEST_TAG_MARKER - 1)
|
||||
}
|
||||
}
|
||||
|
||||
function resolveMatchedPair(
|
||||
tagName: (typeof SPECIAL_TAG_NAMES)[number],
|
||||
body: string,
|
||||
bodyStart: number,
|
||||
pastClose: number
|
||||
): TagResolution {
|
||||
const cls = classifyBody(tagName, body)
|
||||
const resumeAt = resumeForClass(cls, bodyStart, pastClose)
|
||||
|
||||
switch (cls.kind) {
|
||||
case 'payload':
|
||||
return { outcome: 'segment', segment: cls.segment, resumeAt }
|
||||
case 'broken-payload':
|
||||
// Well-formed but the wrong shape — a broken emission. Showing the reader
|
||||
// raw JSON is worse than showing nothing.
|
||||
return { outcome: 'discard', resumeAt }
|
||||
case 'nested-marker':
|
||||
case 'prose-nested-marker':
|
||||
case 'unexamined':
|
||||
case 'never-json':
|
||||
return { outcome: 'literal', resumeAt }
|
||||
}
|
||||
}
|
||||
|
||||
function resolveTagAt(
|
||||
content: string,
|
||||
openIndex: number,
|
||||
tagName: (typeof SPECIAL_TAG_NAMES)[number],
|
||||
isStreaming: boolean,
|
||||
closeCache: IndexOfCache
|
||||
): TagResolution {
|
||||
const openTag = `<${tagName}>`
|
||||
const closeTag = `</${tagName}>`
|
||||
const bodyStart = openIndex + openTag.length
|
||||
const closeIdx = memoizedIndexOf(closeCache, content, closeTag, bodyStart)
|
||||
|
||||
if (closeIdx === -1) {
|
||||
const inspected = inspectWithin(content, bodyStart)
|
||||
if (isStreaming && !unclosedTagCannotResolve(tagName, inspected.text)) {
|
||||
return { outcome: 'pending' }
|
||||
}
|
||||
// Nothing can close it, so only the opener itself is literal. Resuming just
|
||||
// past it (rather than abandoning the message) keeps a genuinely valid tag
|
||||
// later in the same reply parseable.
|
||||
return { outcome: 'literal', resumeAt: bodyStart }
|
||||
}
|
||||
|
||||
return resolveMatchedPair(
|
||||
tagName,
|
||||
content.slice(bodyStart, closeIdx),
|
||||
bodyStart,
|
||||
closeIdx + closeTag.length
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits streamed text into renderable segments, extracting complete special
|
||||
* tags and deciding what to do with the ones that never resolve. Incomplete
|
||||
* tags are suppressed and flagged via `hasPendingTag` so the caller can show a
|
||||
* loading indicator, and a trailing partial opening marker (`<opt`, `<usage_`)
|
||||
* is stripped during streaming so it never flashes as raw markup.
|
||||
*/
|
||||
export function parseSpecialTags(content: string, isStreaming: boolean): ParsedSpecialContent {
|
||||
const segments: ContentSegment[] = []
|
||||
let hasPendingTag = false
|
||||
let cursor = 0
|
||||
|
||||
// Whitespace-only spans are kept, not trimmed away: the literal path emits a
|
||||
// rejected span in several pieces, and a `\n\n` between two of them is a
|
||||
// markdown paragraph break. Dropping it silently merges two paragraphs, because
|
||||
// the renderer concatenates adjacent text segments into one markdown string.
|
||||
const pushText = (text: string) => {
|
||||
if (text) segments.push({ type: 'text', content: text })
|
||||
}
|
||||
|
||||
const openerCache: IndexOfCache = new Map()
|
||||
const closeCache: IndexOfCache = new Map()
|
||||
let discardedTag = false
|
||||
|
||||
while (cursor < content.length) {
|
||||
let nearestStart = -1
|
||||
let nearestTagName: (typeof SPECIAL_TAG_NAMES)[number] | '' = ''
|
||||
|
||||
for (const name of SPECIAL_TAG_NAMES) {
|
||||
const idx = content.indexOf(`<${name}>`, cursor)
|
||||
const idx = memoizedIndexOf(openerCache, content, `<${name}>`, cursor)
|
||||
if (idx !== -1 && (nearestStart === -1 || idx < nearestStart)) {
|
||||
nearestStart = idx
|
||||
nearestTagName = name
|
||||
}
|
||||
}
|
||||
|
||||
if (nearestStart === -1) {
|
||||
// Only the name is tested: the two are assigned together above, so an empty
|
||||
// name and a -1 start are the same state — and the name is the one that
|
||||
// needs narrowing before resolveTagAt below.
|
||||
if (nearestTagName === '') {
|
||||
let remaining = content.slice(cursor)
|
||||
|
||||
if (isStreaming) {
|
||||
// Hide a half-arrived opening marker so it does not flash as text.
|
||||
const partial = remaining.match(/<[a-z_-]*$/i)
|
||||
if (partial) {
|
||||
const fragment = partial[0].slice(1)
|
||||
@@ -550,44 +1121,37 @@ export function parseSpecialTags(content: string, isStreaming: boolean): ParsedS
|
||||
}
|
||||
}
|
||||
|
||||
if (remaining.trim()) {
|
||||
segments.push({ type: 'text', content: remaining })
|
||||
}
|
||||
pushText(remaining)
|
||||
break
|
||||
}
|
||||
|
||||
if (nearestStart > cursor) {
|
||||
const text = content.slice(cursor, nearestStart)
|
||||
if (text.trim()) {
|
||||
segments.push({ type: 'text', content: text })
|
||||
}
|
||||
}
|
||||
pushText(content.slice(cursor, nearestStart))
|
||||
|
||||
const openTag = `<${nearestTagName}>`
|
||||
const closeTag = `</${nearestTagName}>`
|
||||
const bodyStart = nearestStart + openTag.length
|
||||
const closeIdx = content.indexOf(closeTag, bodyStart)
|
||||
const resolution = resolveTagAt(content, nearestStart, nearestTagName, isStreaming, closeCache)
|
||||
|
||||
if (closeIdx === -1) {
|
||||
if (resolution.outcome === 'pending') {
|
||||
hasPendingTag = true
|
||||
cursor = content.length
|
||||
break
|
||||
}
|
||||
|
||||
const body = content.slice(bodyStart, closeIdx)
|
||||
if (!nearestTagName) {
|
||||
cursor = closeIdx + closeTag.length
|
||||
continue
|
||||
}
|
||||
const parsedTag = parseSpecialTagData(nearestTagName, body)
|
||||
if (parsedTag) {
|
||||
segments.push(parsedTag)
|
||||
if (resolution.outcome === 'segment') {
|
||||
segments.push(resolution.segment)
|
||||
} else if (resolution.outcome === 'literal') {
|
||||
pushText(content.slice(nearestStart, resolution.resumeAt))
|
||||
} else {
|
||||
// `discard` deliberately emits nothing. Remembering that it happened is
|
||||
// what keeps the fallback below from undoing it.
|
||||
discardedTag = true
|
||||
}
|
||||
|
||||
cursor = closeIdx + closeTag.length
|
||||
cursor = resolution.resumeAt
|
||||
}
|
||||
|
||||
if (segments.length === 0 && !hasPendingTag) {
|
||||
// A message with no segments is normally a message with nothing in it, and
|
||||
// emitting the raw content is the right floor. But a discard produces no
|
||||
// segment BY DESIGN, so without this guard a message that is only a broken
|
||||
// payload falls through and renders the exact raw JSON the discard removed.
|
||||
if (segments.length === 0 && !hasPendingTag && !discardedTag) {
|
||||
segments.push({ type: 'text', content })
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user