mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-30 19:32:46 +08:00
improvement(mothership): v0.9 (#6815)
* checkpoint
* Checkpoint
* dot fixes
* Make async tool resume delivery recoverable
* Support split table tools and option recovery
* Harden VFS mutation handling
* feat(platform): platform subagent support — docs corpus VFS, search_docs, account context
Squash of the feat/platform-agent branch (sim side): mounts the Sim docs
corpus in the copilot VFS, wires search_docs and retires the legacy docs
search tools, and syncs the generated tool catalog and trace contracts for
the platform subagent.
* Align Copilot tools and resource handling
* Expand workflow log query support
* checkpoint
* Port desktop-improvements-0 desktop and browser-agent work
* fix(desktop): keep browser-agent input alive through live-SPA re-renders
* Harden workflow sanitization and Slack setup
* feat(desktop): coordinate clicks, caret insertion, and drag for the browser agent
* Revert subagent group eager auto-collapse
* fix(chat): keep sends FIFO across the streaming-to-idle drain gap
* Add the steering backend surface for mid-turn sends
* Sync generated contracts for async subagent orchestration
Pulls the mothership tool catalog (wait_agents / tail_agent / steer_agent /
interrupt_agent), trace spans (chat.async_subagent.*, chat.orchestrate.*), and
trace attributes (copilot.async_subagent.*) into the generated TS contracts.
* Add display titles for the async subagent orchestration tools
wait_agents / tail_agent / steer_agent / interrupt_agent get natural-language
running titles (naming the agent id being waited on, tailed, steered, or
stopped) and a Steering→Steered completed-verb rewrite.
* Show orchestrator-chosen subagent names on agent groups
A subagent_start whose payload data carries a name (the orchestrator's new
name trigger parameter) now labels the agent group with that mission name —
the agent-type icon stays. The name flows through the live stream path, the
turn model (AgentNode.displayName) and its serialize/rebuild round-trip, and
persisted transcripts (PersistedContentBlock.name), so reloads keep the label.
* Improve Copilot error handling and logging
* Backfill the subagent display name from the second start event
The dispatch-time subagent_start fires before the trigger args (and therefore
the name parameter) have streamed; the phase-3 start re-announces the lane with
the name. The block builder was dropping that duplicate wholesale, losing the
name on streaming providers — now it backfills subagentName onto the existing
block instead. (The home turn-model path already reconciled this case.)
* Support Slack bot connection flow
* Harden Copilot error and VFS handling
* Harden VFS resource operations
* Show 'Waiting for the first of N agents' for mode-any waits
The wait_agents title ignored the mode argument, so an any-mode wait over
three agents read 'Waiting for 3 agents' while the model narrated waiting for
the first — contradicting the transcript.
* Collapsed-by-default agent cards with live intent status lines
Subagents now narrate their work through <intent>3-5 words</intent> tags (a
fleet-wide prompt protocol on the mothership side). The turn model streams
each subagent's text through a split-safe tag parser: complete tags update the
agent's currentIntent and disappear from the prose, tags split across deltas
are carried until their close arrives, and a tag that never closes flushes
back as plain text.
The agent card renders as one line — display name (or agent label) plus the
latest intent, replaced inline as the agent shifts gears — and never
auto-expands; expanding to the full tool log is a deliberate click. Only an
outstanding permission prompt or a browser hand-back forces a group open.
Intents persist on the subagent block (and through the legacy persisted-
message paths) so reloads keep the last status, and a renamed reinvocation
now takes the latest name instead of pinning the first.
* Add the internal in-band tool execution route for live mothership turns
POST /api/copilot/tools/execute (INTERNAL_API_SECRET, Go→Sim) runs one
sim-server tool through the same server tool router the resume driver uses
and returns the result synchronously — no checkpoint. This is what lets
background (async) subagents write files/tables/knowledge, and lets the main
lane keep streaming (instead of checkpoint-pausing and killing every
background run) while async agents are live.
* Persist resource side effects for in-band tool execution
Files/tables created through the internal execute route now register on the
chat's resources exactly like the resume driver's executions — the route runs
the same handleResourceSideEffects pass (persistence only; an out-of-band
route has no live event sink, so mid-turn chip pushes are a follow-up).
* Extract intents from group text on every path, sync and async
The turn-model intent filter only fires for span-scoped subagent lanes, but
this surface also delivers subagent text through the legacy block path — so
<intent> tags flowed through unparsed and rendered as prose rows. Groups now
extract intents from their accumulated text at append time: the last complete
tag becomes the card's status line and every complete tag is stripped from
the rendered prose. Covers span-scoped, legacy, and persisted-reload paths
for both synchronous and background delegations.
* Fall back to the live tool title for the agent card status line
Persisted data proved tool-first subagents (grok search agents) emit zero
prose, so intent tags never stream no matter what the prompt says. The
collapsed card now always narrates: the agent's own <intent> tag when present,
else the latest tool's display title while the lane is live.
* Catch subagent <intent> tags in the server relay
The relay's subagent text handler now runs the split-safe intent extraction
as chunks stream: the latest complete tag is stamped onto the lane's persisted
subagent block (subagentIntent) and stripped from the stored prose, so live,
persisted, and replayed views all agree. Per-lane carry handles tags split
across chunks; a never-closing tag flushes back as plain text.
* Drop the tool-title fallback: the status line is the agent's intent
With the intent protocol now injected into every spawn's task message, agents
open with an <intent> tag; the card shows that narration or nothing.
* Replace intents with live tool-title status lines on agent cards
Intent parsing is fully removed (turn model, relay handler, persistence
fields, group extraction). The collapsed card's status is the latest tool
call in its RUNNING phrasing — never the completed rewrite, which stays in
the expanded log. Parallel tools show the most recently started still-running
title with a +N for concurrent siblings; between rounds the last title stays
frozen; a closed lane shows the bare name. Nested agent cards compute their
own status recursively from their own items.
* Keep the main Sim lane live-expanded; collapse only real subagent cards
The mothership group is the turn's own narration, not a delegation card —
collapsing it hid main-lane text and tools until manual expand, which read as
mis-ordered streaming while async subagents interleaved. It keeps the
original live-expand behavior and no status suffix.
* Persist subagent lane lifecycle blocks from the span handler
Lane-scoped span events route to the span handler, which only recorded trace
side effects — no subagent start block was ever persisted (verified: a
seven-agent run stored 104 blocks with zero starts). Grouping then fell back
to keying lane content by agent NAME, so a respawned agent of the same type
merged invisibly into the first one's card until it resolved. The handler now
persists the start block (spanId-keyed and deduped, carrying the display
name) and stamps endedAt on close, giving every invocation its own card.
* Name agents in orchestration titles; '+ n more' overflow format
wait/tail/steer/interrupt titles humanize the slugified agent ids back to
their display names ('Waiting for the first of Digest Workflow Build + 4
more'), and the agent card's parallel-tool suffix uses the same '+ n more'
format.
* Harden in-band tool execution and resources
* Route in-band execution through the comprehensive tool dispatcher
The internal execute route used the bare server-tool router, which rejects
VFS tools with 'Unknown server tool: read/glob/grep' — so nearly every
background agent's first discovery call failed (102 in-band calls in one run,
dozens rejected). It now uses the relay's executeTool dispatcher: registered
handlers (VFS, function execute) with permission checks and param
normalization, falling back to the app tool router — the same surface
foreground execution gets.
* Harden chat stream transition handling
* Harden VFS provenance and resource writes
* Standardize tool environment references
* Harden browser panel and chat cleanup
* Descriptive, user-language tool titles across the board
House rules applied everywhere: use every argument the call carries, never
name internal machinery, and never lead with Getting (the Got rewrite is
deleted so it cannot return).
- Deployments name the workflow: Deploying {workflow} as API/chat app/MCP tool
- Workflow reads name the part: Reading {workflow} meta/state/deployment/notes;
generic reads always name the file (Reading {leaf}), never bare Reading file
- Block runs name block and workflow: Running {block} in {workflow}, Running
from {block} in {workflow}, Running {workflow} until {block}, and
Enabling/Disabling {block} in {workflow}
- The six split-table tools get per-operation verbs (Adding column {name},
Updating rows, Wiring automation, Creating view {name}) instead of a wall
of Querying table
- The manage quartet drops X-action system-speak for gerunds
- get_* internal names become user language (Checking run settings, Tracing
block inputs, Reading the deployed version); web_fetch says Fetching
- Scheduled-task titles removed entirely (feature deleted from the Go catalog)
- New verb rewrites: Fetched, Traced, Wired, Configured, Looked, Rotated
* Deploying {workflow} as chat, not as chat app
* Loader gerunds; mv names both ends; mkdir names the folder
search_integration_tools -> Finding the right integration;
load_integration_tool -> Loading {integration} tools; load_skill ->
Loading skill {name}; run_enrichment -> Looking up {subject}. mv prefers the
model's phrasing, else reads 'Moving {files} to {destination}'; mkdir reads
'Creating folder {name}' from the path.
* Overflow counts read '+ n', dropping 'more'
* Unify workspace find and search
* Scale desktop title bar with page zoom
* Serialize account and organization truth into the copilot VFS
Workspace standing, membership, billing, org role, access-control
restrictions, published-block provenance, and fork topology were reachable
only through three parameterless tools (or not at all). They are ambient
read-only facts, so they belong in the VFS where they are greppable, cost no
tool round-trip, and every agent that can read gets them — the same move that
retired get_blocks_and_tools and list_user_workflows.
Adds account/{workspace,workspaces,members,billing}.json (always mounted) and
organization/{organization,access-control,custom-blocks,forks}.json (only when
the workspace is org-hosted). Every file projects an existing use case or util
after getOrMaterializeVFS's access assert — no new queries, no new
authorization. One relation per file, cross-referenced by id-and-name stub, so
overlapping facts cannot disagree. Volatile content (billing, access control,
forks) is lazy, so numbers are read-time fresh and unasked-for reads cost
nothing.
Projection follows the viewer: member emails are admin-only, fork detail
requires workspace admin on a forking-enabled org, and the whole organization/
namespace is absent for a personal workspace — which is itself the answer.
Retires get_account_billing, get_enterprise_context, and list_user_workspaces
along with their handlers; display titles stay for transcript replay.
* Fix insert_text refusing an editable field focused inside a frame
describeFocusedEditable descended shadow roots but not frames, while
activeElementReadback descends both. Focus inside a same-origin frame therefore
surfaced to the first as the FRAME element — not an input, not contentEditable,
not a canvas, no textbox role — so it fell through to 'not-editable' and
insert_text refused a field that press_key had just typed a character into.
Two functions answering 'what is focused' with different answers is the bug;
the descent loops now match exactly.
The refusal also names what actually held focus (tag, role, contenteditable).
A bare 'not-editable' gave the agent nothing to act on, so it guessed at the
cause — a real run spent twenty rounds on the wrong theory and had to be
stopped by the user.
* Keep retired browser takeover renderable in history
The tool is gone from the catalog, so its generated constant went with it and
every path that referenced it stopped compiling. Deleting those paths instead
would have silently downgraded every past transcript containing a takeover card
to a generic tool row, and dropped the no-timeout budget that an in-flight
takeover still needs while a rolling deploy finishes.
retired-tools.ts gives the literal a documented home that says what it is and
why it survives its tool.
* Follow the agent into a tab it opened to work in
browser_open_tab created the page with activate: false, so the agent worked in
a tab the user could not see while the panel sat on a page where nothing was
happening. The panel now follows a tab the agent deliberately opened.
Scoped to that tool only. A page spawning its own tab (popup, target=_blank) is
the site grabbing the view rather than the agent choosing a workspace, and
stays in the background as before — two existing tests pin that and caught the
first version of this change, which moved both.
A tab the user claimed still wins over both: the work starts in the background
instead of pulling the page out from under them mid-read.
* Make the browser tools agree with each other
An audit of the module found the frame-descent bug was one instance of a
pattern: six independent definitions of 'is this editable' and seven of 'what
is focused', disagreeing with each other. A tool refusing what its sibling
accepts on identical page state is invisible at runtime — the agent follows a
snapshot that says one thing into a tool that says another.
- browser_type now accepts role="textbox" like browser_insert_text does. The
snapshot advertises those elements as [textbox] with a ref, so refusing them
meant rejecting exactly what the outline told the model to type into. Both
the native and synthetic paths, and their descendant scans.
- pressKeyOnPage descends shadow roots and frames like every other focus
reader. It was dispatching synthetic keys at the shadow host or <iframe>
element, where they bubble but never reach the editor, while reporting
success — and contradicting the activeElement reported beside it.
- not-editable and ambiguous-editable name what was found: the element's tag
and role, and the candidate fields. Both had the data and discarded it, which
is what turns one blocked step into twenty rounds of guessing.
- obstructedAfterNavigation requires a dialog that ARRIVED with the
navigation. It compared against nothing, so every SPA route change under a
persistent role=dialog reported a successful click as obstructed. The test
that covered this asserted the false positive; it now pins both directions.
- browser_insert_text observes the top document when typing inside a frame,
like every other input tool. A submit that navigates the top page was
invisible to its frame-scoped observation.
* Let hover actually see what it mounted
Four independent defects made browser_hover blind to the most common thing a
hover produces — a row's action bar — so it reported no effect on a hover that
worked, and the agent fell back to clicking pixels off screenshots.
- The popup scan matched only role=tooltip/menu/listbox. Slack's message
shortcuts bar is a labelled toolbar/group, so it registered as nothing at
all. Added toolbar, menubar, labelled group, and [popover].
- The baseline was captured BEFORE prepareElementSurface scrolled the target
into view, so scrollChanged was always set by the tool's own probe. That
pinned every unproductive hover to 'background DOM churn' instead of the
honest 'nothing happened', and hid scrolling the hover really caused.
Re-baselined once the scroll settles and before the pointer moves.
- The MutationObserver attached only on the first observation, while the roots
list is rebuilt every call and grows as shadow roots mount. Components that
appeared later were never observed, so their DOM changes raised no revision.
Roots are now observed as they show up.
- observationTruncated was computed and never read, so a scan capped at 12k
nodes reported 'nothing appeared' with the same confidence as a complete
one — and portalled overlays live at the end of <body>, exactly what the cap
drops. Hover now says the page was too large to scan and to confirm visually.
* Stop the browser agent acting on the wrong element, and say why it refused
Four findings from the module audit, the first of which could silently do the
wrong thing rather than merely fail.
- A ref whose node is gone is re-adopted by structural resemblance, matching on
ORIGIN only so a pushState between snapshot and act does not kill every ref.
That leniency also let a ref to a row control in one view rebind to the
identical control in a view the app had since navigated to — acting on the
wrong message, signalled by nothing louder than recovered: true. Adoption now
requires the same path; a view swap reports the ref stale, and the caller
re-snapshots. Revalidating a still-connected node stays lenient, because that
is literally the node the model chose.
- A hit INSIDE the requested element is its own nested control, not an overlay.
Both produced 'covered by X — close or move the overlay', advice that cannot
be followed because there is nothing to close. Nested hits now say so and
point at retargeting.
- browser_click_at, browser_insert_text, and browser_drag listed targetChanged
in their effect formulas, but none passes an elementId, so no targetState is
ever captured and the term was always false — coverage that read as real.
Removed, with a test pinning the dependency.
- The seven effect formulas are deliberately NOT collapsed into one predicate:
drag must trust domChanged where others must not, hover must ignore field and
focus changes, click counts focus only for editables. Forcing one would make
each tool wrong differently. The differences are now documented in one place
next to the shared computation, so divergence is a declared policy rather
than an accident.
* Let edit_workflow configure block retries
* Updates
* Always focus the resource the agent is working on, and its browser tab
The resource panel had a carve-out: an already-open browser session declined
to replace another selection and only got an attention marker, so agent
browser work happened off-screen. The panel now follows the agent to whatever
it touches — browser included — and an event can still opt out explicitly.
The browser panel also follows the agent BETWEEN tabs: the store already
tracked automationTabId (and the strip marked it), but the visible tab never
changed. It now switches when the agent's target tab changes, so watching the
agent never means hunting for the tab it moved to. Keyed on the target
changing rather than on it being set, so a user who browses elsewhere
mid-run is only pulled along when the agent itself moves.
* Never paint a browser snapshot at stale geometry (the modal-open flash)
Opening a modal locks scroll, which removes the window scrollbar and reflows
the panel — so a capture taken before the lock describes a rect the panel no
longer occupies. The handshake painted that frame anyway and only then
retried, so the replacement landed visibly offset from the page it stands in
for: the flash. A capture is now checked against the host's live rect before
it is painted; a mismatched frame is skipped and re-captured at the settled
layout instead (modal retries go 2 -> 3 to absorb the extra settle).
* Name the workflow in deployment and workflow-scoped tool titles
'Checked deployment status' never said which workflow — nor did the deployed-
state read, run settings, block outputs/inputs, redeploy, promote, or the
global-variable write. These tools carry a workflowId (often defaulting to the
current workflow), so only the client can resolve a name: the enrichment layer
now resolves it for the whole workflow-scoped family and passes it as
workflowName, which every workflow title already reads.
Titles: Checking {workflow} deployment status, Reading deployed {workflow},
Checking {workflow} run settings, Reading {workflow} block outputs, Tracing
{workflow} block inputs, Redeploying {workflow}, Promoting {workflow} version
{n} to live, and 'Adding workflow variable {name} in {workflow}' — each
falling back to its unnamed form when no workflow resolves.
* Name the block that ran; never fall back to a raw block id
run_block and set_block_enabled carry only a blockId, so their titles would
have printed an opaque UUID ('Running 7f3a2b91-… in Invoice Sync'). The
enrichment layer now resolves blockId against the workflow store the same way
it already did for run_from_block's startBlockId, and the base titles no
longer accept an id as a name — an unresolved block reads 'Running block'
rather than a UUID.
* Add the missing Removing -> Removed rewrite
The table work introduced 'Removing automation'/'Removing enrichment' with no
past form, so those rows kept their present tense after completing.
* Name the target resource in the remaining tool titles
Table tools keep their operands nested under args and identify the table by
id, so their rows said 'Adding rows' with no hint where: enrichment now lifts
the nested args and resolves tableId against the cached workspace table list,
giving 'Adding rows to Runtimes', 'Adding column status in Runtimes',
'Reading views of Runtimes'.
Also: a block-schema read names the block instead of the file ('Loading
Slack', 'Loading Google Sheets tips'); browser type/insert show the text they
send, middle-ellipsized; downloads name the file; library-docs searches name
the library and query; knowledge-base searches include the query; generated
media names its output file; and diff_workflows, list_deployment_versions,
and publish_custom_block joined the workflow-name enrichment set.
* Bubble nested agents' tool calls into the parent's status line
The collapsed status only scanned a group's OWN tool items and skipped nested
agent groups, so a parent that had delegated froze on its last own tool while
its child did the actual work — the line described nothing that was running.
Status now walks the whole subtree: any tool at any depth counts, the most
recently started running one is shown, and the rest become the same '+ n'
overflow. With nothing running it falls back to the last tool at any depth,
so an idle parent still reflects where its subtree got to.
* Align nested tool call status rows
* Revert branch-local KB connector error-message edits
Restores apps/sim/connectors/ to staging state. Two copilot-focused commits
on this branch (3a9fc1c5a5, 24b24e5f8e) drove by nine KB source connectors
(airtable, confluence, discord, github, gitlab, google-drive,
microsoft-teams, notion, slack) to enrich credential-validation error
messages. The env-reference resolution machinery those errors supported is
kept; the product connector surface stays unchanged in this branch so the
staging promotion remains scoped to copilot work.
* Fix the failing audits: NUL escape and route-count ratchet
check:source-text: resource-vfs.ts used a raw NUL byte as its folder-index
key separator (comment and template literal), which makes git treat the file
as binary and hide it from review. Written as the '\u0000' escape — the
runtime string is identical.
check:api-validation:strict: baseline 1120 -> 1122 for the two routes added
on this branch since the last bump.
* Add Force Reload (Cmd+Shift+R) to desktop View menu
* Route Force Reload through focused-resource boundary; regen docs manifest
* Align the title-bar surface audit with the zoom rework
'Scale desktop title bar with page zoom' (833d2f126d) changed the CSS contract
in two ways its audit test still pinned the old shape of: the lane vars gained
a max() floor around the platform env() terms so page zoom cannot shrink the
lane below the OS-drawn lights, and the control square became fixed px — CSS px
already scale under zoom — leaving only the centering offset derived from the
lane height. The test required the bare env() prefix and calc() on all three
control vars, so it failed the commit that implemented its own regression
comment.
Pins now assert the env() term inside the clamp (still platform-derived, the
test's actual intent) and split the control vars: offset must stay computed,
size and icon are explicit constants.
* Budget the two structurally slow tests explicitly
sso-trust imports the whole Better Auth module graph (2.5s on an idle
machine) and events.attribution scans call sites across the repo. Under a
fully-parallel uncached run on a loaded machine both blow the default
timeout while passing in isolation and on CI — a verdict decided by machine
load, not by the code. 30s budgets make a loaded local run mean what it says.
* Carry the cross-service trace id in sim log lines
* Improve nested tool status presentation
* Refresh secret schemas and shared test mocks
* Open the deployed graph of org-published blocks, read-only, org-wide
A consuming workspace could see a published block's interface but never what
it does: the backing workflow lives in the publishing workspace, and other
workspaces are nameable, not readable. Publishing a block org-wide is the act
of sharing it, so the graph it executes is now readable from any org
workspace — the DEPLOYED graph, not the publishing workspace's live editor
state, so nothing in-progress leaks and what you read is what runs.
The namespace also adopts the root's index/detail split: custom-blocks.json
slims to names with a detail pointer, organization/custom-blocks/{type}.json
carries provenance plus the deployed graph (loaded lazily through the cached
loadDeployedWorkflowState; credential ids and env references inside it belong
to the publishing workspace and say so), and organization/README.md is the
namespace guide WORKSPACE.md is at the root — files, usage, the in-depth
block inventory, and forks.json documented only when actually mounted.
The block list moves to materialize time (same indexed query the components
pass already runs) because the README, the index, and the per-block key-view
entries all need it; only the graph stays lazy.
* Withhold the deployed graph from external collaborators
Workspace access and org membership are different grants: an external
collaborator can open the workspace and use the published block, so the names
index and the interface schema stay visible to them — but the deployed graph
is org implementation internals, and their detail files now simply do not
exist. isHostOrganizationMember is the viewer bit the host context already
resolves for exactly this distinction.
* Fill out the organization namespace: workspaces, permission groups, credential groups
Three more read-only files, all lazily loaded — the paths appear in the key
view so glob discovers them, but no query runs until a read — and all gated by
registration, so an unpermitted viewer's file simply does not exist:
- workspaces.json (org members): the org's full workspace map with the
viewer's access flag and fork parentage — account/workspaces.json only ever
showed what the viewer can reach. Inaccessible workspaces stay nameable,
not readable.
- permission-groups.json (org admins): every group with member count,
targeted workspaces, and the restrictions its config activates.
access-control.json remains the per-viewer binding. The queries are lifted
into lib/permission-groups/queries.ts because their only prior home was
inline drizzle in the route handlers, which the VFS cannot import.
- credential-groups.json (entitlement-gated): per-option configuration
readiness and enrollment progress — the two facts that decide whether a
credential_group workflow will do anything at runtime. The note teaches the
contract that bit the audit: an active group with zero completed
enrollments yields an empty loop, not an error. Enrollee emails are
workspace-admin-only, matching the settings page; counts come from the
first enrollment page and say so when truncated.
The README documents each file only when mounted for this viewer.
* Stop reporting a click that navigated as a failed click
The field case: 'Begin Assessment' submits a form. The navigation tears the
origin document down while the press completes, so everything after dispatch —
the CDP call's own completion, the synthetic dispatch's return value, the
postcondition reads — fails against a destroyed context, and a maximally
successful click was reported 'Failed clicking element'. The agent's own
follow-up investigation in the transcript diagnosed exactly this.
navigationRescue detects it at the driver level, where the navigation epoch
and URL survive the renderer teardown: when the page provably navigated since
dispatch began, a dispatch-path failure becomes a success carrying
navigatedDuringDispatch and a note explaining why no postconditions exist.
Applied to all four click dispatch paths (unframed CDP, framed native,
synthetic in-page, click_at).
The soft path had the same blindness: with the after-state unreadable,
urlChanged computed false and a navigating click reported 'no observable
change'. navigatedByDriver now folds into navigated/effectObserved, which also
keeps the new-dialog obstruction check meaningful on real navigations.
* Re-hide the browser under a modal after main loses the occlusion lease
The punch-through: a modal opens, the native view hides behind a painted
snapshot, and the renderer records applied: true. Then one heartbeat commit is
skipped — renderer jank past the 2.5s bounds-lease TTL is enough — and main
expires the lease, resetting panelOccluded on its side. The next heartbeat
finds the modal marker still present and calls setDesired(true), but the
lease's dedupe sees applied === desired and sends nothing; the bounds commit
that follows lays out an unoccluded native view above the open modal, and no
later event ever re-hides it. The comment on this branch already claimed it
'reasserts the lease' — the dedupe made that claim false exactly when the
lease had been lost.
While the occlusion marker is present, each heartbeat now drops the applied
belief (assumeRevealed) before setDesired, so the reassert is a real, forced,
idempotent hide IPC — one per second while a modal covers the browser — and
any main-side lease loss self-heals within a heartbeat.
* Make a stale ref name which of its five causes fired
A field run burned five snapshot->click cycles on a STATIC landing page, every
one refused with the same sentence — 'the page changed since the last
snapshot' — and the agent reasonably concluded the page was regenerating its
DOM. It was not; the resolver was refusing, and the message could not say why.
Five distinct conditions produced that one string: an id missing from the
registry, a connected node whose identity drifted, the view-changed adoption
gate, no confident replacement, and a replacement tie.
The resolver now stamps the reason (with the drifted node's current identity,
or the from->to paths for the view gate) and every stale producer carries it
into the driver message. Same pattern as not-editable: a refusal that names
its cause costs one round; an opaque one costs a loop and a wrong theory in
the bug report.
* Pulse throttling off when the browser view is revealed, so it actually paints
The blank-page report: a navigation completes while the view is hidden, the
page 'finishes loading', and the panel shows white until the user re-navigates
by hand. invalidate() on reveal was already there but recomposites the LAST
frame — and the last frame is blank, because background throttling suspended
the rAF the page's SPA paints its first frame from. The reveal now pulses
throttling off (forcing the renderer to produce a real frame), invalidates,
and hands the policy back to the session a second later through
reassertTabThrottling, which preserves the automation-tab exemption.
* Let the agent browser join meetings and finish passkey sign-ins
Camera and microphone: 'media' joins the agent partition's allowlist, but
every grant is gated on the macOS grant first — asked via
systemPreferences.askForMediaAccess so the system prompt appears on first use,
and answered from getMediaAccessStatus on checks — so System Settings stays
the real authority and a page can never hold a grant the OS refused. Granting
site permission without the OS grant produced the misleading NotReadableError
Google Meet showed. Packaging gains the camera entitlement and usage string
(macOS kills the process on prompt without one) and the mic string now covers
meetings.
Passkeys: WebAuthn itself is Chromium-native and nothing in our handlers
blocks it — USB security keys need no permission at all. The hybrid transport
(passkey on a nearby phone via QR) rides Bluetooth, which signed builds
silently lacked: the bluetooth entitlement and usage string enable it.
iCloud-Keychain platform passkeys remain outside what an entitlement here can
grant — Apple restricts that to approved browsers.
* Compile and preview Sim-styled pages
* Wait for the batched prepare intent instead of instantly failing apply_file_edit
The model batches prepare_file_edit and apply_file_edit into one round and
the Go loop runs same-round tools concurrently, so apply could reach the
executor before its prepare staged the intent. The instant no-intent error
cost a model retry round and flashed 'Failed creating …' on the shared file
row before the retry succeeded. The apply handler now polls briefly (10s
cap) for the intent; a truly missing prepare still errors at the deadline.
* Store page source, render docs-styled documents on view
The pdf model for agent pages: the .html file keeps the markdown-shaped
source (frontmatter + prose + sim: fences) and every surface renders the
docs-styled document on demand — preview panel, /api/files/serve, public
shares, and downloads all call the same pure compiler, now shared in
lib/workspace-files. The docs chrome is reproduced from the real fumadocs
source: the left sidebar's exact pill metrics, the clerk TOC with its
animated scroll indicator, divider-style tables; cards and stats left the
vocabulary. Table cells and kv values render inline markdown, and
sim:workflow/table/knowledge/file links resolve to real workspace routes,
bridged out of the sandboxed preview to the app router. Hand-written
imitations of rendered output are rejected at apply_file_edit with a steer
back to source, and a streaming page hides its source behind the live
rendered preview (batched ~2s) the way a generating pdf hides its script.
* Stagger the page rails so the resource panel keeps the docs sidebar
Rails were gated at 1100px of iframe width — the chat resource panel never
reaches that, so pages rendered single-column there. The rails now stagger
the way the docs do on a laptop: >=640px keeps the section sidebar (240px)
beside the content, >=1060px restores the full three-column frame with the
clerk TOC, and only a truly narrow pane collapses to one column.
* Fix in-page anchors escaping the preview; add the docs' toggle, code frames, pagination, and images
Clicking a TOC or section link (or pressing Enter in the section filter,
which clicks one) navigated the sandboxed frame off about:srcdoc in
Electron and landed on a cookie-less sign-in page — the shell now
intercepts every '#' anchor and scrolls directly. The page chrome gains
the docs' exact theme toggle (emcn Sun/Moon, 30px rounded-lg, top right),
framed code blocks with language label and copy button, and footer
previous/next cards from prev/next frontmatter. Workspace images
() compile to /api/files/view and the preview host
inlines them as blob: URLs so the cookie-less frame can render them;
sim:accordion joins the vocabulary as the faq component with title keys.
* Size the section sidebar to its content so it appears at panel widths
The left rail was a fixed 240/300px column, so it only earned its place
once the pane was wide. fit-content caps at the docs width but shrinks to
the longest section title (150px floor for the pills), and the two-column
tier now starts at 560px instead of 640.
* Let the clerk TOC join at 860px by sizing it to its content
Same move as the section sidebar: the TOC column fits its longest link
(capped at the docs' 268px, 150px floor), so the full three-column frame
starts at 860px of pane width instead of 1060.
* Open external links from pages in a new tab
The preview bootstrap cancelled every non-anchor click, so an external
link (the Sim docs, a vendor page) did nothing. External http(s) links now
compile with target=_blank rel=noopener for the standalone and share
surfaces, and the sandboxed preview bridges the click to the host, which
window.opens a new tab — same channel the workspace deep links use.
* Lock Sim pages to the rendered view via an internal record type
The record's contentType is stamped text/x-sim-page when apply_file_edit
detects page source — the file stays .html to the user (serving and
downloads still emit text/html), but every surface now knows what the file
holds before content loads. The viewer forces the rendered view for these
files at every moment: the first streamed chunk (whose frontmatter is
still partial) no longer flashes raw source, the gaps between an agent's
tool calls no longer flip back to raw HTML, and both toggle surfaces (the
Files toolbar and the resource-panel tabs) stop offering a code view for
them. Mid-stream compiles run lenient — a fence still being written is
malformed by definition, so its skip-notice callout is suppressed until
the stream settles.
* Honor the model's declared page type; default copilot .html to a page
An explicit contentType on create_empty_file always wins (the skill now
declares text/x-sim-page for pages, text/html for bespoke raw pages);
with no declaration a copilot-created .html defaults to the page type.
The first apply_file_edit still re-confirms from the actual content, and
the category map knows the internal mime explicitly instead of falling
through to the extension.
* Default undeclared .html back to plain text/html
A file is a Sim page only when the model declares it at creation or the
first written content proves it — never by extension alone.
* Match the docs' PageFooter for page navigation
The invented bordered cards with Previous/Next labels are replaced by the
docs' actual footer: the destination name with a 14px emcn chevron on a
flex-1 hover pill (rounded-lg, px-3 py-3, --surface-active), next
right-justified, and a spacer holding the empty half — verified against
apps/docs/components/docs-layout/page-footer.tsx.
* Scroll the rails invisibly, like the docs
The sticky TOC box is overflow-y auto, and the clerk track's absolutely
positioned SVGs could tip it a few pixels into overflow — Chromium then
painted a full scrollbar beside the rail. Rails now hide their scrollbar
chrome entirely (scrollbar-width none + webkit display none), matching
how the docs scroll their sidebar and TOC.
* Send sim:file links to the Files page, like a markdown link
A workspace-file link in a page now navigates exactly as one tagged in a
.md does: an in-app SPA push to /workspace/{ws}/files/{id} (the Files
page with the file open). The fullscreen /view route stays reserved for
the standalone surface; image refs keep the /api/files/view byte route.
* Inline workspace images after the page compiles, not before
The blob substitution ran on the raw source, where the compiled
/api/files/view src it looks for does not exist yet — so the sandboxed
cookie-less frame fetched every image itself and got 401s (broken image
icons). The substitution now runs on the built document, covering
compiled pages, legacy stored-compiled pages, and bespoke HTML alike.
* Highlight the section you are AT in the left rail, not the last one visible
The rail's current-section pick walked every heading visible in the
viewport and kept the last h2 — so clicking a section landed correctly
but highlighted whichever later section peeked in from below. Current is
now the last h2 at or above the top reading line (matching the 72px
scroll-padding a clicked anchor settles at), falling back to the first
visible section when everything is below the line.
* Stop the TOC jittering sideways as the highlight moves
Active TOC links step from weight 430 to 470, and the rail is a
fit-content column — every active-section change re-measured the longest
link and shifted the rail a pixel or two side to side. Each link now
carries a hidden zero-height ghost of itself at the active weight, so
the column always occupies its bold width and the highlight moves
without the layout moving.
* Absolutize links and images in served page documents
A downloaded page must behave like a downloaded .md whose links are
absolute: clicking a workflow reference opens Sim in the browser at that
workflow. The standalone renderer (plain download, fullscreen viewer,
shares) now compiles sim: links and workspace image refs against
getBaseUrl(); in-app surfaces keep relative paths and SPA navigation.
* Drop the eyebrow; add the docs' top page controls
The docs have no eyebrow line, so compiled pages no longer render one
(old sources still parse; the field is ignored). The title row gains the
docs' top controls: Copy page (copies the page text) and prev/next
chevrons wired to the same neighbors as the footer cards, disabled-dim
when a side is missing.
* Read kv keys and table first columns as labels, the docs way
kv keys dropped the blanket monospace — they render as the docs' row
labels (500, primary, sans), with backticks in the source opting a
code-like key (a path, an env var) into the inline-code chip; keys now
run through inline markdown to make that work. Table body first columns
pick up the same label treatment the docs tables show.
* Platform font and emcn chrome for pages
Pages live inside the app, and the platform — emcn, every workspace
surface — renders the system stack, not the docs' Inter webfont; Inter
made pages read as foreign next to the app around them. The face is now
the platform stack with weights on the platform scale (400/500/600), the
Inter delivery machinery (page-font.ts, the preview data-URI fetch, the
public woff2) is gone, and the section filter wears emcn ChipInput's
exact chrome (30px rounded-lg, --surface-5 fill flipping to --surface-4
in dark, --border, 14px, no focus ring). The docs' geometry — layout,
spacing, rails, tables, code frames — is unchanged.
* Color-only active state in the TOC — width can never move again
Two attempts at reserving the bold width (a hidden ghost, then freezing
measured rail widths) each traded one artifact for another: the ghost
did not stop the fit-content column re-measuring, and the freeze made a
long link wrap to two lines when it gained weight. The root cause was
letting the active state change a width-affecting property at all: the
active TOC link now shifts color only (muted to primary — the clerk
thumb already carries the emphasis), the search input is a plain text
field (the native search clear button is not emcn chrome), and the
ghost/freeze machinery is gone.
* Drop the Copy page control; keep the top chevrons
The title-row actions keep only the previous/next chevrons (rendered
when the page has neighbors); the Copy page button is gone.
* Set-level sidebar: docs-style groups with the current page expanded
Multi-page sets can now carry the whole set's sidebar. nav frontmatter
(groups of labelled page links, identical on every page of the set)
compiles into hidden set-nav markup whose sim: links resolve like any
other; the shell lifts it into the left rail as muted group labels over
page links, recognises the current page by title, and nests that page's
section list beneath it — the docs sidebar's exact shape. Pages without
nav keep the plain section list.
* Center the content column at the docs' measure
On wide panes the 1fr center cell stretched, so content hugged the left
rail with dead space before the TOC. The main column now caps at the
docs' ~760px measure and centers in its cell, matching how the docs
balance a wide viewport.
* Docs Steps, code-tab groups, and API method chips
Three docs components join the vocabulary: sim:steps renders the
numbered timeline (muted circle markers, hairline connector, title and
content per step); sim:tabs renders the docs' grouped code block (mono
tab chips, one pane at a time, the icon copy control targeting the
visible pane); and a METHOD prefix on a set-nav page entry renders the
API-reference chip — sidebar entries only, on the platform badge tokens
(blue and purple added to the mirrors and the live bridge).
Also: preview images switched from blob: to data: URIs — blob URLs are
origin-bound and the sandboxed frame's origin is opaque, so Chromium
refused to render them — and the page view-lock is sticky per file so a
patch stream cannot flash raw source.
* Downloaded pages carry their images
Absolute URLs made LINKS survive a download, but an embedded image
request from a downloaded file is cross-site and carries no session
cookie, so images 401ed outside the app. The standalone renderer now
inlines every workspace image the page references as a data: URI at
serve time — like a pdf carrying its images — capped at 8MB per image,
restricted to the page's own workspace, falling back to the URL
reference on any miss. Applies to serve, download, and public shares.
* Dead-center the content column between equal gutters
The rails are content-sized and unequal, so the old grid (fit-content /
1fr / fit-content) skewed the middle cell toward whichever rail was
narrower. The wide tier now uses the docs' geometry: a fixed 760px
content column centered between two equal flexible gutters, the sidebar
hugging the container's left edge and the TOC its right — rail widths
can no longer move the content.
* Defer title-bar history state out of currententrychange dispatch
The Navigation API fires currententrychange synchronously from the
history mutation that caused it, which can originate inside another
component's useInsertionEffect (style libraries navigating during
commit) — setState there trips React's 'useInsertionEffect must not
schedule updates'. The arrow-state sync now defers to a microtask,
flushing after the commit unwinds, with a disposal guard.
* Show the section sidebar only when there are sections to list
Fewer than two sections (and no set nav): the left rail and its filter
disappear and the reserved left gutter collapses — the content column
leads the container with the TOC trailing. Two or more sections, or a
multi-page set, keep the full centered docs frame.
* Execute the page shell in a DOM harness
jsdom runs the real shell against compiled pages and asserts the layout
decisions: both rails on a many-section page, only the left rail dropped
on a one-section page.
* Medium panes keep the TOC, not the sidebar
Between 560 and 860px the frame showed the section sidebar and hid the
clerk TOC — backwards by our own reasoning, since the sidebar is the
redundant list on a single page. The TOC now survives at medium widths
and the sidebar joins only on wide panes.
* Sidebar is for doc sets only; stagger the rails like the docs
The left rail now exists only for multi-page sets — on a lone page it
just repeated the TOC. A set opens its sidebar at 560px with the TOC
joining on wide panes (the docs stagger); a lone page waits until 700px
and then shows the TOC alone. Set-sidebar spacing tightens to the docs
values, with the current page's nested sections styled as small muted
entries behind a hairline instead of full chips.
* Center the lone-page content and TOC as a pair
On a page with no sidebar the content column stretched while the TOC
hugged the far edge, leaving a field of dead space between them. The
content now caps at reading width with the TOC directly beside it and
the pair centered in the pane, and the TOC waits until 800px so narrow
panes stay single-column a while longer.
* Equalize grid-template specificity across the rail tiers
The 560px tier selects .art-cols:not(.no-side-nav) at (0,2,0), so the
860px tier's bare .art-cols template at (0,1,0) could never win and a
set page on a wide pane kept the 2-column template — wrapping the TOC
to the next grid row, bottom-left. The wide template now carries the
same :not() guard, the 860 block's duplicate of the 800px no-side-nav
rules is gone, and a test pins every template rule to equal specificity
so a future tier can't silently lose the cascade again.
* extract_doc_assets: pull a reference deck's assets into the workspace
Sim-side handler for the new file-agent tool: given an uploaded .pptx
or .docx, unzip it (OOXML is a zip), parse theme1.xml into theme.json
(color scheme as hex, major/minor fonts, slide size from
presentation.xml) and write every ppt|word/media file into a
"<Name> assets" folder with original bytes and real content types.
Re-runs overwrite the set in place. Pure extractor unit-tested against
in-test-built packages; display label "Extracting assets from <file>".
* extract_doc_assets learns .pdf via the doc sandbox
PDFs have no zip structure or declared theme, so extraction runs in the
same vetted sandbox that compiles and renders documents: poppler's
pdfimages dumps every embedded image in its native format (masks
filtered via -list), pdfplumber contributes each image's placement
rects in page points plus the document's font names, and rendered pages
are sampled into an explicitly-inferred color palette. theme.json for a
pdf carries fonts, page size and count, the inferred palette, and a
per-asset placement map.
* Pages have one navigation rail; compile errors go to the agent
The left sidebar leaves the renderer and the DSL: the shell builds only
the content column and the clerk TOC (pair centered at 800px, one bare
.art-cols selector per tier so the cascade cannot invert), the filter
box goes with it, and nav frontmatter is tolerated but no longer
rendered — sidebar METHOD chips and the set-nav markup are gone.
Malformed sim: blocks no longer render a reader-facing "block was
skipped" card: the block is omitted and the failure is reported as a
diagnostic that apply_file_edit appends to its result, so the authoring
agent sees exactly which fence to fix. The lenient flag existed only to
suppress those cards mid-stream and is removed.
The steps timeline connector now derives its position from the marker
size, so it stays centered under the number circles.
* Sync tool catalog: extract_doc_assets accepts pdf
* Asset extraction yields the rebuild recipe, not just the parts
pptx: theme.json now maps every image to its slide-by-slide placements
(slide rels resolve rIds to media names; each pic frame's EMU offset and
extent convert to inches) plus the slide count.
pdf: a second layout.json is written — per page, the text blocks with
content, position, font, size, and fill color; the filled rects
(backgrounds and scrims); and rect-over-image overlay detection with
coverage, which is the "image opacity" effect decks fake with a tinted
rect. Stream alpha is unrecoverable, so overlays name the color and the
rendered page remains the reference for strength.
* Split shared-baseline text runs into separate blocks
Two text boxes sitting at the same height merged into one wide line;
a gap much wider than a space now starts a new block, so columns and
label/value pairs land as distinct entries in layout.json.
* Extract faithful document layout recipes
* Add .chart files: live interactive ECharts docs, static or table-backed
* Size charts by width-driven aspect, not panel height; separate title and legend
* Map table chart rows from storage column ids to display names
* Inject table rows as datasetIndex 0 so specs can transform; stagger array legends
* Give .chart files their own bar-chart icon
* Chart table sources gain groupBy/aggregate/pivot shaping; renderer-owned chrome
* sim:chart page fence: ECharts SSR to themed inline SVG; shared option builder
* sim:chart hydrated embeds: inline or .chart file refs, live table reads per serve
* Finish extensionless pages and document staging
* Render live charts without server hydration
* Cover active-theme page token overrides
* Keep artifact tokens synced with the app theme
* Use tabs for multi-page Sim docs
* Preserve dollar-prefixed tool credentials
* Build chart specs from validated fields
* Sync new integration docs into Copilot manifest
* Add in-document tabs to Sim pages
* Rebuild page TOC on tab changes
* Keep page tabs with docs chrome
* Stabilize tabbed page layout
* Regenerate docs manifest for staging's Modal docs page
* Share one divider between the bar and the chrome tab row
* Pin the keyless OCR path in the unreadable-document test
---------
Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
This commit is contained in:
Siddharth Ganesan
co-authored by
Justin Blumencranz
Vikhyath Mondreti
Justin Blumencranz
Vikhyath Mondreti
parent
6f34dd6248
commit
d9cfd7c68e
@@ -123,6 +123,9 @@ jobs:
|
||||
- name: Repo audits
|
||||
run: bun run check:audits
|
||||
|
||||
- name: Verify docs manifest is in sync
|
||||
run: bun run docs-manifest:check
|
||||
|
||||
- name: Migration safety (zero-downtime) audit
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "pull_request" ]; then
|
||||
|
||||
@@ -9,5 +9,15 @@
|
||||
even after the user grants access in System Settings. -->
|
||||
<key>com.apple.security.device.audio-input</key>
|
||||
<true/>
|
||||
<!-- The agent browser joins real meetings (Google Meet, Zoom web): its
|
||||
getUserMedia grant is gated on the OS grant, and without this key the
|
||||
Hardened Runtime denies the camera no matter what the user allowed. -->
|
||||
<key>com.apple.security.device.camera</key>
|
||||
<true/>
|
||||
<!-- WebAuthn hybrid transport (passkey on the user's phone via QR) rides
|
||||
Bluetooth proximity; without this the QR option silently never
|
||||
completes in signed builds. -->
|
||||
<key>com.apple.security.device.bluetooth</key>
|
||||
<true/>
|
||||
</dict>
|
||||
</plist>
|
||||
|
||||
@@ -58,7 +58,9 @@ mac:
|
||||
# macOS refuses to show the microphone prompt at all — it kills the process —
|
||||
# unless the bundle declares why it wants the device.
|
||||
extendInfo:
|
||||
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat.
|
||||
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat and for meetings you join in the built-in browser.
|
||||
NSCameraUsageDescription: Sim uses your camera for meetings you join in the built-in browser, such as Google Meet.
|
||||
NSBluetoothAlwaysUsageDescription: Sim uses Bluetooth to complete passkey sign-ins with a nearby phone in the built-in browser.
|
||||
entitlements: build/entitlements.mac.plist
|
||||
entitlementsInherit: build/entitlements.mac.plist
|
||||
notarize: true
|
||||
|
||||
@@ -2,8 +2,9 @@ import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.mock('electron', () => import('@/test/electron-mock'))
|
||||
|
||||
import { WebContentsView, type WebFrameMain } from 'electron'
|
||||
import { nativeImage, type WebContents, WebContentsView, type WebFrameMain } from 'electron'
|
||||
import {
|
||||
captureScreenshot,
|
||||
clickAt,
|
||||
ensureInstrumented,
|
||||
evaluateInIsolatedFrame,
|
||||
@@ -482,3 +483,89 @@ describe('browser-agent CDP theme', () => {
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* The browser panel shows a LIVE view, so a capture must not perturb the page.
|
||||
* Chromium serves `clip` by applying device-emulation params to the widget and
|
||||
* syncing visual properties, which the user sees as the page rescaling and
|
||||
* snapping back. Resolution is bounded on the returned image instead.
|
||||
*/
|
||||
describe('browser-agent screenshot capture', () => {
|
||||
function captureFixture(imageSize: { width: number; height: number } | null) {
|
||||
const contents = new WebContentsView().webContents
|
||||
vi.mocked(contents.debugger.sendCommand).mockImplementation((method: string) => {
|
||||
if (method === 'Page.getLayoutMetrics') {
|
||||
return Promise.resolve({ cssLayoutViewport: { clientWidth: 2048, clientHeight: 1024 } })
|
||||
}
|
||||
if (method === 'Page.captureScreenshot') return Promise.resolve({ data: 'c2lt' })
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
const resized = {
|
||||
toJPEG: vi.fn(() => Buffer.from('resized')),
|
||||
}
|
||||
// Shared module-level mock: without this, a later fixture reads the
|
||||
// earlier test's decoded image.
|
||||
vi.mocked(nativeImage.createFromBuffer).mockReset()
|
||||
vi.mocked(nativeImage.createFromBuffer).mockReturnValue({
|
||||
isEmpty: vi.fn(() => imageSize === null),
|
||||
getSize: vi.fn(() => imageSize ?? { width: 0, height: 0 }),
|
||||
resize: vi.fn(() => resized),
|
||||
toJPEG: vi.fn(() => Buffer.alloc(0)),
|
||||
} as unknown as ReturnType<typeof nativeImage.createFromBuffer>)
|
||||
return { contents, resized }
|
||||
}
|
||||
|
||||
function screenshotParams(contents: WebContents): Record<string, unknown> {
|
||||
const call = vi
|
||||
.mocked(contents.debugger.sendCommand)
|
||||
.mock.calls.find(([method]) => method === 'Page.captureScreenshot')
|
||||
if (!call) throw new Error('no capture was requested')
|
||||
return call[1] as Record<string, unknown>
|
||||
}
|
||||
|
||||
it('never sends a clip, which would emulate the live page for the capture', async () => {
|
||||
const { contents } = captureFixture({ width: 4096, height: 2048 })
|
||||
|
||||
await captureScreenshot(contents)
|
||||
|
||||
expect(screenshotParams(contents)).not.toHaveProperty('clip')
|
||||
})
|
||||
|
||||
/**
|
||||
* A 2048px CSS viewport bounded to 1024px is scale 0.5, and the capture
|
||||
* arrives at device resolution (4096px on a 2x display). The resize is what
|
||||
* lands the image on the CSS-relative size the coordinate contract
|
||||
* (cssX = imageX / scale) assumes.
|
||||
*/
|
||||
it('downscales the returned image to the CSS-relative size', async () => {
|
||||
const { contents, resized } = captureFixture({ width: 4096, height: 2048 })
|
||||
|
||||
const shot = await captureScreenshot(contents)
|
||||
|
||||
const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
|
||||
expect(image.resize).toHaveBeenCalledWith({ width: 1024, height: 512, quality: 'good' })
|
||||
expect(resized.toJPEG).toHaveBeenCalled()
|
||||
expect(shot).toEqual({
|
||||
dataUrl: `data:image/jpeg;base64,${Buffer.from('resized').toString('base64')}`,
|
||||
scale: 0.5,
|
||||
})
|
||||
})
|
||||
|
||||
it('skips the re-encode when the capture already matches the target size', async () => {
|
||||
const { contents } = captureFixture({ width: 1024, height: 512 })
|
||||
|
||||
const shot = await captureScreenshot(contents)
|
||||
|
||||
const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
|
||||
expect(image.resize).not.toHaveBeenCalled()
|
||||
expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
|
||||
})
|
||||
|
||||
it('returns the raw capture when the image cannot be decoded', async () => {
|
||||
const { contents } = captureFixture(null)
|
||||
|
||||
const shot = await captureScreenshot(contents)
|
||||
|
||||
expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -10,7 +10,8 @@
|
||||
*/
|
||||
import type { BrowserTheme } from '@sim/browser-protocol'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import type { WebContents, WebFrameMain } from 'electron'
|
||||
import { sleep } from '@sim/utils/helpers'
|
||||
import { nativeImage, type WebContents, type WebFrameMain } from 'electron'
|
||||
|
||||
const logger = createLogger('BrowserAgentCdp')
|
||||
|
||||
@@ -127,12 +128,29 @@ export async function setColorScheme(contents: WebContents, theme: BrowserTheme)
|
||||
})
|
||||
}
|
||||
|
||||
/** Live drag-interception state while a dragPointer call is in flight. */
|
||||
interface DragInterception {
|
||||
intercepted: boolean
|
||||
data: Record<string, unknown> | null
|
||||
}
|
||||
const dragInterceptionsByContents = new WeakMap<WebContents, DragInterception>()
|
||||
|
||||
function handleDebuggerEvent(
|
||||
contents: WebContents,
|
||||
method: string,
|
||||
params: Record<string, unknown>,
|
||||
parentSessionId?: string
|
||||
): void {
|
||||
if (method === 'Input.dragIntercepted') {
|
||||
const interception = dragInterceptionsByContents.get(contents)
|
||||
if (interception) {
|
||||
interception.intercepted = true
|
||||
const data = params.data
|
||||
interception.data =
|
||||
data && typeof data === 'object' ? (data as Record<string, unknown>) : null
|
||||
}
|
||||
return
|
||||
}
|
||||
if (method === 'Target.attachedToTarget') {
|
||||
const sessionId = typeof params.sessionId === 'string' ? params.sessionId : ''
|
||||
const targetInfo = params.targetInfo
|
||||
@@ -352,6 +370,14 @@ export async function evaluateInIsolatedFrame(
|
||||
*/
|
||||
const MAX_SCREENSHOT_EDGE = 1024
|
||||
const SCREENSHOT_QUALITY = 70
|
||||
/**
|
||||
* Quality of the intermediate capture, before the in-process downscale
|
||||
* re-encodes at {@link SCREENSHOT_QUALITY}. Higher than the final quality so
|
||||
* the two lossy passes together land near where one pass did — the model reads
|
||||
* text out of these frames, and compression artifacts on glyphs cost more than
|
||||
* the transient bytes do.
|
||||
*/
|
||||
const SCREENSHOT_CAPTURE_QUALITY = 90
|
||||
|
||||
interface CdpViewport {
|
||||
clientWidth: number
|
||||
@@ -361,13 +387,23 @@ interface CdpViewport {
|
||||
/**
|
||||
* Screenshot via CDP (works while the view is hidden), bounded in resolution.
|
||||
*
|
||||
* `clip.scale` is relative to CSS pixels, so passing the CSS viewport with a
|
||||
* scale of 1 already sidesteps the device pixel ratio — an unclipped capture
|
||||
* on a 2x display returns a 2x image. Scaling further down keeps the longest
|
||||
* edge within {@link MAX_SCREENSHOT_EDGE}. Falls back to an unclipped capture
|
||||
* when layout metrics are unavailable.
|
||||
* The capture is deliberately UNCLIPPED. Chromium implements `clip` by applying
|
||||
* device-emulation parameters (viewport offset and scale) to the widget and
|
||||
* synchronizing visual properties, then restoring them. On a headless target
|
||||
* that is invisible; against the live, composited WebContentsView the Sim
|
||||
* resource panel shows, it is a real visual-properties round-trip, and the page
|
||||
* visibly rescales and snaps back — the screenshot flash. `panel.ts`'s own
|
||||
* snapshot capture refuses to scale a visible surface for the same reason.
|
||||
*
|
||||
* Bounding resolution therefore happens here instead, on the returned image.
|
||||
* The output keeps the dimensions the clipped capture produced, so `scale`
|
||||
* still maps image pixels back to CSS pixels for the coordinate tools
|
||||
* (cssX = imageX / scale) — including on a 2x display, where an unclipped
|
||||
* capture arrives at device resolution and this is what brings it back down.
|
||||
*/
|
||||
export async function captureScreenshot(contents: WebContents): Promise<string> {
|
||||
export async function captureScreenshot(
|
||||
contents: WebContents
|
||||
): Promise<{ dataUrl: string; scale: number }> {
|
||||
const metrics = await send<{
|
||||
cssLayoutViewport?: CdpViewport
|
||||
layoutViewport?: CdpViewport
|
||||
@@ -376,23 +412,34 @@ export async function captureScreenshot(contents: WebContents): Promise<string>
|
||||
const viewport = metrics?.cssLayoutViewport ?? metrics?.layoutViewport
|
||||
const width = viewport?.clientWidth ?? 0
|
||||
const height = viewport?.clientHeight ?? 0
|
||||
const clip =
|
||||
width > 0 && height > 0
|
||||
? {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width,
|
||||
height,
|
||||
scale: Math.min(1, MAX_SCREENSHOT_EDGE / Math.max(width, height)),
|
||||
}
|
||||
: undefined
|
||||
const scale =
|
||||
width > 0 && height > 0 ? Math.min(1, MAX_SCREENSHOT_EDGE / Math.max(width, height)) : 1
|
||||
|
||||
const result = await send<{ data: string }>(contents, 'Page.captureScreenshot', {
|
||||
format: 'jpeg',
|
||||
quality: SCREENSHOT_QUALITY,
|
||||
...(clip ? { clip } : {}),
|
||||
quality: SCREENSHOT_CAPTURE_QUALITY,
|
||||
})
|
||||
return `data:image/jpeg;base64,${result.data}`
|
||||
const captured = `data:image/jpeg;base64,${result.data}`
|
||||
|
||||
const targetWidth = Math.round(width * scale)
|
||||
const targetHeight = Math.round(height * scale)
|
||||
// Without layout metrics there is no CSS frame of reference to resize
|
||||
// against, so the raw capture is the honest answer — the same fallback the
|
||||
// clipped path took.
|
||||
if (targetWidth <= 0 || targetHeight <= 0) return { dataUrl: captured, scale }
|
||||
|
||||
const image = nativeImage.createFromBuffer(Buffer.from(result.data, 'base64'))
|
||||
const size = image.isEmpty() ? { width: 0, height: 0 } : image.getSize()
|
||||
if (size.width === 0 || size.height === 0) return { dataUrl: captured, scale }
|
||||
if (size.width === targetWidth && size.height === targetHeight) {
|
||||
return { dataUrl: captured, scale }
|
||||
}
|
||||
|
||||
const resized = image.resize({ width: targetWidth, height: targetHeight, quality: 'good' })
|
||||
return {
|
||||
dataUrl: `data:image/jpeg;base64,${resized.toJPEG(SCREENSHOT_QUALITY).toString('base64')}`,
|
||||
scale,
|
||||
}
|
||||
}
|
||||
|
||||
/** One half of a trusted key press (`Input.dispatchKeyEvent` params). */
|
||||
@@ -430,7 +477,8 @@ export async function clickAt(
|
||||
contents: WebContents,
|
||||
x: number,
|
||||
y: number,
|
||||
moveBeforePress = true
|
||||
moveBeforePress = true,
|
||||
clickCount = 1
|
||||
): Promise<void> {
|
||||
if (moveBeforePress) await moveMouse(contents, x, y)
|
||||
let pressed = false
|
||||
@@ -439,22 +487,26 @@ export async function clickAt(
|
||||
// response (navigation/process swap). In that ambiguous case a release is
|
||||
// safer than leaving Blink's pointer state stuck down.
|
||||
pressed = true
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mousePressed',
|
||||
x,
|
||||
y,
|
||||
button: 'left',
|
||||
buttons: 1,
|
||||
clickCount: 1,
|
||||
})
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mouseReleased',
|
||||
x,
|
||||
y,
|
||||
button: 'left',
|
||||
buttons: 0,
|
||||
clickCount: 1,
|
||||
})
|
||||
// A multi-click is a sequence of press/release pairs with an increasing
|
||||
// clickCount — Blink synthesizes dblclick from the pair whose count is 2.
|
||||
for (let count = 1; count <= clickCount; count++) {
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mousePressed',
|
||||
x,
|
||||
y,
|
||||
button: 'left',
|
||||
buttons: 1,
|
||||
clickCount: count,
|
||||
})
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mouseReleased',
|
||||
x,
|
||||
y,
|
||||
button: 'left',
|
||||
buttons: 0,
|
||||
clickCount: count,
|
||||
})
|
||||
}
|
||||
pressed = false
|
||||
} finally {
|
||||
if (pressed && !contents.isDestroyed()) {
|
||||
@@ -473,6 +525,131 @@ export async function clickAt(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drags the pointer from one viewport point to another through the trusted
|
||||
* pipeline: press, a threshold-crossing nudge, interpolated moves with the
|
||||
* button held, a settle hold over the target, then release or drop.
|
||||
*
|
||||
* Two drag models are covered by the one call. Pointer-sensor libraries
|
||||
* (dnd-kit, react-beautiful-dnd, canvas apps) treat the held-button move
|
||||
* sequence exactly like a human drag. Native HTML5 `draggable="true"`
|
||||
* sources instead START a Blink drag session on the press+move — with
|
||||
* `Input.setInterceptDrags` enabled, Chromium reports it as
|
||||
* `Input.dragIntercepted` and the remaining movement is delivered as trusted
|
||||
* `Input.dispatchDragEvent` dragEnter/dragOver events ending in a `drop`
|
||||
* (the technique Playwright uses). Both paths are trusted input.
|
||||
*/
|
||||
export async function dragPointer(
|
||||
contents: WebContents,
|
||||
from: { x: number; y: number },
|
||||
to: { x: number; y: number },
|
||||
steps = 12,
|
||||
stepDelayMs = 20
|
||||
): Promise<{ nativeDragIntercepted: boolean }> {
|
||||
const interception: DragInterception = { intercepted: false, data: null }
|
||||
dragInterceptionsByContents.set(contents, interception)
|
||||
let interceptEnabled = false
|
||||
try {
|
||||
await send(contents, 'Input.setInterceptDrags', { enabled: true })
|
||||
interceptEnabled = true
|
||||
} catch {
|
||||
// Chromium without drag interception: the pointer-only path still works
|
||||
// for pointer-sensor drags; native HTML5 sources will report no effect.
|
||||
}
|
||||
await moveMouse(contents, from.x, from.y)
|
||||
let pressed = false
|
||||
let dragEnterSent = false
|
||||
const dragMove = async (x: number, y: number) => {
|
||||
if (interception.intercepted && interception.data) {
|
||||
await sendInput(contents, 'Input.dispatchDragEvent', {
|
||||
type: dragEnterSent ? 'dragOver' : 'dragEnter',
|
||||
x,
|
||||
y,
|
||||
data: interception.data,
|
||||
})
|
||||
dragEnterSent = true
|
||||
} else {
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mouseMoved',
|
||||
x,
|
||||
y,
|
||||
button: 'left',
|
||||
buttons: 1,
|
||||
})
|
||||
}
|
||||
}
|
||||
try {
|
||||
pressed = true
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mousePressed',
|
||||
x: from.x,
|
||||
y: from.y,
|
||||
button: 'left',
|
||||
buttons: 1,
|
||||
clickCount: 1,
|
||||
})
|
||||
// Small first nudge so libraries with a start threshold (commonly 3-8px)
|
||||
// register the drag before the pointer sweeps across the page.
|
||||
await dragMove(from.x + Math.sign(to.x - from.x || 1) * 4, from.y + 2)
|
||||
await sleep(stepDelayMs)
|
||||
const stepCount = Math.max(2, steps)
|
||||
for (let step = 1; step <= stepCount; step++) {
|
||||
const progress = step / stepCount
|
||||
await dragMove(from.x + (to.x - from.x) * progress, from.y + (to.y - from.y) * progress)
|
||||
await sleep(stepDelayMs)
|
||||
}
|
||||
// Hold over the target so drop zones running enter/over animations settle
|
||||
// before the release lands.
|
||||
await sleep(120)
|
||||
if (interception.intercepted && interception.data) {
|
||||
await sendInput(contents, 'Input.dispatchDragEvent', {
|
||||
type: 'drop',
|
||||
x: to.x,
|
||||
y: to.y,
|
||||
data: interception.data,
|
||||
})
|
||||
// Blink ended the intercepted drag session itself; a trailing
|
||||
// mouseReleased would be a stray click on the drop target.
|
||||
pressed = false
|
||||
} else {
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mouseReleased',
|
||||
x: to.x,
|
||||
y: to.y,
|
||||
button: 'left',
|
||||
buttons: 0,
|
||||
clickCount: 1,
|
||||
})
|
||||
pressed = false
|
||||
}
|
||||
return { nativeDragIntercepted: interception.intercepted }
|
||||
} finally {
|
||||
if (pressed && !contents.isDestroyed()) {
|
||||
if (interception.intercepted && interception.data) {
|
||||
await sendInput(contents, 'Input.dispatchDragEvent', {
|
||||
type: 'dragCancel',
|
||||
x: to.x,
|
||||
y: to.y,
|
||||
data: interception.data,
|
||||
}).catch(() => {})
|
||||
} else {
|
||||
await sendInput(contents, 'Input.dispatchMouseEvent', {
|
||||
type: 'mouseReleased',
|
||||
x: to.x,
|
||||
y: to.y,
|
||||
button: 'left',
|
||||
buttons: 0,
|
||||
clickCount: 1,
|
||||
}).catch(() => {})
|
||||
}
|
||||
}
|
||||
dragInterceptionsByContents.delete(contents)
|
||||
if (interceptEnabled && !contents.isDestroyed()) {
|
||||
await send(contents, 'Input.setInterceptDrags', { enabled: false }).catch(() => {})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Inserts text at the focused element's selection (replacing it) through the
|
||||
* native IME path — works in plain fields and code editors alike.
|
||||
|
||||
@@ -1121,6 +1121,20 @@ describe('credential protection', () => {
|
||||
expect(cdpCalls(contents, 'Input.dispatchKeyEvent').length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('still dispatches input after the user has interacted with the visible tab', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, { activeElementSecrecy: 'safe', readActiveElementState: {} })
|
||||
// User interaction claims the visible tab for panel-level ownership
|
||||
// (popups, close protection) but must never block agent input.
|
||||
session.claimActiveTabForUser()
|
||||
expect(session.automationTabClaimedByUser()).toBe(true)
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_press_key', { key: 'a' })
|
||||
|
||||
expect(result.ok).toBe(true)
|
||||
expect(cdpCalls(contents, 'Input.dispatchKeyEvent').length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('sends the keystroke when nothing sensitive is focused', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, { activeElementSecrecy: 'safe', readActiveElementState: {} })
|
||||
@@ -1646,7 +1660,90 @@ describe('credential protection', () => {
|
||||
expect(mainFrame.executeJavaScript).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reports navigation that remains obstructed by a DOM dialog', async () => {
|
||||
// A dialog that was ALREADY open before the click is not obstructing the
|
||||
// navigation it survived — reporting it made every SPA route change under a
|
||||
// persistent role=dialog (cookie banner, side drawer, picker) read as a
|
||||
// failed click. Only a dialog that arrives with the navigation obstructs it.
|
||||
// targetChanged can only fire when pageActionState was given an elementId.
|
||||
// Tools without one listed it in their effect formula for a long time, where
|
||||
// it was silently always false — coverage that read as real. This pins the
|
||||
// dependency so the next tool that adds the term has to earn it.
|
||||
it('cannot observe a target change for a tool that passes no elementId', async () => {
|
||||
const contents = await openPage()
|
||||
let actionReads = 0
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'readActiveElementState')) return Promise.resolve({})
|
||||
if (isPageCall(expression, 'readPageActionState')) {
|
||||
actionReads++
|
||||
// No targetState in either sample: that is what a call without an
|
||||
// elementId returns.
|
||||
return Promise.resolve({
|
||||
url: 'https://example.com/a',
|
||||
title: 'A',
|
||||
focus: 'body',
|
||||
mutationRevision: actionReads === 1 ? 0 : 3,
|
||||
dialogs: [],
|
||||
popups: [],
|
||||
scroll: [0],
|
||||
})
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_press_key', { key: 'Enter' })
|
||||
|
||||
expect(result).toMatchObject({ ok: true })
|
||||
const effect = (result as { result?: { effect?: Record<string, boolean> } }).result?.effect
|
||||
expect(effect?.targetChanged).toBe(false)
|
||||
})
|
||||
|
||||
// The click-that-navigates race from the field: "Begin Assessment" submits a
|
||||
// form, the navigation tears the origin document down, and the CDP dispatch
|
||||
// rejects mid-press. The press already reached the page — the navigation IS
|
||||
// the success — so this must come back dispatched, not failed.
|
||||
it('reports a click whose navigation destroyed the page as a success', async () => {
|
||||
const contents = await openPage()
|
||||
let currentUrl = 'https://example.com/tests/IPIP-BFFM/'
|
||||
vi.mocked(contents.getURL).mockImplementation(() => currentUrl)
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'describePointTarget')) {
|
||||
return Promise.resolve({ found: true, element: 'Begin Assessment', cursor: 'pointer' })
|
||||
}
|
||||
if (isPageCall(expression, 'readActiveElementState')) return Promise.resolve({})
|
||||
if (isPageCall(expression, 'readPageActionState')) {
|
||||
return Promise.resolve({
|
||||
url: currentUrl,
|
||||
title: 'Test',
|
||||
focus: 'body',
|
||||
mutationRevision: 0,
|
||||
dialogs: [],
|
||||
popups: [],
|
||||
scroll: [0],
|
||||
})
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
vi.mocked(contents.debugger.sendCommand).mockImplementation(async (method: string) => {
|
||||
if (method === 'Input.dispatchMouseEvent') {
|
||||
currentUrl = 'https://example.com/tests/IPIP-BFFM/1.php'
|
||||
throw new Error('Execution context was destroyed, most likely because of a navigation.')
|
||||
}
|
||||
return {}
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click_at', { x: 100, y: 200 })
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
dispatched: true,
|
||||
navigatedDuringDispatch: true,
|
||||
effectObserved: true,
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it('ignores a dialog that was already open before the click', async () => {
|
||||
const contents = await openPage()
|
||||
let actionReads = 0
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
@@ -1681,13 +1778,52 @@ describe('credential protection', () => {
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 0 })
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: { effectObserved: true, obstructedAfterNavigation: false, dialogs: ['Search'] },
|
||||
})
|
||||
})
|
||||
|
||||
it('reports navigation obstructed by a dialog that opened with it', async () => {
|
||||
const contents = await openPage()
|
||||
let actionReads = 0
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'clickElement')) {
|
||||
return Promise.resolve({ dispatched: false, x: 24, y: 48, element: 'Search result' })
|
||||
}
|
||||
if (isPageCall(expression, 'readActiveElementState')) return Promise.resolve({})
|
||||
if (isPageCall(expression, 'readPageActionState')) {
|
||||
actionReads++
|
||||
return Promise.resolve(
|
||||
actionReads === 1
|
||||
? {
|
||||
url: 'https://example.com/search',
|
||||
title: 'Search',
|
||||
focus: 'body',
|
||||
mutationRevision: 0,
|
||||
dialogs: [],
|
||||
scroll: [0],
|
||||
}
|
||||
: {
|
||||
url: 'https://example.com/channel/eng-bugs',
|
||||
title: 'eng-bugs',
|
||||
focus: 'body',
|
||||
mutationRevision: 1,
|
||||
dialogs: ['Open in the Slack app?'],
|
||||
scroll: [0],
|
||||
}
|
||||
)
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 0 })
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
effectObserved: true,
|
||||
obstructedAfterNavigation: true,
|
||||
dialogs: ['Search'],
|
||||
note: expect.stringContaining('dialog is still open'),
|
||||
note: expect.stringContaining('Open in the Slack app?'),
|
||||
},
|
||||
})
|
||||
})
|
||||
@@ -1732,4 +1868,355 @@ describe('credential protection', () => {
|
||||
'Element ids are not valid in this tab. Call browser_snapshot and use an id from that result.',
|
||||
})
|
||||
})
|
||||
|
||||
/** Fires every instrumentation listener registered for a WebContents event. */
|
||||
function emitContentsEvent(
|
||||
contents: Awaited<ReturnType<typeof openPage>>,
|
||||
event: string,
|
||||
...args: unknown[]
|
||||
): void {
|
||||
for (const [name, listener] of vi.mocked(contents.on).mock.calls) {
|
||||
if (name === event) (listener as (...listenerArgs: unknown[]) => void)({}, ...args)
|
||||
}
|
||||
}
|
||||
|
||||
it('keeps element ids across a same-document (SPA) navigation', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {})
|
||||
|
||||
emitContentsEvent(contents, 'did-navigate-in-page')
|
||||
const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 0 })
|
||||
|
||||
expect(result).toMatchObject({ ok: true, result: { dispatched: true } })
|
||||
})
|
||||
|
||||
it('still invalidates element ids on a cross-document navigation', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {})
|
||||
|
||||
emitContentsEvent(contents, 'did-navigate')
|
||||
const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 0 })
|
||||
|
||||
expect(result).toEqual({
|
||||
ok: false,
|
||||
error:
|
||||
'Element ids are not valid in this tab. Call browser_snapshot and use an id from that result.',
|
||||
})
|
||||
})
|
||||
|
||||
it('tolerates same-document URL churn during a keypress', async () => {
|
||||
const contents = await openPage()
|
||||
let urlReads = 0
|
||||
vi.mocked(contents.getURL).mockImplementation(() =>
|
||||
++urlReads === 1 ? 'https://example.com/channel-a' : 'https://example.com/channel-b'
|
||||
)
|
||||
respondWith(contents, {
|
||||
activeElementSecrecy: 'safe',
|
||||
readActiveElementState: {},
|
||||
readPageActionState: {
|
||||
url: 'https://example.com/channel-a',
|
||||
title: 'Example',
|
||||
focus: 'body',
|
||||
mutationRevision: 0,
|
||||
dialogs: [],
|
||||
scroll: [0],
|
||||
},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_press_key', { key: 'Escape' })
|
||||
|
||||
expect(result.ok).toBe(true)
|
||||
expect(cdpCalls(contents, 'Input.dispatchKeyEvent').length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('aborts a keypress when a cross-document navigation lands mid-flight', async () => {
|
||||
const contents = await openPage()
|
||||
let navigated = false
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'activeElementSecrecy')) return Promise.resolve('safe')
|
||||
if (isPageCall(expression, 'readActiveElementState')) return Promise.resolve({})
|
||||
if (isPageCall(expression, 'readPageActionState')) {
|
||||
if (!navigated) {
|
||||
navigated = true
|
||||
emitContentsEvent(contents, 'did-navigate')
|
||||
}
|
||||
return Promise.resolve({
|
||||
url: 'https://example.com/login',
|
||||
title: 'Example',
|
||||
focus: 'body',
|
||||
mutationRevision: 0,
|
||||
dialogs: [],
|
||||
scroll: [0],
|
||||
})
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_press_key', { key: 'Escape' })
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/active tab or page changed/)
|
||||
expect(cdpCalls(contents, 'Input.dispatchKeyEvent')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('waits for a late-mounting editor before typing', async () => {
|
||||
const contents = await openPage()
|
||||
let focusReads = 0
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'focusElementForTyping')) {
|
||||
focusReads++
|
||||
return Promise.resolve(
|
||||
focusReads === 1
|
||||
? { error: 'not-editable' }
|
||||
: { focused: true, kind: 'contenteditable', x: 24, y: 48 }
|
||||
)
|
||||
}
|
||||
if (isPageCall(expression, 'activeElementSecrecy')) return Promise.resolve('safe')
|
||||
if (isPageCall(expression, 'readActiveElementState')) {
|
||||
return Promise.resolve({ activeElement: 'div', valueLength: 5 })
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_type', {
|
||||
elementId: 0,
|
||||
text: 'hello',
|
||||
})
|
||||
|
||||
expect(result.ok).toBe(true)
|
||||
expect(focusReads).toBeGreaterThan(1)
|
||||
expect(cdpCalls(contents, 'Input.insertText')).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('reprobes a transiently stale click target before giving up', async () => {
|
||||
const contents = await openPage()
|
||||
let clickReads = 0
|
||||
vi.mocked(contents.executeJavaScript).mockImplementation((expression: string) => {
|
||||
if (isPageCall(expression, 'clickElement')) {
|
||||
clickReads++
|
||||
return Promise.resolve(
|
||||
clickReads === 1
|
||||
? { error: 'stale' }
|
||||
: { dispatched: false, x: 24, y: 48, element: 'Channel row' }
|
||||
)
|
||||
}
|
||||
if (isPageCall(expression, 'readActiveElementState')) return Promise.resolve({})
|
||||
if (isPageCall(expression, 'readPageActionState')) return Promise.resolve({})
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click', { elementId: 0 })
|
||||
|
||||
expect(result).toMatchObject({ ok: true, result: { dispatched: true } })
|
||||
expect(clickReads).toBeGreaterThan(1)
|
||||
})
|
||||
|
||||
it('clicks a coordinate point with native input and reports the target', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: {
|
||||
found: true,
|
||||
element: 'button "Send"',
|
||||
editable: false,
|
||||
secret: false,
|
||||
fileInput: false,
|
||||
cursor: 'pointer',
|
||||
},
|
||||
readActiveElementState: {},
|
||||
readPageActionState: {},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click_at', { x: 120, y: 240 })
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
dispatched: true,
|
||||
trusted: true,
|
||||
clickedAt: { x: 120, y: 240 },
|
||||
target: 'button "Send"',
|
||||
},
|
||||
})
|
||||
const presses = cdpCalls(contents, 'Input.dispatchMouseEvent').filter(
|
||||
([, event]) => (event as { type?: string }).type === 'mousePressed'
|
||||
)
|
||||
expect(presses).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('double-clicks a coordinate point as a rising clickCount sequence', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: { found: true, element: 'canvas', editable: false },
|
||||
readActiveElementState: {},
|
||||
readPageActionState: {},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click_at', {
|
||||
x: 10,
|
||||
y: 20,
|
||||
clickCount: 2,
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({ ok: true, result: { clickCount: 2 } })
|
||||
const counts = cdpCalls(contents, 'Input.dispatchMouseEvent')
|
||||
.filter(([, event]) => (event as { type?: string }).type === 'mousePressed')
|
||||
.map(([, event]) => (event as { clickCount?: number }).clickCount)
|
||||
expect(counts).toEqual([1, 2])
|
||||
})
|
||||
|
||||
it('refuses a coordinate click on a file input', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: { found: true, element: 'input', fileInput: true },
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click_at', { x: 5, y: 5 })
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/file input/)
|
||||
expect(cdpCalls(contents, 'Input.dispatchMouseEvent')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('rejects a coordinate click outside the viewport with mapping guidance', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: { error: 'outside-viewport' },
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_click_at', { x: 9999, y: 5 })
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/divide image pixels by its scale/)
|
||||
})
|
||||
|
||||
it('inserts text into the focused editable at the caret', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
activeElementSecrecy: 'safe',
|
||||
describeFocusedEditable: { editable: true, kind: 'contenteditable' },
|
||||
readActiveElementState: { activeElement: 'div', valueLength: 12 },
|
||||
readPageActionState: {},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_insert_text', {
|
||||
text: 'hello world',
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: { dispatched: true, trusted: true, kind: 'contenteditable', insertedChars: 11 },
|
||||
})
|
||||
expect(cdpCalls(contents, 'Input.insertText')).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('refuses insertion when nothing editable holds focus', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
activeElementSecrecy: 'safe',
|
||||
describeFocusedEditable: { editable: false, reason: 'none' },
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_insert_text', { text: 'x' })
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/No element is focused/)
|
||||
expect(cdpCalls(contents, 'Input.insertText')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('refuses insertion while a password field holds focus', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, { activeElementSecrecy: 'secret' })
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_insert_text', { text: 'x' })
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/Refusing to act on a password field/)
|
||||
expect(cdpCalls(contents, 'Input.insertText')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('drags between coordinate points through the trusted pointer pipeline', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: { found: true, element: 'div "Card"' },
|
||||
readActiveElementState: {},
|
||||
readPageActionState: {},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_drag', {
|
||||
fromX: 40,
|
||||
fromY: 50,
|
||||
toX: 200,
|
||||
toY: 260,
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: { dispatched: true, trusted: true, from: { x: 40, y: 50 }, to: { x: 200, y: 260 } },
|
||||
})
|
||||
const events = cdpCalls(contents, 'Input.dispatchMouseEvent').map(
|
||||
([, event]) => (event as { type?: string }).type
|
||||
)
|
||||
expect(events[0]).toBe('mouseMoved')
|
||||
expect(events).toContain('mousePressed')
|
||||
expect(events[events.length - 1]).toBe('mouseReleased')
|
||||
expect(cdpCalls(contents, 'Input.setInterceptDrags').length).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('drags from a snapshot element to a coordinate target', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
clickElement: { dispatched: false, x: 24, y: 48, element: 'Card "Ship it"' },
|
||||
describePointTarget: { found: true, element: 'section "Done"' },
|
||||
readActiveElementState: {},
|
||||
readPageActionState: {},
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_drag', {
|
||||
fromElementId: 0,
|
||||
toX: 300,
|
||||
toY: 60,
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
result: { dispatched: true, from: { x: 24, y: 48, element: 'Card "Ship it"' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects a drag whose endpoints are the same point', async () => {
|
||||
const contents = await openPage()
|
||||
respondWith(contents, {
|
||||
describePointTarget: { found: true, element: 'div' },
|
||||
})
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_drag', {
|
||||
fromX: 10,
|
||||
fromY: 10,
|
||||
toX: 10,
|
||||
toY: 10,
|
||||
})
|
||||
|
||||
expect(result.ok).toBe(false)
|
||||
expect(result.error).toMatch(/same point/)
|
||||
})
|
||||
|
||||
it('returns the screenshot scale for coordinate mapping', async () => {
|
||||
const contents = await openPage()
|
||||
vi.mocked(contents.debugger.sendCommand).mockImplementation((method: string) => {
|
||||
if (method === 'Page.getLayoutMetrics') {
|
||||
return Promise.resolve({
|
||||
cssLayoutViewport: { clientWidth: 2048, clientHeight: 1024 },
|
||||
})
|
||||
}
|
||||
if (method === 'Page.captureScreenshot') {
|
||||
return Promise.resolve({ data: 'c2lt' })
|
||||
}
|
||||
return Promise.resolve(undefined)
|
||||
})
|
||||
respondWith(contents, { getViewportInfo: { width: 2048, height: 1024 } })
|
||||
|
||||
const result = await driver.executeTool('chat-test', 'browser_screenshot', {})
|
||||
|
||||
expect(result).toMatchObject({ ok: true, result: { scale: 0.5 } })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -46,6 +46,8 @@ import {
|
||||
activeElementSecrecy,
|
||||
clickElement,
|
||||
collectSnapshot,
|
||||
describeFocusedEditable,
|
||||
describePointTarget,
|
||||
focusElementForTyping,
|
||||
getViewportInfo,
|
||||
hoverElement,
|
||||
@@ -174,6 +176,29 @@ function invalidateSnapshot(state = driverScopeState()): void {
|
||||
state.snapshotCaptureEpoch++
|
||||
}
|
||||
|
||||
/**
|
||||
* Cross-document navigation counters, bumped by tab instrumentation. A live
|
||||
* SPA rewrites its URL with pushState/replaceState between a snapshot and the
|
||||
* input dispatched from it, so URL equality cannot distinguish "the document
|
||||
* the model saw is gone" from routine same-document churn — only a real
|
||||
* document swap increments these.
|
||||
*/
|
||||
const crossDocumentNavigations = new WeakMap<WebContents, number>()
|
||||
const crossDocumentFrameNavigations = new WeakMap<WebContents, Map<string, number>>()
|
||||
|
||||
function navigationEpoch(contents: WebContents): number {
|
||||
return crossDocumentNavigations.get(contents) ?? 0
|
||||
}
|
||||
|
||||
function frameEpochKey(processId: number, routingId: number): string {
|
||||
return `${processId}:${routingId}`
|
||||
}
|
||||
|
||||
function frameNavigationEpoch(contents: WebContents, frame: WebFrameMain): number {
|
||||
const key = frameEpochKey(frame.processId, frame.routingId)
|
||||
return crossDocumentFrameNavigations.get(contents)?.get(key) ?? 0
|
||||
}
|
||||
|
||||
const driverScopeStates = new Map<string, DriverScopeState>()
|
||||
const driverScopeAliases = new Map<string, string>()
|
||||
const CANCELLED_TOOL_TTL_MS = 5 * 60_000
|
||||
@@ -312,6 +337,7 @@ function instrumentTab(contents: WebContents): void {
|
||||
contents.on(
|
||||
'did-navigate',
|
||||
inScope(() => {
|
||||
crossDocumentNavigations.set(contents, navigationEpoch(contents) + 1)
|
||||
if (session.automationTab()?.view.webContents === contents) {
|
||||
invalidateSnapshot()
|
||||
}
|
||||
@@ -320,6 +346,21 @@ function instrumentTab(contents: WebContents): void {
|
||||
pushTabsState()
|
||||
})
|
||||
)
|
||||
contents.on(
|
||||
'did-frame-navigate',
|
||||
inScope(
|
||||
(_event, _url, _httpResponseCode, _httpStatusText, _isMainFrame, processId, routingId) => {
|
||||
const frames = crossDocumentFrameNavigations.get(contents) ?? new Map<string, number>()
|
||||
const key = frameEpochKey(processId, routingId)
|
||||
frames.set(key, (frames.get(key) ?? 0) + 1)
|
||||
crossDocumentFrameNavigations.set(contents, frames)
|
||||
}
|
||||
)
|
||||
)
|
||||
// Same-document navigation deliberately does NOT invalidate the snapshot: a
|
||||
// live SPA (Slack) pushStates continuously, and invalidating here made every
|
||||
// ref die between snapshot and act. Element-level identity checks and the
|
||||
// in-page ref resolver keep individual actions honest instead.
|
||||
for (const event of [
|
||||
'did-navigate-in-page',
|
||||
'page-title-updated',
|
||||
@@ -330,12 +371,6 @@ function instrumentTab(contents: WebContents): void {
|
||||
contents.on(
|
||||
event as 'did-navigate',
|
||||
inScope(() => {
|
||||
if (
|
||||
event === 'did-navigate-in-page' &&
|
||||
session.automationTab()?.view.webContents === contents
|
||||
) {
|
||||
invalidateSnapshot()
|
||||
}
|
||||
pushPageState(contents)
|
||||
pushTabsState()
|
||||
})
|
||||
@@ -824,9 +859,12 @@ function unwrapPageResult(result: unknown): unknown {
|
||||
if (isRecordLike(result) && 'error' in result) {
|
||||
const code = (result as { error: string }).error
|
||||
if (code === 'stale') {
|
||||
const reason =
|
||||
isRecordLike(result) && typeof result.reason === 'string' && result.reason
|
||||
? result.reason
|
||||
: 'the page changed since the last snapshot'
|
||||
throw new ToolError(
|
||||
'That element id is stale (the page changed since the last snapshot). ' +
|
||||
'Call browser_snapshot again and use a fresh id.'
|
||||
`That element id is stale (${reason}). Call browser_snapshot again and use a fresh id.`
|
||||
)
|
||||
}
|
||||
if (code === 'password') {
|
||||
@@ -851,17 +889,41 @@ function unwrapPageResult(result: unknown): unknown {
|
||||
`That element is covered by ${blocker}. Close or move the overlay, then take a fresh browser_snapshot.`
|
||||
)
|
||||
}
|
||||
if (code === 'nested-control') {
|
||||
const blocker = String((result as { blocker?: unknown }).blocker || 'a nested control')
|
||||
throw new ToolError(
|
||||
`The point you targeted lands on ${blocker}, which is its own control inside that element — nothing is covering it. Take a fresh browser_snapshot and use the id of the control you actually want.`
|
||||
)
|
||||
}
|
||||
if (code === 'suggestions-open') {
|
||||
throw new ToolError(
|
||||
'That editable field is already focused and covered by its own suggestions popup. Use browser_type on the same element; do not dismiss the popup first.'
|
||||
)
|
||||
}
|
||||
if (code === 'not-editable') {
|
||||
throw new ToolError('That element is not a text input — pick an editable element.')
|
||||
const tag = isRecordLike(result) ? String(result.elementTag ?? '') : ''
|
||||
const role = isRecordLike(result) ? String(result.elementRole ?? '') : ''
|
||||
const described = [tag ? `<${tag}>` : '', role ? `role="${role}"` : '']
|
||||
.filter(Boolean)
|
||||
.join(' ')
|
||||
throw new ToolError(
|
||||
`That element is not a text input${described ? ` (it is ${described})` : ''} — take a fresh browser_snapshot and target the editable field itself.`
|
||||
)
|
||||
}
|
||||
if (code === 'outside-viewport') {
|
||||
throw new ToolError(
|
||||
'That point is outside the visible viewport. Coordinates are CSS pixels within the current viewport — when reading them off a browser_screenshot, divide image pixels by its scale, and scroll the target into view first.'
|
||||
)
|
||||
}
|
||||
if (code === 'ambiguous-editable') {
|
||||
const candidates =
|
||||
isRecordLike(result) && Array.isArray(result.candidates)
|
||||
? result.candidates.map(String).filter(Boolean)
|
||||
: []
|
||||
throw new ToolError(
|
||||
'That composite control contains multiple editable fields. Take a fresh browser_snapshot and target the exact field.'
|
||||
`That composite control contains multiple editable fields${
|
||||
candidates.length > 0 ? ` (${candidates.join(', ')})` : ''
|
||||
}. Take a fresh browser_snapshot and target the exact field.`
|
||||
)
|
||||
}
|
||||
if (code === 'different') {
|
||||
@@ -1037,14 +1099,53 @@ function pageTargetForElement(contents: WebContents, elementId: number): PageExe
|
||||
return target
|
||||
}
|
||||
|
||||
function assertActiveContents(contents: WebContents, expectedUrl?: string): void {
|
||||
// The user's claim on the visible tab (visibleTabUserSelected) deliberately
|
||||
// does NOT gate input here: the user clicking or typing in the panel must
|
||||
// never leave the agent unable to act — hand-off is cooperative via
|
||||
// browser_request_takeover, whose serialized tool slot already keeps agent
|
||||
// input out while the user drives.
|
||||
/**
|
||||
* The click-that-navigates race: a press submits a form, the navigation tears
|
||||
* the origin document down, and everything AFTER the dispatch — the CDP call's
|
||||
* own completion, the confirmation probe, the postcondition reads — fails
|
||||
* against a destroyed context. Reporting that as a failed click is wrong twice
|
||||
* over: the press reached the page, and the navigation IS the strongest
|
||||
* possible evidence it worked. This detects that case at the driver level,
|
||||
* where the navigation epoch and URL survive the renderer teardown.
|
||||
*
|
||||
* Returns a success result when the page provably navigated since dispatch
|
||||
* began, and null otherwise (caller keeps its original failure).
|
||||
*/
|
||||
function navigationRescue(
|
||||
contents: WebContents,
|
||||
epochAtDispatch: number,
|
||||
urlAtDispatch: string,
|
||||
base: { trusted: boolean; activation: string }
|
||||
): Record<string, unknown> | null {
|
||||
if (contents.isDestroyed()) return null
|
||||
const navigated =
|
||||
navigationEpoch(contents) !== epochAtDispatch || contents.getURL() !== urlAtDispatch
|
||||
if (!navigated) return null
|
||||
return {
|
||||
dispatched: true,
|
||||
trusted: base.trusted,
|
||||
activation: base.activation,
|
||||
effectObserved: true,
|
||||
possibleEffectObserved: true,
|
||||
navigatedDuringDispatch: true,
|
||||
effect: { urlChanged: true },
|
||||
dialogs: [],
|
||||
note: 'The click triggered a page navigation, and the origin document was torn down before its result could be read — that teardown is why no postconditions are reported, not a failure. Take a fresh browser_snapshot of the new page.',
|
||||
}
|
||||
}
|
||||
|
||||
function assertActiveContents(contents: WebContents, expectedNavigationEpoch?: number): void {
|
||||
const active = session.automationTab()
|
||||
if (
|
||||
session.automationTabClaimedByUser() ||
|
||||
!active ||
|
||||
active.view.webContents !== contents ||
|
||||
contents.isDestroyed() ||
|
||||
(expectedUrl !== undefined && contents.getURL() !== expectedUrl)
|
||||
(expectedNavigationEpoch !== undefined && navigationEpoch(contents) !== expectedNavigationEpoch)
|
||||
) {
|
||||
throw new ToolError('The active tab or page changed before input could be dispatched.')
|
||||
}
|
||||
@@ -1089,7 +1190,7 @@ function sameWebFrame(left: WebFrameMain, right: WebFrameMain): boolean {
|
||||
function assertFocusedTargetUnchanged(
|
||||
contents: WebContents,
|
||||
target: PageExecutionTarget,
|
||||
expectedFrameUrl?: string
|
||||
expectedFrameNavigationEpoch?: number
|
||||
): void {
|
||||
const focused = focusedPageTarget(contents)
|
||||
const unchanged =
|
||||
@@ -1106,7 +1207,8 @@ function assertFocusedTargetUnchanged(
|
||||
if (
|
||||
frame.isDestroyed() ||
|
||||
frame.detached ||
|
||||
(expectedFrameUrl !== undefined && frame.url !== expectedFrameUrl) ||
|
||||
(expectedFrameNavigationEpoch !== undefined &&
|
||||
frameNavigationEpoch(contents, frame) !== expectedFrameNavigationEpoch) ||
|
||||
!contents.mainFrame.framesInSubtree.some((candidate) => sameWebFrame(candidate, frame))
|
||||
) {
|
||||
throw new ToolError(
|
||||
@@ -1154,16 +1256,25 @@ async function prepareTypingSurface(
|
||||
target: PageExecutionTarget,
|
||||
elementId: number,
|
||||
moveFocus: boolean,
|
||||
executionDeadline?: number
|
||||
executionDeadline?: number,
|
||||
settleGraceMs = 0
|
||||
): Promise<Record<string, unknown>> {
|
||||
const prepared = unwrapPageResult(
|
||||
await execInPage(
|
||||
target,
|
||||
focusElementForTyping,
|
||||
[elementId, moveFocus],
|
||||
false,
|
||||
executionDeadline
|
||||
)
|
||||
settleGraceMs > 0
|
||||
? await execInPageWithSettleGrace(
|
||||
target,
|
||||
focusElementForTyping,
|
||||
[elementId, moveFocus],
|
||||
executionDeadline,
|
||||
settleGraceMs
|
||||
)
|
||||
: await execInPage(
|
||||
target,
|
||||
focusElementForTyping,
|
||||
[elementId, moveFocus],
|
||||
false,
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
if (
|
||||
!isRecordLike(prepared) ||
|
||||
@@ -1178,6 +1289,43 @@ async function prepareTypingSurface(
|
||||
return prepared
|
||||
}
|
||||
|
||||
const TRANSIENT_PROBE_ERRORS = new Set(['stale', 'not-visible', 'not-editable'])
|
||||
const SETTLE_GRACE_MS = 1_000
|
||||
const SETTLE_PROBE_INTERVAL_MS = 250
|
||||
|
||||
/**
|
||||
* First-touch page probe with a short settle grace. A live view re-rendering
|
||||
* under the agent (Slack's virtualized sidebar, its late-mounting composer)
|
||||
* can transiently report an element as stale, invisible, or not yet editable
|
||||
* while its replacement mounts one frame later. These call sites run before
|
||||
* anything is dispatched, so a bounded reprobe safely turns that churn into a
|
||||
* recovery instead of a dead ref.
|
||||
*/
|
||||
async function execInPageWithSettleGrace<Args extends unknown[], Result>(
|
||||
target: PageExecutionTarget,
|
||||
fn: (...args: Args) => Result,
|
||||
args: Args,
|
||||
executionDeadline?: number,
|
||||
graceMs = SETTLE_GRACE_MS
|
||||
): Promise<Result> {
|
||||
const graceDeadline = Date.now() + graceMs
|
||||
for (;;) {
|
||||
const result = await execInPage(target, fn, args, false, executionDeadline)
|
||||
const code =
|
||||
isRecordLike(result) && typeof result.error === 'string' ? String(result.error) : null
|
||||
if (
|
||||
code === null ||
|
||||
!TRANSIENT_PROBE_ERRORS.has(code) ||
|
||||
Date.now() + SETTLE_PROBE_INTERVAL_MS > graceDeadline ||
|
||||
(executionDeadline !== undefined &&
|
||||
Date.now() + SETTLE_PROBE_INTERVAL_MS >= executionDeadline)
|
||||
) {
|
||||
return result
|
||||
}
|
||||
await sleep(SETTLE_PROBE_INTERVAL_MS)
|
||||
}
|
||||
}
|
||||
|
||||
function pointFromPrepared(prepared: Record<string, unknown>): { x: number; y: number } {
|
||||
return { x: prepared.x as number, y: prepared.y as number }
|
||||
}
|
||||
@@ -1194,6 +1342,29 @@ async function pageActionState(
|
||||
return toRecord(state)
|
||||
}
|
||||
|
||||
/**
|
||||
* Which signals each tool accepts as proof its action reached the page.
|
||||
*
|
||||
* The tools deliberately do NOT share one predicate — the differences are real,
|
||||
* and flattening them would make every tool wrong in a different direction:
|
||||
*
|
||||
* - `browser_drag` is the only tool that trusts `domChanged`, because a drop
|
||||
* that reorders a list may change nothing else observable. Everywhere else
|
||||
* background churn (Slack, Gmail) would forge success for an ignored action.
|
||||
* - `browser_hover` ignores `fieldChanged` and `focusChanged`: hovering does not
|
||||
* type or focus, so those would only ever be someone else's effect.
|
||||
* - `browser_click` / `browser_click_at` count `focusChanged` only when the
|
||||
* target was editable — otherwise a click that merely moved focus reads as
|
||||
* success.
|
||||
* - `targetChanged` requires a `targetState`, which `pageActionState` captures
|
||||
* only when given an `elementId`. Tools without one (click_at, insert_text,
|
||||
* drag, press_key) cannot use it; listing it there read as coverage they did
|
||||
* not have, and it was silently always false.
|
||||
*
|
||||
* What IS shared is this function: every signal is computed here once, so a
|
||||
* tool's formula is a statement about which evidence it trusts, not a private
|
||||
* re-derivation of what changed.
|
||||
*/
|
||||
function pageEffect(
|
||||
beforePage: Record<string, unknown>,
|
||||
afterPage: Record<string, unknown>,
|
||||
@@ -1522,7 +1693,7 @@ async function captureSnapshot(contents: WebContents, notAfter?: number): Promis
|
||||
invalidateSnapshot(state)
|
||||
const captureEpoch = state.snapshotCaptureEpoch
|
||||
const capturedTabId = tab.id
|
||||
const capturedUrl = contents.getURL()
|
||||
const capturedNavigationEpoch = navigationEpoch(contents)
|
||||
const targets = new Map<number, PageExecutionTarget>()
|
||||
const targetLineIndexes = new Map<number, number>()
|
||||
const stillCurrent = (): boolean => {
|
||||
@@ -1532,7 +1703,9 @@ async function captureSnapshot(contents: WebContents, notAfter?: number): Promis
|
||||
active?.id === capturedTabId &&
|
||||
active.view.webContents === contents &&
|
||||
!contents.isDestroyed() &&
|
||||
contents.getURL() === capturedUrl
|
||||
// Same-document URL drift (SPA pushState) must not abort the capture;
|
||||
// only a cross-document navigation makes the collected refs meaningless.
|
||||
navigationEpoch(contents) === capturedNavigationEpoch
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1817,7 +1990,8 @@ async function executeToolInner(
|
||||
}
|
||||
}
|
||||
assertCurrentExecution()
|
||||
const tab = session.addAutomationTab()
|
||||
// The agent chose to open this page to work in, so the panel follows it.
|
||||
const tab = session.addAutomationTab({ reveal: true })
|
||||
const contents = tab.view.webContents
|
||||
if (url) {
|
||||
assertCurrentExecution()
|
||||
@@ -1920,19 +2094,21 @@ async function executeToolInner(
|
||||
|
||||
case 'browser_screenshot': {
|
||||
const contents = session.requireAutomationTab().view.webContents
|
||||
const dataUrl = await cdp.captureScreenshot(contents).catch(() => null)
|
||||
if (dataUrl === null) {
|
||||
const shot = await cdp.captureScreenshot(contents).catch(() => null)
|
||||
if (shot === null) {
|
||||
throw new ToolError(
|
||||
'Could not capture the page. Use browser_snapshot or browser_read_text instead.'
|
||||
)
|
||||
}
|
||||
if (dataUrl.length > 8_000_000) {
|
||||
if (shot.dataUrl.length > 8_000_000) {
|
||||
throw new ToolError(
|
||||
'The screenshot result was too large to return safely. Use browser_snapshot or browser_read_text instead.'
|
||||
)
|
||||
}
|
||||
const viewport = await execInPage(contents, getViewportInfo, []).catch(() => null)
|
||||
return { dataUrl, viewport }
|
||||
// scale maps image pixels back to CSS viewport pixels for the
|
||||
// coordinate tools: cssX = imageX / scale.
|
||||
return { dataUrl: shot.dataUrl, viewport, scale: shot.scale }
|
||||
}
|
||||
|
||||
case 'browser_extract': {
|
||||
@@ -1957,11 +2133,10 @@ async function executeToolInner(
|
||||
if (!targetFrame) {
|
||||
assertCurrentExecution()
|
||||
const first = unwrapPageResult(
|
||||
await execInPage(
|
||||
await execInPageWithSettleGrace(
|
||||
target,
|
||||
clickElement,
|
||||
[elementId, false, false],
|
||||
false,
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
@@ -2010,7 +2185,12 @@ async function executeToolInner(
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
const focused = unwrapPageResult(
|
||||
await execInPage(target, clickElement, [elementId, false, true], false, executionDeadline)
|
||||
await execInPageWithSettleGrace(
|
||||
target,
|
||||
clickElement,
|
||||
[elementId, false, true],
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
if (!isRecordLike(focused)) throw new ToolError('Could not resolve that click target.')
|
||||
prepared = focused
|
||||
@@ -2058,6 +2238,8 @@ async function executeToolInner(
|
||||
}
|
||||
|
||||
const observeTopPage = targetFrame !== null || Number(prepared.frameDepth || 0) > 0
|
||||
const epochAtDispatch = navigationEpoch(contents)
|
||||
const urlAtDispatch = contents.getURL()
|
||||
const beforePage = await pageActionState(target, true, elementId)
|
||||
const beforeElement = await activeElementState(target)
|
||||
const beforeTopPage = observeTopPage ? await pageActionState(contents, true) : beforePage
|
||||
@@ -2102,6 +2284,11 @@ async function executeToolInner(
|
||||
trusted = true
|
||||
activation = 'native-pointer'
|
||||
} catch (error) {
|
||||
const rescued = navigationRescue(contents, epochAtDispatch, urlAtDispatch, {
|
||||
trusted: true,
|
||||
activation: 'native-pointer',
|
||||
})
|
||||
if (rescued) return rescued
|
||||
throw new ToolError(
|
||||
`Native click dispatch failed (${getErrorMessage(error)}). The action was not retried because a partial pointer press may already have reached the page. Take a fresh snapshot before continuing.`
|
||||
)
|
||||
@@ -2138,6 +2325,11 @@ async function executeToolInner(
|
||||
activation = 'native-pointer'
|
||||
prepared = finalSurface
|
||||
} catch (error) {
|
||||
const rescued = navigationRescue(contents, epochAtDispatch, urlAtDispatch, {
|
||||
trusted: true,
|
||||
activation: 'native-pointer',
|
||||
})
|
||||
if (rescued) return rescued
|
||||
throw new ToolError(
|
||||
`Native framed click dispatch failed (${getErrorMessage(error)}). The action was not retried because a partial pointer press may already have reached the page. Take a fresh snapshot before continuing.`
|
||||
)
|
||||
@@ -2183,16 +2375,33 @@ async function executeToolInner(
|
||||
} else {
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
const synthetic = unwrapPageResult(
|
||||
await execInPage(
|
||||
target,
|
||||
clickElement,
|
||||
[elementId, true, false],
|
||||
true,
|
||||
executionDeadline
|
||||
let synthetic: unknown
|
||||
try {
|
||||
synthetic = unwrapPageResult(
|
||||
await execInPage(
|
||||
target,
|
||||
clickElement,
|
||||
[elementId, true, false],
|
||||
true,
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
)
|
||||
} catch (error) {
|
||||
// A synchronous form-submit navigation destroys the context
|
||||
// before the dispatch's return value crosses the bridge.
|
||||
const rescued = navigationRescue(contents, epochAtDispatch, urlAtDispatch, {
|
||||
trusted: false,
|
||||
activation: 'synthetic',
|
||||
})
|
||||
if (rescued) return rescued
|
||||
throw error
|
||||
}
|
||||
if (!isRecordLike(synthetic) || synthetic.dispatched !== true) {
|
||||
const rescued = navigationRescue(contents, epochAtDispatch, urlAtDispatch, {
|
||||
trusted: false,
|
||||
activation: 'synthetic',
|
||||
})
|
||||
if (rescued) return rescued
|
||||
throw new ToolError('The frame did not confirm synthetic click dispatch.')
|
||||
}
|
||||
}
|
||||
@@ -2221,6 +2430,13 @@ async function executeToolInner(
|
||||
)),
|
||||
tabChanged,
|
||||
}
|
||||
// Page-read URLs go blind when the click navigated and the origin
|
||||
// context died: the after-state reads {} and urlChanged computes false
|
||||
// for a maximally successful click. The driver-level epoch/URL survive
|
||||
// the teardown, so fold them in.
|
||||
const navigatedByDriver =
|
||||
!contents.isDestroyed() &&
|
||||
(navigationEpoch(contents) !== epochAtDispatch || contents.getURL() !== urlAtDispatch)
|
||||
const clickEffectObserved =
|
||||
observation.effect.urlChanged ||
|
||||
observation.effect.dialogChanged ||
|
||||
@@ -2233,9 +2449,13 @@ async function executeToolInner(
|
||||
topObservation.effect.dialogChanged ||
|
||||
topObservation.effect.popupChanged ||
|
||||
topObservation.effect.targetChanged)
|
||||
const effectObserved = clickEffectObserved || topClickEffectObserved || tabChanged
|
||||
const effectObserved =
|
||||
clickEffectObserved || topClickEffectObserved || tabChanged || navigatedByDriver
|
||||
const possibleEffectObserved =
|
||||
observation.possibleEffectObserved || topObservation.possibleEffectObserved || tabChanged
|
||||
observation.possibleEffectObserved ||
|
||||
topObservation.possibleEffectObserved ||
|
||||
tabChanged ||
|
||||
navigatedByDriver
|
||||
const dialogs = Array.from(
|
||||
new Set([
|
||||
...(Array.isArray(afterPage.dialogs) ? afterPage.dialogs.map(String) : []),
|
||||
@@ -2243,11 +2463,25 @@ async function executeToolInner(
|
||||
])
|
||||
)
|
||||
const navigated =
|
||||
observation.effect.urlChanged || topObservation.effect.urlChanged || tabChanged
|
||||
const obstructedAfterNavigation = navigated && dialogs.length > 0
|
||||
observation.effect.urlChanged ||
|
||||
topObservation.effect.urlChanged ||
|
||||
tabChanged ||
|
||||
navigatedByDriver
|
||||
// Only a dialog that ARRIVED with the navigation obstructs it. Comparing
|
||||
// against the union of what was already open stops the false positive
|
||||
// that fires on every SPA route change under a persistent `role=dialog`
|
||||
// (a cookie banner, a side drawer, an emoji picker) — those are not
|
||||
// blocking anything, and reporting them made successful clicks read as
|
||||
// failures.
|
||||
const dialogsBefore = new Set([
|
||||
...(Array.isArray(beforePage.dialogs) ? beforePage.dialogs.map(String) : []),
|
||||
...(Array.isArray(beforeTopPage.dialogs) ? beforeTopPage.dialogs.map(String) : []),
|
||||
])
|
||||
const newDialogs = dialogs.filter((dialog) => !dialogsBefore.has(dialog))
|
||||
const obstructedAfterNavigation = navigated && newDialogs.length > 0
|
||||
const notes: string[] = []
|
||||
if (obstructedAfterNavigation) {
|
||||
notes.push('The page navigated, but a dialog is still open above it.')
|
||||
notes.push(`The page navigated, but a dialog opened above it (${newDialogs.join(', ')}).`)
|
||||
}
|
||||
if (!effectObserved) {
|
||||
notes.push(
|
||||
@@ -2292,7 +2526,16 @@ async function executeToolInner(
|
||||
// synthetic value-setter when CDP is unavailable.
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
const initialSurface = await prepareTypingSurface(target, elementId, true, executionDeadline)
|
||||
// The settle grace covers a composer whose real contenteditable mounts a
|
||||
// beat after the surrounding view renders (Slack); nothing has been
|
||||
// dispatched yet, so reprobing is safe.
|
||||
const initialSurface = await prepareTypingSurface(
|
||||
target,
|
||||
elementId,
|
||||
true,
|
||||
executionDeadline,
|
||||
SETTLE_GRACE_MS
|
||||
)
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
if (targetFrame) {
|
||||
@@ -2591,9 +2834,10 @@ async function executeToolInner(
|
||||
const requestedKey = requireStr(params, 'key')
|
||||
const combo = parseKeyCombo(requestedKey)
|
||||
const contents = session.requireAutomationTab().view.webContents
|
||||
const pressedPageUrl = contents.getURL()
|
||||
const pressedNavigationEpoch = navigationEpoch(contents)
|
||||
let target: PageExecutionTarget = focusedPageTarget(contents)
|
||||
let pressedFrameUrl = target === contents ? undefined : (target as WebFrameMain).url
|
||||
let pressedFrameEpoch =
|
||||
target === contents ? undefined : frameNavigationEpoch(contents, target as WebFrameMain)
|
||||
// Pasting would move the user's clipboard into the page, where the next
|
||||
// snapshot reports it as an ordinary field value — clipboards routinely
|
||||
// hold a password copied out of a password manager. Copy and cut would
|
||||
@@ -2612,7 +2856,8 @@ async function executeToolInner(
|
||||
const dispatchTarget = focusedPageTarget(contents)
|
||||
if (dispatchTarget !== target) {
|
||||
target = dispatchTarget
|
||||
pressedFrameUrl = target === contents ? undefined : (target as WebFrameMain).url
|
||||
pressedFrameEpoch =
|
||||
target === contents ? undefined : frameNavigationEpoch(contents, target as WebFrameMain)
|
||||
beforePage = await pageActionState(target, true)
|
||||
beforeElement = await activeElementState(target)
|
||||
}
|
||||
@@ -2648,8 +2893,8 @@ async function executeToolInner(
|
||||
let fallbackState: Record<string, unknown> = {}
|
||||
try {
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, pressedPageUrl)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameUrl)
|
||||
assertActiveContents(contents, pressedNavigationEpoch)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameEpoch)
|
||||
await dispatchKeyCombo(contents, combo)
|
||||
} catch (error) {
|
||||
if (error instanceof KeyDispatchError && error.keyDownDispatched) {
|
||||
@@ -2661,8 +2906,8 @@ async function executeToolInner(
|
||||
// CDP unavailable (debugger detached): synthetic DOM fallback. It
|
||||
// cannot trigger default editing actions, so say so in the result.
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, pressedPageUrl)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameUrl)
|
||||
assertActiveContents(contents, pressedNavigationEpoch)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameEpoch)
|
||||
const fallbackSecrecy = await execInPage(target, activeElementSecrecy, []).catch(
|
||||
() => 'opaque'
|
||||
)
|
||||
@@ -2673,8 +2918,8 @@ async function executeToolInner(
|
||||
)
|
||||
}
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, pressedPageUrl)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameUrl)
|
||||
assertActiveContents(contents, pressedNavigationEpoch)
|
||||
assertFocusedTargetUnchanged(contents, target, pressedFrameEpoch)
|
||||
const fallback = unwrapPageResult(
|
||||
await execInPage(
|
||||
target,
|
||||
@@ -2867,16 +3112,29 @@ async function executeToolInner(
|
||||
const elementId = requireNum(params, 'elementId')
|
||||
const target = pageTargetForElement(contents, elementId)
|
||||
const targetFrame = frameExecutionTarget(target, contents)
|
||||
const beforePage = await pageActionState(target, true, elementId)
|
||||
const beforeElement = await activeElementState(target)
|
||||
const beforeTopPage = targetFrame ? await pageActionState(contents, true) : beforePage
|
||||
const beforeTopElement = targetFrame ? await activeElementState(contents) : beforeElement
|
||||
let beforePage = await pageActionState(target, true, elementId)
|
||||
let beforeElement = await activeElementState(target)
|
||||
let beforeTopPage = targetFrame ? await pageActionState(contents, true) : beforePage
|
||||
let beforeTopElement = targetFrame ? await activeElementState(contents) : beforeElement
|
||||
// Preparing the surface scrolls the element into view, so a baseline
|
||||
// taken before it always reports scrollChanged — the tool's own probe,
|
||||
// not the hover's effect. That pinned every unproductive hover to
|
||||
// "background churn" instead of the honest "nothing happened", and hid
|
||||
// real scrolling caused by the hover itself. Re-baseline once the scroll
|
||||
// has settled and before the pointer moves.
|
||||
const rebaseline = async (): Promise<void> => {
|
||||
beforePage = await pageActionState(target, true, elementId)
|
||||
beforeElement = await activeElementState(target)
|
||||
beforeTopPage = targetFrame ? await pageActionState(contents, true) : beforePage
|
||||
beforeTopElement = targetFrame ? await activeElementState(contents) : beforeElement
|
||||
}
|
||||
let trusted = false
|
||||
let result: unknown
|
||||
if (!targetFrame) {
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
let prepared = await prepareElementSurface(target, elementId, executionDeadline, true)
|
||||
await rebaseline()
|
||||
let stable = false
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
assertCurrentExecution()
|
||||
@@ -2900,6 +3158,7 @@ async function executeToolInner(
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
let surface = await prepareElementSurface(target, elementId, executionDeadline, true)
|
||||
await rebaseline()
|
||||
assertCurrentExecution()
|
||||
assertElementActionCurrent(contents, elementId, target)
|
||||
let embedding = await assertFrameEmbeddingVisible(
|
||||
@@ -3000,9 +3259,338 @@ async function executeToolInner(
|
||||
effectObserved,
|
||||
...(!effectObserved
|
||||
? {
|
||||
note: possibleEffectObserved
|
||||
? 'Only background DOM/title churn followed the hover; a tooltip/menu was not confirmed.'
|
||||
: 'No tooltip, menu, focus, or other strong hover effect was observed.',
|
||||
// A capped scan cannot claim nothing appeared: overlays are
|
||||
// commonly portalled to the END of <body>, which is exactly the
|
||||
// part a truncated walk misses. Say so instead of reporting a
|
||||
// partial look with full confidence.
|
||||
note:
|
||||
afterPage.observationTruncated === true ||
|
||||
afterTopPage.observationTruncated === true
|
||||
? 'This page is too large to scan completely, so a tooltip or menu that opened may not have been seen. Confirm with browser_snapshot or browser_screenshot before concluding the hover did nothing.'
|
||||
: possibleEffectObserved
|
||||
? 'Only background DOM/title churn followed the hover; a tooltip/menu was not confirmed.'
|
||||
: 'No tooltip, menu, focus, or other strong hover effect was observed.',
|
||||
}
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
case 'browser_click_at': {
|
||||
const clickedTab = session.requireAutomationTab()
|
||||
const contents = clickedTab.view.webContents
|
||||
const x = requireNum(params, 'x')
|
||||
const y = requireNum(params, 'y')
|
||||
const clickCount = num(params, 'clickCount') ?? 1
|
||||
if (![1, 2, 3].includes(clickCount)) {
|
||||
throw new ToolError('clickCount must be 1 (click), 2 (double-click), or 3 (triple-click).')
|
||||
}
|
||||
const clickNavigationEpoch = navigationEpoch(contents)
|
||||
const urlAtDispatch = contents.getURL()
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents)
|
||||
const pointTarget = unwrapPageResult(
|
||||
await execInPage(contents, describePointTarget, [x, y], false, executionDeadline)
|
||||
)
|
||||
if (!isRecordLike(pointTarget) || pointTarget.found !== true) {
|
||||
throw new ToolError(
|
||||
'Nothing is rendered at that point. Coordinates are CSS pixels in the current viewport — when reading them off a browser_screenshot, divide image pixels by its scale.'
|
||||
)
|
||||
}
|
||||
if (pointTarget.fileInput === true) {
|
||||
throw new ToolError(
|
||||
'Refusing to click a file input because it opens a native chooser the browser agent cannot inspect or complete. Ask the user to upload the file themselves.'
|
||||
)
|
||||
}
|
||||
const beforePage = await pageActionState(contents, true)
|
||||
const beforeElement = await activeElementState(contents)
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, clickNavigationEpoch)
|
||||
try {
|
||||
await cdp.clickAt(contents, x, y, true, clickCount)
|
||||
} catch (error) {
|
||||
const rescued = navigationRescue(contents, clickNavigationEpoch, urlAtDispatch, {
|
||||
trusted: true,
|
||||
activation: 'native-pointer',
|
||||
})
|
||||
if (rescued) return rescued
|
||||
throw new ToolError(
|
||||
`Native click dispatch failed (${getErrorMessage(error)}). The action was not retried because a partial pointer press may already have reached the page. Take a fresh snapshot before continuing.`
|
||||
)
|
||||
}
|
||||
await sleep(150)
|
||||
const afterElement = await activeElementState(contents)
|
||||
const afterPage = await pageActionState(contents)
|
||||
const observation = pageEffect(beforePage, afterPage, beforeElement, afterElement)
|
||||
const activeTab = session.automationTab()
|
||||
const tabChanged = activeTab?.id !== clickedTab.id
|
||||
// targetChanged is deliberately absent: this tool has no elementId, so
|
||||
// pageActionState captures no targetState and the term could only ever
|
||||
// be false. Listing it read as coverage this tool does not have.
|
||||
const effectObserved =
|
||||
observation.effect.urlChanged ||
|
||||
observation.effect.dialogChanged ||
|
||||
observation.effect.popupChanged ||
|
||||
(pointTarget.editable === true && observation.effect.focusChanged) ||
|
||||
tabChanged
|
||||
const dialogs = Array.isArray(afterPage.dialogs) ? afterPage.dialogs.map(String) : []
|
||||
const notes: string[] = []
|
||||
if (pointTarget.secret === true) {
|
||||
notes.push(
|
||||
'The point resolves to a password field. Focusing it is fine, but typing there is refused — call browser_request_takeover for credentials.'
|
||||
)
|
||||
}
|
||||
if (pointTarget.crossOriginFrame === true) {
|
||||
notes.push(
|
||||
'The point lands inside an embedded frame that could not be inspected; the click was dispatched but its target is unverified.'
|
||||
)
|
||||
}
|
||||
if (!effectObserved) {
|
||||
notes.push(
|
||||
observation.possibleEffectObserved || tabChanged
|
||||
? 'Only background DOM/title churn followed the click; inspect the page before treating it as successful.'
|
||||
: 'No strong observable page change followed the click; inspect the page before treating it as successful.'
|
||||
)
|
||||
}
|
||||
return {
|
||||
dispatched: true,
|
||||
trusted: true,
|
||||
activation: 'native-pointer',
|
||||
clickedAt: { x, y },
|
||||
clickCount,
|
||||
target: pointTarget.element,
|
||||
targetCursor: pointTarget.cursor,
|
||||
effectObserved,
|
||||
possibleEffectObserved: observation.possibleEffectObserved || tabChanged,
|
||||
effect: { ...observation.effect, tabChanged },
|
||||
dialogs,
|
||||
...(tabChanged && activeTab
|
||||
? { activeTab: { tabId: activeTab.id, url: activeTab.view.webContents.getURL() } }
|
||||
: {}),
|
||||
...(notes.length > 0 ? { note: notes.join(' ') } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
case 'browser_insert_text': {
|
||||
const text = requireStr(params, 'text')
|
||||
const submit = params.submit === true
|
||||
const contents = session.requireAutomationTab().view.webContents
|
||||
const insertNavigationEpoch = navigationEpoch(contents)
|
||||
const target: PageExecutionTarget = focusedPageTarget(contents)
|
||||
const secrecy = await execInPage(target, activeElementSecrecy, []).catch(() => 'opaque')
|
||||
if (secrecy === 'secret') throw new ToolError(PASSWORD_REFUSAL)
|
||||
if (secrecy === 'opaque') {
|
||||
throw new ToolError(
|
||||
'Focus is inside a cross-origin frame whose contents cannot be inspected, so this ' +
|
||||
'insertion could reach a password field. Call browser_request_takeover if the user needs to type here.'
|
||||
)
|
||||
}
|
||||
const focusState = unwrapPageResult(
|
||||
await execInPage(target, describeFocusedEditable, [], false, executionDeadline)
|
||||
)
|
||||
if (!isRecordLike(focusState) || focusState.editable !== true) {
|
||||
const reason = isRecordLike(focusState) ? String(focusState.reason || '') : ''
|
||||
// Name the element that actually held focus. Without it the agent
|
||||
// cannot tell "I focused the wrong thing" from "this tool cannot type
|
||||
// here", and it retries variations of the same failing approach.
|
||||
const focused = isRecordLike(focusState)
|
||||
? [
|
||||
focusState.focusedTag ? `<${String(focusState.focusedTag)}>` : '',
|
||||
focusState.focusedRole ? `role="${String(focusState.focusedRole)}"` : '',
|
||||
focusState.contentEditable && focusState.contentEditable !== 'unset'
|
||||
? `contenteditable="${String(focusState.contentEditable)}"`
|
||||
: '',
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(' ')
|
||||
: ''
|
||||
throw new ToolError(
|
||||
reason === 'none'
|
||||
? 'No element is focused. Click the field first (browser_click or browser_click_at), then insert text.'
|
||||
: `The focused element does not accept text${reason ? ` (${reason})` : ''}${
|
||||
focused ? `; focus is on ${focused}` : ''
|
||||
}. Click the field you want to type into, then insert text.`
|
||||
)
|
||||
}
|
||||
const beforePage = await pageActionState(target, true)
|
||||
const beforeElement = await activeElementState(target)
|
||||
// Observe the TOP document too when typing inside a frame. A submit that
|
||||
// navigates the top page is invisible to a frame-scoped observation, so a
|
||||
// successful send reported effectObserved: false. Newly reachable now
|
||||
// that the focus check descends into frames at all.
|
||||
const insertInFrame = target !== contents
|
||||
const beforeTopPage = insertInFrame ? await pageActionState(contents, true) : beforePage
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, insertNavigationEpoch)
|
||||
assertFocusedTargetUnchanged(contents, target)
|
||||
try {
|
||||
await cdp.insertText(contents, text)
|
||||
} catch (error) {
|
||||
throw new ToolError(
|
||||
`Native text insertion failed (${getErrorMessage(error)}). Take a fresh snapshot before retrying.`
|
||||
)
|
||||
}
|
||||
let submitDispatched = false
|
||||
if (submit) {
|
||||
await sleep(25)
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, insertNavigationEpoch)
|
||||
try {
|
||||
await dispatchKeyCombo(contents, parseKeyCombo('Enter'))
|
||||
submitDispatched = true
|
||||
} catch {
|
||||
// Reported below through submitDispatched: false.
|
||||
}
|
||||
}
|
||||
await sleep(150)
|
||||
const state = await activeElementState(target)
|
||||
const afterPage = await pageActionState(target)
|
||||
const observation = pageEffect(beforePage, afterPage, beforeElement, state)
|
||||
const topObservation = insertInFrame
|
||||
? pageEffect(beforeTopPage, await pageActionState(contents, true), beforeElement, state)
|
||||
: observation
|
||||
// targetChanged is deliberately absent: this tool has no elementId, so
|
||||
// pageActionState captures no targetState and the term could only ever
|
||||
// be false. Listing it read as coverage this tool does not have.
|
||||
const effectObserved =
|
||||
observation.effect.fieldChanged ||
|
||||
observation.effect.urlChanged ||
|
||||
observation.effect.dialogChanged ||
|
||||
topObservation.effect.urlChanged ||
|
||||
topObservation.effect.dialogChanged
|
||||
return {
|
||||
dispatched: true,
|
||||
trusted: true,
|
||||
kind: focusState.kind,
|
||||
insertedChars: text.length,
|
||||
...state,
|
||||
effectObserved,
|
||||
possibleEffectObserved: observation.possibleEffectObserved,
|
||||
effect: observation.effect,
|
||||
submitRequested: submit,
|
||||
submitDispatched,
|
||||
...(focusState.kind === 'canvas' || focusState.kind === 'textbox-role'
|
||||
? {
|
||||
note: 'The focused editor is canvas/model-backed, so field readback cannot confirm the text — verify visually with browser_screenshot.',
|
||||
}
|
||||
: !effectObserved
|
||||
? {
|
||||
note: 'Insertion produced no observable field or page change; inspect the page before continuing.',
|
||||
}
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
case 'browser_drag': {
|
||||
const draggedTab = session.requireAutomationTab()
|
||||
const contents = draggedTab.view.webContents
|
||||
const dragNavigationEpoch = navigationEpoch(contents)
|
||||
|
||||
const resolveEndpoint = async (
|
||||
which: 'from' | 'to'
|
||||
): Promise<{ x: number; y: number; element?: string }> => {
|
||||
const elementId = num(params, `${which}ElementId`)
|
||||
if (elementId !== undefined) {
|
||||
const target = pageTargetForElement(contents, elementId)
|
||||
if (frameExecutionTarget(target, contents)) {
|
||||
throw new ToolError(
|
||||
`Dragging elements inside embedded frames is not supported. Use ${which}X/${which}Y viewport coordinates instead.`
|
||||
)
|
||||
}
|
||||
const prepared = unwrapPageResult(
|
||||
await execInPageWithSettleGrace(
|
||||
target,
|
||||
clickElement,
|
||||
[elementId, false, false],
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
if (
|
||||
!isRecordLike(prepared) ||
|
||||
typeof prepared.x !== 'number' ||
|
||||
typeof prepared.y !== 'number'
|
||||
) {
|
||||
throw new ToolError(`Could not resolve the ${which} element for the drag.`)
|
||||
}
|
||||
return {
|
||||
x: prepared.x,
|
||||
y: prepared.y,
|
||||
...(typeof prepared.element === 'string' ? { element: prepared.element } : {}),
|
||||
}
|
||||
}
|
||||
const pointX = num(params, `${which}X`)
|
||||
const pointY = num(params, `${which}Y`)
|
||||
if (pointX === undefined || pointY === undefined) {
|
||||
throw new ToolError(
|
||||
`Provide either ${which}ElementId or both ${which}X and ${which}Y for the drag ${which === 'from' ? 'source' : 'target'}.`
|
||||
)
|
||||
}
|
||||
const probe = unwrapPageResult(
|
||||
await execInPage(
|
||||
contents,
|
||||
describePointTarget,
|
||||
[pointX, pointY],
|
||||
false,
|
||||
executionDeadline
|
||||
)
|
||||
)
|
||||
if (!isRecordLike(probe) || probe.found !== true) {
|
||||
throw new ToolError(
|
||||
`Nothing is rendered at the ${which} point. Coordinates are CSS pixels in the current viewport — when reading them off a browser_screenshot, divide image pixels by its scale.`
|
||||
)
|
||||
}
|
||||
return {
|
||||
x: pointX,
|
||||
y: pointY,
|
||||
...(typeof probe.element === 'string' ? { element: probe.element } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents)
|
||||
const from = await resolveEndpoint('from')
|
||||
const to = await resolveEndpoint('to')
|
||||
if (Math.abs(from.x - to.x) < 1 && Math.abs(from.y - to.y) < 1) {
|
||||
throw new ToolError('The drag source and target are the same point; nothing to drag.')
|
||||
}
|
||||
const beforePage = await pageActionState(contents, true)
|
||||
const beforeElement = await activeElementState(contents)
|
||||
assertCurrentExecution()
|
||||
assertActiveContents(contents, dragNavigationEpoch)
|
||||
let interception: { nativeDragIntercepted: boolean }
|
||||
try {
|
||||
interception = await cdp.dragPointer(contents, from, to)
|
||||
} catch (error) {
|
||||
throw new ToolError(
|
||||
`Native drag dispatch failed (${getErrorMessage(error)}). The pointer may have been mid-drag; take a fresh snapshot to see the page's current state before retrying.`
|
||||
)
|
||||
}
|
||||
await sleep(200)
|
||||
const afterElement = await activeElementState(contents)
|
||||
const afterPage = await pageActionState(contents)
|
||||
const observation = pageEffect(beforePage, afterPage, beforeElement, afterElement)
|
||||
// targetChanged is deliberately absent: this tool has no elementId, so
|
||||
// pageActionState captures no targetState and the term could only ever be
|
||||
// false. domChanged IS trusted here — unlike every other tool — because a
|
||||
// drop that reorders a list may change nothing else observable.
|
||||
const effectObserved =
|
||||
observation.effect.domChanged ||
|
||||
observation.effect.urlChanged ||
|
||||
observation.effect.dialogChanged ||
|
||||
observation.effect.scrollChanged
|
||||
const dialogs = Array.isArray(afterPage.dialogs) ? afterPage.dialogs.map(String) : []
|
||||
return {
|
||||
dispatched: true,
|
||||
trusted: true,
|
||||
nativeHtml5Drag: interception.nativeDragIntercepted,
|
||||
from,
|
||||
to,
|
||||
effectObserved,
|
||||
possibleEffectObserved: observation.possibleEffectObserved,
|
||||
effect: observation.effect,
|
||||
dialogs,
|
||||
...(!effectObserved
|
||||
? {
|
||||
note: 'No observable page change followed the drag. Verify with browser_snapshot or browser_screenshot; some drop targets only commit on their own animation frame.',
|
||||
}
|
||||
: {}),
|
||||
}
|
||||
|
||||
@@ -5,6 +5,8 @@ import {
|
||||
activeElementSecrecy,
|
||||
clickElement,
|
||||
collectSnapshot,
|
||||
describeFocusedEditable,
|
||||
describePointTarget,
|
||||
focusElementForTyping,
|
||||
getViewportInfo,
|
||||
hoverElement,
|
||||
@@ -155,6 +157,8 @@ describe('serialization contract', () => {
|
||||
['readPageText', readPageText, []],
|
||||
['pageContainsText', pageContainsText, ['needle']],
|
||||
['getViewportInfo', getViewportInfo, []],
|
||||
['describePointTarget', describePointTarget, [10, 10]],
|
||||
['describeFocusedEditable', describeFocusedEditable, []],
|
||||
]
|
||||
|
||||
it.each(cases)('%s is self-contained', (_name, fn, args) => {
|
||||
@@ -257,8 +261,11 @@ describe('secret-field detection', () => {
|
||||
expect(typeIntoElement(0, 'change', false)).toEqual({ error: 'readonly' })
|
||||
expect(focusElementForTyping(1)).toEqual({ error: 'disabled' })
|
||||
expect(typeIntoElement(1, 'change', false)).toEqual({ error: 'disabled' })
|
||||
expect(focusElementForTyping(2)).toEqual({ error: 'not-editable' })
|
||||
expect(typeIntoElement(2, 'change', false)).toEqual({ error: 'not-editable' })
|
||||
expect(focusElementForTyping(2)).toMatchObject({ error: 'not-editable', elementTag: 'input' })
|
||||
expect(typeIntoElement(2, 'change', false)).toMatchObject({
|
||||
error: 'not-editable',
|
||||
elementTag: 'input',
|
||||
})
|
||||
})
|
||||
|
||||
it('detects a password field reached through a same-origin iframe', () => {
|
||||
@@ -379,7 +386,12 @@ describe('combobox typing surfaces', () => {
|
||||
for (const input of Array.from(document.querySelectorAll('input'))) visible(input)
|
||||
register(ambiguous, secret)
|
||||
|
||||
expect(focusElementForTyping(0)).toEqual({ error: 'ambiguous-editable' })
|
||||
// The candidate list is the whole point of this error — it is the only
|
||||
// thing that lets the agent pick a narrower target.
|
||||
expect(focusElementForTyping(0)).toMatchObject({
|
||||
error: 'ambiguous-editable',
|
||||
candidates: expect.arrayContaining([expect.stringContaining('input')]),
|
||||
})
|
||||
expect(focusElementForTyping(1)).toEqual({ error: 'password' })
|
||||
expect(typeIntoElement(1, 'nope', false)).toEqual({ error: 'password' })
|
||||
})
|
||||
@@ -589,8 +601,11 @@ describe('collectSnapshot', () => {
|
||||
})
|
||||
|
||||
expect(card.contains(nestedButton)).toBe(true)
|
||||
// Nothing is covering the card — its own button owns the point. Reporting
|
||||
// this as an obstruction told the agent to close an overlay that does not
|
||||
// exist; the recovery is to target the nested control instead.
|
||||
expect(clickElement(ref, false)).toEqual({
|
||||
error: 'obstructed',
|
||||
error: 'nested-control',
|
||||
blocker: 'Delete channel',
|
||||
})
|
||||
})
|
||||
@@ -766,7 +781,39 @@ describe('collectSnapshot', () => {
|
||||
document.body.append(replacement)
|
||||
}
|
||||
|
||||
expect(focusElementForTyping(ref)).toEqual({ error: 'stale' })
|
||||
expect(focusElementForTyping(ref)).toMatchObject({ error: 'stale' })
|
||||
})
|
||||
|
||||
it('keeps a connected ref usable after a same-document URL change', () => {
|
||||
document.body.innerHTML = '<button data-testid="composer-send">Send</button>'
|
||||
const button = visible(document.querySelector('button') as HTMLButtonElement)
|
||||
const ref = refFor(outlineOf(collectSnapshot()), 'Send')
|
||||
let clicked = false
|
||||
button.addEventListener('click', () => {
|
||||
clicked = true
|
||||
})
|
||||
|
||||
window.history.pushState({}, '', '/client/T123/C456')
|
||||
|
||||
expect(clickElement(ref)).toMatchObject({ dispatched: true, refRecovered: false })
|
||||
expect(clicked).toBe(true)
|
||||
})
|
||||
|
||||
it('recovers a replaced ref after a same-document URL change', () => {
|
||||
document.body.innerHTML = '<button data-testid="messages-tab">Messages</button>'
|
||||
const original = visible(document.querySelector('button') as HTMLButtonElement)
|
||||
const ref = refFor(outlineOf(collectSnapshot()), 'Messages')
|
||||
const replacement = visible(original.cloneNode(true) as HTMLButtonElement)
|
||||
let clicked = false
|
||||
replacement.addEventListener('click', () => {
|
||||
clicked = true
|
||||
})
|
||||
|
||||
window.history.pushState({}, '', '/client/T123/C456')
|
||||
original.replaceWith(replacement)
|
||||
|
||||
expect(clickElement(ref)).toMatchObject({ dispatched: true, refRecovered: true })
|
||||
expect(clicked).toBe(true)
|
||||
})
|
||||
|
||||
it('refuses to recover a ref when replacement is ambiguous', () => {
|
||||
@@ -777,7 +824,7 @@ describe('collectSnapshot', () => {
|
||||
const second = visible(original.cloneNode(true) as HTMLButtonElement)
|
||||
original.replaceWith(first, second)
|
||||
|
||||
expect(clickElement(ref)).toEqual({ error: 'stale' })
|
||||
expect(clickElement(ref)).toMatchObject({ error: 'stale' })
|
||||
})
|
||||
|
||||
it('invalidates a connected virtual row when its identity is recycled in place', () => {
|
||||
@@ -788,7 +835,7 @@ describe('collectSnapshot', () => {
|
||||
row.textContent = 'random'
|
||||
row.dataset.key = 'channel-2'
|
||||
|
||||
expect(clickElement(ref)).toEqual({ error: 'stale' })
|
||||
expect(clickElement(ref)).toMatchObject({ error: 'stale' })
|
||||
})
|
||||
|
||||
it('invalidates a generic connected row action when its surrounding item is recycled', () => {
|
||||
@@ -802,7 +849,7 @@ describe('collectSnapshot', () => {
|
||||
;(document.querySelector('span') as HTMLSpanElement).textContent = 'random'
|
||||
|
||||
expect(button.isConnected).toBe(true)
|
||||
expect(clickElement(ref)).toEqual({ error: 'stale' })
|
||||
expect(clickElement(ref)).toMatchObject({ error: 'stale' })
|
||||
})
|
||||
|
||||
it('never recycles numeric refs across snapshots', () => {
|
||||
@@ -812,10 +859,32 @@ describe('collectSnapshot', () => {
|
||||
const secondRef = refFor(outlineOf(collectSnapshot()), 'Pins')
|
||||
|
||||
expect(secondRef).toBeGreaterThan(firstRef)
|
||||
expect(clickElement(firstRef)).toEqual({ error: 'stale' })
|
||||
expect(clickElement(firstRef)).toMatchObject({ error: 'stale' })
|
||||
expect(clickElement(secondRef)).toMatchObject({ dispatched: true })
|
||||
})
|
||||
|
||||
// The exact shape of the reported failure: hovering a Slack message mounts an
|
||||
// action bar, but it is a role="toolbar"/"group" — none of the three roles the
|
||||
// popup scan used to match. The hover therefore observed no popup change, no
|
||||
// target change, and so no effect at all, and the agent concluded hovering
|
||||
// did not work and fell back to clicking pixels off screenshots.
|
||||
it('sees a row action bar that mounts on hover', () => {
|
||||
document.body.innerHTML = '<div data-testid="message">Hello</div>'
|
||||
visible(document.querySelector('[data-testid="message"]') as HTMLElement)
|
||||
|
||||
const before = readPageActionState(true) as { popups: string[] }
|
||||
expect(before.popups).toEqual([])
|
||||
|
||||
const toolbar = visible(document.createElement('div'))
|
||||
toolbar.setAttribute('role', 'toolbar')
|
||||
toolbar.setAttribute('aria-label', 'Message shortcuts')
|
||||
document.body.append(toolbar)
|
||||
|
||||
const after = readPageActionState(false) as { popups: string[] }
|
||||
expect(after.popups).toEqual(['Message shortcuts'])
|
||||
expect(after.popups).not.toEqual(before.popups)
|
||||
})
|
||||
|
||||
it('reports a targeted control semantic disappearance after its panel closes', () => {
|
||||
document.body.innerHTML = `
|
||||
<aside aria-label="Thread panel"><button data-testid="close-thread">Close thread</button></aside>
|
||||
@@ -1341,3 +1410,112 @@ describe('pressKeyOnPage', () => {
|
||||
expect(seen).toEqual(['a'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('describePointTarget', () => {
|
||||
function pointAt(el: Element | null): void {
|
||||
Object.defineProperty(document, 'elementFromPoint', {
|
||||
configurable: true,
|
||||
value: () => el,
|
||||
})
|
||||
}
|
||||
|
||||
it('describes the element at a viewport point', () => {
|
||||
document.body.innerHTML = '<button aria-label="Send message">Send</button>'
|
||||
pointAt(document.querySelector('button'))
|
||||
|
||||
expect(describePointTarget(10, 10)).toMatchObject({
|
||||
found: true,
|
||||
tag: 'button',
|
||||
element: 'button "Send message"',
|
||||
editable: false,
|
||||
fileInput: false,
|
||||
secret: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('flags file inputs and password fields at the point', () => {
|
||||
document.body.innerHTML = '<input type="file" />'
|
||||
pointAt(document.querySelector('input'))
|
||||
expect(describePointTarget(10, 10)).toMatchObject({ found: true, fileInput: true })
|
||||
|
||||
document.body.innerHTML = '<input type="password" />'
|
||||
pointAt(document.querySelector('input'))
|
||||
expect(describePointTarget(10, 10)).toMatchObject({
|
||||
found: true,
|
||||
secret: true,
|
||||
editable: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects points outside the viewport', () => {
|
||||
expect(describePointTarget(-5, 10)).toEqual({ error: 'outside-viewport' })
|
||||
expect(describePointTarget(10, window.innerHeight + 5)).toEqual({
|
||||
error: 'outside-viewport',
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('describeFocusedEditable', () => {
|
||||
it('reports no focus when the body holds focus', () => {
|
||||
setActiveElement(document, document.body)
|
||||
expect(describeFocusedEditable()).toEqual({ editable: false, reason: 'none' })
|
||||
})
|
||||
|
||||
// The bug this pins: focus inside a same-origin frame surfaces on the outer
|
||||
// document as the FRAME element, which is not an input, not contentEditable,
|
||||
// not a canvas, and carries no textbox role — so a composer that press-key
|
||||
// typed into fine was reported `not-editable` and insert_text refused. The
|
||||
// descent here must match activeElementReadback's exactly.
|
||||
it('descends a same-origin frame to the editable that really holds focus', () => {
|
||||
document.body.innerHTML = ''
|
||||
const frame = document.createElement('iframe')
|
||||
document.body.append(frame)
|
||||
const inner = frame.contentDocument as Document
|
||||
inner.body.innerHTML = '<div contenteditable="true">composer</div>'
|
||||
const composer = inner.querySelector('div') as HTMLElement
|
||||
Object.defineProperty(composer, 'isContentEditable', { value: true, configurable: true })
|
||||
setActiveElement(inner, composer)
|
||||
setActiveElement(document, frame)
|
||||
|
||||
expect(describeFocusedEditable()).toEqual({ editable: true, kind: 'contenteditable' })
|
||||
})
|
||||
|
||||
it('names the focused element when it refuses, so the agent can recover', () => {
|
||||
document.body.innerHTML = '<div role="button">Send</div>'
|
||||
setActiveElement(document, document.querySelector('div'))
|
||||
|
||||
expect(describeFocusedEditable()).toEqual({
|
||||
editable: false,
|
||||
reason: 'not-editable',
|
||||
focusedTag: 'div',
|
||||
focusedRole: 'button',
|
||||
contentEditable: 'unset',
|
||||
})
|
||||
})
|
||||
|
||||
it('reports a writable input as insertable', () => {
|
||||
document.body.innerHTML = '<input type="text" />'
|
||||
setActiveElement(document, document.querySelector('input'))
|
||||
expect(describeFocusedEditable()).toEqual({ editable: true, kind: 'input:text' })
|
||||
})
|
||||
|
||||
it('reports a read-only input as not insertable', () => {
|
||||
document.body.innerHTML = '<input type="text" readonly />'
|
||||
setActiveElement(document, document.querySelector('input'))
|
||||
expect(describeFocusedEditable()).toEqual({ editable: false, reason: 'readonly' })
|
||||
})
|
||||
|
||||
it('reports a focused contenteditable editor as insertable', () => {
|
||||
document.body.innerHTML = '<div></div>'
|
||||
const editor = document.querySelector('div') as HTMLElement
|
||||
Object.defineProperty(editor, 'isContentEditable', { get: () => true })
|
||||
setActiveElement(document, editor)
|
||||
expect(describeFocusedEditable()).toEqual({ editable: true, kind: 'contenteditable' })
|
||||
})
|
||||
|
||||
it('treats a focused canvas editor surface as insertable', () => {
|
||||
document.body.innerHTML = '<canvas></canvas>'
|
||||
setActiveElement(document, document.querySelector('canvas'))
|
||||
expect(describeFocusedEditable()).toEqual({ editable: true, kind: 'canvas' })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -34,8 +34,14 @@ declare global {
|
||||
root: Node
|
||||
observer: MutationObserver
|
||||
revision: number
|
||||
/** Roots already passed to observe(), so re-observing stays cheap. */
|
||||
observedRoots: WeakSet<ParentNode>
|
||||
}>
|
||||
__simAgentNextElementId?: number
|
||||
/** Why the last __simAgentResolveElement call returned null — read by the
|
||||
* shared stale-error producers so a refusal names its cause instead of
|
||||
* the blanket "the page changed". Cleared on every successful resolve. */
|
||||
__simAgentStaleReason?: string
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,6 +115,7 @@ export function collectSnapshot(startingElementId = 0): unknown {
|
||||
'[onclick]',
|
||||
'[contenteditable="true"]',
|
||||
'[contenteditable=""]',
|
||||
'[contenteditable="plaintext-only"]',
|
||||
].join(', ')
|
||||
const landmarkSelector = [
|
||||
'nav',
|
||||
@@ -593,13 +600,29 @@ export function collectSnapshot(startingElementId = 0): unknown {
|
||||
|
||||
/**
|
||||
* React commonly replaces a control's DOM node while preserving its
|
||||
* semantics. Recover only when the old page URL and a strong semantic
|
||||
* semantics. Recover only when the old page origin and a strong semantic
|
||||
* fingerprint still identify one candidate; a weak or ambiguous match is a
|
||||
* real stale ref, never permission to click something nearby.
|
||||
*/
|
||||
window.__simAgentResolveElement = (id: number) => {
|
||||
const locator = locators[id]
|
||||
if (!locator) return null
|
||||
if (!locator) {
|
||||
window.__simAgentStaleReason = `id ${id} is not in the current snapshot's registry`
|
||||
return null
|
||||
}
|
||||
|
||||
// Origin, not full URL: a live SPA rewrites its path with pushState
|
||||
// between snapshot and act (Slack does so continuously), while the
|
||||
// element the model chose is often still the same mounted node. The
|
||||
// origin still pins the document/frame; the role/name/attribute and
|
||||
// ancestor/context signatures below pin the element itself.
|
||||
const pageOriginOf = (url: string): string => {
|
||||
try {
|
||||
return new URL(url).origin
|
||||
} catch {
|
||||
return url
|
||||
}
|
||||
}
|
||||
|
||||
const stableAttributes = [
|
||||
'id',
|
||||
@@ -620,7 +643,7 @@ export function collectSnapshot(startingElementId = 0): unknown {
|
||||
const identityMatches = (candidate: Element, connected = false): boolean => {
|
||||
if (
|
||||
candidate.tagName.toUpperCase() !== locator.tag ||
|
||||
pageUrlFor(candidate) !== locator.url ||
|
||||
pageOriginOf(pageUrlFor(candidate)) !== pageOriginOf(locator.url) ||
|
||||
roleFor(candidate) !== locator.role
|
||||
) {
|
||||
return false
|
||||
@@ -704,8 +727,40 @@ export function collectSnapshot(startingElementId = 0): unknown {
|
||||
|
||||
const current = registry[id]
|
||||
if (current?.isConnected) {
|
||||
if (!identityMatches(current, true)) return null
|
||||
if (isCurrentlyVisible(current)) return { element: current, recovered: false }
|
||||
if (!identityMatches(current, true)) {
|
||||
window.__simAgentStaleReason = `the node is still mounted but its identity drifted (now <${String(current.tagName || '').toLowerCase()}> role=${roleFor(current) || 'none'} name="${nameFor(current).slice(0, 60)}")`
|
||||
return null
|
||||
}
|
||||
if (isCurrentlyVisible(current)) {
|
||||
window.__simAgentStaleReason = undefined
|
||||
return { element: current, recovered: false }
|
||||
}
|
||||
}
|
||||
|
||||
// Past this point the original node is gone or hidden, so anything returned
|
||||
// is a DIFFERENT node adopted by structural resemblance. Identity matching
|
||||
// compares origins only — deliberately, so a pushState between snapshot and
|
||||
// act does not kill every ref — but that same leniency let a ref to "More
|
||||
// actions" on a row in one view rebind to the identical control in another
|
||||
// view the app had since navigated to, and act on the wrong thing with no
|
||||
// signal beyond `recovered: true`.
|
||||
//
|
||||
// A same-document path change means the view was swapped, so resemblance is
|
||||
// no longer evidence of sameness. Refuse to adopt and report the ref stale:
|
||||
// the caller re-snapshots, which is cheap and always correct. Revalidating
|
||||
// the still-connected node above stays lenient — it is literally the node
|
||||
// the model chose.
|
||||
const pathOf = (url: string): string => {
|
||||
try {
|
||||
const parsed = new URL(url)
|
||||
return `${parsed.origin}${parsed.pathname}`
|
||||
} catch {
|
||||
return url
|
||||
}
|
||||
}
|
||||
if (pathOf(window.location.href) !== pathOf(locator.url)) {
|
||||
window.__simAgentStaleReason = `the view changed since the snapshot (${pathOf(locator.url)} -> ${pathOf(window.location.href)}), so a lookalike must not be adopted`
|
||||
return null
|
||||
}
|
||||
|
||||
const reachable: Element[] = []
|
||||
@@ -786,12 +841,20 @@ export function collectSnapshot(startingElementId = 0): unknown {
|
||||
.filter((entry) => entry.score >= 45)
|
||||
.sort((a, b) => b.score - a.score)
|
||||
|
||||
if (scored.length === 0) return null
|
||||
if (scored.length === 0) {
|
||||
window.__simAgentStaleReason =
|
||||
'the original node left the DOM and no confident replacement matched'
|
||||
return null
|
||||
}
|
||||
const bestScore = scored[0].score
|
||||
const best = scored.filter((entry) => entry.score === bestScore)
|
||||
const chosen = best.length === 1 ? best[0] : undefined
|
||||
if (!chosen) return null
|
||||
if (!chosen) {
|
||||
window.__simAgentStaleReason = `the original node left the DOM and ${best.length} equally-plausible replacements tied`
|
||||
return null
|
||||
}
|
||||
registry[id] = chosen.candidate
|
||||
window.__simAgentStaleReason = undefined
|
||||
return { element: chosen.candidate, recovered: true }
|
||||
}
|
||||
|
||||
@@ -828,7 +891,7 @@ export function clickElement(
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
const isDisabled = (node: Element | null): boolean =>
|
||||
Boolean(
|
||||
node &&
|
||||
@@ -862,7 +925,7 @@ export function clickElement(
|
||||
el.scrollIntoView({ block: 'center', inline: 'center', behavior: 'instant' })
|
||||
|
||||
const view = el.ownerDocument.defaultView
|
||||
if (!view) return { error: 'stale' }
|
||||
if (!view) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
for (let current: Element | null = el; current; ) {
|
||||
const currentView: Window | null = current.ownerDocument.defaultView
|
||||
const style = currentView?.getComputedStyle(current)
|
||||
@@ -1028,7 +1091,7 @@ export function clickElement(
|
||||
addCandidate(el)
|
||||
for (const candidate of Array.from(
|
||||
el.querySelectorAll<HTMLElement>(
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""]'
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""], [contenteditable="plaintext-only"], [role="textbox"]'
|
||||
)
|
||||
)) {
|
||||
addCandidate(candidate)
|
||||
@@ -1087,6 +1150,22 @@ export function clickElement(
|
||||
if (suggestionsCoverFocusedEditable()) {
|
||||
return { error: 'suggestions-open', blocker: blockerLabel(blocker) }
|
||||
}
|
||||
// A hit INSIDE the requested element is not an overlay — it is the ref
|
||||
// wrapping its own control (a row containing a button, a card containing a
|
||||
// link). hitBelongsToTarget rejects both cases identically, so this was
|
||||
// reported as "covered by X, close or move the overlay", advice that cannot
|
||||
// be followed because there is nothing to close. Name it for what it is so
|
||||
// the agent retargets instead of hunting a phantom overlay.
|
||||
let nested = false
|
||||
for (let current = blocker; current; current = composedParent(current)) {
|
||||
if (current === el) {
|
||||
nested = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if (nested) {
|
||||
return { error: 'nested-control', blocker: blockerLabel(blocker) }
|
||||
}
|
||||
return { error: 'obstructed', blocker: blockerLabel(blocker) }
|
||||
}
|
||||
|
||||
@@ -1212,7 +1291,7 @@ export function focusElementForTyping(id: number, moveFocus = true): unknown {
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
|
||||
const isWritableTextField = (
|
||||
field: HTMLInputElement | HTMLTextAreaElement
|
||||
@@ -1236,7 +1315,12 @@ export function focusElementForTyping(id: number, moveFocus = true): unknown {
|
||||
tag === 'TEXTAREA' ||
|
||||
(tag === 'INPUT' &&
|
||||
['text', 'search', 'email', 'url', 'tel', 'number', 'password'].includes(inputType)) ||
|
||||
(node as HTMLElement).isContentEditable
|
||||
(node as HTMLElement).isContentEditable ||
|
||||
// An ARIA-only textbox. The snapshot already advertises these as
|
||||
// `[textbox]` with a ref, and browser_insert_text accepts them, so
|
||||
// refusing here meant one tool rejecting exactly what the outline told
|
||||
// the model to type into and what its sibling would have accepted.
|
||||
node.getAttribute('role') === 'textbox'
|
||||
) {
|
||||
potentialEditables.push(node as HTMLElement)
|
||||
}
|
||||
@@ -1244,14 +1328,34 @@ export function focusElementForTyping(id: number, moveFocus = true): unknown {
|
||||
addEditable(el)
|
||||
for (const candidate of Array.from(
|
||||
el.querySelectorAll<HTMLElement>(
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""]'
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""], [contenteditable="plaintext-only"], [role="textbox"]'
|
||||
)
|
||||
)) {
|
||||
addEditable(candidate)
|
||||
}
|
||||
const editables = Array.from(new Set(potentialEditables))
|
||||
if (editables.length === 0) return { error: 'not-editable' }
|
||||
if (editables.length > 1) return { error: 'ambiguous-editable' }
|
||||
if (editables.length === 0) {
|
||||
// Describe the element instead of only refusing it. Without this the agent
|
||||
// cannot tell "wrong ref" from "this tool cannot type here" and retries
|
||||
// variations of the same failing call.
|
||||
return {
|
||||
error: 'not-editable',
|
||||
elementTag: String(el.tagName || '').toLowerCase(),
|
||||
...(el.getAttribute('role') ? { elementRole: el.getAttribute('role') } : {}),
|
||||
}
|
||||
}
|
||||
if (editables.length > 1) {
|
||||
// The candidate list is right here; discarding it left the agent unable to
|
||||
// pick a narrower target, which is the only recovery this error allows.
|
||||
return {
|
||||
error: 'ambiguous-editable',
|
||||
candidates: editables.slice(0, 5).map((field) => {
|
||||
const fieldTag = String(field.tagName || '').toLowerCase()
|
||||
const label = field.getAttribute('aria-label') || field.getAttribute('placeholder') || ''
|
||||
return label ? `${fieldTag} "${label}"` : fieldTag
|
||||
}),
|
||||
}
|
||||
}
|
||||
const editable = editables[0]
|
||||
const editableTag = tagFor(editable)
|
||||
|
||||
@@ -1274,7 +1378,7 @@ export function focusElementForTyping(id: number, moveFocus = true): unknown {
|
||||
const rawRects = Array.from(editable.getClientRects())
|
||||
if (rawRects.length === 0) rawRects.push(editable.getBoundingClientRect())
|
||||
const view = editable.ownerDocument.defaultView
|
||||
if (!view) return { error: 'stale' }
|
||||
if (!view) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
const rects = rawRects
|
||||
.map((rect) => ({
|
||||
left: Math.max(0, rect.left),
|
||||
@@ -1737,7 +1841,7 @@ export function typeIntoElement(id: number, text: string, submit: boolean): unkn
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
|
||||
const isWritableTextField = (
|
||||
field: HTMLInputElement | HTMLTextAreaElement
|
||||
@@ -1762,7 +1866,12 @@ export function typeIntoElement(id: number, text: string, submit: boolean): unkn
|
||||
candidateTag === 'TEXTAREA' ||
|
||||
(candidateTag === 'INPUT' &&
|
||||
['text', 'search', 'email', 'url', 'tel', 'number', 'password'].includes(inputType)) ||
|
||||
(node as HTMLElement).isContentEditable
|
||||
(node as HTMLElement).isContentEditable ||
|
||||
// An ARIA-only textbox. The snapshot already advertises these as
|
||||
// `[textbox]` with a ref, and browser_insert_text accepts them, so
|
||||
// refusing here meant one tool rejecting exactly what the outline told
|
||||
// the model to type into and what its sibling would have accepted.
|
||||
node.getAttribute('role') === 'textbox'
|
||||
) {
|
||||
potentialEditables.push(node as HTMLElement)
|
||||
}
|
||||
@@ -1770,14 +1879,34 @@ export function typeIntoElement(id: number, text: string, submit: boolean): unkn
|
||||
addEditable(el)
|
||||
for (const candidate of Array.from(
|
||||
el.querySelectorAll<HTMLElement>(
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""]'
|
||||
'input, textarea, [contenteditable="true"], [contenteditable=""], [contenteditable="plaintext-only"], [role="textbox"]'
|
||||
)
|
||||
)) {
|
||||
addEditable(candidate)
|
||||
}
|
||||
const editables = Array.from(new Set(potentialEditables))
|
||||
if (editables.length === 0) return { error: 'not-editable' }
|
||||
if (editables.length > 1) return { error: 'ambiguous-editable' }
|
||||
if (editables.length === 0) {
|
||||
// Describe the element instead of only refusing it. Without this the agent
|
||||
// cannot tell "wrong ref" from "this tool cannot type here" and retries
|
||||
// variations of the same failing call.
|
||||
return {
|
||||
error: 'not-editable',
|
||||
elementTag: String(el.tagName || '').toLowerCase(),
|
||||
...(el.getAttribute('role') ? { elementRole: el.getAttribute('role') } : {}),
|
||||
}
|
||||
}
|
||||
if (editables.length > 1) {
|
||||
// The candidate list is right here; discarding it left the agent unable to
|
||||
// pick a narrower target, which is the only recovery this error allows.
|
||||
return {
|
||||
error: 'ambiguous-editable',
|
||||
candidates: editables.slice(0, 5).map((field) => {
|
||||
const fieldTag = String(field.tagName || '').toLowerCase()
|
||||
const label = field.getAttribute('aria-label') || field.getAttribute('placeholder') || ''
|
||||
return label ? `${fieldTag} "${label}"` : fieldTag
|
||||
}),
|
||||
}
|
||||
}
|
||||
const editable = editables[0]
|
||||
const tag = String(editable.tagName || '').toUpperCase()
|
||||
editable.scrollIntoView({ block: 'nearest', inline: 'nearest', behavior: 'instant' })
|
||||
@@ -1859,9 +1988,37 @@ export function pressKeyOnPage(
|
||||
.some((token) => token === 'current-password' || token === 'new-password')
|
||||
}
|
||||
|
||||
const target = (document.activeElement as HTMLElement | null) ?? document.body
|
||||
// Descend to what is really focused, like every other focus reader here.
|
||||
// Without this the synthetic key lands on the shadow HOST or the <iframe>
|
||||
// element: the event bubbles from there but never enters the shadow tree or
|
||||
// the frame document, so the editor's keymap never sees it — and the result
|
||||
// still reports success. It also made this function's `target` disagree with
|
||||
// the `activeElement` reported alongside it in the same tool result.
|
||||
let target = (document.activeElement as HTMLElement | null) ?? document.body
|
||||
for (let depth = 0; depth < 10; depth++) {
|
||||
const shadow = target.shadowRoot
|
||||
if (shadow?.activeElement) {
|
||||
target = shadow.activeElement as HTMLElement
|
||||
continue
|
||||
}
|
||||
const targetTag = String(target.tagName || '').toUpperCase()
|
||||
if (targetTag === 'IFRAME' || targetTag === 'FRAME') {
|
||||
try {
|
||||
const inner = (target as HTMLIFrameElement).contentDocument
|
||||
if (inner?.activeElement && inner.activeElement !== inner.body) {
|
||||
target = inner.activeElement as HTMLElement
|
||||
continue
|
||||
}
|
||||
} catch {
|
||||
// Cross-origin frame — not inspectable; dispatch to the frame itself.
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
// The driver checks focus before taking the trusted CDP path; this covers
|
||||
// the synthetic fallback, which is reached independently when CDP is down.
|
||||
// Descending first is what makes this guard reach a password field nested in
|
||||
// a shadow root or frame, rather than only inspecting the host.
|
||||
if (isSecretField(target)) return { error: 'password' }
|
||||
const opts = {
|
||||
bubbles: true,
|
||||
@@ -1916,6 +2073,23 @@ export function readPageActionState(resetMutationRevision = false, elementId?: n
|
||||
if (allElements.length >= stateNodeCap) break
|
||||
}
|
||||
|
||||
const mutationOptions: MutationObserverInit = {
|
||||
subtree: true,
|
||||
childList: true,
|
||||
characterData: true,
|
||||
attributes: true,
|
||||
attributeFilter: [
|
||||
'aria-activedescendant',
|
||||
'aria-expanded',
|
||||
'aria-hidden',
|
||||
'aria-selected',
|
||||
'checked',
|
||||
'disabled',
|
||||
'hidden',
|
||||
'open',
|
||||
'selected',
|
||||
],
|
||||
}
|
||||
const mutationStates = (window.__simAgentMutationStates ??= [])
|
||||
let mutationState = observationRoot
|
||||
? mutationStates.find((state) => state.root === observationRoot)
|
||||
@@ -1925,33 +2099,29 @@ export function readPageActionState(resetMutationRevision = false, elementId?: n
|
||||
root: observationRoot,
|
||||
observer: null as unknown as MutationObserver,
|
||||
revision: 0,
|
||||
observedRoots: new WeakSet<ParentNode>(),
|
||||
}
|
||||
const state = mutationState
|
||||
state.observer = new MutationObserver((records) => {
|
||||
state.revision += records.length
|
||||
})
|
||||
for (const root of roots) {
|
||||
state.observer.observe(root, {
|
||||
subtree: true,
|
||||
childList: true,
|
||||
characterData: true,
|
||||
attributes: true,
|
||||
attributeFilter: [
|
||||
'aria-activedescendant',
|
||||
'aria-expanded',
|
||||
'aria-hidden',
|
||||
'aria-selected',
|
||||
'checked',
|
||||
'disabled',
|
||||
'hidden',
|
||||
'open',
|
||||
'selected',
|
||||
],
|
||||
})
|
||||
}
|
||||
mutationStates.push(state)
|
||||
if (mutationStates.length > 10) mutationStates.shift()?.observer.disconnect()
|
||||
}
|
||||
// Observe on EVERY call, not only the first. The roots list is rebuilt each
|
||||
// time and grows as shadow roots mount, so attaching once left every
|
||||
// component that appeared later unobserved — its DOM changes raised no
|
||||
// revision, and an action that mounted UI inside one reported no effect at
|
||||
// all. `observe` on an already-observed root with the same options is a
|
||||
// documented no-op, and observedRoots keeps the common case cheap.
|
||||
if (mutationState) {
|
||||
const state = mutationState
|
||||
for (const root of roots) {
|
||||
if (state.observedRoots.has(root)) continue
|
||||
state.observer.observe(root as Node, mutationOptions)
|
||||
state.observedRoots.add(root)
|
||||
}
|
||||
}
|
||||
if (resetMutationRevision) {
|
||||
mutationState?.observer.takeRecords()
|
||||
if (mutationState) mutationState.revision = 0
|
||||
@@ -2024,8 +2194,18 @@ export function readPageActionState(resetMutationRevision = false, elementId?: n
|
||||
.replace(/[\uD800-\uDBFF]$/, '')
|
||||
)
|
||||
|
||||
// Roles an app uses for something that APPEARS over the page. The first three
|
||||
// were the whole list, which missed the most common hover affordance there
|
||||
// is: a row's action bar (Slack's message shortcuts is role="toolbar"/"group"
|
||||
// with an aria-label). A hover that mounted one produced no popup change, no
|
||||
// target change, and so no observed effect at all — the agent concluded its
|
||||
// hover had failed and escalated to clicking pixels.
|
||||
const visiblePopupLabels = allElements
|
||||
.filter((element) => element.matches('[role="tooltip"], [role="menu"], [role="listbox"]'))
|
||||
.filter((element) =>
|
||||
element.matches(
|
||||
'[role="tooltip"], [role="menu"], [role="listbox"], [role="toolbar"], [role="menubar"], [role="group"][aria-label], [popover]'
|
||||
)
|
||||
)
|
||||
.filter((element) => {
|
||||
const rect = element.getBoundingClientRect()
|
||||
const view = element.ownerDocument.defaultView
|
||||
@@ -2254,7 +2434,8 @@ export function scrollPage(direction: string, amount?: number, elementId?: numbe
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(elementId)
|
||||
const element = resolver ? resolved?.element : (window.__simAgentElements || [])[elementId]
|
||||
if (!element || !element.isConnected) return { error: 'stale' }
|
||||
if (!element || !element.isConnected)
|
||||
return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
const candidates = ancestors(element)
|
||||
target = candidates.find(canMove) ?? candidates[0]
|
||||
source = target && canMove(target) ? 'element' : 'element-boundary'
|
||||
@@ -2361,7 +2542,7 @@ export function selectOptionInElement(id: number, value: string): unknown {
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
if (String(el.tagName || '').toUpperCase() !== 'SELECT') return { error: 'not-select' }
|
||||
const select = el as HTMLSelectElement
|
||||
if (select.disabled || select.getAttribute('aria-disabled') === 'true') {
|
||||
@@ -2401,7 +2582,7 @@ export function readSelectElementState(id: number): unknown {
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
if (String(el.tagName || '').toUpperCase() !== 'SELECT') return { error: 'not-select' }
|
||||
const select = el as HTMLSelectElement
|
||||
return {
|
||||
@@ -2414,7 +2595,7 @@ export function hoverElement(id: number): unknown {
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
el.scrollIntoView({ block: 'center', behavior: 'instant' })
|
||||
const rect = el.getBoundingClientRect()
|
||||
const opts = {
|
||||
@@ -2635,7 +2816,7 @@ export function readPageText(id?: number): unknown {
|
||||
const resolver = window.__simAgentResolveElement
|
||||
const resolved = resolver?.(id)
|
||||
const el = resolver ? resolved?.element : (window.__simAgentElements || [])[id]
|
||||
if (!el || !el.isConnected) return { error: 'stale' }
|
||||
if (!el || !el.isConnected) return { error: 'stale', reason: window.__simAgentStaleReason }
|
||||
text = (el as HTMLElement).innerText ?? el.textContent ?? ''
|
||||
} else {
|
||||
text = document.body?.innerText ?? ''
|
||||
@@ -2662,3 +2843,184 @@ export function getViewportInfo(): unknown {
|
||||
height: window.innerHeight,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes whatever sits at a viewport point, descending through open shadow
|
||||
* roots and same-origin iframes. Coordinate-addressed actions have no
|
||||
* snapshot ref to revalidate, so this probe is their safety check: the driver
|
||||
* refuses file inputs outright and reports what the point resolves to so the
|
||||
* model can confirm it hit what the screenshot showed.
|
||||
*/
|
||||
export function describePointTarget(x: number, y: number): unknown {
|
||||
if (
|
||||
!Number.isFinite(x) ||
|
||||
!Number.isFinite(y) ||
|
||||
x < 0 ||
|
||||
y < 0 ||
|
||||
x >= window.innerWidth ||
|
||||
y >= window.innerHeight
|
||||
) {
|
||||
return { error: 'outside-viewport' }
|
||||
}
|
||||
|
||||
let doc: Document = document
|
||||
let localX = x
|
||||
let localY = y
|
||||
let element: Element | null = null
|
||||
for (let depth = 0; depth < 10; depth++) {
|
||||
if (typeof doc.elementFromPoint !== 'function') break
|
||||
let found: Element | null = doc.elementFromPoint(localX, localY)
|
||||
// Open shadow roots re-hit-test at the same point until a leaf host.
|
||||
for (let shadowDepth = 0; shadowDepth < 10; shadowDepth++) {
|
||||
const shadow = (found as HTMLElement | null)?.shadowRoot
|
||||
const inner = shadow?.elementFromPoint(localX, localY)
|
||||
if (!inner || inner === found) break
|
||||
found = inner
|
||||
}
|
||||
element = found
|
||||
const tag = String(found?.tagName || '').toUpperCase()
|
||||
if ((tag !== 'IFRAME' && tag !== 'FRAME') || !found) break
|
||||
try {
|
||||
const innerDoc = (found as HTMLIFrameElement).contentDocument
|
||||
if (!innerDoc) break
|
||||
const rect = found.getBoundingClientRect()
|
||||
localX -= rect.left + (found as HTMLIFrameElement).clientLeft
|
||||
localY -= rect.top + (found as HTMLIFrameElement).clientTop
|
||||
doc = innerDoc
|
||||
} catch {
|
||||
// Cross-origin frame — cannot inspect further; report the frame itself.
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!element) return { found: false }
|
||||
|
||||
const tag = String(element.tagName || '').toUpperCase()
|
||||
const inputType =
|
||||
tag === 'INPUT' ? String((element as HTMLInputElement).type || 'text').toLowerCase() : ''
|
||||
const secret =
|
||||
tag === 'INPUT' &&
|
||||
(inputType === 'password' ||
|
||||
String(element.getAttribute('autocomplete') || '')
|
||||
.toLowerCase()
|
||||
.split(/\s+/)
|
||||
.some((token) => token === 'current-password' || token === 'new-password'))
|
||||
const editable = Boolean(
|
||||
tag === 'TEXTAREA' ||
|
||||
(tag === 'INPUT' &&
|
||||
['text', 'search', 'email', 'url', 'tel', 'number', 'password'].includes(inputType)) ||
|
||||
(element as HTMLElement).isContentEditable
|
||||
)
|
||||
|
||||
const name = (
|
||||
element.getAttribute('aria-label') ||
|
||||
element.getAttribute('title') ||
|
||||
element.getAttribute('alt') ||
|
||||
((element as HTMLElement).innerText ?? element.textContent ?? '')
|
||||
)
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.slice(0, 80)
|
||||
const role = element.getAttribute('role') || ''
|
||||
const style = element.ownerDocument.defaultView?.getComputedStyle(element)
|
||||
|
||||
return {
|
||||
found: true,
|
||||
element: name
|
||||
? `${tag.toLowerCase()}${role ? `[${role}]` : ''} "${name}"`
|
||||
: `${tag.toLowerCase()}${role ? `[${role}]` : ''}`,
|
||||
tag: tag.toLowerCase(),
|
||||
role,
|
||||
editable,
|
||||
secret,
|
||||
fileInput: tag === 'INPUT' && inputType === 'file',
|
||||
disabled:
|
||||
(element as HTMLInputElement).disabled === true ||
|
||||
element.getAttribute('aria-disabled') === 'true',
|
||||
canvas: tag === 'CANVAS',
|
||||
crossOriginFrame: tag === 'IFRAME' || tag === 'FRAME',
|
||||
cursor: style?.cursor || '',
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether the currently-focused element accepts text insertion, for
|
||||
* browser_insert_text — which types at the caret instead of addressing a
|
||||
* snapshot ref. Secrecy is separately (and authoritatively) probed by
|
||||
* activeElementSecrecy before any insertion.
|
||||
*/
|
||||
export function describeFocusedEditable(): unknown {
|
||||
// Descend shadow roots AND same-origin frames, matching activeElementReadback
|
||||
// exactly. Focus inside a frame surfaces on the outer document as the FRAME
|
||||
// element, which is not an input, not contentEditable, not a canvas and has no
|
||||
// textbox role — so stopping here reported a perfectly writable field as
|
||||
// `not-editable`, while press-key (which does descend) typed into it fine.
|
||||
// Any divergence between these two loops is a tool that refuses what its
|
||||
// sibling accepts on the same page state.
|
||||
let active = document.activeElement as HTMLElement | null
|
||||
for (let depth = 0; active && depth < 10; depth++) {
|
||||
const shadow = active.shadowRoot
|
||||
if (shadow?.activeElement) {
|
||||
active = shadow.activeElement as HTMLElement
|
||||
continue
|
||||
}
|
||||
const activeTag = String(active.tagName || '').toUpperCase()
|
||||
if (activeTag === 'IFRAME' || activeTag === 'FRAME') {
|
||||
try {
|
||||
const inner = (active as HTMLIFrameElement).contentDocument
|
||||
if (inner?.activeElement && inner.activeElement !== inner.body) {
|
||||
active = inner.activeElement as HTMLElement
|
||||
continue
|
||||
}
|
||||
} catch {
|
||||
// Cross-origin frame — not inspectable. The caller refuses separately on
|
||||
// opaque secrecy, so report the frame itself rather than guessing.
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
if (!active || active === document.body) return { editable: false, reason: 'none' }
|
||||
const tag = String(active.tagName || '').toUpperCase()
|
||||
const inputType =
|
||||
tag === 'INPUT' ? String((active as HTMLInputElement).type || 'text').toLowerCase() : ''
|
||||
if (tag === 'INPUT' || tag === 'TEXTAREA') {
|
||||
const field = active as HTMLInputElement | HTMLTextAreaElement
|
||||
if (field.disabled || active.getAttribute('aria-disabled') === 'true') {
|
||||
return { editable: false, reason: 'disabled' }
|
||||
}
|
||||
if (field.readOnly || active.getAttribute('aria-readonly') === 'true') {
|
||||
return { editable: false, reason: 'readonly' }
|
||||
}
|
||||
if (
|
||||
tag === 'INPUT' &&
|
||||
!['text', 'search', 'email', 'url', 'tel', 'number', 'password'].includes(inputType)
|
||||
) {
|
||||
return { editable: false, reason: 'not-text' }
|
||||
}
|
||||
return { editable: true, kind: tag === 'TEXTAREA' ? 'textarea' : `input:${inputType}` }
|
||||
}
|
||||
if (active.isContentEditable) {
|
||||
if (active.getAttribute('aria-disabled') === 'true') {
|
||||
return { editable: false, reason: 'disabled' }
|
||||
}
|
||||
if (active.getAttribute('aria-readonly') === 'true') {
|
||||
return { editable: false, reason: 'readonly' }
|
||||
}
|
||||
return { editable: true, kind: 'contenteditable' }
|
||||
}
|
||||
// A canvas-rendered editor (Google Docs) focuses a hidden proxy or the
|
||||
// canvas region itself; trusted IME insertion still reaches it, so report
|
||||
// it as insertable rather than refusing.
|
||||
if (tag === 'CANVAS' || active.getAttribute('role') === 'textbox') {
|
||||
return { editable: true, kind: tag === 'CANVAS' ? 'canvas' : 'textbox-role' }
|
||||
}
|
||||
// Describe what actually held focus. A bare "not-editable" tells the agent
|
||||
// nothing it can act on, so it guesses at the cause and burns rounds on the
|
||||
// wrong recovery; naming the element lets it click the real field instead.
|
||||
return {
|
||||
editable: false,
|
||||
reason: 'not-editable',
|
||||
focusedTag: tag.toLowerCase(),
|
||||
...(active.getAttribute('role') ? { focusedRole: active.getAttribute('role') } : {}),
|
||||
contentEditable: String(active.getAttribute('contenteditable') ?? 'unset'),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,6 +53,30 @@ describe('panel chat scope', () => {
|
||||
panel = freshPanel()
|
||||
})
|
||||
|
||||
it('returns keyboard focus to the renderer when attaching a view steals it mid-typing', () => {
|
||||
const win = new BrowserWindow()
|
||||
const view = new WebContentsView()
|
||||
const active = { id: 'tab-1', scopeId: 'chat-test', view, pinned: false }
|
||||
vi.mocked(win.webContents.isFocused).mockReturnValue(true)
|
||||
panel.initPanel({
|
||||
getMainWindow: () => win,
|
||||
activeTab: () => active,
|
||||
backgroundColor: () => '#0c0c0c',
|
||||
ensureInitialTab: () => {},
|
||||
onViewDetached: () => {},
|
||||
})
|
||||
panel.activatePanelScope('chat-test')
|
||||
panel.setPanelBounds(PANEL_RECT, win)
|
||||
expect(win.contentView.addChildView).toHaveBeenCalledWith(view)
|
||||
expect(win.webContents.focus).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('leaves focus untouched when the renderer was not focused at attach time', () => {
|
||||
const { win, view } = showPanel(panel)
|
||||
expect(win.contentView.addChildView).toHaveBeenCalledWith(view)
|
||||
expect(win.webContents.focus).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('requires fresh bounds for the newly active chat and ignores stale reports', () => {
|
||||
const { win, view } = showPanel(panel)
|
||||
const previousScope = panel.getActivePanelScopeId()
|
||||
|
||||
@@ -20,6 +20,7 @@ import { getErrorMessage } from '@sim/utils/errors'
|
||||
import type { BrowserWindow, WebContentsView } from 'electron'
|
||||
import { zoomPercentOf } from '@/main/browser-agent/context-menu'
|
||||
import type { AgentTab } from '@/main/browser-agent/session'
|
||||
import { reassertTabThrottling } from '@/main/browser-agent/session'
|
||||
|
||||
const logger = createLogger('BrowserAgentPanel')
|
||||
|
||||
@@ -379,9 +380,17 @@ export function layout(): void {
|
||||
}
|
||||
|
||||
if (attachedView !== active.view) {
|
||||
// addChildView hands keyboard focus to the newly attached WebContentsView.
|
||||
// Agent-driven attaches happen while the user may be typing in the chat
|
||||
// composer, so if the renderer held focus before the attach, give it back —
|
||||
// automation drives the page over CDP and never needs OS focus.
|
||||
const rendererHadFocus = !win.webContents.isDestroyed() && win.webContents.isFocused()
|
||||
win.contentView.addChildView(active.view)
|
||||
hostedWindow = win
|
||||
attachedView = active.view
|
||||
if (rendererHadFocus) {
|
||||
win.webContents.focus()
|
||||
}
|
||||
}
|
||||
bindHostResize(win)
|
||||
const zoom = win.webContents.getZoomFactor()
|
||||
@@ -412,7 +421,19 @@ export function layout(): void {
|
||||
lastAppliedVisibility = visible
|
||||
active.view.setVisible(visible)
|
||||
if (visible && !active.view.webContents.isDestroyed()) {
|
||||
active.view.webContents.invalidate()
|
||||
// invalidate() recomposites the LAST frame — which is blank when the
|
||||
// page finished loading while this view was hidden and background
|
||||
// throttling suspended the rAF its SPA paints from. The page then sits
|
||||
// "loaded" but white until the user re-navigates by hand. Pulse
|
||||
// throttling off so the renderer actually produces a first frame, then
|
||||
// hand the policy back to the session (which keeps the automation-tab
|
||||
// exemption intact).
|
||||
const contents = active.view.webContents
|
||||
contents.setBackgroundThrottling(false)
|
||||
contents.invalidate()
|
||||
setTimeout(() => {
|
||||
if (!contents.isDestroyed()) reassertTabThrottling()
|
||||
}, 1_000)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1715,6 +1715,21 @@ describe('browser-agent session', () => {
|
||||
expect(contents.loadURL).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('brings an agent-opened working tab into view, unless the user claimed the visible one', () => {
|
||||
// browser_open_tab is the agent choosing a page to work in — the panel
|
||||
// follows it so the work is visible, which a popup deliberately does not.
|
||||
const first = session.ensureTab()
|
||||
const working = session.addAutomationTab({ reveal: true })
|
||||
expect(session.activeTab()).toBe(working)
|
||||
expect(working.id).not.toBe(first.id)
|
||||
|
||||
// Once the user claims what they are looking at, the next agent tab opens
|
||||
// behind it rather than yanking the page out from under them.
|
||||
session.claimActiveTabForUser()
|
||||
const background = session.addAutomationTab({ reveal: true })
|
||||
expect(session.activeTab()).not.toBe(background)
|
||||
})
|
||||
|
||||
it('keeps agent popups in the background and context-menu links user-owned', () => {
|
||||
const onTabCreated = vi.fn()
|
||||
session = freshSession(win, { onTabCreated })
|
||||
@@ -1770,28 +1785,39 @@ describe('browser-agent session', () => {
|
||||
expect(event.preventDefault).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('permission handlers deny every request on the agent partition but the copy button', () => {
|
||||
it('permission handlers deny everything on the agent partition but the copy button and media', async () => {
|
||||
const tab = session.ensureTab()
|
||||
const ses = (tab.view as unknown as MockView).webContents.session
|
||||
const requestHandler = ses.setPermissionRequestHandler.mock.calls[0][0] as (
|
||||
wc: unknown,
|
||||
permission: string,
|
||||
callback: (granted: boolean) => void
|
||||
callback: (granted: boolean) => void,
|
||||
details?: unknown
|
||||
) => void
|
||||
const checkHandler = ses.setPermissionCheckHandler.mock.calls[0][0] as (
|
||||
wc: unknown,
|
||||
permission: string
|
||||
permission: string,
|
||||
origin?: string,
|
||||
details?: unknown
|
||||
) => boolean
|
||||
|
||||
// Reading the clipboard would leak whatever the user last copied anywhere
|
||||
// else, so it stays denied alongside everything a page could spy through.
|
||||
for (const permission of ['media', 'geolocation', 'notifications', 'clipboard-read']) {
|
||||
for (const permission of ['geolocation', 'notifications', 'clipboard-read']) {
|
||||
const callback = vi.fn()
|
||||
requestHandler(null, permission, callback)
|
||||
expect(callback).toHaveBeenCalledWith(false)
|
||||
expect(checkHandler(null, permission)).toBe(false)
|
||||
}
|
||||
|
||||
// Media is the deliberate exception — the agent browser joins real
|
||||
// meetings — but the grant is gated on the OS grant (mocked as granted
|
||||
// here), so System Settings remains the real authority.
|
||||
const mediaCallback = vi.fn()
|
||||
requestHandler(null, 'media', mediaCallback, { mediaTypes: ['audio', 'video'] })
|
||||
await vi.waitFor(() => expect(mediaCallback).toHaveBeenCalledWith(true))
|
||||
expect(checkHandler(null, 'media', undefined, { mediaType: 'audio' })).toBe(true)
|
||||
|
||||
// Chromium routes navigator.clipboard.writeText through this one; denying
|
||||
// it silently broke every copy button that does not use execCommand.
|
||||
const writeCallback = vi.fn()
|
||||
|
||||
@@ -35,6 +35,7 @@ import {
|
||||
Menu,
|
||||
nativeTheme,
|
||||
shell,
|
||||
systemPreferences,
|
||||
WebContentsView,
|
||||
} from 'electron'
|
||||
import { isDispatchingAgentInput } from '@/main/browser-agent/cdp'
|
||||
@@ -826,19 +827,61 @@ export async function importAgentCookies(
|
||||
*/
|
||||
const ALLOWED_SITE_PERMISSIONS = new Set(['clipboard-sanitized-write'])
|
||||
|
||||
/**
|
||||
* Grants a getUserMedia request only when macOS has actually authorized the
|
||||
* devices it names. Granting site permission without the OS grant makes the
|
||||
* page fail with a misleading NotReadableError instead of a permission prompt,
|
||||
* and macOS kills the process outright when the bundle lacks usage strings —
|
||||
* so the OS is asked FIRST, which surfaces the system prompt on first use.
|
||||
*/
|
||||
async function ensureOsMediaAccess(mediaTypes: readonly string[] | undefined): Promise<boolean> {
|
||||
if (process.platform !== 'darwin') return true
|
||||
const wanted = mediaTypes && mediaTypes.length > 0 ? mediaTypes : ['audio', 'video']
|
||||
for (const type of wanted) {
|
||||
const device = type === 'video' ? 'camera' : 'microphone'
|
||||
if (systemPreferences.getMediaAccessStatus(device) === 'granted') continue
|
||||
const granted = await systemPreferences.askForMediaAccess(device).catch(() => false)
|
||||
if (!granted) return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Default-deny hardening for the agent partition. Site permissions remain
|
||||
* denied apart from ALLOWED_SITE_PERMISSIONS, while uploads use Chromium's
|
||||
* native file chooser and downloads are saved into the device-level browser
|
||||
* download directory.
|
||||
* denied apart from ALLOWED_SITE_PERMISSIONS and camera/microphone — the agent
|
||||
* browser has to join a Google Meet or a Zoom web client like a real browser,
|
||||
* and those are dead without getUserMedia. The OS grant still gates every
|
||||
* media request, so the user's System Settings choice is the real authority.
|
||||
* Uploads use Chromium's native file chooser and downloads are saved into the
|
||||
* device-level browser download directory.
|
||||
*/
|
||||
function configureAgentPartition(ses: Session): void {
|
||||
if (configuredPartitions.has(ses)) return
|
||||
configuredPartitions.add(ses)
|
||||
ses.setPermissionRequestHandler((_wc, permission, callback) =>
|
||||
ses.setPermissionRequestHandler((_wc, permission, callback, details) => {
|
||||
if (permission === 'media') {
|
||||
const mediaTypes = (details as { mediaTypes?: readonly string[] })?.mediaTypes
|
||||
void ensureOsMediaAccess(mediaTypes).then(callback)
|
||||
return
|
||||
}
|
||||
callback(ALLOWED_SITE_PERMISSIONS.has(permission))
|
||||
)
|
||||
ses.setPermissionCheckHandler((_wc, permission) => ALLOWED_SITE_PERMISSIONS.has(permission))
|
||||
})
|
||||
ses.setPermissionCheckHandler((_wc, permission, _origin, details) => {
|
||||
if (permission === 'media') {
|
||||
if (process.platform !== 'darwin') return true
|
||||
const mediaType = (details as { mediaType?: string })?.mediaType
|
||||
if (mediaType === 'video')
|
||||
return systemPreferences.getMediaAccessStatus('camera') === 'granted'
|
||||
if (mediaType === 'audio') {
|
||||
return systemPreferences.getMediaAccessStatus('microphone') === 'granted'
|
||||
}
|
||||
return (
|
||||
systemPreferences.getMediaAccessStatus('microphone') === 'granted' ||
|
||||
systemPreferences.getMediaAccessStatus('camera') === 'granted'
|
||||
)
|
||||
}
|
||||
return ALLOWED_SITE_PERMISSIONS.has(permission)
|
||||
})
|
||||
// Service workers do not inherit a tab's user agent. With only the tab's set,
|
||||
// the document request carries the browser string while the worker's own
|
||||
// script request still announces Electron — and on a site that routes its
|
||||
@@ -1359,6 +1402,15 @@ export function setAutomationNeedsAttention(needsAttention: boolean): void {
|
||||
* other tab. Call after anything that changes which tab is active, so the
|
||||
* exemption follows the active tab rather than being stranded on the old one.
|
||||
*/
|
||||
/**
|
||||
* Re-applies the tab throttling policy after a caller temporarily suspended it
|
||||
* (the panel's reveal pulse). Exempts the automation-active tab exactly as the
|
||||
* internal policy does.
|
||||
*/
|
||||
export function reassertTabThrottling(): void {
|
||||
applyActiveTabThrottling()
|
||||
}
|
||||
|
||||
function applyActiveTabThrottling(): void {
|
||||
for (const tab of tabs) {
|
||||
if (tab.view.webContents.isDestroyed()) continue
|
||||
@@ -1590,10 +1642,24 @@ export function addTab(): AgentTab {
|
||||
return addTabInternal()
|
||||
}
|
||||
|
||||
/** Opens a tab for agent work without replacing the page the user is viewing. */
|
||||
export function addAutomationTab(): AgentTab {
|
||||
/**
|
||||
* Opens a tab for agent work.
|
||||
*
|
||||
* `reveal` is for the agent deliberately opening a page to work in
|
||||
* (`browser_open_tab`): the panel follows it, so the user watches the work
|
||||
* instead of staring at a page where nothing is happening. It is NOT set when a
|
||||
* page spawns a tab on its own (popups, `target="_blank"`) — that is the site
|
||||
* grabbing the view, not the agent choosing a workspace.
|
||||
*
|
||||
* Even with `reveal`, a tab the user claimed themselves wins: pulling the view
|
||||
* off the page they are reading is the same interruption as a window stealing
|
||||
* focus mid-sentence. The work still starts, just in the background, and the
|
||||
* tab strip shows it arriving.
|
||||
*/
|
||||
export function addAutomationTab({ reveal = false }: { reveal?: boolean } = {}): AgentTab {
|
||||
restoreBrowserSession()
|
||||
const tab = addTabInternal({ activate: false, notify: false })
|
||||
const followTheWork = reveal && !currentScope.visibleTabUserSelected
|
||||
const tab = addTabInternal({ activate: followTheWork, notify: false })
|
||||
currentScope.automationTabId = tab.id
|
||||
applyActiveTabThrottling()
|
||||
persistBrowserSession()
|
||||
@@ -1950,6 +2016,9 @@ export function handleFocusedShortcut(
|
||||
case 'reload-or-clear':
|
||||
shortcutTab.view.webContents.reload()
|
||||
return true
|
||||
case 'hard-reload':
|
||||
shortcutTab.view.webContents.reloadIgnoringCache()
|
||||
return true
|
||||
}
|
||||
|
||||
const zoomAction = zoomActionForShortcut(shortcut)
|
||||
@@ -2194,6 +2263,13 @@ export function automationTab(): AgentTab | null {
|
||||
return tab
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the user has interacted with the visible tab while it is also the
|
||||
* agent's automation tab. Governs panel-level ownership only — popup
|
||||
* adoption and protection against the agent closing the tab out from under
|
||||
* the user. It must never gate agent input: the user clicking or typing in
|
||||
* the panel does not revoke the agent's ability to act there.
|
||||
*/
|
||||
export function automationTabClaimedByUser(): boolean {
|
||||
return (
|
||||
currentScope.visibleTabUserSelected &&
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
parseSearchSuggestionResponse,
|
||||
SearchSuggestionService,
|
||||
} from '@/main/browser-search/suggestions'
|
||||
|
||||
describe('parseSearchSuggestionResponse', () => {
|
||||
it('keeps unique non-empty completions and omits the exact query row', () => {
|
||||
expect(
|
||||
parseSearchSuggestionResponse(
|
||||
[
|
||||
'what is the be',
|
||||
[
|
||||
'what is the be',
|
||||
'what is the best sleeping position',
|
||||
' WHAT IS THE BEST SLEEPING POSITION ',
|
||||
'',
|
||||
42,
|
||||
'what is the benefit of creatine',
|
||||
],
|
||||
],
|
||||
'what is the be'
|
||||
)
|
||||
).toEqual(['what is the best sleeping position', 'what is the benefit of creatine'])
|
||||
})
|
||||
|
||||
it('rejects malformed provider payloads', () => {
|
||||
expect(parseSearchSuggestionResponse(null, 'sim')).toEqual([])
|
||||
expect(parseSearchSuggestionResponse(['sim', {}], 'sim')).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('SearchSuggestionService', () => {
|
||||
it('uses only the fixed provider URL and caches repeated queries', async () => {
|
||||
const fetcher = vi.fn(
|
||||
async (_url: string, _init: RequestInit): Promise<Response> =>
|
||||
Response.json(['sim studio', ['sim studio ai', 'sim studio workflow']])
|
||||
)
|
||||
const service = new SearchSuggestionService(fetcher, () => 1_000)
|
||||
|
||||
await expect(service.suggest(' sim studio ')).resolves.toEqual([
|
||||
'sim studio ai',
|
||||
'sim studio workflow',
|
||||
])
|
||||
await expect(service.suggest('SIM STUDIO')).resolves.toEqual([
|
||||
'sim studio ai',
|
||||
'sim studio workflow',
|
||||
])
|
||||
|
||||
expect(fetcher).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetcher.mock.calls[0]
|
||||
expect(new URL(url).origin).toBe('https://suggestqueries.google.com')
|
||||
expect(new URL(url).searchParams.get('q')).toBe('sim studio')
|
||||
expect(init).toMatchObject({ method: 'GET', redirect: 'error' })
|
||||
})
|
||||
|
||||
it('fails silently for short, oversized, rejected, and unsuccessful requests', async () => {
|
||||
const fetcher = vi.fn(
|
||||
async (_url: string, _init: RequestInit): Promise<Response> =>
|
||||
new Response(null, { status: 503 })
|
||||
)
|
||||
const service = new SearchSuggestionService(fetcher)
|
||||
|
||||
await expect(service.suggest('s')).resolves.toEqual([])
|
||||
await expect(service.suggest('x'.repeat(201))).resolves.toEqual([])
|
||||
await expect(service.suggest('search me')).resolves.toEqual([])
|
||||
|
||||
fetcher.mockRejectedValueOnce(new Error('offline'))
|
||||
await expect(service.suggest('another search')).resolves.toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,102 @@
|
||||
import { net } from 'electron'
|
||||
|
||||
const GOOGLE_SUGGESTIONS_ENDPOINT = 'https://suggestqueries.google.com/complete/search'
|
||||
const SEARCH_SUGGESTION_TIMEOUT_MS = 2_500
|
||||
const SEARCH_SUGGESTION_CACHE_TTL_MS = 5 * 60 * 1_000
|
||||
const MAX_SEARCH_SUGGESTION_CACHE_ENTRIES = 100
|
||||
const MAX_SEARCH_SUGGESTION_QUERY_LENGTH = 200
|
||||
const MAX_SEARCH_SUGGESTION_LENGTH = 256
|
||||
const MAX_SEARCH_SUGGESTIONS = 7
|
||||
|
||||
type SearchSuggestionFetch = (url: string, init: RequestInit) => Promise<Response>
|
||||
|
||||
interface CachedSearchSuggestions {
|
||||
expiresAt: number
|
||||
values: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates the small portion of Google's Firefox-completion response that the
|
||||
* omnibox consumes. Everything else in the provider payload is ignored.
|
||||
*/
|
||||
export function parseSearchSuggestionResponse(payload: unknown, query: string): string[] {
|
||||
if (!Array.isArray(payload) || !Array.isArray(payload[1])) return []
|
||||
|
||||
const queryKey = query.toLocaleLowerCase()
|
||||
const seen = new Set([queryKey])
|
||||
const suggestions: string[] = []
|
||||
for (const candidate of payload[1]) {
|
||||
if (typeof candidate !== 'string') continue
|
||||
const value = candidate.trim()
|
||||
const key = value.toLocaleLowerCase()
|
||||
if (!value || value.length > MAX_SEARCH_SUGGESTION_LENGTH || seen.has(key)) continue
|
||||
seen.add(key)
|
||||
suggestions.push(value)
|
||||
if (suggestions.length === MAX_SEARCH_SUGGESTIONS) break
|
||||
}
|
||||
return suggestions
|
||||
}
|
||||
|
||||
/**
|
||||
* In-memory, bounded search completion client. Queries go only to the fixed
|
||||
* Google suggestions origin and are never logged or written to disk.
|
||||
*/
|
||||
export class SearchSuggestionService {
|
||||
private readonly cache = new Map<string, CachedSearchSuggestions>()
|
||||
|
||||
constructor(
|
||||
private readonly fetcher: SearchSuggestionFetch,
|
||||
private readonly now: () => number = Date.now
|
||||
) {}
|
||||
|
||||
async suggest(rawQuery: unknown): Promise<string[]> {
|
||||
if (typeof rawQuery !== 'string') return []
|
||||
const query = rawQuery.trim()
|
||||
if (query.length < 2 || query.length > MAX_SEARCH_SUGGESTION_QUERY_LENGTH) return []
|
||||
|
||||
const cacheKey = query.toLocaleLowerCase()
|
||||
const cached = this.cache.get(cacheKey)
|
||||
if (cached && cached.expiresAt > this.now()) return [...cached.values]
|
||||
if (cached) this.cache.delete(cacheKey)
|
||||
|
||||
const url = new URL(GOOGLE_SUGGESTIONS_ENDPOINT)
|
||||
url.searchParams.set('client', 'firefox')
|
||||
url.searchParams.set('q', query)
|
||||
const controller = new AbortController()
|
||||
const timeout = setTimeout(() => controller.abort(), SEARCH_SUGGESTION_TIMEOUT_MS)
|
||||
|
||||
try {
|
||||
const response = await this.fetcher(url.toString(), {
|
||||
method: 'GET',
|
||||
redirect: 'error',
|
||||
signal: controller.signal,
|
||||
})
|
||||
if (!response.ok) return []
|
||||
const values = parseSearchSuggestionResponse(await response.json(), query)
|
||||
this.remember(cacheKey, values)
|
||||
return [...values]
|
||||
} catch {
|
||||
return []
|
||||
} finally {
|
||||
clearTimeout(timeout)
|
||||
}
|
||||
}
|
||||
|
||||
private remember(key: string, values: string[]): void {
|
||||
if (this.cache.size >= MAX_SEARCH_SUGGESTION_CACHE_ENTRIES) {
|
||||
const oldest = this.cache.keys().next().value
|
||||
if (typeof oldest === 'string') this.cache.delete(oldest)
|
||||
}
|
||||
this.cache.set(key, {
|
||||
expiresAt: this.now() + SEARCH_SUGGESTION_CACHE_TTL_MS,
|
||||
values: [...values],
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const searchSuggestionService = new SearchSuggestionService((url, init) => net.fetch(url, init))
|
||||
|
||||
/** Fetches live Google completions, failing closed to an empty local-only list. */
|
||||
export function getSearchSuggestions(query: unknown): Promise<string[]> {
|
||||
return searchSuggestionService.suggest(query)
|
||||
}
|
||||
@@ -101,6 +101,8 @@ export interface DesktopSettings {
|
||||
launchAtLogin?: boolean
|
||||
autoDownloadUpdates?: boolean
|
||||
browserEnabled?: boolean
|
||||
/** Whether omnibox typing may request live Google search completions. */
|
||||
browserSearchSuggestionsEnabled?: boolean
|
||||
terminalEnabled?: boolean
|
||||
/** Device-wide browser page appearance; `app` follows Sim. */
|
||||
browserTheme?: 'app' | 'light' | 'dark'
|
||||
@@ -216,6 +218,7 @@ const DEFAULT_SETTINGS: DesktopSettings = {
|
||||
launchAtLogin: false,
|
||||
autoDownloadUpdates: true,
|
||||
browserEnabled: true,
|
||||
browserSearchSuggestionsEnabled: true,
|
||||
terminalEnabled: true,
|
||||
}
|
||||
|
||||
|
||||
@@ -117,6 +117,17 @@ describe('desktop settings service', () => {
|
||||
expect(setTerminalEnabled).toHaveBeenCalledWith(false)
|
||||
})
|
||||
|
||||
it('defaults live browser search suggestions on and persists the privacy switch', () => {
|
||||
const { config, service } = makeService()
|
||||
|
||||
expect(service.getPreferences().browserSearchSuggestionsEnabled).toBe(true)
|
||||
|
||||
const preferences = service.setBrowserSearchSuggestionsEnabled(false)
|
||||
|
||||
expect(config.get('browserSearchSuggestionsEnabled')).toBe(false)
|
||||
expect(preferences.browserSearchSuggestionsEnabled).toBe(false)
|
||||
})
|
||||
|
||||
it('persists browser and terminal appearance with match-Sim defaults', () => {
|
||||
const { config, service, setBrowserTheme, onBrowserThemeChanged } = makeService()
|
||||
expect(service.getPreferences()).toMatchObject({
|
||||
|
||||
@@ -36,6 +36,7 @@ export function isDesktopPreferenceKey(value: unknown): value is DesktopPreferen
|
||||
export interface DesktopSettingsService {
|
||||
getPreferences(): DesktopPreferences
|
||||
setPreference(key: DesktopPreferenceKey, value: boolean): DesktopPreferences
|
||||
setBrowserSearchSuggestionsEnabled(enabled: boolean): DesktopPreferences
|
||||
setAppearancePreference(
|
||||
key: DesktopAppearanceSettingKey,
|
||||
value: DesktopAppearanceTheme
|
||||
@@ -90,6 +91,7 @@ function readPreferences(
|
||||
autoDownloadUpdates: config.get('autoDownloadUpdates') ?? true,
|
||||
trayEnabled: config.get('trayEnabled') ?? true,
|
||||
browserEnabled: config.get('browserEnabled') ?? true,
|
||||
browserSearchSuggestionsEnabled: config.get('browserSearchSuggestionsEnabled') ?? true,
|
||||
terminalEnabled: config.get('terminalEnabled') ?? true,
|
||||
browserTheme: isDesktopAppearanceTheme(browserTheme) ? browserTheme : 'app',
|
||||
browserDefaultZoom: isDesktopZoomPercent(browserDefaultZoom) ? browserDefaultZoom : 100,
|
||||
@@ -153,6 +155,11 @@ export function createDesktopSettingsService(
|
||||
}
|
||||
return read()
|
||||
},
|
||||
setBrowserSearchSuggestionsEnabled(enabled) {
|
||||
deps.config.set('browserSearchSuggestionsEnabled', enabled)
|
||||
deps.config.flush()
|
||||
return read()
|
||||
},
|
||||
setAppearancePreference(key, value) {
|
||||
const previousBrowserTheme = key === 'browserTheme' ? read().browserTheme : undefined
|
||||
deps.config.set(key, value)
|
||||
|
||||
@@ -18,6 +18,10 @@ vi.mock('@/main/browser-import', () => ({
|
||||
})),
|
||||
}))
|
||||
|
||||
vi.mock('@/main/browser-search/suggestions', () => ({
|
||||
getSearchSuggestions: vi.fn(async () => ['sim ai workflow']),
|
||||
}))
|
||||
|
||||
const { terminalThemeProfile } = vi.hoisted(() => ({
|
||||
terminalThemeProfile: {
|
||||
id: 'iterm2:ocean',
|
||||
@@ -124,6 +128,7 @@ import {
|
||||
importChromePasswords,
|
||||
listChromeImportProfiles,
|
||||
} from '@/main/browser-import'
|
||||
import { getSearchSuggestions } from '@/main/browser-search/suggestions'
|
||||
import { trackInputActivity } from '@/main/input-activity'
|
||||
import { type IpcDeps, registerIpcHandlers } from '@/main/ipc'
|
||||
import { LocalFilesystemService } from '@/main/local-filesystem'
|
||||
@@ -252,6 +257,7 @@ describe('registerIpcHandlers', () => {
|
||||
vi.mocked(listChromeImportProfiles).mockClear()
|
||||
vi.mocked(importChromeCookies).mockClear()
|
||||
vi.mocked(importChromePasswords).mockClear()
|
||||
vi.mocked(getSearchSuggestions).mockClear()
|
||||
vi.mocked(findCachedTerminalThemeProfile).mockClear()
|
||||
vi.mocked(listTerminalThemeProfiles).mockClear()
|
||||
vi.mocked(credentialsAvailable).mockClear()
|
||||
@@ -282,6 +288,7 @@ describe('registerIpcHandlers', () => {
|
||||
settings: {
|
||||
getPreferences: vi.fn(() => DEFAULT_DESKTOP_PREFERENCES),
|
||||
setPreference: vi.fn(),
|
||||
setBrowserSearchSuggestionsEnabled: vi.fn(),
|
||||
setAppearancePreference: vi.fn(),
|
||||
setBrowserDefaultZoom: vi.fn(),
|
||||
setTerminalDefaultZoom: vi.fn(),
|
||||
@@ -325,6 +332,22 @@ describe('registerIpcHandlers', () => {
|
||||
expect(shell.openExternal).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('keeps live search suggestions behind the app origin and privacy preference', async () => {
|
||||
const { invoke } = collectHandlers()
|
||||
const handler = invoke.get('browser-agent:search-suggestions')
|
||||
|
||||
expect(await handler?.(evilEvent, 'sim ai')).toEqual([])
|
||||
|
||||
expect(await handler?.(appEvent, 'sim ai')).toEqual(['sim ai workflow'])
|
||||
expect(getSearchSuggestions).toHaveBeenCalledWith('sim ai')
|
||||
|
||||
vi.mocked(deps.settings.getPreferences).mockReturnValue({
|
||||
...DEFAULT_DESKTOP_PREFERENCES,
|
||||
browserSearchSuggestionsEnabled: false,
|
||||
})
|
||||
expect(await handler?.(appEvent, 'sim ai')).toEqual([])
|
||||
})
|
||||
|
||||
it('restricts the OAuth connect handoff to the app origin', async () => {
|
||||
const { invoke } = collectHandlers()
|
||||
const handler = invoke.get('desktop:oauth-connect')
|
||||
|
||||
@@ -70,6 +70,7 @@ import {
|
||||
importChromePasswords,
|
||||
listChromeImportProfiles,
|
||||
} from '@/main/browser-import'
|
||||
import { getSearchSuggestions } from '@/main/browser-search/suggestions'
|
||||
import { listSites } from '@/main/browser-sites'
|
||||
import { isSafeInternalPath } from '@/main/config'
|
||||
import type { DesktopSettingsService } from '@/main/desktop-settings'
|
||||
@@ -661,6 +662,15 @@ export function registerIpcHandlers(deps: IpcDeps): void {
|
||||
? deps.settings.setPreference(key, value)
|
||||
: deps.settings.getPreferences(),
|
||||
},
|
||||
'desktop:settings:set-browser-search-suggestions': {
|
||||
kind: 'invoke',
|
||||
gate: 'app-origin',
|
||||
denied: null,
|
||||
handler: (enabled) =>
|
||||
typeof enabled === 'boolean'
|
||||
? deps.settings.setBrowserSearchSuggestionsEnabled(enabled)
|
||||
: deps.settings.getPreferences(),
|
||||
},
|
||||
'desktop:settings:set-appearance': {
|
||||
kind: 'invoke',
|
||||
gate: 'app-origin',
|
||||
@@ -928,6 +938,16 @@ export function registerIpcHandlers(deps: IpcDeps): void {
|
||||
denied: { sessions: [] },
|
||||
handler: () => getKnownSessions(),
|
||||
},
|
||||
'browser-agent:search-suggestions': {
|
||||
kind: 'invoke',
|
||||
gate: 'app-origin',
|
||||
requires: 'browser',
|
||||
denied: [],
|
||||
handler: (query) =>
|
||||
deps.settings.getPreferences().browserSearchSuggestionsEnabled === false
|
||||
? []
|
||||
: getSearchSuggestions(query),
|
||||
},
|
||||
'browser-agent:clear-browsing-data': {
|
||||
kind: 'invoke',
|
||||
gate: 'app-origin',
|
||||
|
||||
@@ -62,20 +62,27 @@ describe('buildMenuTemplate', () => {
|
||||
'separator',
|
||||
'quit',
|
||||
])
|
||||
expect(submenu(template, 'File').map((item) => item.label ?? item.role ?? item.type)).toEqual([
|
||||
const file = submenu(template, 'File')
|
||||
expect(
|
||||
file.filter((item) => item.visible !== false).map((item) => item.label ?? item.type)
|
||||
).toEqual(['New Window', 'New Chat', 'separator', 'Close Window'])
|
||||
// Resource-scoped shortcuts stay registered but never appear in the menu.
|
||||
expect(file.filter((item) => item.visible === false).map((item) => item.label)).toEqual([
|
||||
'New Tab',
|
||||
'New Window',
|
||||
'New Chat',
|
||||
'separator',
|
||||
'Reopen Closed Tab',
|
||||
'Focus Address Bar',
|
||||
'separator',
|
||||
'Next Tab',
|
||||
'Previous Tab',
|
||||
'Select Tab',
|
||||
'separator',
|
||||
'Tab 1',
|
||||
'Tab 2',
|
||||
'Tab 3',
|
||||
'Tab 4',
|
||||
'Tab 5',
|
||||
'Tab 6',
|
||||
'Tab 7',
|
||||
'Tab 8',
|
||||
'Last Tab',
|
||||
'Close Tab',
|
||||
'Close Window',
|
||||
])
|
||||
expect(submenu(template, 'View').map((item) => item.label ?? item.role ?? item.type)).toEqual([
|
||||
'Search',
|
||||
@@ -83,6 +90,7 @@ describe('buildMenuTemplate', () => {
|
||||
'separator',
|
||||
'Back',
|
||||
'Reload',
|
||||
'Force Reload',
|
||||
'separator',
|
||||
'Actual Size',
|
||||
'Zoom In',
|
||||
@@ -152,7 +160,6 @@ describe('buildMenuTemplate', () => {
|
||||
})
|
||||
)
|
||||
const file = submenu(template, 'File')
|
||||
const selectTabs = submenu(file, 'Select Tab')
|
||||
const focusedWindow = new BrowserWindow()
|
||||
const invoke = (item: MenuItemConstructorOptions | undefined) =>
|
||||
(item?.click as unknown as (menuItem: unknown, browserWindow: BrowserWindow) => void)(
|
||||
@@ -162,7 +169,7 @@ describe('buildMenuTemplate', () => {
|
||||
|
||||
invoke(file.find((item) => item.accelerator === 'Ctrl+Tab'))
|
||||
invoke(file.find((item) => item.accelerator === 'Ctrl+Shift+Tab'))
|
||||
invoke(selectTabs.find((item) => item.accelerator === 'CmdOrCtrl+9'))
|
||||
invoke(file.find((item) => item.accelerator === 'CmdOrCtrl+9'))
|
||||
|
||||
expect(handleFocusedResourceShortcut).toHaveBeenNthCalledWith(1, focusedWindow, 'next-tab')
|
||||
expect(handleFocusedResourceShortcut).toHaveBeenNthCalledWith(2, focusedWindow, 'previous-tab')
|
||||
|
||||
@@ -63,6 +63,7 @@ export function buildMenuTemplate(deps: MenuDeps): MenuItemConstructorOptions[]
|
||||
return {
|
||||
label: number === 9 ? 'Last Tab' : `Tab ${number}`,
|
||||
accelerator: `CmdOrCtrl+${number}`,
|
||||
visible: false,
|
||||
click: resourceShortcut(shortcut),
|
||||
}
|
||||
})
|
||||
@@ -124,6 +125,22 @@ export function buildMenuTemplate(deps: MenuDeps): MenuItemConstructorOptions[]
|
||||
win.webContents.reload()
|
||||
},
|
||||
},
|
||||
/**
|
||||
* Hard refresh, cache ignored. A focused Browser tab claims it first
|
||||
* (same boundary as Reload/Close Tab); otherwise it reloads the Sim
|
||||
* shell — the recovery lever for picking up freshly deployed client
|
||||
* code.
|
||||
*/
|
||||
{
|
||||
label: 'Force Reload',
|
||||
accelerator: 'CmdOrCtrl+Shift+R',
|
||||
click: (_item, focusedWindow) => {
|
||||
const win = focusedOrMain(focusedWindow)
|
||||
if (!win || win.isDestroyed()) return
|
||||
if (deps.handleFocusedResourceShortcut(win, 'hard-reload')) return
|
||||
win.webContents.reloadIgnoringCache()
|
||||
},
|
||||
},
|
||||
{ type: 'separator' },
|
||||
{ label: 'Actual Size', accelerator: 'CmdOrCtrl+0', click: setZoom('reset') },
|
||||
{ label: 'Zoom In', accelerator: 'CmdOrCtrl+Plus', click: setZoom('in') },
|
||||
@@ -153,13 +170,6 @@ export function buildMenuTemplate(deps: MenuDeps): MenuItemConstructorOptions[]
|
||||
{
|
||||
label: 'File',
|
||||
submenu: [
|
||||
{
|
||||
label: 'New Tab',
|
||||
accelerator: 'CmdOrCtrl+T',
|
||||
click: (_item, focusedWindow) => {
|
||||
deps.handleFocusedResourceShortcut(focusedOrMain(focusedWindow), 'new-tab')
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'New Window',
|
||||
accelerator: 'CmdOrCtrl+Shift+N',
|
||||
@@ -167,9 +177,35 @@ export function buildMenuTemplate(deps: MenuDeps): MenuItemConstructorOptions[]
|
||||
},
|
||||
{ label: 'New Chat', accelerator: 'CmdOrCtrl+N', click: deps.newChat },
|
||||
{ type: 'separator' },
|
||||
{
|
||||
label: 'Close Window',
|
||||
accelerator: 'CmdOrCtrl+Shift+W',
|
||||
click: (_item, focusedWindow) => {
|
||||
const win = focusedOrMain(focusedWindow)
|
||||
if (win && !win.isDestroyed()) win.close()
|
||||
},
|
||||
},
|
||||
/**
|
||||
* Resource-scoped shortcuts: these act on whichever Browser/Terminal
|
||||
* panel is focused, not on the app, so they stay out of the visible
|
||||
* File menu. The accelerators still fire — macOS registers a hidden
|
||||
* item's accelerator (`acceleratorWorksWhenHidden` defaults to true).
|
||||
* The numbered tab items sit flat here rather than under a "Select
|
||||
* Tab" submenu because children of a hidden submenu do not reliably
|
||||
* register their accelerators.
|
||||
*/
|
||||
{
|
||||
label: 'New Tab',
|
||||
accelerator: 'CmdOrCtrl+T',
|
||||
visible: false,
|
||||
click: (_item, focusedWindow) => {
|
||||
deps.handleFocusedResourceShortcut(focusedOrMain(focusedWindow), 'new-tab')
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'Reopen Closed Tab',
|
||||
accelerator: 'CmdOrCtrl+Shift+T',
|
||||
visible: false,
|
||||
click: (_item, focusedWindow) => {
|
||||
deps.handleFocusedResourceShortcut(focusedOrMain(focusedWindow), 'reopen-closed-tab')
|
||||
},
|
||||
@@ -177,37 +213,31 @@ export function buildMenuTemplate(deps: MenuDeps): MenuItemConstructorOptions[]
|
||||
{
|
||||
label: 'Focus Address Bar',
|
||||
accelerator: 'CmdOrCtrl+L',
|
||||
visible: false,
|
||||
click: resourceShortcut('focus-omnibox'),
|
||||
},
|
||||
{ type: 'separator' },
|
||||
{
|
||||
label: 'Next Tab',
|
||||
accelerator: 'Ctrl+Tab',
|
||||
visible: false,
|
||||
click: resourceShortcut('next-tab'),
|
||||
},
|
||||
{
|
||||
label: 'Previous Tab',
|
||||
accelerator: 'Ctrl+Shift+Tab',
|
||||
visible: false,
|
||||
click: resourceShortcut('previous-tab'),
|
||||
},
|
||||
{ label: 'Select Tab', submenu: numberedTabItems },
|
||||
{ type: 'separator' },
|
||||
...numberedTabItems,
|
||||
{
|
||||
label: 'Close Tab',
|
||||
accelerator: 'CmdOrCtrl+W',
|
||||
visible: false,
|
||||
click: (_item, focusedWindow) => {
|
||||
const win = focusedOrMain(focusedWindow)
|
||||
deps.handleFocusedResourceShortcut(win, 'close-tab')
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'Close Window',
|
||||
accelerator: 'CmdOrCtrl+Shift+W',
|
||||
click: (_item, focusedWindow) => {
|
||||
const win = focusedOrMain(focusedWindow)
|
||||
if (win && !win.isDestroyed()) win.close()
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
{ role: 'editMenu' },
|
||||
|
||||
@@ -12,6 +12,7 @@ export type FocusedResourceShortcut =
|
||||
| 'focus-omnibox'
|
||||
| ResourceTabSelectionShortcut
|
||||
| 'reload-or-clear'
|
||||
| 'hard-reload'
|
||||
| `zoom-${DesktopZoomAction}`
|
||||
|
||||
export type ResourceTabSelectionShortcut =
|
||||
|
||||
@@ -498,7 +498,8 @@ export class TerminalService {
|
||||
emitRendererCommand: (command: TerminalShortcutCommand, terminalId: string) => void,
|
||||
confirmCloseRunning?: (running: string) => boolean
|
||||
): boolean {
|
||||
if (shortcut === 'focus-omnibox') return false
|
||||
// Hard reload has no terminal meaning — leave it to the Browser or shell.
|
||||
if (shortcut === 'focus-omnibox' || shortcut === 'hard-reload') return false
|
||||
const visibleTabShortcut =
|
||||
shortcut === 'new-tab' || shortcut === 'reopen-closed-tab' || shortcut === 'close-tab'
|
||||
const ownsVisibleTabs =
|
||||
|
||||
@@ -851,7 +851,7 @@ export class TerminalSession {
|
||||
(pending) => ({
|
||||
command: pending.command,
|
||||
output:
|
||||
'This opened a full-screen interactive program, which now holds the terminal until it exits. terminal_read renders its current screen, so you can watch it: if it is doing work the user is waiting on, keep polling with wait + terminal_read until it finishes, exactly as you would a long command. Type into it with terminal_input and stop it with terminal_kill. The user can also drive it in the panel. terminal_run reports BUSY until it exits.',
|
||||
'This opened a full-screen interactive program, which now holds the terminal until it exits. terminal_read renders its current screen, so you can watch it: if it is doing work the user is waiting on, keep polling with wait + terminal_read until it finishes, exactly as you would a long command. Type into it with terminal_input and stop it with terminal_kill. If it is a pager (less, git log, man — the screen ends with ":" or "(END)"), nothing more is coming: exit it by sending terminal_input text "q"; terminal_kill delivers Ctrl-C, which a pager ignores. The user can also drive it in the panel. terminal_run reports BUSY until it exits.',
|
||||
status: 'interactive',
|
||||
exitCode: null,
|
||||
durationMs: Date.now() - pending.startedAt,
|
||||
|
||||
@@ -32,6 +32,8 @@ describe('desktop preload bridge', () => {
|
||||
await exposed.browserAgent.setPanelOccluded(true, 'chat-default')
|
||||
await exposed.browserAgent.setPanelOccluded(false, 'chat-explicit-false', false)
|
||||
await exposed.browserAgent.setPanelOccluded(true, 'chat-force', true)
|
||||
await exposed.browserAgent.getSearchSuggestions?.('sim ai')
|
||||
await exposed.settings.setBrowserSearchSuggestionsEnabled?.(false)
|
||||
|
||||
expect(invoke.mock.calls).toEqual([
|
||||
['browser-agent:cancel-tool', 'tool-1', 'chat-default'],
|
||||
@@ -39,6 +41,8 @@ describe('desktop preload bridge', () => {
|
||||
['browser-agent:set-panel-occluded', true, 'chat-default', false],
|
||||
['browser-agent:set-panel-occluded', false, 'chat-explicit-false', false],
|
||||
['browser-agent:set-panel-occluded', true, 'chat-force', true],
|
||||
['browser-agent:search-suggestions', 'sim ai'],
|
||||
['desktop:settings:set-browser-search-suggestions', false],
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -147,6 +147,8 @@ const api: SimDesktopApi = {
|
||||
getPreferences: (): Promise<DesktopPreferences> => ipcRenderer.invoke('desktop:settings:get'),
|
||||
setPreference: (key: DesktopPreferenceKey, value: boolean): Promise<DesktopPreferences> =>
|
||||
ipcRenderer.invoke('desktop:settings:set', key, value),
|
||||
setBrowserSearchSuggestionsEnabled: (enabled: boolean): Promise<DesktopPreferences> =>
|
||||
ipcRenderer.invoke('desktop:settings:set-browser-search-suggestions', enabled),
|
||||
notify: (payload: DesktopNotificationPayload): Promise<boolean> =>
|
||||
ipcRenderer.invoke('desktop:settings:notify', payload),
|
||||
setBrowserTheme: (theme: DesktopAppearanceTheme): Promise<DesktopPreferences> =>
|
||||
@@ -281,6 +283,8 @@ const api: SimDesktopApi = {
|
||||
ipcRenderer.invoke('browser-agent:get-tabs-state', scopeId),
|
||||
getKnownSessions: (): Promise<BrowserKnownSessionsState> =>
|
||||
ipcRenderer.invoke('browser-agent:get-known-sessions'),
|
||||
getSearchSuggestions: (query: string): Promise<string[]> =>
|
||||
ipcRenderer.invoke('browser-agent:search-suggestions', query),
|
||||
clearBrowsingData: (kinds?: readonly BrowserDataKind[]): Promise<BrowserKnownSessionsState> =>
|
||||
ipcRenderer.invoke('browser-agent:clear-browsing-data', kinds),
|
||||
getDownloadsState: (scopeId: string): Promise<BrowserDownloadsState> =>
|
||||
|
||||
@@ -107,6 +107,17 @@ export const nativeImage = {
|
||||
setTemplateImage: vi.fn(),
|
||||
getSize: vi.fn(() => ({ width: options.width, height: options.height })),
|
||||
})),
|
||||
/**
|
||||
* Decodes nothing: tests pass short base64 stand-ins rather than real images.
|
||||
* Reports empty so callers take their undecodable-capture fallback, and let a
|
||||
* test opt into the resize path by overriding this mock with a sized image.
|
||||
*/
|
||||
createFromBuffer: vi.fn((_buffer: unknown) => ({
|
||||
isEmpty: vi.fn(() => true),
|
||||
getSize: vi.fn(() => ({ width: 0, height: 0 })),
|
||||
resize: vi.fn(),
|
||||
toJPEG: vi.fn(() => Buffer.alloc(0)),
|
||||
})),
|
||||
}
|
||||
|
||||
export class Tray {
|
||||
@@ -226,6 +237,8 @@ export class BrowserWindow {
|
||||
getZoomFactor: vi.fn(() => 1),
|
||||
executeJavaScript: vi.fn(() => Promise.resolve(true)),
|
||||
focus: vi.fn(),
|
||||
isFocused: vi.fn(() => false),
|
||||
isDestroyed: vi.fn(() => false),
|
||||
send: vi.fn(),
|
||||
setWindowOpenHandler: vi.fn(),
|
||||
isDevToolsOpened: vi.fn(() => false),
|
||||
|
||||
@@ -126,7 +126,7 @@ Here's a detailed breakdown of what users can do with each permission level:
|
||||
**What they can do:**
|
||||
- Everything Read users can do, plus:
|
||||
- Create, edit, and delete workflows
|
||||
- Run and deploy workflows
|
||||
- Run workflows
|
||||
- Add, edit, and delete workspace environment variables
|
||||
- Use all available tools and integrations
|
||||
- Collaborate in real-time on workflow editing
|
||||
@@ -140,6 +140,7 @@ Here's a detailed breakdown of what users can do with each permission level:
|
||||
|
||||
**What they can do:**
|
||||
- Everything Write users can do, plus:
|
||||
- Deploy workflows
|
||||
- Invite new users to the workspace with any permission level
|
||||
- Remove users from the workspace
|
||||
- Manage workspace settings and integrations
|
||||
@@ -254,4 +255,4 @@ import { FAQ } from '@/components/ui/faq'
|
||||
{ question: "Who can manage a workspace's credentials and secrets?", answer: "Workspace Admins are automatically Credential Admins of the workspace's shared credentials — OAuth connections, service accounts, and workspace environment variables — so they can use, edit, delete, and share them, and run workflows that rely on them. Organization Owners and Admins get this too because they are workspace Admins everywhere. Read and Write members get use-only access to shared credentials unless they are explicitly made a Credential Admin. Personal environment variables are never shared; they stay private to their owner." },
|
||||
{ question: "What are permission groups and how do they work?", answer: "Permission groups are an Enterprise access control feature that lets organization owners and admins define granular restrictions beyond the standard Read/Write/Admin roles. The organization's default group is org-wide; every other group targets specific workspaces and, by default, governs all members of those workspaces (including external members) — add members to restrict it to specific people. A user is governed by one group per workspace: a group they're an explicit member of takes precedence over an all-members group (one with no members) on that workspace, which takes precedence over the organization's default group. A permission group can hide UI sections (like trace spans, knowledge base, API keys, or deployment options), disable features (MCP tools, custom tools, skills, invitations), and restrict which integrations and model providers its members can access. Only one group per organization can be the default; it ignores members and governs everyone not covered by a workspace group, including external members. Restrictions are enforced based on the organization that owns the workflow's workspace, not on which workspace you're currently viewing." },
|
||||
{ question: "How should I set up permissions for a new team member?", answer: "Start with the lowest permission level they need. Invite them with Read workspace access if they only need visibility, Write if they need to create and run workflows, or Admin if they need to manage the workspace and its users, and leave Membership on Member. For clients, partners, and contractors, choose External so they collaborate without joining your organization or using a seat — this requires them to already be on a paid Sim plan, either their own Pro or Max subscription or another organization that seats them." },
|
||||
]} />
|
||||
]} />
|
||||
|
||||
@@ -86,8 +86,10 @@ describe('desktop title-bar surface audit', () => {
|
||||
// Regression: a px clearance shrinks under page zoom while the OS-drawn lights
|
||||
// do not, so they end up drawn over the sidebar toggle.
|
||||
it('reserves the traffic-light lane from the platform, not hardcoded pixels', () => {
|
||||
expect(globalStyles).toContain('--desktop-title-bar-height: env(titlebar-area-height,')
|
||||
expect(globalStyles).toContain('--desktop-title-bar-inset-x: env(titlebar-area-x,')
|
||||
// The env() term is the platform-derived part; the max() floor keeps the
|
||||
// lane from collapsing below the OS-drawn lights under page zoom.
|
||||
expect(globalStyles).toContain('--desktop-title-bar-height: max(env(titlebar-area-height,')
|
||||
expect(globalStyles).toContain('--desktop-title-bar-inset-x: max(env(titlebar-area-x,')
|
||||
|
||||
// Scoped to the desktop block: the `:root` zeros are the deliberate
|
||||
// no-lane case, so only the overrides must stay derived.
|
||||
@@ -95,13 +97,17 @@ describe('desktop title-bar surface audit', () => {
|
||||
/html\[data-sim-desktop-title-bar="inset"\]\s*\{([^}]*)\}/
|
||||
)?.[1]
|
||||
expect(insetBlock).toBeTypeOf('string')
|
||||
// The zoom rework split the lane from the control: the lane stays
|
||||
// platform-derived (clamped env() above), while the control is a fixed
|
||||
// square — CSS px already scale under page zoom, so only the centering
|
||||
// offset must remain computed from the lane height.
|
||||
expect(insetBlock).toMatch(/--desktop-title-bar-control-offset: calc\(/)
|
||||
expect(insetBlock).not.toMatch(/--desktop-title-bar-control-offset:\s*[\d.]+px/)
|
||||
for (const name of [
|
||||
'--desktop-title-bar-control-size',
|
||||
'--desktop-title-bar-control-icon-size',
|
||||
'--desktop-title-bar-control-offset',
|
||||
]) {
|
||||
expect(insetBlock).toMatch(new RegExp(`${name}: calc\\(`))
|
||||
expect(insetBlock).not.toMatch(new RegExp(`${name}:\\s*[\\d.]+px`))
|
||||
expect(insetBlock).toMatch(new RegExp(`${name}:\\s*[\\d.]+px`))
|
||||
}
|
||||
|
||||
// Both lane consumers read those vars; a literal in either is the bug.
|
||||
|
||||
@@ -96,23 +96,24 @@
|
||||
}
|
||||
|
||||
/**
|
||||
* Electron's `titleBarOverlay` publishes the window controls' geometry as these
|
||||
* `titlebar-area-*` env vars, and Chromium rescales them under page zoom so the
|
||||
* lane holds its physical size. Fallbacks are the platform's measured values, for
|
||||
* a shell predating the overlay; the `:root` zeros cover the different case of the
|
||||
* attribute being absent entirely, where this block never matches.
|
||||
* The lane scales with page zoom like the rest of the UI (Codex-style): the
|
||||
* 38px/81px terms are CSS px, so zooming in grows the lane and its controls
|
||||
* with the content while the OS-drawn traffic lights hold their physical size
|
||||
* inside it. The `env(titlebar-area-*)` terms — which Chromium rescales under
|
||||
* zoom to keep physical geometry — act only as a floor, so zooming OUT can
|
||||
* never shrink the lane under the lights or slide the controls beneath them.
|
||||
* Fallbacks are the platform's measured values, for a shell predating the
|
||||
* overlay; the `:root` zeros cover the different case of the attribute being
|
||||
* absent entirely, where this block never matches.
|
||||
*/
|
||||
html[data-sim-desktop-title-bar="inset"] {
|
||||
--sidebar-collapsed-width: 0px;
|
||||
--desktop-title-bar-height: env(titlebar-area-height, 38px);
|
||||
--desktop-title-bar-inset-x: env(titlebar-area-x, 81px);
|
||||
/* 0.79 = 30px of the 38px lane, and 0.53 of that = 16px. Proportions rather
|
||||
than px because px would scale with page zoom while the OS-drawn lights
|
||||
would not — and calc cannot divide a length by a length to get a scale.
|
||||
`navigator.windowControlsOverlay` could, but reading it in JS would race
|
||||
first paint for a value the blocking script needs. */
|
||||
--desktop-title-bar-control-size: calc(var(--desktop-title-bar-height) * 0.79);
|
||||
--desktop-title-bar-control-icon-size: calc(var(--desktop-title-bar-control-size) * 0.53);
|
||||
--desktop-title-bar-height: max(env(titlebar-area-height, 38px), 38px);
|
||||
--desktop-title-bar-inset-x: max(env(titlebar-area-x, 81px), 81px);
|
||||
/* 30px of the 38px lane, and 16px of that — CSS px on purpose, so the
|
||||
controls zoom with the page instead of staying pinned to physical size. */
|
||||
--desktop-title-bar-control-size: 30px;
|
||||
--desktop-title-bar-control-icon-size: 16px;
|
||||
--desktop-title-bar-control-offset: calc(
|
||||
(var(--desktop-title-bar-height) - var(--desktop-title-bar-control-size)) /
|
||||
2
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { createMockRequest } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockAuthenticate, mockGetLatestRunForStream, mockRequestStreamSteering, mockAppend } =
|
||||
vi.hoisted(() => ({
|
||||
mockAuthenticate: vi.fn(),
|
||||
mockGetLatestRunForStream: vi.fn(),
|
||||
mockRequestStreamSteering: vi.fn(),
|
||||
mockAppend: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/http', () => ({
|
||||
authenticateCopilotRequestSessionOnly: mockAuthenticate,
|
||||
}))
|
||||
vi.mock('@/lib/copilot/async-runs/repository', () => ({
|
||||
getLatestRunForStream: mockGetLatestRunForStream,
|
||||
}))
|
||||
vi.mock('@/lib/copilot/request/session/steer', () => ({
|
||||
requestStreamSteering: mockRequestStreamSteering,
|
||||
}))
|
||||
vi.mock('@/lib/copilot/chat/messages-store', () => ({
|
||||
appendCopilotChatMessages: mockAppend,
|
||||
}))
|
||||
|
||||
import { POST } from '@/app/api/copilot/chat/steer/route'
|
||||
|
||||
function steerRequest(overrides: Record<string, unknown> = {}) {
|
||||
return createMockRequest('POST', {
|
||||
streamId: 'stream-1',
|
||||
chatId: 'chat-1',
|
||||
steeringId: 'steer-1',
|
||||
content: 'focus on the tests',
|
||||
...overrides,
|
||||
})
|
||||
}
|
||||
|
||||
describe('POST /api/copilot/chat/steer', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockAuthenticate.mockResolvedValue({ userId: 'user-1', isAuthenticated: true })
|
||||
mockGetLatestRunForStream.mockResolvedValue({ chatId: 'chat-1', workspaceId: 'workspace-1' })
|
||||
mockRequestStreamSteering.mockResolvedValue({ queued: true, status: 200 })
|
||||
mockAppend.mockResolvedValue(undefined)
|
||||
})
|
||||
|
||||
it('queues steering with Go and persists the user message', async () => {
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(await response.json()).toMatchObject({ ok: true, queued: true })
|
||||
expect(mockRequestStreamSteering).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
streamId: 'stream-1',
|
||||
chatId: 'chat-1',
|
||||
steeringId: 'steer-1',
|
||||
content: 'focus on the tests',
|
||||
userId: 'user-1',
|
||||
})
|
||||
)
|
||||
expect(mockAppend).toHaveBeenCalledWith(
|
||||
'chat-1',
|
||||
[
|
||||
expect.objectContaining({
|
||||
id: 'steer-1',
|
||||
role: 'user',
|
||||
content: 'focus on the tests',
|
||||
}),
|
||||
],
|
||||
{ streamId: 'stream-1' }
|
||||
)
|
||||
})
|
||||
|
||||
it('returns 409 when Go rejects the steer so the client falls back to a normal send', async () => {
|
||||
mockRequestStreamSteering.mockResolvedValue({ queued: false, status: 429 })
|
||||
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(409)
|
||||
expect(await response.json()).toMatchObject({ ok: false, queued: false })
|
||||
expect(mockAppend).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('returns 409 when the Go forward throws', async () => {
|
||||
mockRequestStreamSteering.mockRejectedValue(new Error('network down'))
|
||||
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(409)
|
||||
expect(mockAppend).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects a chat that does not own the stream', async () => {
|
||||
mockGetLatestRunForStream.mockResolvedValue({ chatId: 'other-chat' })
|
||||
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(403)
|
||||
expect(mockRequestStreamSteering).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects unauthenticated callers', async () => {
|
||||
mockAuthenticate.mockResolvedValue({ userId: null, isAuthenticated: false })
|
||||
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(401)
|
||||
expect(mockRequestStreamSteering).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an empty content body', async () => {
|
||||
const response = await POST(steerRequest({ content: '' }))
|
||||
|
||||
expect(response.status).toBe(400)
|
||||
expect(mockRequestStreamSteering).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('still reports queued when history persistence fails', async () => {
|
||||
mockAppend.mockRejectedValue(new Error('db down'))
|
||||
|
||||
const response = await POST(steerRequest())
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(await response.json()).toMatchObject({ ok: true, queued: true })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,128 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { getErrorMessage } from '@sim/utils/errors'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { copilotChatSteerBodySchema } from '@/lib/api/contracts/copilot'
|
||||
import { validationErrorResponse } from '@/lib/api/server'
|
||||
import { getLatestRunForStream } from '@/lib/copilot/async-runs/repository'
|
||||
import { appendCopilotChatMessages } from '@/lib/copilot/chat/messages-store'
|
||||
import { CopilotSteerOutcome } from '@/lib/copilot/generated/trace-attribute-values-v1'
|
||||
import { TraceAttr } from '@/lib/copilot/generated/trace-attributes-v1'
|
||||
import { TraceSpan } from '@/lib/copilot/generated/trace-spans-v1'
|
||||
import { authenticateCopilotRequestSessionOnly } from '@/lib/copilot/request/http'
|
||||
import { withIncomingGoSpan } from '@/lib/copilot/request/otel'
|
||||
import { requestStreamSteering } from '@/lib/copilot/request/session/steer'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
|
||||
const logger = createLogger('CopilotChatSteerAPI')
|
||||
|
||||
// POST /api/copilot/chat/steer — queues a mid-turn steering message with the
|
||||
// Go side for a LIVE stream. Acceptance means "queued", not "applied": Go
|
||||
// acknowledges application with a `run`/`steering_applied` stream event; a
|
||||
// client that never sees that ack before the stream ends re-sends the content
|
||||
// as an ordinary message. A 409 here tells the client to take that ordinary
|
||||
// path immediately.
|
||||
export const POST = withRouteHandler((request: NextRequest) =>
|
||||
withIncomingGoSpan(
|
||||
request.headers,
|
||||
TraceSpan.CopilotChatSteerStream,
|
||||
undefined,
|
||||
async (rootSpan) => {
|
||||
const { userId: authenticatedUserId, isAuthenticated } =
|
||||
await authenticateCopilotRequestSessionOnly()
|
||||
if (!isAuthenticated || !authenticatedUserId) {
|
||||
rootSpan.setAttribute(TraceAttr.CopilotSteerOutcome, CopilotSteerOutcome.BadRequest)
|
||||
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
|
||||
}
|
||||
|
||||
// boundary-raw-json: tolerant parse; validation happens via the contract schema below
|
||||
const body = await request.json().catch(() => ({}))
|
||||
const validation = copilotChatSteerBodySchema.safeParse(body)
|
||||
if (!validation.success) {
|
||||
rootSpan.setAttribute(TraceAttr.CopilotSteerOutcome, CopilotSteerOutcome.BadRequest)
|
||||
return validationErrorResponse(validation.error, 'Invalid request body')
|
||||
}
|
||||
const { streamId, chatId, steeringId, content } = validation.data
|
||||
rootSpan.setAttributes({
|
||||
[TraceAttr.StreamId]: streamId,
|
||||
[TraceAttr.ChatId]: chatId,
|
||||
[TraceAttr.UserId]: authenticatedUserId,
|
||||
[TraceAttr.CopilotSteeringContentChars]: content.length,
|
||||
})
|
||||
|
||||
// Ownership pre-check on the Sim side (Go re-proves it independently):
|
||||
// the stream must belong to a run of the authenticated user, and the
|
||||
// claimed chat must match that run.
|
||||
const run = await getLatestRunForStream(streamId, authenticatedUserId).catch((err) => {
|
||||
logger.warn('getLatestRunForStream failed while resolving steer context', {
|
||||
streamId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
return null
|
||||
})
|
||||
if (run?.chatId && run.chatId !== chatId) {
|
||||
rootSpan.setAttribute(TraceAttr.CopilotSteerOutcome, CopilotSteerOutcome.BadRequest)
|
||||
return NextResponse.json({ error: 'Stream does not belong to this chat' }, { status: 403 })
|
||||
}
|
||||
|
||||
let queued = false
|
||||
let goStatus = 0
|
||||
try {
|
||||
const result = await requestStreamSteering({
|
||||
streamId,
|
||||
userId: authenticatedUserId,
|
||||
chatId,
|
||||
steeringId,
|
||||
content,
|
||||
workspaceId: run?.workspaceId ?? undefined,
|
||||
})
|
||||
queued = result.queued
|
||||
goStatus = result.status
|
||||
} catch (err) {
|
||||
logger.warn('Steer forward to Go failed', {
|
||||
streamId,
|
||||
chatId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
}
|
||||
|
||||
if (!queued) {
|
||||
rootSpan.setAttribute(TraceAttr.CopilotSteerOutcome, CopilotSteerOutcome.NoActiveTurn)
|
||||
// 409 = "could not queue; send it as an ordinary message instead".
|
||||
return NextResponse.json({ ok: false, queued: false, goStatus }, { status: 409 })
|
||||
}
|
||||
|
||||
// Persist the steering text as a user message so reloads include it.
|
||||
// Failure here must not fail the steer — the message is already queued
|
||||
// with Go and will reach the model; persistence is display-only.
|
||||
try {
|
||||
await appendCopilotChatMessages(
|
||||
chatId,
|
||||
[
|
||||
{
|
||||
id: steeringId,
|
||||
role: 'user',
|
||||
content,
|
||||
timestamp: new Date().toISOString(),
|
||||
},
|
||||
],
|
||||
{ streamId }
|
||||
)
|
||||
} catch (err) {
|
||||
logger.warn('Failed to persist steering message to chat history', {
|
||||
chatId,
|
||||
steeringId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
}
|
||||
|
||||
rootSpan.setAttribute(TraceAttr.CopilotSteerOutcome, CopilotSteerOutcome.Queued)
|
||||
logger.info('Queued mid-turn steering message', {
|
||||
streamId,
|
||||
chatId,
|
||||
steeringId,
|
||||
contentChars: content.length,
|
||||
})
|
||||
return NextResponse.json({ ok: true, queued: true })
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,114 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const { mockCheckInternalApiKey, mockPrepareEnvironmentContext, mockHandler } = vi.hoisted(() => ({
|
||||
mockCheckInternalApiKey: vi.fn(),
|
||||
mockPrepareEnvironmentContext: vi.fn(),
|
||||
mockHandler: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/http', () => ({
|
||||
checkInternalApiKey: mockCheckInternalApiKey,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/environment-context', () => ({
|
||||
prepareCopilotEnvironmentContext: mockPrepareEnvironmentContext,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/tool-executor', () => ({
|
||||
ensureHandlersRegistered: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/tool-executor/executor', () => ({
|
||||
executeTool: (
|
||||
_toolName: string,
|
||||
params: Record<string, unknown>,
|
||||
context: Record<string, unknown>
|
||||
) => mockHandler(params, context),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/tools/resources', () => ({
|
||||
handleResourceSideEffects: vi.fn().mockResolvedValue(undefined),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/otel', () => ({
|
||||
withIncomingGoSpan: (
|
||||
_headers: Headers,
|
||||
_span: string,
|
||||
_attrs: undefined,
|
||||
fn: (span: { setAttributes: () => void }) => Promise<Response>
|
||||
) => fn({ setAttributes: () => {} }),
|
||||
}))
|
||||
|
||||
import { POST } from '@/app/api/copilot/tools/execute/route'
|
||||
|
||||
function makeRequest(body: Record<string, unknown>): Request {
|
||||
return new Request('http://localhost/api/copilot/tools/execute', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(body),
|
||||
})
|
||||
}
|
||||
|
||||
const BASE_BODY = {
|
||||
toolCallId: 'call-1',
|
||||
toolName: 'read',
|
||||
params: { path: 'files/a.md' },
|
||||
userId: 'user-1',
|
||||
workspaceId: 'ws-1',
|
||||
chatId: 'chat-1',
|
||||
messageId: 'msg-1',
|
||||
}
|
||||
|
||||
describe('POST /api/copilot/tools/execute (in-band)', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockCheckInternalApiKey.mockReturnValue({ success: true })
|
||||
// A fresh, complete registry per test: the module-level turn cache is keyed
|
||||
// by messageId, so each test uses a distinct messageId to avoid cross-test
|
||||
// cache hits.
|
||||
mockPrepareEnvironmentContext.mockImplementation(async () => ({
|
||||
resolvedSecretTraceRegistry: new ResolvedSecretTraceRegistry([]),
|
||||
}))
|
||||
})
|
||||
|
||||
it('threads a per-call registry fork into the handler and returns the projected result', async () => {
|
||||
mockHandler.mockResolvedValue({ success: true, output: { content: 'hello' } })
|
||||
const res = await POST(makeRequest({ ...BASE_BODY, messageId: 'msg-fork' }) as never)
|
||||
expect(res.status).toBe(200)
|
||||
await expect(res.json()).resolves.toEqual({ success: true, output: { content: 'hello' } })
|
||||
|
||||
const [, handlerContext] = mockHandler.mock.calls[0]
|
||||
expect(handlerContext.resolvedSecretTraceRegistry).toBeInstanceOf(ResolvedSecretTraceRegistry)
|
||||
expect(handlerContext.userId).toBe('user-1')
|
||||
expect(handlerContext.copilotToolExecution).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps a clean tool failure message intact when the registry is available', async () => {
|
||||
mockHandler.mockResolvedValue({ success: false, error: 'File not found: files/a.md' })
|
||||
const res = await POST(makeRequest({ ...BASE_BODY, messageId: 'msg-clean-error' }) as never)
|
||||
const body = await res.json()
|
||||
expect(body.success).toBe(false)
|
||||
expect(body.error).toBe('File not found: files/a.md')
|
||||
})
|
||||
|
||||
it('withholds results when no egress registry can be built', async () => {
|
||||
mockPrepareEnvironmentContext.mockRejectedValue(new Error('env unavailable'))
|
||||
mockHandler.mockResolvedValue({ success: true, output: { content: 'sensitive' } })
|
||||
const res = await POST(makeRequest({ ...BASE_BODY, messageId: 'msg-no-registry' }) as never)
|
||||
const body = await res.json()
|
||||
expect(body).toEqual({ success: true })
|
||||
})
|
||||
|
||||
it('reuses one turn registry across calls that share a messageId', async () => {
|
||||
mockHandler.mockResolvedValue({ success: true, output: {} })
|
||||
await POST(makeRequest({ ...BASE_BODY, messageId: 'msg-shared' }) as never)
|
||||
await POST(
|
||||
makeRequest({ ...BASE_BODY, toolCallId: 'call-2', messageId: 'msg-shared' }) as never
|
||||
)
|
||||
expect(mockPrepareEnvironmentContext).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,206 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { getErrorMessage } from '@sim/utils/errors'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { copilotToolExecuteInternalBodySchema } from '@/lib/api/contracts/copilot'
|
||||
import { validationErrorResponse } from '@/lib/api/server'
|
||||
import { prepareCopilotEnvironmentContext } from '@/lib/copilot/environment-context'
|
||||
import { TraceAttr } from '@/lib/copilot/generated/trace-attributes-v1'
|
||||
import { TraceSpan } from '@/lib/copilot/generated/trace-spans-v1'
|
||||
import { checkInternalApiKey } from '@/lib/copilot/request/http'
|
||||
import { withIncomingGoSpan } from '@/lib/copilot/request/otel'
|
||||
import {
|
||||
inspectToolResultForCopilot,
|
||||
projectToolErrorMessageForCopilot,
|
||||
} from '@/lib/copilot/request/tools/resolved-secret-result'
|
||||
import { handleResourceSideEffects } from '@/lib/copilot/request/tools/resources'
|
||||
import type { ToolCallResult } from '@/lib/copilot/request/types'
|
||||
import { ensureHandlersRegistered } from '@/lib/copilot/tool-executor'
|
||||
import { executeTool } from '@/lib/copilot/tool-executor/executor'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('CopilotToolExecuteInternalAPI')
|
||||
|
||||
/**
|
||||
* In-band calls are stateless one-offs, but the turn they serve is not: secret
|
||||
* provenance activated by one tool call must stay visible to the next call's
|
||||
* egress projection, exactly as the request-lifecycle registry accumulates
|
||||
* across a turn. Keyed by the turn (messageId) so one background lane shares
|
||||
* one registry; TTL-evicted since nothing signals turn end on this route.
|
||||
*/
|
||||
const TURN_REGISTRY_TTL_MS = 10 * 60 * 1000
|
||||
const TURN_REGISTRY_CACHE_MAX = 256
|
||||
const turnRegistryCache = new Map<
|
||||
string,
|
||||
{ registry: ResolvedSecretTraceRegistry; expiresAt: number }
|
||||
>()
|
||||
|
||||
async function getTurnEgressRegistry(
|
||||
userId: string,
|
||||
workspaceId: string | undefined,
|
||||
messageId: string | undefined
|
||||
): Promise<ResolvedSecretTraceRegistry> {
|
||||
const key = `${userId}\u0000${workspaceId ?? ''}\u0000${messageId ?? ''}`
|
||||
const now = Date.now()
|
||||
const hit = turnRegistryCache.get(key)
|
||||
if (hit && hit.expiresAt > now) {
|
||||
hit.expiresAt = now + TURN_REGISTRY_TTL_MS
|
||||
return hit.registry
|
||||
}
|
||||
const environmentContext = await prepareCopilotEnvironmentContext(userId, workspaceId)
|
||||
for (const [cachedKey, cached] of turnRegistryCache) {
|
||||
if (cached.expiresAt <= now) turnRegistryCache.delete(cachedKey)
|
||||
}
|
||||
if (turnRegistryCache.size >= TURN_REGISTRY_CACHE_MAX) {
|
||||
const oldest = turnRegistryCache.keys().next().value
|
||||
if (oldest !== undefined) turnRegistryCache.delete(oldest)
|
||||
}
|
||||
turnRegistryCache.set(key, {
|
||||
registry: environmentContext.resolvedSecretTraceRegistry,
|
||||
expiresAt: now + TURN_REGISTRY_TTL_MS,
|
||||
})
|
||||
return environmentContext.resolvedSecretTraceRegistry
|
||||
}
|
||||
|
||||
// POST /api/copilot/tools/execute — internal (Go → Sim) in-band execution of
|
||||
// one sim-server tool announced on a LIVE mothership turn. This is what lets
|
||||
// background (async) subagents — and the main lane while background agents are
|
||||
// running — use sim-executed tools without a checkpoint pause: Go calls here
|
||||
// synchronously instead of parking the turn, and the tool runs through the
|
||||
// same server tool router the resume driver uses. Trusted server-to-server
|
||||
// only: Go supplies the acting user, proven by the internal API secret.
|
||||
//
|
||||
// Results cross a model boundary here just as they do in the resume driver, so
|
||||
// this route mirrors its provenance discipline: a per-call registry fork feeds
|
||||
// the handler, the settled result is projected before it returns to Go, and
|
||||
// the fork is merged back only when the projection was safe and the fork
|
||||
// stayed complete. Without this, every result crossed unprojected and every
|
||||
// thrown error was replaced by the opaque "could not be returned safely"
|
||||
// sentinel (the projection fails closed on a missing registry).
|
||||
export const POST = withRouteHandler((request: NextRequest) =>
|
||||
withIncomingGoSpan(
|
||||
request.headers,
|
||||
TraceSpan.CopilotToolsExecuteInband,
|
||||
undefined,
|
||||
async (rootSpan) => {
|
||||
const authResult = checkInternalApiKey(request)
|
||||
if (!authResult.success) {
|
||||
return NextResponse.json(
|
||||
{ success: false, error: authResult.error || 'Authentication failed' },
|
||||
{ status: 401 }
|
||||
)
|
||||
}
|
||||
|
||||
// boundary-raw-json: tolerant parse; validation happens via the contract schema below
|
||||
const body = await request.json().catch(() => ({}))
|
||||
const validation = copilotToolExecuteInternalBodySchema.safeParse(body)
|
||||
if (!validation.success) {
|
||||
return validationErrorResponse(validation.error, 'Invalid request body')
|
||||
}
|
||||
const {
|
||||
toolCallId,
|
||||
toolName,
|
||||
params,
|
||||
userId,
|
||||
workflowId,
|
||||
workspaceId,
|
||||
chatId,
|
||||
messageId,
|
||||
parentToolCallId,
|
||||
userPermission,
|
||||
} = validation.data
|
||||
rootSpan.setAttributes({
|
||||
[TraceAttr.ToolName]: toolName,
|
||||
[TraceAttr.ToolCallId]: toolCallId,
|
||||
[TraceAttr.UserId]: userId,
|
||||
})
|
||||
|
||||
let toolRegistry: ResolvedSecretTraceRegistry | undefined
|
||||
let turnRegistry: ResolvedSecretTraceRegistry | undefined
|
||||
try {
|
||||
turnRegistry = await getTurnEgressRegistry(userId, workspaceId, messageId)
|
||||
toolRegistry = turnRegistry.forkForInputPaths([])
|
||||
} catch (err) {
|
||||
logger.error('In-band egress registry unavailable; results will be withheld', {
|
||||
toolName,
|
||||
toolCallId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
}
|
||||
|
||||
try {
|
||||
// The relay's comprehensive dispatcher: registry handlers (VFS
|
||||
// glob/read/grep, function execute, ...) plus the server tool router
|
||||
// fallback — the plain server-tool adapter alone rejects VFS tools
|
||||
// with "Unknown server tool".
|
||||
ensureHandlersRegistered()
|
||||
const result = await executeTool(toolName, params, {
|
||||
userId,
|
||||
workflowId: workflowId ?? '',
|
||||
workspaceId,
|
||||
chatId,
|
||||
messageId,
|
||||
toolCallId,
|
||||
parentToolCallId,
|
||||
userPermission,
|
||||
copilotToolExecution: true,
|
||||
resolvedSecretTraceRegistry: toolRegistry,
|
||||
})
|
||||
const projection = inspectToolResultForCopilot(result, toolRegistry, toolName)
|
||||
const projected = projection.result
|
||||
if (projection.safe && toolRegistry?.isComplete() && turnRegistry) {
|
||||
turnRegistry.mergeToolCallRegistry(toolRegistry)
|
||||
}
|
||||
if (!projected.success) {
|
||||
logger.warn('In-band tool execution failed', {
|
||||
toolName,
|
||||
toolCallId,
|
||||
error: projected.error,
|
||||
runtimeSucceeded: result.success,
|
||||
projectionSafe: projection.safe,
|
||||
})
|
||||
}
|
||||
if (result.success && chatId) {
|
||||
// Persist created/deleted resources on the chat (file chips, table
|
||||
// links) exactly like the resume driver does. No live event sink
|
||||
// exists for an out-of-band route, so chips surface from the
|
||||
// persisted chat resources rather than a mid-turn push. Side effects
|
||||
// read the raw result; only model-facing content is projected.
|
||||
const asToolResult = { success: result.success, output: result.output } as ToolCallResult
|
||||
await handleResourceSideEffects(
|
||||
toolName,
|
||||
params,
|
||||
asToolResult,
|
||||
{ success: projected.success, output: projected.output } as ToolCallResult,
|
||||
chatId,
|
||||
undefined,
|
||||
() => false
|
||||
).catch((err) => {
|
||||
logger.warn('In-band resource side effects failed', {
|
||||
toolName,
|
||||
toolCallId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
})
|
||||
}
|
||||
return NextResponse.json({
|
||||
success: projected.success,
|
||||
...(projected.output !== undefined ? { output: projected.output } : {}),
|
||||
...(projected.error ? { error: projected.error } : {}),
|
||||
})
|
||||
} catch (err) {
|
||||
const message = projectToolErrorMessageForCopilot(
|
||||
getErrorMessage(err),
|
||||
toolRegistry,
|
||||
toolName
|
||||
)
|
||||
logger.error('In-band tool execution threw', {
|
||||
toolName,
|
||||
toolCallId,
|
||||
error: getErrorMessage(err),
|
||||
})
|
||||
return NextResponse.json({ success: false, error: message }, { status: 500 })
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -12,6 +12,8 @@ import { enforcePublicFileRateLimit } from '@/lib/public-shares/rate-limit'
|
||||
import { resolveActiveShareByToken } from '@/lib/public-shares/share-manager'
|
||||
import { downloadFile } from '@/lib/uploads/core/storage-service'
|
||||
import { resolveServableImageBytes } from '@/lib/uploads/server/image-derivative'
|
||||
import { isSimPageSource, SIM_PAGE_CONTENT_TYPE } from '@/lib/workspace-files/page-compile'
|
||||
import { renderSimPageDocumentWithAssets } from '@/lib/workspace-files/page-document.server'
|
||||
import {
|
||||
createErrorResponse,
|
||||
createFileResponse,
|
||||
@@ -89,6 +91,24 @@ export const GET = withRouteHandler(
|
||||
if (servable.kind === 'artifact') {
|
||||
buffer = servable.buffer
|
||||
contentType = servable.contentType
|
||||
} else if (
|
||||
// Sim pages store an extensionless name — the record type marks them;
|
||||
// legacy pages still carry .html.
|
||||
(file.contentType === SIM_PAGE_CONTENT_TYPE ||
|
||||
file.originalName.toLowerCase().endsWith('.html')) &&
|
||||
isSimPageSource(raw.toString('utf8'))
|
||||
) {
|
||||
// The pdf model for pages: the stored .html is source; a share serves
|
||||
// the fully styled compiled document, matching the preview and serve
|
||||
// routes. sim: links resolve to workspace routes (a viewer without
|
||||
// workspace access simply lands on the sign-in gate).
|
||||
buffer = Buffer.from(
|
||||
await renderSimPageDocumentWithAssets(raw.toString('utf8'), {
|
||||
workspaceId: file.workspaceId ?? undefined,
|
||||
}),
|
||||
'utf8'
|
||||
)
|
||||
contentType = 'text/html'
|
||||
} else if (preview) {
|
||||
// Only for a render request: the Download button omits `preview`, so a saved
|
||||
// file is always the bytes that were shared.
|
||||
|
||||
@@ -22,6 +22,8 @@ import { resolveStoredFileContext } from '@/lib/uploads/server/metadata'
|
||||
import { inferContextFromKey } from '@/lib/uploads/utils/file-utils'
|
||||
import { internalWorkspaceFileServeAuth } from '@/lib/workspace-files/api'
|
||||
import { readWorkspaceFileContentByKey } from '@/lib/workspace-files/application/read-workspace-file-content-by-key'
|
||||
import { isSimPageSource, SIM_PAGE_CONTENT_TYPE } from '@/lib/workspace-files/page-compile'
|
||||
import { renderSimPageDocumentWithAssets } from '@/lib/workspace-files/page-document.server'
|
||||
import { verifyFileAccess } from '@/app/api/files/authorization'
|
||||
import {
|
||||
createErrorResponse,
|
||||
@@ -76,12 +78,39 @@ async function resolveServableBytes(params: {
|
||||
options: ServeOptions
|
||||
ownerKey: string | undefined
|
||||
filePrincipal?: Principal
|
||||
/** The stored record's content type, where the caller has the record. */
|
||||
fileType?: string
|
||||
signal: AbortSignal | undefined
|
||||
}): Promise<{ buffer: Buffer; contentType: string }> {
|
||||
const { buffer, filename, storageKey, workspaceId, options, ownerKey, filePrincipal, signal } =
|
||||
params
|
||||
const {
|
||||
buffer,
|
||||
filename,
|
||||
storageKey,
|
||||
workspaceId,
|
||||
options,
|
||||
ownerKey,
|
||||
filePrincipal,
|
||||
fileType,
|
||||
signal,
|
||||
} = params
|
||||
if (options.raw) return { buffer, contentType: getContentType(filename) }
|
||||
|
||||
// The pdf model for pages: a page file stores its SOURCE (frontmatter +
|
||||
// markdown + sim: fences) and serving compiles it to the rendered document,
|
||||
// the same way a .pdf key stores its script and serves the binary. Raw
|
||||
// requests above still return the source; bespoke/legacy HTML falls through
|
||||
// untouched. Sim pages store an EXTENSIONLESS name — the record type marks
|
||||
// them; legacy pages still carry .html.
|
||||
if (fileType === SIM_PAGE_CONTENT_TYPE || filename.toLowerCase().endsWith('.html')) {
|
||||
const text = buffer.toString('utf8')
|
||||
if (isSimPageSource(text)) {
|
||||
return {
|
||||
buffer: Buffer.from(await renderSimPageDocumentWithAssets(text, { workspaceId }), 'utf8'),
|
||||
contentType: 'text/html',
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (options.preview) {
|
||||
// Images resolve independently of the document path: a HEIF has no compiled-source
|
||||
// concept, so it never reaches the doc branch.
|
||||
@@ -271,6 +300,7 @@ async function handleWorkspaceFile(
|
||||
options,
|
||||
ownerKey,
|
||||
filePrincipal: principal,
|
||||
fileType: file.type,
|
||||
signal: request.signal,
|
||||
})
|
||||
|
||||
|
||||
@@ -237,11 +237,21 @@ export function encodeFilenameForHeader(storageKey: string): string {
|
||||
}
|
||||
|
||||
export function createFileResponse(file: FileResponse): NextResponse {
|
||||
const { contentType, disposition } = getSecureFileHeaders(file.filename, file.contentType)
|
||||
// Sim pages store an extensionless name and serve/download as compiled
|
||||
// HTML — re-append the extension so the saved file opens in a browser.
|
||||
// Decided from the CALLER's content type (getSecureFileHeaders downgrades
|
||||
// text/html), and BEFORE the header decision, so the .html name gets the
|
||||
// same forced-attachment treatment a legacy .html file gets.
|
||||
const servedFilename =
|
||||
file.contentType === 'text/html' && !/\.[A-Za-z0-9]{1,8}$/.test(file.filename)
|
||||
? `${file.filename}.html`
|
||||
: file.filename
|
||||
|
||||
const { contentType, disposition } = getSecureFileHeaders(servedFilename, file.contentType)
|
||||
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': contentType,
|
||||
'Content-Disposition': `${disposition}; ${encodeFilenameForHeader(file.filename)}`,
|
||||
'Content-Disposition': `${disposition}; ${encodeFilenameForHeader(servedFilename)}`,
|
||||
// Default to PRIVATE: this response is served only after access verification, so it must never be
|
||||
// stored by a shared cache/CDN and re-served cross-user. Genuinely public assets (avatars, OG images,
|
||||
// workspace logos) pass an explicit `cacheControl` (see PUBLIC_ASSET_CACHE_CONTROL in the serve route).
|
||||
|
||||
@@ -237,8 +237,13 @@ describe('mothership private trace provenance transport', () => {
|
||||
'user-1',
|
||||
'secret-value __var_FOREIGN',
|
||||
'workspace-1',
|
||||
'chat-1'
|
||||
'chat-1',
|
||||
expect.any(Object)
|
||||
)
|
||||
|
||||
const contextRegistry = mockProcessContextsServer.mock.calls.at(-1)?.[5]
|
||||
const lifecycleOptions = mockRunHeadlessCopilotLifecycle.mock.calls.at(-1)?.[1]
|
||||
expect(contextRegistry).toBe(lifecycleOptions.environmentContext?.resolvedSecretTraceRegistry)
|
||||
})
|
||||
|
||||
it('keeps context routing and display inputs raw until the lifecycle boundary', async () => {
|
||||
@@ -290,7 +295,8 @@ describe('mothership private trace provenance transport', () => {
|
||||
'user-1',
|
||||
'hello',
|
||||
'workspace-1',
|
||||
'chat-1'
|
||||
'chat-1',
|
||||
expect.any(Object)
|
||||
)
|
||||
})
|
||||
|
||||
|
||||
@@ -244,7 +244,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
userId,
|
||||
lastUserMessage,
|
||||
workspaceId,
|
||||
effectiveChatId
|
||||
effectiveChatId,
|
||||
activeResolvedSecretTraceRegistry
|
||||
).catch((error) => {
|
||||
reqLogger.warn('Failed to resolve agent contexts for execution', {
|
||||
error: toError(error).message,
|
||||
|
||||
@@ -21,14 +21,13 @@ import { getUserEntityPermissions } from '@/lib/workspaces/permissions/utils'
|
||||
import { getWorkspaceOrganizationId } from '@/lib/workspaces/utils'
|
||||
|
||||
/**
|
||||
* Gate for the v2 tables HTTP API (`tables-v2-api` flag). Returns a 404 response
|
||||
* when the flag is off for the caller — the surface behaves as if it doesn't
|
||||
* exist — or `null` to proceed. Gated by userId + the workspace's org cohort.
|
||||
*
|
||||
* **Call this AFTER the authz check, never before.** Ahead of authz it does a
|
||||
* primary-DB read keyed on a caller-supplied `workspaceId`, and the 404-vs-403
|
||||
* split tells an unauthorized caller whether that workspace's org is in the
|
||||
* rollout cohort.
|
||||
* Gate for the internal predicate-grammar table query route (`tables-v2-api`
|
||||
* flag). Runs AFTER authorization, so the caller has already proven read
|
||||
* access to the table — hiding the gate behind a bare 404 at that point
|
||||
* serves nobody and reads as data loss (live incident: the table_v2 block
|
||||
* hard-"Not found"-ing on every query while the copilot gateway, which
|
||||
* bypasses HTTP, found the rows). Authorized callers get an honest 403
|
||||
* naming the gate instead.
|
||||
*/
|
||||
export async function tablesV2GateError(
|
||||
userId: string,
|
||||
@@ -36,7 +35,13 @@ export async function tablesV2GateError(
|
||||
): Promise<NextResponse | null> {
|
||||
const orgId = await getWorkspaceOrganizationId(workspaceId)
|
||||
if (await isFeatureEnabled('tables-v2-api', { userId, orgId })) return null
|
||||
return NextResponse.json({ error: 'Not found' }, { status: 404 })
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: 'The v2 table query API is not enabled for this workspace',
|
||||
code: 'tables_v2_disabled',
|
||||
},
|
||||
{ status: 403 }
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -852,8 +852,13 @@ describe('workflow execute async route', () => {
|
||||
status: 'pending',
|
||||
},
|
||||
{ id: 'copilot-run-1', userId: 'session-user-1', workflowId: 'workflow-1' },
|
||||
403,
|
||||
'COPILOT_WORKFLOW_TOOL_BINDING_AWAITING_APPROVAL',
|
||||
],
|
||||
[
|
||||
// A finished call is a benign duplicate, not a defect: some other runner
|
||||
// already owns this tool call, so it reports the same conflict the
|
||||
// execution claim does and the client stays silent.
|
||||
'terminal tool row',
|
||||
{
|
||||
toolCallId: 'copilot-tool-1',
|
||||
@@ -863,6 +868,8 @@ describe('workflow execute async route', () => {
|
||||
status: 'completed',
|
||||
},
|
||||
{ id: 'copilot-run-1', userId: 'session-user-1', workflowId: 'workflow-1' },
|
||||
409,
|
||||
'COPILOT_WORKFLOW_EXECUTION_CONFLICT',
|
||||
],
|
||||
[
|
||||
'different workflow target',
|
||||
@@ -874,6 +881,8 @@ describe('workflow execute async route', () => {
|
||||
status: 'running',
|
||||
},
|
||||
{ id: 'copilot-run-1', userId: 'session-user-1', workflowId: 'workflow-1' },
|
||||
403,
|
||||
'COPILOT_WORKFLOW_TOOL_BINDING_WORKFLOW_MISMATCH',
|
||||
],
|
||||
[
|
||||
'different execution actor',
|
||||
@@ -885,19 +894,28 @@ describe('workflow execute async route', () => {
|
||||
status: 'running',
|
||||
},
|
||||
{ id: 'copilot-run-1', userId: 'other-user', workflowId: 'workflow-1' },
|
||||
403,
|
||||
'COPILOT_WORKFLOW_TOOL_BINDING_FOREIGN_OWNER',
|
||||
],
|
||||
])('rejects a Copilot binding owned by a %s', async (_caseName, toolCall, run) => {
|
||||
mockGetAsyncToolCall.mockResolvedValueOnce(toolCall)
|
||||
mockGetRunSegment.mockResolvedValueOnce(run)
|
||||
['missing tool row', null, null, 404, 'COPILOT_WORKFLOW_TOOL_BINDING_UNKNOWN'],
|
||||
])(
|
||||
'rejects a Copilot binding owned by a %s',
|
||||
async (_caseName, toolCall, run, expectedStatus, expectedCode) => {
|
||||
mockGetAsyncToolCall.mockResolvedValueOnce(toolCall)
|
||||
mockGetRunSegment.mockResolvedValueOnce(run)
|
||||
|
||||
const response = await POST(createBoundCopilotExecutionRequest(), {
|
||||
params: Promise.resolve({ id: 'workflow-1' }),
|
||||
})
|
||||
const response = await POST(createBoundCopilotExecutionRequest(), {
|
||||
params: Promise.resolve({ id: 'workflow-1' }),
|
||||
})
|
||||
|
||||
expect(response.status).toBe(403)
|
||||
expect(mockExecuteWorkflowCore).not.toHaveBeenCalled()
|
||||
expect(loggingSessionMockFns.mockSetTrustedExecutionCorrelation).not.toHaveBeenCalled()
|
||||
})
|
||||
expect(response.status).toBe(expectedStatus)
|
||||
// The reason must be machine-readable — an opaque 403 is what stopped the
|
||||
// client telling a benign duplicate from a real failure.
|
||||
await expect(response.json()).resolves.toMatchObject({ code: expectedCode })
|
||||
expect(mockExecuteWorkflowCore).not.toHaveBeenCalled()
|
||||
expect(loggingSessionMockFns.mockSetTrustedExecutionCorrelation).not.toHaveBeenCalled()
|
||||
}
|
||||
)
|
||||
|
||||
it('rejects Copilot workflow bindings outside the interactive SSE surface', async () => {
|
||||
const response = await POST(createBoundCopilotExecutionRequest({ stream: false }), {
|
||||
|
||||
@@ -21,7 +21,6 @@ import {
|
||||
type BillingAttributionSnapshot,
|
||||
requireBillingAttributionHeader,
|
||||
} from '@/lib/billing/core/billing-attribution'
|
||||
import { isWorkflowToolExecutionClaimable } from '@/lib/copilot/async-runs/lifecycle'
|
||||
import {
|
||||
claimWorkflowToolExecution,
|
||||
getAsyncToolCall,
|
||||
@@ -29,10 +28,12 @@ import {
|
||||
releaseWorkflowToolExecutionClaim,
|
||||
} from '@/lib/copilot/async-runs/repository'
|
||||
import { COPILOT_WORKFLOW_EXECUTION_CONFLICT_CODE } from '@/lib/copilot/constants'
|
||||
import { CopilotDegradedReason } from '@/lib/copilot/generated/trace-attribute-values-v1'
|
||||
import { recordDegraded } from '@/lib/copilot/request/metrics'
|
||||
import {
|
||||
ASYNC_WORKFLOW_DEPLOYMENT_ERRORS,
|
||||
isWorkflowToolName,
|
||||
resolveWorkflowToolTargetId,
|
||||
type CopilotWorkflowToolBindingResult,
|
||||
classifyWorkflowToolBinding,
|
||||
} from '@/lib/copilot/tools/workflow-tools'
|
||||
import { admissionRejectedResponse, tryAdmit } from '@/lib/core/admission/gate'
|
||||
import {
|
||||
@@ -168,25 +169,19 @@ const SERVER_EXECUTION_ID_CLAIM_ATTEMPTS = 3
|
||||
export const runtime = 'nodejs'
|
||||
export const dynamic = 'force-dynamic'
|
||||
|
||||
async function isValidCopilotWorkflowToolBinding(params: {
|
||||
async function resolveCopilotWorkflowToolBinding(params: {
|
||||
toolCallId: string
|
||||
userId: string
|
||||
workflowId: string
|
||||
}): Promise<boolean> {
|
||||
}): Promise<CopilotWorkflowToolBindingResult> {
|
||||
const toolCall = await getAsyncToolCall(params.toolCallId)
|
||||
if (
|
||||
!toolCall ||
|
||||
!isWorkflowToolName(toolCall.toolName) ||
|
||||
!isWorkflowToolExecutionClaimable(toolCall.status, toolCall.permissionDecision)
|
||||
) {
|
||||
return false
|
||||
}
|
||||
|
||||
const run = await getRunSegment(toolCall.runId)
|
||||
return (
|
||||
run?.userId === params.userId &&
|
||||
resolveWorkflowToolTargetId(toolCall.args, run.workflowId) === params.workflowId
|
||||
)
|
||||
const run = toolCall ? await getRunSegment(toolCall.runId) : null
|
||||
return classifyWorkflowToolBinding({
|
||||
toolCall,
|
||||
run,
|
||||
userId: params.userId,
|
||||
workflowId: params.workflowId,
|
||||
})
|
||||
}
|
||||
|
||||
function createExecutionJsonResponse(
|
||||
@@ -1021,18 +1016,29 @@ async function handleExecutePost(
|
||||
)
|
||||
}
|
||||
|
||||
if (
|
||||
copilotToolCallId &&
|
||||
!(await isValidCopilotWorkflowToolBinding({
|
||||
if (copilotToolCallId) {
|
||||
const binding = await resolveCopilotWorkflowToolBinding({
|
||||
toolCallId: copilotToolCallId,
|
||||
userId,
|
||||
workflowId,
|
||||
}))
|
||||
) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Copilot workflow tool binding was not found' },
|
||||
{ status: 403 }
|
||||
)
|
||||
})
|
||||
if (!binding.ok) {
|
||||
// This rejection happens before any LoggingSession exists, so it leaves
|
||||
// no execution log and no workflow span — log the reason or it is
|
||||
// invisible everywhere except the browser console.
|
||||
// This rejection happens before a LoggingSession or any workflow span
|
||||
// exists, so the counter is the only place it becomes visible.
|
||||
recordDegraded(CopilotDegradedReason.BindingRejected)
|
||||
reqLogger.warn('Rejected Copilot workflow tool execution', {
|
||||
copilotToolCallId,
|
||||
workflowId,
|
||||
reason: binding.rejection.code,
|
||||
})
|
||||
return NextResponse.json(
|
||||
{ error: binding.rejection.message, code: binding.rejection.code },
|
||||
{ status: binding.rejection.statusCode }
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (inputFromExecutionId) {
|
||||
|
||||
+19
-9
@@ -35,7 +35,10 @@ vi.mock('@sim/emcn/icons', () => ({
|
||||
X: () => <span data-icon='x' />,
|
||||
}))
|
||||
|
||||
import { TableFind, type TableFindProps } from './table-find'
|
||||
import {
|
||||
FindBar,
|
||||
type FindBarProps,
|
||||
} from '@/app/workspace/[workspaceId]/components/find-bar/find-bar'
|
||||
|
||||
let container: HTMLDivElement
|
||||
let root: Root
|
||||
@@ -53,16 +56,15 @@ afterEach(() => {
|
||||
container.remove()
|
||||
})
|
||||
|
||||
function render(overrides: Partial<TableFindProps> = {}) {
|
||||
const props: TableFindProps = {
|
||||
function render(overrides: Partial<FindBarProps> = {}) {
|
||||
const props: FindBarProps = {
|
||||
ariaLabel: 'Find in table',
|
||||
query: '',
|
||||
onQueryChange: vi.fn(),
|
||||
onNext: vi.fn(),
|
||||
onPrev: vi.fn(),
|
||||
onSubmit: vi.fn(),
|
||||
onClose: vi.fn(),
|
||||
isStale: false,
|
||||
canNavigate: true,
|
||||
count: 0,
|
||||
currentIndex: 0,
|
||||
truncated: false,
|
||||
@@ -70,7 +72,7 @@ function render(overrides: Partial<TableFindProps> = {}) {
|
||||
inputRef: createRef<HTMLInputElement>(),
|
||||
...overrides,
|
||||
}
|
||||
act(() => root.render(<TableFind {...props} />))
|
||||
act(() => root.render(<FindBar {...props} />))
|
||||
return props
|
||||
}
|
||||
|
||||
@@ -96,7 +98,7 @@ function press(key: string, init: KeyboardEventInit = {}) {
|
||||
})
|
||||
}
|
||||
|
||||
describe('TableFind counter', () => {
|
||||
describe('FindBar counter', () => {
|
||||
it('shows nothing before the user has typed', () => {
|
||||
render({ query: '' })
|
||||
expect(counterText()).toBe('')
|
||||
@@ -138,7 +140,7 @@ describe('TableFind counter', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('TableFind keyboard', () => {
|
||||
describe('FindBar keyboard', () => {
|
||||
it('navigates on Enter rather than submitting a search', () => {
|
||||
const props = render({ query: 'a', count: 3 })
|
||||
press('Enter')
|
||||
@@ -190,9 +192,17 @@ describe('TableFind keyboard', () => {
|
||||
expect(props.onSubmit).toHaveBeenCalledTimes(2)
|
||||
expect(props.onPrev).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
// A surface that matches synchronously passes neither flag; Enter must still
|
||||
// step rather than stall on the defaults.
|
||||
it('steps on Enter when the surface declares no staleness model', () => {
|
||||
const props = render({ query: 'a', count: 3, onSubmit: undefined })
|
||||
press('Enter')
|
||||
expect(props.onNext).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('TableFind controls', () => {
|
||||
describe('FindBar controls', () => {
|
||||
it('offers a clear button only once there is text', () => {
|
||||
render({ query: '' })
|
||||
expect(container.querySelector('button[aria-label="Clear search"]')).toBeNull()
|
||||
+36
-20
@@ -5,61 +5,77 @@ import { memo } from 'react'
|
||||
import { Button, ChipInput } from '@sim/emcn'
|
||||
import { ChevronDown, ChevronUp, Loader, Search, X } from '@sim/emcn/icons'
|
||||
|
||||
export interface TableFindProps {
|
||||
export interface FindBarProps {
|
||||
/** Accessible name for the input, naming the surface: "Find in table", "Find in files". */
|
||||
ariaLabel: string
|
||||
query: string
|
||||
onQueryChange: (query: string) => void
|
||||
onNext: () => void
|
||||
onPrev: () => void
|
||||
/** Adopts the typed term immediately, skipping the debounce. */
|
||||
onSubmit: () => void
|
||||
onClose: () => void
|
||||
/** Whether the typed term has yet to be searched, so Enter should commit it. */
|
||||
isStale: boolean
|
||||
/**
|
||||
* Adopts the typed term immediately, skipping the surface's debounce. Only
|
||||
* meaningful on a surface that debounces; omit it and Enter always steps.
|
||||
*/
|
||||
onSubmit?: () => void
|
||||
/**
|
||||
* Whether the typed term has yet to be searched, so Enter should commit it
|
||||
* rather than step. Defaults to false — a surface that searches
|
||||
* synchronously is never stale.
|
||||
*/
|
||||
isStale?: boolean
|
||||
/**
|
||||
* Whether the matches on screen belong to the term that was searched. False
|
||||
* while a term's own results are in flight, when the count still describes
|
||||
* the previous term and stepping through it would land on a cell the box no
|
||||
* longer names.
|
||||
* longer names. Defaults to true for surfaces that match synchronously.
|
||||
*/
|
||||
canNavigate: boolean
|
||||
/** Number of matches after dropping columns not in the current view. */
|
||||
canNavigate?: boolean
|
||||
/** Number of matches after dropping any the current view cannot show. */
|
||||
count: number
|
||||
/** 0-based index of the active match. Ignored when `count` is 0. */
|
||||
currentIndex: number
|
||||
/** Whether the server capped the match set. */
|
||||
/** Whether the producer capped the match set. */
|
||||
truncated: boolean
|
||||
isLoading: boolean
|
||||
inputRef: React.RefObject<HTMLInputElement | null>
|
||||
}
|
||||
|
||||
/**
|
||||
* Memoized: while the bar is open it is a child of the grid, which re-renders
|
||||
* on scroll, hover and selection. Every prop is a primitive or a stable
|
||||
* The find bar every Cmd/Ctrl+F surface shares (tables, files, ...). Purely
|
||||
* presentational and fully controlled: the surface owns the query, the match
|
||||
* model and the stepping; this renders the input, the tally and the
|
||||
* next/prev/close controls. Positioned absolutely against the nearest
|
||||
* relative container, top-right, the way an in-page find sits in Chrome.
|
||||
*
|
||||
* Memoized: while the bar is open it is a child of a view that re-renders on
|
||||
* scroll, hover and selection. Every prop is a primitive or a stable
|
||||
* identity, so this collapses to renders where a find value actually changed.
|
||||
*/
|
||||
export const TableFind = memo(function TableFind({
|
||||
export const FindBar = memo(function FindBar({
|
||||
ariaLabel,
|
||||
query,
|
||||
onQueryChange,
|
||||
onNext,
|
||||
onPrev,
|
||||
onSubmit,
|
||||
onClose,
|
||||
isStale,
|
||||
canNavigate,
|
||||
onSubmit,
|
||||
isStale = false,
|
||||
canNavigate = true,
|
||||
count,
|
||||
currentIndex,
|
||||
truncated,
|
||||
isLoading,
|
||||
inputRef,
|
||||
}: TableFindProps) {
|
||||
}: FindBarProps) {
|
||||
const handleKeyDown = (e: React.KeyboardEvent<HTMLInputElement>) => {
|
||||
if (e.key === 'Enter') {
|
||||
e.preventDefault()
|
||||
// Commit an unsearched term; otherwise step — but only once the results
|
||||
// describe it. In between (committed, still loading) Enter does nothing
|
||||
// rather than walk the previous term's matches; the auto-reveal lands on
|
||||
// the first hit as soon as they arrive.
|
||||
if (isStale) onSubmit()
|
||||
// rather than walk the previous term's matches; the surface's auto-reveal
|
||||
// lands on the first hit as soon as they arrive.
|
||||
if (isStale && onSubmit) onSubmit()
|
||||
else if (!canNavigate) return
|
||||
else if (e.shiftKey) onPrev()
|
||||
else onNext()
|
||||
@@ -89,7 +105,7 @@ export const TableFind = memo(function TableFind({
|
||||
ref={inputRef}
|
||||
value={query}
|
||||
placeholder='Search'
|
||||
aria-label='Find in table'
|
||||
aria-label={ariaLabel}
|
||||
spellCheck={false}
|
||||
autoComplete='off'
|
||||
icon={Search}
|
||||
@@ -6,6 +6,8 @@ export { ResourceNotFound } from '@/app/workspace/[workspaceId]/components/resou
|
||||
export { ConversationListItem } from './conversation-list-item'
|
||||
export type { ErrorBoundaryProps, ErrorStateProps } from './error'
|
||||
export { ErrorShell, ErrorState } from './error'
|
||||
export type { FindBarProps } from './find-bar/find-bar'
|
||||
export { FindBar } from './find-bar/find-bar'
|
||||
export { InlineRenameInput } from './inline-rename-input'
|
||||
export { IntegrationTabsHeader } from './integration-tabs-header'
|
||||
export { MessageActions } from './message-actions'
|
||||
@@ -52,4 +54,5 @@ export { selectionLabel } from './resource/selection-label'
|
||||
export type { ResourceRowSelection } from './resource/use-resource-row-selection'
|
||||
export { useResourceRowSelection } from './resource/use-resource-row-selection'
|
||||
export { ResourceTile } from './resource-tile'
|
||||
export { SearchHighlight } from './search-highlight/search-highlight'
|
||||
export { SkillTile } from './skill-tile'
|
||||
|
||||
@@ -31,6 +31,7 @@ import { FloatingOverflowText } from '@/app/workspace/[workspaceId]/components/r
|
||||
import type { BreadcrumbDropConfig } from '@/app/workspace/[workspaceId]/components/resource/components/resource-header'
|
||||
import { ResourceHeader } from '@/app/workspace/[workspaceId]/components/resource/components/resource-header'
|
||||
import { ResourceOptions } from '@/app/workspace/[workspaceId]/components/resource/components/resource-options'
|
||||
import { SearchHighlight } from '@/app/workspace/[workspaceId]/components/search-highlight/search-highlight'
|
||||
|
||||
export interface ResourceColumn {
|
||||
id: string
|
||||
@@ -72,6 +73,12 @@ export interface ResourceCell {
|
||||
* layout, and the rename field replaces the label entirely while it is open.
|
||||
*/
|
||||
pinned?: boolean
|
||||
/**
|
||||
* Find term to tint inside the label (Cmd/Ctrl+F match). Honoured only on the
|
||||
* plain label cell, like `pinned` — a `content` cell owns its own rendering
|
||||
* and the rename field replaces the label while open.
|
||||
*/
|
||||
highlight?: string
|
||||
}
|
||||
|
||||
export interface ResourceRow {
|
||||
@@ -559,6 +566,7 @@ interface CellContentProps {
|
||||
content?: ReactNode
|
||||
editing?: ResourceCellEditing
|
||||
pinned?: boolean
|
||||
highlight?: string
|
||||
}
|
||||
|
||||
const CellContent = memo(function CellContent({
|
||||
@@ -567,6 +575,7 @@ const CellContent = memo(function CellContent({
|
||||
content,
|
||||
editing,
|
||||
pinned,
|
||||
highlight,
|
||||
}: CellContentProps) {
|
||||
if (editing) {
|
||||
return (
|
||||
@@ -586,7 +595,9 @@ const CellContent = memo(function CellContent({
|
||||
return (
|
||||
<span className={cn('flex min-w-0 items-center', chipContentGap)}>
|
||||
{icon && <span className={cellIconNodeClass}>{icon}</span>}
|
||||
<FloatingOverflowText label={label} className={cn('block', chipContentLabelClass)} />
|
||||
<FloatingOverflowText label={label} className={cn('block', chipContentLabelClass)}>
|
||||
{highlight ? <SearchHighlight text={label} searchQuery={highlight} /> : undefined}
|
||||
</FloatingOverflowText>
|
||||
{pinned && (
|
||||
<Pin
|
||||
className='size-[12px] shrink-0 text-[var(--text-icon)]'
|
||||
@@ -767,6 +778,7 @@ const DataRow = memo(function DataRow({
|
||||
content={cell?.content}
|
||||
editing={cell?.editing}
|
||||
pinned={cell?.pinned}
|
||||
highlight={cell?.highlight}
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
|
||||
+65
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
import { useEffect, useLayoutEffect, useRef, useState } from 'react'
|
||||
import { cn } from '@sim/emcn'
|
||||
import { PanelLeft } from '@sim/emcn/icons'
|
||||
import { ArrowLeft, ArrowRight, PanelLeft } from '@sim/emcn/icons'
|
||||
import { usePathname } from 'next/navigation'
|
||||
import { getDesktopBridge } from '@/lib/desktop'
|
||||
import { applyDesktopTitleBarMode, type DesktopTitleBarMode } from '@/app/_shell/desktop-title-bar'
|
||||
@@ -86,6 +86,69 @@ interface WorkspaceChromeProps {
|
||||
initialSidebarCollapsed?: boolean
|
||||
}
|
||||
|
||||
/** Chromium Navigation API slice (absent from TS lib.dom). */
|
||||
type ChromiumNavigation = EventTarget & { canGoBack: boolean; canGoForward: boolean }
|
||||
|
||||
const LANE_NAV_BUTTON =
|
||||
'flex size-[var(--desktop-title-bar-control-size)] items-center justify-center rounded-lg transition-colors disabled:pointer-events-none disabled:opacity-40 hover-hover:bg-[var(--surface-active)]'
|
||||
const LANE_NAV_ICON = 'size-[var(--desktop-title-bar-control-icon-size)] text-[var(--text-icon)]'
|
||||
|
||||
/**
|
||||
* Back/forward history arrows in the desktop title-bar lane, right of the
|
||||
* sidebar toggle. Only the macOS shell sets the inset attribute, so the web
|
||||
* app never shows them; the shell's renderer is Chromium, so the Navigation
|
||||
* API is always there for the arrow state.
|
||||
*/
|
||||
function TitleBarHistoryNav() {
|
||||
const [can, setCan] = useState({ back: false, forward: false })
|
||||
|
||||
useEffect(() => {
|
||||
const nav = (window as { navigation?: ChromiumNavigation }).navigation
|
||||
if (!nav) return
|
||||
let disposed = false
|
||||
// currententrychange dispatches SYNCHRONOUSLY from the history mutation
|
||||
// that caused it, which can originate inside another component's
|
||||
// useInsertionEffect (style libraries navigate during commit). Scheduling
|
||||
// state there is forbidden ("useInsertionEffect must not schedule
|
||||
// updates"), so the update defers to a microtask, which flushes after
|
||||
// the commit's synchronous work unwinds.
|
||||
const sync = () => {
|
||||
queueMicrotask(() => {
|
||||
if (!disposed) setCan({ back: nav.canGoBack, forward: nav.canGoForward })
|
||||
})
|
||||
}
|
||||
sync()
|
||||
nav.addEventListener('currententrychange', sync)
|
||||
return () => {
|
||||
disposed = true
|
||||
nav.removeEventListener('currententrychange', sync)
|
||||
}
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<div className='absolute top-[var(--desktop-title-bar-control-offset)] left-[calc(var(--desktop-title-bar-inset-x)+var(--desktop-title-bar-control-size)+4px)] z-30 hidden h-[var(--desktop-title-bar-control-size)] items-center gap-[2px] [-webkit-app-region:no-drag] [[data-sim-desktop-title-bar=inset]_&]:flex'>
|
||||
<button
|
||||
type='button'
|
||||
aria-label='Back'
|
||||
disabled={!can.back}
|
||||
onClick={() => window.history.back()}
|
||||
className={LANE_NAV_BUTTON}
|
||||
>
|
||||
<ArrowLeft className={LANE_NAV_ICON} />
|
||||
</button>
|
||||
<button
|
||||
type='button'
|
||||
aria-label='Forward'
|
||||
disabled={!can.forward}
|
||||
onClick={() => window.history.forward()}
|
||||
className={LANE_NAV_BUTTON}
|
||||
>
|
||||
<ArrowRight className={LANE_NAV_ICON} />
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function isFullscreenPath(pathname: string | null): boolean {
|
||||
return FULLSCREEN_SUFFIXES.some((s) => pathname?.endsWith(s))
|
||||
}
|
||||
@@ -374,6 +437,7 @@ export function WorkspaceChrome({
|
||||
</SidebarTooltip>
|
||||
</div>
|
||||
)}
|
||||
{!isFullscreen && <TitleBarHistoryNav />}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
'use client'
|
||||
|
||||
import { memo, useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { getErrorMessage } from '@sim/utils/errors'
|
||||
import type { EChartsOption } from 'echarts'
|
||||
import { useTheme } from 'next-themes'
|
||||
import { buildChartRenderOption } from '@/lib/charts/option'
|
||||
import {
|
||||
CHART_ROWS_DEFAULT,
|
||||
CHART_ROWS_MAX,
|
||||
type ChartSpec,
|
||||
mapRowsToColumnNames,
|
||||
parseChartSpec,
|
||||
shapeTableRows,
|
||||
} from '@/lib/charts/spec'
|
||||
import { useTable, useTableRowsSample } from '@/hooks/queries/tables'
|
||||
import { PreviewLoadingFrame } from './preview-shared'
|
||||
|
||||
function buildOption(spec: ChartSpec, rows: Array<Record<string, unknown>> | null): EChartsOption {
|
||||
return buildChartRenderOption({ title: spec.title, option: spec.option, rows }) as EChartsOption
|
||||
}
|
||||
|
||||
function ChartErrorCard({ message, content }: { message: string; content: string }) {
|
||||
return (
|
||||
<div className='overflow-hidden rounded-lg border border-[var(--border)]'>
|
||||
<div className='flex items-center justify-between border-[var(--border)] border-b bg-[var(--surface-3)] px-3 py-1.5'>
|
||||
<span className='text-[11px] text-[var(--text-tertiary)]'>chart</span>
|
||||
<span className='text-[11px] text-[var(--text-muted)]'>{message}</span>
|
||||
</div>
|
||||
<div className='code-editor-theme bg-[var(--surface-5)]'>
|
||||
<pre className='m-0 overflow-x-auto whitespace-pre p-4 font-mono text-[13px] text-[var(--text-primary)] leading-[1.6]'>
|
||||
<code>{content}</code>
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function ChartErrorPanel({ message, content }: { message: string; content: string }) {
|
||||
return (
|
||||
<div className='min-h-0 flex-1 overflow-auto p-6'>
|
||||
<ChartErrorCard message={message} content={content} />
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
type EChartsModule = typeof import('echarts')
|
||||
|
||||
/**
|
||||
* Renders a `.chart` document with ECharts, lazy-loading the (heavy) library
|
||||
* on first use. Static sources render inline rows; table sources read the
|
||||
* table live through React Query, so the chart reflects the table's current
|
||||
* data every time it is opened.
|
||||
*/
|
||||
export const ChartPreview = memo(function ChartPreview({
|
||||
content,
|
||||
workspaceId,
|
||||
isStreaming = false,
|
||||
}: {
|
||||
content: string
|
||||
workspaceId: string
|
||||
isStreaming?: boolean
|
||||
}) {
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const [echartsLib, setEchartsLib] = useState<EChartsModule | null>(null)
|
||||
const [loadError, setLoadError] = useState<string | null>(null)
|
||||
const [renderError, setRenderError] = useState<string | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
let active = true
|
||||
import('echarts')
|
||||
.then((mod) => {
|
||||
if (active) setEchartsLib(mod)
|
||||
})
|
||||
.catch((e) => {
|
||||
if (active) setLoadError(getErrorMessage(e, 'failed to load the chart renderer'))
|
||||
})
|
||||
return () => {
|
||||
active = false
|
||||
}
|
||||
}, [])
|
||||
|
||||
const { spec, error: parseError } = useMemo(() => parseChartSpec(content), [content])
|
||||
|
||||
const tableSource = spec?.source?.type === 'table' ? spec.source : null
|
||||
const rowsQuery = useTableRowsSample({
|
||||
workspaceId,
|
||||
tableId: tableSource?.tableId,
|
||||
filter: tableSource?.filter,
|
||||
sort: tableSource?.sort,
|
||||
limit: Math.min(tableSource?.limit ?? CHART_ROWS_DEFAULT, CHART_ROWS_MAX),
|
||||
enabled: Boolean(tableSource),
|
||||
})
|
||||
const tableQuery = useTable(tableSource ? workspaceId : undefined, tableSource?.tableId)
|
||||
|
||||
const rows = useMemo(() => {
|
||||
if (!spec) return null
|
||||
if (spec.source?.type === 'static') return spec.source.rows ?? null
|
||||
if (!tableSource) return null
|
||||
const fetched = rowsQuery.data?.rows
|
||||
const columns = tableQuery.data?.schema.columns
|
||||
if (!fetched || !columns) return null
|
||||
return shapeTableRows(mapRowsToColumnNames(fetched, columns), tableSource)
|
||||
}, [spec, tableSource, rowsQuery.data, tableQuery.data])
|
||||
|
||||
const option = useMemo(() => (spec ? buildOption(spec, rows) : null), [spec, rows])
|
||||
/** Stable identity for the effect below — option is a fresh object per memo. */
|
||||
const optionKey = useMemo(() => (option ? JSON.stringify(option) : ''), [option])
|
||||
|
||||
const { resolvedTheme } = useTheme()
|
||||
|
||||
useEffect(() => {
|
||||
setRenderError(null)
|
||||
const el = containerRef.current
|
||||
if (!el || !echartsLib || !option) return
|
||||
const chart = echartsLib.init(el, resolvedTheme === 'dark' ? 'dark' : undefined)
|
||||
try {
|
||||
chart.setOption(option)
|
||||
} catch (e) {
|
||||
setRenderError(getErrorMessage(e, 'invalid ECharts option'))
|
||||
chart.dispose()
|
||||
return
|
||||
}
|
||||
const resizeObserver = new ResizeObserver(() => chart.resize())
|
||||
resizeObserver.observe(el)
|
||||
return () => {
|
||||
resizeObserver.disconnect()
|
||||
chart.dispose()
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [echartsLib, optionKey, resolvedTheme])
|
||||
|
||||
if (parseError) {
|
||||
// A file the agent is still writing is expected to be truncated JSON.
|
||||
if (isStreaming) return <PreviewLoadingFrame className='h-full flex-1' />
|
||||
return <ChartErrorPanel message={parseError} content={content} />
|
||||
}
|
||||
if (loadError) return <ChartErrorPanel message={loadError} content={content} />
|
||||
if (tableSource && rowsQuery.isError) {
|
||||
return (
|
||||
<ChartErrorPanel
|
||||
message={getErrorMessage(rowsQuery.error, 'failed to read the table')}
|
||||
content={content}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
if (tableSource && tableQuery.isError) {
|
||||
return (
|
||||
<ChartErrorPanel
|
||||
message={getErrorMessage(tableQuery.error, 'failed to read the table')}
|
||||
content={content}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
const waitingOnRows = Boolean(tableSource) && rows === null
|
||||
// Width-driven aspect box, not full-bleed: a chart stretched to the whole
|
||||
// panel height is unreadable in a tall resource pane. ECharts follows the
|
||||
// box through the ResizeObserver above.
|
||||
//
|
||||
// A render error (setOption threw) HIDES the chart box rather than
|
||||
// unmounting it: the render effect only re-runs when the option changes,
|
||||
// and it needs the container mounted at that moment to re-initialize —
|
||||
// an unmounted container would leave the fixed chart blank until a
|
||||
// second edit.
|
||||
return (
|
||||
<div className='min-h-0 flex-1 overflow-auto p-6'>
|
||||
{renderError !== null && <ChartErrorCard message={renderError} content={content} />}
|
||||
<div className={renderError !== null ? 'hidden' : 'relative mx-auto w-full max-w-[1024px]'}>
|
||||
{(!echartsLib || waitingOnRows) && (
|
||||
<PreviewLoadingFrame className='absolute inset-0 z-10' />
|
||||
)}
|
||||
<div ref={containerRef} className='aspect-[16/10] min-h-[280px] w-full' />
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
})
|
||||
@@ -8,6 +8,8 @@ const TEXT_EDITABLE_MIME_TYPES = new Set([
|
||||
'application/x-yaml',
|
||||
'text/csv',
|
||||
'text/html',
|
||||
// Sim pages: internal record type; the file itself is .html.
|
||||
'text/x-sim-page',
|
||||
'text/xml',
|
||||
'application/xml',
|
||||
'text/css',
|
||||
@@ -20,6 +22,8 @@ const TEXT_EDITABLE_MIME_TYPES = new Set([
|
||||
'text/x-sql',
|
||||
'image/svg+xml',
|
||||
'text/x-mermaid',
|
||||
// Chart documents: declarative ECharts specs rendered live (chart-preview.tsx).
|
||||
'text/x-sim-chart',
|
||||
])
|
||||
|
||||
const TEXT_EDITABLE_EXTENSIONS = new Set([
|
||||
|
||||
+226
-8
@@ -1,23 +1,39 @@
|
||||
'use client'
|
||||
|
||||
import { memo, useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { useRouter } from 'next/navigation'
|
||||
import { useTheme } from 'next-themes'
|
||||
import '@sim/emcn/components/code/code.css'
|
||||
import { CSV_PREVIEW_MAX_ROWS } from '@/lib/api/contracts/workspace-file-table'
|
||||
import { getFileExtension } from '@/lib/uploads/utils/file-utils'
|
||||
import {
|
||||
SIM_ARTIFACT_SHELL,
|
||||
SIM_ARTIFACT_STYLESHEET,
|
||||
simTokenOverrides,
|
||||
usesSimArtifactStyles,
|
||||
} from '@/lib/workspace-files/artifact-stylesheet'
|
||||
import { compileSimPage, isSimPageSource } from '@/lib/workspace-files/page-compile'
|
||||
import { useHorizontalWheelScroll } from '@/app/workspace/[workspaceId]/files/components/file-viewer/use-horizontal-wheel-scroll'
|
||||
import { useWorkspaceFileBinary } from '@/hooks/queries/workspace-files'
|
||||
import { ChartPreview } from './chart-preview'
|
||||
import { type CsvImportFileDescriptor, useCsvTruncationImport } from './csv-import'
|
||||
import { DataTable } from './data-table'
|
||||
import { MermaidDiagram } from './mermaid-diagram'
|
||||
import { ZoomablePreview } from './zoomable-preview'
|
||||
|
||||
type PreviewType = 'markdown' | 'html' | 'csv' | 'svg' | 'mermaid' | null
|
||||
type PreviewType = 'markdown' | 'html' | 'csv' | 'svg' | 'mermaid' | 'chart' | null
|
||||
|
||||
const PREVIEWABLE_MIME_TYPES: Record<string, PreviewType> = {
|
||||
// Sim pages store an EXTENSIONLESS name — without this mapping the record
|
||||
// type resolves to no preview at all and the viewer renders a blank pane
|
||||
// (including the live compile-as-it-streams view).
|
||||
'text/x-sim-page': 'html',
|
||||
'text/markdown': 'markdown',
|
||||
'text/html': 'html',
|
||||
'text/csv': 'csv',
|
||||
'image/svg+xml': 'svg',
|
||||
'text/x-mermaid': 'mermaid',
|
||||
'text/x-sim-chart': 'chart',
|
||||
}
|
||||
|
||||
const PREVIEWABLE_EXTENSIONS: Record<string, PreviewType> = {
|
||||
@@ -27,6 +43,7 @@ const PREVIEWABLE_EXTENSIONS: Record<string, PreviewType> = {
|
||||
csv: 'csv',
|
||||
svg: 'svg',
|
||||
mmd: 'mermaid',
|
||||
chart: 'chart',
|
||||
}
|
||||
|
||||
/** All extensions that have a rich preview renderer. */
|
||||
@@ -66,7 +83,16 @@ export const PreviewPanel = memo(function PreviewPanel({
|
||||
}: PreviewPanelProps) {
|
||||
const previewType = resolvePreviewType(mimeType, filename)
|
||||
|
||||
if (previewType === 'html') return <HtmlPreview content={content} />
|
||||
if (previewType === 'html')
|
||||
return (
|
||||
<HtmlPreview
|
||||
content={content}
|
||||
isStreaming={isStreaming}
|
||||
workspaceId={workspaceId}
|
||||
fileId={fileId}
|
||||
fileKey={fileKey}
|
||||
/>
|
||||
)
|
||||
if (previewType === 'csv')
|
||||
return (
|
||||
<CsvPreview
|
||||
@@ -79,6 +105,8 @@ export const PreviewPanel = memo(function PreviewPanel({
|
||||
if (previewType === 'svg') return <SvgPreview content={content} />
|
||||
if (previewType === 'mermaid')
|
||||
return <MermaidFilePreview content={content} isStreaming={isStreaming} />
|
||||
if (previewType === 'chart')
|
||||
return <ChartPreview content={content} workspaceId={workspaceId} isStreaming={isStreaming} />
|
||||
|
||||
return null
|
||||
})
|
||||
@@ -112,6 +140,12 @@ const HTML_PREVIEW_BOOTSTRAP = `<script>
|
||||
const href = anchor.getAttribute('href') || ''
|
||||
if (allowHref(href)) return
|
||||
event.preventDefault()
|
||||
// The sandbox can neither navigate nor open windows, so hand the click
|
||||
// to the host: workspace routes go through the app router, external
|
||||
// http(s) links open a new tab.
|
||||
if (href.startsWith('/workspace/') || /^https?:\\/\\//i.test(href)) {
|
||||
parent.postMessage({ __simPageNav: href }, '*')
|
||||
}
|
||||
},
|
||||
true
|
||||
)
|
||||
@@ -127,27 +161,211 @@ const HTML_PREVIEW_BOOTSTRAP = `<script>
|
||||
})()
|
||||
</script>`
|
||||
|
||||
function buildHtmlPreviewDocument(content: string): string {
|
||||
function stampTheme(html: string, theme: 'dark' | 'light'): string {
|
||||
if (/<html[\s>]/i.test(html)) {
|
||||
return html.replace(/<html(\s[^>]*)?>/i, (match, attrs: string | undefined) =>
|
||||
/\sdata-theme=/i.test(attrs ?? '') ? match : `<html data-theme="${theme}"${attrs ?? ''}>`
|
||||
)
|
||||
}
|
||||
return html
|
||||
}
|
||||
|
||||
export function buildHtmlPreviewDocument(
|
||||
content: string,
|
||||
theme: 'dark' | 'light' = 'light',
|
||||
workspaceId?: string
|
||||
): string {
|
||||
// The pdf model: a page file STORES its source (frontmatter + markdown +
|
||||
// sim: fences) and every rendering surface compiles on demand. Partial
|
||||
// source mid-stream compiles too, so the page builds up live as the agent
|
||||
// appends. Raw HTML (bespoke and legacy stored-compiled pages) skips this.
|
||||
if (isSimPageSource(content)) {
|
||||
content = compileSimPage(content, { workspaceId })
|
||||
}
|
||||
const headInjection = [
|
||||
'<meta charset="utf-8">',
|
||||
`<base href="${HTML_PREVIEW_BASE_URL}">`,
|
||||
`<meta http-equiv="Content-Security-Policy" content="${HTML_PREVIEW_CSP}">`,
|
||||
// Ahead of the page's own <style>, so anything it declares still wins. The
|
||||
// token overrides follow the sheet so the app's live values beat its
|
||||
// fallbacks, and the shell follows both.
|
||||
usesSimArtifactStyles(content)
|
||||
? `<style>${SIM_ARTIFACT_STYLESHEET}</style><style>${simTokenOverrides(theme)}</style>${SIM_ARTIFACT_SHELL}`
|
||||
: '',
|
||||
HTML_PREVIEW_BOOTSTRAP,
|
||||
].join('')
|
||||
|
||||
if (/<head[\s>]/i.test(content)) {
|
||||
return content.replace(/<head(\s[^>]*)?>/i, (match) => `${match}${headInjection}`)
|
||||
return stampTheme(
|
||||
content.replace(/<head(\s[^>]*)?>/i, (match) => `${match}${headInjection}`),
|
||||
theme
|
||||
)
|
||||
}
|
||||
|
||||
if (/<html[\s>]/i.test(content)) {
|
||||
return content.replace(/<html(\s[^>]*)?>/i, (match) => `${match}<head>${headInjection}</head>`)
|
||||
return stampTheme(
|
||||
content.replace(/<html(\s[^>]*)?>/i, (match) => `${match}<head>${headInjection}</head>`),
|
||||
theme
|
||||
)
|
||||
}
|
||||
|
||||
return `<!DOCTYPE html><html><head>${headInjection}</head><body>${content}</body></html>`
|
||||
return `<!DOCTYPE html><html data-theme="${theme}"><head>${headInjection}</head><body>${content}</body></html>`
|
||||
}
|
||||
|
||||
const HtmlPreview = memo(function HtmlPreview({ content }: { content: string }) {
|
||||
const wrappedContent = buildHtmlPreviewDocument(content)
|
||||
/**
|
||||
* Batches iframe content updates while an agent streams. Every content change
|
||||
* replaces the srcDoc (a full document reload), so applying each chunk as it
|
||||
* arrives would reload the page several times a second; ~2s batches keep the
|
||||
* growing page readable. Off-stream, the live value passes straight through.
|
||||
*/
|
||||
function useStreamBatchedValue(value: string, streaming: boolean, intervalMs: number): string {
|
||||
const [batched, setBatched] = useState(value)
|
||||
const lastAppliedAtRef = useRef(0)
|
||||
useEffect(() => {
|
||||
if (!streaming) {
|
||||
lastAppliedAtRef.current = 0
|
||||
setBatched(value)
|
||||
return
|
||||
}
|
||||
const elapsed = Date.now() - lastAppliedAtRef.current
|
||||
if (elapsed >= intervalMs) {
|
||||
lastAppliedAtRef.current = Date.now()
|
||||
setBatched(value)
|
||||
return
|
||||
}
|
||||
const timer = setTimeout(() => {
|
||||
lastAppliedAtRef.current = Date.now()
|
||||
setBatched(value)
|
||||
}, intervalMs - elapsed)
|
||||
return () => clearTimeout(timer)
|
||||
}, [value, streaming, intervalMs])
|
||||
return streaming ? batched : value
|
||||
}
|
||||
|
||||
/**
|
||||
* The sandboxed frame carries no cookies, so a workspace image
|
||||
* (`/api/files/view/<id>`, what `` compiles to) would
|
||||
* 401 inside it. The host fetches the bytes with its own session and hands
|
||||
* the frame blob: URLs, which the preview CSP already allows.
|
||||
*/
|
||||
function useInlinedWorkspaceImages(content: string): string {
|
||||
const [resolved, setResolved] = useState<Record<string, string>>({})
|
||||
const cacheRef = useRef<Map<string, string> | null>(null)
|
||||
cacheRef.current ??= new Map()
|
||||
useEffect(() => {
|
||||
const cache = cacheRef.current
|
||||
if (!cache) return
|
||||
const urls = [...content.matchAll(/src="(\/api\/files\/view\/[^"]+)"/g)].map((m) => m[1])
|
||||
const missing = [...new Set(urls)].filter((url) => !cache.has(url))
|
||||
if (missing.length === 0) return
|
||||
let cancelled = false
|
||||
for (const url of missing) {
|
||||
// boundary-raw-fetch: binary image bytes for the cookie-less sandboxed preview
|
||||
fetch(url)
|
||||
.then((response) => (response.ok ? response.blob() : null))
|
||||
.then(
|
||||
(blob) =>
|
||||
new Promise<string | null>((resolve) => {
|
||||
if (!blob) return resolve(null)
|
||||
// data: URIs, NOT blob: URLs — blob URLs are origin-bound and
|
||||
// the sandboxed frame's origin is opaque, so Chromium refuses
|
||||
// to render a parent-origin blob inside it. data: is
|
||||
// origin-independent and already allowed by the preview CSP.
|
||||
const reader = new FileReader()
|
||||
reader.onload = () =>
|
||||
resolve(typeof reader.result === 'string' ? reader.result : null)
|
||||
reader.onerror = () => resolve(null)
|
||||
reader.readAsDataURL(blob)
|
||||
})
|
||||
)
|
||||
.then((dataUri) => {
|
||||
if (!dataUri || cancelled) return
|
||||
cache.set(url, dataUri)
|
||||
setResolved((prev) => ({ ...prev, [url]: dataUri }))
|
||||
})
|
||||
.catch(() => {})
|
||||
}
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [content])
|
||||
return useMemo(
|
||||
() =>
|
||||
content.replace(/src="(\/api\/files\/view\/[^"]+)"/g, (match, url: string) => {
|
||||
const blobUrl = cacheRef.current?.get(url) ?? resolved[url]
|
||||
return blobUrl ? `src="${blobUrl}"` : match
|
||||
}),
|
||||
[content, resolved]
|
||||
)
|
||||
}
|
||||
|
||||
const HtmlPreview = memo(function HtmlPreview({
|
||||
content,
|
||||
isStreaming,
|
||||
workspaceId,
|
||||
fileId,
|
||||
fileKey,
|
||||
}: {
|
||||
content: string
|
||||
isStreaming?: boolean
|
||||
workspaceId?: string
|
||||
fileId?: string
|
||||
fileKey?: string
|
||||
}) {
|
||||
const { resolvedTheme } = useTheme()
|
||||
const router = useRouter()
|
||||
const batchedContent = useStreamBatchedValue(content, isStreaming === true, 2000)
|
||||
// A SAVED sim page prefers the server-compiled document — the pptx/docx
|
||||
// model: the serve route resolves chart references (reading a table's
|
||||
// CURRENT rows under the viewer's authorization) and inlines the chart
|
||||
// runtime, and reopening/refocusing refetches, so the page recompiles on
|
||||
// reload. While it loads — and always while streaming/editing — the client
|
||||
// compile below stands in, with chart figures as placeholders.
|
||||
const isSavedPage =
|
||||
Boolean(fileId && fileKey && workspaceId) && isStreaming !== true && isSimPageSource(content)
|
||||
const served = useWorkspaceFileBinary(workspaceId ?? '', fileId ?? '', fileKey ?? '', {
|
||||
enabled: isSavedPage,
|
||||
})
|
||||
const servedHtml = useMemo(
|
||||
() => (isSavedPage && served.data ? new TextDecoder().decode(served.data) : null),
|
||||
[isSavedPage, served.data]
|
||||
)
|
||||
const builtContent = buildHtmlPreviewDocument(
|
||||
servedHtml ?? batchedContent,
|
||||
resolvedTheme === 'dark' ? 'dark' : 'light',
|
||||
workspaceId
|
||||
)
|
||||
// AFTER the build: workspace image srcs (/api/files/view/…) only exist in
|
||||
// the COMPILED document — the raw source says sim:file/… — so substituting
|
||||
// the host-fetched blob: URLs must run on the built output, or the
|
||||
// sandboxed (cookie-less) frame 401s every image.
|
||||
const wrappedContent = useInlinedWorkspaceImages(builtContent)
|
||||
|
||||
// Receives sim-resource link clicks bridged out of the sandboxed page and
|
||||
// routes them in the app. Only workspace-internal paths are honored.
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
const href = (event.data as { __simPageNav?: unknown } | null)?.__simPageNav
|
||||
if (typeof href !== 'string') return
|
||||
if (href.startsWith('/workspace/')) {
|
||||
router.push(href)
|
||||
} else if (/^https?:\/\//i.test(href)) {
|
||||
// The server-compiled document absolutizes workspace links (so a
|
||||
// DOWNLOADED copy reaches Sim) — recognize our own origin and route
|
||||
// in-app rather than spawning a new tab of the whole app.
|
||||
try {
|
||||
const url = new URL(href)
|
||||
if (url.origin === window.location.origin && url.pathname.startsWith('/workspace/')) {
|
||||
router.push(`${url.pathname}${url.search}${url.hash}`)
|
||||
return
|
||||
}
|
||||
} catch {}
|
||||
window.open(href, '_blank', 'noopener,noreferrer')
|
||||
}
|
||||
}
|
||||
window.addEventListener('message', onMessage)
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [router])
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const [isRenderable, setIsRenderable] = useState(false)
|
||||
const [resumeNonce, setResumeNonce] = useState(0)
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
} from '@/lib/copilot/chat/selection-context'
|
||||
import type { WorkspaceFileRecord } from '@/lib/uploads/contexts/workspace'
|
||||
import { getFileExtension } from '@/lib/uploads/utils/file-utils'
|
||||
import { isSimPageSource, SIM_PAGE_CONTENT_TYPE } from '@/lib/workspace-files/page-compile'
|
||||
import { useAddToChat } from '@/hooks/use-add-to-chat'
|
||||
import type { ChatContext } from '@/stores/panel'
|
||||
import { EditorContextMenu } from './editor-context-menu'
|
||||
@@ -20,6 +21,9 @@ import { PreviewLoadingFrame } from './preview-shared'
|
||||
import { useEditableFileContent } from './use-editable-file-content'
|
||||
import { useSelectionCopyBridge } from './use-selection-copy-bridge'
|
||||
|
||||
/** File ids observed rendering as Sim pages this session (see the sticky lock). */
|
||||
const KNOWN_PAGE_FILE_IDS = new Set<string>()
|
||||
|
||||
const SIM_DARK_RULES: MonacoEditorTypes.ITokenThemeRule[] = [
|
||||
{ token: 'comment', foreground: '606060', fontStyle: 'italic' },
|
||||
{ token: 'string', foreground: '3ab872' },
|
||||
@@ -584,7 +588,32 @@ export const TextEditor = memo(function TextEditor({
|
||||
|
||||
const previewType = resolvePreviewType(file.type, file.name)
|
||||
const isIframeRendered = previewType === 'html' || previewType === 'svg'
|
||||
const effectiveMode = isStreaming && isIframeRendered ? 'editor' : previewMode
|
||||
// A Sim page NEVER shows its source here — the pdf model: the rendered
|
||||
// document is the file's face on every surface, at every moment. The
|
||||
// record's internal type decides instantly (stamped at write, known before
|
||||
// content loads); content detection is the fallback for files written
|
||||
// before the stamp existed. During the very first streamed chunk even the
|
||||
// frontmatter is partial, so an html stream that has not yet contradicted
|
||||
// the page shape (empty, or an opening '---') already counts — that kills
|
||||
// the raw-header flash at stream start and the raw flips between an
|
||||
// agent's tool calls. Bespoke raw HTML keeps the old behavior.
|
||||
const trimmedContent = content.trimStart()
|
||||
const detectedSimPage =
|
||||
file.type === SIM_PAGE_CONTENT_TYPE ||
|
||||
(previewType === 'html' &&
|
||||
(isSimPageSource(content) ||
|
||||
(isStreaming && (trimmedContent === '' || trimmedContent.startsWith('-')))))
|
||||
// Sticky per file: a PATCH stream carries only the replacement snippet,
|
||||
// which is not frontmatter-shaped — without memory the lock would drop
|
||||
// mid-edit and flash raw source. Once a file is known to be a page, it
|
||||
// stays one for the session.
|
||||
if (detectedSimPage) KNOWN_PAGE_FILE_IDS.add(file.id)
|
||||
const isSimPageFile = detectedSimPage || KNOWN_PAGE_FILE_IDS.has(file.id)
|
||||
const effectiveMode = isSimPageFile
|
||||
? 'preview'
|
||||
: isStreaming && isIframeRendered
|
||||
? 'editor'
|
||||
: previewMode
|
||||
const showEditor = effectiveMode !== 'preview'
|
||||
const showPreviewPane = effectiveMode !== 'editor'
|
||||
|
||||
|
||||
@@ -47,18 +47,21 @@ import {
|
||||
SUPPORTED_IMAGE_EXTENSIONS,
|
||||
SUPPORTED_VIDEO_EXTENSIONS,
|
||||
} from '@/lib/uploads/utils/validation'
|
||||
import { SIM_PAGE_CONTENT_TYPE } from '@/lib/workspace-files/page-compile'
|
||||
import type {
|
||||
BreadcrumbItem,
|
||||
FilterTag,
|
||||
ResourceAction,
|
||||
ResourceColumn,
|
||||
ResourceRow,
|
||||
ResourceTableHandle,
|
||||
SearchConfig,
|
||||
SortConfig,
|
||||
} from '@/app/workspace/[workspaceId]/components'
|
||||
import {
|
||||
EMPTY_CELL_PLACEHOLDER,
|
||||
FILTER_SECTION_LABEL_CLASS,
|
||||
FindBar,
|
||||
OwnerAvatar,
|
||||
ownerCell,
|
||||
Resource,
|
||||
@@ -222,11 +225,13 @@ const MIME_TYPE_LABELS: Record<string, string> = {
|
||||
'text/csv': 'CSV',
|
||||
'text/plain': 'Text',
|
||||
'text/html': 'HTML',
|
||||
'text/x-sim-page': 'Page',
|
||||
'text/markdown': 'Markdown',
|
||||
}
|
||||
|
||||
const EMPTY_WORKSPACE_FILES: WorkspaceFileRecord[] = []
|
||||
const EMPTY_WORKSPACE_FILE_FOLDERS: WorkspaceFileFolderApi[] = []
|
||||
const EMPTY_FIND_MATCH_IDS: readonly string[] = Object.freeze([])
|
||||
|
||||
const hasExternalFiles = (dataTransfer: DataTransfer): boolean =>
|
||||
dataTransfer.types.includes('Files')
|
||||
@@ -780,6 +785,79 @@ export function Files() {
|
||||
})
|
||||
}, [baseRows, listRename.editingId, listRename.editValue, listRename.isSaving])
|
||||
|
||||
// Find (Cmd/Ctrl+F): the shared find bar over the visible list, stepping
|
||||
// through rows whose name matches. The list is client-side, so matching is
|
||||
// synchronous — no debounce or loading states.
|
||||
const [findOpen, setFindOpen] = useState(false)
|
||||
const [findQuery, setFindQuery] = useState('')
|
||||
const [findIndex, setFindIndex] = useState(0)
|
||||
const findInputRef = useRef<HTMLInputElement>(null)
|
||||
const tableApiRef = useRef<ResourceTableHandle | null>(null)
|
||||
|
||||
const trimmedFindQuery = findQuery.trim().toLowerCase()
|
||||
const findMatchIds = useMemo<readonly string[]>(() => {
|
||||
if (!findOpen || trimmedFindQuery.length === 0) return EMPTY_FIND_MATCH_IDS
|
||||
const ids = rows
|
||||
.filter((row) => (row.cells.name?.label ?? '').toLowerCase().includes(trimmedFindQuery))
|
||||
.map((row) => row.id)
|
||||
return ids.length > 0 ? ids : EMPTY_FIND_MATCH_IDS
|
||||
}, [rows, findOpen, trimmedFindQuery])
|
||||
const findMatchIdsRef = useRef(findMatchIds)
|
||||
findMatchIdsRef.current = findMatchIds
|
||||
const findIndexRef = useRef(findIndex)
|
||||
findIndexRef.current = findIndex
|
||||
|
||||
const goToFindMatch = useCallback((index: number) => {
|
||||
const matches = findMatchIdsRef.current
|
||||
if (matches.length === 0) return
|
||||
const wrapped = ((index % matches.length) + matches.length) % matches.length
|
||||
setFindIndex(wrapped)
|
||||
tableApiRef.current?.scrollToRow(matches[wrapped])
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* A new term resets to and reveals its first match. Keyed on the term, not
|
||||
* the match set: rows regenerate on renames, uploads and SSE refreshes, and
|
||||
* re-revealing then would yank a user who has stepped elsewhere back to
|
||||
* match one.
|
||||
*/
|
||||
useEffect(() => {
|
||||
setFindIndex(0)
|
||||
if (trimmedFindQuery.length === 0) return
|
||||
const first = findMatchIdsRef.current[0]
|
||||
if (first) tableApiRef.current?.scrollToRow(first)
|
||||
}, [trimmedFindQuery])
|
||||
|
||||
const handleFindNext = useCallback(() => {
|
||||
goToFindMatch(findIndexRef.current + 1)
|
||||
}, [goToFindMatch])
|
||||
|
||||
const handleFindPrev = useCallback(() => {
|
||||
goToFindMatch(findIndexRef.current - 1)
|
||||
}, [goToFindMatch])
|
||||
|
||||
/** Closing clears the search: term, highlights and cursor all go. */
|
||||
const handleFindClose = useCallback(() => {
|
||||
setFindOpen(false)
|
||||
setFindQuery('')
|
||||
setFindIndex(0)
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Rows for the table, with the active term tinted into matching name cells.
|
||||
* Layered over `rows` so selection, drag-drop and keyboard nav keep reading
|
||||
* the canonical list.
|
||||
*/
|
||||
const displayRows: ResourceRow[] = useMemo(() => {
|
||||
if (findMatchIds.length === 0) return rows
|
||||
const matchSet = new Set(findMatchIds)
|
||||
return rows.map((row) =>
|
||||
matchSet.has(row.id)
|
||||
? { ...row, cells: { ...row.cells, name: { ...row.cells.name, highlight: findQuery } } }
|
||||
: row
|
||||
)
|
||||
}, [rows, findMatchIds, findQuery])
|
||||
|
||||
const visibleRowIds = useMemo(() => rows.map((row) => row.id), [rows])
|
||||
|
||||
const {
|
||||
@@ -1445,6 +1523,71 @@ export function Files() {
|
||||
return () => window.removeEventListener('keydown', handleKeyDown)
|
||||
}, [handleSave])
|
||||
|
||||
const selectedRowIdsRef = useRef(selectedRowIds)
|
||||
selectedRowIdsRef.current = selectedRowIds
|
||||
const visibleRowIdsRef = useRef(visibleRowIds)
|
||||
visibleRowIdsRef.current = visibleRowIds
|
||||
const listRenameActiveRef = useRef(listRename.editingId)
|
||||
listRenameActiveRef.current = listRename.editingId
|
||||
const handleBulkDeleteRef = useRef(handleBulkDelete)
|
||||
handleBulkDeleteRef.current = handleBulkDelete
|
||||
|
||||
useEffect(() => {
|
||||
const handleListKeyDown = (e: KeyboardEvent) => {
|
||||
if (fileIdFromRouteRef.current) return
|
||||
const active = document.activeElement
|
||||
if (
|
||||
active &&
|
||||
(active.tagName === 'INPUT' ||
|
||||
active.tagName === 'TEXTAREA' ||
|
||||
(active as HTMLElement).isContentEditable)
|
||||
)
|
||||
return
|
||||
if (listRenameActiveRef.current) return
|
||||
|
||||
if ((e.key === 'Delete' || e.key === 'Backspace') && selectedRowIdsRef.current.size > 0) {
|
||||
e.preventDefault()
|
||||
handleBulkDeleteRef.current()
|
||||
return
|
||||
}
|
||||
|
||||
if (e.key === 'Escape' && selectedRowIdsRef.current.size > 0) {
|
||||
e.preventDefault()
|
||||
clearSelection()
|
||||
return
|
||||
}
|
||||
|
||||
if ((e.metaKey || e.ctrlKey) && e.key === 'a' && visibleRowIdsRef.current.length > 0) {
|
||||
e.preventDefault()
|
||||
replaceSelection(visibleRowIdsRef.current)
|
||||
}
|
||||
}
|
||||
window.addEventListener('keydown', handleListKeyDown)
|
||||
return () => window.removeEventListener('keydown', handleListKeyDown)
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Overrides the browser's Cmd/Ctrl+F with the in-list find while the list is
|
||||
* showing. Skipped when a file is open — its editor owns the shortcut there —
|
||||
* and when another surface already claimed the press.
|
||||
*/
|
||||
useEffect(() => {
|
||||
const handleFindShortcut = (e: KeyboardEvent) => {
|
||||
if (!(e.metaKey || e.ctrlKey) || e.altKey || e.shiftKey) return
|
||||
if (e.key.toLowerCase() !== 'f') return
|
||||
if (fileIdFromRouteRef.current) return
|
||||
if (e.defaultPrevented) return
|
||||
e.preventDefault()
|
||||
setFindOpen(true)
|
||||
requestAnimationFrame(() => {
|
||||
findInputRef.current?.focus()
|
||||
findInputRef.current?.select()
|
||||
})
|
||||
}
|
||||
document.addEventListener('keydown', handleFindShortcut)
|
||||
return () => document.removeEventListener('keydown', handleFindShortcut)
|
||||
}, [])
|
||||
|
||||
const handleCyclePreviewMode = useCallback(() => {
|
||||
setPreviewMode((prev) => {
|
||||
if (prev === 'editor') return 'split'
|
||||
@@ -1466,8 +1609,10 @@ export function Files() {
|
||||
const canPreview = isPreviewable(selectedFile) && !streamOnly
|
||||
// Markdown renders in the single-surface inline editor, which has no raw/split/preview modes.
|
||||
const isInlineMarkdown = isMarkdownFile(selectedFile)
|
||||
const hasSplitView = canEditText && canPreview && !isInlineMarkdown
|
||||
const showPreviewToggle = canPreview && !isInlineMarkdown
|
||||
// A Sim page is locked to its rendered view — no code view to toggle to.
|
||||
const isSimPage = selectedFile.type === SIM_PAGE_CONTENT_TYPE
|
||||
const hasSplitView = canEditText && canPreview && !isInlineMarkdown && !isSimPage
|
||||
const showPreviewToggle = canPreview && !isInlineMarkdown && !isSimPage
|
||||
|
||||
const nextModeLabel =
|
||||
previewMode === 'editor' ? 'Split' : previewMode === 'split' ? 'Preview' : 'Edit'
|
||||
@@ -2019,7 +2164,8 @@ export function Files() {
|
||||
/>
|
||||
<Resource.Table
|
||||
columns={isSearching ? SEARCH_COLUMNS : COLUMNS}
|
||||
rows={rows}
|
||||
rows={displayRows}
|
||||
apiRef={tableApiRef}
|
||||
emptyState={
|
||||
listState === 'empty' ? (
|
||||
<FilesEmptyState
|
||||
@@ -2040,6 +2186,21 @@ export function Files() {
|
||||
onRowContextMenu={handleRowContextMenu}
|
||||
overlay={
|
||||
<>
|
||||
{findOpen && (
|
||||
<FindBar
|
||||
ariaLabel='Find in files'
|
||||
query={findQuery}
|
||||
onQueryChange={setFindQuery}
|
||||
onNext={handleFindNext}
|
||||
onPrev={handleFindPrev}
|
||||
onClose={handleFindClose}
|
||||
count={findMatchIds.length}
|
||||
currentIndex={Math.min(findIndex, Math.max(0, findMatchIds.length - 1))}
|
||||
truncated={false}
|
||||
isLoading={false}
|
||||
inputRef={findInputRef}
|
||||
/>
|
||||
)}
|
||||
<ResourceActionBar
|
||||
selectedCount={selectedRowIds.size}
|
||||
onDownload={handleBulkDownload}
|
||||
|
||||
+81
@@ -213,6 +213,13 @@ describe('AgentGroup browser takeover', () => {
|
||||
})
|
||||
|
||||
expect(container.querySelector('.animate-stream-fade-in')).toBeNull()
|
||||
// Groups never auto-expand: the answered question lives inside the
|
||||
// collapsed log until the user opens it manually.
|
||||
const headerToggle = container.querySelector('button[class*="group/agent"]')
|
||||
expect(headerToggle).not.toBeNull()
|
||||
act(() => {
|
||||
headerToggle?.dispatchEvent(new MouseEvent('click', { bubbles: true }))
|
||||
})
|
||||
const resumedLog = container.querySelector('[data-state="open"]')
|
||||
const answeredQuestion = container.querySelector('[data-takeover-answer="true"]')
|
||||
expect(answeredQuestion?.textContent).toContain(reason)
|
||||
@@ -222,3 +229,77 @@ describe('AgentGroup browser takeover', () => {
|
||||
act(() => root.unmount())
|
||||
})
|
||||
})
|
||||
|
||||
describe('AgentGroup nested status line', () => {
|
||||
let container: HTMLDivElement
|
||||
let root: Root
|
||||
|
||||
beforeEach(() => {
|
||||
container = document.createElement('div')
|
||||
document.body.appendChild(container)
|
||||
root = createRoot(container)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
act(() => root.unmount())
|
||||
container.remove()
|
||||
})
|
||||
|
||||
const namedTool = (
|
||||
displayTitle: string,
|
||||
status: ToolCallStatus,
|
||||
startedAt?: number
|
||||
): AgentGroupItem => ({
|
||||
type: 'tool',
|
||||
data: {
|
||||
id: `${displayTitle}-${startedAt ?? 0}`,
|
||||
toolName: 'grep',
|
||||
displayTitle,
|
||||
status,
|
||||
startedAt,
|
||||
},
|
||||
})
|
||||
|
||||
const render = (items: AgentGroupItem[]) => {
|
||||
act(() => {
|
||||
root.render(
|
||||
createElement(AgentGroup, {
|
||||
agentName: 'workflow',
|
||||
agentLabel: 'Workflow Agent',
|
||||
items,
|
||||
isStreaming: true,
|
||||
isLaneOpen: true,
|
||||
})
|
||||
)
|
||||
})
|
||||
return container.textContent ?? ''
|
||||
}
|
||||
|
||||
it("shows a nested agent's running tool instead of the parent's finished one", () => {
|
||||
const header = render([
|
||||
namedTool('Reading workflow', 'success' as ToolCallStatus, 1),
|
||||
group([namedTool('Deploying Invoice Sync as API', 'executing' as ToolCallStatus, 2)]),
|
||||
])
|
||||
expect(header).toContain('Workflow Agent — Deploying Invoice Sync as API')
|
||||
})
|
||||
|
||||
it('counts running tools across depths with the + n suffix', () => {
|
||||
const header = render([
|
||||
namedTool('Reading workflow', 'executing' as ToolCallStatus, 1),
|
||||
group([
|
||||
namedTool('Deploying Invoice Sync as API', 'executing' as ToolCallStatus, 3),
|
||||
namedTool('Checking deployment status', 'executing' as ToolCallStatus, 2),
|
||||
]),
|
||||
])
|
||||
// Latest start wins; the other two running become the overflow count.
|
||||
expect(header).toContain('Deploying Invoice Sync as API + 2')
|
||||
})
|
||||
|
||||
it('falls back to the last tool at any depth when nothing is running', () => {
|
||||
const header = render([
|
||||
namedTool('Reading workflow', 'success' as ToolCallStatus, 1),
|
||||
group([namedTool('Deploying Invoice Sync as API', 'success' as ToolCallStatus, 2)]),
|
||||
])
|
||||
expect(header).toContain('Workflow Agent — Deploying Invoice Sync as API')
|
||||
})
|
||||
})
|
||||
|
||||
+62
-21
@@ -1,10 +1,10 @@
|
||||
'use client'
|
||||
|
||||
import { useEffect, useLayoutEffect, useRef, useState } from 'react'
|
||||
import { useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { ChevronDown, cn, Expandable, ExpandableContent } from '@sim/emcn'
|
||||
import { ShimmerText } from '@/components/ui'
|
||||
import { isBrowserAgentAvailable } from '@/lib/browser-agent/transport'
|
||||
import { BrowserRequestTakeover } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { RETIRED_BROWSER_REQUEST_TAKEOVER_ID } from '@/lib/copilot/tools/retired-tools'
|
||||
import { useSmoothText } from '@/hooks/use-smooth-text'
|
||||
import { type ToolCallData, ToolCallStatus } from '../../../../types'
|
||||
import { getAgentIcon, isToolDone } from '../../utils'
|
||||
@@ -43,6 +43,27 @@ interface AgentGroupProps {
|
||||
isLaneOpen?: boolean
|
||||
}
|
||||
|
||||
function toolStatusTitle(tool: ToolCallData): string {
|
||||
return tool.displayTitle || String(tool.toolName ?? '')
|
||||
}
|
||||
|
||||
/**
|
||||
* Every tool in a group, in stream order, including those run by nested
|
||||
* agents. A parent's status line speaks for the whole subtree it delegated,
|
||||
* so a grandchild's work is what surfaces while the parent itself waits.
|
||||
*/
|
||||
function collectGroupTools(items: AgentGroupItem[]): ToolCallData[] {
|
||||
const tools: ToolCallData[] = []
|
||||
const walk = (list: AgentGroupItem[]) => {
|
||||
for (const item of list) {
|
||||
if (item.type === 'tool') tools.push(item.data)
|
||||
else if (item.type === 'agent_group') walk(item.group.items)
|
||||
}
|
||||
}
|
||||
walk(items)
|
||||
return tools
|
||||
}
|
||||
|
||||
/** True when any row in this group (or a nested one) is waiting on a permission decision. */
|
||||
function hasAwaitingApproval(items: AgentGroupItem[]): boolean {
|
||||
return items.some((item) => {
|
||||
@@ -63,7 +84,7 @@ function getActiveBrowserTakeover(items: AgentGroupItem[]): ActiveBrowserTakeove
|
||||
const item = items[index]
|
||||
if (item.type !== 'tool') continue
|
||||
if (
|
||||
item.data.toolName === BrowserRequestTakeover.id &&
|
||||
item.data.toolName === RETIRED_BROWSER_REQUEST_TAKEOVER_ID &&
|
||||
item.data.status === ToolCallStatus.executing
|
||||
) {
|
||||
const reason = item.data.params?.reason
|
||||
@@ -114,6 +135,30 @@ export function AgentGroup({
|
||||
isLaneOpen = false,
|
||||
}: AgentGroupProps) {
|
||||
const AgentIcon = getAgentIcon(agentName)
|
||||
const isMainAgent = agentName === 'mothership'
|
||||
// Collapsed status line: the latest tool call, always in its RUNNING
|
||||
// phrasing — it never flips to the completed rewrite (that lives in the
|
||||
// expanded log). Work delegated further down bubbles up, so a group whose
|
||||
// own turn is idle still narrates what its nested agent is doing rather
|
||||
// than freezing on its last own tool. With several tools running at any
|
||||
// depth, the most recently started wins and the rest become "+ n"; between
|
||||
// rounds the last tool's title stays frozen; a closed lane shows the bare
|
||||
// name.
|
||||
const status = useMemo(() => {
|
||||
if (isMainAgent || !isLaneOpen) return undefined
|
||||
const tools = collectGroupTools(items)
|
||||
const running = tools.filter((tool) => tool.status === ToolCallStatus.executing)
|
||||
if (running.length > 0) {
|
||||
const latest = running.reduce((newest, tool) =>
|
||||
(tool.startedAt ?? 0) >= (newest.startedAt ?? 0) ? tool : newest
|
||||
)
|
||||
const title = toolStatusTitle(latest)
|
||||
return running.length > 1 ? `${title} + ${running.length - 1}` : title
|
||||
}
|
||||
const last = tools.at(-1)
|
||||
return last ? toolStatusTitle(last) : undefined
|
||||
}, [isLaneOpen, isMainAgent, items])
|
||||
const headerText = status ? `${agentLabel} — ${status}` : agentLabel
|
||||
const hasItems = items.length > 0
|
||||
const resolved = isAgentGroupResolved(items)
|
||||
const browserAgentAvailable = isBrowserAgentAvailable()
|
||||
@@ -123,17 +168,13 @@ export function AgentGroup({
|
||||
const isWorking =
|
||||
!activeBrowserTakeover && ((isDelegating && !resolved) || (isStreaming && isLaneOpen))
|
||||
|
||||
// Expand while the turn is live and any of: the lane is open (the subagent is
|
||||
// actively running), this is the current/latest section, or there is unresolved
|
||||
// work. A finished group stays open until the NEXT section starts (it is no
|
||||
// longer the latest), instead of collapsing the instant its own work resolves.
|
||||
// Keying "still running" off the lane-open signal (not `resolved` alone) avoids
|
||||
// a collapse/reopen flicker on parallel siblings: a subagent's tools all
|
||||
// momentarily read "done" in the gap between its last search and its `respond`
|
||||
// ("Gathering thoughts") tool, transiently flipping `resolved` true; the open
|
||||
// lane bridges that gap so the row never collapses mid-run. The turn ending
|
||||
// (isStreaming false) collapses everything; a manual toggle pins the choice.
|
||||
const autoExpanded = isStreaming && (isCurrentSection || isLaneOpen || !resolved)
|
||||
// SUBAGENT groups never auto-expand: the collapsed row IS the live view —
|
||||
// label plus latest running tool title. Expanding is a deliberate user
|
||||
// action; only a pending permission prompt or a browser hand-back forces
|
||||
// one open. The MAIN lane ("Sim") is not a delegation card: its narration
|
||||
// and tool calls are the turn itself, so it keeps the original live-expand
|
||||
// behavior (open while streaming/current, settles when superseded).
|
||||
const autoExpanded = isMainAgent && isStreaming && (isCurrentSection || isLaneOpen || !resolved)
|
||||
const [manualExpanded, setManualExpanded] = useState<boolean | null>(null)
|
||||
const [expandedTakeoverId, setExpandedTakeoverId] = useState<string | null>(null)
|
||||
// An outstanding permission prompt overrides a manual collapse: the turn
|
||||
@@ -160,32 +201,32 @@ export function AgentGroup({
|
||||
<button
|
||||
type='button'
|
||||
onClick={toggleExpanded}
|
||||
className='group/agent flex cursor-pointer items-center gap-2'
|
||||
className='group/agent flex w-full min-w-0 cursor-pointer items-center gap-2 text-left'
|
||||
>
|
||||
<div className='flex size-[16px] flex-shrink-0 items-center justify-center'>
|
||||
<AgentIcon className='size-[16px] text-[var(--text-icon)]' />
|
||||
</div>
|
||||
{isWorking ? (
|
||||
<ShimmerText className='text-sm'>{agentLabel}</ShimmerText>
|
||||
<ShimmerText className='min-w-0 truncate text-sm'>{headerText}</ShimmerText>
|
||||
) : (
|
||||
<span className='text-[var(--text-body)] text-sm'>{agentLabel}</span>
|
||||
<span className='min-w-0 truncate text-[var(--text-body)] text-sm'>{headerText}</span>
|
||||
)}
|
||||
<ChevronDown
|
||||
className={cn(
|
||||
'size-[14px] text-[var(--text-icon)] opacity-0 transition-[transform,opacity] duration-150 group-hover/agent:opacity-100 group-focus-visible/agent:opacity-100',
|
||||
'size-[14px] flex-shrink-0 text-[var(--text-icon)] opacity-0 transition-[transform,opacity] duration-150 group-hover/agent:opacity-100 group-focus-visible/agent:opacity-100',
|
||||
!expanded && '-rotate-90'
|
||||
)}
|
||||
/>
|
||||
</button>
|
||||
) : (
|
||||
<div className='flex items-center gap-2'>
|
||||
<div className='flex min-w-0 items-center gap-2'>
|
||||
<div className='flex size-[16px] flex-shrink-0 items-center justify-center'>
|
||||
<AgentIcon className='size-[16px] text-[var(--text-icon)]' />
|
||||
</div>
|
||||
{isWorking ? (
|
||||
<ShimmerText className='text-sm'>{agentLabel}</ShimmerText>
|
||||
<ShimmerText className='min-w-0 truncate text-sm'>{headerText}</ShimmerText>
|
||||
) : (
|
||||
<span className='text-[var(--text-body)] text-sm'>{agentLabel}</span>
|
||||
<span className='min-w-0 truncate text-[var(--text-body)] text-sm'>{headerText}</span>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
+9
-7
@@ -2,14 +2,14 @@ import { useEffect, useMemo, useState } from 'react'
|
||||
import { isPlainRecord } from '@sim/utils/object'
|
||||
import { ShimmerText } from '@/components/ui'
|
||||
import {
|
||||
BrowserRequestTakeover,
|
||||
CallIntegrationTool,
|
||||
PrepareFileEdit,
|
||||
Read as ReadTool,
|
||||
Terminal as TerminalTool,
|
||||
Wait as WaitTool,
|
||||
WorkspaceFile,
|
||||
} from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { getReadTargetBlock } from '@/lib/copilot/tools/client/read-block'
|
||||
import { RETIRED_BROWSER_REQUEST_TAKEOVER_ID } from '@/lib/copilot/tools/retired-tools'
|
||||
import { extractStreamingStringArgument } from '@/lib/copilot/tools/streaming-args'
|
||||
import { getToolStatusDisplayTitle, getWaitCountdownTitle } from '@/lib/copilot/tools/tool-display'
|
||||
import { BrandIcon } from '@/blocks/brand-icon'
|
||||
@@ -138,7 +138,7 @@ export function ToolCallItem({
|
||||
}, [toolName, params, streamingArgs])
|
||||
|
||||
const liveWorkspaceFileTitle = useMemo(() => {
|
||||
if (toolName !== WorkspaceFile.id || !streamingArgs) return null
|
||||
if (toolName !== PrepareFileEdit.id || !streamingArgs) return null
|
||||
const titleMatch = streamingArgs.match(/"title"\s*:\s*"([^"]+)"/)
|
||||
if (!titleMatch?.[1]) return null
|
||||
const opMatch = streamingArgs.match(/"operation"\s*:\s*"(\w+)"/)
|
||||
@@ -168,7 +168,7 @@ export function ToolCallItem({
|
||||
|
||||
const displayState = resolveToolDisplayState(status)
|
||||
const isExecuting = displayState === 'spinner'
|
||||
const isBrowserTakeover = toolName === BrowserRequestTakeover.id
|
||||
const isBrowserTakeover = toolName === RETIRED_BROWSER_REQUEST_TAKEOVER_ID
|
||||
|
||||
const isCountingDown = toolName === WaitTool.id && isExecuting
|
||||
const elapsedMs = useElapsedMs(isCountingDown, startedAt)
|
||||
@@ -236,14 +236,16 @@ export function ToolCallItem({
|
||||
}
|
||||
|
||||
return (
|
||||
<div className='flex items-center gap-[6px] pl-6'>
|
||||
<div className='flex min-w-0 items-center gap-[6px] pl-6'>
|
||||
{BlockIcon && <BrandIcon icon={BlockIcon} className='size-[14px] flex-shrink-0' />}
|
||||
{isExecuting ? (
|
||||
<ShimmerText className='text-[13px] [--shimmer-rest:var(--text-secondary)]'>
|
||||
<ShimmerText className='min-w-0 truncate text-[13px] leading-[18px] [--shimmer-rest:var(--text-secondary)]'>
|
||||
{title}
|
||||
</ShimmerText>
|
||||
) : (
|
||||
<span className='text-[13px] text-[var(--text-secondary)]'>{title}</span>
|
||||
<span className='min-w-0 truncate text-[13px] text-[var(--text-secondary)] leading-[18px]'>
|
||||
{title}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
|
||||
+156
@@ -1218,3 +1218,159 @@ describe('parser properties', () => {
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('flattened options payload recovery', () => {
|
||||
// Observed in the wild: no <options> wrapper, and the LAST entry lost its
|
||||
// braces so its title sits on the numeric key with the description hoisted.
|
||||
const flattened =
|
||||
'options {"1": {"title": "Test files and a second turn in the same DM thread", "description": "watermark + file-delta live check"}, "2": {"title": "Put /chat/the-elder on Brain with a password", "description": "finish chat cutover"}, "3": "Add thinking-status keepalive for long Brain runs", "description": "refresh shimmer past 2 minutes"}'
|
||||
|
||||
it('renders it as an options card instead of raw JSON', () => {
|
||||
const { segments } = parseSpecialTags(flattened, false)
|
||||
const options = segments.find((segment) => segment.type === 'options')
|
||||
expect(options).toBeDefined()
|
||||
expect(Object.keys((options as { data: Record<string, unknown> }).data)).toEqual([
|
||||
'1',
|
||||
'2',
|
||||
'3',
|
||||
])
|
||||
})
|
||||
|
||||
it('rebuilds the flattened entry from the hoisted description', () => {
|
||||
const { segments } = parseSpecialTags(flattened, false)
|
||||
const data = (segments.find((s) => s.type === 'options') as { data: Record<string, unknown> })
|
||||
.data
|
||||
expect(data['3']).toEqual({
|
||||
title: 'Add thinking-status keepalive for long Brain runs',
|
||||
description: 'refresh shimmer past 2 minutes',
|
||||
})
|
||||
})
|
||||
|
||||
it('drops the bare "options" label rather than leaving it as prose', () => {
|
||||
const { segments } = parseSpecialTags(flattened, false)
|
||||
expect(segments.some((segment) => segment.type === 'text')).toBe(false)
|
||||
})
|
||||
|
||||
it('repairs the same corruption inside a well-formed tag', () => {
|
||||
const tagged =
|
||||
'<options>{"1": {"title": "A", "description": "a"}, "2": "B", "description": "b"}</options>'
|
||||
const { segments } = parseSpecialTags(tagged, false)
|
||||
const data = (segments.find((s) => s.type === 'options') as { data: Record<string, unknown> })
|
||||
.data
|
||||
expect(data['2']).toEqual({ title: 'B', description: 'b' })
|
||||
})
|
||||
|
||||
it('leaves prose JSON alone', () => {
|
||||
const prose = 'Here is the config: {"name": "x", "description": "y"}'
|
||||
const { segments } = parseSpecialTags(prose, false)
|
||||
expect(segments.some((segment) => segment.type === 'options')).toBe(false)
|
||||
})
|
||||
|
||||
it('does not fire mid-stream', () => {
|
||||
expect(parseSpecialTags(flattened, true).segments.some((s) => s.type === 'options')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('bare options with a capitalized label', () => {
|
||||
// Well-formed payload, but no wrapper and a "Options:" label instead of the
|
||||
// lowercase bare word — the second shape seen in the wild.
|
||||
const labeled =
|
||||
'Options: {"1": {"title": "Live-test ASK top-level, ASK thread, and a mention in another channel", "description": "confirm the new accept rule in Slack"}, "2": {"title": "Test files and a second turn in the same DM thread", "description": "watermark + file-delta live check"}}'
|
||||
|
||||
it('renders it as an options card', () => {
|
||||
const { segments } = parseSpecialTags(labeled, false)
|
||||
const options = segments.find((segment) => segment.type === 'options')
|
||||
expect(options).toBeDefined()
|
||||
expect(Object.keys((options as { data: Record<string, unknown> }).data)).toEqual(['1', '2'])
|
||||
})
|
||||
|
||||
it('leaves no stray "Options:" prose above the card', () => {
|
||||
const { segments } = parseSpecialTags(labeled, false)
|
||||
expect(segments.some((segment) => segment.type === 'text')).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps real prose that precedes the payload', () => {
|
||||
const withProse = `Here is what I suggest.\n\n${labeled}`
|
||||
const { segments } = parseSpecialTags(withProse, false)
|
||||
const text = segments.find((segment) => segment.type === 'text') as { content: string }
|
||||
expect(text.content).toBe('Here is what I suggest.')
|
||||
expect(segments.some((segment) => segment.type === 'options')).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('bare options JSON with no label at all', () => {
|
||||
const naked =
|
||||
'{"1": {"title": "Live-test ASK top-level, ASK thread, and a mention in another channel", "description": "confirm the new accept rule in Slack"}, "2": {"title": "Test files and a second turn in the same DM thread", "description": "watermark + file-delta live check"}}'
|
||||
|
||||
it('renders the payload alone as an options card', () => {
|
||||
const { segments } = parseSpecialTags(naked, false)
|
||||
const options = segments.find((segment) => segment.type === 'options')
|
||||
expect(options).toBeDefined()
|
||||
expect(Object.keys((options as { data: Record<string, unknown> }).data)).toEqual(['1', '2'])
|
||||
expect(segments.some((segment) => segment.type === 'text')).toBe(false)
|
||||
})
|
||||
|
||||
it('renders it after prose with no label between them', () => {
|
||||
const { segments } = parseSpecialTags(`Two ways to go from here.\n\n${naked}`, false)
|
||||
const text = segments.find((segment) => segment.type === 'text') as { content: string }
|
||||
expect(text.content).toBe('Two ways to go from here.')
|
||||
expect(segments.some((segment) => segment.type === 'options')).toBe(true)
|
||||
})
|
||||
|
||||
it('recovers a flattened payload with no label either', () => {
|
||||
const flattenedNaked = '{"1": {"title": "A", "description": "a"}, "2": "B", "description": "b"}'
|
||||
const { segments } = parseSpecialTags(flattenedNaked, false)
|
||||
const data = (segments.find((s) => s.type === 'options') as { data: Record<string, unknown> })
|
||||
.data
|
||||
expect(data['2']).toEqual({ title: 'B', description: 'b' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('bare question payload recovery', () => {
|
||||
const bare =
|
||||
'{"type": "single_select", "prompt": "Which channel should the bot post to?", "options": [{"id": "a", "label": "#general"}, {"id": "b", "label": "#alerts"}]}'
|
||||
|
||||
it('renders an unwrapped question payload as a question card', () => {
|
||||
const { segments } = parseSpecialTags(bare, false)
|
||||
const question = segments.find((segment) => segment.type === 'question')
|
||||
expect(question).toBeDefined()
|
||||
expect((question as { data: Array<{ prompt: string }> }).data[0].prompt).toBe(
|
||||
'Which channel should the bot post to?'
|
||||
)
|
||||
expect(segments.some((segment) => segment.type === 'text')).toBe(false)
|
||||
})
|
||||
|
||||
it('accepts an array payload and strips a bare label', () => {
|
||||
const { segments } = parseSpecialTags(`Question: [${bare}]`, false)
|
||||
expect(segments.some((segment) => segment.type === 'question')).toBe(true)
|
||||
expect(segments.some((segment) => segment.type === 'text')).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps prose that precedes the payload', () => {
|
||||
const { segments } = parseSpecialTags(`I need one detail.\n\n${bare}`, false)
|
||||
const text = segments.find((segment) => segment.type === 'text') as { content: string }
|
||||
expect(text.content).toBe('I need one detail.')
|
||||
})
|
||||
|
||||
it('does not fire mid-stream', () => {
|
||||
expect(parseSpecialTags(bare, true).segments.some((s) => s.type === 'question')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('ordinary JSON in prose is never turned into a card', () => {
|
||||
const prose = [
|
||||
'Here is the config: {"name": "elder", "description": "the bot", "enabled": true}',
|
||||
'The API returned {"1": "ok", "2": "ok"}',
|
||||
'Response shape: {"type": "object", "prompt": "n/a", "options": []}',
|
||||
'Use {"type": "single_select"} as the discriminator.',
|
||||
'Rows: [{"id": "1", "label": "one"}, {"id": "2", "label": "two"}]',
|
||||
'Payload: {"data": {"title": "x", "description": "y"}}',
|
||||
'Env: {"0": {"title": "a", "description": "b"}}',
|
||||
]
|
||||
|
||||
it.each(prose)('leaves %s as text', (content) => {
|
||||
const { segments } = parseSpecialTags(content, false)
|
||||
expect(segments.some((s) => s.type === 'options' || s.type === 'question')).toBe(false)
|
||||
expect(segments.some((s) => s.type === 'text')).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
+45
@@ -1413,3 +1413,48 @@ describe('parseSpecialTags sim_key placeholder', () => {
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('recoverTrailingBareOptions', () => {
|
||||
const bareOptions =
|
||||
'{"1": {"title": "Fix the tracker", "description": "debug"}, "2": {"title": "Inspect the miss", "description": "look"}}'
|
||||
|
||||
it('renders a trailing bare-JSON options payload as an options card', () => {
|
||||
const { segments } = parseSpecialTags(`Here they are.\n${bareOptions}`, false)
|
||||
const last = segments[segments.length - 1]
|
||||
expect(last.type).toBe('options')
|
||||
if (last.type === 'options') {
|
||||
expect(last.data['1']?.title).toBe('Fix the tracker')
|
||||
}
|
||||
expect(segments[0]).toEqual({ type: 'text', content: 'Here they are.' })
|
||||
})
|
||||
|
||||
it('never recovers mid-stream — a partial JSON tail must not flicker into a card', () => {
|
||||
const { segments } = parseSpecialTags(`Here they are.\n${bareOptions}`, true)
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
})
|
||||
|
||||
it('leaves ordinary JSON prose alone', () => {
|
||||
const { segments } = parseSpecialTags('The config is {"retries": 3, "mode": "fast"}', false)
|
||||
expect(segments.every((segment) => segment.type === 'text')).toBe(true)
|
||||
})
|
||||
|
||||
it('does not double-render when a real options tag already parsed', () => {
|
||||
const { segments } = parseSpecialTags(`Pick one <options>${bareOptions}</options>`, false)
|
||||
expect(segments.filter((segment) => segment.type === 'options')).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('recovers an options payload wrapped in the singular <option> near-miss tag', () => {
|
||||
const body =
|
||||
'{"1": {"title": "Demo wait — sleep until an agent finishes", "description": "wait_agents demo"}, "2": {"title": "Demo steer — redirect a running agent mid-task", "description": "steer_agent demo"}, "3": {"title": "Demo stop — interrupt a running agent", "description": "interrupt_agent demo"}}'
|
||||
const { segments } = parseSpecialTags(
|
||||
`Next up is usually **wait**, **steer**, or **stop**.\n\n<option>${body}</option>`,
|
||||
false
|
||||
)
|
||||
const last = segments[segments.length - 1]
|
||||
expect(last.type).toBe('options')
|
||||
if (last.type === 'options') {
|
||||
expect(Object.keys(last.data)).toEqual(['1', '2', '3'])
|
||||
expect(last.data['1']?.title).toBe('Demo wait — sleep until an agent finishes')
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
+196
-6
@@ -370,6 +370,56 @@ function isOptionsItemData(value: unknown): value is OptionsItemData {
|
||||
return typeof value.title === 'string' && typeof value.description === 'string'
|
||||
}
|
||||
|
||||
/**
|
||||
* Repairs the one malformed options payload seen in the wild: the LAST entry
|
||||
* loses its object braces, so its title lands directly on the numeric key and
|
||||
* its description is hoisted to a sibling of the entries —
|
||||
*
|
||||
* {"1": {…}, "2": {…}, "3": "Third title", "description": "Third desc"}
|
||||
*
|
||||
* — which fails both the per-item shape check and the numeric-key gate, so the
|
||||
* whole card was dropped and the raw JSON rendered as prose.
|
||||
*
|
||||
* Deliberately narrow: a bare string is only accepted under a numeric key, and
|
||||
* only a single stray `description` is absorbed, attaching to the last numeric
|
||||
* entry that lacks one. Anything else is returned untouched so a JSON object
|
||||
* quoted in prose still cannot masquerade as an options card. Returns the input
|
||||
* unchanged when there is nothing to repair.
|
||||
*/
|
||||
/** Whether keys are exactly "1".."N" in order, the shape the options contract emits. */
|
||||
function isSequentialOptionKeys(keys: string[]): boolean {
|
||||
return keys.length > 0 && keys.every((key, index) => key === String(index + 1))
|
||||
}
|
||||
|
||||
function repairFlattenedOptionEntry(value: unknown): unknown {
|
||||
if (!isRecordLike(value) || Array.isArray(value)) return value
|
||||
|
||||
const entries = Object.entries(value)
|
||||
const numericKeys = entries.filter(([key]) => /^\d+$/.test(key))
|
||||
if (numericKeys.length === 0) return value
|
||||
|
||||
// The hoisted description IS the signature of this corruption. Without one,
|
||||
// a numeric-keyed map of plain strings is just data ({"1": "ok", "2": "ok"})
|
||||
// and must stay prose.
|
||||
const strayKeys = entries.filter(([key]) => !/^\d+$/.test(key))
|
||||
if (strayKeys.length !== 1) return value
|
||||
const [strayKey, strayValue] = strayKeys[0]
|
||||
if (strayKey !== 'description' || typeof strayValue !== 'string') return value
|
||||
|
||||
// Exactly one entry lost its braces, and it is the last one — every earlier
|
||||
// entry must already be a well-formed option.
|
||||
const flattenedCount = numericKeys.filter(([, item]) => typeof item === 'string').length
|
||||
if (flattenedCount !== 1) return value
|
||||
const [lastKey, lastItem] = numericKeys[numericKeys.length - 1]
|
||||
if (typeof lastItem !== 'string') return value
|
||||
|
||||
const repaired: Record<string, unknown> = {}
|
||||
for (const [key, item] of numericKeys) {
|
||||
repaired[key] = key === lastKey ? { title: lastItem, description: strayValue } : item
|
||||
}
|
||||
return repaired
|
||||
}
|
||||
|
||||
/**
|
||||
* Arrays are accepted alongside keyed objects: an agent that emits
|
||||
* `<options>[{title,description},…]</options>` still renders, with the array
|
||||
@@ -592,11 +642,14 @@ export function parseQuestionTagBody(body: string): QuestionTagData | null {
|
||||
|
||||
export function parseJsonTagBody<T>(
|
||||
body: string,
|
||||
isExpectedShape: (value: unknown) => value is T
|
||||
isExpectedShape: (value: unknown) => value is T,
|
||||
/** Optional normalizer applied before the shape check, for known model typos. */
|
||||
repair?: (value: unknown) => unknown
|
||||
): T | null {
|
||||
try {
|
||||
const parsed = JSON.parse(body) as unknown
|
||||
return isExpectedShape(parsed) ? parsed : null
|
||||
const candidate = repair ? repair(parsed) : parsed
|
||||
return isExpectedShape(candidate) ? candidate : null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
@@ -667,7 +720,7 @@ function parseSpecialTagData(
|
||||
}
|
||||
|
||||
if (tagName === 'options') {
|
||||
const data = parseJsonTagBody(body, isOptionsTagData)
|
||||
const data = parseJsonTagBody(body, isOptionsTagData, repairFlattenedOptionEntry)
|
||||
return data ? { type: 'options', data } : null
|
||||
}
|
||||
|
||||
@@ -1455,8 +1508,139 @@ export function parseSpecialTags(content: string, isStreaming: boolean): ParsedS
|
||||
segments.push({ type: 'text', content })
|
||||
}
|
||||
|
||||
if (!isStreaming) {
|
||||
recoverTrailingBareOptions(segments)
|
||||
recoverTrailingBareQuestion(segments)
|
||||
}
|
||||
|
||||
return { segments, hasPendingTag }
|
||||
}
|
||||
/**
|
||||
* Recovers a trailing bare-JSON options payload the model emitted WITHOUT the
|
||||
* `<options>` wrapper (observed when an automation prompt asks the model to
|
||||
* "(re)send suggested actions" and it answers with the JSON as content). The
|
||||
* shape check is strict — a non-empty object whose every value is
|
||||
* { title, description } with numeric-string keys — so ordinary JSON in prose
|
||||
* cannot false-positive. Only a message's FINAL text segment is considered,
|
||||
* mirroring the tag contract (options go last), and only when no options tag
|
||||
* already parsed. Never applied mid-stream: a partial JSON tail must not
|
||||
* flicker between prose and a card.
|
||||
*/
|
||||
const NEAR_MISS_OPTIONS_WRAPPER = /<option>\s*(\{[\s\S]*\})\s*<\/option>\s*$/
|
||||
|
||||
/**
|
||||
* Locates a trailing bare JSON payload: the earliest opening delimiter from
|
||||
* which the whole remainder parses. Nested objects mean the FIRST `{` is not
|
||||
* necessarily the payload's start, so positions are probed left to right and
|
||||
* bounded so a brace-heavy prose block cannot become a hot loop.
|
||||
*/
|
||||
function findTrailingJsonPayload(
|
||||
text: string,
|
||||
openers: string[]
|
||||
): { start: number; body: string } | null {
|
||||
const trimmed = text.trimEnd()
|
||||
if (!openers.some((opener) => trimmed.endsWith(opener === '[' ? ']' : '}'))) return null
|
||||
let probe = -1
|
||||
for (const opener of openers) {
|
||||
const index = text.indexOf(opener)
|
||||
if (index !== -1 && (probe === -1 || index < probe)) probe = index
|
||||
}
|
||||
for (let attempts = 0; probe !== -1 && attempts < 20; attempts++) {
|
||||
const body = text.slice(probe).trim()
|
||||
try {
|
||||
JSON.parse(body)
|
||||
return { start: probe, body }
|
||||
} catch {
|
||||
let next = -1
|
||||
for (const opener of openers) {
|
||||
const index = text.indexOf(opener, probe + 1)
|
||||
if (index !== -1 && (next === -1 || index < next)) next = index
|
||||
}
|
||||
probe = next
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
/** Drops a bare `question` / `<question>` label sitting just before a payload. */
|
||||
const BARE_QUESTION_LABEL = /(^|\n)\s*<?questions?>?\s*:?\s*$/i
|
||||
|
||||
/**
|
||||
* Recovers a trailing `<question>` payload the model emitted WITHOUT the
|
||||
* wrapper, mirroring {@link recoverTrailingBareOptions}.
|
||||
*
|
||||
* Safe to attempt on bare JSON because the question shape is far more
|
||||
* distinctive than the options one: `parseQuestionTagBody` requires a `type` of
|
||||
* exactly `single_select` or `multi_select`, a non-empty `prompt`, and a
|
||||
* non-empty `options` array of `{id, label}`. Ordinary JSON in prose — a config
|
||||
* blob, an API response, a code sample — does not carry that combination, so
|
||||
* the strict validator is the whole gate. Accepts an array or a single object,
|
||||
* exactly as the tagged path does.
|
||||
*/
|
||||
function recoverTrailingBareQuestion(segments: ContentSegment[]): void {
|
||||
const last = segments[segments.length - 1]
|
||||
if (!last || last.type !== 'text') return
|
||||
if (segments.some((segment) => segment.type === 'question')) return
|
||||
const payload = findTrailingJsonPayload(last.content, ['{', '['])
|
||||
if (!payload) return
|
||||
const data = parseQuestionTagBody(payload.body)
|
||||
if (!data) return
|
||||
const prefix = last.content
|
||||
.slice(0, payload.start)
|
||||
.replace(/\s+$/, '')
|
||||
.replace(BARE_QUESTION_LABEL, '$1')
|
||||
.replace(/\s+$/, '')
|
||||
segments.pop()
|
||||
if (prefix) segments.push({ type: 'text', content: prefix })
|
||||
segments.push({ type: 'question', data })
|
||||
}
|
||||
|
||||
function recoverTrailingBareOptions(segments: ContentSegment[]): void {
|
||||
const last = segments[segments.length - 1]
|
||||
if (!last || last.type !== 'text') return
|
||||
if (segments.some((segment) => segment.type === 'options')) return
|
||||
let text = last.content
|
||||
// A near-miss wrapper — the singular `<option>` tag observed in the wild —
|
||||
// is neither a parseable tag nor bare JSON (the trailing `</option>` fails
|
||||
// the brace gate below). Unwrap it and let the strict shape check decide.
|
||||
const nearMiss = NEAR_MISS_OPTIONS_WRAPPER.exec(text)
|
||||
if (nearMiss) {
|
||||
text = `${text.slice(0, nearMiss.index)}${nearMiss[1]}`
|
||||
}
|
||||
if (!text.trimEnd().endsWith('}')) return
|
||||
// The payload nests objects, so the START brace is the first one from which
|
||||
// the remainder parses — probe brace positions left to right (bounded).
|
||||
let start = -1
|
||||
let parsed: unknown
|
||||
let probe = text.indexOf('{')
|
||||
for (let attempts = 0; probe !== -1 && attempts < 20; attempts++) {
|
||||
try {
|
||||
parsed = JSON.parse(text.slice(probe).trim())
|
||||
start = probe
|
||||
break
|
||||
} catch {
|
||||
probe = text.indexOf('{', probe + 1)
|
||||
}
|
||||
}
|
||||
if (start === -1) return
|
||||
const repaired = repairFlattenedOptionEntry(parsed)
|
||||
if (!isOptionsTagData(repaired) || Object.keys(repaired as object).length === 0) return
|
||||
// The contract numbers options from 1 upward. Demanding the exact run — not
|
||||
// merely "every key is numeric" — keeps a numeric-keyed map that happens to
|
||||
// hold {title, description} values (e.g. {"0": {…}}) from becoming a card.
|
||||
if (!isSequentialOptionKeys(Object.keys(repaired as object))) return
|
||||
// A bare `options` / `<options>` label immediately before the payload is the
|
||||
// wrapper the model meant to emit, not prose — drop it rather than leaving a
|
||||
// stray word above the card.
|
||||
const prefix = text
|
||||
.slice(0, start)
|
||||
.replace(/\s+$/, '')
|
||||
.replace(/(^|\n)\s*<?options>?\s*:?\s*$/i, '$1')
|
||||
.replace(/\s+$/, '')
|
||||
segments.pop()
|
||||
if (prefix) segments.push({ type: 'text', content: prefix })
|
||||
segments.push({ type: 'options', data: repaired })
|
||||
}
|
||||
|
||||
interface SpecialTagsProps {
|
||||
segment: Exclude<ContentSegment, { type: 'text' }>
|
||||
@@ -2708,10 +2892,16 @@ export function CredentialDisplay({
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* The message is the whole user-facing story. The raw `code` is an internal
|
||||
* identifier ("async_resume_aborted") that means nothing to a reader and made
|
||||
* every error line end in parenthesized jargon; it stays in the tag payload for
|
||||
* logs and support, just not on screen.
|
||||
*/
|
||||
function MothershipErrorDisplay({ data }: { data: MothershipErrorTagData }) {
|
||||
const detail = data.code ? `${data.message} (${data.code})` : data.message
|
||||
|
||||
return <p className='text-[13px] text-[var(--text-secondary)] italic leading-[20px]'>{detail}</p>
|
||||
return (
|
||||
<p className='text-[13px] text-[var(--text-secondary)] italic leading-[20px]'>{data.message}</p>
|
||||
)
|
||||
}
|
||||
|
||||
function UsageUpgradeDisplay({ data }: { data: UsageUpgradeTagData }) {
|
||||
|
||||
+14
-12
@@ -135,7 +135,7 @@ describe('parseBlocks span-identity tree', () => {
|
||||
subagentStart('workflow', 'S1', 'main'),
|
||||
subagentToolCall('t1', 'create_workflow', 'S1', 'workflow'),
|
||||
subagentStart('deploy', 'S2', 'S1'),
|
||||
subagentToolCall('t2', 'check_deployment_status', 'S2', 'deploy'),
|
||||
subagentToolCall('t2', 'get_deployment_status', 'S2', 'deploy'),
|
||||
]
|
||||
|
||||
const segments = parseBlocks(blocks)
|
||||
@@ -185,9 +185,9 @@ describe('parseBlocks span-identity tree', () => {
|
||||
it('creates distinct groups for repeated deploy invocations (no collision)', () => {
|
||||
const blocks: ContentBlock[] = [
|
||||
subagentStart('deploy', 'S2', 'main'),
|
||||
subagentToolCall('t1', 'deploy_api', 'S2', 'deploy'),
|
||||
subagentToolCall('t1', 'deploy_as_api', 'S2', 'deploy'),
|
||||
subagentStart('deploy', 'S4', 'main'),
|
||||
subagentToolCall('t2', 'deploy_api', 'S4', 'deploy'),
|
||||
subagentToolCall('t2', 'deploy_as_api', 'S4', 'deploy'),
|
||||
]
|
||||
|
||||
const segments = parseBlocks(blocks)
|
||||
@@ -311,7 +311,7 @@ describe('parseBlocks span-identity tree', () => {
|
||||
it('absorbs the dispatch tool of a nested file subagent from its parent span group', () => {
|
||||
const blocks: ContentBlock[] = [
|
||||
subagentStart('workflow', 'S1', 'main'),
|
||||
subagentToolCall('t1', 'workspace_file', 'S1', 'workflow'),
|
||||
subagentToolCall('t1', 'prepare_file_edit', 'S1', 'workflow'),
|
||||
{ type: 'subagent', content: 'file', spanId: 'S2', parentSpanId: 'S1', timestamp: 2 },
|
||||
{ type: 'subagent_text', content: 'writing', spanId: 'S2', timestamp: 3 },
|
||||
]
|
||||
@@ -321,7 +321,7 @@ describe('parseBlocks span-identity tree', () => {
|
||||
const workflow = segments[0]
|
||||
if (workflow.type !== 'agent_group') throw new Error('expected workflow group')
|
||||
|
||||
// The workspace_file dispatch tool is absorbed (not shown as a sibling tool);
|
||||
// The prepare_file_edit dispatch tool is absorbed (not shown as a sibling tool);
|
||||
// only the nested file subagent remains under workflow.
|
||||
expect(workflow.items.some((item) => item.type === 'tool')).toBe(false)
|
||||
const nested = workflow.items.find((item) => item.type === 'agent_group')
|
||||
@@ -476,16 +476,18 @@ describe('completed tool titles', () => {
|
||||
type: 'tool_call',
|
||||
toolCall: {
|
||||
id: 'undeploy-api',
|
||||
name: 'deploy_api',
|
||||
name: 'deploy_as_api',
|
||||
status: 'success',
|
||||
params: { action: 'undeploy' },
|
||||
},
|
||||
timestamp: 1,
|
||||
},
|
||||
])
|
||||
).toBe('Undeployed API')
|
||||
).toBe('Undeployed as API')
|
||||
|
||||
expect(firstToolTitle([mainToolCall('deploy-mcp', 'deploy_mcp')])).toBe('Deployed MCP tool')
|
||||
expect(firstToolTitle([mainToolCall('deploy-mcp', 'deploy_as_mcp')])).toBe(
|
||||
'Deployed as MCP tool'
|
||||
)
|
||||
})
|
||||
|
||||
it('renders Compared after the full diff_workflows wire lifecycle succeeds', () => {
|
||||
@@ -591,9 +593,9 @@ describe('completed tool titles', () => {
|
||||
expect(failures).toEqual([])
|
||||
})
|
||||
|
||||
it('keeps present tense while executing and on error', () => {
|
||||
it('keeps present tense while executing; failed rows say so', () => {
|
||||
expect(firstToolTitle([queryLogsCall('executing')])).toBe('Querying logs')
|
||||
expect(firstToolTitle([queryLogsCall('error')])).toBe('Querying logs')
|
||||
expect(firstToolTitle([queryLogsCall('error')])).toBe('Failed querying logs')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -755,7 +757,7 @@ describe('assistantMessageHasVisibleExecutingTool', () => {
|
||||
const blocks: ContentBlock[] = [
|
||||
{
|
||||
type: 'tool_call',
|
||||
toolCall: { id: 'dispatch-1', name: 'workspace_file', status: 'executing' },
|
||||
toolCall: { id: 'dispatch-1', name: 'prepare_file_edit', status: 'executing' },
|
||||
timestamp: 1,
|
||||
},
|
||||
{
|
||||
@@ -785,7 +787,7 @@ describe('deriveThinkingLabel', () => {
|
||||
|
||||
it('shows Dispatching for the dispatch call, then yields to the opened lane', () => {
|
||||
expect(deriveThinkingLabel([mainToolCall('t1', 'workflow')])).toBe('Dispatching…')
|
||||
expect(deriveThinkingLabel([mainToolCall('t1', 'workspace_file')])).toBe('Dispatching…')
|
||||
expect(deriveThinkingLabel([mainToolCall('t1', 'prepare_file_edit')])).toBe('Dispatching…')
|
||||
expect(deriveThinkingLabel([mainToolCall('t1', 'grep')])).toBe('Thinking…')
|
||||
expect(deriveThinkingLabel([subagentStart('workflow', 'S1', 'main')])).toBeNull()
|
||||
})
|
||||
|
||||
+4
-2
@@ -11,7 +11,7 @@ import {
|
||||
useState,
|
||||
} from 'react'
|
||||
import { cn } from '@sim/emcn'
|
||||
import { Read as ReadTool, WorkspaceFile } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { PrepareFileEdit, Read as ReadTool } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { isToolHiddenInUi } from '@/lib/copilot/tools/client/hidden-tools'
|
||||
import { resolveToolDisplay } from '@/lib/copilot/tools/client/store-utils'
|
||||
import { ClientToolCallState } from '@/lib/copilot/tools/client/tool-call-state'
|
||||
@@ -127,7 +127,7 @@ const SUBAGENT_KEYS = new Set(Object.keys(SUBAGENT_LABELS))
|
||||
* group is absorbed so it doesn't render as a separate Mothership entry.
|
||||
*/
|
||||
const SUBAGENT_DISPATCH_TOOLS: Record<string, string> = {
|
||||
[FILE_SUBAGENT_ID]: WorkspaceFile.id,
|
||||
[FILE_SUBAGENT_ID]: PrepareFileEdit.id,
|
||||
}
|
||||
|
||||
function isToolResultRead(params?: Record<string, unknown>): boolean {
|
||||
@@ -380,6 +380,7 @@ function parseBlocksWithSpanTree(blocks: ContentBlock[]): MessageSegment[] {
|
||||
const dispatchToolName = SUBAGENT_DISPATCH_TOOLS[block.content]
|
||||
if (dispatchToolName) absorbDispatchTool(dispatchToolName, block.parentSpanId)
|
||||
const g = ensureSpanGroup(block.content, block.spanId, block.parentSpanId)
|
||||
if (block.subagentName) g.agentLabel = block.subagentName
|
||||
if (block.endedAt !== undefined) {
|
||||
// Persisted backend path: the lane was stamped closed (endedAt) without
|
||||
// a separate subagent_end block (the Sim backend stamps endedAt only;
|
||||
@@ -623,6 +624,7 @@ function parseBlocksLegacy(blocks: ContentBlock[]): MessageSegment[] {
|
||||
}
|
||||
groupsByKey.delete(groupKey('mothership', undefined))
|
||||
const { group: g } = ensureGroup(key, block.parentToolCallId)
|
||||
if (block.subagentName) g.agentLabel = block.subagentName
|
||||
if (inheritedDelegation) g.isDelegating = true
|
||||
g.isOpen = true
|
||||
activeGroupKey = resolveGroupKey(key, block.parentToolCallId)
|
||||
|
||||
@@ -33,19 +33,19 @@ const TOOL_ICONS: Record<string, IconComponent> = {
|
||||
mv: FolderCode,
|
||||
cp: Layout,
|
||||
mkdir: FolderCode,
|
||||
search_online: Search,
|
||||
scrape_page: Search,
|
||||
get_page_contents: Search,
|
||||
web_search: Search,
|
||||
web_scrape: Search,
|
||||
web_fetch: Search,
|
||||
search_library_docs: Library,
|
||||
manage_mcp_tool: Settings,
|
||||
manage_mcp_connection: Settings,
|
||||
manage_skill: Asterisk,
|
||||
user_memory: Database,
|
||||
function_execute: TerminalWindow,
|
||||
run_function: TerminalWindow,
|
||||
run_code: TerminalWindow,
|
||||
superagent: Blimp,
|
||||
user_table: TableIcon,
|
||||
workspace_file: File,
|
||||
edit_content: File,
|
||||
prepare_file_edit: File,
|
||||
apply_file_edit: File,
|
||||
create_workflow: Layout,
|
||||
edit_workflow: Pencil,
|
||||
workflow: Hammer,
|
||||
@@ -64,6 +64,7 @@ const TOOL_ICONS: Record<string, IconComponent> = {
|
||||
research: Search,
|
||||
scout: Search,
|
||||
search: Search,
|
||||
platform: Library,
|
||||
context_compaction: Asterisk,
|
||||
open_resource: Eye,
|
||||
file: File,
|
||||
|
||||
+32
-3
@@ -664,6 +664,28 @@ export function MothershipChat({
|
||||
handleEditQueued(tail.id)
|
||||
}, [handleEditQueued])
|
||||
|
||||
/**
|
||||
* A drag-selection that overshoots a message's last line crosses the row
|
||||
* wrappers' block boundaries, which the clipboard serializer renders as
|
||||
* trailing newlines — every copied response pasted with blank lines
|
||||
* appended. Rewrite the plain-text flavor trimmed; the rich flavor is
|
||||
* re-serialized from the selection so formatted pastes keep working.
|
||||
*/
|
||||
const handleCopy = useCallback((event: React.ClipboardEvent) => {
|
||||
const selection = window.getSelection()
|
||||
if (!selection || selection.isCollapsed || !event.clipboardData) return
|
||||
const text = selection.toString()
|
||||
const trimmed = text.replace(/\s+$/, '')
|
||||
if (trimmed === text) return
|
||||
event.preventDefault()
|
||||
event.clipboardData.setData('text/plain', trimmed)
|
||||
const html = document.createElement('div')
|
||||
for (let i = 0; i < selection.rangeCount; i++) {
|
||||
html.appendChild(selection.getRangeAt(i).cloneContents())
|
||||
}
|
||||
event.clipboardData.setData('text/html', html.innerHTML)
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Land at the most recent message once per chat — on open and when switching
|
||||
* chats. The ref tracks which `chatId` we last scrolled for (seeded with
|
||||
@@ -696,7 +718,7 @@ export function MothershipChat({
|
||||
onWorkspaceResourceSelect={onWorkspaceResourceSelect}
|
||||
>
|
||||
<div className={cn('flex h-full min-h-0 flex-col', className)}>
|
||||
<div ref={setScrollElement} className={styles.scrollContainer}>
|
||||
<div ref={setScrollElement} className={styles.scrollContainer} onCopy={handleCopy}>
|
||||
{isLoading && !hasMessages ? (
|
||||
<MothershipChatSkeleton layout={layout} />
|
||||
) : (
|
||||
@@ -714,8 +736,15 @@ export function MothershipChat({
|
||||
key={virtualItem.key}
|
||||
data-index={index}
|
||||
ref={virtualizer.measureElement}
|
||||
className='absolute top-0 left-0 w-full'
|
||||
style={{ transform: `translateY(${virtualItem.start}px)` }}
|
||||
/* Positioned with a real `top`, NOT `top-0` + translateY:
|
||||
text selection maps a drag's start point to a text
|
||||
position via the rows' LAYOUT boxes, and with every row
|
||||
laid out at y=0 a drag starting in the gutter anchors in
|
||||
the wrong row — selections ran upward from a downward
|
||||
drag. Transforms move paint and hit-testing but not the
|
||||
layout box that mapping falls back to. */
|
||||
className='absolute left-0 w-full'
|
||||
style={{ top: virtualItem.start }}
|
||||
>
|
||||
{msg.role === 'user' ? (
|
||||
interactionPairing.hiddenUserByIndex[index] ? null : (
|
||||
|
||||
+42
@@ -30,6 +30,7 @@ import {
|
||||
createBrowserPanelGeometryOcclusionLease,
|
||||
hasNativeSurfaceOcclusion,
|
||||
NATIVE_SURFACE_OCCLUSION_SELECTOR,
|
||||
snapshotMatchesHost,
|
||||
useBrowserPanelOcclusion,
|
||||
} from '@/app/workspace/[workspaceId]/home/components/mothership-view/components/resource-content/components/browser-session/browser-panel-occlusion'
|
||||
|
||||
@@ -582,3 +583,44 @@ describe('useBrowserPanelOcclusion modal lifecycle', () => {
|
||||
hook.unmount()
|
||||
})
|
||||
})
|
||||
|
||||
describe('snapshotMatchesHost', () => {
|
||||
const rect = (x: number, y: number, width: number, height: number) =>
|
||||
({ x, y, width, height }) as DOMRect
|
||||
|
||||
it('accepts a capture that still describes the host rect', () => {
|
||||
expect(
|
||||
snapshotMatchesHost(
|
||||
{ viewportBounds: { x: 10, y: 20, width: 800, height: 600 } },
|
||||
rect(10, 20, 800, 600)
|
||||
)
|
||||
).toBe(true)
|
||||
})
|
||||
|
||||
it('tolerates sub-pixel drift from rounding', () => {
|
||||
expect(
|
||||
snapshotMatchesHost(
|
||||
{ viewportBounds: { x: 10, y: 20, width: 800, height: 600 } },
|
||||
rect(10.4, 19.6, 800.5, 599.5)
|
||||
)
|
||||
).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects a capture taken before a scroll lock reflowed the panel', () => {
|
||||
// Modal scroll lock removes the scrollbar: the host widens by 15px, so the
|
||||
// pre-lock capture would paint misaligned — the flash this guards.
|
||||
expect(
|
||||
snapshotMatchesHost(
|
||||
{ viewportBounds: { x: 10, y: 20, width: 800, height: 600 } },
|
||||
rect(10, 20, 815, 600)
|
||||
)
|
||||
).toBe(false)
|
||||
})
|
||||
|
||||
it('accepts captures with no viewport bounds (host-tracking fallback style)', () => {
|
||||
expect(snapshotMatchesHost({ viewportBounds: undefined }, rect(0, 0, 100, 100))).toBe(true)
|
||||
expect(
|
||||
snapshotMatchesHost({ viewportBounds: { x: 0, y: 0, width: 10, height: 10 } }, null)
|
||||
).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
+35
-2
@@ -184,10 +184,34 @@ async function decodeSnapshot(dataUrl: string): Promise<boolean> {
|
||||
* changing layers never reveals or recaptures the native view, and the view is
|
||||
* revealed only after the final reason disappears.
|
||||
*/
|
||||
|
||||
/** Largest tolerated drift, in CSS px, between a capture and the live host rect. */
|
||||
const SNAPSHOT_GEOMETRY_TOLERANCE_PX = 1
|
||||
|
||||
/**
|
||||
* Whether a captured frame still describes the rectangle the panel occupies.
|
||||
* A capture with no viewport bounds is positioned by the fallback
|
||||
* `absolute inset-0` style, which tracks the host by construction.
|
||||
*/
|
||||
export function snapshotMatchesHost(
|
||||
frame: Pick<BrowserPanelSnapshot, 'viewportBounds'>,
|
||||
hostRect: DOMRect | null
|
||||
): boolean {
|
||||
const bounds = frame.viewportBounds
|
||||
if (!bounds || !hostRect) return true
|
||||
return (
|
||||
Math.abs(bounds.x - hostRect.x) <= SNAPSHOT_GEOMETRY_TOLERANCE_PX &&
|
||||
Math.abs(bounds.y - hostRect.y) <= SNAPSHOT_GEOMETRY_TOLERANCE_PX &&
|
||||
Math.abs(bounds.width - hostRect.width) <= SNAPSHOT_GEOMETRY_TOLERANCE_PX &&
|
||||
Math.abs(bounds.height - hostRect.height) <= SNAPSHOT_GEOMETRY_TOLERANCE_PX
|
||||
)
|
||||
}
|
||||
|
||||
export function useBrowserPanelOcclusion(
|
||||
scopeId: string,
|
||||
activeTabId: string | null,
|
||||
panelVisible = true
|
||||
panelVisible = true,
|
||||
getHostRect?: () => DOMRect | null
|
||||
): BrowserPanelOcclusion {
|
||||
const [snapshotRender, setSnapshotRender] = useState<SnapshotRender | null>(null)
|
||||
const [activeOverlay, setActiveOverlay] = useState<BrowserPanelOverlay | null>(null)
|
||||
@@ -204,8 +228,10 @@ export function useBrowserPanelOcclusion(
|
||||
const paintFramesRef = useRef<number[]>([])
|
||||
const reconcileChainRef = useRef<Promise<boolean>>(Promise.resolve(true))
|
||||
const mountedRef = useRef(true)
|
||||
const getHostRectRef = useRef(getHostRect)
|
||||
activeTabIdRef.current = activeTabId
|
||||
panelVisibleRef.current = panelVisible
|
||||
getHostRectRef.current = getHostRect
|
||||
|
||||
const updateSnapshotRender = useCallback((render: SnapshotRender | null) => {
|
||||
snapshotRenderRef.current = render
|
||||
@@ -303,7 +329,7 @@ export function useBrowserPanelOcclusion(
|
||||
|
||||
// Modal scroll locking can alter panel geometry between capture and the
|
||||
// final native hide. One fresh capture retries that now-settled layout.
|
||||
const maxAttempts = desired === 'modal' ? 2 : 1
|
||||
const maxAttempts = desired === 'modal' ? 3 : 1
|
||||
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
||||
const frame = await captureBrowserPanelSnapshot(scopeId).catch(() => null)
|
||||
if (!mountedRef.current || version !== transitionVersionRef.current) return false
|
||||
@@ -318,6 +344,13 @@ export function useBrowserPanelOcclusion(
|
||||
desired = desiredLayer()
|
||||
if (!desired || !decoded) continue
|
||||
|
||||
// Painting a capture whose geometry no longer matches the host is the
|
||||
// flash: a modal's scroll lock changes the window's content width
|
||||
// between capture and paint, so the replacement lands offset from the
|
||||
// page it is standing in for. Skip that frame and re-capture at the
|
||||
// settled layout instead of showing a misaligned one.
|
||||
if (!snapshotMatchesHost(frame, getHostRectRef.current?.() ?? null)) continue
|
||||
|
||||
const paintId = ++paintIdRef.current
|
||||
const painted = new Promise<boolean>((resolve) => {
|
||||
pendingPaintRef.current = { paintId, resolve }
|
||||
|
||||
+4
@@ -190,6 +190,10 @@ describe('initialUrlSuggestionIndex', () => {
|
||||
expect(initialUrlSuggestionIndex('https://sim.ai', 3)).toBeNull()
|
||||
})
|
||||
|
||||
it('selects the exact search row after typing on an existing page', () => {
|
||||
expect(initialUrlSuggestionIndex('https://sim.ai', 3, 'what is the best')).toBe(0)
|
||||
})
|
||||
|
||||
it('selects nothing when there are no suggestions', () => {
|
||||
expect(initialUrlSuggestionIndex('', 0)).toBeNull()
|
||||
})
|
||||
|
||||
+102
-27
@@ -39,6 +39,7 @@ import { onFocusVisibleBrowserOmnibox } from '@/lib/browser-agent/renderer-short
|
||||
import {
|
||||
fillBrowserCredential,
|
||||
loadBrowserFillOptions,
|
||||
loadBrowserSearchSuggestions,
|
||||
loadBrowserSuggestionSources,
|
||||
onBrowserAddToChat,
|
||||
onBrowserAppearanceThemeChanged,
|
||||
@@ -84,12 +85,16 @@ import {
|
||||
import { BrowserTabStrip } from '@/app/workspace/[workspaceId]/home/components/mothership-view/components/resource-content/components/browser-session/browser-tab-strip'
|
||||
import { BrowserThemeNotice } from '@/app/workspace/[workspaceId]/home/components/mothership-view/components/resource-content/components/browser-session/browser-theme-notice'
|
||||
import {
|
||||
buildOmniboxSuggestions,
|
||||
googleSearchUrl,
|
||||
isSearchQueryInput,
|
||||
mergeSuggestionSources,
|
||||
moveActiveIndex,
|
||||
rankSuggestions,
|
||||
type OmniboxSuggestion,
|
||||
type UrlSuggestion,
|
||||
} from '@/app/workspace/[workspaceId]/home/components/mothership-view/components/resource-content/components/browser-session/url-suggestions'
|
||||
import { ResourceZoomMenuItems } from '@/app/workspace/[workspaceId]/home/components/mothership-view/components/resource-content/components/resource-zoom-menu-items'
|
||||
import { useDebounce } from '@/hooks/use-debounce'
|
||||
import { useSettingsNavigation } from '@/hooks/use-settings-navigation'
|
||||
import { useBrowserSessionStore } from '@/stores/browser-session/store'
|
||||
import { MOTHERSHIP_WIDTH } from '@/stores/constants'
|
||||
@@ -97,6 +102,7 @@ import type { ChatContext } from '@/stores/panel'
|
||||
|
||||
/** Ties the omnibox to its listbox for assistive tech. */
|
||||
const SUGGESTIONS_LIST_ID = 'browser-url-suggestions'
|
||||
const SEARCH_SUGGESTIONS_DEBOUNCE_MS = 160
|
||||
const NEW_TAB_CONFIRM_TIMEOUT_MS = 10_000
|
||||
const EMPTY_BROWSER_TABS: BrowserTabState[] = []
|
||||
|
||||
@@ -156,14 +162,10 @@ export function browserSelectionContext({
|
||||
export function resolveUrlBarInput(raw: string): string {
|
||||
const input = raw.trim()
|
||||
if (/^https?:\/\//i.test(input)) return input
|
||||
const hostLike =
|
||||
/^([a-z0-9-]+(\.[a-z0-9-]+)+|localhost|\d{1,3}(\.\d{1,3}){3}|\[[0-9a-f:]+\])(:\d+)?([/?#].*)?$/i
|
||||
if (!input.includes(' ') && hostLike.test(input)) {
|
||||
const isLocal =
|
||||
/^(localhost|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|0\.0\.0\.0|\[::1?\])(:\d+)?([/?#]|$)/i.test(input)
|
||||
return `${isLocal ? 'http' : 'https'}://${input}`
|
||||
}
|
||||
return `https://www.google.com/search?q=${encodeURIComponent(input)}`
|
||||
if (isSearchQueryInput(input)) return googleSearchUrl(input)
|
||||
const isLocal =
|
||||
/^(localhost|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|0\.0\.0\.0|\[::1?\])(:\d+)?([/?#]|$)/i.test(input)
|
||||
return `${isLocal ? 'http' : 'https'}://${input}`
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -302,13 +304,14 @@ export function shouldOpenUrlSuggestions(
|
||||
return activeOverlay === 'suggestions' && suggestionCount > 0
|
||||
}
|
||||
|
||||
/** New tabs submit the best suggestion; existing pages submit their current URL. */
|
||||
/** Typed searches and new tabs select the first row; an untouched page URL remains literal. */
|
||||
export function initialUrlSuggestionIndex(
|
||||
pageUrl: string | undefined,
|
||||
suggestionCount: number
|
||||
suggestionCount: number,
|
||||
query = ''
|
||||
): number | null {
|
||||
if (suggestionCount === 0) return null
|
||||
return !pageUrl || pageUrl === 'about:blank' ? 0 : null
|
||||
return query.trim() || !pageUrl || pageUrl === 'about:blank' ? 0 : null
|
||||
}
|
||||
|
||||
/** A new-tab request is complete only after the authoritative strip grows and activates a new id. */
|
||||
@@ -369,6 +372,9 @@ export function BrowserSession({
|
||||
const suspended = useBrowserSessionStore((state) => state.sessions[scopeId]?.suspended ?? false)
|
||||
const panelRef = useRef<HTMLDivElement>(null)
|
||||
const hostRef = useRef<HTMLDivElement>(null)
|
||||
// Lets the occlusion handshake reject a capture taken before a modal's
|
||||
// scroll lock reflowed the panel — painting that stale rect is the flash.
|
||||
const getHostRect = useCallback(() => hostRef.current?.getBoundingClientRect() ?? null, [])
|
||||
const urlInputRef = useRef<HTMLInputElement>(null)
|
||||
const findInputRef = useRef<HTMLInputElement>(null)
|
||||
const fillButtonRef = useRef<HTMLButtonElement>(null)
|
||||
@@ -429,6 +435,15 @@ export function BrowserSession({
|
||||
const [suggestionsVisible, setSuggestionsVisible] = useState(false)
|
||||
/** Empty on initial focus; follows the typed text once the user edits it. */
|
||||
const [suggestionQuery, setSuggestionQuery] = useState<string | null>(null)
|
||||
/** Live completions tagged with the query that produced them, so late replies cannot leak in. */
|
||||
const [searchCompletions, setSearchCompletions] = useState<{
|
||||
query: string
|
||||
values: string[]
|
||||
}>({ query: '', values: [] })
|
||||
const debouncedSuggestionQuery = useDebounce(
|
||||
suggestionQuery ?? '',
|
||||
SEARCH_SUGGESTIONS_DEBOUNCE_MS
|
||||
)
|
||||
/** Whether the find bar is docked above the page. */
|
||||
const [findOpen, setFindOpen] = useState(false)
|
||||
const {
|
||||
@@ -438,7 +453,7 @@ export function BrowserSession({
|
||||
requestOverlay,
|
||||
closeOverlay,
|
||||
onSnapshotError,
|
||||
} = useBrowserPanelOcclusion(scopeId, activeTabId, panelVisible)
|
||||
} = useBrowserPanelOcclusion(scopeId, activeTabId, panelVisible, getHostRect)
|
||||
|
||||
// The resource picker lives above this component in the panel tab bar. Give
|
||||
// that one external browser overlay access to the same capture/hide handshake
|
||||
@@ -502,6 +517,23 @@ export function BrowserSession({
|
||||
}
|
||||
}, [panelVisible])
|
||||
|
||||
/** Debounced live completions never block the immediate local/search row. */
|
||||
useEffect(() => {
|
||||
const query = debouncedSuggestionQuery.trim()
|
||||
if (!suggestionsVisible || !isSearchQueryInput(query)) {
|
||||
setSearchCompletions({ query: '', values: [] })
|
||||
return
|
||||
}
|
||||
|
||||
let active = true
|
||||
void loadBrowserSearchSuggestions(query).then((values) => {
|
||||
if (active) setSearchCompletions({ query, values })
|
||||
})
|
||||
return () => {
|
||||
active = false
|
||||
}
|
||||
}, [debouncedSuggestionQuery, suggestionsVisible])
|
||||
|
||||
useEffect(() => {
|
||||
if (appearanceTheme) {
|
||||
const next = resolveDesktopAppearanceTheme(appearanceTheme, theme)
|
||||
@@ -579,6 +611,24 @@ export function BrowserSession({
|
||||
|
||||
useEffect(() => onBrowserOmniboxFocus(focusOmnibox, scopeId), [focusOmnibox, scopeId])
|
||||
|
||||
// Follow the agent's tab. The panel already marks the automated tab in the
|
||||
// strip; this makes it the VISIBLE one, so watching the agent never means
|
||||
// hunting for which tab it moved to. Keyed on the automation target
|
||||
// CHANGING, not on it merely being set — the user can still browse a
|
||||
// different tab mid-run and is only pulled along when the agent itself
|
||||
// moves to another tab.
|
||||
const followedAutomationTabRef = useRef<string | null>(null)
|
||||
useEffect(() => {
|
||||
if (!automationActive || !automationTabId) {
|
||||
if (!automationActive) followedAutomationTabRef.current = null
|
||||
return
|
||||
}
|
||||
if (followedAutomationTabRef.current === automationTabId) return
|
||||
followedAutomationTabRef.current = automationTabId
|
||||
if (automationTabId === activeTabId) return
|
||||
sendBrowserPanelAction('switch-tab', { tabId: automationTabId }, scopeId)
|
||||
}, [activeTabId, automationActive, automationTabId, scopeId])
|
||||
|
||||
// Sim owns keyboard events while its renderer has focus. Claim Cmd+L here
|
||||
// before the workspace's global "Go to Logs" command can navigate away.
|
||||
useEffect(() => {
|
||||
@@ -748,6 +798,17 @@ export function BrowserSession({
|
||||
// then report bounds: the WebContentsView is attached hidden from its
|
||||
// very first compositor frame. This also reasserts the lease after a
|
||||
// minimized/throttled window temporarily loses its bounds.
|
||||
//
|
||||
// The reassert must be REAL, not deduped. Main's bounds lease expires
|
||||
// after 2.5s of missed reports (renderer jank, a skipped commit) and
|
||||
// resets panelOccluded — while this side still remembers `applied:
|
||||
// true`. Without dropping that belief, setDesired(true) is a no-op,
|
||||
// the next bounds commit lays out an unoccluded native view, and the
|
||||
// browser punches above the still-open modal with nothing left to
|
||||
// ever re-hide it. Forgetting `applied` costs one idempotent hide IPC
|
||||
// per heartbeat while a modal is up, and makes any main-side lease
|
||||
// loss self-heal within a second.
|
||||
if (nativeSurfaceOcclusionPresent) geometryOcclusionLease.assumeRevealed()
|
||||
void geometryOcclusionLease.setDesired(nativeSurfaceOcclusionPresent).then((settled) => {
|
||||
if (disposed) return
|
||||
const latestOcclusionPresent = atomicPanelOcclusion && hasNativeSurfaceOcclusion()
|
||||
@@ -833,17 +894,18 @@ export function BrowserSession({
|
||||
* Programmatic focus on a new tab keeps the omnibox ready for typing without
|
||||
* opening this list. A pointer interaction or typed edit opts into suggestions.
|
||||
*/
|
||||
const suggestions = useMemo(
|
||||
() =>
|
||||
suggestionsVisible && suggestionQuery !== null
|
||||
? rankSuggestions(suggestionCorpus, suggestionQuery)
|
||||
: [],
|
||||
[suggestionCorpus, suggestionQuery, suggestionsVisible]
|
||||
)
|
||||
const suggestions = useMemo((): OmniboxSuggestion[] => {
|
||||
if (!suggestionsVisible || suggestionQuery === null) return []
|
||||
const query = suggestionQuery.trim()
|
||||
const live = searchCompletions.query === query ? searchCompletions.values : []
|
||||
return buildOmniboxSuggestions(suggestionCorpus, suggestionQuery, live)
|
||||
}, [searchCompletions, suggestionCorpus, suggestionQuery, suggestionsVisible])
|
||||
|
||||
useEffect(() => {
|
||||
setActiveSuggestion(initialUrlSuggestionIndex(suggestionOriginUrl, suggestions.length))
|
||||
}, [suggestionOriginUrl, suggestions])
|
||||
setActiveSuggestion(
|
||||
initialUrlSuggestionIndex(suggestionOriginUrl, suggestions.length, suggestionQuery ?? '')
|
||||
)
|
||||
}, [suggestionOriginUrl, suggestionQuery, suggestions])
|
||||
|
||||
// The suggestion list is renderer UI that extends over the native page.
|
||||
// Keep the page's exact captured frame underneath it while it is open so
|
||||
@@ -1075,6 +1137,7 @@ export function BrowserSession({
|
||||
placeholder='Search Google or enter a URL'
|
||||
autoComplete='off'
|
||||
role='combobox'
|
||||
aria-autocomplete='list'
|
||||
aria-expanded={suggestionsOpen}
|
||||
aria-controls={SUGGESTIONS_LIST_ID}
|
||||
aria-activedescendant={
|
||||
@@ -1092,6 +1155,7 @@ export function BrowserSession({
|
||||
onChange={(event) => {
|
||||
setSuggestionsVisible(true)
|
||||
setSuggestionQuery(event.target.value)
|
||||
setSearchCompletions({ query: '', values: [] })
|
||||
setUrlDraft(event.target.value)
|
||||
// The old highlight pointed at a row that may no longer be
|
||||
// in the list, let alone in the same position.
|
||||
@@ -1150,7 +1214,11 @@ export function BrowserSession({
|
||||
>
|
||||
{suggestions.map((suggestion, index) => (
|
||||
<PopoverItem
|
||||
key={suggestion.hostname}
|
||||
key={
|
||||
suggestion.kind === 'site'
|
||||
? `site:${suggestion.hostname}`
|
||||
: `search:${suggestion.query}`
|
||||
}
|
||||
id={suggestionRowId(index)}
|
||||
role='option'
|
||||
aria-selected={index === activeSuggestion}
|
||||
@@ -1161,17 +1229,24 @@ export function BrowserSession({
|
||||
onClick={() => navigateTo(suggestion.url)}
|
||||
>
|
||||
<div className='flex w-full min-w-0 items-center gap-2'>
|
||||
<BrowserSuggestionIcon suggestion={suggestion} />
|
||||
{suggestion.name ? (
|
||||
{suggestion.kind === 'search' ? (
|
||||
<>
|
||||
<Search className='size-4 shrink-0 text-[var(--text-icon)]' />
|
||||
<span className='truncate'>{suggestion.query}</span>
|
||||
</>
|
||||
) : (
|
||||
<BrowserSuggestionIcon suggestion={suggestion} />
|
||||
)}
|
||||
{suggestion.kind === 'site' && suggestion.name ? (
|
||||
<div className='flex min-w-0 flex-1 items-center gap-1.5'>
|
||||
<span className='min-w-0 flex-1 truncate'>{suggestion.name}</span>
|
||||
<span className='max-w-[45%] shrink-0 truncate text-[var(--text-muted)]'>
|
||||
— {suggestion.hostname}
|
||||
</span>
|
||||
</div>
|
||||
) : (
|
||||
) : suggestion.kind === 'site' ? (
|
||||
<span className='truncate'>{suggestion.hostname}</span>
|
||||
)}
|
||||
) : null}
|
||||
</div>
|
||||
</PopoverItem>
|
||||
))}
|
||||
|
||||
+62
@@ -2,6 +2,9 @@ import type { BrowserKnownSession, BrowserSessionEvidence } from '@sim/browser-p
|
||||
import type { BrowserCredentialMetadata, BrowserSiteInfo } from '@sim/desktop-bridge'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
buildOmniboxSuggestions,
|
||||
googleSearchUrl,
|
||||
isSearchQueryInput,
|
||||
mergeSuggestionSources,
|
||||
moveActiveIndex,
|
||||
rankSuggestions,
|
||||
@@ -421,6 +424,65 @@ describe('rankSuggestions', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildOmniboxSuggestions', () => {
|
||||
it('leads with the exact search, keeps matching sites, then adds live completions', () => {
|
||||
const results = buildOmniboxSuggestions(
|
||||
[suggestion('mail.google.com', 100, 'Gmail')],
|
||||
'gmail',
|
||||
['gmail login', 'gmail account']
|
||||
)
|
||||
|
||||
expect(results.map((result) => [result.kind, result.url])).toEqual([
|
||||
['search', googleSearchUrl('gmail')],
|
||||
['site', 'https://mail.google.com'],
|
||||
['search', googleSearchUrl('gmail login')],
|
||||
['search', googleSearchUrl('gmail account')],
|
||||
])
|
||||
})
|
||||
|
||||
it('does not send URL-looking input through search completions', () => {
|
||||
const results = buildOmniboxSuggestions([suggestion('github.com', 100)], 'github.com', [
|
||||
'github.com login',
|
||||
])
|
||||
|
||||
expect(results).toHaveLength(1)
|
||||
expect(results[0]).toMatchObject({ kind: 'site', hostname: 'github.com' })
|
||||
})
|
||||
|
||||
it('deduplicates completions and caps the combined dropdown', () => {
|
||||
const results = buildOmniboxSuggestions(
|
||||
[],
|
||||
'sim ai',
|
||||
['sim ai', 'SIM AI', 'sim ai workflow', 'sim ai agents'],
|
||||
2
|
||||
)
|
||||
|
||||
expect(results.map((result) => result.kind === 'search' && result.query)).toEqual([
|
||||
'sim ai',
|
||||
'sim ai workflow',
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps an empty omnibox local-only', () => {
|
||||
const results = buildOmniboxSuggestions([suggestion('github.com', 100)], '', [
|
||||
'ignored remote completion',
|
||||
])
|
||||
|
||||
expect(results).toHaveLength(1)
|
||||
expect(results[0]).toMatchObject({ kind: 'site', hostname: 'github.com' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('isSearchQueryInput', () => {
|
||||
it('distinguishes searches from navigable addresses', () => {
|
||||
expect(isSearchQueryInput('what is the best browser')).toBe(true)
|
||||
expect(isSearchQueryInput('electron')).toBe(true)
|
||||
expect(isSearchQueryInput('sim.ai/docs')).toBe(false)
|
||||
expect(isSearchQueryInput('https://sim.ai')).toBe(false)
|
||||
expect(isSearchQueryInput('localhost:3000')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('moveActiveIndex', () => {
|
||||
it('highlights nothing until the user arrows in, so Enter still means "go to what I typed"', () => {
|
||||
expect(moveActiveIndex(null, 1, 3)).toBe(0)
|
||||
|
||||
+55
@@ -47,6 +47,25 @@ export interface UrlSuggestion {
|
||||
visits?: number
|
||||
}
|
||||
|
||||
export type OmniboxSuggestion =
|
||||
| ({ kind: 'site' } & UrlSuggestion)
|
||||
| { kind: 'search'; query: string; url: string }
|
||||
|
||||
const HOST_LIKE_INPUT =
|
||||
/^([a-z0-9-]+(\.[a-z0-9-]+)+|localhost|\d{1,3}(\.\d{1,3}){3}|\[[0-9a-f:]+\])(:\d+)?([/?#].*)?$/i
|
||||
|
||||
/** Whether an omnibox value should search rather than navigate directly. */
|
||||
export function isSearchQueryInput(raw: string): boolean {
|
||||
const input = raw.trim()
|
||||
if (!input || /^https?:\/\//i.test(input)) return false
|
||||
return input.includes(' ') || !HOST_LIKE_INPUT.test(input)
|
||||
}
|
||||
|
||||
/** The canonical Google results URL used by search rows and bare submission. */
|
||||
export function googleSearchUrl(query: string): string {
|
||||
return `https://www.google.com/search?q=${encodeURIComponent(query.trim())}`
|
||||
}
|
||||
|
||||
function timestamp(value: string | undefined): number {
|
||||
if (!value) return 0
|
||||
const parsed = Date.parse(value)
|
||||
@@ -189,6 +208,42 @@ export function rankSuggestions(
|
||||
return scored.slice(0, limit).map((entry) => entry.suggestion)
|
||||
}
|
||||
|
||||
/**
|
||||
* Combines immediate navigation/search actions with the user's known sites and
|
||||
* live completions. The exact typed search leads, known sites retain priority,
|
||||
* and remote completions fill whatever room remains.
|
||||
*/
|
||||
export function buildOmniboxSuggestions(
|
||||
siteCorpus: readonly UrlSuggestion[],
|
||||
rawQuery: string,
|
||||
searchCompletions: readonly string[] = [],
|
||||
limit: number = MAX_URL_SUGGESTIONS
|
||||
): OmniboxSuggestion[] {
|
||||
if (limit <= 0) return []
|
||||
const query = rawQuery.trim()
|
||||
const sites = rankSuggestions(siteCorpus, query, limit)
|
||||
if (!query || !isSearchQueryInput(query)) {
|
||||
return sites.map((site) => ({ ...site, kind: 'site' }))
|
||||
}
|
||||
|
||||
const results: OmniboxSuggestion[] = [{ kind: 'search', query, url: googleSearchUrl(query) }]
|
||||
for (const site of sites) {
|
||||
if (results.length === limit) return results
|
||||
results.push({ ...site, kind: 'site' })
|
||||
}
|
||||
|
||||
const seen = new Set([query.toLocaleLowerCase()])
|
||||
for (const candidate of searchCompletions) {
|
||||
const completion = candidate.trim()
|
||||
const key = completion.toLocaleLowerCase()
|
||||
if (!completion || seen.has(key)) continue
|
||||
seen.add(key)
|
||||
results.push({ kind: 'search', query: completion, url: googleSearchUrl(completion) })
|
||||
if (results.length === limit) break
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
/**
|
||||
* How well the browser knows a host, then how much it is used, then how
|
||||
* recently, then alphabetically so the same corpus always comes back in the
|
||||
|
||||
+1
@@ -256,6 +256,7 @@ export const ResourceContent = memo(function ResourceContent({
|
||||
tableId={resource.id}
|
||||
embedded
|
||||
viewsEnabled={tableViewsEnabled}
|
||||
initialViewId={resource.viewId}
|
||||
/>
|
||||
)
|
||||
|
||||
|
||||
+2
-2
@@ -243,9 +243,9 @@ const ResourceTabItem = memo(function ResourceTabItem({
|
||||
>
|
||||
{config.renderTabIcon(resource, 'mr-1.5 size-[14px]')}
|
||||
{displayName}
|
||||
{hasActivity && !isActive && (
|
||||
{hasActivity && !isActive && !isHovered && (
|
||||
<span
|
||||
className='ml-1 size-1.5 shrink-0 rounded-full bg-[var(--brand-primary)]'
|
||||
className='-translate-y-1/2 absolute top-1/2 right-[9px] size-1.5 rounded-full bg-[var(--brand-primary)]'
|
||||
aria-label='Background activity'
|
||||
/>
|
||||
)}
|
||||
|
||||
+5
-1
@@ -4,6 +4,7 @@ import { forwardRef, memo, useCallback, useMemo, useRef, useState } from 'react'
|
||||
import { cn } from '@sim/emcn'
|
||||
import type { FilePreviewSession } from '@/lib/copilot/request/session'
|
||||
import { getFileExtension } from '@/lib/uploads/utils/file-utils'
|
||||
import { SIM_PAGE_CONTENT_TYPE } from '@/lib/workspace-files/page-compile'
|
||||
import type { PreviewMode } from '@/app/workspace/[workspaceId]/files/components/file-viewer'
|
||||
import {
|
||||
isCsvStreamOnly,
|
||||
@@ -164,7 +165,10 @@ export const MothershipView = memo(
|
||||
// the record before deciding so the toggle doesn't flash on for a large CSV — but don't gate
|
||||
// other rich types (html, svg, …) on the file list loading.
|
||||
!(isActiveCsv && filesLoading) &&
|
||||
!(activeFile && isCsvStreamOnly(activeFile))
|
||||
!(activeFile && isCsvStreamOnly(activeFile)) &&
|
||||
// A Sim page is locked to its rendered view (the pdf model — the raw
|
||||
// source is not a mode this surface offers), so no toggle either.
|
||||
activeFile?.type !== SIM_PAGE_CONTENT_TYPE
|
||||
|
||||
return (
|
||||
<div
|
||||
|
||||
@@ -55,7 +55,13 @@ import {
|
||||
UserInput,
|
||||
type UserInputHandle,
|
||||
} from './components'
|
||||
import { getMothershipUseChatOptions, useChat, useMothershipResize } from './hooks'
|
||||
import {
|
||||
getMothershipUseChatOptions,
|
||||
type ResourceEventOptions,
|
||||
shouldActivateResourceEvent,
|
||||
useChat,
|
||||
useMothershipResize,
|
||||
} from './hooks'
|
||||
import type {
|
||||
FileAttachmentForApi,
|
||||
MothershipResource,
|
||||
@@ -211,14 +217,14 @@ export function Home({ chatId, userName, userId, tableViewsEnabled }: HomeProps)
|
||||
const activeResourceParamRef = useRef(activeResourceParam)
|
||||
activeResourceParamRef.current = activeResourceParam
|
||||
|
||||
function handleResourceEvent(resourceId: string) {
|
||||
// Agent work should always make the resource surface available, but it
|
||||
// must never replace an existing selection. Activity in another resource
|
||||
// stays in the background and gets an attention marker instead.
|
||||
function handleResourceEvent(resourceId: string, options?: ResourceEventOptions) {
|
||||
// Agent work surfaces the resource and switches to it as it is created or
|
||||
// edited; only the browser session stays in the background behind an
|
||||
// existing selection (see shouldActivateResourceEvent).
|
||||
if (isResourceCollapsedRef.current) setIsResourceCollapsed(false)
|
||||
|
||||
const activeResourceId = activeResourceParamRef.current
|
||||
if (activeResourceId && activeResourceId !== resourceId) {
|
||||
if (!shouldActivateResourceEvent(activeResourceId, resourceId, options)) {
|
||||
setResourceActivityIds((current) => new Set(current).add(resourceId))
|
||||
return
|
||||
}
|
||||
@@ -228,7 +234,10 @@ export function Home({ chatId, userName, userId, tableViewsEnabled }: HomeProps)
|
||||
next.delete(resourceId)
|
||||
return next
|
||||
})
|
||||
if (activeResourceId !== resourceId) setActiveResourceUrl(resourceId)
|
||||
if (activeResourceId !== resourceId) {
|
||||
activeResourceParamRef.current = resourceId
|
||||
setActiveResourceUrl(resourceId)
|
||||
}
|
||||
}
|
||||
|
||||
const {
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
export type { ResourceEventOptions } from './use-chat'
|
||||
export {
|
||||
getMothershipUseChatOptions,
|
||||
getWorkflowCopilotUseChatOptions,
|
||||
shouldActivateResourceEvent,
|
||||
useChat,
|
||||
} from './use-chat'
|
||||
export { useMothershipResize } from './use-mothership-resize'
|
||||
|
||||
@@ -4,7 +4,7 @@ import {
|
||||
MothershipStreamV1ToolPhase,
|
||||
MothershipStreamV1ToolStatus,
|
||||
} from '@/lib/copilot/generated/mothership-stream-v1'
|
||||
import { WorkspaceFile } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { ApplyFileEdit, PrepareFileEdit } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import type { PersistedStreamEventEnvelope } from '@/lib/copilot/request/session/contract'
|
||||
import {
|
||||
extractResourcesFromToolResult,
|
||||
@@ -79,7 +79,7 @@ function runToolResultSideEffects(ctx: StreamLoopContext, node: ToolNode): void
|
||||
invalidateResourceQueries(deps.queryClient, deps.workspaceId, resource.type, resource.id)
|
||||
}
|
||||
|
||||
if ((name === 'edit_content' || name === WorkspaceFile.id) && isSuccess) {
|
||||
if ((name === ApplyFileEdit.id || name === PrepareFileEdit.id) && isSuccess) {
|
||||
const out = output as Record<string, unknown> | undefined
|
||||
const editData =
|
||||
out && typeof out.data === 'object' && out.data !== null
|
||||
@@ -102,17 +102,20 @@ function runToolResultSideEffects(ctx: StreamLoopContext, node: ToolNode): void
|
||||
deps.onToolResultRef.current?.(name, isSuccess, output)
|
||||
|
||||
const workspaceFileOperation =
|
||||
name === WorkspaceFile.id && typeof params?.operation === 'string'
|
||||
name === PrepareFileEdit.id && typeof params?.operation === 'string'
|
||||
? params.operation
|
||||
: undefined
|
||||
const shouldKeepWorkspacePreviewOpen =
|
||||
name === WorkspaceFile.id &&
|
||||
name === PrepareFileEdit.id &&
|
||||
(workspaceFileOperation === 'append' ||
|
||||
workspaceFileOperation === 'update' ||
|
||||
workspaceFileOperation === 'patch')
|
||||
|
||||
if ((name === WorkspaceFile.id || name === 'edit_content') && !shouldKeepWorkspacePreviewOpen) {
|
||||
if (name === WorkspaceFile.id) {
|
||||
if (
|
||||
(name === PrepareFileEdit.id || name === ApplyFileEdit.id) &&
|
||||
!shouldKeepWorkspacePreviewOpen
|
||||
) {
|
||||
if (name === PrepareFileEdit.id) {
|
||||
deps.removePreviewSessionImmediate(node.id)
|
||||
}
|
||||
const fileResource = extractedResources.find((r) => r.type === 'file')
|
||||
@@ -128,7 +131,7 @@ function runToolResultSideEffects(ctx: StreamLoopContext, node: ToolNode): void
|
||||
|
||||
/**
|
||||
* Side effects for tool events. State (the tool node, its status, args, and the
|
||||
* edit_content row merge) is owned by `reduceEvent`; this handler routes preview
|
||||
* apply_file_edit row merge) is owned by `reduceEvent`; this handler routes preview
|
||||
* phases, fires client workflow tools, and runs result side effects, then
|
||||
* flushes the model-derived snapshot.
|
||||
*/
|
||||
|
||||
@@ -2,35 +2,37 @@ import { createLogger } from '@sim/logger'
|
||||
import { isRecordLike } from '@sim/utils/object'
|
||||
import {
|
||||
CallIntegrationTool,
|
||||
CrawlWebsite,
|
||||
CreateFile,
|
||||
CreateEmptyFile,
|
||||
CreateWorkflow,
|
||||
DeployApi,
|
||||
DeployChat,
|
||||
DeployMcp,
|
||||
DeployAsApi,
|
||||
DeployAsChat,
|
||||
DeployAsMcp,
|
||||
EditWorkflow,
|
||||
FunctionExecute,
|
||||
Glob,
|
||||
Grep,
|
||||
ManageCredential,
|
||||
ManageCustomTool,
|
||||
ManageMcpTool,
|
||||
ManageMcpConnection,
|
||||
ManageSkill,
|
||||
PrepareFileEdit,
|
||||
PrepareFileEditOperation,
|
||||
QueryLogs,
|
||||
Redeploy,
|
||||
Rm,
|
||||
RunFromBlock,
|
||||
RunFunction,
|
||||
RunWorkflow,
|
||||
RunWorkflowUntilBlock,
|
||||
ScrapePage,
|
||||
SearchOnline,
|
||||
WorkspaceFile,
|
||||
WorkspaceFileOperation,
|
||||
WebCrawl,
|
||||
WebScrape,
|
||||
WebSearch,
|
||||
} from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import { extractStreamingStringArgument } from '@/lib/copilot/tools/streaming-args'
|
||||
import { getToolDisplayTitle, mvDisplayVerb } from '@/lib/copilot/tools/tool-display'
|
||||
import { getQueryClient } from '@/app/_shell/providers/get-query-client'
|
||||
import type { ContentBlock } from '@/app/workspace/[workspaceId]/home/types'
|
||||
import { ToolCallStatus } from '@/app/workspace/[workspaceId]/home/types'
|
||||
import { tableKeys } from '@/hooks/queries/utils/table-keys'
|
||||
import { getWorkflowById } from '@/hooks/queries/utils/workflow-cache'
|
||||
import { useWorkflowRegistry } from '@/stores/workflows/registry/store'
|
||||
import { useWorkflowStore } from '@/stores/workflows/workflow/store'
|
||||
@@ -40,9 +42,9 @@ const logger = createLogger('StreamHelpers')
|
||||
export const FILE_SUBAGENT_ID = 'file'
|
||||
|
||||
export const DEPLOY_TOOL_NAMES: Set<string> = new Set([
|
||||
DeployApi.id,
|
||||
DeployChat.id,
|
||||
DeployMcp.id,
|
||||
DeployAsApi.id,
|
||||
DeployAsChat.id,
|
||||
DeployAsMcp.id,
|
||||
Redeploy.id,
|
||||
])
|
||||
|
||||
@@ -123,6 +125,33 @@ function resolveTargetWorkflowName(args: Record<string, unknown> | undefined): s
|
||||
return resolveWorkflowNameForDisplay(args?.workflowId ?? registry.hydration.workflowId)
|
||||
}
|
||||
|
||||
/**
|
||||
* Table name for a nested `args.tableId`. Tables reach the client through
|
||||
* React Query rather than a Zustand store, so the cached workspace list is
|
||||
* the synchronous source a title can read; an uncached id simply stays
|
||||
* unnamed rather than blocking the row.
|
||||
*/
|
||||
function resolveTableNameForDisplay(tableId: unknown): string | undefined {
|
||||
const id = stringParam(tableId)
|
||||
if (!id) return undefined
|
||||
const cache = getQueryClient().getQueryCache()
|
||||
for (const query of cache.findAll({ queryKey: tableKeys.lists() })) {
|
||||
const data = query.state.data
|
||||
const tables = Array.isArray(data)
|
||||
? data
|
||||
: isRecordLike(data) && Array.isArray((data as { tables?: unknown }).tables)
|
||||
? ((data as { tables: unknown[] }).tables as unknown[])
|
||||
: []
|
||||
for (const table of tables) {
|
||||
if (!isRecordLike(table)) continue
|
||||
if (stringParam(table.id) !== id) continue
|
||||
const name = stringParam(table.name)
|
||||
if (name) return name
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function resolveBlockNameForDisplay(blockId: unknown): string | undefined {
|
||||
const id = stringParam(blockId)
|
||||
if (!id) return undefined
|
||||
@@ -139,13 +168,13 @@ function resolveWorkspaceFileDisplayTitle(
|
||||
let verb = 'Writing'
|
||||
|
||||
switch (operation) {
|
||||
case WorkspaceFileOperation.append:
|
||||
case PrepareFileEditOperation.append:
|
||||
verb = 'Adding'
|
||||
break
|
||||
case WorkspaceFileOperation.patch:
|
||||
case PrepareFileEditOperation.patch:
|
||||
verb = 'Editing'
|
||||
break
|
||||
case WorkspaceFileOperation.update:
|
||||
case PrepareFileEditOperation.update:
|
||||
verb = 'Writing'
|
||||
break
|
||||
}
|
||||
@@ -186,6 +215,40 @@ export function resolveIntegrationToolDisplayTitle(tool: {
|
||||
return tool.integrationDescription
|
||||
}
|
||||
|
||||
/**
|
||||
* Tools whose subject is one workflow. They accept a `workflowId` (or imply
|
||||
* the current workflow), so their titles can only name the workflow once the
|
||||
* client resolves the id against the workflow registry.
|
||||
*/
|
||||
const TABLE_SCOPED_TOOL_IDS = new Set<string>([
|
||||
'table_automations',
|
||||
'table_columns',
|
||||
'table_enrichments',
|
||||
'table_manage',
|
||||
'table_rows',
|
||||
'table_views',
|
||||
])
|
||||
|
||||
const WORKFLOW_SCOPED_TOOL_IDS = new Set<string>([
|
||||
'deploy_as_api',
|
||||
'diff_workflows',
|
||||
'list_deployment_versions',
|
||||
'publish_custom_block',
|
||||
'deploy_as_chat',
|
||||
'deploy_as_mcp',
|
||||
'get_block_outputs',
|
||||
'get_block_upstream_references',
|
||||
'get_deployed_workflow_state',
|
||||
'get_deployment_status',
|
||||
'get_workflow_data',
|
||||
'get_workflow_run_options',
|
||||
'promote_to_live',
|
||||
'redeploy',
|
||||
'run_block',
|
||||
'set_block_enabled',
|
||||
'set_global_workflow_variables',
|
||||
])
|
||||
|
||||
export function resolveToolDisplayTitle(name: string, args?: Record<string, unknown>): string {
|
||||
// Cases that enrich the title with live workspace/block names from the client
|
||||
// stores. Everything else is resolved by the shared name+args resolver, which
|
||||
@@ -224,6 +287,39 @@ export function resolveToolDisplayTitle(name: string, args?: Record<string, unkn
|
||||
if (workflowName) return `Querying logs for ${workflowName}`
|
||||
}
|
||||
|
||||
// Workflow-scoped tools carry an id, not a name — and often not even that,
|
||||
// defaulting to the current workflow. Resolve the name here and hand it to
|
||||
// the shared resolver as `workflowName`, which every workflow title already
|
||||
// reads, so deployments, reads, and block work all say WHICH workflow.
|
||||
// Table tools keep their operands nested under `args`, and identify the
|
||||
// table by id — so the shared resolver sees neither the column being added
|
||||
// nor which table it belongs to. Lift both here.
|
||||
if (TABLE_SCOPED_TOOL_IDS.has(name)) {
|
||||
const nested = isRecordLike(args?.args) ? (args?.args as Record<string, unknown>) : undefined
|
||||
const tableName =
|
||||
stringParam(args?.tableName) ?? resolveTableNameForDisplay(nested?.tableId ?? args?.tableId)
|
||||
if (nested || tableName) {
|
||||
return getToolDisplayTitle(name, {
|
||||
...args,
|
||||
...(nested ?? {}),
|
||||
...(tableName ? { tableName } : {}),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if (WORKFLOW_SCOPED_TOOL_IDS.has(name) && !stringParam(args?.workflowName)) {
|
||||
const workflowName = resolveTargetWorkflowName(args)
|
||||
// Block-scoped tools carry a blockId for the same reason; resolve it too,
|
||||
// so a row says which block ran rather than an opaque id (or nothing).
|
||||
const blockName = stringParam(args?.blockName) ?? resolveBlockNameForDisplay(args?.blockId)
|
||||
const enriched = {
|
||||
...args,
|
||||
...(workflowName ? { workflowName } : {}),
|
||||
...(blockName ? { blockName } : {}),
|
||||
}
|
||||
if (workflowName || blockName) return getToolDisplayTitle(name, enriched)
|
||||
}
|
||||
|
||||
return getToolDisplayTitle(name, args)
|
||||
}
|
||||
|
||||
@@ -270,11 +366,11 @@ export function resolveStreamingToolDisplayTitle(
|
||||
name: string,
|
||||
streamingArgs: string
|
||||
): string | undefined {
|
||||
if (name === FunctionExecute.id) {
|
||||
if (name === RunFunction.id) {
|
||||
return functionExecuteTitle(matchStreamingStringArg(streamingArgs, 'title'))
|
||||
}
|
||||
|
||||
if (name === WorkspaceFile.id) {
|
||||
if (name === PrepareFileEdit.id) {
|
||||
return resolveWorkspaceFileDisplayTitle(
|
||||
matchStreamingStringArg(streamingArgs, 'operation'),
|
||||
matchStreamingStringArg(streamingArgs, 'title'),
|
||||
@@ -282,7 +378,7 @@ export function resolveStreamingToolDisplayTitle(
|
||||
)
|
||||
}
|
||||
|
||||
if (name === CreateFile.id) {
|
||||
if (name === CreateEmptyFile.id) {
|
||||
const target =
|
||||
matchStreamingStringArg(streamingArgs, 'path') ??
|
||||
matchStreamingStringArg(streamingArgs, 'fileName')
|
||||
@@ -299,7 +395,7 @@ export function resolveStreamingToolDisplayTitle(
|
||||
return workflowId ? resolveToolDisplayTitle(name, { workflowId }) : undefined
|
||||
}
|
||||
|
||||
if (name === SearchOnline.id) {
|
||||
if (name === WebSearch.id) {
|
||||
const toolTitle = matchStreamingStringArg(streamingArgs, 'toolTitle')
|
||||
return toolTitle ? `Searching online for ${toolTitle}` : undefined
|
||||
}
|
||||
@@ -349,12 +445,12 @@ export function resolveStreamingToolDisplayTitle(
|
||||
return toolTitle ? `Deleting ${toolTitle}` : undefined
|
||||
}
|
||||
|
||||
if (name === ScrapePage.id) {
|
||||
if (name === WebScrape.id) {
|
||||
const url = matchStreamingStringArg(streamingArgs, 'url')
|
||||
return url ? `Scraping ${url}` : undefined
|
||||
}
|
||||
|
||||
if (name === CrawlWebsite.id) {
|
||||
if (name === WebCrawl.id) {
|
||||
const url = matchStreamingStringArg(streamingArgs, 'url')
|
||||
return url ? `Crawling ${url}` : undefined
|
||||
}
|
||||
@@ -363,7 +459,7 @@ export function resolveStreamingToolDisplayTitle(
|
||||
return resolveStreamingManagedResourceTitle(name, streamingArgs, ['toolTitle', 'title', 'name'])
|
||||
}
|
||||
|
||||
if (name === ManageMcpTool.id) {
|
||||
if (name === ManageMcpConnection.id) {
|
||||
return resolveStreamingManagedResourceTitle(name, streamingArgs, [
|
||||
'serverName',
|
||||
'name',
|
||||
|
||||
+10
-10
@@ -68,7 +68,7 @@ describe('streaming resource titles', () => {
|
||||
})
|
||||
|
||||
// A main-agent file delegation: trigger tool (main lane), subagent span, inner
|
||||
// workspace_file, span end, delegation result.
|
||||
// prepare_file_edit, span end, delegation result.
|
||||
function fileDelegationEvents(): PersistedStreamEventEnvelope[] {
|
||||
const sub: Scope = {
|
||||
lane: 'subagent',
|
||||
@@ -89,13 +89,13 @@ function fileDelegationEvents(): PersistedStreamEventEnvelope[] {
|
||||
env(
|
||||
4,
|
||||
'tool',
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'workspace_file' },
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'prepare_file_edit' },
|
||||
{ lane: 'subagent', spanId: 'S1' }
|
||||
),
|
||||
env(
|
||||
5,
|
||||
'tool',
|
||||
{ phase: 'result', toolCallId: 'wf-1', toolName: 'workspace_file', success: true },
|
||||
{ phase: 'result', toolCallId: 'wf-1', toolName: 'prepare_file_edit', success: true },
|
||||
{ lane: 'subagent', spanId: 'S1' }
|
||||
),
|
||||
env(
|
||||
@@ -124,7 +124,7 @@ describe('modelToContentBlocks', () => {
|
||||
expect(trigger?.toolCall?.status).toBe('success')
|
||||
|
||||
const innerTool = blocksByType(blocks, 'tool_call').find(
|
||||
(b) => b.toolCall?.name === 'workspace_file'
|
||||
(b) => b.toolCall?.name === 'prepare_file_edit'
|
||||
)
|
||||
expect(innerTool?.spanId).toBe('S1')
|
||||
expect(innerTool?.toolCall?.calledBy).toBe('file')
|
||||
@@ -220,7 +220,7 @@ describe('modelToContentBlocks', () => {
|
||||
env(
|
||||
4,
|
||||
'tool',
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'workspace_file' },
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'prepare_file_edit' },
|
||||
{ lane: 'subagent', spanId: 'S1' }
|
||||
),
|
||||
env(
|
||||
@@ -233,7 +233,7 @@ describe('modelToContentBlocks', () => {
|
||||
])
|
||||
)
|
||||
const types = blocks.map((b) => b.type)
|
||||
const innerIdx = blocks.findIndex((b) => b.toolCall?.name === 'workspace_file')
|
||||
const innerIdx = blocks.findIndex((b) => b.toolCall?.name === 'prepare_file_edit')
|
||||
const endIdx = types.indexOf('subagent_end')
|
||||
const afterIdx = blocks.findIndex((b) => b.type === 'text' && b.content === 'after')
|
||||
// subagent_end sits after the inner work and before the trailing main text — no sibling jumps.
|
||||
@@ -275,7 +275,7 @@ describe('modelToContentBlocks', () => {
|
||||
env(
|
||||
3,
|
||||
'tool',
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'workspace_file' },
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'prepare_file_edit' },
|
||||
{ lane: 'subagent', spanId: 'S1' }
|
||||
),
|
||||
])
|
||||
@@ -309,7 +309,7 @@ describe('modelToContentBlocks', () => {
|
||||
env(1, 'tool', {
|
||||
phase: 'call',
|
||||
toolCallId: 'wf',
|
||||
toolName: 'workspace_file',
|
||||
toolName: 'prepare_file_edit',
|
||||
arguments: { operation: 'create', title: 'My Doc' },
|
||||
}),
|
||||
])
|
||||
@@ -322,7 +322,7 @@ describe('modelToContentBlocks', () => {
|
||||
const blocks = modelToContentBlocks(build(fileDelegationEvents()))
|
||||
const startIdx = blocks.findIndex((b) => b.type === 'subagent')
|
||||
const innerIdx = blocks.findIndex(
|
||||
(b) => b.type === 'tool_call' && b.toolCall?.name === 'workspace_file'
|
||||
(b) => b.type === 'tool_call' && b.toolCall?.name === 'prepare_file_edit'
|
||||
)
|
||||
const endIdx = blocks.findIndex((b) => b.type === 'subagent_end')
|
||||
expect(startIdx).toBeGreaterThanOrEqual(0)
|
||||
@@ -425,7 +425,7 @@ describe('contentBlocksToModel round-trip', () => {
|
||||
env(
|
||||
3,
|
||||
'tool',
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'workspace_file' },
|
||||
{ phase: 'call', toolCallId: 'wf-1', toolName: 'prepare_file_edit' },
|
||||
{ lane: 'subagent', spanId: 'S1' }
|
||||
),
|
||||
])
|
||||
|
||||
@@ -167,6 +167,7 @@ export function modelToContentBlocks(model: TurnModel): ContentBlock[] {
|
||||
block: {
|
||||
type: 'subagent',
|
||||
content: node.agentId,
|
||||
...(node.displayName ? { subagentName: node.displayName } : {}),
|
||||
spanId: node.spanId,
|
||||
parentSpanId: node.parentSpanId,
|
||||
...(node.triggerToolCallId ? { parentToolCallId: node.triggerToolCallId } : {}),
|
||||
@@ -267,7 +268,10 @@ export function contentBlocksToModel(blocks: ContentBlock[]): TurnModel {
|
||||
kind: 'subagent',
|
||||
event: 'start',
|
||||
agent: block.content,
|
||||
data: block.parentToolCallId ? { tool_call_id: block.parentToolCallId } : {},
|
||||
data: {
|
||||
...(block.parentToolCallId ? { tool_call_id: block.parentToolCallId } : {}),
|
||||
...(block.subagentName ? { name: block.subagentName } : {}),
|
||||
},
|
||||
},
|
||||
scopeFor(block),
|
||||
block.timestamp
|
||||
|
||||
@@ -144,17 +144,17 @@ describe('reduceEvent — tool lifecycle', () => {
|
||||
|
||||
it('accumulates streaming args across deltas', () => {
|
||||
const m = apply([
|
||||
toolCall(1, 'tc-1', 'workspace_file'),
|
||||
toolCall(1, 'tc-1', 'prepare_file_edit'),
|
||||
envelope(2, 'tool', {
|
||||
phase: 'args_delta',
|
||||
toolCallId: 'tc-1',
|
||||
toolName: 'workspace_file',
|
||||
toolName: 'prepare_file_edit',
|
||||
argumentsDelta: '{"a":',
|
||||
}),
|
||||
envelope(3, 'tool', {
|
||||
phase: 'args_delta',
|
||||
toolCallId: 'tc-1',
|
||||
toolName: 'workspace_file',
|
||||
toolName: 'prepare_file_edit',
|
||||
argumentsDelta: '1}',
|
||||
}),
|
||||
])
|
||||
@@ -164,11 +164,11 @@ describe('reduceEvent — tool lifecycle', () => {
|
||||
|
||||
it('clears streamingArgs once the result settles the tool', () => {
|
||||
const m = apply([
|
||||
toolCall(1, 'tc-1', 'workspace_file'),
|
||||
toolCall(1, 'tc-1', 'prepare_file_edit'),
|
||||
envelope(2, 'tool', {
|
||||
phase: 'args_delta',
|
||||
toolCallId: 'tc-1',
|
||||
toolName: 'workspace_file',
|
||||
toolName: 'prepare_file_edit',
|
||||
argumentsDelta: '{"operation":"create"',
|
||||
}),
|
||||
toolResult(3, 'tc-1', true),
|
||||
@@ -213,11 +213,11 @@ describe('reduceEvent — tool lifecycle', () => {
|
||||
|
||||
it('ignores preview phases (decoupled from tool status)', () => {
|
||||
const m = apply([
|
||||
toolCall(1, 'tc-1', 'workspace_file'),
|
||||
toolCall(1, 'tc-1', 'prepare_file_edit'),
|
||||
envelope(2, 'tool', {
|
||||
previewPhase: 'file_preview_content',
|
||||
toolCallId: 'tc-1',
|
||||
toolName: 'workspace_file',
|
||||
toolName: 'prepare_file_edit',
|
||||
content: 'x',
|
||||
contentMode: 'delta',
|
||||
fileName: 'f',
|
||||
@@ -237,6 +237,29 @@ describe('reduceEvent — subagent lifecycle', () => {
|
||||
expect(agent(m, 'S1').parentSpanId).toBe(MAIN_SPAN)
|
||||
})
|
||||
|
||||
it('captures the orchestrator-chosen display name from span start data', () => {
|
||||
const m = apply([
|
||||
envelope(
|
||||
1,
|
||||
'span',
|
||||
{
|
||||
kind: 'subagent',
|
||||
event: 'start',
|
||||
agent: 'research',
|
||||
data: { tool_call_id: 'tc-r', name: 'Pricing research' },
|
||||
},
|
||||
{
|
||||
lane: 'subagent',
|
||||
spanId: 'S1',
|
||||
parentSpanId: MAIN_SPAN,
|
||||
parentToolCallId: 'tc-r',
|
||||
agentId: 'research',
|
||||
}
|
||||
),
|
||||
])
|
||||
expect(agent(m, 'S1').displayName).toBe('Pricing research')
|
||||
})
|
||||
|
||||
it('settles an agent error when span end carries an error', () => {
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
@@ -270,8 +293,8 @@ describe('reduceEvent — subagent lifecycle', () => {
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-a'),
|
||||
spanStart(2, 'S2', 'file', 'tc-b'),
|
||||
toolCall(3, 'wf-a', 'workspace_file', { lane: 'subagent', spanId: 'S1' }),
|
||||
toolCall(4, 'wf-b', 'workspace_file', { lane: 'subagent', spanId: 'S2' }),
|
||||
toolCall(3, 'wf-a', 'prepare_file_edit', { lane: 'subagent', spanId: 'S1' }),
|
||||
toolCall(4, 'wf-b', 'prepare_file_edit', { lane: 'subagent', spanId: 'S2' }),
|
||||
toolResult(5, 'wf-a', true),
|
||||
spanEnd(6, 'S1', 'file'),
|
||||
toolResult(7, 'wf-b', true),
|
||||
@@ -327,7 +350,7 @@ describe('reduceEvent — idempotency', () => {
|
||||
it('rebuilds the identical model when replayed into a fresh model', () => {
|
||||
const events = [
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
toolCall(2, 'wf', 'workspace_file', { lane: 'subagent', spanId: 'S1' }),
|
||||
toolCall(2, 'wf', 'prepare_file_edit', { lane: 'subagent', spanId: 'S1' }),
|
||||
toolResult(3, 'wf', true),
|
||||
spanEnd(4, 'S1', 'file'),
|
||||
complete(5),
|
||||
@@ -340,42 +363,42 @@ describe('reduceEvent — idempotency', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('reduceEvent — edit_content row merge', () => {
|
||||
it('folds an edit_content write into its span workspace_file row', () => {
|
||||
describe('reduceEvent — apply_file_edit row merge', () => {
|
||||
it('folds an apply_file_edit write into its span prepare_file_edit row', () => {
|
||||
const sub: Scope = { lane: 'subagent', spanId: 'S1' }
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
toolCall(2, 'wf-1', 'workspace_file', sub),
|
||||
toolCall(2, 'wf-1', 'prepare_file_edit', sub),
|
||||
toolResult(3, 'wf-1', true, undefined, sub),
|
||||
toolCall(4, 'ec-1', 'edit_content', sub),
|
||||
toolCall(4, 'ec-1', 'apply_file_edit', sub),
|
||||
])
|
||||
// No separate edit_content node; the workspace_file row reopened for the edit.
|
||||
// No separate apply_file_edit node; the prepare_file_edit row reopened for the edit.
|
||||
expect(m.nodes.has('ec-1')).toBe(false)
|
||||
expect(tool(m, 'wf-1').status).toBe('running')
|
||||
expect(m.toolAlias.get('ec-1')).toBe('wf-1')
|
||||
})
|
||||
|
||||
it('settles the merged row on the edit_content result', () => {
|
||||
it('settles the merged row on the apply_file_edit result', () => {
|
||||
const sub: Scope = { lane: 'subagent', spanId: 'S1' }
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
toolCall(2, 'wf-1', 'workspace_file', sub),
|
||||
toolCall(3, 'ec-1', 'edit_content', sub),
|
||||
toolCall(2, 'wf-1', 'prepare_file_edit', sub),
|
||||
toolCall(3, 'ec-1', 'apply_file_edit', sub),
|
||||
toolResult(4, 'ec-1', true, undefined, sub),
|
||||
])
|
||||
expect(tool(m, 'wf-1').status).toBe('success')
|
||||
expect(m.nodes.has('ec-1')).toBe(false)
|
||||
})
|
||||
|
||||
it('folds an edit_content result that raced ahead of its call into the merged row', () => {
|
||||
it('folds an apply_file_edit result that raced ahead of its call into the merged row', () => {
|
||||
const sub: Scope = { lane: 'subagent', spanId: 'S1' }
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
toolCall(2, 'wf-1', 'workspace_file', sub),
|
||||
// Result for edit_content arrives BEFORE its call (buffered under ec-1)...
|
||||
toolCall(2, 'wf-1', 'prepare_file_edit', sub),
|
||||
// Result for apply_file_edit arrives BEFORE its call (buffered under ec-1)...
|
||||
toolResult(3, 'ec-1', true, undefined, sub),
|
||||
// ...then the call lands and aliases ec-1 -> wf-1, draining the buffer.
|
||||
toolCall(4, 'ec-1', 'edit_content', sub),
|
||||
toolCall(4, 'ec-1', 'apply_file_edit', sub),
|
||||
])
|
||||
expect(tool(m, 'wf-1').status).toBe('success')
|
||||
expect(tool(m, 'wf-1').result?.success).toBe(true)
|
||||
@@ -386,13 +409,13 @@ describe('reduceEvent — edit_content row merge', () => {
|
||||
const sub: Scope = { lane: 'subagent', spanId: 'S1' }
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
// Section 1: the workspace_file row is reopened by its edit_content, but the
|
||||
// Section 1: the prepare_file_edit row is reopened by its apply_file_edit, but the
|
||||
// edit's closing result is reordered/dropped — wf-1 is left running.
|
||||
toolCall(2, 'wf-1', 'workspace_file', sub),
|
||||
toolCall(2, 'wf-1', 'prepare_file_edit', sub),
|
||||
toolResult(3, 'wf-1', true, undefined, sub),
|
||||
toolCall(4, 'ec-1', 'edit_content', sub),
|
||||
toolCall(4, 'ec-1', 'apply_file_edit', sub),
|
||||
// Section 2 opens before section 1's edit result lands.
|
||||
toolCall(5, 'wf-2', 'workspace_file', sub),
|
||||
toolCall(5, 'wf-2', 'prepare_file_edit', sub),
|
||||
])
|
||||
// The previous section settles instead of spinning until the turn terminal...
|
||||
expect(tool(m, 'wf-1').status).toBe('success')
|
||||
@@ -498,7 +521,7 @@ describe('turn-terminal propagation', () => {
|
||||
// A file subagent opened but no span end arrived (mid-stream error/disconnect).
|
||||
const m = apply([
|
||||
spanStart(1, 'S1', 'file', 'tc-file'),
|
||||
toolCall(2, 'wf-1', 'workspace_file', { lane: 'subagent', spanId: 'S1' }),
|
||||
toolCall(2, 'wf-1', 'prepare_file_edit', { lane: 'subagent', spanId: 'S1' }),
|
||||
])
|
||||
expect(agent(m, 'S1').endSeq).toBeUndefined()
|
||||
applyTurnTerminal(m, 'error')
|
||||
@@ -556,3 +579,93 @@ describe('reduceEvent — span-start owner reconciliation', () => {
|
||||
expect((lane as AgentNode).agentId).toBe('workflow')
|
||||
})
|
||||
})
|
||||
|
||||
describe('reduceEvent — span end settles stale lane tools', () => {
|
||||
const laneScope = { lane: 'subagent', spanId: 'S1', parentToolCallId: 'd1' } as Scope
|
||||
|
||||
it('marks still-running tools success when their lane ends cleanly', () => {
|
||||
const model = apply([
|
||||
envelope(
|
||||
1,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'start', agent: 'browser', data: { tool_call_id: 'd1' } },
|
||||
laneScope
|
||||
),
|
||||
toolCall(2, 'click-1', 'browser_click', laneScope),
|
||||
// No result for click-1 — dropped/reordered past the lane end.
|
||||
envelope(
|
||||
3,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'end', agent: 'browser', data: {} },
|
||||
laneScope
|
||||
),
|
||||
])
|
||||
|
||||
const click = model.nodes.get('click-1')
|
||||
if (click?.kind !== 'tool') throw new Error('expected tool node')
|
||||
expect(click.status).toBe('success')
|
||||
|
||||
const laneId = model.agentBySpanId.get('S1')
|
||||
const lane = laneId ? model.nodes.get(laneId) : undefined
|
||||
if (lane?.kind !== 'agent') throw new Error('expected agent lane')
|
||||
expect(lane.status).toBe('success')
|
||||
})
|
||||
|
||||
it('marks still-running tools error when the lane ends with an error', () => {
|
||||
const model = apply([
|
||||
envelope(
|
||||
1,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'start', agent: 'browser', data: { tool_call_id: 'd1' } },
|
||||
laneScope
|
||||
),
|
||||
toolCall(2, 'click-1', 'browser_click', laneScope),
|
||||
envelope(
|
||||
3,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'end', agent: 'browser', data: { error: 'boom' } },
|
||||
laneScope
|
||||
),
|
||||
])
|
||||
|
||||
const click = model.nodes.get('click-1')
|
||||
if (click?.kind !== 'tool') throw new Error('expected tool node')
|
||||
expect(click.status).toBe('error')
|
||||
})
|
||||
|
||||
it('leaves settled tools alone and lets a late result overwrite the settle', () => {
|
||||
const model = apply([
|
||||
envelope(
|
||||
1,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'start', agent: 'browser', data: { tool_call_id: 'd1' } },
|
||||
laneScope
|
||||
),
|
||||
toolCall(2, 'click-1', 'browser_click', laneScope),
|
||||
envelope(
|
||||
3,
|
||||
'span',
|
||||
{ kind: 'subagent', event: 'end', agent: 'browser', data: {} },
|
||||
laneScope
|
||||
),
|
||||
// Late result arrives after the settle — it must win.
|
||||
envelope(
|
||||
4,
|
||||
'tool',
|
||||
{
|
||||
phase: 'result',
|
||||
toolCallId: 'click-1',
|
||||
toolName: 'browser_click',
|
||||
success: false,
|
||||
error: 'nope',
|
||||
},
|
||||
laneScope
|
||||
),
|
||||
])
|
||||
|
||||
const click = model.nodes.get('click-1')
|
||||
if (click?.kind !== 'tool') throw new Error('expected tool node')
|
||||
expect(click.status).toBe('error')
|
||||
expect(click.result?.error).toBe('nope')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -85,6 +85,8 @@ export interface AgentNode extends NodeBase {
|
||||
agentId: string
|
||||
/** The outer delegation tool_use that triggered this run; links the trigger tool node. */
|
||||
triggerToolCallId?: string
|
||||
/** Orchestrator-chosen display name for this delegation (falls back to the agent label). */
|
||||
displayName?: string
|
||||
status: NodeStatus
|
||||
/** Wire seq at which the run terminated (span end), for ordering the close marker. */
|
||||
endSeq?: number
|
||||
@@ -121,7 +123,7 @@ export interface TurnModel {
|
||||
>
|
||||
/**
|
||||
* Maps a tool call id to another tool node it folds into. Used for the
|
||||
* `edit_content` -> `workspace_file` row merge so the write streams into the
|
||||
* `apply_file_edit` -> `prepare_file_edit` row merge so the write streams into the
|
||||
* single "writing" row rather than a second row.
|
||||
*/
|
||||
toolAlias: Map<string, string>
|
||||
@@ -142,16 +144,16 @@ export function createTurnModel(): TurnModel {
|
||||
}
|
||||
}
|
||||
|
||||
const WORKSPACE_FILE_TOOL = 'workspace_file'
|
||||
const EDIT_CONTENT_TOOL = 'edit_content'
|
||||
const WORKSPACE_FILE_TOOL = 'prepare_file_edit'
|
||||
const EDIT_CONTENT_TOOL = 'apply_file_edit'
|
||||
|
||||
/** Resolves a tool call id through the alias map (e.g. edit_content -> its workspace_file row). */
|
||||
/** Resolves a tool call id through the alias map (e.g. apply_file_edit -> its prepare_file_edit row). */
|
||||
export function resolveToolId(model: TurnModel, id: string): string {
|
||||
return model.toolAlias.get(id) ?? id
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds the most recent `workspace_file` tool node in a span so an `edit_content`
|
||||
* Finds the most recent `prepare_file_edit` tool node in a span so an `apply_file_edit`
|
||||
* write folds into it (the single "writing" row). Co-location in the file
|
||||
* subagent's span is the link — no coupling to preview phases. The caller
|
||||
* reopens whatever this returns, including an already-settled row (an edit after
|
||||
@@ -169,11 +171,11 @@ function findWorkspaceFileNodeInSpan(model: TurnModel, spanId: string): ToolNode
|
||||
}
|
||||
|
||||
/**
|
||||
* The file agent writes a file as strictly sequential `workspace_file` +
|
||||
* `edit_content` section pairs, waiting for each to finish before the next. So
|
||||
* when a new section's `workspace_file` opens, any earlier `workspace_file` row
|
||||
* The file agent writes a file as strictly sequential `prepare_file_edit` +
|
||||
* `apply_file_edit` section pairs, waiting for each to finish before the next. So
|
||||
* when a new section's `prepare_file_edit` opens, any earlier `prepare_file_edit` row
|
||||
* still `running` in the same span is a completed section whose closing
|
||||
* `edit_content` result was reordered or dropped — finalize it as success so its
|
||||
* `apply_file_edit` result was reordered or dropped — finalize it as success so its
|
||||
* "writing" spinner resolves when the next section starts, instead of lingering
|
||||
* until the turn-terminal sweep. A no-op on the happy path (prior rows already
|
||||
* settled on their own result).
|
||||
@@ -331,8 +333,8 @@ function appendText(
|
||||
/**
|
||||
* Applies a result that raced ahead of its tool `call` (buffered under `fromId`)
|
||||
* onto `node`, then clears the buffer. Used by the normal call path and by the
|
||||
* edit_content -> workspace_file merge, where the buffer is keyed by the
|
||||
* edit_content id but folds into the workspace_file row.
|
||||
* apply_file_edit -> prepare_file_edit merge, where the buffer is keyed by the
|
||||
* apply_file_edit id but folds into the prepare_file_edit row.
|
||||
*/
|
||||
function drainBufferedResult(model: TurnModel, fromId: string, node: ToolNode): void {
|
||||
const buffered = model.bufferedResults.get(fromId)
|
||||
@@ -473,12 +475,12 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
ensureSubagentLane(model, spanId, scope, seq, tsMs)
|
||||
const phase = payload.phase
|
||||
if (phase === MothershipStreamV1ToolPhase.call) {
|
||||
// edit_content folds into its span's workspace_file row (the write
|
||||
// apply_file_edit folds into its span's prepare_file_edit row (the write
|
||||
// continues in the single "writing" row), reopening it for the edit.
|
||||
if (toolName === EDIT_CONTENT_TOOL) {
|
||||
// A re-emitted edit_content call (same tool call id — duplicate/replay)
|
||||
// A re-emitted apply_file_edit call (same tool call id — duplicate/replay)
|
||||
// must keep its ORIGINAL target row. Re-running the span lookup can
|
||||
// return a newer workspace_file, and folding into that would leave the
|
||||
// return a newer prepare_file_edit, and folding into that would leave the
|
||||
// first (already reopened) row running with no result ever closing it —
|
||||
// a spinner stuck until the turn-terminal sweep. So once aliased, reuse.
|
||||
const aliasedId = model.toolAlias.get(rawToolCallId)
|
||||
@@ -492,7 +494,7 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
parent.status = 'running'
|
||||
parent.result = undefined
|
||||
// A result that raced ahead of this call was buffered under the
|
||||
// edit_content id; fold it into the reopened workspace_file row.
|
||||
// apply_file_edit id; fold it into the reopened prepare_file_edit row.
|
||||
drainBufferedResult(model, rawToolCallId, parent)
|
||||
break
|
||||
}
|
||||
@@ -563,6 +565,7 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
const triggerToolCallId =
|
||||
scope?.parentToolCallId ?? asString(data?.tool_call_id) ?? asString(data?.toolCallId)
|
||||
const agentId = asString(payload.agent) ?? scope?.agentId ?? ''
|
||||
const displayName = asString(data?.name)
|
||||
const resolvedSpanId =
|
||||
scope?.spanId ?? (triggerToolCallId ? `span:${triggerToolCallId}` : `span:${seq}`)
|
||||
const parentSpanId = scope?.parentSpanId ?? MAIN_SPAN
|
||||
@@ -581,6 +584,7 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
// scope.agentId can name the forwarding caller (e.g. superagent),
|
||||
// while this start's payload.agent is the authoritative lane owner.
|
||||
if (agentId && existing.agentId !== agentId) existing.agentId = agentId
|
||||
if (displayName) existing.displayName = displayName
|
||||
if (!existing.triggerToolCallId && triggerToolCallId) {
|
||||
existing.triggerToolCallId = triggerToolCallId
|
||||
}
|
||||
@@ -602,6 +606,7 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
seq: seq,
|
||||
...(tsMs !== undefined ? { startedAtMs: tsMs } : {}),
|
||||
...(triggerToolCallId ? { triggerToolCallId } : {}),
|
||||
...(displayName ? { displayName } : {}),
|
||||
}
|
||||
model.nodes.set(node.id, node)
|
||||
model.order.push(node.id)
|
||||
@@ -611,10 +616,25 @@ export function reduceEvent(model: TurnModel, envelope: PersistedStreamEventEnve
|
||||
if (data?.pending === true) break
|
||||
breakLane(model, resolvedSpanId, tsMs)
|
||||
const node = model.nodes.get(resolvedSpanId)
|
||||
const spanErrored = Boolean(data && asString(data.error))
|
||||
if (node && node.kind === 'agent' && !isNodeTerminal(node.status)) {
|
||||
node.status = data && asString(data.error) ? 'error' : 'success'
|
||||
node.status = spanErrored ? 'error' : 'success'
|
||||
node.endSeq = seq
|
||||
}
|
||||
// The lane is over: settle any tool row still `running` in it (its
|
||||
// result was dropped or reordered past the end). Left open, the row
|
||||
// pins the whole group expanded and shimmering for the rest of the
|
||||
// turn even though the subagent already returned. A late result event
|
||||
// still corrects this — applyToolResult overwrites unconditionally.
|
||||
for (const id of model.order) {
|
||||
const stale = model.nodes.get(id)
|
||||
if (stale?.kind === 'tool' && stale.spanId === resolvedSpanId) {
|
||||
if (stale.status === 'running') {
|
||||
stale.status = spanErrored ? 'error' : 'success'
|
||||
stale.streamingArgs = undefined
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
@@ -12,7 +12,10 @@ import {
|
||||
getReplayCompletedWorkflowToolCallIds,
|
||||
panelForExecutingClientTool,
|
||||
reconcileLiveAssistantTurn,
|
||||
selectDeletedWorkflowResources,
|
||||
selectReconnectReplayState,
|
||||
shouldActivateResourceEvent,
|
||||
shouldQueueOutgoingMessage,
|
||||
waitForDetachedChatResolution,
|
||||
} from '@/app/workspace/[workspaceId]/home/hooks/use-chat'
|
||||
import type {
|
||||
@@ -30,6 +33,76 @@ vi.mock('next/navigation', () => ({
|
||||
}),
|
||||
}))
|
||||
|
||||
describe('selectDeletedWorkflowResources', () => {
|
||||
const resource = (id: string) => ({ type: 'workflow' as const, id, title: id })
|
||||
const cached = (id: string) => ({
|
||||
id,
|
||||
name: id,
|
||||
lastModified: new Date(0),
|
||||
createdAt: new Date(0),
|
||||
sortOrder: 0,
|
||||
})
|
||||
|
||||
it('selects a hydrated workflow the server no longer has', () => {
|
||||
expect(selectDeletedWorkflowResources([resource('wf-gone')], new Set(), [])).toEqual([
|
||||
resource('wf-gone'),
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps a workflow present in the fetched list', () => {
|
||||
expect(selectDeletedWorkflowResources([resource('wf-1')], new Set(['wf-1']), [])).toEqual([])
|
||||
})
|
||||
|
||||
it('keeps a workflow the stream inserted into the cache after the list snapshot', () => {
|
||||
expect(
|
||||
selectDeletedWorkflowResources([resource('wf-new')], new Set(), [cached('wf-new')])
|
||||
).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('shouldActivateResourceEvent', () => {
|
||||
it('surfaces browser work even when another resource is selected', () => {
|
||||
expect(shouldActivateResourceEvent('file-1', 'browser-session')).toBe(true)
|
||||
})
|
||||
|
||||
it('surfaces every other resource the agent touches', () => {
|
||||
expect(shouldActivateResourceEvent('file-1', 'workflow-1')).toBe(true)
|
||||
expect(shouldActivateResourceEvent('browser-session', 'terminal-session')).toBe(true)
|
||||
expect(shouldActivateResourceEvent(null, 'browser-session')).toBe(true)
|
||||
})
|
||||
|
||||
it('honors an explicit request to activate', () => {
|
||||
expect(shouldActivateResourceEvent('file-1', 'browser-session', { activate: true })).toBe(true)
|
||||
})
|
||||
|
||||
it('lets an event opt out of stealing focus', () => {
|
||||
expect(shouldActivateResourceEvent('file-1', 'browser-session', { activate: false })).toBe(
|
||||
false
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('shouldQueueOutgoingMessage', () => {
|
||||
it('queues while a send is in flight', () => {
|
||||
expect(shouldQueueOutgoingMessage(true, false, 0)).toBe(true)
|
||||
})
|
||||
|
||||
it('queues while a stop is still settling', () => {
|
||||
expect(shouldQueueOutgoingMessage(false, true, 0)).toBe(true)
|
||||
})
|
||||
|
||||
it('queues behind messages still waiting after the turn ended', () => {
|
||||
// The regression: a message queued mid-stream must dispatch before one
|
||||
// typed in the idle gap after the turn stopped — a direct send here would
|
||||
// jump the queue and swap the user's message order.
|
||||
expect(shouldQueueOutgoingMessage(false, false, 1)).toBe(true)
|
||||
})
|
||||
|
||||
it('sends directly on an idle chat with an empty queue', () => {
|
||||
expect(shouldQueueOutgoingMessage(false, false, 0)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
function userMessage(id: string): PersistedMessage {
|
||||
return {
|
||||
id,
|
||||
|
||||
@@ -30,7 +30,7 @@ import { onOpenInBrowserPanel } from '@/lib/browser-agent/open-in-panel'
|
||||
import {
|
||||
cancelActiveBrowserTools,
|
||||
initBrowserAgentTransport,
|
||||
sendBrowserPanelAction,
|
||||
openUrlInNewBrowserTab,
|
||||
} from '@/lib/browser-agent/transport'
|
||||
import { getMothershipAttachmentPreviewUrl } from '@/lib/copilot/chat/attachment-preview'
|
||||
import { toDisplayMessage } from '@/lib/copilot/chat/display-message'
|
||||
@@ -126,8 +126,10 @@ import { getFolderMap } from '@/hooks/queries/utils/folder-cache'
|
||||
import { invalidateWorkflowSelectors } from '@/hooks/queries/utils/invalidate-workflow-lists'
|
||||
import { getTopInsertionSortOrder } from '@/hooks/queries/utils/top-insertion-sort-order'
|
||||
import { getWorkflowById, getWorkflows } from '@/hooks/queries/utils/workflow-cache'
|
||||
import { getWorkflowListQueryOptions } from '@/hooks/queries/utils/workflow-list-query'
|
||||
import { workflowKeys } from '@/hooks/queries/workflows'
|
||||
import { useExecutionStream } from '@/hooks/use-execution-stream'
|
||||
import { snapAllSmoothText } from '@/hooks/use-smooth-text'
|
||||
import { useExecutionStore } from '@/stores/execution/store'
|
||||
import { useMothershipQueueStore } from '@/stores/mothership-queue/store'
|
||||
import type {
|
||||
@@ -228,6 +230,7 @@ const RECONNECT_TAIL_ERROR =
|
||||
const MAX_RECONNECT_ATTEMPTS = 10
|
||||
const RECONNECT_BASE_DELAY_MS = 1000
|
||||
const RECONNECT_MAX_DELAY_MS = 30_000
|
||||
const RECONNECT_EXHAUSTED_RECHECK_MS = 30_000
|
||||
const STREAM_BATCH_FETCH_TIMEOUT_MS = 10_000
|
||||
const STREAM_CHAT_ID_RESOLVE_TIMEOUT_MS = 10_000
|
||||
const CHAT_HISTORY_RECOVERY_TIMEOUT_MS = 10_000
|
||||
@@ -1188,8 +1191,65 @@ function ensureWorkflowInRegistry(resourceId: string, title: string, workspaceId
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Hydrated workflow resources whose workflow exists neither in the fetched
|
||||
* server list nor in the local cache. The cache term protects a workflow the
|
||||
* agent created after the list snapshot was taken — the stream's registry
|
||||
* insert lands it in the cache before any refetch does.
|
||||
*/
|
||||
export function selectDeletedWorkflowResources(
|
||||
workflowResources: MothershipResource[],
|
||||
fetchedWorkflowIds: ReadonlySet<string>,
|
||||
cachedWorkflows: readonly WorkflowMetadata[]
|
||||
): MothershipResource[] {
|
||||
const cachedIds = new Set(cachedWorkflows.map((workflow) => workflow.id))
|
||||
return workflowResources.filter(
|
||||
(resource) => !fetchedWorkflowIds.has(resource.id) && !cachedIds.has(resource.id)
|
||||
)
|
||||
}
|
||||
|
||||
export interface ResourceEventOptions {
|
||||
activate?: boolean
|
||||
}
|
||||
|
||||
export type ResourceEventHandler = (resourceId: string, options?: ResourceEventOptions) => void
|
||||
|
||||
/**
|
||||
* Whether a streamed resource event should activate its tab. The panel always
|
||||
* follows the agent: whatever it is creating, editing, or driving becomes the
|
||||
* visible resource, browser sessions included. The parameters are retained so
|
||||
* callers stay explicit about the resource in play, and so a future opt-out
|
||||
* (an event that deliberately declines focus) has a place to live.
|
||||
*/
|
||||
export function shouldActivateResourceEvent(
|
||||
_activeResourceId: string | null,
|
||||
_resourceId: string,
|
||||
options?: ResourceEventOptions
|
||||
): boolean {
|
||||
return options?.activate !== false
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a fresh outbound message must join the chat's send queue instead of
|
||||
* dispatching directly. Queueing while a send or stop is in flight is the
|
||||
* obvious half; the queued-ahead term preserves FIFO across the
|
||||
* streaming→idle boundary — a message queued while the previous turn streamed
|
||||
* must reach the model before one typed after that turn ended but before the
|
||||
* queue drained. Without it the fresh send jumps the queue and both the
|
||||
* transcript and the model see the user's messages in swapped order. The two
|
||||
* signals never gap mid-dispatch: a queued message stays in the queue until
|
||||
* its optimistic send applies, which is after the in-flight flag is set.
|
||||
*/
|
||||
export function shouldQueueOutgoingMessage(
|
||||
sendInFlight: boolean,
|
||||
stopPending: boolean,
|
||||
queuedAheadCount: number
|
||||
): boolean {
|
||||
return sendInFlight || stopPending || queuedAheadCount > 0
|
||||
}
|
||||
|
||||
export interface UseChatOptions {
|
||||
onResourceEvent?: (resourceId: string) => void
|
||||
onResourceEvent?: ResourceEventHandler
|
||||
apiPath?: string
|
||||
stopPath?: string
|
||||
workflowId?: string
|
||||
@@ -1437,6 +1497,10 @@ export function useChat(
|
||||
() => {}
|
||||
)
|
||||
const recoveringQueuedSendHandoffRef = useRef<ActiveQueuedSendHandoffRecovery | null>(null)
|
||||
const recoverActiveStreamRef = useRef<
|
||||
(reason: 'pageshow' | 'visible' | 'online' | 'exhausted_recheck') => Promise<void>
|
||||
>(async () => {})
|
||||
const reconnectExhaustedRecheckTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null)
|
||||
|
||||
const abortControllerRef = useRef<AbortController | null>(null)
|
||||
const detachedChatResolutionControllersRef = useRef<Set<AbortController>>(new Set())
|
||||
@@ -1814,6 +1878,37 @@ export function useChat(
|
||||
}
|
||||
}, [])
|
||||
|
||||
/**
|
||||
* Drops hydrated workflow tabs whose workflow no longer exists, so an old
|
||||
* chat cannot resurrect a deleted workflow. The check is against a fetched
|
||||
* workflow list rather than the cache: seeding the registry from the chat's
|
||||
* persisted resources (what hydration previously did unconditionally) put
|
||||
* phantom entries in the sidebar that 404 on click. Removal also deletes the
|
||||
* resource from the chat's persisted set, so the tab stays gone next open.
|
||||
*/
|
||||
const reconcileHydratedWorkflowResources = useCallback(
|
||||
async (chatId: string, workflowResources: MothershipResource[]) => {
|
||||
let existing: WorkflowMetadata[]
|
||||
try {
|
||||
existing = await getQueryClient().fetchQuery(getWorkflowListQueryOptions(workspaceId))
|
||||
} catch {
|
||||
// Existence is unknowable right now; keep the tabs rather than delete
|
||||
// resources on a network failure. The next hydration retries.
|
||||
return
|
||||
}
|
||||
const deleted = selectDeletedWorkflowResources(
|
||||
workflowResources,
|
||||
new Set(existing.map((workflow) => workflow.id)),
|
||||
getWorkflows(workspaceId)
|
||||
)
|
||||
for (const resource of deleted) {
|
||||
if ((chatIdRef.current ?? selectedChatIdRef.current) !== chatId) return
|
||||
removeResource('workflow', resource.id)
|
||||
}
|
||||
},
|
||||
[workspaceId, removeResource]
|
||||
)
|
||||
|
||||
const reorderResources = useCallback((newOrder: MothershipResource[]) => {
|
||||
setResources(newOrder)
|
||||
const persistChatId = chatIdRef.current ?? selectedChatIdRef.current
|
||||
@@ -1924,12 +2019,14 @@ export function useChat(
|
||||
)
|
||||
|
||||
const openBrowserResource = useCallback(() => {
|
||||
// Browser work surfaces like any other agent activity: the panel follows
|
||||
// the agent to the browser whether or not the session was already open.
|
||||
addResource({
|
||||
type: 'browser',
|
||||
id: BROWSER_SESSION_RESOURCE_ID,
|
||||
title: 'Browser',
|
||||
})
|
||||
onResourceEventRef.current?.(BROWSER_SESSION_RESOURCE_ID)
|
||||
onResourceEventRef.current?.(BROWSER_SESSION_RESOURCE_ID, { activate: true })
|
||||
}, [addResource])
|
||||
|
||||
const getResourceActivityTracker = useCallback(
|
||||
@@ -2047,7 +2144,11 @@ export function useChat(
|
||||
useEffect(() => {
|
||||
return onOpenInBrowserPanel((url) => {
|
||||
openBrowserResource()
|
||||
sendBrowserPanelAction('navigate', { url }, desktopScopeIdRef.current)
|
||||
void openUrlInNewBrowserTab(url, desktopScopeIdRef.current).catch((error) => {
|
||||
logger.warn('Failed to open chat link in a new browser tab', {
|
||||
error: getErrorMessage(error),
|
||||
})
|
||||
})
|
||||
})
|
||||
}, [openBrowserResource])
|
||||
|
||||
@@ -2355,9 +2456,12 @@ export function useChat(
|
||||
setActiveResourceId(hydratedActiveResourceId)
|
||||
}
|
||||
|
||||
for (const resource of persistedResources) {
|
||||
if (resource.type !== 'workflow') continue
|
||||
ensureWorkflowInRegistry(resource.id, resource.title, workspaceId)
|
||||
// Restored workflow tabs are verified against the server instead of
|
||||
// seeded into the registry: a chat can outlive its workflows, and
|
||||
// fabricating entries for deleted ones polluted the sidebar.
|
||||
const workflowResources = persistedResources.filter((r) => r.type === 'workflow')
|
||||
if (workflowResources.length > 0) {
|
||||
void reconcileHydratedWorkflowResources(chatHistory.id, workflowResources)
|
||||
}
|
||||
} else if (hasPersistedStreamingFile) {
|
||||
activeResourceIdRef.current = null
|
||||
@@ -2442,6 +2546,7 @@ export function useChat(
|
||||
flushPendingResources,
|
||||
openBrowserResource,
|
||||
openTerminalResource,
|
||||
reconcileHydratedWorkflowResources,
|
||||
recoverPendingClientWorkflowTools,
|
||||
seedPreviewSessions,
|
||||
setTransportIdle,
|
||||
@@ -3199,7 +3304,29 @@ export function useChat(
|
||||
maxAttempts: MAX_RECONNECT_ATTEMPTS,
|
||||
})
|
||||
if (streamGenRef.current === gen) {
|
||||
/**
|
||||
* Never give up silently: surface the failure so the pane shows why
|
||||
* the live stream stopped instead of a torn-down transcript. Callers
|
||||
* own the finalize on a false return (every call site finalizes with
|
||||
* error: true), which refetches the persisted transcript; if the
|
||||
* server turn is still running, the visibility/online recovery path
|
||||
* re-attaches on the next pageshow/visible/online event.
|
||||
*/
|
||||
setIsReconnecting(false)
|
||||
setError(RECONNECT_TAIL_ERROR)
|
||||
/**
|
||||
* The tab may stay visible (no pageshow/visible/online event will ever
|
||||
* fire) while the server turn keeps running detached. One bounded
|
||||
* recheck re-enters recovery once the transient network condition has
|
||||
* had time to clear; recovery itself no-ops when nothing is active.
|
||||
*/
|
||||
if (reconnectExhaustedRecheckTimerRef.current) {
|
||||
clearTimeout(reconnectExhaustedRecheckTimerRef.current)
|
||||
}
|
||||
reconnectExhaustedRecheckTimerRef.current = setTimeout(() => {
|
||||
reconnectExhaustedRecheckTimerRef.current = null
|
||||
void recoverActiveStreamRef.current('exhausted_recheck')
|
||||
}, RECONNECT_EXHAUSTED_RECHECK_MS)
|
||||
}
|
||||
return false
|
||||
},
|
||||
@@ -3208,7 +3335,7 @@ export function useChat(
|
||||
retryReconnectRef.current = retryReconnect
|
||||
|
||||
const recoverActiveStreamFromRedis = useCallback(
|
||||
async (reason: 'pageshow' | 'visible' | 'online'): Promise<void> => {
|
||||
async (reason: 'pageshow' | 'visible' | 'online' | 'exhausted_recheck'): Promise<void> => {
|
||||
const startingChatId = chatIdRef.current
|
||||
const startingSelectedChatId = selectedChatIdRef.current
|
||||
const chatId = startingChatId ?? startingSelectedChatId
|
||||
@@ -3343,6 +3470,7 @@ export function useChat(
|
||||
},
|
||||
[getActiveStreamIdForChat, queryClient, resumeOrFinalize, setTransportReconnecting]
|
||||
)
|
||||
recoverActiveStreamRef.current = recoverActiveStreamFromRedis
|
||||
|
||||
useEffect(() => {
|
||||
if (typeof window === 'undefined' || typeof document === 'undefined') return
|
||||
@@ -3374,6 +3502,10 @@ export function useChat(
|
||||
document.removeEventListener('visibilitychange', handleVisibilityChange)
|
||||
window.removeEventListener('pageshow', handlePageShow)
|
||||
window.removeEventListener('online', handleOnline)
|
||||
if (reconnectExhaustedRecheckTimerRef.current) {
|
||||
clearTimeout(reconnectExhaustedRecheckTimerRef.current)
|
||||
reconnectExhaustedRecheckTimerRef.current = null
|
||||
}
|
||||
}
|
||||
}, [recoverActiveStreamFromRedis])
|
||||
|
||||
@@ -4164,12 +4296,23 @@ export function useChat(
|
||||
|
||||
// An in-flight send drains the queue from `finalize`; a pending stop kicks
|
||||
// the dispatcher itself, since nothing else will once the stop settles.
|
||||
if (sendingRef.current || pendingStopPromiseRef.current) {
|
||||
// A non-empty queue forces queueing even on an idle chat: messages
|
||||
// queued while the previous turn streamed must go out first, so a fresh
|
||||
// send lands behind them instead of jumping the line in the drain gap
|
||||
// after a turn ends.
|
||||
const queuedAheadCount = (queueStore.queues[activeChatKey] ?? EMPTY_MESSAGE_QUEUE).length
|
||||
if (
|
||||
shouldQueueOutgoingMessage(
|
||||
Boolean(sendingRef.current),
|
||||
Boolean(pendingStopPromiseRef.current),
|
||||
queuedAheadCount
|
||||
)
|
||||
) {
|
||||
queueStore.enqueue(
|
||||
activeChatKey,
|
||||
createQueuedMessage(message, fileAttachments, contexts, options?.resumeUserMessageId)
|
||||
)
|
||||
if (pendingStopPromiseRef.current) {
|
||||
if (pendingStopPromiseRef.current || (queuedAheadCount > 0 && !sendingRef.current)) {
|
||||
void enqueueQueueDispatchRef.current({ type: 'send_head' })
|
||||
}
|
||||
return
|
||||
@@ -4518,6 +4661,9 @@ export function useChat(
|
||||
abortControllerRef.current?.abort('user_stop:client_stopGeneration')
|
||||
abortControllerRef.current = null
|
||||
setTransportIdle()
|
||||
// The paced reveal may still hold up to a drain-horizon of buffered text;
|
||||
// after an explicit Stop it must not keep typing itself out.
|
||||
snapAllSmoothText()
|
||||
|
||||
try {
|
||||
if (activeChatId) {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import type { ChatContext } from '@/stores/panel'
|
||||
import type { BrowserTextSelection, TerminalTextSelection } from '@/stores/panel/types'
|
||||
|
||||
const EDIT_CONTENT_TOOL_ID = 'edit_content'
|
||||
const EDIT_CONTENT_TOOL_ID = 'apply_file_edit'
|
||||
const RUN_SUBAGENT_ID = 'run'
|
||||
|
||||
export type {
|
||||
@@ -114,6 +114,8 @@ export interface ContentBlock {
|
||||
type: ContentBlockType
|
||||
content?: string
|
||||
subagent?: string
|
||||
/** Orchestrator-chosen display name for a `subagent` start block (shown instead of the generic agent label). */
|
||||
subagentName?: string
|
||||
toolCall?: ToolCallInfo
|
||||
options?: OptionItem[]
|
||||
timestamp?: number
|
||||
@@ -189,9 +191,13 @@ export const SUBAGENT_LABELS: Record<string, string> = {
|
||||
custom_tool: 'Custom Tool Agent',
|
||||
scout: 'Scout Agent',
|
||||
search: 'Search Agent',
|
||||
platform: 'Platform Agent',
|
||||
superagent: 'Superagent',
|
||||
run: 'Run Agent',
|
||||
agent: 'Tools Agent',
|
||||
// The extensions subagent's wire/scope AgentID stays `agent` (pre-rename);
|
||||
// `extensions` is its current model-facing trigger tool name.
|
||||
agent: 'Extensions Agent',
|
||||
extensions: 'Extensions Agent',
|
||||
// `job` retained as a backward-compat alias so historical transcripts still render a label.
|
||||
job: 'Job Agent',
|
||||
file: 'File Agent',
|
||||
|
||||
+4
-7
@@ -29,6 +29,7 @@ import {
|
||||
SLACK_CAPABILITIES,
|
||||
SLACK_MANAGED_USER_AUTHORIZATION_CAPABILITY,
|
||||
} from '@/triggers/slack/capabilities'
|
||||
import { buildSlackCustomBotRequestUrl } from '@/triggers/webhook-url'
|
||||
|
||||
const logger = createLogger('ConnectSlackBotModal')
|
||||
|
||||
@@ -119,13 +120,9 @@ export function ConnectSlackBotModal({
|
||||
}
|
||||
}, [open, created, isReconnect, initialDisplayName, initialDescription])
|
||||
|
||||
// NEXT_PUBLIC_APP_URL, not window.location.origin: Slack's servers must be
|
||||
// able to reach this URL, so it has to be the app's public base (e.g. the
|
||||
// tunnel host in dev), not whatever host the browser happens to be on.
|
||||
const requestUrl = useMemo(
|
||||
() => `${getBaseUrl()}/api/webhooks/slack/custom/${credentialId}`,
|
||||
[credentialId]
|
||||
)
|
||||
// Shared server-side derivation: uses the app public base (not
|
||||
// window.location.origin) so Slack's servers can reach it.
|
||||
const requestUrl = useMemo(() => buildSlackCustomBotRequestUrl(credentialId), [credentialId])
|
||||
|
||||
const manifestJson = useMemo(() => {
|
||||
const managedUserAuthorization = selected.has(SLACK_MANAGED_USER_AUTHORIZATION_CAPABILITY.id)
|
||||
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
EMPTY_CELL_PLACEHOLDER,
|
||||
Resource,
|
||||
ResourceNotFound,
|
||||
SearchHighlight,
|
||||
} from '@/app/workspace/[workspaceId]/components'
|
||||
import {
|
||||
FOLDERED_RESOURCE_HEADERS,
|
||||
@@ -55,7 +56,7 @@ import {
|
||||
documentParsers,
|
||||
documentUrlKeys,
|
||||
} from '@/app/workspace/[workspaceId]/knowledge/[id]/[documentId]/search-params'
|
||||
import { ActionBar, SearchHighlight } from '@/app/workspace/[workspaceId]/knowledge/[id]/components'
|
||||
import { ActionBar } from '@/app/workspace/[workspaceId]/knowledge/[id]/components'
|
||||
import { getDocumentIcon } from '@/app/workspace/[workspaceId]/knowledge/components'
|
||||
import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/providers/workspace-permissions-provider'
|
||||
import { useContextMenu } from '@/app/workspace/[workspaceId]/w/components/sidebar/hooks'
|
||||
|
||||
@@ -64,6 +64,7 @@ import {
|
||||
isResourceListEmpty,
|
||||
Resource,
|
||||
ResourceNotFound,
|
||||
SearchHighlight,
|
||||
} from '@/app/workspace/[workspaceId]/components'
|
||||
import {
|
||||
FOLDERED_RESOURCE_HEADERS,
|
||||
@@ -81,7 +82,6 @@ import {
|
||||
ConnectorsSection,
|
||||
DocumentContextMenu,
|
||||
RenameDocumentModal,
|
||||
SearchHighlight,
|
||||
} from '@/app/workspace/[workspaceId]/knowledge/[id]/components'
|
||||
import {
|
||||
addConnectorParam,
|
||||
|
||||
@@ -6,4 +6,3 @@ export { ConnectorsSection } from './connectors-section'
|
||||
export { DocumentContextMenu } from './document-context-menu'
|
||||
export { EditConnectorModal } from './edit-connector-modal'
|
||||
export { RenameDocumentModal } from './rename-document-modal'
|
||||
export { SearchHighlight } from './search-highlight'
|
||||
|
||||
-1
@@ -1 +0,0 @@
|
||||
export { SearchHighlight } from './search-highlight'
|
||||
@@ -87,8 +87,22 @@ vi.mock('@sim/emcn', () => ({
|
||||
</select>
|
||||
),
|
||||
Label: ({ children }: { children: ReactNode }) => <span>{children}</span>,
|
||||
Switch: ({ checked }: { checked: boolean }) => (
|
||||
<button type='button' role='switch' aria-checked={checked} />
|
||||
Switch: ({
|
||||
checked,
|
||||
disabled,
|
||||
onCheckedChange,
|
||||
}: {
|
||||
checked: boolean
|
||||
disabled?: boolean
|
||||
onCheckedChange?: (checked: boolean) => void
|
||||
}) => (
|
||||
<button
|
||||
type='button'
|
||||
role='switch'
|
||||
aria-checked={checked}
|
||||
disabled={disabled}
|
||||
onClick={() => onCheckedChange?.(!checked)}
|
||||
/>
|
||||
),
|
||||
toast: mockToast,
|
||||
}))
|
||||
@@ -235,6 +249,7 @@ const IMPORTED_BOTH: BrowserChromeImportResult = {
|
||||
|
||||
interface BridgeOverrides {
|
||||
browserEnabled?: boolean
|
||||
browserSearchSuggestionsEnabled?: boolean
|
||||
browserTheme?: 'app' | 'light' | 'dark'
|
||||
browserDefaultZoom?: DesktopZoomPercent
|
||||
browserDownloadDirectory?: string
|
||||
@@ -247,6 +262,7 @@ interface BridgeOverrides {
|
||||
|
||||
function createBridge({
|
||||
browserEnabled = true,
|
||||
browserSearchSuggestionsEnabled = true,
|
||||
browserTheme = 'app',
|
||||
browserDefaultZoom = 100,
|
||||
browserDownloadDirectory = '/Users/sim/Downloads',
|
||||
@@ -263,6 +279,7 @@ function createBridge({
|
||||
launchAtLogin: false,
|
||||
autoDownloadUpdates: true,
|
||||
browserEnabled,
|
||||
browserSearchSuggestionsEnabled,
|
||||
browserTheme,
|
||||
browserDefaultZoom,
|
||||
browserDownloadDirectory,
|
||||
@@ -271,6 +288,10 @@ function createBridge({
|
||||
settings: {
|
||||
getPreferences: vi.fn(async () => preferences),
|
||||
setBrowserEnabled: vi.fn(),
|
||||
setBrowserSearchSuggestionsEnabled: vi.fn(async (enabled: boolean) => ({
|
||||
...preferences,
|
||||
browserSearchSuggestionsEnabled: enabled,
|
||||
})),
|
||||
setBrowserTheme: vi.fn(async (theme: 'app' | 'light' | 'dark') => ({
|
||||
...preferences,
|
||||
browserTheme: theme,
|
||||
@@ -343,6 +364,7 @@ describe('Browser settings', () => {
|
||||
|
||||
expect(container.textContent).toContain('Let Chat browse the web')
|
||||
expect(container.textContent).toContain('Theme')
|
||||
expect(container.textContent).toContain('Search suggestions')
|
||||
expect(container.textContent).not.toContain(
|
||||
'Pages open in a browser built into Sim, signed in separately from your own.'
|
||||
)
|
||||
@@ -352,6 +374,22 @@ describe('Browser settings', () => {
|
||||
expect(container.textContent).not.toContain('Saved passwords are never deleted here')
|
||||
})
|
||||
|
||||
it('explains and persists the live-search privacy switch', async () => {
|
||||
const bridge = createBridge({ browserSearchSuggestionsEnabled: true })
|
||||
mockBridge.current = bridge
|
||||
await render()
|
||||
|
||||
expect(container.textContent).toContain(
|
||||
'Send address-bar typing to Google for live completions'
|
||||
)
|
||||
const switches = [...container.querySelectorAll<HTMLButtonElement>('[role="switch"]')]
|
||||
expect(switches[1]).toHaveAttribute('aria-checked', 'true')
|
||||
|
||||
await click(switches[1])
|
||||
|
||||
expect(bridge.settings.setBrowserSearchSuggestionsEnabled).toHaveBeenCalledWith(false)
|
||||
})
|
||||
|
||||
it('persists an independent website theme', async () => {
|
||||
const bridge = createBridge({ browserTheme: 'app' })
|
||||
mockBridge.current = bridge
|
||||
|
||||
@@ -114,6 +114,14 @@ export function Browser() {
|
||||
setPreferences
|
||||
)
|
||||
|
||||
const { pending: searchSuggestionsPending, mutate: setSearchSuggestionsEnabled } =
|
||||
useDesktopPreferenceMutation(
|
||||
async (bridge, enabled: boolean) =>
|
||||
bridge.settings.setBrowserSearchSuggestionsEnabled?.(enabled),
|
||||
'Could not update browser search suggestions',
|
||||
setPreferences
|
||||
)
|
||||
|
||||
const clear = useCallback(async (kinds: readonly BrowserDataKind[]) => {
|
||||
const desktop = getDesktopBridge()
|
||||
if (!desktop) return
|
||||
@@ -145,6 +153,9 @@ export function Browser() {
|
||||
}
|
||||
|
||||
const enabled = preferences.browserEnabled
|
||||
const supportsSearchSuggestions = Boolean(
|
||||
getDesktopBridge()?.settings.setBrowserSearchSuggestionsEnabled
|
||||
)
|
||||
const target = confirming === 'all' ? null : confirming
|
||||
|
||||
return (
|
||||
@@ -172,6 +183,23 @@ export function Browser() {
|
||||
/>
|
||||
</div>
|
||||
|
||||
{supportsSearchSuggestions && (
|
||||
<div className='flex items-center justify-between gap-4'>
|
||||
<div className='flex min-w-0 flex-col gap-1'>
|
||||
<Label htmlFor='browser-search-suggestions'>Search suggestions</Label>
|
||||
<p className='text-[var(--text-muted)] text-caption'>
|
||||
Send address-bar typing to Google for live completions
|
||||
</p>
|
||||
</div>
|
||||
<Switch
|
||||
id='browser-search-suggestions'
|
||||
checked={preferences.browserSearchSuggestionsEnabled ?? true}
|
||||
disabled={searchSuggestionsPending}
|
||||
onCheckedChange={(checked) => void setSearchSuggestionsEnabled(checked)}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className='flex items-center justify-between gap-4'>
|
||||
<Label>Theme</Label>
|
||||
<AppearanceThemeSelect
|
||||
|
||||
+3
-2
@@ -28,6 +28,7 @@ import { columnTypeOf } from '@/lib/table/column-types'
|
||||
import { TABLE_LIMITS } from '@/lib/table/constants'
|
||||
import { cellValueFilterConditions } from '@/lib/table/query-builder/cell-filter'
|
||||
import { SEARCH_DEBOUNCE_MS } from '@/lib/url-state'
|
||||
import { FindBar } from '@/app/workspace/[workspaceId]/components'
|
||||
import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/providers/workspace-permissions-provider'
|
||||
import type { RemoteTableSelection } from '@/app/workspace/[workspaceId]/tables/[tableId]/hooks/use-table-room'
|
||||
import type { BlockedTableAction } from '@/app/workspace/[workspaceId]/tables/[tableId]/lock-copy'
|
||||
@@ -63,7 +64,6 @@ import { ADD_COL_WIDTH, COL_WIDTH, SELECTION_TINT_BG } from './constants'
|
||||
import { DataRow } from './data-row'
|
||||
import { ColumnHeaderMenu, WorkflowGroupMetaCell } from './headers'
|
||||
import { RemoteSelectionOverlay } from './remote-selection-overlay'
|
||||
import { TableFind } from './table-find'
|
||||
import { AddRowButton, SelectAllCheckbox, TableColGroup } from './table-primitives'
|
||||
import type { DisplayColumn } from './types'
|
||||
import {
|
||||
@@ -4592,7 +4592,8 @@ export function TableGrid({
|
||||
<div ref={containerRef} className='flex h-full flex-col overflow-hidden'>
|
||||
<div className='relative flex min-h-0 flex-1'>
|
||||
{findOpen && (
|
||||
<TableFind
|
||||
<FindBar
|
||||
ariaLabel='Find in table'
|
||||
query={findQuery}
|
||||
onQueryChange={setFindQuery}
|
||||
onNext={handleFindNext}
|
||||
|
||||
@@ -125,6 +125,13 @@ interface TableProps {
|
||||
* today's Filter/Sort behavior.
|
||||
*/
|
||||
viewsEnabled?: boolean
|
||||
/**
|
||||
* Saved view to adopt on first seed instead of the table's default —
|
||||
* embedded mode only, set when the agent opened this table pinned to a
|
||||
* view. Participates only in the one-time adoption branch, so it never
|
||||
* fights a later user switch.
|
||||
*/
|
||||
initialViewId?: string
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -191,6 +198,7 @@ interface ViewConfigKeep {
|
||||
*/
|
||||
export function Table({
|
||||
embedded,
|
||||
initialViewId,
|
||||
workspaceId: propWorkspaceId,
|
||||
tableId: propTableId,
|
||||
tableLocksEnabled = false,
|
||||
@@ -585,17 +593,23 @@ export function Table({
|
||||
const legacyAllWithDefault = activeViewId === ALL_VIEW_PARAM && defaultView !== null
|
||||
|
||||
if (activeViewId === null || inheritedParams || legacyAllWithDefault) {
|
||||
// Embedded mode may pin the view the agent opened this table on
|
||||
// (`initialViewId`); the pin outranks the persisted default for this
|
||||
// first adoption.
|
||||
const pinnedView =
|
||||
embedded && initialViewId ? views.find((view) => view.id === initialViewId) : undefined
|
||||
const viewToApply = pinnedView ?? defaultView
|
||||
// `sort` rides the same host URL, so when the view id is inherited the
|
||||
// sort beside it is too — not local work, and it must not suppress the
|
||||
// default view's own sort.
|
||||
const keep = inheritedParams ? { ...localWork(), sort: false } : localWork()
|
||||
if (defaultView) {
|
||||
appliedViewRevisionRef.current = getTableViewRevision(defaultView)
|
||||
setTableParams({ view: defaultView.id })
|
||||
preserveViewState(defaultView.id, keep)
|
||||
applyViewConfig(resolveTableViewConfig(tableData?.metadata, defaultView.config), keep)
|
||||
resolvePendingLayout(defaultView.id)
|
||||
flushPendingViewConfig(defaultView.id)
|
||||
if (viewToApply) {
|
||||
appliedViewRevisionRef.current = getTableViewRevision(viewToApply)
|
||||
setTableParams({ view: viewToApply.id })
|
||||
preserveViewState(viewToApply.id, keep)
|
||||
applyViewConfig(resolveTableViewConfig(tableData?.metadata, viewToApply.config), keep)
|
||||
resolvePendingLayout(viewToApply.id)
|
||||
flushPendingViewConfig(viewToApply.id)
|
||||
return
|
||||
}
|
||||
// No view to adopt. Deliberately does NOT apply an empty config — that
|
||||
|
||||
+4
-1
@@ -1070,7 +1070,10 @@ export async function executeWorkflowWithFullLogging(
|
||||
error: errorMessage,
|
||||
httpStatus: response.status,
|
||||
})
|
||||
throw new Error(errorMessage)
|
||||
// Keep the status and code on the thrown error. Downgrading to a bare Error
|
||||
// discarded both, so callers could not tell a Copilot binding rejection from
|
||||
// any other 4xx — and the reason never reached the agent that could fix it.
|
||||
throw new ExecutionStreamHttpError(errorMessage, response.status, errorCode)
|
||||
}
|
||||
|
||||
if (!response.body) {
|
||||
|
||||
@@ -139,6 +139,26 @@ export function VideoIcon(props: SVGProps<SVGSVGElement>) {
|
||||
)
|
||||
}
|
||||
|
||||
export function ChartFileIcon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
viewBox='0 0 24 24'
|
||||
fill='none'
|
||||
stroke='currentColor'
|
||||
strokeWidth='1.5'
|
||||
strokeLinecap='round'
|
||||
strokeLinejoin='round'
|
||||
xmlns='http://www.w3.org/2000/svg'
|
||||
{...props}
|
||||
>
|
||||
<path d='M3 3v17a1 1 0 0 0 1 1h17' />
|
||||
<rect x='7' y='12' width='3' height='6' rx='0.5' />
|
||||
<rect x='12.5' y='8' width='3' height='10' rx='0.5' />
|
||||
<rect x='18' y='5' width='3' height='13' rx='0.5' />
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
|
||||
export function HtmlIcon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg
|
||||
@@ -277,6 +297,16 @@ export function getDocumentIcon(
|
||||
return PptxIcon
|
||||
}
|
||||
|
||||
if (mimeType === 'text/x-sim-chart' || extension === 'chart') {
|
||||
return ChartFileIcon
|
||||
}
|
||||
|
||||
// Sim pages present as plain documents, not as HTML artifacts — the .html
|
||||
// is an implementation detail (legacy pages still carry the extension).
|
||||
if (mimeType === 'text/x-sim-page') {
|
||||
return DefaultFileIcon
|
||||
}
|
||||
|
||||
if (mimeType === 'text/html' || extension === 'html' || extension === 'htm') {
|
||||
return HtmlIcon
|
||||
}
|
||||
|
||||
@@ -30,6 +30,7 @@ import { useQueryState } from 'nuqs'
|
||||
import { saveDiscardActions } from '@/components/settings/save-discard-actions'
|
||||
import type { ShareAuthType } from '@/lib/api/contracts/public-shares'
|
||||
import { isBlockTypeAccessControlExempt } from '@/lib/permission-groups/block-access'
|
||||
import { PLATFORM_CATEGORY_ORDER, PLATFORM_FEATURES } from '@/lib/permission-groups/features'
|
||||
import type { PermissionGroupConfig } from '@/lib/permission-groups/types'
|
||||
import { UnsavedChangesModal } from '@/app/workspace/[workspaceId]/components/credential-detail'
|
||||
import {
|
||||
@@ -174,151 +175,6 @@ function AuthModeField({ label, value, onChange, options, disabled }: AuthModeFi
|
||||
)
|
||||
}
|
||||
|
||||
/** Render order for the platform-feature category sections; unlisted ones follow. */
|
||||
const PLATFORM_CATEGORY_ORDER = [
|
||||
'Sidebar',
|
||||
'Deploy Tabs',
|
||||
'Chat',
|
||||
'Collaboration',
|
||||
'Workflow Panel',
|
||||
'Tools',
|
||||
'Features',
|
||||
'Settings Tabs',
|
||||
'Logs',
|
||||
'Files',
|
||||
]
|
||||
|
||||
const PLATFORM_FEATURES = [
|
||||
{
|
||||
id: 'hide-knowledge-base',
|
||||
label: 'Knowledge Base',
|
||||
category: 'Sidebar',
|
||||
configKey: 'hideKnowledgeBaseTab' as const,
|
||||
hint: 'Hide the Knowledge Base module from the sidebar.',
|
||||
},
|
||||
{
|
||||
id: 'hide-tables',
|
||||
label: 'Tables',
|
||||
category: 'Sidebar',
|
||||
configKey: 'hideTablesTab' as const,
|
||||
hint: 'Hide the Tables module from the sidebar.',
|
||||
},
|
||||
{
|
||||
id: 'hide-copilot',
|
||||
label: 'Chat',
|
||||
category: 'Workflow Panel',
|
||||
configKey: 'hideCopilot' as const,
|
||||
hint: 'Hide the Chat panel so users cannot build or edit with natural language.',
|
||||
},
|
||||
{
|
||||
id: 'hide-integrations',
|
||||
label: 'Integrations',
|
||||
category: 'Settings Tabs',
|
||||
configKey: 'hideIntegrationsTab' as const,
|
||||
hint: 'Hide the Integrations settings tab (OAuth connections).',
|
||||
},
|
||||
{
|
||||
id: 'hide-secrets',
|
||||
label: 'Secrets',
|
||||
category: 'Settings Tabs',
|
||||
configKey: 'hideSecretsTab' as const,
|
||||
hint: 'Hide the Secrets (environment variables) settings tab.',
|
||||
},
|
||||
{
|
||||
id: 'hide-api-keys',
|
||||
label: 'API Keys',
|
||||
category: 'Settings Tabs',
|
||||
configKey: 'hideApiKeysTab' as const,
|
||||
hint: 'Hide the API Keys settings tab.',
|
||||
},
|
||||
{
|
||||
id: 'hide-files',
|
||||
label: 'Files',
|
||||
category: 'Settings Tabs',
|
||||
configKey: 'hideFilesTab' as const,
|
||||
hint: 'Hide the Files settings tab.',
|
||||
},
|
||||
{
|
||||
id: 'hide-deploy-api',
|
||||
label: 'API',
|
||||
category: 'Deploy Tabs',
|
||||
configKey: 'hideDeployApi' as const,
|
||||
hint: 'Hide the API deployment option.',
|
||||
},
|
||||
{
|
||||
id: 'hide-deploy-mcp',
|
||||
label: 'MCP',
|
||||
category: 'Deploy Tabs',
|
||||
configKey: 'hideDeployMcp' as const,
|
||||
hint: 'Hide the MCP server deployment option.',
|
||||
},
|
||||
{
|
||||
id: 'disable-mcp',
|
||||
label: 'MCP Tools',
|
||||
category: 'Tools',
|
||||
configKey: 'disableMcpTools' as const,
|
||||
hint: 'Block agents from calling MCP tools.',
|
||||
},
|
||||
{
|
||||
id: 'disable-custom-tools',
|
||||
label: 'Custom Tools',
|
||||
category: 'Tools',
|
||||
configKey: 'disableCustomTools' as const,
|
||||
hint: 'Block agents from calling user-defined custom tools.',
|
||||
},
|
||||
{
|
||||
id: 'disable-skills',
|
||||
label: 'Skills',
|
||||
category: 'Tools',
|
||||
configKey: 'disableSkills' as const,
|
||||
hint: 'Block agents from loading skills.',
|
||||
},
|
||||
{
|
||||
id: 'hide-trace-spans',
|
||||
label: 'Trace Spans',
|
||||
category: 'Logs',
|
||||
configKey: 'hideTraceSpans' as const,
|
||||
hint: 'Hide per-block trace spans in logs.',
|
||||
},
|
||||
{
|
||||
id: 'disable-invitations',
|
||||
label: 'Invitations',
|
||||
category: 'Collaboration',
|
||||
configKey: 'disableInvitations' as const,
|
||||
hint: 'Prevent users from inviting others to workspaces.',
|
||||
},
|
||||
{
|
||||
id: 'hide-inbox',
|
||||
label: 'Sim Mailer',
|
||||
category: 'Features',
|
||||
configKey: 'hideInboxTab' as const,
|
||||
hint: 'Hide the Sim Mailer inbox.',
|
||||
},
|
||||
{
|
||||
id: 'disable-public-api',
|
||||
label: 'Public API',
|
||||
category: 'Features',
|
||||
configKey: 'disablePublicApi' as const,
|
||||
hint: 'Disable public API access to deployed workflows.',
|
||||
},
|
||||
// Chat and Files get a category of their own so their nested auth-mode
|
||||
// dropdown (see `featureExtras`) reads as part of the toggle it qualifies.
|
||||
{
|
||||
id: 'hide-deploy-chatbot',
|
||||
label: 'Deployment',
|
||||
category: 'Chat',
|
||||
configKey: 'hideDeployChatbot' as const,
|
||||
hint: 'Hide the chat deployment option.',
|
||||
},
|
||||
{
|
||||
id: 'disable-public-file-sharing',
|
||||
label: 'Public Sharing',
|
||||
category: 'Files',
|
||||
configKey: 'disablePublicFileSharing' as const,
|
||||
hint: 'Disable public file-share links.',
|
||||
},
|
||||
]
|
||||
|
||||
interface OrganizationMemberOption {
|
||||
userId: string
|
||||
user: {
|
||||
@@ -963,7 +819,7 @@ export function GroupDetail({
|
||||
}, [searchedPlatformFeatures, statusFilter, editingConfig])
|
||||
|
||||
const platformCategories = useMemo(() => {
|
||||
const categories: Record<string, typeof PLATFORM_FEATURES> = {}
|
||||
const categories: Record<string, (typeof PLATFORM_FEATURES)[number][]> = {}
|
||||
for (const feature of filteredPlatformFeatures) {
|
||||
if (!categories[feature.category]) {
|
||||
categories[feature.category] = []
|
||||
|
||||
@@ -84,6 +84,8 @@ import {
|
||||
ModelNotAllowedError,
|
||||
ProviderNotAllowedError,
|
||||
PublicFileSharingNotAllowedError,
|
||||
resolveUserAccessControlContext,
|
||||
resolveVerifiedUserAccessControlContext,
|
||||
SkillsNotAllowedError,
|
||||
ToolNotAllowedError,
|
||||
validateBlockType,
|
||||
@@ -229,6 +231,126 @@ describe('getUserPermissionConfig (org + entitlement gating)', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('resolveUserAccessControlContext', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
mockGetAllowedIntegrationsFromEnv.mockReturnValue(null)
|
||||
})
|
||||
|
||||
it('describes a personal workspace without changing the config-only result', async () => {
|
||||
mockGetWorkspaceWithOwner.mockResolvedValue({ organizationId: null })
|
||||
|
||||
await expect(resolveUserAccessControlContext('user-123', 'workspace-1')).resolves.toEqual({
|
||||
organizationId: null,
|
||||
entitled: false,
|
||||
permissionGroup: null,
|
||||
config: null,
|
||||
})
|
||||
await expect(getUserPermissionConfig('user-123', 'workspace-1')).resolves.toBeNull()
|
||||
})
|
||||
|
||||
it('returns the explicit governing group and its effective config', async () => {
|
||||
setEnterpriseOrgWorkspace()
|
||||
queueGroupResolution([
|
||||
{
|
||||
id: 'group-explicit',
|
||||
name: 'Engineering',
|
||||
config: { disableMcpTools: true },
|
||||
isMember: true,
|
||||
hasMembers: true,
|
||||
},
|
||||
])
|
||||
|
||||
await expect(resolveUserAccessControlContext('user-123', 'workspace-1')).resolves.toEqual({
|
||||
organizationId: 'org-1',
|
||||
entitled: true,
|
||||
permissionGroup: {
|
||||
id: 'group-explicit',
|
||||
name: 'Engineering',
|
||||
resolution: 'explicit-member',
|
||||
},
|
||||
config: expect.objectContaining({ disableMcpTools: true }),
|
||||
})
|
||||
})
|
||||
|
||||
it('identifies an all-members governing group', async () => {
|
||||
setEnterpriseOrgWorkspace()
|
||||
queueGroupResolution([
|
||||
{
|
||||
id: 'group-all-members',
|
||||
name: 'All workspace members',
|
||||
config: { disableCustomTools: true },
|
||||
isMember: false,
|
||||
hasMembers: false,
|
||||
},
|
||||
])
|
||||
|
||||
const context = await resolveUserAccessControlContext('user-123', 'workspace-1')
|
||||
|
||||
expect(context.permissionGroup).toEqual({
|
||||
id: 'group-all-members',
|
||||
name: 'All workspace members',
|
||||
resolution: 'all-members',
|
||||
})
|
||||
})
|
||||
|
||||
it('uses a verified workspace organization without loading the workspace again', async () => {
|
||||
mockIsOrganizationOnEnterprisePlan.mockResolvedValue(true)
|
||||
queueGroupResolution([
|
||||
{
|
||||
id: 'group-verified',
|
||||
name: 'Verified group',
|
||||
config: { disableSkills: true },
|
||||
isMember: true,
|
||||
hasMembers: true,
|
||||
},
|
||||
])
|
||||
|
||||
const context = await resolveVerifiedUserAccessControlContext(
|
||||
'user-123',
|
||||
'workspace-1',
|
||||
'org-verified'
|
||||
)
|
||||
|
||||
expect(mockGetWorkspaceWithOwner).not.toHaveBeenCalled()
|
||||
expect(mockIsOrganizationOnEnterprisePlan).toHaveBeenCalledWith('org-verified')
|
||||
expect(context).toMatchObject({
|
||||
organizationId: 'org-verified',
|
||||
entitled: true,
|
||||
permissionGroup: {
|
||||
id: 'group-verified',
|
||||
resolution: 'explicit-member',
|
||||
},
|
||||
config: { disableSkills: true },
|
||||
})
|
||||
})
|
||||
|
||||
it('identifies the default group and preserves the environment allowlist', async () => {
|
||||
setEnterpriseOrgWorkspace()
|
||||
mockGetAllowedIntegrationsFromEnv.mockReturnValue(['slack'])
|
||||
queueGroupResolution(
|
||||
[],
|
||||
[
|
||||
{
|
||||
id: 'group-default',
|
||||
name: 'Organization default',
|
||||
config: { allowedIntegrations: ['slack', 'github'] },
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
const context = await resolveUserAccessControlContext('user-123', 'workspace-1')
|
||||
|
||||
expect(context.permissionGroup).toEqual({
|
||||
id: 'group-default',
|
||||
name: 'Organization default',
|
||||
resolution: 'default',
|
||||
})
|
||||
expect(context.config?.allowedIntegrations).toEqual(['slack'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('getUserPermissionConfig (workspace-group precedence)', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
|
||||
@@ -141,9 +141,30 @@ function mergeEnvAllowlist(config: PermissionGroupConfig | null): PermissionGrou
|
||||
export interface ResolvedPermissionGroup {
|
||||
permissionGroupId: string
|
||||
groupName: string
|
||||
resolution: 'explicit-member' | 'all-members' | 'default'
|
||||
config: PermissionGroupConfig
|
||||
}
|
||||
|
||||
export interface UserAccessControlContext {
|
||||
organizationId: string | null
|
||||
entitled: boolean
|
||||
permissionGroup: {
|
||||
id: string
|
||||
name: string
|
||||
resolution: ResolvedPermissionGroup['resolution']
|
||||
} | null
|
||||
config: PermissionGroupConfig | null
|
||||
}
|
||||
|
||||
function inactiveUserAccessControlContext(organizationId: string | null): UserAccessControlContext {
|
||||
return {
|
||||
organizationId,
|
||||
entitled: false,
|
||||
permissionGroup: null,
|
||||
config: mergeEnvAllowlist(null),
|
||||
}
|
||||
}
|
||||
|
||||
/** The organization's single default group (`isDefault`), or `null`. */
|
||||
async function resolveDefaultGroup(
|
||||
organizationId: string
|
||||
@@ -167,6 +188,7 @@ async function resolveDefaultGroup(
|
||||
return {
|
||||
permissionGroupId: defaultGroup.id,
|
||||
groupName: defaultGroup.name,
|
||||
resolution: 'default',
|
||||
config: parsePermissionGroupConfig(defaultGroup.config),
|
||||
}
|
||||
}
|
||||
@@ -222,12 +244,14 @@ export async function resolveWorkspaceGroup(
|
||||
)
|
||||
.orderBy(asc(permissionGroup.createdAt), asc(permissionGroup.id))
|
||||
|
||||
const winner = rows.find((row) => row.isMember) ?? rows.find((row) => !row.hasMembers)
|
||||
const explicitMemberGroup = rows.find((row) => row.isMember)
|
||||
const winner = explicitMemberGroup ?? rows.find((row) => !row.hasMembers)
|
||||
|
||||
if (winner) {
|
||||
return {
|
||||
permissionGroupId: winner.id,
|
||||
groupName: winner.name,
|
||||
resolution: explicitMemberGroup ? 'explicit-member' : 'all-members',
|
||||
config: parsePermissionGroupConfig(winner.config),
|
||||
}
|
||||
}
|
||||
@@ -246,26 +270,70 @@ export async function resolveWorkspaceGroup(
|
||||
* The env-level integration allowlist is always merged last so self-hosted
|
||||
* deployments can constrain integrations without touching the DB.
|
||||
*/
|
||||
async function resolveUserAccessControlContextForOrganization(
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
organizationId: string | null
|
||||
): Promise<UserAccessControlContext> {
|
||||
if (!organizationId) return inactiveUserAccessControlContext(null)
|
||||
|
||||
const isEnterprise = await isOrganizationOnEnterprisePlan(organizationId)
|
||||
if (!isEnterprise) {
|
||||
return inactiveUserAccessControlContext(organizationId)
|
||||
}
|
||||
|
||||
const resolved = await resolveWorkspaceGroup(userId, organizationId, workspaceId)
|
||||
return {
|
||||
organizationId,
|
||||
entitled: true,
|
||||
permissionGroup: resolved
|
||||
? {
|
||||
id: resolved.permissionGroupId,
|
||||
name: resolved.groupName,
|
||||
resolution: resolved.resolution,
|
||||
}
|
||||
: null,
|
||||
config: mergeEnvAllowlist(resolved?.config ?? null),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves Access Control from an organization ID obtained from an already
|
||||
* access-checked workspace. This function does not independently authorize the
|
||||
* user for the workspace; callers must establish that boundary first.
|
||||
*/
|
||||
export async function resolveVerifiedUserAccessControlContext(
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
organizationId: string | null
|
||||
): Promise<UserAccessControlContext> {
|
||||
if (!isHosted && !isAccessControlEnabled) {
|
||||
return inactiveUserAccessControlContext(null)
|
||||
}
|
||||
return resolveUserAccessControlContextForOrganization(userId, workspaceId, organizationId)
|
||||
}
|
||||
|
||||
export async function resolveUserAccessControlContext(
|
||||
userId: string,
|
||||
workspaceId: string
|
||||
): Promise<UserAccessControlContext> {
|
||||
if (!isHosted && !isAccessControlEnabled) {
|
||||
return inactiveUserAccessControlContext(null)
|
||||
}
|
||||
|
||||
const workspace = await getWorkspaceWithOwner(workspaceId, { includeArchived: true })
|
||||
return resolveUserAccessControlContextForOrganization(
|
||||
userId,
|
||||
workspaceId,
|
||||
workspace?.organizationId ?? null
|
||||
)
|
||||
}
|
||||
|
||||
export async function getUserPermissionConfig(
|
||||
userId: string,
|
||||
workspaceId: string
|
||||
): Promise<PermissionGroupConfig | null> {
|
||||
if (!isHosted && !isAccessControlEnabled) {
|
||||
return mergeEnvAllowlist(null)
|
||||
}
|
||||
|
||||
const ws = await getWorkspaceWithOwner(workspaceId, { includeArchived: true })
|
||||
if (!ws?.organizationId) {
|
||||
return mergeEnvAllowlist(null)
|
||||
}
|
||||
|
||||
const isEnterprise = await isOrganizationOnEnterprisePlan(ws.organizationId)
|
||||
if (!isEnterprise) {
|
||||
return mergeEnvAllowlist(null)
|
||||
}
|
||||
|
||||
const resolved = await resolveWorkspaceGroup(userId, ws.organizationId, workspaceId)
|
||||
return mergeEnvAllowlist(resolved?.config ?? null)
|
||||
return (await resolveUserAccessControlContext(userId, workspaceId)).config
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -501,6 +501,47 @@ export function useFindTableRows({ workspaceId, tableId, q, filter, sort }: Find
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Bounded single-page row read for chart files (`.chart` previews): one fetch
|
||||
* of up to `limit` rows with the spec's verbatim filter/sort. Charts accept a
|
||||
* truncated page — they visualize a sample, not a drained table.
|
||||
*/
|
||||
export function useTableRowsSample({
|
||||
workspaceId,
|
||||
tableId,
|
||||
filter,
|
||||
sort,
|
||||
limit,
|
||||
enabled = true,
|
||||
}: {
|
||||
workspaceId?: string
|
||||
tableId?: string
|
||||
/** Passed through verbatim from the chart document; the contract validates. */
|
||||
filter?: unknown
|
||||
sort?: unknown
|
||||
limit: number
|
||||
enabled?: boolean
|
||||
}) {
|
||||
const paramsKey = JSON.stringify({ filter: filter ?? null, sort: sort ?? null, limit })
|
||||
return useQuery({
|
||||
// rq-lint-allow: tableId is globally unique; workspaceId is only the authz scope
|
||||
queryKey: tableKeys.sample(tableId ?? '', paramsKey),
|
||||
queryFn: ({ signal }) =>
|
||||
fetchTableRows({
|
||||
workspaceId: workspaceId as string,
|
||||
tableId: tableId as string,
|
||||
limit,
|
||||
filter: (filter ?? null) as TablePredicate | null,
|
||||
sort: (sort ?? null) as SortSpec | null,
|
||||
includeTotal: false,
|
||||
signal,
|
||||
}),
|
||||
enabled: Boolean(workspaceId && tableId) && enabled,
|
||||
staleTime: TABLE_ROWS_STALE_TIME,
|
||||
placeholderData: keepPreviousData,
|
||||
})
|
||||
}
|
||||
|
||||
export function tableRowsInfiniteOptions({
|
||||
workspaceId,
|
||||
tableId,
|
||||
|
||||
@@ -34,6 +34,9 @@ export const tableKeys = {
|
||||
infiniteRows: (tableId: string, paramsKey: string) =>
|
||||
[...tableKeys.infiniteRowsRoot(tableId), paramsKey] as const,
|
||||
rowWrites: (tableId: string) => [...tableKeys.rowsRoot(tableId), 'write'] as const,
|
||||
/** Bounded single-page row read for chart files (`.chart` previews). */
|
||||
sample: (tableId: string, paramsKey: string) =>
|
||||
[...tableKeys.rowsRoot(tableId), 'sample', paramsKey] as const,
|
||||
find: (tableId: string, paramsKey: string) =>
|
||||
[...tableKeys.rowsRoot(tableId), 'find', paramsKey] as const,
|
||||
/** Deliberately NOT under `detail` — the non-exact `invalidateQueries` on that
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user