6397 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
3 people	afccfe98a2	fix(selectors): respect trigger credentials in trigger mode (#6952) ... * fix(selectors): respect trigger credentials in trigger mode * fix(selectors): isolate trigger selector context * fix(selectors): isolate action credential fallback * fix(selectors): scope fork reconfigs to trigger mode --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-21 19:10:28 -07:00
Waleed	c16883e9dc	improvement(tabs): give the resource tabs a quieter floating look (#6960) ... * improvement(tabs): give the resource tabs a quieter floating look Adds a `floating` variant to the shared TabStrip and uses it for the mothership resource tabs. Only the active tab carries a shape; the rest are bare labels divided by a hairline, sized to their content up to a cap. The browser and terminal strips keep the attached look, which stays the default. * improvement(tabs): align the floating tab ramp and icon size with the platform Each surface token now does the job it is named for: a bare tab hovers to --surface-hover instead of the Button variant's --surface-4, and a selected tab takes the rung between hover and --surface-active. Tab icons match the action icons beside them at 16px.	2026-08-21 19:01:37 -07:00
Vikhyath MondretiandClaude Opus 5	71129cd112	feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces (#6950) ... * feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces Joining a custom block's child run into its caller's trace shipped on by default, gated at read time by whether the person reading could already open the source workspace. That gate is doing the wrong job: a custom block's whole point is that consumers need no access to the source, so the check refuses exactly the readers the feature exists for, and it makes the answer depend on who is looking rather than on what the block's owner agreed to publish. The decision moves to the party whose data it is. `custom_block.trace_child_runs` is set by the publisher in Settings, applies org-wide, and is the entire policy — nothing downstream re-checks a caller. `getCustomBlockAuthority` already resolves per invocation and is the one lookup both the canvas handler and the Agent-tool runner pass through, so one column covers both surfaces and no consumer input can assert it. It defaults to FALSE. With the viewer check gone, an opted-in block publishes the source workflow's block names, inputs, outputs, and prompts to anyone who can read a consuming workflow's log. That is the same boundary curated outputs and redacted errors hold, so it opens by an affirmative act of the publisher or not at all — never as the residue of a column default on rows nobody revisited. Closed means the handle is withheld outright rather than persisted behind a flag: with no `childExecutionId` there is nothing for a reader, a migration, or a later refactor to join. What replaces it is a `_childTraceDisabled` marker, because a boundary span with no children renders exactly like a leaf block and an untraced run would otherwise read as one that did nothing. The consumer-facing failure `ref` is untouched either way — it is the only thing that makes an untraced failure reportable. Custom blocks invoked as Agent tools now join too. The child's handle already reached the agent's persisted `toolCalls[].result` (`postProcessToolOutput` strips only `__`-prefixed keys); nothing lifted it onto the tool span. Both span builders lift and strip it, and `hydrateChildTraces` needs no change — its boundary walk already recurses. The same handle is stripped from the model-facing copy of the tool result in `executeProviderTool`, the single point where the raw and model copies diverge: an opaque execution id in a tool result reads to a model like data the tool returned. The live SSE stream keeps one condition beyond the policy: an identified consumer. Not an authorization check — no workspace query — but chat deployments and the public API leave `liveTraceViewerUserId` unset because their consumer may be anonymous, and opting into org-wide tracing is not consent to stream a publisher's raw agent tokens to the internet. Copilot deliberately cannot set the field; exposing a team's internals org-wide is a human decision, not one an agent makes while publishing on their behalf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): read the publisher's trace policy at read time, not from the handle's presence Treating a persisted `childExecutionId` as proof of publisher consent is only true for handles this PR's writer produced. Every handle written before it meant something else — "a child ran; authorize the reader" — and the rows carrying them outlive the migration, so removing the reader check turned them into an open door: a consumer could open an old parent log and receive the source workflow's block names, inputs, outputs, and prompts from a block whose publisher never opted in. `hydrateChildTraces` now resolves the policy live, per boundary, from `custom_block.trace_child_runs`. The child log row's `workflowId` is the key — publish enforces one block per workflow — which also covers an Agent-tool boundary, whose span carries no block type to look up. A workflow with no block row (never published, or since deleted) has no publisher left to consent and stays shut, as does a failed policy read. This is not redundant with the write-time withholding. The handler still emits no handle for a block that was closed when the run executed, so such a run stays closed forever even if the block is opened later; this check decides whether the runs that DO carry a handle may still be shown. Turning the policy off therefore also closes what is already recorded, which is what a governance switch has to do to mean anything. Reported by Greptile on #6950. Also drops `any` from the trace-policy tests: outputs read through `Record<string, unknown>` (the handler's declared return does not name these internal keys) and failures narrow through `ChildWorkflowError.isChildWorkflowError`, which pins the failure type as well as its fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(logs): sum the child-trace drop counters from the struct, not a hand-listed set `totalDropped` re-listed four of the five counters, so a read whose only drops were policy refusals computed zero and skipped the log entirely. That is the commonest drop there is now — every handle written before the publisher policy existed refuses at that gate — so the one signal telling an operator the live check is closing joins went silent exactly when it started mattering. Summed from the struct instead. A hand-maintained list beside a struct is stale the moment a field is added, which is precisely how `policyClosed` was left out. Reported by Cursor Bugbot on #6950. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(db): renumber the custom-block trace migration around a 0299 collision Staging landed its own 0299 (`table_run_dispatches.heartbeat_at`) while this branch was open. The two migrations are independent — different tables, no shared statement — so only the number and drizzle's snapshot chain collided. Regenerated rather than hand-merged: a drizzle snapshot is a full-schema dump whose `prevId` links it to its parent, so editing one by hand to sit after a migration it was not generated against is how the chain silently stops matching the database. Staging's 0299 and its snapshot are taken verbatim; this is 0300, generated against them, and its SQL is byte-identical to what it replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 18:21:08 -07:00
Vikhyath MondretiandClaude Opus 5	fe0b92bb6d	feat(secrets): show where a secret is referenced, beside its usage log (#6947) ... * feat(secrets): show where a secret is referenced, beside its usage log "See usage" answered who has run something with a key. It could not answer the question a rotation actually starts from — where is this wired in — because a secret four blocks depend on but nothing has executed yet has no usage rows at all, so the panel read "This secret has not been used yet" for a live key. The usage view now carries two tabs. Logs is the existing trail, unchanged and still the default, since that is what the header action has always opened. References is new: the blocks that name the secret as {{KEY}}, grouped under their workflow, then the custom tools and MCP servers whose own bodies carry it. Detection is the workspace-fork remapper's. remapSubBlocks already walks nested tool-input params, resolves canonical basic/advanced pairs, and skips dormant and condition-hidden members, so calling it per block inherits every rule a fork already obeys. Only the aggregation is new: scanWorkflowReferences collapses its output to unique (kind, sourceId) pairs and discards the workflow — right for building a mapping table, wrong for locating a key. Nothing under ee/workspace-forking changed. - Candidates come from strpos(sub_blocks::text, name) > 0, deliberately not LIKE: `_` is a LIKE single-character wildcard and nearly every env key contains one, so SB_ACTION_ROUTER_SECRET would match text it does not occur in. The prefilter can over-match but never under-match; the scanner decides. The plan is an index scan on workflow by workspace, nested-looped into workflow_blocks, so cost tracks the workspace rather than the table. - Scope gates the read but does not narrow it. A {{KEY}} names a key, not a scope, so the same sites answer for a workspace secret and the personal one it shadows; narrowing here would report a personal secret as unreferenced the moment a workspace variable of the same name existed. - References reports one field per block, not a list. The remapper dedupes a block's references by (kind, sourceId), so a block naming the secret twice yields one entry — the type says so and a test pins it, because the row renders that field as its whole description. - Reads live state, not deployed: a draft workflow referencing the key must show. Blocks are capped and the cap is reported as `truncated` rather than silently trimming the list. - Authorization is the existing usage gate, renamed requireSecretTrailReadAccess and shared verbatim, so the two tabs can never disagree about who may look. UI is existing primitives only — ChipModalTabs for the strip, DetailSection per workflow over RESOURCE_LIST_STACK rows, IntegrationTile for the block glyph so a block reads here as it does on an integrations row, SettingsEmptyState for the gates. No new component, no new class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): close the reference-scan scope bypass and bound its output Review round 1. - use-cases.ts: `scope` was a caller-controlled assertion the reference scan never narrowed by, so `scope=personal` returned from the shared gate before any check and handed any workspace member the admin-gated reference map for any workspace secret. The usage trail can trust that scope because it filters the read by `secretOwnerUserId`; a name-based workspace-wide scan cannot. References now authorize on what the NAME resolves to — a workspace secret under that name is admin-gated outright, and absent one the caller must actually hold a personal secret of that name, which also stops a member enumerating arbitrary names. `scope` is dropped from the input, the contract, the hook and the query key rather than merely ignored: a parameter that does not exist cannot be asserted. The trail gate keeps its old name and a note saying why only a scope-narrowed read may reuse it. - scan.ts: the prefilter matched the bare name, so `API_KEY` also read every block holding `{{API_KEY_TEST}}` or the words "the API_KEY value" — and those false positives counted against the row cap, so on a workspace with enough of them genuine references sorted later were never read at all. It now matches the reference syntax (`{{name}}`, with the whitespace ENV_REF_PATTERN allows), so a candidate is a real occurrence and the cap means what it says. A name outside the env-key charset short-circuits, which is also what makes it safe to inline into the regex unescaped. Verified against a real workspace: the exact key still returns its 16 blocks, its prefix now returns 0 where it previously matched all 16, and a metacharacter name touches no query. - scan.ts: capping tool and server ROWS did not bound the output — one MCP server emits an entry per matching header plus one for its url, so 200 rows could expand past the contract's 400-entry bound and make the route reject its own response, turning a successful scan into a 500 and the tab into "Could not load references." Emission now stops at the bound and reports `truncated`. - secret-references-panel.tsx: the empty-state early return preceded the truncation banner, so a capped scan that filtered everything out claimed the secret was unreferenced. Both paths now share one note, and silence from a capped scan reads as absence of evidence rather than evidence of absence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): cover unicode whitespace, legacy keys, and the shadowed-personal tab Review round 2 — all three follow from round 1's own fixes. - scan.ts: the syntax prefilter anchored on `[[:space:]]`, but the two engines disagree about what whitespace is. `ENV_REF_PATTERN`'s `\s` accepts U+00A0, U+202F and U+3000; Postgres `[[:space:]]` matches only the ASCII set. So a value pasted with a non-breaking space inside the braces is a reference the executor resolves and the prefilter silently dropped — the one failure direction this feature must never take, since the answer it gives is "unused, safe to delete". Anchoring on `[^[:alnum:]_]` instead accepts every whitespace encoding while still rejecting a longer key on either side, and needs no code-point list that could drift. It can admit a non-reference like `{{-NAME-}}`; that costs one candidate row, and the scanner re-checks every candidate regardless. Erring loose here is deliberate. (Greptile's `{{\tAPI_KEY\t}}` example was already handled — tab is ASCII — but the unicode half of the finding was real.) - use-cases.ts: the gate read `keyAccess.knownKeys` as "a workspace secret exists under this name", but that set only covers names with an `env_workspace` credential row. A legacy value written before the ACL existed has no row and still wins at run time, so it fell through to the personal branch and handed a non-admin the reference map for exactly the oldest keys. It now reads the authoritative `workspace_environment.variables` map through a new `hasWorkspaceEnvValue`, which is documented against `knownKeys` so the two are not confused again. `getWorkspaceEnvKeyAdminAccess` keeps its existing contract — its `knownKeys` still answers the ACL question its other callers ask. - secret-references-panel.tsx: a personal secret shadowed by a same-named workspace variable could open the view (its owner may read their own Logs) but References always hit the workspace refusal and rendered a generic load error — a tab offered in a state where it cannot succeed. The refusal is correct; the tab now states the shadowing instead of asking for a map it will be denied, reusing the wording the detail page already shows. No request is made in that state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): re-check the reference gate's volatile input after the scan Review round 3. The name-resolution gate reads whether a workspace value exists, then scans. A workspace secret created between the two makes the map now in hand admin-gated, so a personal owner could receive it without workspace-secret administration. The window is small and the data is derivable — a workspace member can already open every workflow and read its `{{KEY}}` references — but the gate's stated contract is that references follow the same predicate as revealing the value, and a point-in-time check that can be overtaken does not honour that. An advisory lock or a snapshot transaction would serialize a read-only view against secret writes for it, which is the wrong trade. Instead the one volatile input is re-read after the scan and the request fails closed if it flipped. `requireSecretReferencesReadAccess` now reports which branch authorized: an `admin` grant holds however the name resolves and pays nothing, while a `personal` grant — the only one resting on absence — is re-checked. A non-admin loses nothing they were entitled to keep; the request is refused the way it would have been a moment later. Adds a `listSecretReferencesUseCase` suite covering both denial paths, the legacy value, the personal owner, the admin short-circuit, and the race itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): accept JSON-escaped whitespace in the reference prefilter Review round 4. The prefilter reads a `::text` rendering of a JSON column, and `jsonb::text` renders a real tab inside a string value as the literal pair `\` `t`. `t` is alphanumeric, so `[^[:alnum:]_]` could not consume it and the row was discarded before `ENV_REF_PATTERN` ever ran — the References tab omitting a live reference and reporting `truncated: false` while doing it. Round 3's fix was verified against a raw text value rather than the JSON rendering, which is exactly why it looked correct: `E'{{\tAPI_KEY\t}}'` matches, `jsonb_build_object('v', E'{{\tAPI_KEY\t}}')::text` does not. The gap between `{{` and the name now accepts three encodings at once — raw characters (covering every Unicode space, which Postgres `[[:space:]]` misses), JSON two-character escapes, and `\uXXXX` (how a vertical tab survives the same rendering). Verified against the real jsonb rendering: tab, newline, carriage return, vertical tab and form feed all recover, U+00A0 / U+3000 / space / plain keep matching, and `{{API_KEY_TEST}}`, `{{MY_API_KEY}}` and prose are still rejected — so the row cap keeps meaning what it says. Plain-text columns (`custom_tools.code`, `mcp_servers.url`) carry no JSON escaping, but tool code is JavaScript source and can contain the same escape sequences literally, so the one predicate is right for every column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(secrets): land the References link on the block, and name its field Feedback round. - Logs leads the tab strip. It was already the default tab; the order now says so. - The usage view drops its resource heading for a plain "Usage" title. The back chip already names the secret, so the tile and the subtitle underneath were saying it a second time. `CredentialDetailLayout` gains an optional `title` that renders the same element, class and column position the settings shell gives `SettingsPanel` — which is how the sibling Forks "Activity" view titles itself. Existing callers pass nothing and are unchanged. - A block row now lands on the block instead of the workflow's default framing. The editor had no URL params at all, so `?block=` is its first: read once on arrival, acted on, and stripped. It is a navigation signal rather than canvas state — the carve-out in sim-url-state.md is about pan, zoom, selection and drag, which are socket-synced or high-frequency; this is neither, and it rides in the link so a middle-click or reload keeps it where an in-memory handoff could not. The consuming effect mirrors the note-search reveal in the same file, including the three details that make that one work: read from `displayNodes` so a target arriving before its node mounts is retried on the mounting commit, route selection through `resolveSelectionConflicts`, and latch in a ref. It also claims `userFocusedWorkflowIdRef` the way a node click does, because `onInit` re-reads that inside its own rAF and would otherwise `fitView` over the camera — and that ref is reset by exactly the `workflowIdParam` change a deep link causes. The panel opens for free: `syncPanelWithSelection` already follows selection. `useSearchParams` needs a Suspense boundary and the editor's ancestry has none, so the read lives in a leaf under its own `fallback={null}` rather than wrapping the editor and adding a `loading.tsx` to its mount path. `next build` passes. - A tool-input reference showed `tools-tool-0-code`. Those `{subBlockId}-tool-{index}-{paramId}` keys are documented as an ephemeral, client-only projection of the canonical `tool-input` value and are not meant to be persisted, but older rows carry them — so the scanner reported whichever the record yielded last. They are dropped before scanning, which is right even where the two disagree: `tool.params` is what executes, so a mirror the canonical no longer matches describes a reference that no longer runs. - The row now shows the field's label from the block config rather than its storage id — "API Key", "Tools", "Code", "Bot Token" — falling back to the id when the block or field is unregistered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): make the reference prefilter exactly as tight as the authority Review round 6. The gap between `{{` and the name accepted any non-word character, so `{{-API_KEY-}}` and `{{"API_KEY"}}` matched in SQL while `ENV_REF_PATTERN` rejects them. The previous commit called that free — "costs a candidate row and nothing else" — which was wrong: a candidate row is a slot under BLOCK_SCAN_LIMIT, so enough near-misses sorted earlier exhaust the cap before a genuine reference is read, and the tab reports a live key as unused. That is the same failure the tightening in round 1 was meant to remove, reintroduced by the round 4 loosening that fixed JSON-escaped whitespace. The gap now enumerates exactly the whitespace `\s` accepts, in each encoding it can arrive in: `[[:space:]]` for raw ASCII, `\\[tnrf]` and `\\u000[bB]` for the JSON escapes, and an explicit class for the Unicode spaces Postgres emits verbatim but `[[:space:]]` does not match. That class is generated from a code-point table rather than written literally. Writing it by hand put a run of invisible characters in the source — a reviewer cannot check them, and a formatter or editor can silently mangle them. The table is the readable form and `toPgEscape` renders it. Verified against the real jsonb rendering, 17 cases: raw space, tab, newline, carriage return, vertical tab, form feed, U+00A0, U+202F, U+3000 and an embedded reference all match; `{{-API_KEY-}}`, `{{"API_KEY"}}`, `{{API_KEY_TEST}}`, `{{MY_API_KEY}}`, prose and an across-braces span all do not. Every candidate the SQL admits is now a real occurrence, so the cap counts references and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): cap the reference scan on results, not candidates Review round 7. The prefilter now matches reference syntax exactly, but `remapSubBlocks` filters further on semantics SQL cannot see: it drops dormant canonical members and condition-hidden fields. So a block whose only `{{KEY}}` sits in a hidden field is a genuine candidate that yields nothing, and with the cap counting candidates, enough of those sorted earlier displaced active references out of the answer. Unlike the previous two rounds this is not fixable by tightening the prefilter — no SQL predicate can evaluate canonical modes or field conditions. So the cap moves to what it should have counted all along: blocks REPORTED. Candidates are read a page at a time up to a ceiling far above the result limit, so filtered rows are absorbed as extra reads instead of taking result slots. Paging rather than one large read because the alternative is holding every candidate block's `sub_blocks` in memory at once; peak memory is now one page. `blockId` joins the ordering as a final tiebreak, since OFFSET paging over a non-unique sort can repeat or skip rows across pages — which here would double-report a block or silently lose one. This does not make the scan unconditionally complete, and the ceiling says so: bounded work and guaranteed completeness cannot both hold, so the only real choice is where the bound sits and whether it counts something the reader can see. It now counts results. Query plan re-checked with the OFFSET in place: still an index scan on workflow by workspace, nested-looped into workflow_blocks. Test added that pins the fix — 2,500 prose candidates sorted ahead of one real reference, which the previous cap dropped entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): drop the paging that caused drift, and stop false truncation Review round 8. - Paging removed. It bought headroom and paid with drift: `OFFSET` is positional, so a block renamed, inserted or deleted between page queries shifts the result set, and the scan skips a live reference or reports one twice. That is a worse failure than the one paging was added to fix, and it was self-inflicted last round. Candidates are read in one statement again — one statement is one snapshot, so neither skew nor duplication is possible — with the ceiling lowered to 4,000 so a single read stays a sane amount of memory. Result-capping survives, which was the actual point: filtered rows are still absorbed as extra reads rather than taking result slots. - `truncated` no longer fires on an exact landing. The block path now uses the limit-plus-one read and strict `>` the resource paths already used, so a scan that ends precisely on a bound reports complete instead of warning about references that were all returned. - The deep-link target is released when its block does not exist. It was cleared only on a match, so a link to a since-deleted block left the id set with the param already stripped: the effect re-checked on every canvas update forever and shadowed a later link to the same block. Once any node has mounted the canvas is populated, so an id still absent is gone and the target is dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): gate the deep-link release on the workflow being ready Review round 9. Round 8 released a deep-link target once `displayNodes` was non-empty, reading that as "the canvas is populated, so a missing id is deleted". It is not: arriving from another workflow the store still holds that graph, so nodes are present while the linked workflow is still hydrating — and a valid `?block=` target was dropped before its own blocks ever mounted. The file already had the right predicate. `isWorkflowReady` pins `hydration.phase === 'ready'`, `hydration.workflowId === workflowIdParam` and `activeWorkflowId === workflowIdParam`, which is exactly "the graph now loaded is this workflow's". Absence is only conclusive under that, and a node count never was — it says something mounted, not whose. This is the second fix to this release condition in two rounds, both from guessing at a readiness signal instead of using the one the component already computes for the same question. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 17:30:17 -07:00
Waleed	b440f4ec11	improvement(mship): preview recent runs in @ mention, with the logs icon (#6951) ... The Logs family flooded the `@` picker with up to 50 near-identical rows, named after their workflow and drawn with the workflow icon, so they read as workflow snapshots and buried every other family. - Preview the 5 most recent runs while the query is empty; typing still searches the full fetched set. The cap spans the workspace rather than one workflow, so a few background runs cannot evict a run just started - Draw the row with the Logs icon, matching the sidebar, the search palette, and the chip the selection turns into - Trail the row with relative time, and with the dot `Badge` draws at `sm` for a run that did not simply succeed, so runs of one workflow are told apart at a glance - Make `@logs` reach the family, which nothing in a row's text names Mentioning a log also resolved to nothing: a log row is keyed by `id` but its run is addressed by `execution_id`, and the picker sent the former where the server resolves the latter. The run id now rides on the resource and every menu builds that resource through one helper, rather than eight inline literals that each silently dropped it.	2026-08-21 16:30:07 -07:00
Waleed	f0a0970d2f	improvement(tabs): port resource tabs onto the shared TabStrip (#6953) ... Replace the hand-rolled resource tab bar with the shared TabStrip primitive, and generalize the strip where this caller needed more than it offered.	2026-08-21 16:29:47 -07:00
Justin Blumencranz	cd935e8ea4	feat(workflows): generate short machine-nature names (#6906) ... * feat(workflows): generate short machine-nature names * fix(workflows): remove vehicle name terms * feat(workflows): expand generated name vocabulary	2026-08-21 15:57:37 -07:00
Justin Blumencranz	2252ac0ee6	fix(workflows): deduplicate generated workflow names (#6935) ... * fix(workflows): deduplicate generated workflow names * fix(workflows): retry deduplicated name races (#6935) - recompute generated names after workflow-name conflicts - preserve exact-name and unrelated constraint behavior - cover the concurrent-create retry path	2026-08-21 15:29:50 -07:00
Waleed	7b6c58113d	fix(knowledge,tables): recover abandoned dispatches, bound the sweep and the workbook preview (#6945) ... * fix(tables,knowledge): recover abandoned dispatches and bound the sweep Three defects measured in production this afternoon. A dispatcher killed by an OOM left `table_run_dispatches` at `dispatching` forever. Every terminal transition on that table is user- or flow-initiated, so nothing reclaimed the row: four dispatches were stranded in one afternoon, pinning each table's "X running" overlay and blocking re-runs, with no way to clear them from the product. The `table_run_dispatches_watchdog_idx` index has existed for this sweep since the table was created, unused. Liveness comes from a new `heartbeat_at`, stamped by the per-window writes that already advance `cursor` and `processed_count`, so a slow-but-live dispatch is spared however long it runs — the in-process path has no duration ceiling, so ageing from `requested_at` would reclaim live self-hosted work. The sweep reads `COALESCE(heartbeat_at, requested_at)` so rows written before the column stay reclaimable rather than NULL-false forever, and runs as the last arm of the existing stale-execution cron at the same 95-minute window its table-job sibling uses. Rows are cancelled, not completed: the scope never finished. The OOM itself is not a leak. Peak RSS is a flat plateau — 457 MB at 20-45s and 461 MB past 200s, so ten times the duration buys four megabytes — that has crept about two percent per release for a month, from 446 MB in late July to 545 MB, past the 512 MiB `small-1x` ceiling. CPU peaks at 0.19, so the larger preset is bought for RAM alone. `maxAttempts` never covered the kill either: Trigger.dev retries `TASK_PROCESS_OOM_KILLED` only when `retry.outOfMemory.machine` names a preset, and all four runs recorded `attempt_count = 1` while the docstring claimed they resumed from the persisted cursor. The connector stuck-document sweep dispatched without a bound. Its chunk size paced the loop but the candidate query had no limit, so one connector enqueued 2,959 documents in fifteen seconds onto the queue every workspace shares. Nothing was double-billed — those documents were genuinely unindexed — but one connector monopolized the queue, and each dispatch mints a fresh requestId, so the idempotency key differs every pass and none of it deduplicates. Candidates are now taken oldest-first and capped per sync; a deeper backlog is deferred to the next sync rather than dropped. * fix(file-parsers): read officeparser's entry point across module systems `officeparser` is CommonJS — `main: officeParser.js`, no `type`, no `exports` map — so what `await import('officeparser')` yields depends on who built the code. Node and webpack synthesize named exports from `module.exports`, so `.parseOfficeAsync` is there. esbuild, which builds the Trigger.dev worker bundle, puts `module.exports` on `.default` and leaves the named export undefined, and the package is in neither `build.external` nor `additionalPackages`, so it is bundled. Reading the named export directly therefore worked everywhere except the worker, where calling it threw `TypeError: parseOfficeAsync is not a function`. All four parsers treat that as "the library failed" and answer with a scrape of the archive, which returns `degraded: true`, and the document pipeline rejects a degraded parse outright. The visible result was every `.pptx` and legacy `.doc` reporting "No text could be extracted from this file — it may be scanned, image-only, or password-protected", naming a cause that had nothing to do with the fault. 118 pptx and 14 doc failures landed in a single burst when one connector's sync first succeeded after ten consecutive crashes. Resolved in one shared loader rather than per bundler: externalizing the package has to be repeated in every build config this code runs under and regresses silently the day one is missed. The shape handling is split into a pure `resolveParseOfficeAsync` because the failing shape cannot be reproduced by mocking the specifier — Vitest's module-namespace proxy throws on a missing export rather than yielding the `undefined` a real bundle produces, so a test going through `import` can only assert the shape that already worked. That is also why the existing parser suites never caught this: each mocks `officeparser` with a fabricated named export, which presupposes the interop being broken here. * fix(knowledge): bound the workbook preview to the rows it emits `sheet_to_json` allocates from a worksheet's DECLARED `!ref` range rather than its populated cells, and Excel routinely writes an inflated range from stray formatting. The 1,000-row preview cap was applied to the result, so it bounded the emitted string while the allocation it was meant to bound had already happened. An 880 KB workbook exhausted an 8 GB worker; the same content exhausted 16 GB when this ran inside the connector sync. No machine size fixes that, because the allocation scales with a number the file declares about itself — fleet p99 for this task is 691 MB against 8 GB, so this is a cliff, not pressure. Passing the window into the conversion is what makes the cap real. `defval` goes with it: defaulting every cell in the range made each row dense, so allocation scaled with columns x declared rows rather than with populated cells, and because no row was left empty it silently defeated the `blankrows: false` beside it. Reported totals still come from the declared range, so bounding the conversion does not change what the metadata says the workbook holds. The eleven documents killed this way recorded `attempt_count = 1`: `maxAttempts` does not cover `TASK_PROCESS_OOM_KILLED`, which Trigger.dev retries only when a larger preset is named. Adding that escalation is a safety net rather than the fix, and the same gap the dispatcher had. Also corrects the machine comment, which claimed `large-1x` was 2 vCPU / 2 GB. It is 4 vCPU / 8 GB, and believing the stale figure makes a resize look like the answer when the parser is what is unbounded. * fix(tables): keep a cancelled dispatch cancelled when a step claims it `dispatcherStep` reads the dispatch, then awaits the table load before writing `dispatching`. Keying that write on the id alone resurrected a dispatch cancelled inside that window — a Stop-all, or now the stale-dispatch sweep — and the fresh heartbeat it writes would then buy the resurrected row another full window before the sweep could reclaim it again. The race predates the sweep, but the sweep is a new writer of `cancelled` that no user action drives, so it is newly reachable without anyone touching Stop. Re-asserting the status the step already read is the whole fix. * fix(tables,knowledge): spare a live window, and restore the truncation notice A lease needs its heartbeat interval to sit well under its TTL. The dispatch heartbeat is stamped between windows, not during them, and `batchTriggerAndWait` checkpoints the loop for the whole window — so the interval is really "one window", which nothing bounds: the window ends when its cells do, and the in-process path has no ceiling at all. A window outliving the stale threshold had its dispatch cancelled while it was plainly alive. Its cells carry the signal the checkpointed parent cannot — `updatedAt` on every in-flight row execution, written by the cell tasks themselves. Both signals must be stale before a dispatch is reclaimed, so a slow window is spared for as long as its cells keep reporting while a run with nothing beating and nothing executing is still collected. The subquery rides the partial `(table_id, status)` index that already covers exactly those three statuses. Bounding the workbook conversion also made its truncation notice unreachable: the converted length can no longer exceed the window it was compared against, so every sheet larger than the preview cap silently stopped reporting that it had been cut. Compared against the declared row count instead, which is what the comparison meant before the conversion was bounded. * fix(tables,knowledge): act on the claim outcome and scope liveness to the dispatch Three defects, two of them created by the previous round's fixes. Guarding the pending-to-dispatching claim without reading its outcome was the worse half of a fix. When a Stop-all or the stale sweep won the race the row correctly stayed `cancelled`, while the step went on to announce `dispatching`, stamp cells and enqueue a window for it — and an empty window would then reach the unguarded `markDispatchComplete` and overwrite `cancelled` with `complete`. The step now ends when it did not claim the row. The cell-liveness probe was table-scoped, and `table_row_executions` carries no dispatch column, so a live dispatch's cells vouched for an abandoned dispatch beside it and the abandoned row was never reclaimed — turning the stuck overlay this sweep exists to clear into a permanent one. Narrowed to the dispatch's own groups, which it already stores. Two active dispatches over the same groups can still mask each other, but that is the state `markActiveDispatchesCancelled` already prevents. Truncation asks whether the window cut the sheet short — a question about the declared range against the cap. Comparing the converted length to the cap made it unreachable once the conversion was bounded; comparing the declared count to the converted length then reported truncation for any sheet merely containing blank rows, which are now skipped rather than defaulted into existence. * fix(tables): scope dispatch liveness to its rows, not just its groups The previous round narrowed the cell-liveness probe to the dispatch's groups on the reasoning that two active dispatches over the same groups cannot coexist, because starting a run cancels prior work on its scope. That reasoning was wrong. `cancelPriorRuns` in `workflow-columns` requires `isManualRun`, so auto-fired runs never cancel anything, and the per-row path is explicitly a no-op for dispatch cancellation. Same-group coexistence is ordinary. A dispatch that names rows now only accepts liveness from those rows, which covers the auto-fired and row-scoped runs that reach this state. What remains is two table-wide dispatches over the same groups, where nothing in the row execution says whose work it is; closing that needs a `dispatch_id` column on `table_row_executions` threaded through six write sites, including the shared cell-write path every cell task uses. That residue is a delay rather than a permanent mask — the live dispatch's cells stop updating when it finishes, and the next sweep after a quiet window reclaims the abandoned row. * refactor(tables): name the dispatch liveness predicate and bound its fan-out Extracts the cell-activity check into `hasRecentCellActivity`, so the stale predicate reads as its two conditions — nothing beating, nothing executing — rather than a twenty-line SQL blob nested inside an `and()`. No behaviour change; this is the code three review rounds found defects in, and being able to read it is what makes those defects findable. Bounds the terminal-event fan-out with `mapWithConcurrency`, matching how the scheduler already fans out. The sibling cancel paths emit over one table's dispatches; this sweep can carry a whole tick's worth across many tables, and each event is its own write. Also repairs the test that covers it. `collectChunks` walks into the `tableRowExecutions` table object the fragment interpolates, so every column name appears in the chunks whether the predicate references it or not — the group, row, table and timestamp assertions all passed with their predicates deleted. Matching the literal SQL instead makes them fail, which mutating each clause now confirms. * fix(tables): make the row bypass NULL-safe and guard the post-wait completion `jsonb_typeof(scope -> 'rowIds') <> 'array'` was the table-wide bypass, but a table-wide dispatch has no `rowIds`: the extraction is SQL NULL, `jsonb_typeof` returns NULL, and `NULL <> 'array'` is UNKNOWN rather than TRUE. The bypass never fired, so no live cell could satisfy the probe and the sweep reclaimed exactly the long-running table-wide dispatches the row filter was added to protect — inverting it. `IS DISTINCT FROM` is the NULL-safe form, and the same pitfall is already handled with `coalesce` in `markActiveDispatchesCancelled`. `completeDispatch` also wrote through the unguarded `markDispatchComplete`. Both its callers run AFTER the window's wait, so a Stop-all or the sweep landing during that wait leaves the row `cancelled` and the write overwrote it with `complete`, publishing a completion event after the cancellation one. The claim guard cannot cover this — the cancel arrives long after the claim. It now goes through `completeDispatchIfActive`, which already exists for exactly this, and emits nothing when the transition does not land. * fix(knowledge): give connector sync logs a retention pass Nothing pruned `knowledge_connector_sync_log`, so it grew by one row per sync run forever — a connector on a fifteen-minute interval writes about 35,000 rows a year by itself. That cost lands on `loadPreviousListingObservation`, which reads the newest `completed` row per connector through an index covering `connector_id` alone, so every retained row makes the sort behind the deletion-safety corroboration slower. Added as another arm of the cleanup cron, batched the same way as its two sibling prunes. Two `exists` guards are load-bearing rather than defensive: the newest row per connector always survives, and so does the newest `completed` one, because that is the row `loadPreviousListingObservation` reconstructs the previous listing from — and that reconstruction decides whether a suspect listing is corroborated, i.e. whether reconciliation may delete documents. Pruning it would silently change deletion behaviour. `started` rows are never eligible; they are in flight or waiting on the scheduler's own sweep. * fix(tables): funnel every post-claim completion through the guarded write The empty-window exit still wrote through the unguarded `markDispatchComplete`, and it runs after the claim like the other two — so a cancel landing during its window query was overwritten with `complete`. Shorter window than the two post-wait exits, same defect, and leaving one of three unguarded is how this came back twice already. All three now route through `completeDispatch`, so the guard lives in one place and covering it once covers every exit. The redundant test for this path went with it: it could not be made to fail against the mock, and a test that cannot fail is worse than none — the guard is held by the test on the shared funnel. * fix(tables): bound how long cell activity may spare a dispatch The liveness probe cannot tell whose cells it is looking at when two table-wide dispatches share a group, because `table_row_executions` carries no dispatch column. On a quiet table that is only a delay — the neighbour finishes and the next sweep reclaims — but a busy table with continuous auto-fired work can keep an abandoned dispatch masked indefinitely, which is the stuck overlay this sweep exists to clear. A ceiling bounds it: past a day without a heartbeat, a dispatch is reclaimed whatever its cells are doing. That is safe because a live dispatch stamps its heartbeat between windows regardless of cell activity, so only a single window outliving the ceiling could be reclaimed wrongly, and no window lasts a day on any path — the Trigger.dev run ceiling is ninety minutes. The real fix is a `dispatch_id` on the executions row. Threading it through the patch layer and the upserts underneath it is a change to the hottest write path in tables and belongs in its own review, not on the sixth round of this one. * refactor(tables): give the stale predicate one definition of "last beat" `COALESCE(heartbeat_at, requested_at)` was written twice — once for the stale threshold and again for the absolute ceiling — so the two could drift into disagreeing about what proof of life means. One `lastBeat` fragment, one `notBeatingSince(cutoff)` helper, both cutoffs expressed through it. Also corrects the ceiling's comment: it triggers a day past the stale threshold, not a day past now. * fix(tables): delete the unguarded completion rather than guard it a fourth time The two pre-claim exits — table missing, no target groups — still wrote through `markDispatchComplete`. Last round I argued they run before the claim, "where forcing a terminal state is the intent". That was wrong twice over: the table lookup is awaited, so a cancel lands in that window like any other, and a dispatch cancelled mid-lookup has not completed its scope any more than one cancelled mid-window has. Routing them through `completeDispatchIfActive` left `markDispatchComplete` with no callers, so it is gone. That is the part worth having: this is the fourth place the same defect appeared, each time because an unguarded writer was sitting there to be reached. With it deleted, `completeDispatchIfActive` is the only way to complete a dispatch and the class cannot recur. * fix(tables): re-read the dispatch before committing a window Several round trips separate the claim from the enqueue — the window query, the executions prefetch, the tombstone filter — and nothing rechecked the dispatch across them. A Stop-all or the stale sweep landing in that gap had the step stamp cells and run a whole window for a dispatch already recorded as cancelled; the existing recheck sits after the window, which is too late to prevent it. Mirrors that existing check on the other side of the enqueue. It narrows the gap to a single statement rather than closing it — a cancel arriving after this read still races the enqueue, and no check can fix that. The cell-level `cancellationGuard` and the `isExecCancelledAfter` tombstone filter are what catch the remainder.	2026-08-21 15:02:12 -07:00
Vikhyath Mondreti	81ebb37563	fix(delivery): stable provider idempotency tokens, and correctness fixes around them (#6943) ... * fix(delivery): stable provider idempotency tokens, and correctness fixes around them Six independent fixes found while investigating a Slack transport failure. None of them are that failure; all of them are live. **Money writes could be delivered twice.** Square (8 tools), Brex (5) and Outlook Calendar minted their provider idempotency token with `generateId()` at request-build time. That is stable inside the transport retry loop — `prepareToolRequest` runs once above it — but a BLOCK-level retry re-enters the handler and mints a fresh one, defeating the provider's dedupe. A builder ticking "retry" on a Square block turned a committed write into a second card charge, and a Brex one into a second money transfer. Tokens now come from `deriveDeliveryKey`, a pure function of execution + block + tool + invocation, so every retry layer derives the same value. Stripe joins them: all ~50 tools previously sent no `Idempotency-Key` at all. The comment on `brex/create_transfer.ts` claimed a fresh key per transfer *prevents* duplicate money movement. That is inverted — fresh per *attempt* is what permits it — and is corrected here. **`invocationId` is required, not optional.** Deriving from `executionId` alone would be worse than the bug: five loop iterations paying five invoices would share one token, the provider would honour the first and silently drop four real payments, and it would look like five successes. Also: - `INTERNAL_API_BASE_URL` is now ignored on Trigger.dev workers. It names a route that resolves only inside the app container, and several modules run in both runtimes — `guardrails/mask-client.ts` says so in its own TSDoc — so setting it produced `PII redaction failed: Unable to connect` on every worker-side redaction. Mirrored into `packages/testing`'s urls mock, which reimplements the function and would otherwise have diverged. - `engines.bun` raised to >=1.3.14. Measured on 1.2.15, which the old floor permitted: a fully-delivered POST is silently replayed and the caller sees 200. - CloudWatch `put-metric-data` pinned to `maxAttempts: 1`. The AWS SDK default of 3 already replayed it, and `PutMetricData` aggregates rather than overwrites, so a duplicate silently corrupts the customer's metric series and alarm thresholds. - `webhookIdempotency` given a bounded in-progress lease. An untimed run held a SEVEN DAY lease while concurrent duplicates polled it once a second. Verified: `tsc --noEmit` clean, 30,100 tests pass. * fix(ci): restore staging files the branch split had reverted, and format Three problems, all from assembling this branch by checking paths out of a WIP branch built on an older `staging`. Anything `staging` changed since that base came back as a revert. - Root `package.json` had lost the `opentype.js` / `@types/opentype.js` dependencies `staging` added, which desynced `bun.lock` and failed `bun audit`. It also carried a `check:outbound-delivery` script belonging to other work. Every `package.json` is now taken from `staging` with only the `engines.bun` line re-applied. - `executor/utils/block-data.test.ts` — a 77-line file `staging` added — was deleted outright. Restored. - `executor/handlers/generic/generic-handler.test.ts` had lost a test `staging` added. Restored, with only the one `blockId` assertion re-applied. Also formats `keyed-invocation-identity.test.ts` and sorts imports in `internal-api-base-url.test.ts`, which is what `lint:check` failed on. * fix(providers): thread the model's tool-call id into keyed tool execution `prepareToolExecution` accepted an `invocationId` but no provider supplied one, so a keyed tool invoked through an agent always hit the incomplete-context fallback and minted a fresh token — leaving Stripe, Square, Brex and Outlook Calendar writes able to double-deliver under the hosted-key retry layer even though the block path was fixed. The id is now a positional parameter rather than another optional field on `request`, so a provider that cannot supply one fails to compile instead of silently falling through. 22 of the 27 call sites already had the OpenAI-shaped `toolCall` in scope; `tsc` identified the other five, of which Anthropic (`toolUse.id`) and Bedrock (`toolUse.toolUseId`) name it differently. Gemini is left deliberately unthreaded and documented: its function-call parts carry no model-supplied identifier — the streaming loop has to synthesize a local one — and a positional index would not survive the model re-emitting the call. A token that only looks stable is worse than the loud fallback, which names the missing fields. * fix(executor): keep execution order monotonic across a resume `executionOrder` is not carried in the pause snapshot, so a resumed run restarted the counter at 0 and a loop or parallel body executing on both sides of a pause could reuse a pre-pause value. That was cosmetic while the number only ordered logs. It stops being cosmetic once identity is derived from it: a `keyed` tool takes its provider idempotency token from this value, so two distinct writes would present the same token and the provider would silently drop the second. Suppressing a real payment is worse than the duplicate the token exists to prevent, because it looks like success. The counter is now seeded from the highest `executionOrder` among the restored block logs rather than from a new snapshot field, so snapshots written before this change are repaired on resume instead of needing a migration. * fix(providers): thread the tool-call id through the streaming loops too The previous commit only reached call sites written as a single-line three-argument call. The streaming loops are formatted across lines, so openai-compat (Groq, DeepSeek and everything else routing through it), Anthropic and Bedrock still omitted the id and kept falling back to a fresh token. Both Gemini paths now pass `part.functionCall?.id` — the RAW model id, not the `ensureToolCallId` value used for stream events. That helper allocates an execution-local id when Gemini supplies none, and it is freshly allocated per attempt: passing it would complete the keyed context, silencing the "could not derive" warning, while leaving the token just as unstable. Gemini frequently omits the id, in which case this is `undefined` and the loud fallback stands. All 27 call sites are now covered. * fix(providers): make the tool-call id argument required, not optional The TSDoc claimed a provider that cannot supply an id would fail to compile, but the parameter was declared `toolCallId?: string` — so a new call site could omit it entirely, typecheck, and silently take the unstable-token path the positional parameter exists to close. The comment promised a guarantee the type did not enforce. It is now `string | undefined`: required in position, nullable in value. A provider with no model-supplied id must pass `undefined` explicitly and take the loud fallback, rather than being able to forget the argument. All 27 existing call sites already pass it, so this is enforcement only. Verified by deleting the argument at one site: `tsc` rejects it.	2026-08-21 13:35:20 -07:00
Vikhyath Mondreti	85a3226678	fix(security): stop redirects replaying request bodies and leaking credentials (#6941) ... * fix(security): stop redirects replaying request bodies and leaking credentials `secureFetchWithPinnedIP` passed its options straight into the redirect recursion, so a 301/302/303 replayed the original method and body — delivering a non-idempotent write twice — and forwarded `Authorization` and every other caller header to whatever origin the upstream named. `followRedirectsGuarded`, a hundred lines above it in the same file, already had the correct RFC 9110 rules. The two had drifted, and the drift is the bug. Both now route through one `resolveRedirectHop`: - 303, and 301/302 on POST, degrade to a bodyless GET and drop the entity headers that described the removed body. - A cross-origin hop drops every caller header, not just `Authorization`. - A cross-origin hop that would forward a body is refused. `stripAuthOnRedirect` still narrows same-origin hops for endpoints that redirect to a target carrying its own signed URL. Verified by stashing the fix and re-running: 4 of the 6 new tests fail against the old code. The 2 that pass either way cover same-origin behaviour that was already correct. * fix(api): preserve HTTP redirect compatibility	2026-08-21 13:33:52 -07:00
Siddharth Ganesan	58aa6379e0	feat(cli): add chat command (#6937) ... * feat(cli): add chat command * fix(cli): harden chat command execution	2026-08-21 12:59:08 -07:00
Vikhyath Mondreti	cd0516cade	fix(billing): separate Enterprise reporting periods from Stripe terms (#6942) ... * fix(billing): separate Enterprise reporting periods from Stripe terms * fix(billing): reconcile accepted legacy intents * fix(billing): keep accepted legacy intents fail-closed * fix(billing): reconcile accepted retired intents * fix(billing): retire invalid legacy intents	2026-08-21 12:52:49 -07:00
Vikhyath MondretiandSim Pi Agent	6a45b0d4a6	feat(library): Best AI Agent Platforms for Connecting Your Existing Tools (#6946) ... Co-authored-by: Sim Pi Agent <pi@sim.ai>	2026-08-21 12:45:59 -07:00
Theodore Li	4c41fc6c3e	fix(setup): bump sim-setup to 1.0.1 (#6944)	2026-08-21 12:24:06 -07:00
Waleed	6b7fd1a88d	fix(webhooks): stop the generic webhook publishing a closed output schema (#6939) ... * fix(webhooks): stop the generic webhook publishing a closed output schema Declaring outputs on the generic webhook trigger did not add three reference completions — it made those three the only legal fields on the block. `collectBlockData` registers any non-empty output declaration as an exhaustive schema, and `resolveBlockReference` then throws `InvalidFieldError` for any reference outside it that resolves to `undefined`. A generic webhook receives whatever the caller sends, so every workflow reading a body field started failing the moment a delivery omitted that field, instead of resolving to `undefined` and letting the condition evaluate falsy as it always had. Revert the declaration to `{}` and record why it has to stay that way. The request metadata is still merged into the workflow input by the provider's `formatInput`; it is only undeclared, which is what keeps the shape open. Offering these as editor completions needs a way to mark outputs as hints rather than a closed schema — a change to `getRegistrySchema`, not to this list. Pins the behavior at the executor level rather than on the trigger config, since the config assertion is what passed while the block was broken. * chore(audits): re-record the workspace module-graph baseline `check:tool-registry-boundary` fails on CI for any branch right now: the knowledge page measures 2255 modules against a 2209 baseline, one module past the max(25, 2%) allowance. It passes locally at 2253, which is why it only shows up in CI — the two platforms resolve a couple of modules differently, and the route happened to sit inside that gap. The drift is not from any one change. 26 of the 34 recorded routes have grown since the baseline was last written, by up to +44. Measuring this branch's route with and without its own diff gives 2253 either way, so it contributes nothing; it is just the branch that happened to cross the line. Re-records all 34 entries, which is what the script prescribes. No gateway was added or removed on any route — only the module counts moved — so the boundary this audit exists to protect is unchanged and the first assertion, that the tool registry stays out of every workspace page graph, still passes. Worth a separate look at why the workspace pages have grown this much; this commit only stops a stale number from blocking unrelated work. * chore(tests): give the block-data test helper an explicit return type	2026-08-21 11:32:57 -07:00
Waleed	00d8a3fbfe	fix(knowledge): refund a processing attempt whose dispatch never happened (#6938) ... * fix(knowledge): refund a processing attempt whose dispatch never happened `markDocumentsQueued` spends one attempt from the processing budget on every dispatch, and `clearDocumentsQueued` already withdraws the queue stamp on the one path that proves nothing was dispatched — but it left the attempt spent. The budget exists to stop re-billing a document that keeps failing the same way in processing. An attempt that never reached a worker teaches it nothing, so an infrastructure outage — a Trigger.dev region error, an exhausted quota — burned the allowance without a single run. MAX_PROCESSING_ATTEMPTS such outages dead-letter the document, and the connector sweep filters on `processingAttempts < MAX_PROCESSING_ATTEMPTS`, so automatic recovery stops for a document that was never processed once. Refunded in the same guarded statement as the stamp, so it can only ever give back the charge this call made, and floored at zero. Also corrects the stale-lock TTL doc, which still described the reclaim as measuring `updatedAt` after it moved to `COALESCE(syncLockLeaseAt, updatedAt)`, and the attempt-budget rationale, which predates the refund. * chore(audits): re-record the route module-graph baseline Every workspace route had drifted past what the baseline records, uniformly: the shared `[workspaceId]/layout.tsx` grew +28 and each route inheriting it grew +27 to +29. The knowledge page carries +18 of its own on top, which put it +46 over a +44 allowance and was the only entry actually failing. Bisecting the growth across the last five commits on staging shows 2249 → 2251 → 2252 → 2253 → 2255 — one or two modules per unrelated feature PR, not a single regression dragging in a fat dependency. That is the organic creep the `max(25, 2%)` ratchet is meant to absorb and the re-record is meant to settle. Counts only: all 34 entries are preserved and every entry's gateway set is unchanged, so the registry-reachability gate and the per-entry ratchet keep exactly the strength they had.	2026-08-21 11:21:26 -07:00
Theodore Li	a0a14383f0	feat(enrichments): add LinkedIn profile lookup (#6926) ... * feat(enrichments): add LinkedIn profile lookup * fix(enrichments): request Findymail profile data * fix(enrichments): avoid Findymail profile charge * refactor(enrichments): omit Findymail profile option	2026-08-21 14:05:23 -04:00
Siddharth Ganesan	e578cfe5dd	feat(files): mship file writing improvements- #6933 (#6933) ... feat(files): mship file writing improvements (#6933)	2026-08-21 10:31:51 -07:00
Siddharth Ganesan	29acfb10ff	improvement(mship): mship file fixes (#6918) ... * improvement(files): preserve editable page source on round trips * improvement(pages): harden previews and mobile rendering * fix(copilot): keep generated API keys accessible * fix(files): rewrite page sources after upload finalization * fix(pages): expose compile diagnostics through VFS * fix(copilot): surface classified tool access errors * fix(copilot): surface actionable server tool errors	2026-08-21 10:09:31 -07:00
Waleed	63569a2459	fix(connectors): count hard-kill failures and cap deletion blast radius (#6909)	2026-08-21 09:41:21 -07:00
Vikhyath MondretiandClaude Opus 5	01795e1ed2	fix(combobox): report every open, not just the dismissals Radix initiates (#6931) ... The Combobox owns its `open` state but renders a controlled Radix `Popover` (`PopoverAnchor` + `open={open}`, no `PopoverTrigger`), and the consumer's `onOpenChange` hung off that Popover alone. A controlled popover reports only transitions it initiates itself, so outside-click and Escape arrived and nothing else did: the trigger, the chevron, focus, Enter/Space/ArrowDown, and select-to-close were all the component's own `setOpen`, invisible from outside. Every consumer refreshed on open or reset on close, so the damage stayed quiet — the credential selectors, MCP tool selector, workspace-file picker, connector modal, and the sub-block dropdown's remote option list simply never refreshed when opened. Then #6881 gated the agent block's `toolGroups` on the same signal to keep the group build off the canvas's hot path, and a picker that could not learn it was open built nothing: the dropdown rendered "No tools found" over the full block registry. Every transition now goes through one `changeOpen`, which Radix's own `onOpenChange` also feeds, so `setOpen` has exactly one caller and the callback cannot be missed. It dedupes through a ref, because several paths both close and let the popover dismiss — a redundancy the raw setState absorbed silently but a consumer callback would not — and reading that ref lets the toggles resolve their next value without re-creating their handlers on every open. Tests cover the transitions Radix never reported (trigger click both ways, keyboard open, Escape) and the consumer shape that made this visible: options supplied only once the dropdown says it opened must render, not the empty state. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 08:40:30 -07:00
Waleed	8be58682ca	fix(custom-blocks): give a custom block's logo a tile its header chip can paint (#6929) ... A custom block with an uploaded logo declared bgColor 'transparent', which meant "the image is the whole tile" — true on tile surfaces, where the image fills the box, but wrong for the canvas node header. The header chip sets its label beside the icon rather than under it, so an unpainted chip left the label nothing to contrast: perceivedBrightness('transparent') is null, the foreground fell back to white, and the block name rendered white on the white card. The header showed a bare logo where every other block shows a chip. Custom blocks with an image now wear the same white plate as every other light-tiled provider, so they read as an ordinary integration everywhere. Icon and tile resolve together from one precedence rule, so a block can never paint a tile its icon disagrees with.	2026-08-21 08:20:03 -07:00
Vikhyath Mondreti	fd5ea3e399	fix(billing): cast outbox payloads for JSONB operators (#6928)	2026-08-20 23:22:27 -07:00
Vikhyath Mondreti	ab66ce9149	fix(admin): harden dashboard billing operations (#6914) ... * feat(admin): make dashboard billing operations durable * fix(admin): close dashboard recovery gaps * fix(admin): preserve member operation lock order * fix(admin): harden durable operation boundaries * fix(admin): report member operation failures accurately * test(invitations): cover locked seat admission * feat(enterprise): gate owner activation on acceptance * fix(enterprise): keep owner activation recoverable	2026-08-20 22:31:11 -07:00
Waleed	dbbe99e473	fix(integrations): validation pass over Crunchbase, PitchBook, and CB Insights (#6925) ... * fix(crunchbase): widen tier-gated collection allowlists and cap the deleted feed The deleted-entity, autocomplete, and fields-metadata allowlists each held only the collections the narrowest package tier publishes, so requests valid on a richer package were rejected locally before any request went out. An Advanced Financials key could not read the funding-round deletion feed at all. - deleted-entity collections: 9 -> the 14-collection union across all tiers - autocomplete and fields-metadata: 14 -> all 43 collections - clamp the deleted feed to its documented max of 25, not Search's 1000 - offer "All collections" so the cross-collection feed stays reachable - name the richer-tier card additions instead of presenting the base set as exhaustive Also rewrites a test that asserted the broken behavior and tightens a substring URL assertion that passed on the value it was meant to reject. * fix(pitchbook): stop a rejected API key reaching block output and logs PitchBook's 401 body echoes the submitted key back inside `message`. No PitchBook tool declared an `errorExtractor`, so the failure fell through to the generic chain, whose first entry returns `data.message` verbatim — putting the credential in the block error, the run log, and any agent context reading the failure. The existing scrubber sat in `transformResponse`, which never runs on a non-ok response. - add a `pitchbook-errors` extractor that replaces the unauthorized message with a fixed string, and wire it through all 91 tools - the extractor returns undefined unless the body carries a `message`, so a foreign 401 on the shared fallback chain is never labelled a PitchBook failure - correct `investor_preferences.preferredIndustry` to the shape the API returns - make `company_industries.emergingSpaces` opaque; its item shape is undocumented - reject a non-list of article ids instead of throwing a bare TypeError * fix(cbinsights): reject malformed input instead of silently rescoping a billed query CB Insights is metered, so a filter that fails to parse must fail the request — dropping it does not narrow the result, it charges for a query the caller never asked for. - reject an unrecognized boolean rather than dropping it, which had been widening a VC-backed firmographics search - reject a non-numeric limit instead of falling back to the endpoint default - reject non-text filter entries instead of stringifying them to "[object Object]" - accept only asc/desc for sort direction; a typo had returned the bottom of a metered result set as though it were the top - treat a whitespace-only numeric bound as unset, not as zero - drop `totalHits`/`totalHitsRelation` from list business relationships; that endpoint reports no total, so both were permanently null - trim `nextPageToken`, matching the id fields Also moves the token cache onto `lru-cache` per the in-process caching rule, replacing hand-rolled TTL arithmetic and a manual prune. * chore(harmonic): drop the team-key help text from the credential descriptor * chore(tools): regenerate tool metadata for the validation fixes * fix(tools): redact the retained error body, not just the message Scrubbing the extracted message left the raw provider body reachable: `createTransformedErrorFromErrorInfo` attaches `errorInfo.data` to the thrown error and the executor surfaces it on the failed tool's `output.data`, so a PitchBook key rejected with an echoing 401 still reached block output and agent tool results via `output.data.message`. - add an optional `redactData` to the error-extractor contract, so an extractor that exists because a provider echoes a credential can replace the body too - retain `redactErrorData(errorInfo, extractorId)` in place of the raw body - PitchBook replaces only the unauthorized body; every other failure is untouched - cover the executor path itself, since asserting on the redactor directly still passes when nothing is wired to it	2026-08-20 22:22:28 -07:00
Vikhyath Mondreti	aca152c7fb	fix(env): put runtime config on <html> so client reads can't outrun it (#6923) ... * fix(env): put runtime config on <html> so client reads can't outrun it The inline script that assigns `window.__ENV` is rendered from the component tree, so it lands ~13KB after the `<script async>` bootstrap tags React emits in the preamble. `appBootstrap` calls `hydrate()` synchronously whenever `self.__next_s` is empty — which it always is now that the script is a plain tag rather than a `beforeInteractive` one, that queue having been the only thing sequencing the assignment ahead of hydration. So module bodies and the first commit could both read env before the assignment landed: the socket URL fell back to the page origin for the life of the document, `getBaseUrl()` threw, and every module-scope flag in `env-flags` froze on nothing. Carry the same snapshot on `<html>`, the document's first tag, and read it in `getEnv` when `window.__ENV` is not yet assigned. Parsing is memoized against the raw attribute rather than against having run once, so the cache can never serve a value the document no longer carries. `window.__ENV` stays the public global and the preferred read, and both transports are built from one function so they cannot drift. Alongside: guard the read-only webhook-URL field so a base URL it cannot resolve is a blank field rather than a dead canvas; report what the workflow error boundary catches, which it previously swallowed entirely; and enable PostHog's native exception capture, since error boundaries only ever see their own subtree and chunk-load failures, rejected promises and throws from event or socket callbacks reached nothing. * fix(realtime): count each failed connect attempt once `manager.reconnect()` calls `open()`, whose error path emits `error` — which the socket re-emits as `connect_error` — and then emits `reconnect_error` itself. A failed reconnect therefore reached both handlers and advanced the counter twice, so the outage report tripped on the second real attempt while claiming three. Count in `connect_error` alone: it is the only handler that fires exactly once for both the initial failure and every retry. `reconnect_error` keeps its log line and states why it deliberately does not count. * fix(workflow): let an unresolvable webhook URL fail loudly Reverts the guard added earlier in this PR. A blank row titled "Webhook URL" is a worse outcome than a crash: it explains nothing, and the value is one a user copies into a third-party provider, so any substitute — a guessed page origin, an empty string — is a URL that provider accepts and then never delivers to. The read can no longer come back empty from the hydration race this PR fixes, so reaching it at all means the deployment has no application base URL, which breaks webhook registration and callbacks regardless. The error boundary now reports what it caught, so the throw names its own cause instead of surfacing as an unexplained fallback.	2026-08-20 22:12:36 -07:00
Justin Blumencranz	4ad1d53de9	fix(copilot): sanitize edit_workflow result state before returning it to the agent (#6904)	2026-08-20 20:22:16 -07:00
Waleed	f9eaadfc70	fix(knowledge): give failed documents a grace while their retries are still scheduled (#6924) ... The sweep treats failed as terminal — "the run that produced it has ended" — and that was true when processing ran inline. Since dispatch became asynchronous the worker writes failed and then rethrows, and the processing task retries up to three times, so failed is a resting state between attempts rather than a final one. The sweep gave it no grace at all, so it reclaimed a document mid-retry chain, deleted its embeddings, and re-dispatched with a fresh pass id: two live runs, two indexing passes, two bills, through the one state deliberately left unguarded. Aged from processingCompletedAt, which every failure write stamps, so it reads exactly when the last attempt ended rather than when the document was dispatched. Sized at the queue grace rather than a run-duration bound. What is being waited on between attempts is a re-queue behind the same global concurrency limit, not another run, so on a large backlog the next attempt starts hours after the previous one ended and a run-duration bound would still reclaim live work. The grace itself is now an expression rather than a constant — corpus over queue concurrency, times occupancy, times a contention factor — with each input documented where it can be re-measured. The previous 240 was derived from a 2,600-document corpus and sat below the drain time of the 7,730-document connector it was written for. Rounding is up: too small silently re-bills live work, too large only delays recovery of documents nothing is processing. Also withdraws the queue stamp when every dispatch in a batch fails, so a provably-undispatched document keeps next-sync recovery instead of waiting out a grace it never earned. Scoped to the batch's own ids, to rows still pending, and by compare-and-set on the exact stamp that call wrote, so a concurrent batch that re-stamped a document keeps its grace. Best-effort by design, while the stamp write still throws: a failed stamp means the grace cannot be promised and dispatching anyway is the unsafe direction, whereas a failed withdrawal only delays recovery and must not mask the dispatch error underneath it.	2026-08-20 19:36:52 -07:00
Waleed	1ced0b6065	fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents (#6921) ... * fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents The sweep gave 'processing' a staleness cutoff but gave 'pending' and 'failed' none, and the only cross-run guard was uploadedAt < syncStartedAt, which scopes within a sync but not across them. That was harmless while document processing awaited inline: documents were terminal by the time a sync ended. Since dispatch became asynchronous they sit 'pending' until a worker picks them up, so the next sync deleted their embeddings, reset them, and re-dispatched while the original runs were still executing — and each re-dispatch mints a fresh pass id, so it billed again. Queued documents now get a grace period derived from queue drain time rather than run duration: the processing queue's concurrency is 20 and global across workspaces, so a 2,600-document corpus takes roughly two hours to drain. The existing 45-minute threshold bounds a run, not a wait, and would still reclaim live documents on any corpus over about 900. No column records dispatch time, so the signal is processingStartedAt falling back to uploadedAt. The sweep now stamps that column when it re-dispatches, and the user-triggered retry stamps it instead of clearing it — without both, only a document's first dispatch was protected and the sweep churned once per sync forever on anything that kept waiting. The grace narrows the duplicate-billing window but cannot close it: a re-dispatch mints a new request id and the Trigger idempotency key is scoped per dispatch by design. Closing it durably needs a document-scoped key or a dispatch-generation column, recorded in TSDoc and deliberately not built here. * fix(knowledge): record dispatch time in its own column instead of overloading the start time Two review findings on this PR traced to the same root: the queue grace was reading processingStartedAt, a column that means something else. Externally, a pending row carrying a dispatch timestamp reported a processing start time for work that had not started. Internally, updateDocument sets a document pending and refreshes uploadedAt while leaving the previous run's processingStartedAt in place, and completion never clears it — so the sweep aged a re-dispatched document from a leftover stamp rather than its actual dispatch, and could reclaim it inside the grace window while the earlier queue entry was still live. That reopens the duplicate-billing race this PR closes. processing_queued_at is written on every re-dispatch and read by the sweep; processingStartedAt goes back to meaning what its name says, so its external contract is byte-identical to before this PR. Not stamped on first dispatch: the row is created and dispatched inside the same sync run, so uploadedAt is already accurate there and a guarded write per upload would buy no behavior. The internal document route returns a full table row through a passthrough schema, so the new column would have shipped as an undeclared raw Date. Declared and serialized explicitly instead. * fix(knowledge): stamp the queue time in the dispatch funnel, not at each call site Giving dispatch its own column was not enough. The connector content-update path updates a row in place with pending status and a fresh uploadedAt while leaving every prior-run processing column intact, so a document dispatched once already carried a stale queue stamp, the refreshed uploadedAt was never consulted, and the row aged from a dead run's timestamp — the same bug one level down. "Written on dispatch and only on dispatch" is now structural rather than a convention three call sites remember: the stamp lives in processDocumentsWithQueue, which every one of the nine dispatch paths already goes through, so none can forget it. The sweep's and the retry's own stamps are kept because they land inside the same transaction as their reset, so a document keeps its grace even if the dispatch that follows throws. The funnel also clears processingStartedAt, guarded on the row still being pending so it cannot disturb a worker's compare-and-set. That closes the API-facing half at its source: a pending row can no longer report a start time from any path, including the content-update path no serializer would have seen.	2026-08-20 18:59:31 -07:00
Vikhyath MondretiandSim Pi Agent	2074afe8ed	feat(library): Best AI Automation Tools in 2026 (#6922) ... Co-authored-by: Sim Pi Agent <pi@sim.ai>	2026-08-20 18:42:29 -07:00
Vikhyath Mondreti	9a66fbb774	fix(auth): bound the three unbounded session-policy caches (#6919) ... * fix(auth): bound the three unbounded session-policy caches security-policy.ts and session-policy.ts are read from Better Auth's session create and update hooks, so they run on every session validation. All three caches were plain Maps with a hand-rolled 'Date.now() - fetchedAt < TTL' read and no ceiling: entries were released only by an explicit invalidate, so they grew for the life of the process. membershipCache is the sharpest of the three because it is keyed by user, not organization — one entry per user who ever authenticated on that instance. Move all three to LRUCache, already a direct dependency and the pattern copilot/entitlements.ts and providers/client-cache.ts use. The library owns the TTL and the ceiling; every existing invalidate* keeps working unchanged. Two things to preserve, both now pinned by tests: - membership results keep their asymmetric TTL (a non-member result expires far sooner, so a user who joins through a path this codebase never sees cannot dodge the new org's policy). Expressed as membershipCacheTtlMs rather than an inline ternary, since it is a security property and not a tuning knob. - reads test '!== undefined', because a version is a number and a membership is nullable — a truthiness check would treat both as a miss. security-policy.ts had no test file; adds one covering caching, invalidation, failure fallbacks and the TTL asymmetry. * fix(auth): raise the cache ceilings to a memory backstop getSessionCookieCacheVersion feeds Better Auth's session.cookieCache.version, so these are read on every session read, not just create/refresh. At max: 1000 a busy instance could exceed the live key set inside the 60s TTL and start evicting early — never a wrong answer (a miss is one indexed lookup, exactly the pre-cache behaviour) but a hit-rate cliff on a hot path. Entries are a few dozen bytes, so headroom is nearly free: orgs 1k -> 20k, users 10k -> 100k. That is single-digit MB at worst and puts the ceiling far above any plausible per-instance working set, leaving it as the memory backstop it was meant to be. * docs(rules): write down the caching decision tree The lifecycle-map-vs-TTL-cache distinction, the ceiling-as-backstop sizing, and the fetchMethod-unless-you-need-a-hang-deadline call each took real digging to settle. Recording them so the next cache does not re-derive the same answers — or re-introduce the unbounded tenant-keyed Map this branch just removed. Notes the two non-obvious traps behind that: ttl alone does not bound memory without a ceiling, and React cache() is a no-op in Trigger workers, so a gate that looks free on a settings page is uncached and per-block on the executor.	2026-08-20 18:29:48 -07:00
Waleed	b125cfdc05	fix(knowledge): dispatch document processing from inside Trigger.dev runs (#6920) ... isTriggerAvailable() returns false inside Trigger.dev worker runs, so every connector document has been downloaded, parsed, chunked and embedded inside the sync task at concurrency 5 rather than dispatched to knowledge-process-document. The per-document fan-out is inert in production. Confirmed from a crashed run's spans: the dispatch log line carries backend: "direct", and no knowledge-process-document runs exist for that knowledge base while other knowledge bases dispatched normally in the same window. On the largest connector — 7,730 documents, individual XLSX files over 20MB — that exhausts the heap and OOMs the sync. Inside a run the answer is unconditionally yes: the platform is what is executing the process, so no environment guess can beat the run marker. Two independent signals, because this has now been got wrong twice — the SDK's ambient taskContext.isInsideTask, already load-bearing in getAsyncBackendType for the same carve-out, and a marker written by the global init lifecycle hook, which uses only the documented public surface. Neither reads TRIGGER_SECRET_KEY or TRIGGER_DEV_ENABLED. Resolving true inside a run is safe under either hypothesis about which conjunct was false: if a run process lacks the secret key the SDK rejects the batch trigger and dispatch falls back in-process, which is where a false predicate lands today. The first evaluation per process now logs the resolved inputs, naming which conjunct is false. A predicate unit test would not have caught either shipment; both were deploy-environment differences only observable from inside a worker.	2026-08-20 18:06:26 -07:00
Waleed	d8d983859a	fix(harmonic): correct the destructive-clear copy and stop double-billing enrichment (#6915) ... * fix(harmonic): correct the destructive-clear copy and stop double-billing enrichment Follow-up to #6902, from a final validation pass against Harmonic's OpenAPI and API reference. No endpoint, method, or response mapping changed. - The `personUrns` field told users and the LLM that Clear Net-New Results "clears everything when omitted". That is the raw provider behavior the clearScope guard was added to block; omitting it now throws. The field is shared across three operations, so the wrong sentence was being served as guidance on all of them. - Bulk email enrichment deduplicated LinkedIn URLs before canonicalising them, so `.../in/foo?utm_source=x` and `.../in/foo` were submitted as two people. Harmonic bills per submitted entry, so this spent quota twice and double-counted against the 5,000 cap. Deduplicate after canonicalising. - The two documented bulk-enrichment failures carry a code in `error` and no message anywhere, so quota exhaustion surfaced as "Request failed with status 429". Render the code with its counters instead. Gated on those counters being present: `extractErrorMessage` without an explicit id walks every extractor in order, and claiming a bare `error` key swallowed OAuth's `error_description`. - An enrichment 404 whose detail carries only the URN no longer discards it. - Report the identifier conflict before complaining about an individual URL. - Validate `companyContextUrns` as company URNs, like every other URN param. Forward-compat: `user_saved_search_type` is passed through rather than checked against a fixed set. It is display metadata nothing branches on, and Harmonic owns the enum — an allow-list turned any value they add into a hard failure of the whole list while the selector reading the same rows kept working. Also drops `USER_CONNECTION`, which Harmonic documents as unsupported via the API, removes three superseded types and one dead helper, and extends the "credential never reaches a URL or body" assertion from 4 tools to all 13. * fix(harmonic): fold equivalent profile URLs and stop blank entries failing a batch Review round on #6915. - Blank and non-string `personLinkedinUrls` entries are dropped before the mutual-exclusivity check. Moving the filter after per-URL validation meant a list like `['']` alongside person URNs reported "not both" — naming a conflict the caller never created — or failed the URL parse instead of reading as absent. - Deduplicate on a profile key rather than the canonical string, so `linkedin.com/in/x`, `www.linkedin.com/in/x` and a trailing slash count once. Harmonic canonicalizes and silently deduplicates server-side and reserves quota afterwards, so this does not change what is billed; it keeps Sim's own 1-5000 accounting in step with the set Harmonic accepts, so a batch of equivalent URLs is not rejected locally for a cap it never reaches. Regional subdomains stay distinct: folding `uk.linkedin.com` into `www.` would assert an equivalence Harmonic does not document, and the URL kept for display must remain the one the caller supplied. * fix(harmonic): fold only recognized profile URLs, never pass-through ones Review round on #6915. The profile key added last round was applied to every entry, but it is built from host and path alone. A URL forwarded verbatim for Harmonic to adjudicate keeps its query, port and fragment significant, so two distinct identifiers collapsed to one key and the later one was dropped before Harmonic ever saw it. The key now applies only to a URL `normalizeLinkedinProfileUrl` already canonicalized — where the query and fragment are gone by construction, so folding host and trailing slash is safe. Anything passed through deduplicates on its exact text.	2026-08-20 17:59:28 -07:00
Theodore Li	5b28da1989	fix(tables): accept plain row query predicates (#6916) ... * fix(tables): accept plain row query predicates * fix(cli): show table predicate group syntax	2026-08-20 20:48:47 -04:00
Theodore Li	8eebd6e687	fix(enrichments): project provider failures (#6917) ... * fix(enrichments): project provider failures * fix(enrichments): share Prospeo failure projection	2026-08-20 20:47:17 -04:00
3 people	42f6287911	feat(byok): add organization-wide key inheritance (#6834) ... * feat(byok): add organization key management * feat(byok): inherit organization keys at runtime * feat(byok): add organization scope to BYOK settings * fix(byok): refresh org key state after mutations * fix(byok): hide stale inherited status badges * chore(db): drop colliding byok migration ahead of staging merge Staging independently claimed 0293. Remove ours so the merge is clean; it is regenerated at the next free index right after. * chore(db): regenerate byok migration at 0296 Staging claimed 0293-0295 during the merge; the regenerated SQL is byte-identical to the dropped 0293. * docs(byok): document organization scope, precedence, and the full provider list The BYOK section described workspace-scoped keys only. Add the organization scope, its Enterprise requirement, the per-provider precedence rule, what an entitlement lapse does, and the Pi sandbox exposure. Refresh the provider table from the settings page, which had drifted from 14 to 34 entries. * feat(byok): open organization keys to every organization plan Organization BYOK was gated on Enterprise, but an organization is the only thing that can hold the keys, so every plan that can own an organization should qualify — Pro for Teams, Max for Teams, and Enterprise. Add checkOrgPlan/resolveOrganizationPlan beside the Enterprise pair rather than widening checkEnterprisePlan, so the Enterprise-only gates (Access Control, whitelabeling) are untouched, and restore resolveOrganizationEnterprisePlan to module-private now that BYOK no longer needs it. * perf(byok): cache the organization entitlement, not the key material getBYOKKey runs once per agent block and once per hosted-capable tool call, so a loop over N items resolved N times — and each organization-inheriting resolution paid three sequential billing queries on top of the two key reads. Split the two reads by staleness tolerance. Key rows stay fresh, because revocation must be immediate. The entitlement is a billing gate that tolerates bounded staleness in the harmless direction (a lapsed organization keeps using its own key for <=60s), so cache it per organization with an in-flight share so concurrent blocks issue one query set. The management surfaces keep reading it fresh, so an organization that just upgraded is never told otherwise. Also run the block check and subscription read in parallel inside resolveOrganizationPlan, and carry the resolved scope on BYOKKeyResult so a log line can say whether a run used the workspace's key or an inherited one. * feat(byok): let workspaces store the Z.ai and Cohere keys the runtime reads Both ids were already in the BYOK contract enum and both are resolved at execution time — getApiKeyWithBYOK reaches 'zai' (GLM models are in the hosted catalog, so the BYOK branch runs), and 'cohere' backs both the Embeddings block and Knowledge Base reranking — but neither appeared in the settings list, so there was no way to store the key either path looks for. Cohere had no icon; add one from the official multi-color mark so it stays legible on a light and a dark page. Cohere's embed-v4.0 is kbEligible:false, so the description says 'Embeddings and Knowledge Base reranking' rather than claiming KB embeddings. * improvement(byok): shorten the workspace scope chip to 'Workspace' It sits beside 'Organization', so the scope reads from the pair; 'This' only added width. * fix(byok): do not cache a billing outage as an unentitled organization resolveOrganizationPlan maps a failed billing read to false, which is indistinguishable from a real plan lapse. The entitlement cache stored that, so one transient outage held the gate shut for the full TTL and every inheriting run silently fell back to a metered hosted key — and the cache's rejection path, which exists to prevent exactly this, was unreachable. Give the resolver the onError option its neighbours already have and let the cached read ask for 'throw', so a failure stays out of the cache and the next resolution retries. Behavior for the call that saw the error is unchanged: getBYOKKey still fails closed. Reported by Cursor Bugbot. * fix(byok): propagate the subscription read's failure too The previous commit threaded onError through resolveOrganizationPlan's own catch, but getOrganizationSubscriptionUsable soft-fails to null on its own, so a failed subscription read still arrived as an ordinary 'no usable subscription' and returned a successful false — which the entitlement cache then stored for the full TTL. Thread the option into that call as well. Test it at the billing layer rather than the cache layer: the entitlement test mocks resolveOrganizationPlan wholesale, so it could never have caught this. Verified the new test fails against the previous commit. Reported by Cursor Bugbot. * refactor(byok): cache the entitlement with LRUCache, like copilot entitlements The hand-rolled version reinvented three things the codebase already has a canonical answer for. lru-cache is a declared dependency of apps/sim and lib/copilot/entitlements.ts already caches an entitlement with it — by storing the in-flight Promise, which is what makes concurrent callers collapse onto one resolution with no in-flight bookkeeping at all. TTL and the size bound come from the library. That removes the second Map, the manual eviction (and its interaction with an in-flight entry), and the dead value-while-refreshing state: 23 executable lines. The one thing the library does not cover is dropping a rejected promise so a billing outage is not cached for the TTL, which is kept and pinned by a test that fails without it. TTL expiry is no longer re-tested — that is the library's behavior, not ours, and lru-cache reads its clock at module load so faking timers never moved it. * refactor(byok): coalesce the entitlement read with the shared singleflight lib/concurrency/singleflight.ts is the codebase's coalescing primitive and oauth/credential-service.ts already pairs it with a read-through cache. Adopting that shape fixes a case caching the promise directly did not: a *hung* billing read wedged every caller for the full 60s TTL, where coalesceLocally evicts and rejects at its settle deadline. It also removes the hand-rolled rejection eviction — the cache is written only on the success path, so an outage leaves no entry by construction. The cache now holds booleans, which introduces the one trap worth a test: a truthiness check would read a cached false as a miss and re-query billing on every resolution for lapsed organizations. Pinned. * fix(byok): keep an abandoned entitlement producer from writing the cache coalesceLocally does not cancel a producer it timed out — its docstring says so explicitly — so writing the cache from inside the producer let a late billing result overwrite a fresher answer a retry had already cached, and hold it for a full TTL. Move the write onto the value the caller actually received. A caller that timed out throws before reaching it, so an abandoned producer now resolves into nothing. The test reproduces the overwrite and fails against the previous shape. Reported by Cursor Bugbot. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-20 17:45:28 -07:00
Waleed	e4a1fbeaae	fix(og): put the landing model and integration cards on the brandbook template (#6913) ... * fix(og): put the landing model and integration cards on the brandbook template The five models/integrations cards were the last ones still rendering the retired dark card with the green square-icon logo. They now share the cover renderer the library, docs, and shared-file cards use, which retires og-utils and the assets only it referenced. Captions wrap to two lines so a catalog description survives the move; the eyebrow, pill row, and domain label have no slot in the reference template and are dropped, with the counts they carried folded into the two index captions instead. * fix(og): drop the dead hard-space helper and restore the unknown-context copy withHardSpaces lost its last caller when captions moved onto wrapLines, which packs the non-breaking space inline; its rationale moves to wrapLines with it. A missing context window rendered as a bare "context window", reading as a field label mid-sentence. formatTokenCount already returns "Unknown" for a null value, so the ternary was both redundant and wrong.	2026-08-20 17:34:58 -07:00
Waleed	040f40cd61	fix(connectors): stop rendering running and crashed syncs as successes (#6910) ... * fix(connectors): stop rendering running and crashed syncs as successes The sync-history row derived its state from status literals the engine never writes — 'running', 'syncing', 'error' are connector statuses, not sync-log statuses. The in-progress branch was therefore false for every row that will ever exist, so a live sync rendered as a green success tick for its whole duration and a killed run left a permanent fake success reading "No changes". - Derive the row state from the three statuses the engine actually writes - Narrow the sync-log status contract from z.string() to an enum, and make the render switch exhaustive, so producer drift fails response validation and consumer drift fails the build - Add an interrupted state for a started row older than the stale-lock TTL: it is a crashed run, not a live one, and rendering it as a permanent spinner would trade one false signal for another * docs(connectors): record why the stale TTL is a hard ceiling for the interrupted state	2026-08-20 17:34:03 -07:00
Waleed	451d2ccbde	fix(knowledge): stop billing a document once per processing attempt (#6911) ... The usage sourceReference carried a per-attempt timestamp, which defeated by construction the eventKey deduplication it flows into. With three Trigger attempts plus the stuck-document sweep, one document could be billed four times for a single indexing pass. Key on the dispatch request id instead — already the codebase's name for one indexing pass, fixed across attempts of a run and fresh on every new dispatch — so retries collapse while a genuine re-index still bills. Falls back to the embedding pricing id where no pass id is threaded. usage_log has no retention, so keying on documentId alone would have suppressed legitimate re-indexing permanently. Thread requestId through the Trigger worker, which previously dropped it before the call — the path that actually retries.	2026-08-20 17:33:50 -07:00
3 people	ea70f8dcf1	feat(harmonic): add contact workflow integration (#6902) ... * feat(harmonic): add contact workflow integration * fix(harmonic): sync docs manifest * fix(harmonic): address integration review findings * feat(harmonic): add the missing people endpoints and fix two error paths Extends the integration from 4 to 13 tools, covering every non-deprecated people-scoped Harmonic endpoint, and repairs two defects found by validating the existing tools against Harmonic's OpenAPI and API reference. New tools: - Enrich Person (POST /persons) — the only path from a LinkedIn URL or email a workflow already holds to a Harmonic contact. - Get Person, Get Company Employees — account-based sourcing; employees returns URNs that chain into Batch Get People. - Saved-search net-new results and their acknowledgement, so a monitor stops reprocessing the entire result set on every poll. - Bulk email enrichment: submit, poll, and quota, plus Get Enrichment Status. Fixes: - The error extractor dropped Harmonic's string and object `detail` envelopes. A tool that names an extractor gets no fallback chain, so every FastAPI abort surfaced as "Request failed with status 403". The enrichment 404 also carries the scheduled `enrichment_urn`, which was being discarded — that URN is the only handle on the job, so it is now kept in the message. - The saved-search selector failed the whole dropdown instead of degrading: the response cap was half the sibling value on an endpoint that is unpaginated and returns every saved search with its full query object, and the option ceiling threw rather than truncating. Raised to 1MB and switched to truncate-and-warn, matching the other data-driven selectors. Clearing net-new results now requires an explicit scope. Harmonic treats an absent `entity_urns` as "clear everything", so an empty field would have silently discarded the backlog. Scope deliberately excludes company-side, deal, typeahead, network, and Scout streaming endpoints, and every endpoint retiring on 2026-11-05. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-20 17:12:57 -07:00
Waleed	893e729b3f	fix(og): put the shared-file card on the brandbook cover template (#6907) ... * fix(og): put the shared-file card on the brandbook cover template * fix(og): bound the cover title and caption to the fixed canvas * fix(og): measure cover text with the font's real advance widths An average glyph width under-measures caps-heavy names and over-measures narrow ones, so a viewer-supplied file name could still clip off the fixed canvas. Measure against the same font Satori is handed instead, matching the library cover generator; the tests parse the font independently so the assertion is not made with the estimator it is checking. * fix(og): apply the leading compensation to the title, not the whole footer On the footer the 14px nudge dragged the caption into the bottom padding as well, and the caption has no phantom leading to correct for. Matches how the sibling cover renderers scope the same offset.	2026-08-20 16:28:09 -07:00
Waleed	865f8173ab	feat(affinity): add Affinity CRM integration (#6908) ... * feat(affinity): add Affinity CRM integration Adds the Affinity v2 API as a block with 70 tools, covering 86 of the 87 documented endpoints. Only Send Feedback is omitted — it reports product feedback to Affinity rather than doing workflow work. Endpoint families that differ only by an entity segment are one tool with an entityType param, so companies/persons field, list, row, and relationship reads, the company/person merge endpoints, and entity notes each collapse into a single operation. * chore(affinity): regenerate the docs manifest for the new integration page	2026-08-20 16:27:29 -07:00
Theodore Li	a99f61bee9	feat(api): add v2 resource management endpoints (#6900) ... * feat(api): add v2 resource management endpoints * fix(cli): gate destructive v2 commands * fix(test): update v2 request-slice count * feat(cli): add shared workspace profiles	2026-08-20 18:55:57 -04:00
Vikhyath MondretiandClaude Opus 5	85290693c4	refactor(search): one definition of what a search occurrence is (#6905) ... Follow-up to #6901, closing two places where the workflow search index and the Note card that mirrors it could drift apart. Neither is a live bug; both are the shape that produced one — the card silently disagreeing with the panel about which hit is which, counted in one place and painted in another. THE SCAN. #6901 shared `foldSearchWhitespace` but left the scan around it duplicated: normalize, then non-overlapping `indexOf` stepping by `max(len, 1)`, written out once in the indexer and once in the renderer package. They agree today. They would stop agreeing the moment either grew whole-word matching, diacritic folding, or a regex mode, and the failure is silent. Both now call one `forEachSearchOccurrence` in `@sim/utils/string` — the only place either package can share, since the card renders from a package that cannot import from `apps/*`. THE DECLARATION. The indexer projects markdown escapes only for a field declaring `searchTextFormat: 'markdown'`; the card projects unconditionally, because it cannot read the block registry. Dropping that one line from the Note config would leave them disagreeing with nothing to catch it, so a test now pins it and explains why. Net negative in lines: this deletes a duplicated loop rather than adding a layer. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 15:44:22 -07:00
Vikhyath MondretiandClaude Opus 5	2e111f615c	fix(search): answer a Note match on the canvas card (#6901) ... * fix(search): answer a Note match on the canvas card A workflow search match inside a Note counted towards the result total and then highlighted nowhere: the editor panel renders nothing for a Note, and `clearCurrentBlock` — which the panel calls to refuse one — also cleared the shared `activeSearchTarget`, destroying the very target the card was about to paint. Searching a 15k-character note reported "1 of 6" and moved nothing. The card's read view is now the surface that answers: - A rehype plugin marks every rendered occurrence; the current one is picked by an ordinal counted with the same scan the indexer uses, and travels by context so cycling matches does not re-parse the document. - `<Streamdown>` is keyed on the query. Its memo comparator ignores `rehypePlugins`/`components`, so a plugin change alone cannot re-render it — marks appeared only when something else remounted the card, and then outlived the query that produced them. - The canvas selects, centres and expands the Note, because a compact card resets its scroll region to the top and cannot hold a position deep in its own body. Scrolling to the mark is `scrollTop` arithmetic, never `scrollIntoView`, which would drag ReactFlow's transformed viewport off-frame. - Title matches mark the name too. `activeSearchTarget` is re-published with a fresh identity on most of the search panel's renders, so subscribers take primitives. Holding the object in `WorkflowContent` — the panel's own ancestor — closed an unbounded update loop. Separately, the serializer escaped every underscore, writing `SB\_ACTION\_ROUTER\_SECRET` into the document. CommonMark's intraword rule means that backslash carries no meaning, and search matches the stored markdown, so it made anything with an underscore unfindable in a note that plainly showed it. Dropped outside code regions, where the serializer emits verbatim and a backslash is the author's own character. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(search): honour fences and folded whitespace in Note highlighting Two review findings, both real. The intraword-underscore cleanup guarded code with a pattern that recognised only the shortest delimiter forms — a bare ``` pair and a single-backtick span. A ````-fenced block, a tilde fence, or a ``multi-backtick`` span ended the region early and handed the rest of the author's code to the rewrite, turning `a\_b` into `a_b` inside their code sample. Fenced blocks are now walked a line at a time, tracking the opening delimiter exactly the way stripEmptyListItemLines already does (three or more, closed only by a run at least as long), and the inline branch matches a backtick RUN closed by one of equal length. The note scanner claimed to be the same scan as the indexer's `findTextRanges` but did not fold whitespace, which staging added since this branch was written. The indexer folds every `\s` to a space, so a phrase matches across a soft line break — which `remark-breaks` renders as a `<br>`, splitting the phrase over two text nodes that a per-node scan could never see. The hit counted in the panel and highlighted nowhere, the exact bug this branch exists to fix. The plugin now scans runs of continuously-readable text rather than single nodes, so a match spanning an inline boundary (a soft break, a bold word) is wrapped as several marks sharing one ordinal. Runs end at any non-inline element, so two paragraphs are never joined into a phrase the reader cannot see. `foldSearchWhitespace` moved to `@sim/utils/string`: the canvas card renders from a package, which cannot import from `apps/*`, and two copies of that rule silently disagreeing is precisely what produced the second finding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(markdown): close a fence only on a bare delimiter run A closing fence carries nothing but its delimiter run; a line that merely starts with one is content. The guard matched the prefix alone, so an interior line like ` ````example ` inside a same-length fence ended the block, and every cleanup below then processed the author's remaining code as prose — dropping the backslashes from their `a\_b`. Both fence walks in this file shared that flaw, so both now go through one `closesFence`, which requires the run to be followed by nothing but whitespace. Strictly more conservative: a fence stays open longer, so more content is left verbatim. Scope, stated plainly: the serializer always opens a block with one more delimiter than the longest run inside it, so its own output cannot reach this shape today, and `postProcessSerializedMarkdown` only ever sees serializer output. This is a correctness fix that removes an unstated coupling to that choice, not a live corruption path. The tests therefore exercise `postProcessSerializedMarkdown` directly — a round-trip test of the same input would pass either way, which is exactly the vacuous check worth avoiding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(markdown): skip quoted fences, and stop joining runs across inline tags Two review findings, both real, both the same shape: a rule that looked at the rendered form and forgot what the source actually says. QUOTED FENCES. The fence walk only recognised a bare delimiter run, but the serializer writes a fence inside a blockquote or a `[!NOTE]` callout with a `>` on every line. Code state was therefore never entered there and the block's interior was cleaned up as prose: `> x = a\_b` round-tripped to `> x = a_b`, losing the author's backslash. Unlike the fence-length cases this one is reachable today — verified against the real serializer before and after. Both fence walks now unquote the line first. INLINE JOINS. Runs concatenated the visible text of every inline tag, so `a<strong>b</strong>c` read as `abc` — a hit that cannot exist in the markdown the indexer scans, where `**` sits between the words. That is worse than a spurious mark: `occurrenceIndex` counts SOURCE occurrences, so a fabricated hit earlier in the document steals the current ordinal and paints the mark on text the search never matched. Only `<br>` continues a run now, because it alone stands for a character the source really has (a `\n`, folded to a space). Everything else stands for syntax the render drops. Nothing real is lost: a match spanning `a**b**c` would have to contain the asterisks to exist at all, and a match wholly inside an element is still found — the element simply starts its own run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(search): match a Note body as it renders, instead of rewriting the file Replaces the serializer change with one that writes nothing. The editor backslash-escapes every markdown-significant character in prose, so a Note the reader sees as `{{TE_SERET}}` is stored as `{{TE\_SERET}}` and search — which matches the stored value — could not find it. The previous approach undid that escape in `postProcessSerializedMarkdown`, which meant re-deriving markdown structure from the serialized string with regexes so it knew what was code. That is a losing game: three review rounds, each finding another construct it did not model (longer fences, then delimiter-prefixed lines, then quoted fences), and each miss REWROTE somebody's code. `markdown-fidelity.ts` is back to staging, byte for byte. The escape is now undone on the matching side only. A field declares `searchTextFormat: 'markdown'` (the Note body is the only one), and the indexer matches it against `projectEscapedMarkdownForSearch(value)` — a total, structure-free function that returns the rendered text plus an index back into the source. Ranges stay in source coordinates, so replace still rewrites the whole `\_` and never strands a backslash. The asymmetry is the whole point: a matcher that de-escapes something a fence would have kept literal changes only which text highlights, and no caller writes it back. A rewriter making the identical mistake corrupts the file. So there is nothing here that needs to know about fences at all. Two consequences worth having: existing notes are searchable immediately rather than after their next edit, and no stored byte changes, so no document the editor has ever written can be affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 15:29:25 -07:00
Waleed	a27f376164	feat(connectors): add Bitbucket, Databricks, Google Chat, and Workday Help KB connectors (#6895) ... * feat(connectors): add Bitbucket, Databricks, Google Chat, and Workday Help KB connectors Adds four knowledge base connectors, closing the gap where Sim shipped tool blocks for these services but could not index their content. - Bitbucket: repository source files and pull request descriptions over the existing Bitbucket OAuth credential - Databricks: notebooks (Workspace API) and saved SQL queries, PAT auth - Google Chat: spaces indexed as message transcripts, new google-chat OAuth service under the shared Google client - Workday Help: knowledge article versions via the public helpArticle/v1 API * fix(connectors): second-pass validation fixes and test coverage Adversarial re-validation of all four connectors plus a combined-change regression audit. - bitbucket: stop declaring incremental sync (deletion reconciliation is disabled for incremental runs, so deleted files were never removed on the default code configuration); drop a wasted listing round-trip after the frontier drains; add 33 tests - databricks: reject an explicit maxDocuments of 0, which meant unlimited; add 34 tests - google-chat: correct the sender displayName documentation (user auth populates only name and type) and emit second-precision RFC-3339 in the message filter; 15 -> 24 tests - workday: fix a crash when maxVersions is persisted as a number; refuse a configuration whose status filter Workday did not honor; cap the unresolved-name error; 18 -> 27 tests - document the Google Chat service-account omission - docs: list all four connectors and correct the connector count * fix(connectors): index Google Chat spaces with no messages in the window Review round 1. - orderBy takes a full ordering expression, not a bare direction. The reference documents the default as `createTime ASC`, so send `createTime DESC`; a bare `DESC` either 400s every hydration or is ignored, which would make the cap keep the oldest traffic and the later reverse render the transcript backwards. - getDocument no longer returns null when the message window is empty. A space with no messages is still a live space, and null is the "document is gone" signal the engine treats as last-known-good: returning it dropped spaces whose only prose is their description or guidelines, and left a stale transcript indexed after a space was cleared or lookbackDays was tightened past every message. - The transcript header is omitted when no message contributed text. * fix(connectors): only flag a Bitbucket listing capped when the cap withheld something Review round 2. takeIndexableWithinCap reports capReached as soon as the running total equals maxItems, which is also true of a listing that ended at exactly that count. Setting listingCapped there suppressed deletion reconciliation for a complete listing, so upstream-deleted files and pull requests could stay in the knowledge base indefinitely. applyMaxItemsCap now takes whether Bitbucket had more content beyond the page -- a next link, or directories still queued on the frontier -- and flags the listing only when the cap actually withheld something, matching the Databricks, Google Chat, and Workday connectors. * fix(connectors): keep the Bitbucket cap flag set when it skips the pull request phase Review round 3. Fixes a regression from 622a21bd9a. maxItems is shared across the code and pull request phases, so a code walk that ends with exactly maxItems documents and no next link or frontier stops pagination before the pull request phase runs. Scoping listingCapped to "this phase had more" left the flag unset in that case, and the engine then treated the run as a complete enumeration and could hard-delete previously indexed pr:* documents that were never listed. The cap flag now asks whether anything the connector was configured to list remains unlisted -- including a later phase the cap is about to stop us reaching.	2026-08-20 14:46:29 -07:00
Waleedandmini.jeong	d374ebce6f	fix(webhooks): accept the methods and expose the request metadata the generic webhook advertises (#6893) ... * feat(webhooks): support query parameters and GET deliveries on generic webhooks The generic webhook Setup Instructions promised that query parameters would be available in the workflow and that any HTTP method would be accepted, but neither was true: query parameters were never carried past the route, and every GET that was not a provider challenge got a 405. Carry the request query string into the execution payload and expose it to providers through FormatInputContext. The generic provider merges it into the workflow input under a reserved `query` key, leaving the body's own fields untouched so existing payloads resolve exactly as before. Add an opt-in `acceptsGetDelivery` provider capability and enable it for the generic provider, so a workflow can be triggered by a plain URL fetch such as a link in an email. Providers that have not opted in still answer 405, and unknown paths keep answering 405 on GET so probes cannot distinguish them. Update the Setup Instructions to describe what the endpoint actually accepts. Signed-off-by: mini.jeong <mini.jeong@navercorp.com> * feat(webhooks): expose generic webhook request headers The generic webhook's Setup Instructions promised that request headers would be available in the workflow, but formatInput returned only the body: headers were used solely for the idempotency key and provider signature checks. Expose them under a reserved `headers` key, withholding the ones that carry credentials. Exposing a credential would copy it into execution logs and trace spans, where it outlives the request, so a fixed denylist (authorization, cookie, x-api-key, ...) is combined with the webhook's own configured secretHeaderName. A denylist rather than an allowlist keeps arbitrary custom headers usable, which is the point of the feature. Generalize the query-parameter merge so query and headers share the same key-wise body-precedence rule. Also correct the authentication instruction: only the configured method is accepted, not either one. Refs #6888 Signed-off-by: mini.jeong <mini.jeong@navercorp.com> * feat(webhooks): accept PUT, PATCH and DELETE deliveries and expose the request method The generic webhook's Setup Instructions promised any HTTP method, and the /api CORS policy already advertises PUT, PATCH and DELETE, yet the route answered 405 for everything except POST and GET. Open the remaining methods for providers that opt in, which today is only the generic webhook. Expose the method on the trigger input as well. Without it a workflow behind one URL cannot tell a create from a delete, which makes multi-method delivery half a feature. The payload field is optional so jobs already queued at deploy time keep executing. Turn the GET-only opt-in into a per-provider method set, and let the request metadata merge carry scalar values so `method` follows the same key-wise body-precedence rule as query and headers. Refs #6888 Signed-off-by: mini.jeong <mini.jeong@navercorp.com> * feat(webhooks): declare the generic webhook trigger outputs The trigger declared no outputs, so the reference dropdown in the editor offered no completions for it and users had to type paths like `query.id` by hand after reading the setup instructions. Declare the request metadata that is known ahead of time. Body fields stay undeclared because a generic webhook receives whatever JSON the caller sends. Refs #6888 Signed-off-by: mini.jeong <mini.jeong@navercorp.com> * fix(webhooks): stop provider challenges from intercepting other providers' deliveries The challenge handlers run before webhook lookup and are provider-blind, so two query parameter names are effectively reserved across every path. Now that a generic webhook can be triggered by a URL fetch, a link carrying either name answers the challenge instead of running the workflow: - `?validationToken=x` is echoed back as a Microsoft Graph subscription validation. Graph sends that validation as a POST, so ignore the parameter on every other method. - `hub.mode`, `hub.verify_token` and `hub.challenge` answer 403 when no WhatsApp webhook on the path expects a token. A path with no such webhook is not a failed verification - the parameters belong to whoever owns that path - so fall through and let the delivery route normally. A token mismatch against a WhatsApp webhook still fails with 403. Refs #6888 Signed-off-by: mini.jeong <mini.jeong@navercorp.com> * fix(webhooks): make the request metadata opt-in per webhook The four commits below make the generic webhook do what its Setup Instructions promise. They do it through a provider-level capability, which applies to every generic webhook row the moment it deploys: each one begins accepting GET, PUT, PATCH and DELETE, and each one's workflow input gains `method` and `headers`, on POST deliveries too. No webhook owner chose either. Gate both behind `providerConfig` flags written by two switches, off by default. A webhook deployed before these existed has neither flag, so it answers POST only and its input is exactly the body, as before. `query` stays ungated: it is dropped today, only appears when the caller's own URL carries it, and yields to a body field of the same name. Generalize the Microsoft Teams challenge fix. Every challenge handler runs before the webhook lookup and matches on payload shape alone, so any of them will answer a delivery addressed to another provider on the same path. Gate them centrally to POST via `challengeMethods`, which WhatsApp widens to GET for Meta's handshake, rather than guarding one handler inline. Also: - Widen the credential header denylist to 24 names and withhold the webhook's own token by value as well as by name, since a denylist is leaky by construction. - Condition the `method` and `headers` trigger outputs on their switches, so the reference dropdown cannot offer a field the webhook will not send. - Give PUT, PATCH and DELETE their own contracts instead of reusing the POST one, whose `method: 'POST'` had become untrue. - Parse, challenge and generate a request ID once per delivery rather than twice on GET, which was logging one request under two IDs. - Offer the challenge handlers the request before admission, so Meta's GET handshake cannot be answered with a 429 by a busy instance. - Answer every non-POST rejection with the same 405 plus `Allow`, whether the path is unknown, holds only non-path triggers, or holds a trigger that has not opted in. - Read flags through a helper treating only `true`/`'true'` as on: the editor writes booleans, but a YAML- or Copilot-authored workflow can write the string `'false'`, which is truthy. - Name the methods switch "Accept Other HTTP Methods": HEAD and OPTIONS still answer 405, so claiming "all" would reintroduce the overstatement this whole change set exists to remove. - Drop the per-delivery metadata warn logs to debug. --------- Signed-off-by: mini.jeong <mini.jeong@navercorp.com> Co-authored-by: mini.jeong <mini.jeong@navercorp.com>	2026-08-20 14:38:08 -07:00
3 people	d9cfd7c68e	improvement(mothership): v0.9 (#6815) ... * checkpoint * Checkpoint * dot fixes * Make async tool resume delivery recoverable * Support split table tools and option recovery * Harden VFS mutation handling * feat(platform): platform subagent support — docs corpus VFS, search_docs, account context Squash of the feat/platform-agent branch (sim side): mounts the Sim docs corpus in the copilot VFS, wires search_docs and retires the legacy docs search tools, and syncs the generated tool catalog and trace contracts for the platform subagent. * Align Copilot tools and resource handling * Expand workflow log query support * checkpoint * Port desktop-improvements-0 desktop and browser-agent work * fix(desktop): keep browser-agent input alive through live-SPA re-renders * Harden workflow sanitization and Slack setup * feat(desktop): coordinate clicks, caret insertion, and drag for the browser agent * Revert subagent group eager auto-collapse * fix(chat): keep sends FIFO across the streaming-to-idle drain gap * Add the steering backend surface for mid-turn sends * Sync generated contracts for async subagent orchestration Pulls the mothership tool catalog (wait_agents / tail_agent / steer_agent / interrupt_agent), trace spans (chat.async_subagent.*, chat.orchestrate.*), and trace attributes (copilot.async_subagent.*) into the generated TS contracts. * Add display titles for the async subagent orchestration tools wait_agents / tail_agent / steer_agent / interrupt_agent get natural-language running titles (naming the agent id being waited on, tailed, steered, or stopped) and a Steering→Steered completed-verb rewrite. * Show orchestrator-chosen subagent names on agent groups A subagent_start whose payload data carries a name (the orchestrator's new name trigger parameter) now labels the agent group with that mission name — the agent-type icon stays. The name flows through the live stream path, the turn model (AgentNode.displayName) and its serialize/rebuild round-trip, and persisted transcripts (PersistedContentBlock.name), so reloads keep the label. * Improve Copilot error handling and logging * Backfill the subagent display name from the second start event The dispatch-time subagent_start fires before the trigger args (and therefore the name parameter) have streamed; the phase-3 start re-announces the lane with the name. The block builder was dropping that duplicate wholesale, losing the name on streaming providers — now it backfills subagentName onto the existing block instead. (The home turn-model path already reconciled this case.) * Support Slack bot connection flow * Harden Copilot error and VFS handling * Harden VFS resource operations * Show 'Waiting for the first of N agents' for mode-any waits The wait_agents title ignored the mode argument, so an any-mode wait over three agents read 'Waiting for 3 agents' while the model narrated waiting for the first — contradicting the transcript. * Collapsed-by-default agent cards with live intent status lines Subagents now narrate their work through <intent>3-5 words</intent> tags (a fleet-wide prompt protocol on the mothership side). The turn model streams each subagent's text through a split-safe tag parser: complete tags update the agent's currentIntent and disappear from the prose, tags split across deltas are carried until their close arrives, and a tag that never closes flushes back as plain text. The agent card renders as one line — display name (or agent label) plus the latest intent, replaced inline as the agent shifts gears — and never auto-expands; expanding to the full tool log is a deliberate click. Only an outstanding permission prompt or a browser hand-back forces a group open. Intents persist on the subagent block (and through the legacy persisted- message paths) so reloads keep the last status, and a renamed reinvocation now takes the latest name instead of pinning the first. * Add the internal in-band tool execution route for live mothership turns POST /api/copilot/tools/execute (INTERNAL_API_SECRET, Go→Sim) runs one sim-server tool through the same server tool router the resume driver uses and returns the result synchronously — no checkpoint. This is what lets background (async) subagents write files/tables/knowledge, and lets the main lane keep streaming (instead of checkpoint-pausing and killing every background run) while async agents are live. * Persist resource side effects for in-band tool execution Files/tables created through the internal execute route now register on the chat's resources exactly like the resume driver's executions — the route runs the same handleResourceSideEffects pass (persistence only; an out-of-band route has no live event sink, so mid-turn chip pushes are a follow-up). * Extract intents from group text on every path, sync and async The turn-model intent filter only fires for span-scoped subagent lanes, but this surface also delivers subagent text through the legacy block path — so <intent> tags flowed through unparsed and rendered as prose rows. Groups now extract intents from their accumulated text at append time: the last complete tag becomes the card's status line and every complete tag is stripped from the rendered prose. Covers span-scoped, legacy, and persisted-reload paths for both synchronous and background delegations. * Fall back to the live tool title for the agent card status line Persisted data proved tool-first subagents (grok search agents) emit zero prose, so intent tags never stream no matter what the prompt says. The collapsed card now always narrates: the agent's own <intent> tag when present, else the latest tool's display title while the lane is live. * Catch subagent <intent> tags in the server relay The relay's subagent text handler now runs the split-safe intent extraction as chunks stream: the latest complete tag is stamped onto the lane's persisted subagent block (subagentIntent) and stripped from the stored prose, so live, persisted, and replayed views all agree. Per-lane carry handles tags split across chunks; a never-closing tag flushes back as plain text. * Drop the tool-title fallback: the status line is the agent's intent With the intent protocol now injected into every spawn's task message, agents open with an <intent> tag; the card shows that narration or nothing. * Replace intents with live tool-title status lines on agent cards Intent parsing is fully removed (turn model, relay handler, persistence fields, group extraction). The collapsed card's status is the latest tool call in its RUNNING phrasing — never the completed rewrite, which stays in the expanded log. Parallel tools show the most recently started still-running title with a +N for concurrent siblings; between rounds the last title stays frozen; a closed lane shows the bare name. Nested agent cards compute their own status recursively from their own items. * Keep the main Sim lane live-expanded; collapse only real subagent cards The mothership group is the turn's own narration, not a delegation card — collapsing it hid main-lane text and tools until manual expand, which read as mis-ordered streaming while async subagents interleaved. It keeps the original live-expand behavior and no status suffix. * Persist subagent lane lifecycle blocks from the span handler Lane-scoped span events route to the span handler, which only recorded trace side effects — no subagent start block was ever persisted (verified: a seven-agent run stored 104 blocks with zero starts). Grouping then fell back to keying lane content by agent NAME, so a respawned agent of the same type merged invisibly into the first one's card until it resolved. The handler now persists the start block (spanId-keyed and deduped, carrying the display name) and stamps endedAt on close, giving every invocation its own card. * Name agents in orchestration titles; '+ n more' overflow format wait/tail/steer/interrupt titles humanize the slugified agent ids back to their display names ('Waiting for the first of Digest Workflow Build + 4 more'), and the agent card's parallel-tool suffix uses the same '+ n more' format. * Harden in-band tool execution and resources * Route in-band execution through the comprehensive tool dispatcher The internal execute route used the bare server-tool router, which rejects VFS tools with 'Unknown server tool: read/glob/grep' — so nearly every background agent's first discovery call failed (102 in-band calls in one run, dozens rejected). It now uses the relay's executeTool dispatcher: registered handlers (VFS, function execute) with permission checks and param normalization, falling back to the app tool router — the same surface foreground execution gets. * Harden chat stream transition handling * Harden VFS provenance and resource writes * Standardize tool environment references * Harden browser panel and chat cleanup * Descriptive, user-language tool titles across the board House rules applied everywhere: use every argument the call carries, never name internal machinery, and never lead with Getting (the Got rewrite is deleted so it cannot return). - Deployments name the workflow: Deploying {workflow} as API/chat app/MCP tool - Workflow reads name the part: Reading {workflow} meta/state/deployment/notes; generic reads always name the file (Reading {leaf}), never bare Reading file - Block runs name block and workflow: Running {block} in {workflow}, Running from {block} in {workflow}, Running {workflow} until {block}, and Enabling/Disabling {block} in {workflow} - The six split-table tools get per-operation verbs (Adding column {name}, Updating rows, Wiring automation, Creating view {name}) instead of a wall of Querying table - The manage quartet drops X-action system-speak for gerunds - get_* internal names become user language (Checking run settings, Tracing block inputs, Reading the deployed version); web_fetch says Fetching - Scheduled-task titles removed entirely (feature deleted from the Go catalog) - New verb rewrites: Fetched, Traced, Wired, Configured, Looked, Rotated * Deploying {workflow} as chat, not as chat app * Loader gerunds; mv names both ends; mkdir names the folder search_integration_tools -> Finding the right integration; load_integration_tool -> Loading {integration} tools; load_skill -> Loading skill {name}; run_enrichment -> Looking up {subject}. mv prefers the model's phrasing, else reads 'Moving {files} to {destination}'; mkdir reads 'Creating folder {name}' from the path. * Overflow counts read '+ n', dropping 'more' * Unify workspace find and search * Scale desktop title bar with page zoom * Serialize account and organization truth into the copilot VFS Workspace standing, membership, billing, org role, access-control restrictions, published-block provenance, and fork topology were reachable only through three parameterless tools (or not at all). They are ambient read-only facts, so they belong in the VFS where they are greppable, cost no tool round-trip, and every agent that can read gets them — the same move that retired get_blocks_and_tools and list_user_workflows. Adds account/{workspace,workspaces,members,billing}.json (always mounted) and organization/{organization,access-control,custom-blocks,forks}.json (only when the workspace is org-hosted). Every file projects an existing use case or util after getOrMaterializeVFS's access assert — no new queries, no new authorization. One relation per file, cross-referenced by id-and-name stub, so overlapping facts cannot disagree. Volatile content (billing, access control, forks) is lazy, so numbers are read-time fresh and unasked-for reads cost nothing. Projection follows the viewer: member emails are admin-only, fork detail requires workspace admin on a forking-enabled org, and the whole organization/ namespace is absent for a personal workspace — which is itself the answer. Retires get_account_billing, get_enterprise_context, and list_user_workspaces along with their handlers; display titles stay for transcript replay. * Fix insert_text refusing an editable field focused inside a frame describeFocusedEditable descended shadow roots but not frames, while activeElementReadback descends both. Focus inside a same-origin frame therefore surfaced to the first as the FRAME element — not an input, not contentEditable, not a canvas, no textbox role — so it fell through to 'not-editable' and insert_text refused a field that press_key had just typed a character into. Two functions answering 'what is focused' with different answers is the bug; the descent loops now match exactly. The refusal also names what actually held focus (tag, role, contenteditable). A bare 'not-editable' gave the agent nothing to act on, so it guessed at the cause — a real run spent twenty rounds on the wrong theory and had to be stopped by the user. * Keep retired browser takeover renderable in history The tool is gone from the catalog, so its generated constant went with it and every path that referenced it stopped compiling. Deleting those paths instead would have silently downgraded every past transcript containing a takeover card to a generic tool row, and dropped the no-timeout budget that an in-flight takeover still needs while a rolling deploy finishes. retired-tools.ts gives the literal a documented home that says what it is and why it survives its tool. * Follow the agent into a tab it opened to work in browser_open_tab created the page with activate: false, so the agent worked in a tab the user could not see while the panel sat on a page where nothing was happening. The panel now follows a tab the agent deliberately opened. Scoped to that tool only. A page spawning its own tab (popup, target=_blank) is the site grabbing the view rather than the agent choosing a workspace, and stays in the background as before — two existing tests pin that and caught the first version of this change, which moved both. A tab the user claimed still wins over both: the work starts in the background instead of pulling the page out from under them mid-read. * Make the browser tools agree with each other An audit of the module found the frame-descent bug was one instance of a pattern: six independent definitions of 'is this editable' and seven of 'what is focused', disagreeing with each other. A tool refusing what its sibling accepts on identical page state is invisible at runtime — the agent follows a snapshot that says one thing into a tool that says another. - browser_type now accepts role="textbox" like browser_insert_text does. The snapshot advertises those elements as [textbox] with a ref, so refusing them meant rejecting exactly what the outline told the model to type into. Both the native and synthetic paths, and their descendant scans. - pressKeyOnPage descends shadow roots and frames like every other focus reader. It was dispatching synthetic keys at the shadow host or <iframe> element, where they bubble but never reach the editor, while reporting success — and contradicting the activeElement reported beside it. - not-editable and ambiguous-editable name what was found: the element's tag and role, and the candidate fields. Both had the data and discarded it, which is what turns one blocked step into twenty rounds of guessing. - obstructedAfterNavigation requires a dialog that ARRIVED with the navigation. It compared against nothing, so every SPA route change under a persistent role=dialog reported a successful click as obstructed. The test that covered this asserted the false positive; it now pins both directions. - browser_insert_text observes the top document when typing inside a frame, like every other input tool. A submit that navigates the top page was invisible to its frame-scoped observation. * Let hover actually see what it mounted Four independent defects made browser_hover blind to the most common thing a hover produces — a row's action bar — so it reported no effect on a hover that worked, and the agent fell back to clicking pixels off screenshots. - The popup scan matched only role=tooltip/menu/listbox. Slack's message shortcuts bar is a labelled toolbar/group, so it registered as nothing at all. Added toolbar, menubar, labelled group, and [popover]. - The baseline was captured BEFORE prepareElementSurface scrolled the target into view, so scrollChanged was always set by the tool's own probe. That pinned every unproductive hover to 'background DOM churn' instead of the honest 'nothing happened', and hid scrolling the hover really caused. Re-baselined once the scroll settles and before the pointer moves. - The MutationObserver attached only on the first observation, while the roots list is rebuilt every call and grows as shadow roots mount. Components that appeared later were never observed, so their DOM changes raised no revision. Roots are now observed as they show up. - observationTruncated was computed and never read, so a scan capped at 12k nodes reported 'nothing appeared' with the same confidence as a complete one — and portalled overlays live at the end of <body>, exactly what the cap drops. Hover now says the page was too large to scan and to confirm visually. * Stop the browser agent acting on the wrong element, and say why it refused Four findings from the module audit, the first of which could silently do the wrong thing rather than merely fail. - A ref whose node is gone is re-adopted by structural resemblance, matching on ORIGIN only so a pushState between snapshot and act does not kill every ref. That leniency also let a ref to a row control in one view rebind to the identical control in a view the app had since navigated to — acting on the wrong message, signalled by nothing louder than recovered: true. Adoption now requires the same path; a view swap reports the ref stale, and the caller re-snapshots. Revalidating a still-connected node stays lenient, because that is literally the node the model chose. - A hit INSIDE the requested element is its own nested control, not an overlay. Both produced 'covered by X — close or move the overlay', advice that cannot be followed because there is nothing to close. Nested hits now say so and point at retargeting. - browser_click_at, browser_insert_text, and browser_drag listed targetChanged in their effect formulas, but none passes an elementId, so no targetState is ever captured and the term was always false — coverage that read as real. Removed, with a test pinning the dependency. - The seven effect formulas are deliberately NOT collapsed into one predicate: drag must trust domChanged where others must not, hover must ignore field and focus changes, click counts focus only for editables. Forcing one would make each tool wrong differently. The differences are now documented in one place next to the shared computation, so divergence is a declared policy rather than an accident. * Let edit_workflow configure block retries * Updates * Always focus the resource the agent is working on, and its browser tab The resource panel had a carve-out: an already-open browser session declined to replace another selection and only got an attention marker, so agent browser work happened off-screen. The panel now follows the agent to whatever it touches — browser included — and an event can still opt out explicitly. The browser panel also follows the agent BETWEEN tabs: the store already tracked automationTabId (and the strip marked it), but the visible tab never changed. It now switches when the agent's target tab changes, so watching the agent never means hunting for the tab it moved to. Keyed on the target changing rather than on it being set, so a user who browses elsewhere mid-run is only pulled along when the agent itself moves. * Never paint a browser snapshot at stale geometry (the modal-open flash) Opening a modal locks scroll, which removes the window scrollbar and reflows the panel — so a capture taken before the lock describes a rect the panel no longer occupies. The handshake painted that frame anyway and only then retried, so the replacement landed visibly offset from the page it stands in for: the flash. A capture is now checked against the host's live rect before it is painted; a mismatched frame is skipped and re-captured at the settled layout instead (modal retries go 2 -> 3 to absorb the extra settle). * Name the workflow in deployment and workflow-scoped tool titles 'Checked deployment status' never said which workflow — nor did the deployed- state read, run settings, block outputs/inputs, redeploy, promote, or the global-variable write. These tools carry a workflowId (often defaulting to the current workflow), so only the client can resolve a name: the enrichment layer now resolves it for the whole workflow-scoped family and passes it as workflowName, which every workflow title already reads. Titles: Checking {workflow} deployment status, Reading deployed {workflow}, Checking {workflow} run settings, Reading {workflow} block outputs, Tracing {workflow} block inputs, Redeploying {workflow}, Promoting {workflow} version {n} to live, and 'Adding workflow variable {name} in {workflow}' — each falling back to its unnamed form when no workflow resolves. * Name the block that ran; never fall back to a raw block id run_block and set_block_enabled carry only a blockId, so their titles would have printed an opaque UUID ('Running 7f3a2b91-… in Invoice Sync'). The enrichment layer now resolves blockId against the workflow store the same way it already did for run_from_block's startBlockId, and the base titles no longer accept an id as a name — an unresolved block reads 'Running block' rather than a UUID. * Add the missing Removing -> Removed rewrite The table work introduced 'Removing automation'/'Removing enrichment' with no past form, so those rows kept their present tense after completing. * Name the target resource in the remaining tool titles Table tools keep their operands nested under args and identify the table by id, so their rows said 'Adding rows' with no hint where: enrichment now lifts the nested args and resolves tableId against the cached workspace table list, giving 'Adding rows to Runtimes', 'Adding column status in Runtimes', 'Reading views of Runtimes'. Also: a block-schema read names the block instead of the file ('Loading Slack', 'Loading Google Sheets tips'); browser type/insert show the text they send, middle-ellipsized; downloads name the file; library-docs searches name the library and query; knowledge-base searches include the query; generated media names its output file; and diff_workflows, list_deployment_versions, and publish_custom_block joined the workflow-name enrichment set. * Bubble nested agents' tool calls into the parent's status line The collapsed status only scanned a group's OWN tool items and skipped nested agent groups, so a parent that had delegated froze on its last own tool while its child did the actual work — the line described nothing that was running. Status now walks the whole subtree: any tool at any depth counts, the most recently started running one is shown, and the rest become the same '+ n' overflow. With nothing running it falls back to the last tool at any depth, so an idle parent still reflects where its subtree got to. * Align nested tool call status rows * Revert branch-local KB connector error-message edits Restores apps/sim/connectors/ to staging state. Two copilot-focused commits on this branch (3a9fc1c5a5, 24b24e5f8e) drove by nine KB source connectors (airtable, confluence, discord, github, gitlab, google-drive, microsoft-teams, notion, slack) to enrich credential-validation error messages. The env-reference resolution machinery those errors supported is kept; the product connector surface stays unchanged in this branch so the staging promotion remains scoped to copilot work. * Fix the failing audits: NUL escape and route-count ratchet check:source-text: resource-vfs.ts used a raw NUL byte as its folder-index key separator (comment and template literal), which makes git treat the file as binary and hide it from review. Written as the '\u0000' escape — the runtime string is identical. check:api-validation:strict: baseline 1120 -> 1122 for the two routes added on this branch since the last bump. * Add Force Reload (Cmd+Shift+R) to desktop View menu * Route Force Reload through focused-resource boundary; regen docs manifest * Align the title-bar surface audit with the zoom rework 'Scale desktop title bar with page zoom' (833d2f126d) changed the CSS contract in two ways its audit test still pinned the old shape of: the lane vars gained a max() floor around the platform env() terms so page zoom cannot shrink the lane below the OS-drawn lights, and the control square became fixed px — CSS px already scale under zoom — leaving only the centering offset derived from the lane height. The test required the bare env() prefix and calc() on all three control vars, so it failed the commit that implemented its own regression comment. Pins now assert the env() term inside the clamp (still platform-derived, the test's actual intent) and split the control vars: offset must stay computed, size and icon are explicit constants. * Budget the two structurally slow tests explicitly sso-trust imports the whole Better Auth module graph (2.5s on an idle machine) and events.attribution scans call sites across the repo. Under a fully-parallel uncached run on a loaded machine both blow the default timeout while passing in isolation and on CI — a verdict decided by machine load, not by the code. 30s budgets make a loaded local run mean what it says. * Carry the cross-service trace id in sim log lines * Improve nested tool status presentation * Refresh secret schemas and shared test mocks * Open the deployed graph of org-published blocks, read-only, org-wide A consuming workspace could see a published block's interface but never what it does: the backing workflow lives in the publishing workspace, and other workspaces are nameable, not readable. Publishing a block org-wide is the act of sharing it, so the graph it executes is now readable from any org workspace — the DEPLOYED graph, not the publishing workspace's live editor state, so nothing in-progress leaks and what you read is what runs. The namespace also adopts the root's index/detail split: custom-blocks.json slims to names with a detail pointer, organization/custom-blocks/{type}.json carries provenance plus the deployed graph (loaded lazily through the cached loadDeployedWorkflowState; credential ids and env references inside it belong to the publishing workspace and say so), and organization/README.md is the namespace guide WORKSPACE.md is at the root — files, usage, the in-depth block inventory, and forks.json documented only when actually mounted. The block list moves to materialize time (same indexed query the components pass already runs) because the README, the index, and the per-block key-view entries all need it; only the graph stays lazy. * Withhold the deployed graph from external collaborators Workspace access and org membership are different grants: an external collaborator can open the workspace and use the published block, so the names index and the interface schema stay visible to them — but the deployed graph is org implementation internals, and their detail files now simply do not exist. isHostOrganizationMember is the viewer bit the host context already resolves for exactly this distinction. * Fill out the organization namespace: workspaces, permission groups, credential groups Three more read-only files, all lazily loaded — the paths appear in the key view so glob discovers them, but no query runs until a read — and all gated by registration, so an unpermitted viewer's file simply does not exist: - workspaces.json (org members): the org's full workspace map with the viewer's access flag and fork parentage — account/workspaces.json only ever showed what the viewer can reach. Inaccessible workspaces stay nameable, not readable. - permission-groups.json (org admins): every group with member count, targeted workspaces, and the restrictions its config activates. access-control.json remains the per-viewer binding. The queries are lifted into lib/permission-groups/queries.ts because their only prior home was inline drizzle in the route handlers, which the VFS cannot import. - credential-groups.json (entitlement-gated): per-option configuration readiness and enrollment progress — the two facts that decide whether a credential_group workflow will do anything at runtime. The note teaches the contract that bit the audit: an active group with zero completed enrollments yields an empty loop, not an error. Enrollee emails are workspace-admin-only, matching the settings page; counts come from the first enrollment page and say so when truncated. The README documents each file only when mounted for this viewer. * Stop reporting a click that navigated as a failed click The field case: 'Begin Assessment' submits a form. The navigation tears the origin document down while the press completes, so everything after dispatch — the CDP call's own completion, the synthetic dispatch's return value, the postcondition reads — fails against a destroyed context, and a maximally successful click was reported 'Failed clicking element'. The agent's own follow-up investigation in the transcript diagnosed exactly this. navigationRescue detects it at the driver level, where the navigation epoch and URL survive the renderer teardown: when the page provably navigated since dispatch began, a dispatch-path failure becomes a success carrying navigatedDuringDispatch and a note explaining why no postconditions exist. Applied to all four click dispatch paths (unframed CDP, framed native, synthetic in-page, click_at). The soft path had the same blindness: with the after-state unreadable, urlChanged computed false and a navigating click reported 'no observable change'. navigatedByDriver now folds into navigated/effectObserved, which also keeps the new-dialog obstruction check meaningful on real navigations. * Re-hide the browser under a modal after main loses the occlusion lease The punch-through: a modal opens, the native view hides behind a painted snapshot, and the renderer records applied: true. Then one heartbeat commit is skipped — renderer jank past the 2.5s bounds-lease TTL is enough — and main expires the lease, resetting panelOccluded on its side. The next heartbeat finds the modal marker still present and calls setDesired(true), but the lease's dedupe sees applied === desired and sends nothing; the bounds commit that follows lays out an unoccluded native view above the open modal, and no later event ever re-hides it. The comment on this branch already claimed it 'reasserts the lease' — the dedupe made that claim false exactly when the lease had been lost. While the occlusion marker is present, each heartbeat now drops the applied belief (assumeRevealed) before setDesired, so the reassert is a real, forced, idempotent hide IPC — one per second while a modal covers the browser — and any main-side lease loss self-heals within a heartbeat. * Make a stale ref name which of its five causes fired A field run burned five snapshot->click cycles on a STATIC landing page, every one refused with the same sentence — 'the page changed since the last snapshot' — and the agent reasonably concluded the page was regenerating its DOM. It was not; the resolver was refusing, and the message could not say why. Five distinct conditions produced that one string: an id missing from the registry, a connected node whose identity drifted, the view-changed adoption gate, no confident replacement, and a replacement tie. The resolver now stamps the reason (with the drifted node's current identity, or the from->to paths for the view gate) and every stale producer carries it into the driver message. Same pattern as not-editable: a refusal that names its cause costs one round; an opaque one costs a loop and a wrong theory in the bug report. * Pulse throttling off when the browser view is revealed, so it actually paints The blank-page report: a navigation completes while the view is hidden, the page 'finishes loading', and the panel shows white until the user re-navigates by hand. invalidate() on reveal was already there but recomposites the LAST frame — and the last frame is blank, because background throttling suspended the rAF the page's SPA paints its first frame from. The reveal now pulses throttling off (forcing the renderer to produce a real frame), invalidates, and hands the policy back to the session a second later through reassertTabThrottling, which preserves the automation-tab exemption. * Let the agent browser join meetings and finish passkey sign-ins Camera and microphone: 'media' joins the agent partition's allowlist, but every grant is gated on the macOS grant first — asked via systemPreferences.askForMediaAccess so the system prompt appears on first use, and answered from getMediaAccessStatus on checks — so System Settings stays the real authority and a page can never hold a grant the OS refused. Granting site permission without the OS grant produced the misleading NotReadableError Google Meet showed. Packaging gains the camera entitlement and usage string (macOS kills the process on prompt without one) and the mic string now covers meetings. Passkeys: WebAuthn itself is Chromium-native and nothing in our handlers blocks it — USB security keys need no permission at all. The hybrid transport (passkey on a nearby phone via QR) rides Bluetooth, which signed builds silently lacked: the bluetooth entitlement and usage string enable it. iCloud-Keychain platform passkeys remain outside what an entitlement here can grant — Apple restricts that to approved browsers. * Compile and preview Sim-styled pages * Wait for the batched prepare intent instead of instantly failing apply_file_edit The model batches prepare_file_edit and apply_file_edit into one round and the Go loop runs same-round tools concurrently, so apply could reach the executor before its prepare staged the intent. The instant no-intent error cost a model retry round and flashed 'Failed creating …' on the shared file row before the retry succeeded. The apply handler now polls briefly (10s cap) for the intent; a truly missing prepare still errors at the deadline. * Store page source, render docs-styled documents on view The pdf model for agent pages: the .html file keeps the markdown-shaped source (frontmatter + prose + sim: fences) and every surface renders the docs-styled document on demand — preview panel, /api/files/serve, public shares, and downloads all call the same pure compiler, now shared in lib/workspace-files. The docs chrome is reproduced from the real fumadocs source: the left sidebar's exact pill metrics, the clerk TOC with its animated scroll indicator, divider-style tables; cards and stats left the vocabulary. Table cells and kv values render inline markdown, and sim:workflow/table/knowledge/file links resolve to real workspace routes, bridged out of the sandboxed preview to the app router. Hand-written imitations of rendered output are rejected at apply_file_edit with a steer back to source, and a streaming page hides its source behind the live rendered preview (batched ~2s) the way a generating pdf hides its script. * Stagger the page rails so the resource panel keeps the docs sidebar Rails were gated at 1100px of iframe width — the chat resource panel never reaches that, so pages rendered single-column there. The rails now stagger the way the docs do on a laptop: >=640px keeps the section sidebar (240px) beside the content, >=1060px restores the full three-column frame with the clerk TOC, and only a truly narrow pane collapses to one column. * Fix in-page anchors escaping the preview; add the docs' toggle, code frames, pagination, and images Clicking a TOC or section link (or pressing Enter in the section filter, which clicks one) navigated the sandboxed frame off about:srcdoc in Electron and landed on a cookie-less sign-in page — the shell now intercepts every '#' anchor and scrolls directly. The page chrome gains the docs' exact theme toggle (emcn Sun/Moon, 30px rounded-lg, top right), framed code blocks with language label and copy button, and footer previous/next cards from prev/next frontmatter. Workspace images () compile to /api/files/view and the preview host inlines them as blob: URLs so the cookie-less frame can render them; sim:accordion joins the vocabulary as the faq component with title keys. * Size the section sidebar to its content so it appears at panel widths The left rail was a fixed 240/300px column, so it only earned its place once the pane was wide. fit-content caps at the docs width but shrinks to the longest section title (150px floor for the pills), and the two-column tier now starts at 560px instead of 640. * Let the clerk TOC join at 860px by sizing it to its content Same move as the section sidebar: the TOC column fits its longest link (capped at the docs' 268px, 150px floor), so the full three-column frame starts at 860px of pane width instead of 1060. * Open external links from pages in a new tab The preview bootstrap cancelled every non-anchor click, so an external link (the Sim docs, a vendor page) did nothing. External http(s) links now compile with target=_blank rel=noopener for the standalone and share surfaces, and the sandboxed preview bridges the click to the host, which window.opens a new tab — same channel the workspace deep links use. * Lock Sim pages to the rendered view via an internal record type The record's contentType is stamped text/x-sim-page when apply_file_edit detects page source — the file stays .html to the user (serving and downloads still emit text/html), but every surface now knows what the file holds before content loads. The viewer forces the rendered view for these files at every moment: the first streamed chunk (whose frontmatter is still partial) no longer flashes raw source, the gaps between an agent's tool calls no longer flip back to raw HTML, and both toggle surfaces (the Files toolbar and the resource-panel tabs) stop offering a code view for them. Mid-stream compiles run lenient — a fence still being written is malformed by definition, so its skip-notice callout is suppressed until the stream settles. * Honor the model's declared page type; default copilot .html to a page An explicit contentType on create_empty_file always wins (the skill now declares text/x-sim-page for pages, text/html for bespoke raw pages); with no declaration a copilot-created .html defaults to the page type. The first apply_file_edit still re-confirms from the actual content, and the category map knows the internal mime explicitly instead of falling through to the extension. * Default undeclared .html back to plain text/html A file is a Sim page only when the model declares it at creation or the first written content proves it — never by extension alone. * Match the docs' PageFooter for page navigation The invented bordered cards with Previous/Next labels are replaced by the docs' actual footer: the destination name with a 14px emcn chevron on a flex-1 hover pill (rounded-lg, px-3 py-3, --surface-active), next right-justified, and a spacer holding the empty half — verified against apps/docs/components/docs-layout/page-footer.tsx. * Scroll the rails invisibly, like the docs The sticky TOC box is overflow-y auto, and the clerk track's absolutely positioned SVGs could tip it a few pixels into overflow — Chromium then painted a full scrollbar beside the rail. Rails now hide their scrollbar chrome entirely (scrollbar-width none + webkit display none), matching how the docs scroll their sidebar and TOC. * Send sim:file links to the Files page, like a markdown link A workspace-file link in a page now navigates exactly as one tagged in a .md does: an in-app SPA push to /workspace/{ws}/files/{id} (the Files page with the file open). The fullscreen /view route stays reserved for the standalone surface; image refs keep the /api/files/view byte route. * Inline workspace images after the page compiles, not before The blob substitution ran on the raw source, where the compiled /api/files/view src it looks for does not exist yet — so the sandboxed cookie-less frame fetched every image itself and got 401s (broken image icons). The substitution now runs on the built document, covering compiled pages, legacy stored-compiled pages, and bespoke HTML alike. * Highlight the section you are AT in the left rail, not the last one visible The rail's current-section pick walked every heading visible in the viewport and kept the last h2 — so clicking a section landed correctly but highlighted whichever later section peeked in from below. Current is now the last h2 at or above the top reading line (matching the 72px scroll-padding a clicked anchor settles at), falling back to the first visible section when everything is below the line. * Stop the TOC jittering sideways as the highlight moves Active TOC links step from weight 430 to 470, and the rail is a fit-content column — every active-section change re-measured the longest link and shifted the rail a pixel or two side to side. Each link now carries a hidden zero-height ghost of itself at the active weight, so the column always occupies its bold width and the highlight moves without the layout moving. * Absolutize links and images in served page documents A downloaded page must behave like a downloaded .md whose links are absolute: clicking a workflow reference opens Sim in the browser at that workflow. The standalone renderer (plain download, fullscreen viewer, shares) now compiles sim: links and workspace image refs against getBaseUrl(); in-app surfaces keep relative paths and SPA navigation. * Drop the eyebrow; add the docs' top page controls The docs have no eyebrow line, so compiled pages no longer render one (old sources still parse; the field is ignored). The title row gains the docs' top controls: Copy page (copies the page text) and prev/next chevrons wired to the same neighbors as the footer cards, disabled-dim when a side is missing. * Read kv keys and table first columns as labels, the docs way kv keys dropped the blanket monospace — they render as the docs' row labels (500, primary, sans), with backticks in the source opting a code-like key (a path, an env var) into the inline-code chip; keys now run through inline markdown to make that work. Table body first columns pick up the same label treatment the docs tables show. * Platform font and emcn chrome for pages Pages live inside the app, and the platform — emcn, every workspace surface — renders the system stack, not the docs' Inter webfont; Inter made pages read as foreign next to the app around them. The face is now the platform stack with weights on the platform scale (400/500/600), the Inter delivery machinery (page-font.ts, the preview data-URI fetch, the public woff2) is gone, and the section filter wears emcn ChipInput's exact chrome (30px rounded-lg, --surface-5 fill flipping to --surface-4 in dark, --border, 14px, no focus ring). The docs' geometry — layout, spacing, rails, tables, code frames — is unchanged. * Color-only active state in the TOC — width can never move again Two attempts at reserving the bold width (a hidden ghost, then freezing measured rail widths) each traded one artifact for another: the ghost did not stop the fit-content column re-measuring, and the freeze made a long link wrap to two lines when it gained weight. The root cause was letting the active state change a width-affecting property at all: the active TOC link now shifts color only (muted to primary — the clerk thumb already carries the emphasis), the search input is a plain text field (the native search clear button is not emcn chrome), and the ghost/freeze machinery is gone. * Drop the Copy page control; keep the top chevrons The title-row actions keep only the previous/next chevrons (rendered when the page has neighbors); the Copy page button is gone. * Set-level sidebar: docs-style groups with the current page expanded Multi-page sets can now carry the whole set's sidebar. nav frontmatter (groups of labelled page links, identical on every page of the set) compiles into hidden set-nav markup whose sim: links resolve like any other; the shell lifts it into the left rail as muted group labels over page links, recognises the current page by title, and nests that page's section list beneath it — the docs sidebar's exact shape. Pages without nav keep the plain section list. * Center the content column at the docs' measure On wide panes the 1fr center cell stretched, so content hugged the left rail with dead space before the TOC. The main column now caps at the docs' ~760px measure and centers in its cell, matching how the docs balance a wide viewport. * Docs Steps, code-tab groups, and API method chips Three docs components join the vocabulary: sim:steps renders the numbered timeline (muted circle markers, hairline connector, title and content per step); sim:tabs renders the docs' grouped code block (mono tab chips, one pane at a time, the icon copy control targeting the visible pane); and a METHOD prefix on a set-nav page entry renders the API-reference chip — sidebar entries only, on the platform badge tokens (blue and purple added to the mirrors and the live bridge). Also: preview images switched from blob: to data: URIs — blob URLs are origin-bound and the sandboxed frame's origin is opaque, so Chromium refused to render them — and the page view-lock is sticky per file so a patch stream cannot flash raw source. * Downloaded pages carry their images Absolute URLs made LINKS survive a download, but an embedded image request from a downloaded file is cross-site and carries no session cookie, so images 401ed outside the app. The standalone renderer now inlines every workspace image the page references as a data: URI at serve time — like a pdf carrying its images — capped at 8MB per image, restricted to the page's own workspace, falling back to the URL reference on any miss. Applies to serve, download, and public shares. * Dead-center the content column between equal gutters The rails are content-sized and unequal, so the old grid (fit-content / 1fr / fit-content) skewed the middle cell toward whichever rail was narrower. The wide tier now uses the docs' geometry: a fixed 760px content column centered between two equal flexible gutters, the sidebar hugging the container's left edge and the TOC its right — rail widths can no longer move the content. * Defer title-bar history state out of currententrychange dispatch The Navigation API fires currententrychange synchronously from the history mutation that caused it, which can originate inside another component's useInsertionEffect (style libraries navigating during commit) — setState there trips React's 'useInsertionEffect must not schedule updates'. The arrow-state sync now defers to a microtask, flushing after the commit unwinds, with a disposal guard. * Show the section sidebar only when there are sections to list Fewer than two sections (and no set nav): the left rail and its filter disappear and the reserved left gutter collapses — the content column leads the container with the TOC trailing. Two or more sections, or a multi-page set, keep the full centered docs frame. * Execute the page shell in a DOM harness jsdom runs the real shell against compiled pages and asserts the layout decisions: both rails on a many-section page, only the left rail dropped on a one-section page. * Medium panes keep the TOC, not the sidebar Between 560 and 860px the frame showed the section sidebar and hid the clerk TOC — backwards by our own reasoning, since the sidebar is the redundant list on a single page. The TOC now survives at medium widths and the sidebar joins only on wide panes. * Sidebar is for doc sets only; stagger the rails like the docs The left rail now exists only for multi-page sets — on a lone page it just repeated the TOC. A set opens its sidebar at 560px with the TOC joining on wide panes (the docs stagger); a lone page waits until 700px and then shows the TOC alone. Set-sidebar spacing tightens to the docs values, with the current page's nested sections styled as small muted entries behind a hairline instead of full chips. * Center the lone-page content and TOC as a pair On a page with no sidebar the content column stretched while the TOC hugged the far edge, leaving a field of dead space between them. The content now caps at reading width with the TOC directly beside it and the pair centered in the pane, and the TOC waits until 800px so narrow panes stay single-column a while longer. * Equalize grid-template specificity across the rail tiers The 560px tier selects .art-cols:not(.no-side-nav) at (0,2,0), so the 860px tier's bare .art-cols template at (0,1,0) could never win and a set page on a wide pane kept the 2-column template — wrapping the TOC to the next grid row, bottom-left. The wide template now carries the same :not() guard, the 860 block's duplicate of the 800px no-side-nav rules is gone, and a test pins every template rule to equal specificity so a future tier can't silently lose the cascade again. * extract_doc_assets: pull a reference deck's assets into the workspace Sim-side handler for the new file-agent tool: given an uploaded .pptx or .docx, unzip it (OOXML is a zip), parse theme1.xml into theme.json (color scheme as hex, major/minor fonts, slide size from presentation.xml) and write every ppt|word/media file into a "<Name> assets" folder with original bytes and real content types. Re-runs overwrite the set in place. Pure extractor unit-tested against in-test-built packages; display label "Extracting assets from <file>". * extract_doc_assets learns .pdf via the doc sandbox PDFs have no zip structure or declared theme, so extraction runs in the same vetted sandbox that compiles and renders documents: poppler's pdfimages dumps every embedded image in its native format (masks filtered via -list), pdfplumber contributes each image's placement rects in page points plus the document's font names, and rendered pages are sampled into an explicitly-inferred color palette. theme.json for a pdf carries fonts, page size and count, the inferred palette, and a per-asset placement map. * Pages have one navigation rail; compile errors go to the agent The left sidebar leaves the renderer and the DSL: the shell builds only the content column and the clerk TOC (pair centered at 800px, one bare .art-cols selector per tier so the cascade cannot invert), the filter box goes with it, and nav frontmatter is tolerated but no longer rendered — sidebar METHOD chips and the set-nav markup are gone. Malformed sim: blocks no longer render a reader-facing "block was skipped" card: the block is omitted and the failure is reported as a diagnostic that apply_file_edit appends to its result, so the authoring agent sees exactly which fence to fix. The lenient flag existed only to suppress those cards mid-stream and is removed. The steps timeline connector now derives its position from the marker size, so it stays centered under the number circles. * Sync tool catalog: extract_doc_assets accepts pdf * Asset extraction yields the rebuild recipe, not just the parts pptx: theme.json now maps every image to its slide-by-slide placements (slide rels resolve rIds to media names; each pic frame's EMU offset and extent convert to inches) plus the slide count. pdf: a second layout.json is written — per page, the text blocks with content, position, font, size, and fill color; the filled rects (backgrounds and scrims); and rect-over-image overlay detection with coverage, which is the "image opacity" effect decks fake with a tinted rect. Stream alpha is unrecoverable, so overlays name the color and the rendered page remains the reference for strength. * Split shared-baseline text runs into separate blocks Two text boxes sitting at the same height merged into one wide line; a gap much wider than a space now starts a new block, so columns and label/value pairs land as distinct entries in layout.json. * Extract faithful document layout recipes * Add .chart files: live interactive ECharts docs, static or table-backed * Size charts by width-driven aspect, not panel height; separate title and legend * Map table chart rows from storage column ids to display names * Inject table rows as datasetIndex 0 so specs can transform; stagger array legends * Give .chart files their own bar-chart icon * Chart table sources gain groupBy/aggregate/pivot shaping; renderer-owned chrome * sim:chart page fence: ECharts SSR to themed inline SVG; shared option builder * sim:chart hydrated embeds: inline or .chart file refs, live table reads per serve * Finish extensionless pages and document staging * Render live charts without server hydration * Cover active-theme page token overrides * Keep artifact tokens synced with the app theme * Use tabs for multi-page Sim docs * Preserve dollar-prefixed tool credentials * Build chart specs from validated fields * Sync new integration docs into Copilot manifest * Add in-document tabs to Sim pages * Rebuild page TOC on tab changes * Keep page tabs with docs chrome * Stabilize tabbed page layout * Regenerate docs manifest for staging's Modal docs page * Share one divider between the bar and the chrome tab row * Pin the keyless OCR path in the unreadable-document test --------- Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-20 14:05:46 -07:00
Theodore Li	6f34dd6248	fix(credentials): hide gated service accounts from connected list (#6898)	2026-08-20 16:52:30 -04:00