5793 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Waleed	ccc2ec9507	fix(file-parsers): guard .doc uploads against zip-bomb memory exhaustion (#6166) ... * fix(file-parsers): guard .doc uploads against zip-bomb memory exhaustion DocParser handed the raw upload straight to officeparser and then mammoth, both of which inflate every ZIP entry into memory before any app-level size cap applies. The extension is only a routing hint, so a bomb-bearing OOXML archive renamed to .doc selected the one parser that skipped the guard its docx/pptx/xlsx siblings all call. Adds assertOoxmlArchiveWithinLimits to DocParser.parseBuffer, and centrally in file-parsers parseBuffer so a future parser cannot silently opt out. The guard reads the central directory's declared sizes without decompressing, and no-ops for non-ZIP buffers, so legacy OLE .doc files are unaffected. * fix(file-parsers): verify actual inflation, not just declared ZIP sizes The declared uncompressed sizes in a ZIP central directory are attacker- controlled, so a bomb can under-report them and pass the size and ratio checks untouched. officeparser and mammoth only detect the mismatch after inflating the entry in full: a 498 KB archive declaring 1000 bytes per entry drove 559 MB resident through the .doc parser and 538 MB through .docx, then failed. SheetJS and officeparser reject the container earlier, so xlsx/pptx were not affected, but doc and docx both were. Each entry is now inflated during verification under a maxOutputLength bound equal to the size it declared. Node's zlib aborts the moment output would exceed that bound, so a lying entry costs only its declared size and the inflated bytes are discarded immediately; both bomb variants now reject at +0 MB across every extension. Stored entries are checked against their own compressed size, and unsupported compression methods fail closed. Verification walks the contiguous run of central-directory records rather than the EOCD's declared entry count, since that run is what a decompression library allocates per entry — a lied-down count must not hide an entry from verification. Cost is ~0.45 ms per MB of uncompressed content (22 ms for a 50 MB archive), against parse times an order of magnitude larger. All 17 real Word-produced .docx fixtures in mammoth's test data are still accepted. * fix(file-parsers): require central and local ZIP headers to agree The parsers disagree about which header to trust. JSZip skips the local header outright and decompresses using the central directory's method, while SheetJS's parse_local_file switches on the local header's method and inflates from there. An entry claiming STORED centrally and DEFLATE locally therefore took the guard's stored branch, skipping bounded inflation, and was still expanded downstream — a 398 KB archive hiding a 400 MB deflate payload. Verification now rejects any entry whose two headers disagree on compression method, and on declared sizes when the local header carries them (the data-descriptor flag and ZIP64 sentinels legitimately omit them, and those entries stay covered by the bounded inflate). Caught by Greptile review. All 17 real Word-produced .docx fixtures in mammoth's test data are still accepted. * fix(file-parsers): charge hidden central-directory entries against the cap sumDeclaredUncompressedSize walked only the entry count the EOCD declares, while verification walks the contiguous run of records. JSZip's readCentralDir loops on the record signature and keeps every entry it finds — a count mismatch is explicitly not an error there — so an archive that under-reported its count could hide honestly-large entries from the total-size cap and still have the parser expand them. The sum now walks the same contiguous run as the verification pass and readZipCentralDirectoryStats, and fails closed when the run is shorter than the declared count. Caught by Cursor Bugbot review.	2026-08-01 17:30:23 -07:00
Waleed	5686b7bb83	fix(realtime): enforce room access continuously, not only at join (#6170) ... * fix(realtime): enforce room access continuously, not only at join File-doc, table, and workspace-list rooms authorized once at JOIN and never again, so a member whose workspace access was revoked or downgraded kept live collaborative write access — including durable Yjs document writes — for the whole lifetime of an already-open socket. The access-revalidation sweep explicitly skipped every non-workflow room. - sweep every room type, authorizing each against its own resource - share one membership policy (ROOM_MEMBERSHIP_ACTIONS) between the join check and the sweep, so a file-doc room keeps requiring write in both - gate file-doc document frames and table cell selections on the cached permission, evicting on a confirmed loss of access - re-check the cached decision before a join commits, so a join that authorized just before a revocation cannot re-enter the room - never let a join's own cache write clobber a revocation recorded mid-flight - surface room-access-revoked to clients; the file-doc editor falls back to read-only instead of accepting keystrokes that go nowhere * refactor(realtime): one shared eviction path for revoked room access The sweep and both per-frame gates each open-coded emit + leave + local-state cleanup. Route them all through evictSocketFromRoom so they cannot diverge on what eviction means; workflow keeps its historical access-revoked payload. * fix(realtime): re-check access before workspace-list room joins too The workspace-files / workspace-tables joins committed straight from their authorize result, so a join that authorized just before a revocation could put the socket back in a room the sweep had already evicted it from. Mirrors the guard the file-doc and table joins already had. * fix(realtime): order role-cache writes by read start, not write time Two authorizations can start in one order and finish in the other, so the decision written last can come from the older read. A join that authorized before a revocation but returned after the sweep's denial would bury it, handing the socket another full cache TTL of access. Every writer now takes a monotonic ticket before it queries and yields only to a later-started read. * chore(realtime): drop the test-only unguarded role-cache writer Tests can express the same setup with commitRoomPermission + a read ticket, so the cache has exactly one write path and no export without a production caller. * fix(realtime): keep handler-initiated table eviction retryable Evicting leaves the Socket.IO room synchronously, which is also how the sweep discovers work — so a presence removal failing in the per-frame path could never be retried and left a ghost collaborator until disconnect. Failed (or unconfirmed) removals now hand off to the sweep's existing cleanup lane instead of a second retry loop. * fix(realtime): re-resolve access at join commit instead of peeking the cache The pre-commit recheck peeked the role cache, which reports an EXPIRED entry as unknown and fails open — so a join stalled longer than the cache TTL could re-enter a room the sweep had already evicted it from, including a file-doc room where the next cold-cache frame is accepted as a durable write. All three joins now re-resolve the way the workflow join always has; it is normally a cache hit, since the join's own authorize just warmed it. * fix(realtime): keep the join generation guard after the access re-check The access re-resolve added in the previous commit sat AFTER the generation / superseded guard in the table and workspace-list joins, so a leave or a newer join landing during that await no longer cancelled the stale join — it would go on to leave the room the client had switched to and commit the abandoned one. The guard is now the last thing before the commit in all three handlers, as it already was for file-doc and workflow. * test(realtime): use the shared sleep helper in the new join tests check:utils bans the inline new Promise(setTimeout) form; the two stalled-join tests were the only new offenders. * fix(realtime): leave the prior table room only once the join is certain A table switch left the previous room before the access re-check ran, so a denial there aborted the join and left the client in no table room at all — silently dropped from one it may still be allowed to occupy. The leave now happens after the re-check, matching the file-doc and workspace-list joins. * fix(realtime): close the table join window between re-check and commit Moving the prior-room leave after the access re-check left Redis awaits between that check and socket.join, and superseded() only watches the join generation — so a sweep revocation landing in that window could still put a revoked socket back in the room. A synchronous cache peek immediately before the commit closes it without reintroducing the await; the authoritative resolve moments earlier wrote a fresh entry, so a differing read IS the revocation being guarded.	2026-08-01 17:20:06 -07:00
Waleed	bf78c4c1d6	improvement(settings): unify save/discard across editable surfaces (#6173) ... Save is now always rendered (primary, disabled until there is something to save) and Discard only appears once there is something to discard. Previously sandboxes showed no action at all until dirty — so a new sandbox had no visible Create button — while skills always showed a disabled Save and no Discard. - saveDiscardActions moves to @/components/settings and owns the one rule - adds SettingsActionChip / SettingsActionChips so the settings shell and the ReactNode-slot headers render actions through the same chip path - skills, secrets, connected credentials, custom tools and the secrets manager drop their hand-rolled Save chips for the shared helper - credential-detail drafts seed once per credential instead of re-seeding from every refetch, which was acting as an implicit discard	2026-08-01 17:19:51 -07:00
Waleed	be8a93de81	fix(security): validate cloud region/project inputs and bind vertex credentials to the executing workspace (#6167) ... * fix(security): validate cloud region/project inputs and bind vertex credentials to the executing workspace Two credential-exposure bugs on the Vertex AI path. 1. `vertexLocation` reached `new GoogleGenAI({ location })` unvalidated. The SDK interpolates that value straight into the API hostname (`https://${location}-aiplatform.googleapis.com/`), so a value like `attacker.tld/x` terminates the authority component and relocates the request — with the workspace's GCP bearer token attached by the auth client — to an arbitrary host. Reachable from agent/router/evaluator blocks and from `POST /api/guardrails/validate` with only a session cookie. `vertexProject` (interpolated into the URL path) and the Bedrock region (interpolated into the endpoint hostname) had the same shape of problem. Adds `validateGoogleCloudLocation` / `validateGoogleCloudProject` to the shared input-validation module and applies them, plus the existing `validateAwsRegion`, at the provider chokepoints. Every path to the SDK goes through `executeRequest`, so no caller can bypass them. `azureEndpoint` already routes through the DNS-pinning SSRF guard. 2. `resolveVertexCredential` enforced only the user↔credential predicate and never the workflow-workspace↔credential predicate that `authorizeCredentialUse` applies on the HTTP path. A credential held in workspace B could be pasted into a workflow in workspace A and consumed by workspace-A principals with no access to B — including on deployed runs, where `enforceCredentialAccess` is false and `ctx.userId` is the workflow owner rather than the trigger caller. Service-account credentials mint a `cloud-platform`-scoped token, so this handed workspace A the use of workspace B's GCP identity. The resolver now takes the executing `workspaceId` and rejects a credential belonging to a different workspace, mirroring `credential-access.ts`. All four executor call sites pass `ctx.workspaceId`. * fix(providers): normalize vertex location case before validating Hostnames are case-insensitive, so a mixed-case location like US-Central1 reaches Google today. Lowercase it before the region check rather than rejecting an input that currently works. * fix(security): accept AWS European Sovereign Cloud regions in validateAwsRegion eusc-de-east-1 is a real Bedrock-enabled region that the existing pattern did not cover, so applying the validator to bedrockRegion would have rejected a config that worked before. Adds the eusc-<country> partition.	2026-08-01 17:12:52 -07:00
mzxchandraandWaleed Latif	cd50a445fd	feat(chat): highlight-to-chat — reference file/table selections in Chat (#6087) ... * feat(chat): highlight-to-chat for file and table selections Adds an IDE-style "add to chat" affordance so the Sim agent can reference an exact passage of a file or a specific set of table rows/cells instead of a whole resource. - Two new ChatContext kinds: file_selection (inline selected text + line range) and table_selection (authoritative row ids, optional column ids; the server re-fetches current rows by id). Chip registry, client serializer, boundary contract, and server resolver all extend along their existing switch(kind) seams. - Producers: Monaco context menu, Tiptap bubble menu, and the table grid context menu, all using the Sim Chat block icon. Adding a selection opens the split slideover (chat + resource) with the chip dropped straight into the input; from a standalone Files/Tables page the context is stashed and drained on chat mount. - Cmd+C / Cmd+V: a selection copied from a file/table rides a custom text/x-sim-selection clipboard MIME and pastes into chat as the same reference chip; the chat input round-trips a sole selection chip on copy/cut too. - Guards: selection payloads are length/row/column bounded and truncated within the schema bound; labels carry a deterministic key so distinct same-size selections don't collide; a shared resource tab closes only once no remaining chip references it; stale cell-range column ids drop the context rather than dumping the full table. * fix(chat): widen selection code fence so embedded backticks can't truncate it Cursor: resolveFileSelectionResource wrapped the selected passage in a fixed ``` fence, so a selection that itself contained a fenced code block closed the outer fence early and the agent received a truncated snippet. The fence is now one backtick longer than the longest backtick run in the content (floored at three), matching CommonMark's close rule. * fix(chat): carry the selection chip on a column-header copy Cursor: a column-header Cmd+C always took the async paged clipboard path, which replaces the whole clipboard with text/plain only and can't carry a custom MIME - so pasting a column copy into Chat couldn't rebuild the table_selection chip that Add-to-chat produces for the same selection. When every row is loaded and within the chat-selection cap, the column copy now does a synchronous event write so the scoped table_selection rides alongside text/plain (mirroring the row-'some' and cell-range paths); oversized/partially-loaded columns keep the async plain-text path. * refactor(chat): clean up highlight-to-chat selections Correctness: - Stop reporting fabricated line numbers for rich-markdown selections. `doc.textBetween` counts ProseMirror block boundaries, not markdown source lines, so the chip label and the agent prompt both claimed line ranges that don't exist in the file. Line info is now emitted only by Monaco. - Bound a table_selection's rendered markdown by characters, not just row and column counts — 500 wide rows dwarfed the 20k-char file-selection budget. Rows are emitted until the budget is spent, and the content says what was omitted. - Complete `areContextsEqual` for both selection kinds, so re-adding the same selection dedupes while a different passage of an already-referenced file registers as new. Consistency: - Carry the resource display name on the context instead of recovering it by regex from the chip label, deleting fileNameFromSelectionLabel and tableNameFromSelectionLabel. - Replace the user-visible `#k3f9` hash disambiguator with a readable ordinal applied at insert time (`Sales (3 rows) (2)`), via a shared prepareContextForInsert used by both the add-to-chat and paste paths. - Fold MothershipPendingContextStorage into MothershipHandoffStorage as a chip-only handoff (optional message), removing the parallel storage class, the second drain effect, and its StrictMode guard ref. - Replace `window.location.assign` with `router.push`, matching the existing "Troubleshoot in Chat" handoff. - Collapse three near-duplicate synchronous copy branches in table-grid into shared buildTableSelectionContext / writeLoadedRowsWithChip helpers, also reused by the add-to-chat handler. - Trim multi-paragraph inline comments to TSDoc stating each reason once. Tests: new coverage for the character budget, the label ordinal, selection equality, and chip-only handoff accumulation; each verified to fail when its fix is reverted. * refactor(chat): tighten table copy fallback and helper placement - writeLoadedRowsWithChip now requires a chip to carry; with none it falls through to the canonical paged path (preserving its row loading and truncation notice) instead of doing a bare synchronous write. - Reuse selectedColumnIds in the context-menu memo. - Restore resolveTableSelectionResource's TSDoc, orphaned when renderTableCell was extracted between the doc and its function. * fix(chat): apply chip handoffs as one batch; widen table copy chip path Cursor Bugbot: a multi-context chip handoff dispatched one event per context, and insertContextChip resolved label collisions against selectedContexts read through a ref that only refreshes on render. Each dispatch therefore saw the same stale list, so a second same-label selection was never ordinalized and addContext dropped it while its @token still landed in the text. The event now carries the whole batch and insertContextChips threads each resolved context forward as it goes. Greptile: an explicit multi-row ('some') selection no longer requires every selected row to be loaded before taking the chip-carrying sync path. That gate assumed the paged fall-through would copy more, but its loadRows returns rowsRef.current unchanged for 'some' — the same rows, minus the chip. Renamed the parameter to to say what it actually gates on. Remaining chip-less cases are inherent to the async Clipboard API, which replaces the whole clipboard and cannot hold a custom MIME: a filtered select-all (must page in more rows) and selections past the 500-row chip cap. 'Add to chat' covers both — it is not gesture-bound and drains to the cap. * fix(chat): distinguish a line-less file-selection label from the whole file Cursor Bugbot: a rich-markdown selection labelled itself with the bare file name, which is exactly the whole-file chip's label. Menu-driven inserts reject any context whose label is already taken (isContextAlreadySelected closes the menu silently), so once a markdown selection was attached, mentioning that same file was quietly dropped. One-way: the reverse order works because programmatic inserts ordinalize through prepareContextForInsert. Fallen out of dropping the fabricated line range from this editor — with no range left, the label collapsed onto the file name. Fixed at the single source: buildFileSelectionLabel now returns 'notes.md (selection)' when there is no line range, so it stays honest about location while remaining distinguishable. * fix(chat): don't revive an aged-out chip handoff when accumulating Cursor Bugbot: pendingContexts merged a prior chip-only handoff without checking its age, while store stamps a fresh timestamp on every write. An abandoned handoff that had already passed max-age would therefore ride along on the next 'Add to chat' and fire on the following navigation as if current. Introduced by the accumulation behavior added earlier in this PR. Applied the same freshness bar consume uses, and hoisted the 60s window into a single MAX_AGE_MS constant so the two paths cannot drift. * fix(chat): reference every selected row in a table chip, not just loaded ones Cursor Bugbot: for a 'some' row selection the chip's rowIds came from the loaded-page intersection (currentRows filtered by the selection) rather than rowSel.ids. The chip carries ids and the server re-fetches them via getRowsByIds, so a selected row that simply had not been paged into the grid was silently dropped from the agent's context — select 600 rows with 200 loaded and the agent saw 200. Add to chat had the same loaded-only narrowing through contextMenuRowIds. Both now send the full selection, still bounded by MAX_TABLE_SELECTION_ROWS. Only the pasted text stays limited to loaded rows, which is inherent — there are no cell values to serialize for a row that has not been fetched. * docs(chat): scope the table copy 'complete' comment to the text path It read as though the whole selection were complete when ids are unloaded, which is now only true of the serialized text — the chip deliberately carries every selected id for the server to re-fetch. * fix(chat): compare selection ids as sets, not sequences Cursor Bugbot: sameIds compared rowIds/columnIds by index, but a table selection's ids iterate in click order (they come from a Set), so the same rows picked in a different order — or reached via a cell range rather than the gutter — compared unequal. prepareContextForInsert then added a second ordinalized chip pointing at rows already referenced instead of no-opping. More reachable since the previous commit started sourcing rowIds from rowSel.ids directly, where insertion order tracks the user's clicks. * fix(chat): enforce the table selection budget over the whole rendered content Cursor Bugbot: the budget subtracted only the header and divider before packing rows, then prepended the 'Selected ...' prose and the newlines afterward, so the final content could exceed MAX_TABLE_SELECTION_CONTENT_LENGTH whenever the last accepted row left less slack than the prefix needed. The cap the TSDoc promises was not actually enforced. The prior test passed while missing this: its rows were wide enough that packing stopped far short of the limit, so the boundary was never exercised. Replaced with rows sized to fill the budget almost exactly, asserting both that the content stays within the cap and that it still approaches it (so the assertion can't pass by emitting an empty table). Also capitalize Chat in the three 'Add to Chat' menu labels, matching the constitution's module naming and the existing 'Fix in Chat' / 'Troubleshoot in Chat' UI strings. * fix(chat): don't swallow a paste whose selection chip is already attached Cursor Bugbot: the selection-paste path called preventDefault before prepareContextForInsert, so when that returned null (the same selection is already a chip) the handler returned having claimed the event — no chip inserted and no text/plain pasted either. Cmd+V did nothing. preventDefault now waits until there is a chip to insert; a duplicate falls through to the plain-text paste below, which is the reasonable reading of the gesture. * fix(chat): derive the budget reserve from the same clause it reserves for Cursor Bugbot: worstCaseSizeClause built its own string that omitted the row/rows word the real clause always carries, so the reserve ran ~5 characters short and a tightly packed selection could still exceed MAX_TABLE_SELECTION_CONTENT_LENGTH. A bug in the previous fix, from duplicating the format instead of sharing it. Replaced with one sizeClause(shown, omitted) used for both the up-front reserve and the final prose, so the two cannot describe the count differently. The reserve passes (rows.length, rows.length) — max digits on both counts and the plural forced — which is an upper bound on any real clause. The earlier tight-packing test could not see this: a single cell width leaves whatever remainder it leaves, and 100 left more than 5 characters. Added a sweep over widths 60-75 that collects overflows so a failure names the width; it catches the reported bug at width 74 (20002 vs 20000). * fix(chat): align MothershipChat's onContextRemove with the surface contract The remaining-contexts argument was added to ChatSurfaceContextValue but not to MothershipChat's own prop type, which forwards straight into it. Nothing passes the handler there today so it typechecked (fewer params is assignable), but the two declarations of the same wiring disagreed. Does not change behavior: home still wires the remove handler only to the empty-state surface, as it did before this PR. * refactor(chat): apply cleanup pass findings emcn design: the table context menu built its Add-to-Chat label in the parent and never pluralized it, so right-clicking a single row read 'Add rows to Chat' directly above 'Delete row'. Derive it inside ContextMenu from selectedRowCount like every sibling label, with an addToChatCellScoped boolean mirroring workflowCellScoped. Also more correct: selectedRowCount accounts for a select-all beyond the loaded page, which contextMenuRowIds.length does not. callbacks: drop two useCallback wrappers whose identity nothing observes — both handleAddSelectionToChat handlers feed unmemoized components (one through an inline arrow). buildSelectionContext stays wrapped; it is a real dep of the copy-bridge effect. comments: fold a duplicated rationale paragraph in handleAddSelectionToChat left by two commits stacking, drop a {@link MothershipHandoff} that resolves to nothing (the type is not imported there), and trim a rowIds doc that restated its own type. Skipped, deliberately: the effects pass proposed moving buildContext out of useSelectionCopyBridge's deps behind a latest-ref. buildSelectionContext is already stable, so churn is near-zero, and it would leave two sibling useCallbacks purposeless. * fix(chat): don't attach a selection chip to a copy from a nested input Cursor Bugbot: a copy from a field inside the editor — Monaco's find box being the common one — bubbles to the container while the document still holds a highlight, so the bridge attached the editor selection to text the user never copied. Chat paste then prefers the custom MIME and inserts a reference chip instead of the search term. Skips INPUT targets only. Copying the table grid's INPUT/TEXTAREA guard would have suppressed the chip on the main copy path this hook exists for: Monaco's own editing surface is a hidden textarea, unlike the grid's cell editors, which really are form fields. Tests cover both directions — chip attached from the textarea surface, skipped from a nested input — and were verified to fail against the missing guard and against the INPUT+TEXTAREA variant. * fix(chat): persist the source names a selection chip renders from Cursor Bugbot: fileName/tableName were mapped into the optimistic message and accepted by the API schema, but PersistedMessageContext, buildPersistedUserMessage and toDisplayContexts never carried them. After a reload a file_selection chip fell back to its label for getDocumentIcon — and the label carries a location suffix ('notes.md:12-40'), so extension detection broke. Persists only the two names the display path reads. The rest of the payload (text, rowIds, columnIds, line numbers) stays unpersisted on purpose: it exists to resolve the selection server-side at send time, is never re-read when rendering a past message, and would put a selection-sized blob — up to the 20k char cap — in every stored message. Test asserts both halves of that, and was verified to fail with the mapping removed. * refactor(chat): remove a needless alias and correct an eslint-disable reason Self-audit for shortcuts, prompted by 'nothing hacky': - handlePaste kept `const prepared = preparedSelection`, an alias added only to avoid renaming two downstream lines. Uses the real name now. - The home.tsx drain's eslint-disable claimed handleContextAdd is 'a stable body function'. It is a body function, so it is a NEW value every render — the justification was false. Replaced with the actual reason: it is omitted to keep the drain one-shot, and doing so is harmless because consume() clears atomically, so a re-run would find nothing. Audited the rest of the diff for suppressions, casts and swallowed errors. The three catch blocks are documented graceful degradations with explicit fallbacks (row-drain failure, browsers rejecting a custom clipboard MIME mid-gesture, malformed clipboard JSON); the one double cast is a DataTransfer stub in a test. * fix(chat): bound the sync copy path by the text limit, not the chip cap Cursor Bugbot: writeLoadedRowsWithChip bailed once loaded rows exceeded MAX_TABLE_SELECTION_ROWS, but buildTableSelectionContext already slices rowIds to that cap. So selecting more than 500 loaded rows fell through to the async path, which cannot carry a custom MIME, and silently lost the chip — while Add to Chat on the very same selection still produced a 500-row chip. The two limits govern different things: the chip's cap is how many rows a table_selection can reference, the text's is TABLE_LIMITS.MAX_COPY_ROWS. Gate on the latter. Past it the paged path still takes over, because it owns truncation and the accompanying notice. * refactor(tables): extract selection-to-chip helpers into utils so they are testable Follow-up I owed on the previous round: the copy path's eligibility rule and the context builder were module-private in a ~4,600-line component with no test file, so the last two fixes to them were reasoned rather than covered — and both were wrong on the first attempt. Moves selectedColumnIds and buildTableSelectionContext to the existing table-grid/utils.ts (which already owns RowSelection, DisplayColumn and getColumnId), and extracts the copy decision as canWriteRowsWithChip. writeLoadedRowsWithChip keeps only the clipboard and toast effects, so the pure rule can be tested without a DOM. utils.ts stays free of side effects. New utils.test.ts covers the two limits that were conflated — a selection past the chip cap still qualifies (the context caps its own rowIds), one past MAX_COPY_ROWS defers to the paged path — plus the all-columns collapse and both caps. Verified to fail against the old chip-cap gate. * fix(chat): scope selections to the columns actually picked, and stop under-counting rows Three findings from one Bugbot round. Hidden columns widened cell ranges (reported twice). buildTableSelectionContext and contextMenuColumnIds collapsed a range to an open scope when it covered 'every column', comparing against displayColumns.length — which drops hidden columns AND expands workflow groups, so it never meant 'the whole schema'. Selecting every visible column therefore cleared columnIds and the server re-fetched columns the user had hidden. The collapse is removed rather than re-based on a schema count: no count available to a caller describes the schema, and an explicit column list is what the user actually selected. totalColumnCount is gone from the signature. Add-to-chat label undercounted rows. The menu derived its count from selectedRowCount (loaded rows only) while the chip was built from the full rowSel.ids set, so the label could promise fewer rows than were sent. Both now read one addToChatRowIds memo, with the count passed through explicitly since it legitimately differs from the count the delete/run labels use. Monaco line range was off by one. A full-line highlight ends at column 1 of the FOLLOWING line, so endLineNumber named a line that contributed no text — the chip label and the agent prompt both claimed an extra line. The collapse test I added last round asserted the buggy behavior as correct; it now pins the opposite, and fails if the collapse returns. * fix(tables): cap the Add-to-Chat label at the rows a chip can carry Cursor Bugbot: last round's fix for the label undercounting rows introduced the opposite error. addToChatRowCount passed the raw selection size, but buildTableSelectionContext caps rowIds at MAX_TABLE_SELECTION_ROWS, so a 2,000-row selection advertised 2,000 while the chip referenced 500 — breaking the same invariant the fix claimed to establish. Routes the count through a chipRowCount helper next to the builder that applies the cap, so the label cannot drift from the payload again. utils.test.ts now asserts the invariant directly rather than the formula: across 1, 42, 500, 750 and 50,000 requested rows, chipRowCount equals the rowIds length the context actually carries. Verified to fail if the cap is dropped. * fix(chat): stop a removed chip lingering when its label prefixes another Cursor Bugbot: the mention sync tests each label with a lookahead that rejects only word characters, so '-', ')' and space all let a shorter label match INSIDE a longer token. '@notes.md:12' matches within '@notes.md:12-40', and '@Sales (3 rows)' within '@Sales (3 rows) (2)'. Deleting the shorter chip left its context attached, and it was still sent with the message. The label class is pre-existing, but this PR made it routine: line ranges and uniqueContextLabel ordinals generate prefix pairs for any two selections of the same file or table. Fixed at the sync rather than by reshaping labels to dodge the prefix — a label format chosen to avoid a matcher bug would just relocate it. Contexts are now tested longest-label-first, and each matched token is blanked before shorter labels are tested, so every context is judged against text its own token owns. Blanked in place, not removed, so the (^|\s) boundary of whatever sits next to it is preserved; prev order is still what's returned. Shared with the workflow copilot input, so tests cover both directions: the two prefix pairs are dropped when only the longer token remains, both survive when both tokens are present, order is preserved, and trailing punctuation after a mention still keeps its chip. * fix(chat): include the line range in file-selection equality Cursor Bugbot: areContextsEqual compared only fileId and text for file_selection, while the comment directly above claimed equality was the selected range. A line that occurs twice in a file — a repeated import, a closing brace — highlighted at both places produced identical text, so prepareContextForInsert called the second a duplicate and dropped its chip, even though the labels (notes.md:12 vs notes.md:50) were plainly different. Comparing startLine/endLine as well makes the code match what the comment promised. The rich-markdown editor omits the range, so both sides are undefined there and identical text still dedupes — correct, since two identical passages are genuinely indistinguishable without line numbers. Tests cover both: distinct ranges stay distinct, an exact repeat still dedupes, and the no-line-number path still dedupes. Verified the first fails against text-only equality. * fix(tables): drain past the cap so exclusions can't shrink a select-all chip Cursor Bugbot: for a gutter select-all the menu count comes from selectedRowCount (capped), but the chip was built by loading exactly MAX_TABLE_SELECTION_ROWS and filtering exclusions AFTER. Any excluded row inside that prefix left the chip short of the advertised count. Third appearance of the same invariant — label vs payload — this time in the select-all path specifically, which the earlier fixes did not touch. Loads the cap plus the exclusion count, which covers the worst case where every exclusion falls inside the prefix, so the filtered result still reaches the cap whenever the table has the rows. Extracted as drainTargetForChip so the compensation is stated and tested rather than an inline arithmetic detail. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-01 16:59:45 -07:00
Waleed	d89ab4a8ee	fix(security): bound response bodies in secureFetchWithPinnedIP by default (#6169) ... secureFetchWithPinnedIP only capped a response body when the caller passed maxResponseBytes; with the option absent the body streamed into an unbounded Buffer.concat. Several tool proxies whose target host is user-supplied (Jupyter, ClickHouse, Grafana, 1Password Connect) and the RSS poller called it without that option, so an attacker-controlled server answering with an endless chunked body could grow the shared process heap until it was OOM-killed. Make the cap fail-safe: default to 100MB (and treat a non-positive value as the default) so there is no unlimited mode, then pass tighter explicit caps at the user-supplied-host sites. Responses that carry no body (HEAD, 204, 304) are exempted from the content-length pre-check — they advertise the resource size as metadata, which would otherwise spuriously fail a HEAD probe of a large file or an RSS conditional-GET 304.	2026-08-01 16:54:34 -07:00
Waleed	47e8f1eb5b	fix(ssh/sftp): cap remote file reads on received bytes, not stat() size (#6168) ... The SFTP/SSH download routes buffered a remote file into memory with the only size guard being sftp.stat().size — a value the caller-supplied SSH server controls. A server that reports a tiny size and then streams endlessly drove unbounded heap growth until OOM. Read through the existing readNodeStreamToBufferWithLimit limiter, which enforces the cap on bytes actually received and destroys the stream on breach, via a small readSftpFileCapped wrapper in sftp/utils. All four SFTP-reading tool routes now share it — download, download-file, read-file-content, and the append path of write-file-content, which had no cap at all. The wrapper keeps a no-op error listener on the stream for its whole life: ssh2 rejects still-pending SFTP requests when the channel closes, which arrives as a late error event after the limiter has detached its own handlers, and an error event with no listener would take the process down. Too-large responses now return 413 to match how the rest of the routes surface PayloadSizeLimitError, and responses report the actual byte count rather than the server-reported stat size.	2026-08-01 16:53:46 -07:00
3 people	18214158b9	fix(sanitization): secret exposure in function and agent trace spans (#6000) ... * fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-01 16:37:08 -07:00
3 people	3740c62112	feat(library): Agentic AI Coding Tools: What They Are and How the Top Options Compare (#6164) ... * feat(library): Agentic AI Coding Tools: What They Are and How the Top Options Compare * feat(library): add generated cover for agentic AI coding tools post --------- Co-authored-by: Sim Pi Agent <pi@sim.ai> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-01 11:51:13 -07:00
Waleed	51560390dd	fix(styles): restore transition timing Tailwind was silently dropping (#6165) ... `tailwindcss-animate` adds a `duration-*` utility for `animation-duration`, which collides with core's `transition-duration`. For a named value Tailwind emits both rules, but for an arbitrary value it cannot pick one — so it drops the class entirely and warns. The result is 52 usages across 13 files that produced **no CSS at all**. Elements marked `transition-transform duration-[1700ms] ease-[cubic-bezier(0.22,1,0.36,1)]` had neither a duration nor an easing function: they snapped instead of animating. Most of it is the landing hero. Verified by compiling the stylesheet on both sides: transition-duration: 1700ms before=0 after=1 transition-duration: 420ms before=0 after=1 transition-duration: 175ms before=0 after=1 transition-timing-function: cubic-bezier(0.22,1,0.36,1) before=0 after=1 transition-timing-function: cubic-bezier(0.23,1,0.32,1) before=0 after=1 The stylesheet grows 1,425 bytes — exactly the rules that were missing — and the 16 ambiguity warnings on every dev start go to zero. Every usage is in a transition context (`transition-opacity`, `transition-transform`, `transition-[...]`); none sits alongside an `animate-*` utility, so `transition-duration`/`transition-timing-function` is the right half of the collision in all 52 cases.	2026-08-01 11:50:15 -07:00
Waleed	03649e934c	refactor(dev): remove the minimal-registry escape hatch (#6163) ... * refactor(dev): remove the minimal-registry escape hatch `dev:minimal` existed because the tool registry was 71-82% of every workspace route's module graph and aliasing it away was the only way to make dev bearable. The metadata work removed that reason, so the hatch now buys almost nothing: before this stack 31.7s -> 20.0s cold (-37%) after this stack 22.9s -> 20.7s cold (-10%) A 10% cold-compile win, on the run that happens once — restarts are ~4.2s either way — is not worth what it costs. `tools/registry.minimal.ts` and `blocks/registry-maps.minimal.ts` are 283 lines of hand-curated duplicates of the real registries that **nothing keeps in sync** (no lint, no CI check, no test); they are correct today only because someone remembered. And the mode is actively misleading: it silently drops ~250 services and ~280 blocks, so anything reproduced under it may not reproduce for real. Removes both files, the `SIM_DEV_MINIMAL_REGISTRY` branch from `next.config.ts` (including the whole `webpack()` hook, which existed only for this), and the `dev:minimal` / `dev:full:minimal-registry` scripts. Verified after removal: `tsc` clean, boundary + metadata + skills + monorepo gates pass, and `next dev` starts and serves the canvas at 22.6s cold / HTTP 200. * fix(setup): stop the wizard offering the removed minimal-registry mode The setup wizard prompted for a dev server on machines under 16GB and **defaulted** to `dev:full:minimal-registry` — a script this stack deletes. Anyone running `bun run setup` on a low-RAM machine would have accepted the default and hit "Script not found", which is exactly the contributor the mode existed to help. Repointed at `dev:full:capped`, which still exists and caps Node at 4GB without dropping ~250 integrations — a strictly better answer to the same question. The hints were also stale: they warned the full registry "can use 4-5GB+ on its own", which was true when a dev server sat at 11.5GB. It now sits at ~4GB, so they say that instead. Missed by an earlier sweep because the pattern searched for `dev:minimal` and `registry.minimal`, and this string is `dev:full:minimal-registry` — the two halves reversed. Re-swept across every file type for all spellings: zero references remain. Also audited every script value the wizard can return, so the class of bug is checked, not just this instance.	2026-08-01 11:27:38 -07:00
Waleed	e8894a8764	perf(tools): guard the tool-registry client boundary in CI (#6156) ... * perf(tools): guard the tool-registry client boundary in CI The registry was 71-82% of every workspace route's module graph, and the two edges that put it there were invisible at the call site: `providers/utils.ts` imported `mergeToolParameters`, and `mcp-dynamic-args.tsx` imported `formatParameterLabel`. Neither import looks remotely like "pull in 4,700 modules of SDK clients", which is why this needs a lint rather than a convention. `check-tool-registry-boundary.ts` walks the value-import graph (skipping `import type`, which is erased) from the workspace layout and the four routes that mount inside it, and fails if `@/tools/registry` is reachable — printing the exact chain that reintroduced it. Verified it fails: reintroducing a `getTool` import in `serializer/index.ts` exits 1 and names the chain through `stores/workflow-diff/store.ts`; removing it returns to 0. There is deliberately no allowlist. The fix for a failure is always to move the symbol the file actually needs into a registry-free module, not to exempt the route. Documents the guard in the tool-registry-boundary skill. * fix(tools): close two edge-detection gaps in the registry boundary guard Review found the walker missed two forms, both verified against a matrix of every import/export shape: export * as ns from '…' namespace re-export — the star branch had no alias import('…') dynamic import A dynamic import splits the registry into its own chunk rather than the route's initial one, so it does not show up in cold-compile time — but it still puts 4,300 tools' worth of executable config on a client path, which is what this guard exists to prevent. It counts as reaching the registry. No such import exists today; this is purely closing the hole. Adding both raised the measured counts (tables 1,217 -> 1,261, files 1,310 -> 1,419) because lazily-loaded modules are now counted. The registry stays unreachable from all five entries. Also checked and rejected: side-effect imports (`import '@/x'`) were reported as missed, but are matched both standalone and after another import — the `from` clause is already optional. * fix(tools): resolve extensionful specifiers in the boundary guard `resolveSpecifier` probed `base + ext` and `base/index + ext` but never `base` itself, so an already-extensioned specifier resolved to null and its edge vanished from the walk — `import { tools } from '@/tools/registry.ts'` would have passed the guard silently. Not theoretical: `executor/execution/block-executor.ts` already imports `@/executor/human-in-the-loop/utils.ts` with the extension, so real edges were being dropped. Counts rise slightly now that they are followed (canvas 2,023 -> 2,029). Verified: the extensionful import exits 1, and removing it returns to 0. * fix(tools): discover guard entries instead of listing them Review caught the guard checking the wrong shell: it named `app/workspace/layout.tsx` as "the shared shell every route mounts inside", but that file only wraps `SocketProvider`. The real shell is `app/workspace/[workspaceId]/layout.tsx`, which pulls in `WorkspaceChrome`, the loaders and the providers — and it was never checked. Worse, layouts are composed by Next.js convention rather than imported, so a page's graph never reaches its layout at all. Walking pages alone left every layout module outside the guard. So entries are now discovered: every `page.tsx` and `layout.tsx` under `app/workspace`, 35 of them instead of a hand-written 5. A list goes stale silently; discovery cannot. Refuses to pass vacuously if the walk finds none. Immediately found a real edge the hand-written list had missed — the settings route reaching the registry through a dynamically-imported access-control panel (fixed in the previous commit). Full walk takes ~2s. Also restores the extensionful-specifier fix, which a bad merge had dropped from this file. Re-verified both directions: an extensionful `@/tools/registry.ts` import exits 1, removing it returns to 0. * fix(tools): restore the dynamic-import and namespace-alias edge detection A bad merge during a rebase reverted this file to a pre-fix revision, silently dropping `DYNAMIC_IMPORT_RE` and the `export * as ns from` alias branch that earlier commits on this branch had already added. The guard still passed, which is the worst way for a lint to break — it simply stopped following edges. Caught it because the per-route counts fell after the rebase (files 1,424 -> 1,314, logs 1,610 -> 1,545) rather than staying put. A guard that reports fewer modules after a no-op merge is not passing, it is blind. Now verified against every bypass form rather than the one I happened to think of, so a future regression of this kind fails loudly: CAUGHT extensionful import { tools } from '@/tools/registry.ts' CAUGHT dynamic import('@/tools/registry') CAUGHT ns re-export export * as ns from '@/tools/registry' CAUGHT side-effect import '@/tools/registry' CAUGHT plain named import { tools } from '@/tools/registry' clean tree passes * fix(tools): traverse require() edges in the boundary guard Review flagged `require()` as an untraversed edge form, and it is not hypothetical here — this codebase uses lazy `require('@/…')` to break import cycles, including from a client-reachable file (`tools/params.ts` reaches `@/blocks` that way). Those edges are as real as static imports; a `require` of the registry would have walked straight past the guard. The audit now covers every form a module can be reached by, each verified rather than assumed: CAUGHT plain named import { tools } from '@/tools/registry' CAUGHT side-effect import '@/tools/registry' CAUGHT extensionful import { tools } from '@/tools/registry.ts' CAUGHT ns re-export export * as ns from '@/tools/registry' CAUGHT dynamic import('@/tools/registry') CAUGHT require require('@/tools/registry') clean tree passes No new violations surfaced — the 35 guarded page/layout graphs stay clean with require edges followed.	2026-08-01 11:27:37 -07:00
Waleed	452d82a636	perf(tools): read tool metadata instead of the registry on client paths (#6155) ... * perf(tools): read tool metadata instead of the registry on client paths Cuts the last four edges that pulled `@/tools/registry` into the workspace shell. Every workspace route drops ~4,700 modules: route before after /w (canvas) 6,592 1,908 -71% /logs 6,227 1,543 -75% /tables 5,903 1,217 -79% /files 5,996 1,310 -78% workspace layout 5,751 1,063 -82% Dev cold compile of the canvas, n=3, cache cleared between runs: before 32.3s / 31.4s / 30.1s RSS 9.0-12.5 GB after 22.4s / 22.2s / 21.6s RSS 7.8-9.2 GB That lands where the `dev:minimal` escape hatch measured (20.0s / 6.7 GB) without its downside — `dev:minimal` swaps in curated registries that drop ~250 services, whereas this keeps every tool working. Rewired: - `block-outputs` -> `getToolOutputsMetadata` (needed `outputs`) - `serializer` -> `getToolParams` (needed `params`) - `validation` -> `hasToolId` (needed existence only) - `tools/params` -> `getToolMetadata` (needed `params`, `oauth`, `name`) `tools/params.ts` was the stubborn one: `mcp-dynamic-args.tsx` imports only `formatParameterLabel` from it, so the whole registry rode in behind a string helper — the same shape as the `mergeToolParameters` edge cut earlier. Adds a third generated artifact, `tool-ids.ts` (~110 KB). Resolution needs only the key set, so `@/tools/metadata` and `@/tools/metadata-outputs` both resolve through it and stay independent of each other, and an existence check costs ~110 KB instead of ~4 MB. Behaviour preservation was the risk here: `getTool` resolves an unversioned name onto its newest version, and a plain key lookup would have silently reported 246 versioned tools as missing. `resolveToolId` is reproduced against the id set and differentially tested — 4,404 probes (every id, every stripped base name, and an unknown) comparing old vs new resolution and existence: 0 mismatches. `ToolWithParameters.toolConfig` and `SubBlocksForToolInput.toolConfig` narrow from `ToolConfig` to `ToolMetadata`. The only external reader is `tool-input.tsx`, which uses `.name`. * docs(tools): point the boundary skill at the three metadata modules The skill still routed `hasToolMetadata` and `getToolIds` to `@/tools/metadata`, but this PR moved id resolution into `@/tools/tool-ids`. Left as-is it would send the next caller to the 4 MB module for an existence check that costs 110 KB — the exact mistake the skill exists to prevent. Also records the two properties a caller can silently get wrong: lookups guard with `Object.hasOwn` (a bare bracket lookup returns inherited prototype members), and they resolve unversioned names (246 tools are versioned, and a plain lookup reports them missing rather than crashing). * fix(tools): cut the settings-route registry edge and fix serializer test mocks Two findings from review, both real. The settings route still reached the registry: settings/[section]/page.tsx -> settings.tsx -> (dynamic import) ee/access-control/components/access-control.tsx -> group-detail.tsx -> tools/utils.ts -> tools/registry.ts It reads `getTool(id)?.name` — metadata — so it moves to `getToolMetadata`. The earlier audit missed it because it walked only from the canvas route, and the edge hides behind a dynamic `import()` that a static walk skips. Serializer tests mocked the wrong module. `Serializer` now reads params via `getToolParams` from `@/tools/metadata`, but the tests still only mocked `@/tools/utils`, so they controlled nothing and passed because the real generated artifacts happen to agree with the fixtures. Adds `toolsMetadataMock` to `@sim/testing/mocks`, backed by the same `mockToolConfigs` as `toolsUtilsMock` so a test mocking both sees one consistent tool universe, and mocks it in the three serializer suites. Verified the mock is now load-bearing: pointing it at a sentinel param makes the three user-only-required validation tests fail, and restoring it returns all 110 serializer tests to green. Before this they passed either way. * fix(tools): freeze the tool id array handed out by getToolIds `getToolIds()` returned the module's internal array by reference, so a caller doing `getToolIds().sort()` would reorder it in place and silently corrupt every later lookup — the in-place-mutation footgun `.claude/rules/sim-react-performance.md` calls out. Frozen rather than copied: the array is consumed in loops, so copying would allocate on every call. Freezing makes the mutation throw instead of corrupt, and `[...getToolIds()].sort()` still works. Return type is now `readonly string[]`, so the mistake is a compile error rather than a runtime surprise. No caller mutates it today; this is closing the hole, not fixing a live bug. * test(tools): enforce that the two tool-id resolvers never diverge `resolveToolId` now exists twice on purpose — `@/tools/utils` resolves against the live registry (so a tool added before regeneration still resolves at runtime), `@/tools/tool-ids` against the generated id list (so client code resolves without importing 4,300 tools). Nothing structurally kept them in step; a change to versioning logic in one would silently drift from the other. `tool-metadata:check` now asserts they agree across every id, every stripped base name, and an unknown — 4,404 probes — and only after the staleness check passes, so a missing regeneration reports as staleness rather than as drift. Verified it fails: breaking resolution for `gmail*` exits 1; restoring it passes. It cannot live in a vitest suite. `vitest.setup.ts` globally mocks `@/tools/registry` to an empty map, so `getTool` resolves nothing there — a parity test written as a spec passes or fails for the wrong reason. Both facts are recorded where the code is. Both resolvers stay exported. An earlier pass here un-exported the `@/tools/utils` one as dead; `tools/utils.server.ts` imports it through a multi-line import that a grep missed, and `tsc` caught it. Its doc now says which resolver a caller should reach for instead of leaving two identically-named functions unexplained.	2026-08-01 11:27:37 -07:00
Waleed	d6e08d38d7	perf(tools): generate serializable tool metadata artifacts (#6153) ... * perf(tools): generate serializable tool metadata artifacts Adds `scripts/sync-tool-metadata.ts`, which projects the executable tool registry down to the data half nobody needs a closure for, plus typed accessors over the result. No consumer is rewired yet — that is the next PR. `@/tools/registry` is a ~9,000-line barrel over 4,366 tools. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures (`request.headers`, `transformResponse`, `directExecution`, `postProcess`), and those closures reach every integration's SDK client and parser — which is why reaching the barrel costs ~4,700 modules. Every client-reachable caller was audited: none of them need a closure. They need `outputs`, `params`, or an existence check. Two artifacts, not one. `outputs` is ~4 MB of the ~8 MB and has a single consumer, so it is emitted separately and exposed from its own module; callers needing only params never load it. The data is a JSON string parsed at runtime rather than an imported `.json` or an object literal. That is not stylistic — with `resolveJsonModule` (enabled repo-wide) a `.json` import makes TypeScript infer a literal type for all 4,366 entries: tsc --noEmit, baseline 12.6s tsc --noEmit, with `.json` imports 8m07s (38x) tsc --noEmit, with string literals 12.0s An ambient `declare module` does not short-circuit it (measured: 8m18s), and an object literal is the same inference work. A single string literal is one cheap token for the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. The generator refuses to emit any function value, so shipping executable config to the client fails loudly instead of silently. `hosting` and `schemaEnrichment` are excluded on those grounds — both hold functions and are server-only. Also strips empty param entries: the registry has one (`stt_deepgram_v2`, an `undefined`) which crashes callers that read `param.type` while iterating. `JSON.stringify` drops `undefined` on its own, so the guard is there for an explicit `null` — which serializes faithfully and would reach consumers — and to warn either way. Wires `tool-metadata:check` into CI alongside the other generated-contract gates, and ignores the generated directory in biome (it exceeds the 1 MB limit and was being skipped with a notice on every commit). Adds a `tool-registry-boundary` skill covering which module to import, the three non-obvious properties of the artifacts, and how to verify an edge is actually cut — the canvas route reaches the registry through four redundant paths, so cutting one alone moves the module count by ~1. * fix(tools): harden the metadata accessors against inherited keys Review found two real defects in the generated-metadata layer. `JSON.parse` returns an object with the normal prototype, so a bare bracket lookup resolved inherited members: `getToolMetadata('constructor')` returned a *function* typed as `ToolMetadata`, and `getToolOutputsMetadata('toString')` likewise — silently violating the accessors' documented "undefined if unknown" contract. Guarded with `Object.hasOwn`, with a parameterised regression test over `constructor`, `toString`, `valueOf`, `hasOwnProperty` and `__proto__`. The generator's no-functions scan also gave up past ten levels of nesting. Param and output schemas nest arbitrarily, so a deeper closure would have been dropped silently by `JSON.stringify` while generation reported success — shipping an incomplete schema and defeating the guarantee the scan exists to provide. The depth cap is gone; a `WeakSet` handles the cycles that exposes. * docs(tools): tell tool authors to regenerate the metadata artifacts A new tool now has a second registration step. Client code reads `params` and `outputs` from the generated artifacts rather than from the registry, so a tool added without regenerating them is registered but invisible to the UI — and CI fails on the stale artifacts. `add-tools` and `add-integration` are where someone actually adds a tool, so the step goes in both, next to the registry edit and in each checklist. * docs(blocks): note when a block change needs tool-metadata regeneration Adding a block alone needs no regeneration — it references existing tool IDs and changes no tool's shape. But a change that touches a tool alongside the block does, and this is where that is easy to miss: a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata, so a stale artifact makes the block's declared outputs disagree with what the panel renders (and fails CI). Completes the tool-authoring surface alongside add-tools and add-integration. * docs(tools): cover tool removal in the regeneration guidance The three tool-authoring skills said to regenerate after adding or changing a tool, but not after removing one. Removal is equally breaking and equally guarded: deleting a tool from `tools/registry.ts` without regenerating fails `tool-metadata:check` (verified — exit 1), so a contributor following the skill literally would have hit a CI failure the skill never warned about.	2026-08-01 11:27:36 -07:00
Waleed	9299cee58d	perf(tools): move mergeToolParameters into a registry-free leaf module (#6152) ... * perf(tools): move mergeToolParameters into a registry-free leaf module `providers/utils.ts` imports exactly one thing from `@/tools/params`: `mergeToolParameters`. That function performs no tool lookup at all — it merges two plain param objects. But `params.ts` imports `getTool` from `@/tools/utils`, which statically imports the 4,300-entry `@/tools/registry` barrel, so that one-symbol import was dragging the entire tool registry into every module graph that reached it. Measured with a module-graph walk from each entry: tools/params.ts 4,926 modules (registry reachable) providers/utils.ts 4,926 modules (registry reachable) -> 22 modules ✅ tools/merge-params.ts 2 modules (registry NOT reachable) `mergeToolParameters`, `deepMergeInputMapping` and `isNonEmpty` move to `@/tools/merge-params`, which is forbidden from importing `@/tools/utils`, `@/tools/registry` or `@/tools/params`. `params.ts` now imports `isNonEmpty` from there; its `isRecordLike` and `isEmptyTagValue` imports became unused and are dropped. The two consumers (`providers/utils.ts`, `executor/handlers/pi/sim-tools.ts`) import from the new module directly rather than via a re-export, per the no-re-exports rule. This is preparation, not the payoff. The canvas route still reaches the registry through three other edges (block-outputs, serializer, sanitization/validation) — all four are redundant paths and must all be cut before the route's module count moves. Those follow in the metadata-manifest PRs. Behaviour is unchanged: the moved functions are copied verbatim. * refactor(tools): make deepMergeInputMapping module-private It was exported from `@/tools/params` and imported by nothing — a private helper of `mergeToolParameters` that had leaked into the public surface. Since this move created the module, the export goes with it rather than being carried forward. Verified zero consumers repo-wide before dropping it.	2026-08-01 11:27:36 -07:00
Waleed	28abbc94c6	perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) (#6151) ... * perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) `turbopackFileSystemCacheForDev` has been `false` since #5408 — a landing-page homepage redesign whose description covers hero cards, feature-card aspect ratios, eyebrow chips and a voice-input button color, and never mentions Turbopack, caching, or dev performance. It was collateral, not a decision, and it overrode the Next default (true since v16.1). It is not the flag #6078/#6080 measured. That A/B was `...ForBuild` and its conclusion stands — the build cache is a 3.2x regression and stays off. The two flags look alike and are opposite decisions; both are now commented as such. Measured on `/workspace/[workspaceId]/w`, n=3 per arm, SIGINT between runs: cache OFF 31.4s / 30.1s / 31.9s RSS 9.0-9.8 GB cache ON 5.6s / 5.6s / 5.5s RSS 4.4-5.1 GB 5.4x faster restarts, ~2x less resident memory. Cold compile against an empty cache is unchanged (~32s either way) — the cache only pays back on restart, which is the loop that actually hurts. The cache is unbounded on disk: the abandoned one on this machine had reached 78 GB across 1,848 SST files, and a stale cache is slower to read back, so left alone it erodes the win it exists to provide. `prune-turbopack-cache.ts` runs on `predev` and drops it past a cap (default 20 GB, `SIM_TURBOPACK_CACHE_MAX_GB` to override); `bun run dev:cache:prune` forces it. It never blocks `next dev` on a maintenance failure. Adds a `dev-performance` skill recording the cost model, the reference numbers, and the benchmarking method — including that stopping the server with `kill -9` mid-cache-write discards the cache and makes this exact win read as no win. * improvement(dev): chain the cache prune into dev scripts instead of a predev hook Review read the root `bun run dev` path as bypassing the `predev` hook and so never capping the newly-enabled cache. Turbo does fire `pre*` hooks — verified live, the run prints the prune before `next dev` — but the concern is fair in that the guarantee rested on package-manager lifecycle semantics that are invisible at the call site. Chaining it explicitly removes the question entirely: every `dev` variant now runs `bun run dev:cache:cap && …`, which holds on any invocation path, is visible in the command itself, and drops the three duplicated `predev:*` entries for one shared script. Verified on both paths — direct `bun run dev` and root `turbo run dev`, the latter printing: sim:dev: $ bun run dev:cache:cap && next dev --port 3000 sim:dev: $ bun run ../../scripts/prune-turbopack-cache.ts * docs(dev): document cache-corruption recovery, the cost of enabling the cache Stress-tested the failure mode rather than assuming it: deliberately corrupting an SST block makes Turbopack abort with a FATAL panic — it does not self-heal. FATAL: An unexpected Turbopack error occurred. Cache corruption detected: checksum mismatch in block 4 of 00000221.sst `bun run dev:cache:prune` and restart fixes it; verified the canvas serves 200 again afterwards. Documented in the skill and in the script's header, since the symptom is a hard crash and the remedy is not guessable. This is the honest cost of turning the cache on. It is worth paying — a 5.4x faster restart against a rare, loud, single-command failure — but it should be written down rather than discovered. Worth distinguishing from the adjacent case: an ordinary hard kill does *not* corrupt the cache. Turbopack discards a partially-written cache and rebuilds it silently, which is exactly why a `kill -9`-based benchmark reads as "no cache win" (noted in the benchmarking section). * refactor(dev): drop the dev-performance skill, keep its findings at the code A whole skill was too much for what this is. The parts that are load-bearing — why the two lookalike cache flags are opposite decisions, the measured numbers, the corruption remedy, and the benchmarking trap — now live in the config and script they describe, where someone changing the flag actually reads them. The trap is the piece worth keeping: `next dev` compiles on demand so startup time is meaningless, and stopping the server with `kill -9` makes Turbopack discard a partially-written cache and rebuild silently — which reads as 'the cache does nothing' and is how this flag stayed wrong for a month. Dropped rather than relocated: generic advice that was not specific to this repo (antivirus, Docker-on-macOS, orphaned processes) and a measured no-op (`optimizePackageImports` for lucide-react changed nothing, 31.6s vs 31.7s). * docs(dev): record the measured cost and concurrency behaviour of cache pruning Stress-tested the maintenance path rather than assuming it is free. Cost: the size walk is ~30ms on a real cache and ~85ms at 2,000 files — under 2% of a 4.2s warm restart, and invisible against a cold one. It runs before every dev start, so it needed to be cheap; it is. Concurrency: pruning while a dev server is live (which happens when a second server is started from the same checkout) does not crash it. The running server keeps its in-memory state and kept serving HTTP 200 with zero panics. It does stop persisting for the rest of that session, so its next start is cold once — verified recovering at 23.4s then 4.5s. Worth writing down because the directory silently never reappears mid-session, which looks like a bug if you go looking. The cap is a backstop, not routine: a normal session sits at 1-2 GB against a 20 GB default. * fix(dev): cap every app's Turbopack cache, not just apps/sim `apps/docs` is a Next app too (`next dev --port 3001`) and overrides nothing, so it uses the Next default where the dev filesystem cache is on. It already had an uncapped 1.1 GB cache here, and the root `bun run dev` (`turbo run dev`) starts it — so a teammate using the documented command was accumulating a cache nothing would ever prune. The script now resolves its target from the working directory instead of hardcoding `apps/sim`, and each app chains its own cap. Per-app rather than one sweep on purpose: a single pass would let one app's dev start delete a cache another app is holding open, which costs that session its persistence. Verified both: `apps/sim` and `apps/docs` each report and cap their own 1.1 GB cache, and both dev servers start clean (`Ready in 299ms` / `229ms`, docs serving). * refactor(dev): drop dev:cache:prune in favour of the existing dev:clean `dev:cache:prune` duplicated `dev:clean`, which already existed in `apps/sim` and does strictly more (`rm -rf .next/dev/cache` covers the Turbopack cache plus the fetch and image caches). Two commands for one job is worse than one, and the docs pointed at the newer, narrower of the two. Removes it from both apps and gives `apps/docs` the `dev:clean` that `apps/sim` already had, so the recovery command is the same everywhere. `dev:cache:cap` stays — it is the chained step, used by more than one dev variant, and naming it keeps the relative script path out of each command. Verified `dev:clean` is a real remedy: corrupt a cache block, run it, restart — canvas serves 200 with no panic. Also corrects an overstatement. A damaged cache does not *always* abort Turbopack; whether it panics depends on whether the damaged region is read, so it is not reliably reproducible. Both notes now say "can abort" and give the same remedy either way.	2026-08-01 11:27:35 -07:00
Justin Blumencranz	273ca06ddb	fix(files): serve rendered documents instead of source code (#6139) ... * fix(files): serve rendered documents to attachments and workflow reads Generated documents store their generation source under a .pdf/.docx name and keep the compiled binary in a separate content-addressed artifact store, so any consumer doing a raw read handed out source text under a document name. - Route attachments and readUserFileContent through the servable resolver so they get the compiled artifact, and stop the internal generation-source MIME marker reaching providers as a content type. - Render on read when the artifact is missing. The artifact key is (workspace, source hash), so forking a workspace, moving a file, or editing the source outside a recompiling writer orphaned it permanently and reported "still being generated" forever. Rendering self-heals those and stores the result. - Fall back to serving stored bytes as application/octet-stream when a render fails, instead of failing forever, and remember not to retry those bytes. - Only compile without a workspace context when the file's type positively says it is generation source, so unrelated stored bytes are never executed. - Make the doc-not-ready error opt-in per caller and give it a 409 via HttpError, so output decoration degrades instead of failing completed work. * fix(files): keep render failures retryable and never relabel unrendered bytes Addresses the first review round. - Only memoize a render failure when it is deterministic. A DocCompileUserError means the source will never render, so remembering it is safe; sandbox outages, timeouts, and cancellations are transient and were stranding valid documents for the life of the process. Infra failures now propagate, which also restores DocCompileUserError reaching callers that map it to 409. - Refuse to hand back bytes the resolver could not render. readUserFileContent returns a string, so the resolver's honest application/octet-stream could not travel with it and attachment builders re-inferred a document MIME from the filename — shipping generation source to a provider as a PDF. The file-serve route keeps the graceful passthrough, where a human downloading the bytes is useful. - Normalize the declared type once so a padded or upper-cased source marker cannot pass the resolver gate on one code path and fail it on the other. - Import the doc-not-ready guard lazily. The static import pulled the doc-compile module graph (remote sandbox, task runner, execution limits) into every hydration consumer and broke an unrelated test's module mock in CI. * perf(files): coalesce concurrent renders of the same missing artifact An artifact miss is identical for every concurrent reader — a freshly forked workspace whose document several viewers open at once, or one request whose blocks read the same file — and each was paying for its own compile of the same bytes. Share one in-flight render per (workspace, source, ext) key and drop the entry as soon as it settles, so a later read still re-renders normally. * fix(files): refuse unrendered bytes at the download boundary Addresses the second review round. - Throw UnrenderableDocumentError from downloadServableFileFromStorage instead of returning bytes with an `unrendered` flag. Around 45 call sites (email attachments, cloud uploads, zip entries, provider attachments) receive only a Buffer and re-infer the type from the filename, so a flag they must remember to check is a flag they will not check. Those callers already handled the previous not-ready throw, so failing is the shape they expect. The file-serve route is unaffected — it resolves bytes directly and keeps the graceful passthrough, where a human downloading the file has a use for it. - Surface that failure through hydration: with throwOnDocNotReady set, the caller cannot use a file with no content, so an unrenderable document now reaches it verbatim instead of degrading to null and reporting a misleading "may exceed size limit or no longer accessible". - Stop a shared render inheriting one caller's cancellation. The coalesced run no longer carries any caller's signal; each caller races its own instead, so an aborting reader gives up promptly while the render finishes for the others and still lands in the cache. * fix(files): move the unrenderable error out of the 'use server' module file-utils.server.ts carries 'use server', whose exports must all be async functions, so exporting an error class from it failed the production build with 67 cascading errors. The class now lives in the plain file-utils.ts beside the other shared file helpers, which also lets the hydration path import it directly instead of through a dynamic import. Also bounds how long a failed render is remembered. The isolated-vm engine cannot tell a bad source from a sandbox outage, so a permanent entry let one transient failure block re-rendering that source for the life of the process. Entries now expire after five minutes: long enough to stop a read loop spending a sandbox run per read, short enough that an outage self-heals without a deploy. * fix(files): finish the render cancellation and failure-surfacing edges - Race the E2B render against the caller's signal too. Only the isolated-vm branch did, so an aborted request on the E2B path waited for the sandbox to finish and could return a success the caller no longer wanted. - Attach a terminal handler to the shared render. Every caller races it against its own signal, so all of them can walk away; a later rejection with no waiters left would otherwise surface as an unhandled rejection. - Stop narrowing what throwOnDocNotReady rethrows. readUserFileContent now runs document compiles and can fail in ways this module has no business enumerating; narrowing produced three consecutive review rounds of "this particular failure is still swallowed". The flag means "do not degrade". - Do not mark an unrendered response immutable. The serve route caches versioned responses for a year, which would pin a one-off render failure to that URL long after a later compile succeeds on the same version. * revert(files): drop the concurrent-render coalescing The coalescing was an optional efficiency win — rendering is content-addressed and idempotent, so duplicate concurrent renders produced the same artifact and cost only extra sandbox time on an artifact miss. It bought that at the price of the most intricate code in the change set, and produced three concurrency findings across two review rounds: a shared render inheriting one caller's cancellation, an E2B/isolated-vm asymmetry in how the signal was raced, and orphaned rejections once every caller could race away. Removing it also restores true cancellation on the isolated-vm path: the caller's signal now reaches runSandboxTask again, so an abort cancels the sandbox work rather than only abandoning the wait for it. * fix(review): simplify generated document attachments * fix(files): mock servable downloads in hydration tests * fix(files): preserve rendered attachment semantics * fix(files): preserve cached artifact size * fix(files): refuse unresolved xlsx source * fix(files): resolve execution artifact workspace	2026-08-01 10:56:02 -07:00
Waleed	06506bbd38	improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path (#6149) ... * improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path - Cache the agent-stream ProseMirror↔Yjs binding metadata per session and reuse it across streamed frames instead of rebuilding it (O(doc)) every frame; matches how y-tiptap's own binding maintains the mapping in place. Safe because the shadow doc only ever sees the agent's own reconciles. - Default createFileResponse to a private, no-cache policy so auth-gated bytes are never stored in a shared cache; genuinely-public serve routes opt into public caching explicitly. - Serve content-addressed (key=) embedded images with an immutable private cache to avoid re-downloading them on every doc re-open; fileId= embeds and public shares keep revalidating (the underlying key can change / a share can be revoked). - Drop the dead conflict.version field from the collab-doc persist result and remove the wasted getWorkspaceFile re-read on the conflict path (the relay treats missing and conflict identically and never read version). - Decorate-sort the files/folders lists (compute each sort key once) and index the move-menu subtree build (O(N) vs O(N^2)); ordering is preserved exactly. * fix(files): serve inline images with no-cache so deletion/authorization is enforced per request Embedded images are authenticated content whose backing file can be deleted or have its access revoked at any time. Both key= and fileId= embeds serve private, no-cache, must-revalidate, so every request re-runs the server-side deletion/authorization check instead of serving a possibly-stale image from cache.	2026-07-31 20:09:45 -07:00
Waleedandmzxchandra	10bfb5d139	feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991) ... * feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on b8f28b04b: - Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to avoid clobbering a local in-progress resize), so refetching the definition on a peer never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not a no-op refetch. Structural changes still propagate via 'schema'. - Overlay now also observes the content layer with the ResizeObserver, so a column resize (which grows the content, not the scroll container) re-measures remote outlines. - Presence-merge comment now states both sides of the trade-off. * fix(tables): re-broadcast local selection on (re)join Cursor Medium: a selection made before the room join completes (or held across a reconnect) was dropped server-side and never re-sent, so peers didn't see it until the local user moved it again. Track the current selection in a ref (set on every emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is joined. * fix(tables): re-broadcast selection when a peer's row change shifts it End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable (rowId, columnId) only on selection/editing change, not when a live edit/schema refetch inserted/deleted/reordered rows. The index-based local selection then sat on a different logical row than the rowId peers held, so your outline showed on the old row until you moved. Re-run the emit on rows/displayColumns change and dedup an unchanged result (also drops the redundant null-on-open emit) so the broadcast stays consistent with the local highlight. * fix(tables): schema invalidates run-state/enrichment + guard stale join Cursor round on cdc8796b8 (2 Medium): - schema handler used detail exact:true, so it skipped the activeDispatches + enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a prefix match. After a peer deletes/restructures a workflow group, peers could keep a stale running badge or enrichment panel. Now invalidates both siblings too (rows stay on the debounce). - Guard against a stale join stealing the room: a fast table A->B switch could let A's async authorize finish after B, leave B, and strand the socket in A. Added a per-socket monotonic join generation checked after authorize (mirrors the file-doc relay's guard) + a test. * feat(tables): live column width/pin/order sync Collaborators now see each other's column resizes, pins, and reorders live — the last piece of Google-Sheets-style layout parity. - New lightweight `metadata` durable event kind (distinct from `schema`): only the table definition carries UI metadata, so peers refetch the definition alone — no rows/run-state refetch. The metadata PUT route now signals it. - The grid reconciles server metadata against its in-progress gesture: the column being actively resized keeps its live local width, and an in-flight column drag blocks a reorder apply — so a peer's change never reverts the local action. Each field is reference-guarded (React Query structural sharing keeps unchanged sub-objects stable), so an unrelated peer change doesn't re-apply the others. * fix(tables): escalate to schema signal when a reorder scrubs group deps Independent audit of the metadata-sync commit found a stale-run-state hole: a columnOrder PUT that moves a column left of a workflow group's leftmost column makes updateTableMetadata scrub that group's dependencies and write a new schema — a real structural change. But the route only fired the lightweight 'metadata' signal (detail-only refetch), so peers' and the actor's activeDispatches / enrichmentDetails queries stayed stale (a lingering running badge / enrichment panel) — exactly what the 'schema' handler exists to prevent. updateTableMetadata now reports whether it scrubbed the schema; the route emits signalTableSchemaChanged in that case and the light signalTableMetadataChanged otherwise. Width/pin/plain-reorder stay on the cheap detail-only path. * feat(realtime): accurate in-file presence + collaborative-caret polish (#5965) Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests. * feat(files): smarter bullet delete/indent and untitled-file title sync (#5971) * improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption Backspace at the start of a list item now outdents a nested item or clears a top-level item to a paragraph in place instead of deleting the row and jumping the caret to the previous block; Enter on an empty nested item outdents. Empty non-trailing top-level items still collapse cleanly since they cannot round-trip as a lifted paragraph. Also strips nested empty list-item marker lines on serialize: a nested empty bullet re-parsed as a Setext heading underline, silently turning its parent line into an H2 and dropping the bullet. Top-level empty items are preserved. * feat(files): sync an untitled file's name with its leading heading While a file is still named untitled(.md), typing a leading heading auto-renames the file after it (debounced), and renaming the file first seeds a leading H1 from the new name. One-shot: coupling stops once the file has a real name, and the heading seed always prepends so existing content is never clobbered. * fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace Addresses review findings on the list Backspace logic: - Emptiness now uses the caret block's content.size (counts inline images/mentions), not textContent, so a bullet holding only a non-text atom is no longer treated as empty and deleted. - An empty first block whose item has sibling blocks removes only that block instead of lifting the whole item out of the list. * fix(files): preserve the untitled to named heading seed across a rename during editor load The parent captures the file name at mount (before content/session finish loading) and passes it as the transition baseline, so a rename that lands in the loading window is still seen as an untitled to named transition and the leading heading seed is not skipped. * fix(files): drop the name-to-heading seed, keep title sync one-way Removes the effect that inserted a leading H1 when an untitled file was renamed. On the collaborative Files page every open client observed the untitled-to-named transition and inserted into the shared doc, producing duplicate headings; it could also re-insert a heading a user had just deleted while a rename was in flight. Seeding document content from an async rename transition is the wrong model on a shared editor. The primary direction — typing a leading heading renames a still-untitled file — is unaffected (it never mutates the doc). * fix(files): keep empty lines between paragraphs on reload The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds from runs of blank lines — a saved visual blank line silently vanished on the next load (the settle/reopen re-seed goes through the chunker). The whole-document parser preserves them, but whether a gap yields an empty paragraph is a global, block-type- dependent decision (kept between two paragraphs, dropped after a heading), so it can't be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes; ordinary single-blank-line separation still takes the fast chunked path. Adds a suite asserting chunked output matches the whole-document parser for leading/trailing/between gaps and around lists/headings. * fix(files): only auto-name an untitled file when the user can edit The debounced untitled→filename hook ran on every onUpdate — including the mount-time seed and for view-only viewers — without checking edit permission, so a read-only user could schedule a rename they have no permission to make (a spurious, server-rejected write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready, the same signal the autosave path uses), at both schedule and fire time. * fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry - markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic \r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked, dropping empties. Normalize line endings once up front so the routing guards, the chunker, and the parser all see the same \n. +CRLF/CR test cases. - keymap: Enter on an empty first block of a multi-block item now removes only that block (removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks case — the trailing check no longer swallows multi-block items. +test. * fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip) A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness) surfaced these: - HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines → serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are unaffected. +regression tests. - Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits only; false for non-collab surfaces). - Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty item DIRECTLY under a shallower parent line — matching the function's own docstring intent. Probe-verified. +test. - Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling removed during review. * fix(files): a remote edit must not cancel the local rename debounce The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so a peer's edit arriving within the 600ms window cancelled the local user's pending rename. Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it. * docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip The stacked trailing-empty-paragraph strip made the older comment overstate a correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks, and advertise dead CRLF handling. Reword to match what the code actually does. --------- Co-authored-by: Waleed Latif <walif6@gmail.com> * fix(realtime): access-revalidation multi-room safety + cleanup pass Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the access-revalidation sweep (staging's workflow-only #5917) treated every entry in socket.rooms as a workflow id, but the generalized multi-room model puts namespaced files/tables/file-doc rooms on the same io. It would resolve those as bogus workflows, get null, and evict files/tables collaborators every ~30s. collectScanTargets now decodes each room name with parseRoomName and sweeps only workflow rooms; added a regression test and fixed the now-false TSDoc. Other audit fixes (all behavior-preserving): - workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated from avatar.ts) - PresenceAvatars: mr-1 was baked into the shared component, silently adding a margin to the workflow sidebar stack; moved to an optional layout className, re-applied on the tables/file-doc header surfaces only - table DELETE routes only signal collaborators when rows were actually removed (matches PUT) - events.ts definition kind: drop the never-emitted reason:'schema', fix its doc - event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis success path too, not just the memory fallback) - remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals in handleWorkflowDeletion; gate the table columnIndexById map on remote selections; move file-doc module TSDoc off the FileDocOwner interface; fix a stale @returns * fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps Validated each issue with subagents before implementing the cleanest fix. - tables LEAVE in-flight-join race (B8): the table handler tracked no current-table intent, so an unscoped/same-table leave during an in-flight authorize left the socket stranded in the room (present in the roster, broadcasting a ghost until disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that advances joinGeneration to cancel the racing join. + 3 regression tests. - v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update. Add the signals at those call sites (add-only, matching the existing route seam) — never in the service, so execution writes can't double-emit. Sync-only for copilot bulk ops, guarded on affected/deleted count; async job branches stay covered by their kind:'job' events; create/delete/get untouched. - table join read consolidation (B5): sweepStalePresence returns its roster so the same-tab dedup reuses it instead of a second getRoomUsers. - shared authorize slice (B6): extract only the guard-safe authorize->allowed branch into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays inline — file-doc's generation capture sits mid-ladder and must not move). - resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps local widths leading until the width PUT settles. - embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the embedded surface stops broadcasting cell selections the server drops. * fix(tables): close two copilot live-collab signal gaps + harden presence sweep Follow-ups from a comprehensive review of the branch: - copilot batch_update_rows and import_file's inline append branch wrote rows but emitted no live-collab signal, so collaborators didn't see those edits live (the append's sibling replace branch already signalled). Add the guarded signal to both, matching the internal route. - sweepStalePresence now reads the roster before the fetchSockets liveness probe and returns it on a probe failure, so same-tab dedup still runs during a transient fetchSockets outage instead of being skipped. - reword an internal comment off the retired "mothership" term. * fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup Review round on #5991: - Table join re-checked the generation only once after authorize, then awaited leave/sweep/avatar before joining + registering presence. A table switch or leave in that window stranded the socket in the wrong room, and the failure catch could tear down a newer successful join. Resolve the avatar up-front, re-check generation immediately before the membership commit (matching the file-doc join), and skip the rollback/error for a superseded join. + a post-authorize-window regression test. - access-revalidation cleanup treated removeUserFromRoom's no-op false as a transport failure and re-enqueued a still-connected socket forever. Only retry when the socket is still mapped to the room (a healthy null mapping means the entry is already gone). Repurposed the expired-mapping test to lock it. * fix(realtime): guard table join leave-prior against superseding join Round 2 on #5991: a superseded join's leave-prior could still run — during its getRoomForSocket await a newer join commits to its room, so currentRoom is that newer room and the superseded join would leave/remove/broadcast it before the final guard aborts. Re-check the generation immediately after the lookup await, before the leave mutation. Extended the post-authorize-window test to assert the superseded join never tears down the newer join's room. * fix(realtime): roll back a table join superseded during addUserToRoom Round 3 on #5991: after the final generation guard, A could join + register presence while a newer join B commits to its room during addUserToRoom's await — B's leave-prior can't observe A's half-written entry, so A's late write wins and strands the socket. Re-check after addUserToRoom and roll back A's own Socket.IO join + presence (scoped to A's room, never touching B). + a regression test hanging addUserToRoom mid-commit. * refactor(realtime): DRY table-join supersession guards; fix stale comment Cleanliness pass after the review rounds (no behavior change): - Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected` checks into a named `superseded()` helper (the catch keeps its intentionally narrower check). - Remove a stale guard comment that was left stranded above the avatar resolve. - Document the best-effort rollback catch. * fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost Two Cursor findings on the file-doc client-id ownership rebind: - On a document switch, the prior room was left BEFORE the ownership check, so a CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining the new one (contradicting its own comment). Run the ownership check first, and leave the previous doc only once the rebind is guaranteed to succeed. - Reclaiming a client id removed the stale prior socket from owners + awareness only; its socketToRoomName + Socket.IO membership remained, and handleMessage's SYNC path gates on socketToRoomName (not owners), so it stayed able to write document frames until disconnect. Fully evict the reclaimed socket. + 2 tests. * refactor(realtime): serialize table join/leave to fix map-corruption at the root Round 4 on #5991 surfaced a race the generation guards structurally cannot fix: two concurrent joins for one socket race on the single-valued socket→room map — a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry to A, and the rollback then wipes it, stranding the socket (map empty while it holds table B). Guards protect JS suspension points; they can't stop an in-flight Redis write from landing late. Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket promise chain so their multi-step async Redis commits can never interleave — restoring the atomic-commit property the synchronous sibling handlers get for free. This DELETES the leave-prior guard and the post-commit rollback (the code that caused the bug); four generation guards collapse to two identical superseded() checks (skip a superseded queued op + one pre-commit check). Reworked the interleaving-specific tests into a fast-switch-skips-superseded test; the leave- cancels-join tests are unchanged. Local to the tables handler — no shared-infra change. * fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim Two review findings: - Table join: the catch skipped rollback when superseded, but a socket.join that landed before addUserToRoom threw leaves the socket in the Socket.IO room with no matching socket->room map entry — unreclaimable by any later op (cleanup keys off the map). Under serialization the skip is unnecessary (the newer op hasn't committed), so always roll back. Simpler + fixes the strand. - File-doc reclaim: fully evicting the prior socket didn't release the seeder role if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set) never re-elected and an unseeded doc stayed empty until the deadline. Clear the role on eviction so the join's election picks a new seeder. + 2 regression tests. * fix(realtime): close revoke-race ghost presence in workflow join An access-revalidation revoke landing between socket.join and addUserToRoom socketsLeaves the socket while its presence mapping does not yet exist, so cleanupEvictedSocket finds nothing to remove and the join then writes presence for a socket already out of the room — a ghost collaborator until the stale sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth check so the whole re-auth -> socket.join -> addUserToRoom section is await-free, matching the invariant the handler already relies on for the pre-join re-auth. + ordering regression test. * refactor(realtime): serialize workflow join/leave; drop dead room-authz limb Comprehensive independent audit follow-ups: - workflow.ts join/leave now use the same opChain + joinGeneration serialization as the sibling handlers (tables, file-doc, workspace-files). It was the only async presence path left unserialized, so a rapid workflow switch A->B (or a leave racing an in-flight join) could strand presence in room A — a ghost collaborator still receiving A's operation broadcasts until disconnect. The join now aborts a superseded op at start and again right before the membership commit, and the catch always rolls back a partial join. - leave-workflow drops the '&& session' gate: an idle user whose 1h session key expired (while the 24h room mapping is still live) can now leave cleanly instead of being stranded until disconnect. The room ref alone suffices. - authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its getActiveWorkflowContext import. Workflow authorizes through its own path and never flows through authorizeRoom; the map now honestly covers only the workspace-scoped types (files, file-doc, table). - Remove unused isSameRoom (zero callers) and a needless useMemo in PresenceAvatars (plain derivation, single copy). - Tests: 4 workflow serialization/leave regressions. All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol tests, biome, api-validation, boundaries, prune 14/25. * fix(realtime): align session TTL, harden committed joins from post-success rollback Per-module comprehensive audit follow-ups: - redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room set outlived the session, and since getRoomForSocket reads the room set while the workflow handlers gate edits/presence on `room && session`, an active-but-idle collaborator got wedged into 'session expired' after 1h — sticky until reload (activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it). Both keys refresh together, so they now expire together (restores the pre-refactor consistency, where both shared one TTL). - workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live collaborator after success was already acked). - workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow re-join no longer leave→re-adds and flickers peers' presence), and the join ack is liveness-filtered via filterVisiblePresence — both for parity with the tables handler. - platform-authz: honest docstring + 400 message for the workspace-scoped-only authorizeRoom map (workflow authorizes via its own path). - caret-presence: corrected an over-stated batching comment. - +1 workflow regression test (post-success failure keeps the user joined). Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast Two Cursor findings on the prior audit-fix commit: - The 'committed' guard in join-workflow/join-table was too broad: a failure BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit 'if (committed) return' and emitted neither success nor error, hanging the client while it sat in the room. Replaced with the narrower shape: only the purely-decorative post-success steps (peer broadcast + log-metric) are wrapped best-effort; anything before the success ack still rolls back and surfaces a retryable error, so the client retries instead of hanging — while the original goal (a benign broadcast/metric blip never kicking a live, acked user) holds. - broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis transport error to []. On a disconnect broadcast that emitted an empty roster and cleared every remaining collaborator's presence until the next healthy update. Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the broadcast on a read failure (getRoomUsers keeps its swallow contract). - Tests: pre-success failure rolls back + retryable error (no hang); post-success failure keeps the user joined. Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune. * fix(realtime): harden seeder recovery, join-generation, and misc robustness Final line-by-line audit follow-ups (all LOW/MED, no P0/P1): - file-doc: a sole client whose seed FETCH fails was added to triedSeeders, re-election found nobody, and the document stayed permanently empty until reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before giving up. Also bound clientId to a non-negative integer (it is an ownership key). - tables + workspace-files: validate the room id BEFORE advancing joinGeneration, so a malformed/rejected join can't cancel a legitimate in-flight join. - workflow + tables + workspace-files: suppress the client-facing join error when the op was already superseded (a retryable error naming the abandoned room could make a client re-join and cancel its newer join). The rollback still runs. - redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it on failure) so isReady() can't report ready while the Lua SHAs are null. - connection: apply the presence-bearing filter to the manager-removed set too (symmetry with the fallback path). - http.ts: validate workflowId on the four workflow endpoints (matching the files one). - client: clear a pending join-retry timer before rescheduling (reconnect churn no longer orphans a stray extra join); clear caret fade timers on plugin destroy; seed-effect cleanup reports NOT-ready (safe direction). - Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip, presenceEventName. Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): gate isReady() on loaded script SHAs, not just connection Follow-up to the prior isConnected change, which was incomplete: the redis client's 'ready' event flips isConnected=true on connect — before initialize() loads the Lua scripts — so a bare isConnected check reports ready while removeUserFromRoom / updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs too, so the POST endpoints return a retryable 503 during that startup window instead of proceeding against unloaded scripts. Standard readiness-probe discipline. * fix(files): offline read-only fallback when the realtime doc never syncs When the realtime server is unreachable (offline, server down, socket never connects), the collaborative editor would sit blank and read-only forever — content only arrives via provider sync events that never fire. Add a bounded connect-deadline to the Yjs provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a synthetic non-retryable join-error — the exact path a real fatal rejection already uses, which seeds the file's stored content read-only. Latching fatal also stops a late reconnect from syncing server state in and merge-duplicating the locally-seeded content (the documented Yjs 'non-empty doc ignores initial value' gotcha). Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap can't run the markdown->Yjs conversion server-side (Collaboration extension errors under jsdom), and a durable binary snapshot would create a dual source of truth with the markdown file (edited by copilot / PUT / download). The client-seeder + bounded re-election is the correct architecture for a markdown-is-truth model; this closes its one real user-facing gap without persistence, a migration, or dual-truth. Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests. Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification. * fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target Two Cursor findings: - join-workflow bumped joinGeneration before validating workflowId (unlike tables / workspace-files, which I'd already fixed). A malformed/empty join could advance the counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test. - The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in client-id reclaim), that binding still points at the socket's PRIOR, valid document — so the rollback tore down a document the socket was validly in. Only run that cleanup when the binding already points at THIS join's target; otherwise the socket never registered as an owner here and the only leftover is a freshly-created empty room, dropped by destroyRoomIfIdle. Gates: realtime tsc 0, 209 tests, biome, boundaries, prune. * fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset Final safety-audit findings: - CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect deadline latched fatal and the editor fell back to a read-only local seed, a late SyncStep2 (slow server / flaky network / deploy) was still applied — merging server state into the seeded doc (content duplication) and flipping synced=true, which un-gated autosave and would persist the duplicate to the real file. fatal guarded (re)join but not inbound sync. Now handleMessage returns early when fatal. +test (late SyncStep2 after the deadline is ignored, doc stays empty + gated). - file-doc join catch now suppresses the client-facing error when superseded (matches workflow/tables/workspace-files) so a retryable error for an abandoned file can't make a client re-join and cancel the newer one. - table/workspace-files room hooks reset the retry budget on (re)connect so a prior full exhaustion doesn't block retries after a reconnect. - presence-visibility: corrected a stale TTL comment. Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune. * feat(collab-doc): server-authoritative Yjs seeding (#6008) Server-authoritative Yjs seeding for collaborative file documents: the realtime relay fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and applies it once per room, replacing the client-seeder election/handshake entirely. - DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc) reusing the client markdown engine for parity by construction - Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the client readiness deadline, guard-release for join-driven retry, read-only fallback on persistent failure - Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone build (serverExternalPackages + outputFileTracingIncludes) Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C. * feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation) buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them. Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the current state and applies the returned diff. Tested incl. concurrent-edit no-clobber. * feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff, applies it (relaying to every editor, reconciled with concurrent user edits), and reports 'no-live-room' so the caller falls back to a direct file write when nothing is open. - Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a shared POST helper + fetchFileDocSeed/fetchFileDocMerge - POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied } - Tests for the seeded-room merge relay and the no-live-room fallback * feat(collab-doc): stream copilot edits into open editors (Stage C) edit_content now, after its durable file write, best-effort merges the same markdown into the file's live collaborative document (markdown files only). If a collaborator has it open, the edit streams into their editor as a CRDT merge — reconciled with their concurrent typing — instead of the file silently changing under them; the editor's existing autosave mirrors the merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit. * fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown - buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as the seed does. Copilot passes full-file content, so without this the frontmatter merged into the doc as editor content and autosave wrote it back over the file (corruption). - Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the collaborative editor never renders. * fix(collab-doc): make frontmatter collaborative so a merge can't revert it The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge (which triggers an autosave with no user action) could write stale YAML back over a copilot frontmatter change — silently dropping it. Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed stores it, the merge updates it (only when it actually changed, preserving the no-op diff), and the editor re-attaches THAT value on save — falling back to the locked copy before the seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it. * fix(collab-doc): re-sync draft on a frontmatter-only merge A server edit that changes only the frontmatter updates the config map but not the body fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time frontmatter, and an explicit save could revert the live change. Observe the config map and, on a frontmatter-only change, re-attach the new frontmatter to the current body and push a fresh draft (guarded on a synced body so it never races the seed's own onUpdate). * fix(collab-doc): order the apply-edit/merge timeouts (outer > inner) The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge timeout (8s), so the outer call could abort while the relay was still merging — the relay then applied the merge after edit_content had returned, racing a follow-on edit. Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s so it always outlives the inner merge. Cross-referenced in comments to prevent drift. * fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races) A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core patterns are idiomatic) surfaced these real issues: - The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it. - The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness) living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in @sim/realtime-protocol with a test asserting the ordering; both apps import it. - The copilot live-merge gate used an extension-only check, but the editor treats any text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware isMarkdownFile mirroring the client, so those files stream too. - applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each diff the same stale snapshot and apply out of order. Serialize per file via a promise chain. - Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift). - Add .max() bounds to the merge contract body; document the best-effort-merge failure window honestly (open editor + merge failure can drop a copilot edit until reload — closed by the deferred durable-doc work). * feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence Make collaborative file-doc editing correct across multiple ECS tasks (the per-process Y.Doc previously assumed one replica per file). - Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each file's stream is the ordered, replayable log of updates; a multiplexed XREAD tailer converges every task's in-memory doc. Coordinated single-seeder election (SET NX + empty-stream recheck) fixes split-brain seeding. - Doc-sync fans out to local clients + the stream; awareness stays on the Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries. - Server-side persistence: project the live doc back to markdown via a new /api/internal/file-doc/persist endpoint, debounced during editing and flushed on last-disconnect, from the authoritative stream state. Collaborative editors no longer client-autosave, closing the copilot clobber-window. - Copilot merges apply through the stream (reach the live doc on any task) and serialize cross-task via a Redis merge lock. - Degrades to the original single-replica behavior when REDIS_URL is unset. * fix(collab-doc): harden distributed locks and durability from review Address Greptile + Cursor review of the multi-replica backend: - Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never merges against a shared base while a peer holds the lock) and AWAIT the stream write before releasing, so the next task never diffs a stale base. - Distributed locks (seed/merge/compact) now use ownership tokens + a compare-and-delete release (Lua), so a lock that expired and was re-acquired by another task is never stolen; acquisition fails CLOSED on Redis error. - Seed: publish the seed to the stream AWAITED under the lock before releasing, so a later seeder's empty-stream fence always sees it (closes the fence's publish-after-release gap); TTL kept at the readiness deadline. - Persist: persist the AUTHORITATIVE stream state even when this task's local doc was never seeded, and capture the local fallback synchronously so a last-disconnect flush never encodes an already-destroyed doc. - Publish: retry a transient xAdd failure so a Redis blip can't silently drop an edit from the shared log. * fix(collab-doc): only persist a doc a user actually edited Cursor review: a copilot durable write landing while a doc is being seeded could have the stale seed projected back over it on last-disconnect, clobbering the copilot edit even with no user changes. Gate server-side persistence on a genuine user edit (socket-origin update): a seed-only or merge-only doc is never projected back to the file (copilot writes the file durably itself), so it can't clobber a concurrent external write. * fix(collab-doc): close review-round race/durability gaps Address Greptile P1 + Cursor findings: - Seed publishes to the shared stream AWAITED *before* seeding the local doc, so a publish failure leaves the doc unseeded and the stream empty for a clean retry rather than serving an unpublished local seed a peer would re-seed over (split-brain). - flushPersist falls back to the synchronously-captured local snapshot when getStreamState throws (not only when it returns null), so a transient Redis read no longer drops the final durable write as the room is torn down. - streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no longer let the seed fence pass and double-seed. - Client collabReady initializes from the collaborative prop, so a collaborative editor never has a mount-window where client autosave could clobber the server write. * fix(collab-doc): unstick seed retry and persist tailed edits Cursor review: - ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the room unseeded with no retry. - Persistence dirty-tracking now marks a doc edited on any post-seed update, including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a seededObserved flag. The last task to leave persists real edits even if it only tailed them; the seed transition itself is still never counted, so a seeded-but-unedited doc is never projected back over the file. * fix(collab-doc): match client markdown post-processing on server persist Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only, but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter. Server persist could therefore write markdown differing from a client save (empty list markers, callout un-escaping) — spurious blob churn / round-trip drift despite the byte-identical claim. Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is byte-identical to a client save and the client's dirty-check baseline. Add a regression test guarding the composition. * fix(collab-doc): close durability gaps at deploy boundaries + audit polish From a comprehensive from-scratch audit (correctness, SOTA, cleanliness, feature-completeness): - Persist max-wait: a continuous edit burst kept resetting the 5s debounce and never persisted; cap it so a burst flushes at least every 20s, bounding unpersisted edits. - Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling deploy / scale-in secures open edited rooms to durable markdown before exit, instead of relying on the stream + a surviving task. - Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a snapshot folds seed+edits into one frame, so a task catching up purely from it no longer treats real edits as an unedited seed and skips persisting. - Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer object-identity guard; fix stale comments (seed route, edit-content autosave). * improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle - collab editor no longer latches a spurious 'Unsaved changes' prompt: report dirty only when the client owns durability (canAutosave), since in a collaborative session the relay persists the doc server-side - guard shutdown against a double SIGINT/SIGTERM running teardown twice - disconnect local sockets before httpServer.close so shutdown exits gracefully instead of hitting the forced-exit timer (local-only, deploy-safe) - return 400 (not silent 200) on an invalid workspaceId in the files-changed fanout - clear a pending join-retry timer on reconnect so it can't fire a duplicate join * fix(realtime): make collab-doc seeding atomic to close split-brain window An adversarial concurrency audit found a split-brain double-seed vector: the seed used an advisory SET NX PX lock + a SEPARATE xLen fence + an unconditional xAdd (a non-atomic check-then-append). If the seed lock's TTL expired mid-seed (a >4s stall after the 8s seed fetch), a second task could acquire the freed lock over a still-empty stream, both fences read empty, and both append seeds with distinct Yjs client ids -> duplicated document content. - add an atomic SEED_IF_EMPTY_SCRIPT (append-iff-empty in one Redis step) + store.seedIfEmpty(); the emptiness check and append are now inseparable, so two tasks racing (even both past an expired lock) can never both seed - ensureServerSeed uses seedIfEmpty instead of streamHasContent-fence + publishAndWait; the seed lock is now purely an efficiency optimization (avoid a duplicate fetch), not a correctness dependency - fix the misleading comment that claimed a copilot merge is not counted as an edit in the multi-replica path (it round-trips as REDIS_ORIGIN and does count; a safe idempotent over-persist, never a lost edit) - add interleaving tests: seedIfEmpty atomicity/fence, the split-brain regression under an expired lock, a peer edit during attach catch-up, and concurrent two-task compaction * docs(realtime): align seed comments with the atomic-append correctness model Greptile flagged lingering doc drift: the module + shouldSeed comments still credited the seed lock + empty-stream check as the split-brain fix. Reframe them so the atomic seedIfEmpty is the exactly-once guarantee and shouldSeed is an efficiency gate only. * feat(realtime): live workspace tables list, sharing one invalidation-room impl (#6053) * feat(realtime): live workspace tables list, sharing one invalidation-room impl Bring the tables list to parity with the files list: a create/rename/move/delete/ restore now propagates to every viewer live instead of waiting out the 30s staleTime. Following the files pattern, but factoring the two into one shared implementation rather than copy-pasting. - add ROOM_TYPES.WORKSPACE_TABLES + its authz resolver (workspace-id-addressed, reuses the workspace resolver like workspace-files) - extract setupWorkspaceInvalidationRoom (server) and useWorkspaceInvalidationRoom (client) — the presence-free, workspace-scoped live-list room; files and tables now both bind to it, so they can never drift. Event/room names derive from the room type. Replaces the standalone workspace-files handler + hook - notifyWorkspaceTablesChanged fanout fired from the table service (createTable, renameTable, moveTableToFolder, deleteTable, restoreTable) so it covers both the HTTP routes AND copilot, which call the service directly - relay /api/workspace-tables-changed endpoint; wire the hook into the tables page - consolidate the handler test into one suite run against both room types * feat(realtime): live tables list also covers table-folder mutations Fold in the follow-up: a table folder create/rename/move/delete/restore now propagates to the tables list live too, so the browser is fully consistent. - generic notifyFolderResourceChanged(resourceType, workspaceId) dispatches the workspace live-list signal by resource type (a map, not a special-case if), so file/knowledge_base/workflow are no-ops today and gain liveness by adding a map entry when they adopt an invalidation room - fired from the shared folder lifecycle (createFolder/updateFolder/deleteFolder/ restoreFolder), covering routes AND copilot - the tables room hook now invalidates the table folders query too, not just the tables list, since the page renders both * fix(realtime): skip per-table live-list notify during a folder cascade A folder delete/restore already fires one folder-level notifyFolderResourceChanged for the whole subtree, but the cascade also calls deleteTable/restoreTable per table — each awaiting its own notifyWorkspaceTablesChanged. A folder with many tables would run N+1 sequential relay calls (each bounded by NOTIFY_TIMEOUT_MS), blocking the mutation. Add a skipNotify option the cascade passes so only the one folder-level notify fires. * feat(collab-doc): Hocuspocus binary persistence + Next 16 seed/persist fixes (#6059) * fix(collab-doc): make server-side seed conversion work under Next 16 / Turbopack Opening a file left both collaborators read-only and stalled ~12s: the server-side seed (markdown -> Yjs, run through the headless editor engine) was failing, so the doc never seeded and the editor never left its readiness gate. Two root causes, both latent until a real build/runtime (typecheck + unit tests don't exercise either), surfaced by the Next 16 upgrade: 1. Build boundary: the server seed route imported the shared editor schema (`createMarkdownContentExtensions`), which pulled in the React node-view components (`useEffect`) -> 'client component in a Server Component'. Split each node's React-free schema into its own `*-schema.ts` (code-block, image, raw-markdown-snippet); the client editor still injects the React node views via the existing `nodeViews` param, unchanged. 2. Runtime DOM: the converter installs a jsdom `window` on `globalThis`, but Turbopack's server bundle gives bundled `@tiptap/core` a `window` that does NOT read `globalThis`, so `elementFromString` threw 'no window object available'. Externalize the `@tiptap/*` packages the converter uses (native Node require, so their `window` reads the real global) and fix the converter's DOM guard to gate on `window` (what TipTap checks) with no sticky flag. Verified: seed route returns 200 with the Yjs update; 514 collab-doc + editor tests pass; schema byte-identical after the split. * feat(collab-doc): persist the Yjs binary and load it on cold-start (Hocuspocus pattern) Adopt the industry-standard Hocuspocus store/load-document pattern so a cold room open loads the file's last-persisted Yjs binary directly instead of re-converting markdown -> Yjs on every open. Rebuilding the CRDT from markdown on each connect is the exact anti-pattern Tiptap/Yjs warn against (fresh client ids -> duplicated content); it also forced the fragile server-side headless-editor conversion on every open. Now conversion runs only on a genuine first open or an external markdown edit. - new table workspace_file_collab_state(file_id PK->workspace_files cascade, doc_state bytea, source_hash, updated_at): the Yjs binary + a hash of the markdown it was derived from (bounded <=~1MB by the 256KB round-trip gate). Mirrors Hocuspocus's extension-database (binary in a DB column). Migration 0275. - persist upserts the binary (tagged with the exact markdown just written) - cold-start seed returns the cached binary when its source_hash matches the file's current markdown; otherwise converts (and the next persist refreshes the cache) - also externalize yjs / y-protocols / lib0 alongside @tiptap: bundling loaded a second yjs copy, so @tiptap/y-tiptap's 'instanceof Y.XmlElement' failed on app-created nodes ('Unexpected case') during Yjs -> markdown Verified end-to-end: seed -> persist -> seed returns the exact persisted binary (a cache hit, no re-conversion). 18 collab-doc + 51 realtime file-doc tests pass. * fix(collab-doc): best-effort cache read + drop dead barrel - seed: a cache-read failure (transient DB error, not-yet-migrated cache table) no longer aborts a cold room open — the durable markdown is already in hand, so fall through to conversion. Symmetric with persist's best-effort cache write. Addresses the Cursor Bugbot finding on the read/write asymmetry. - remove the collab-doc index.ts barrel: nothing imported it (every consumer uses direct ./seed / ./merge / ./converter imports), so it was dead re-export surface. De-export COLLAB_DOC_FIELD accordingly — it is used only inside converter.ts. * fix(collab-doc): stream every external file write into open editors, not just edit_content (#6070) * fix(collab-doc): stream every external file write into open editors, not just edit_content A copilot/mothership edit to an open markdown file did not appear live in another user's editor: the live-doc merge bridge (mergeEditIntoLiveFileDoc) was wired into the edit_content tool ONLY. Every other server-side write — the file tool (/api/tools/file/manage), function_execute (/api/function/execute via writeWorkspaceFileByPath), create_file overwrite, and the PUT /content route — went straight to updateWorkspaceFileContent and skipped the merge, so the durable file changed but the open editor never updated. Confirmed from live logs (the mothership 'Prepend sentence' ran read + file + function_execute — zero apply-edit calls) and Redis (the prepended text was absent from the doc stream). Centralize the merge at the one chokepoint every external writer shares: - updateWorkspaceFileContent gains an opt-out "syncLiveDoc" (default on) and, after the durable write, merges markdown writes into any open collaborative doc (best-effort; no-op when nobody has it open). Any current OR future writer is covered automatically. - persist.ts opts out (syncLiveDoc:false) — it IS the doc→markdown projection, so merging it back would be a persist→merge→persist self-loop. - create_file opts its empty shell out (real content arrives via a later write) so an open editor never flickers to empty on overwrite; threaded through writeWorkspaceFileByPath. - edit_content drops its now-redundant explicit merge call (the chokepoint handles it). - binary writers (image/video/audio/ffmpeg/download) are naturally excluded — the merge is gated to markdown, the only format the collaborative editor renders. Also bump the api-validation route baseline 994→996 to match the true route count already on this branch (pre-existing ratchet drift from an earlier merge; NOT added by this PR). * fix(collab-doc): defer setEditable out of the render phase (flushSync warning) The collab editability-reapply effect called editor.setEditable synchronously. In collab mode isEditable flips from readiness (synced + seeded), which is driven by a Yjs config.observe firing synchronously inside Y.applyUpdate — so the effect can run while React is mid-render. TipTap's React binding commits setEditable's transaction with flushSync, which throws "flushSync was called from inside a lifecycle method. React cannot flush when React is already rendering." Defer the setEditable to a microtask (runs right after the current commit, before paint), guarding against a destroyed editor or a stale value before it fires. Only the collab path (this effect) hit the warning; the streaming/settle effect's setEditable calls run on the non-collab path where isEditable isn't driven by a mid-render Yjs observer. * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) (#6073) * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) The non-collaborative streaming/settle effect called editor.setContent / setEditable / setTextSelection / focus directly in the effect body. setContent mounts the custom node views synchronously through the @tiptap/react flushSync path (tiptap#3764), so when this effect runs while React is mid-render it throws "flushSync was called from inside a lifecycle method." This is the second flushSync source (the collab editability effect was the first, fixed separately); it fires on the agent-streaming-into-a-non-collab-editor surface. Defer the effect-body view mutations to a microtask via a small runOffRender helper (runs right after the current commit, before paint; no-ops if the editor was torn down). The settle block is deferred as ONE microtask so setContent -> collapse selection -> setEditable -> focus keep their order. The streaming rAF tick is left untouched — it already runs off-render, so it keeps writing content directly. queueMicrotask is TipTap's own documented remedy for this warning. 497 rich-markdown-editor tests (incl. stream-settle-selection) pass; tsc + lint + api-validation + boundary + prune green. Needs a live check: stream an agent into a non-collab markdown file and confirm it still renders smoothly. * chore(rich-markdown-editor): trim verbose flushSync-defer comments * fix(rich-markdown-editor): drop superseded settle/stream microtasks via a run token runOffRender previously only guarded editor.isDestroyed, so if React ran the next reconcile pass (a newer stream or settle) before a queued microtask flushed, the stale microtask could still apply setContent/setEditable/setTextSelection over the newer state. Tag each effect run with an incrementing token; a deferred mutation applies only when its run is still the latest (and the editor is alive). A run token fits this effect's several early-return exits better than a per-exit cleanup flag. Addresses Greptile/Cursor review. * fix(rich-markdown-editor): never drop the settle selection-collapse under a superseded run The run token drops a superseded settle's microtask, but the settle had already flipped its state flags synchronously — so a pre-empting steady-sync run took the non-settle path and never collapsed the selection, leaving a post-stream select-all painting the leaf-in-selection decoration. Track the collapse as a debt (pendingCollapseRef): whichever deferred run ultimately applies — settle or the steady-sync path — clears it, so the collapse runs exactly once on the latest content. Addresses the Cursor review finding. * feat(tables): show live cell-selection carets in the embedded chat panel (#6081) The table cell-selection presence room was joined only on the dedicated /tables/[id] page (useTableRoom was passed an empty id in embedded mode). Join it in embedded too, so the mothership chat resource panel shows collaborators' live cell selections and broadcasts the local one. tableId is already resolved from props in embedded (the data event stream already uses it un-gated), and authz runs on join, so this is safe. Avatars are unaffected — they render only in the !embedded Resource.Header, so the panel gets carets without avatars. * feat(collab-doc): If-Match optimistic concurrency so persist never clobbers an out-of-band edit (#6085) * feat(collab-doc): optimistic-concurrency guard so persist never clobbers an out-of-band edit The relay projected the live Yjs doc back to durable markdown unconditionally (last-write-wins), so a persist already in flight when an external write landed could overwrite it. Add RFC 7232 If-Match optimistic concurrency end to end, reconciling through the CRDT (never rejecting user work): - updateWorkspaceFileContent gains an expectedUpdatedAt guard: the write commits only if the file is still at that version (checked against the SELECT ... FOR UPDATE-locked row, so it is atomic with the write), else it throws the new ContentVersionConflictError without clobbering. - persistFileDoc takes expectedVersion and returns a discriminated result (persisted | missing | conflict). On conflict it returns the current durable content + version instead of writing. - The relay tracks the durable version its live doc is synced to — set on seed, advanced when a durable write is merged in (apply-edit carries the version), and on each successful persist. It is held cluster-wide in Redis (filedoc:syncver:{name}) so whichever task persists reads the same version, with the per-room value as the single-pod fallback. - flushPersist sends that version as If-Match. On a conflict it merges the current durable content into the live doc (so the out-of-band edit AND the live edits converge) and retries (bounded), so even a last-leave flush racing an external write persists the reconciled result rather than losing the session's edits. Threads the version through the seed + persist contracts and the apply-edit payload. No schema change (reuses workspace_files.updatedAt as the version token). Tests: app-side CAS (match writes, mismatch throws + cleans up the orphan upload), relay conflict handled gracefully without clobber/loop; 236 realtime + 76 sim collab/uploads tests, tsc x2, lint, api-validation, boundaries, prune all green. * chore(collab-doc): heartbeat-refresh the synced-version key TTL alongside its stream Keep filedoc:syncver:{name} alive as long as the room's stream (it was only re-set on seed/merge/persist), so an open-but-idle doc's persist If-Match token can't expire and force a needless reconcile. * fix(collab-doc): stop persist-conflict retries when there is no live doc to reconcile On an If-Match conflict with no live doc to reconcile into (last collaborator gone, no shared stream), applyMarkdownToLiveFileDoc returns no-live-room; re-projecting the same pre-teardown snapshot would only re-conflict, so break the retry loop immediately and leave the out-of-band (durable) content authoritative — the intended conflict policy. Addresses Greptile review. * fix(collab-doc): close three optimistic-concurrency edge cases from review - Single-pod persist retry projected the pre-reconcile snapshot (captureState always returned the initial localState), while the synced version had been advanced by the reconcile — so the If-Match could pass and clobber the reconciled edit. captureState now re-reads the live doc on each attempt (falling back to the pre-teardown snapshot only once the room is gone). - The synced version was recorded from this task's own seed FETCH before knowing whether this task's seed actually won; a peer winning with a different version could leave a newer token than the stream content. Record it only inside the didSeed branch (the task whose seed won); peer-seeded tasks read the winner's cluster value. - Persist wrote UNCONDITIONALLY when no version was available (relay version momentarily missing), which could clobber non-empty durable content. It now returns conflict for a non-empty file with no version (reconcile/retry once the version is re-established); an empty file's first write stays unconditional. * fix(collab-doc): defer (not reconcile) on missing version, and use the freshest version token - Missing-version persist now returns 'deferred' instead of 'conflict'. A missing version token (a Redis blip on a peer-seeded task) is NOT a genuine out-of-band change, so triggering a reconcile would wipe live edits (incoming-wins) even though nothing changed durably. Deferred means: don't write, don't reconcile — leave the edits in the stream and let a later persist write them once the version is re-established. - currentVersion now takes the MAX of the cluster (Redis) and local room versions rather than always preferring Redis, so a lagged/failed fire-and-forget Redis set can't shadow a newer local value and cause spurious If-Match conflicts. Versions are monotonic epoch-ms, so the larger is the later sync. * fix(collab-doc): make persist If-Match teardown-race-immune and recover missing version on final flush Close two last-leave concurrency holes Cursor flagged: - Thread the reconciled version LOCALLY through the persist retry loop. After a conflict+reconcile the correct next If-Match is exactly result.version, so carry it in a local var instead of re-deriving from room.syncedVersion/Redis. On a last-leave flush destroyRoomIfIdle removes the room from the map before the async flush finishes, so mergeMarkdownIntoRoom's recordVersion can no longer update room.syncedVersion — threading makes each retry's precondition correct by construction, immune to that dropped mutation and to a best-effort Redis re-read. - Cache the resolved version back into room.syncedVersion in currentVersion() so a peer-seeded/tail-only task (which never sets it locally) or a later transient Redis read failure still resolves it from the last value seen (monotonic max, never regresses). - On a FINAL flush, briefly retry resolving the If-Match when the version read momentarily fails, rather than deferring and stranding the session's edits in the TTL'd stream — the version is cluster-wide and heartbeat-refreshed. * fix(collab-doc): stamp cluster sync version the moment the seed wins, before the liveness guard The winning seeder set the If-Match token (room + Redis filedoc:syncver) only after the liveness/seeded guard that follows seedIfEmpty. But the tailer can integrate the just-appended seed DURING the seedIfEmpty await, so isDocSeeded(room.doc) is already true when the guard runs and it returns early — leaving the stream holding seed content with no cluster version. Later persists then send no If-Match, the app returns `deferred`, and session edits stay only in the TTL'd stream (the exact stranding this PR prevents elsewhere). Move the version stamp to immediately after seedIfEmpty wins, before the guard. Recording it only once our seed won (not from the fetch) is preserved, so it still can't shadow a peer's winning seed. * fix(collab-doc): make the synced-version token monotonic at every write site The If-Match token is written fire-and-forget from the seed stamp, merges, and persists, both locally and to Redis. An out-of-order write (e.g. a seed's lagged setSyncedVersion landing after a later merge's) could regress it below the version the live doc already incorporates, causing spurious If-Match conflicts — and on a last-leave flush with no live room to reconcile into, a spurious conflict leaves durable authoritative and drops the session's edits. - setSyncedVersion now writes via SET_VERSION_IF_NEWER_SCRIPT (Redis-side compare-and-set): it overwrites only when the new value is greater, refreshing the TTL either way. - recordVersion / the persisted branch / the seed stamp all take Math.max instead of assigning room.syncedVersion directly. Versions are monotonic epoch-ms, so "newer" is a plain numeric compare, exact within a Lua double. * fix(collab-doc): close three last-leave persist edge cases from review - Stale snapshot after reconcile (High): the multi-task captureState fell back to the pre-await localState snapshot even after a reconcile advanced ifMatch, so a failed stream re-read could persist the pre-reconcile state against the new version and clobber the out-of-band edit the reconcile just incorporated. NULL localState after a reconcile so a failed read aborts instead. - Lock miss aborts reconcile (Medium): a merge-lock acquisition failure returned 'no-live-room', indistinguishable from an absent stream, so flushPersist treated transient contention as terminal. Return a distinct 'merge-unavailable' and handle it as retry-later (edits stay in the stream), never as "nothing to reconcile into". - Peer syncver never recovers (Medium): the winner's setSyncedVersion was fire-and-forget with swallowed errors — the only way a peer-seeded task learns the durable version — so a dropped write left that peer deferring forever. Make it retry (bounded) like appendUpdate/seedIfEmpty; the monotonic script keeps a racing retry a no-op. * fix(collab-doc): scope the persist If-Match to a content version so metadata bumps can't clobber edits The optimistic-concurrency validator was `updatedAt`, which rename/move/delete/restore also bump with no content change. A racing live-doc persist then saw a stale token, got `conflict`, reconciled the pre-edit durable body via updateYFragment (incoming-wins on overlap), and wiped the user's in-flight edits. Scope the validator to content (RFC 7232 semantics — validate the representation, not the row): - New `workspace_files.content_updated_at` (NOT NULL, `now()` fast-default — no table rewrite). Advances ONLY on content writes (upload / overwrite / create); metadata writes never touch it. - The FOR UPDATE CAS, the merge-notify version, and the seed version all use `content_updated_at`. A rename now leaves it unchanged, so the persist If-Match still matches -> no spurious conflict, no reconcile, no lost edits. Genuine out-of-band content writes still conflict and reconcile. - Consolidated the collab schema into one migration (the collab-state table + the new column) per request, rather than a separate follow-up migration. Relay/store/contracts unchanged (still a numeric monotonic version). * chore(collab-doc): condense the densest persist comments (no behavior change) Cleanup pass: tighten the three longest comment blocks added while hardening the persist path (currentVersion cache, ifMatch threading, final-flush version retry) without dropping any invariant. No dead code found (biome lint clean; all new symbols referenced). * fix(collab-doc): persist must return the content version, not updatedAt Follow-up to the content-scoped If-Match: persistFileDoc still returned `updatedAt` as the version in both the persisted and conflict results, while the CAS/seed/merge all guard on `content_updated_at`. A content write sets both to the same instant, so it was coincidentally correct — until they diverge: if a metadata write bumps `updatedAt` past `content_updated_at`, the conflict path returned the larger `updatedAt`, so the relay's re-persist sent an If-Match the CAS (which checks `content_updated_at`) could never match → perpetual conflict → dropped reconciled edits. Return `contentUpdatedAt` in both paths so the relay's token always matches what it's checked against. * fix(collab-doc): defer persist whenever the version is missing; guard the content-version test - Empty-file CAS race (Medium): the unconditional-write carve-out for size===0 read `record.size` outside the write transaction, so a concurrent first content write could land after the check and be clobbered. With content_updated_at NOT NULL every existing file always has a real version, so a missing expectedVersion is always transient — always defer, never write unconditionally. Removes the TOCTOU hole. - Content-version test (Low): the merge-chokepoint test kept updatedAt == contentUpdatedAt, so it passed even if wired to the wrong field. Mock distinct values and assert contentUpdatedAt, so a regression to updatedAt now fails the test. * fix(collab-doc): don't reconcile a conflict the live doc already reflects (would wipe newer edits) flushPersist reconciled the durable body into the live doc on every conflict. But when the conflict comes from a racing self-persist (or an apply-edit the chokepoint already merged), the durable body is a STALE SUBSET of the live stream, and the incoming-wins updateYFragment merge moves the doc backward — wiping newer in-flight edits, which the retry then persists. Before reconciling, re-check the freshest synced version. If it already covers the conflict version, the live doc has already incorporated that content (or is ahead), so skip the reconcile and just retry with the freshest version as If-Match — the re-projection captures the current live stream, preserving every edit. Only a genuine out-of-band change the live doc hasn't incorporated (freshest < conflict version) is reconciled in. freshest never exceeds the durable version, so this can't loop. * fix(collab-doc): make content_updated_at monotonic per file; skip-reconcile can't loop The If-Match token was stamped with app-local new Date() on each content write, so cross-instance clock skew could stamp a later write with an EARLIER content_updated_at — breaking the version ordering the whole optimistic-concurrency scheme (and the skip-reconcile branch's freshest>=version assumption) depends on. Under skew the relay's monotonic syncedVersion could exceed the durable version, sticking the If-Match: persist conflicts forever, exhausts retries, drops the session's edits. - Stamp content_updated_at strictly after the current committed value (we hold the row's FOR UPDATE lock): new Date(max(now, currentFile.contentUpdatedAt + 1ms)). Monotonic per file regardless of clocks; also removes same-millisecond collisions. updatedAt stays plain wall-clock (display/sort). - Skip-reconcile branch retries with result.version (the durable value the CAS will match), never freshest (which could exceed it and loop). Belt-and-suspenders now that the version is monotonic. * refactor(collab-doc): drop the destructive in-persist reconcile; adopt-version-and-retry on conflict The in-persist reconcile projected the durable body back over the live doc via updateYFragment ("make the doc match"). That is destructive: when the live stream is already ahead — the common case, because the write chokepoint (mergeEditIntoLiveFileDoc) already merged the out-of-band change into the stream — it moved the doc backward and wiped newer in-flight edits. This produced a run of races (stale snapshot, wipe-newer-edits, version-lag skip miss) that a full-document reconcile fundamentally can't avoid, since deciding when it's safe relies on a laggy cross-task version token. Remove it. On conflict, adopt the durable version as the new If-Match and retry: captureState re-reads the current stream (which holds the out-of-band change AND the live edits), so the re-projection persists the converged result. The durable change reaches the live doc via the chokepoint, never here. Trade-off: the only unmerged out-of-band write is one whose chokepoint merge itself failed (rare, logged), which we accept over the frequent reconcile-wipes-edits race. - flushPersist: conflict -> ifMatch = result.version, retry (bounded). No applyMarkdownToLiveFileDoc. - conflict response drops `markdown` (contract + relay type + persist) — no body needed, saves a blob fetch. applyMarkdownToLiveFileDoc stays (still used by the apply-edit route / the chokepoint). * fix(collab-doc): don't let a last-leave conflict retry clobber via the stale local snapshot Regression from dropping the reconcile: on conflict the retry adopts result.version and re-reads captureState. But after single-pod last-leave teardown the room is already destroyed, so captureState falls back to the pre-teardown localState (which lacks the out-of-band change); the retry then CAS-passes and overwrites the committed external write — undoing the external-wins last-leave policy. Null localState on the first conflict, so the retry can only use freshly-read authoritative state (stream / live doc). When none is available (single-pod room gone, or a transient stream-read failure) captureState returns null and the retry stops, leaving durable content authoritative. Covers both the single-pod and multi-task-stream-unavailable variants of the stale-snapshot clobber. * fix(collab-doc): stop (don't re-persist) on a persist conflict — closes the commit-window clobber The conflict retry adopted the durable version and immediately re-persisted the current stream, assuming the stream already held the out-of-band change. But an external write commits durable BEFORE its chokepoint merge (mergeEditIntoLiveFileDoc) reaches the stream, so a persist landing in that window CAS-passed with a stream that still lacked the external content and clobbered the committed write — not just the rare merge-failed path, but a race on every external write, worst at last-leave flushes. Make persist a single attempt: on conflict, STOP and leave durable authoritative. The chokepoint merges the change into the stream and — only once it is actually there — advances the synced version via its own recordVersion; a later flush (debounced or final) then projects the converged stream with a matching token. The session's edits stay in the stream meanwhile. The conflict handler deliberately does NOT advance the synced version, or the next flush would clobber with a still-behind stream. Removes the retry loop and PERSIST_CONFLICT_RETRIES. * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run (#6094) * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run These three table operations mutate row data but emitted no `rows` change signal, so open editors' grids stayed stale until a manual refresh (enrichment *results* already stream live via `cell` events; these are the bulk paths that don't emit per-cell events): - Async row delete (`runTableDelete`): signal as rows drop out (throttled with the existing progress event) and once more on completion — the `job` progress event only drives the delete meter, not the rows query. Covers the delete-async route and the copilot bulk-delete, since both share the runner. - Cancel runs (`cancel-runs` route): cancelling clears each affected row's exec state; the `dispatch: cancelled` events drop the run overlay but the client then renders authoritative DB state, so refetch. Only when something was actually cancelled. - Run column (`columns/run` route): starting a run bulk-clears the target group's cells to pending; refetch so the cleared cells show. Only when a dispatch was actually created. Guarded so no signal fires on a no-op/failure. Adds a delete-runner test asserting the completion signal. * fix(tables): guarantee the live-rows signal on every mutating path (review) - Delete runner (Greptile P1): a batch could commit and the job then cancel/supersede before the next throttled progress signal or `markJobReady`, bypassing both signals and leaving deleted rows on screen. Track `deletedAny` and fire the grid refetch in a `finally`, so it runs on EVERY exit — completion, cancel/supersede, mid-batch lock, or a rethrown error after a partial delete. - cancel-runs / columns/run routes (Cursor): the `cancelled > 0` / `if (dispatchId)` guards don't always reflect DB row changes — cancel tombstones exec state even when 0 dispatches were active, and a run bulk-clears cells then can return a null dispatchId. Signal unconditionally; a stale-but-harmless refetch beats a missed one. - Tests: assert the delete signal fires on the mid-run-cancel-after-delete path and NOT when nothing was deleted. * fix(tables): mark deletedAny before the page delete so a mid-page lock still refreshes the grid `deletePageByIds` commits in internal batches, so a delete lock landing mid-page can persist earlier batches and THEN throw TableLockedError — the catch returns without a count, so setting `deletedAny` from the return value missed it and the finally skipped the grid refetch. Set `deletedAny = true` before the call (any attempt may commit rows); an attempt that commits nothing only over-refetches (harmless). Adds a test asserting the signal fires when a page throws a mid-page lock. * fix(files): make embedded resource file view collaborative (#6095) * fix(files): make embedded resource file view collaborative The /chat resource panel rendered saved files through FileViewer without the collaborative opt-in, so a file open on the Files page and the same file open in the embedded panel never joined the same file-doc room — no live carets and no live content sync between the two surfaces. Pass collaborative on the EmbeddedFile FileViewer. Collaboration still self-gates on canEdit + non-streaming + workspace doc, so the agent token-stream preview (the dedicated streaming-file path, canEdit=false) is untouched. * fix(files): refcount file-doc room membership per shared socket Two collaborative surfaces in one tab (the Files editor and the embedded chat resource panel) share one Socket.IO connection, so both providers for the same file JOIN the same room over that socket. The server's LEAVE does socket.leave(name) with no membership refcount, so the first provider's destroy() would strand the second still-mounted one — no more live content or presence. Count live providers per file per socket (keyed by the stable Socket object, so it survives reconnects) and emit LEAVE only when the last provider for a file tears down. The single-provider path is unchanged (0->1->0). * feat(tables): propagate shared saved-view changes to collaborators live (#6100) Table views (named filter/sort/layout presets) are table-wide shared state — every reader sees every view — but view create/update/delete had no realtime signal, so a collaborator only saw another user's view changes on their own staleTime/focus refetch. Add a 'views' table event kind + signalTableViewsChanged, emitted from the views service (createTableView/updateTableView/deleteTableView, on real success only), and a client handler that invalidates the views query alone (no rows/definition refetch — a view is presentation state on the loaded table). Mirrors how row/schema/metadata changes already propagate. * test(tables): cover the views realtime signal (emit + on-success-only) (#6101) - events.test.ts: signalTableViewsChanged appends a single 'views' event carrying the tableId (through the real memory buffer). - views/service.test.ts: create/update/delete emit signalTableViewsChanged on real success, and DON'T on a no-op (a PATCH/DELETE targeting a missing view changes nothing, so it must not signal). Mirrors delete-runner's signal-path coverage; drives the DB via the shared dbChainMock. - Add tableViews to the comprehensive @sim/db/schema test mock so the service tests can queue the in-transaction existence row. * chore(ci): reconcile api-validation baselines after the staging merge The staging merge unioned realtime-rooms's own `as unknown as` cast (lib/collab-doc/converter.ts) with staging's zod-recursive-type cast (lib/api/contracts/tables.ts), so the non-test double-cast count is 9 — both casts pre-existed and were individually accepted on their branches. Also tighten rawJsonReads 6->5 to the true current count. Fixes the strict API contract boundary audit on realtime-rooms. * feat(copilot): stream file edits into the live collaborative Y.Doc (keep embedded view collaborative) (#6108) * feat(copilot): stream file edits into the live collaborative Y.Doc Copilot's file edits previously only reached the live doc once, at the final edit_content write, so a collaborative editor watching the file saw nothing until completion (streaming looked broken) and the client-side preview path was suppressed in collab mode. Make copilot a CRDT peer: as it streams append/update/patch content, merge the growing markdown into the file's live Y.Doc via the existing apply-edit path (a minimal updateYFragment diff, concurrent-edit-safe), throttled to ~250ms. version is omitted for these intermediate merges — they advance the live doc for viewers but are not durable checkpoints; the final edit_content write carries the real contentUpdatedAt and reconciles the durable file. Per the relay's persist gating, server-internal merges never schedule a persist, so a copilot-only stream produces zero intermediate file writes. - notify.ts: mergeEditIntoLiveFileDoc version is now optional (streaming omits it). - file-preview-adapter.ts: throttled live-doc merge at the edit_content stream hook. * fix(copilot): order + gate streaming live-doc merges; fast collab first render Harden the streaming merge (adversarial review): - Order + bound: dispatch through a per-file in-flight guard (drop-while-in-flight) so a stale out-of-order snapshot can never land after a newer one and regress the doc, and relay load is capped at one request per file regardless of rate. - No wipe: gate append/patch on the base file content having loaded — a base-less snapshot would diff to a delete-everything wipe of the seeded doc; update streams a full rewrite from scratch and needs no base. - Markdown-only gate: non-markdown files have no collaborative room, so skip the wasted relay round-trip. Fast collab first render (Issue 2): render the already-fetched markdown read-only via generateHTML while the collaborative doc seeds, with the editor mounted-but- hidden in the same layout box for a seamless swap on collabReady. Pure HTML — it never touches the Y.Doc (client seeding duplicates the doc), and generateHTML escapes text (raw-HTML snippets render escaped), so no XSS. * test(copilot): cover streaming file edits into the live collaborative Y.Doc Drives edit_content args_delta stream events through processFilePreviewStreamEvent and asserts the live-doc merge: fires with the growing FULL previewText and no version arg; is throttled (~250ms per file); is skipped for non-markdown files and for a base-less append (the delete-everything wipe guard); and runs at most one-in-flight per file. Verified to fail if any gate/guard is removed. * fix(collab-doc): coordinate live-doc merge ordering in one place; close durable-clobber race The second review found a residual: the durable edit_content write went through a different path than the adapter's in-flight guard, so a late straggler streaming merge could land after it and, via a persist, clobber the durable file's tail. Move the per-file coordination into mergeEditIntoLiveFileDoc (the one place both the streaming and durable paths call): a streaming (versionless) merge is dropped while one is in flight for the file; a durable (versioned) write instead WAITS for the in-flight streaming merge, so the final content is always the last merge applied and can't be regressed by a straggler. Simplifies the adapter (drops its Set + helper). Relocate the one-in-flight test to notify.test.ts (streaming-drops-while-busy + durable-waits-then-applies-last); the adapter test keeps throttle/gates/previewText. * fix(copilot): address review — order merges, exclude update, gate throttle, unhide stream Review round on #6108: - Greptile P1 (durable merges lose ordering): serialize ALL merges per file on one chain in mergeEditIntoLiveFileDoc (each chains after the current tail), so concurrent durable writes can't resume-and-fire out of order. notify now exposes isLiveDocMergeInFlight. - Cursor High (update stream blanks the doc): only append/patch stream — they build on the loaded base; update is a from-scratch rewrite whose partial snapshot would diff the full doc toward a fragment, so it applies atomically at the durable write. - Cursor Medium (throttle advances on a dropped merge): the adapter gates on !isLiveDocMergeInFlight, so the send throttle advances only on an actual dispatch — no lag, no backlog behind a slow relay. - Cursor Medium (placeholder hides a live stream): show the fast-render placeholder only when not streaming, so a stream that starts before the doc seeds shows through the editor. - Soften merge.ts/notify.ts comments per the lifecycle audit: only UNTOUCHED regions are preserved; a region the merge rewrites reconciles toward copilot's content. Tests updated: notify covers chain ordering + isLiveDocMergeInFlight; adapter covers append streaming, throttle, non-markdown/base-less/update skips, and the in-flight skip. * fix(collab-doc): reject stale durable merges at the relay (cross-process ordering) The in-process merge chain only orders merges within one apps/sim process. Two durable writes for the same file on DIFFERENT processes could reach the relay out of dispatch order; the relay recorded the version monotonically but still APPLIED the older markdown, regressing the live doc while the token stayed high (a later persist could then write the stale content back over the durable file). Enforce ordering at the relay — the single cross-process coordination point — using the existing Redis primitives: under the per-file Redis merge lock, read the cluster-wide synced version and SKIP a versioned merge that is not newer (a newer durable write already landed). Make recordVersion await setSyncedVersion so it is durable before the lock releases, so the next holder's staleness check reads a consistent value. Streaming (versionless) merges are unaffected — they carry no durable version and are ordered per-process by the caller. Adds a relay test asserting a stale/idempotent versioned merge returns 'stale' and never computes or publishes a diff. * fix(copilot): match durable path — detect markdown by MIME type + name at the stream gate The streaming gate checked isMarkdownFile with only the filename, while the durable merge uses type + name — so a text/markdown file without a .md extension was skipped mid-stream (it self-corrected at the durable write). Pass editIntent.contentType so streaming detects the same set of markdown files as the durable path. * test(copilot): assert throttle follow-through after an in-flight merge clears * fix(collab-doc): order streaming merges by streamedAt so a late snapshot can't regress a newer durable write * refactor(collab-doc): tidy merge-order docs + relay order object; cover multi-replica streaming stale-check * fix(collab-doc): order streaming merges by causal base version, not wall-clock A streaming snapshot now carries baseVersion (the durable contentUpdatedAt it was built from) instead of a wall-clock streamedAt. The relay drops the snapshot when a newer durable write landed since that base, so a concurrent human save can no longer be clobbered in the live doc and then persisted over the durable file. Skew-immune: both keys are DB-monotonic contentUpdatedAt values. * fix(collab-doc): derive streaming baseVersion as contentUpdatedAt ?? updatedAt Match the version line the seed/persist use so a legacy file with no content version still ships an ordered streaming snapshot instead of an unordered one. * fix(collab-doc): fail-closed on a streaming snapshot with no baseVersion The live-merge gate now requires a numeric baseVersion, not just loaded base content. A rare base with no file record (hence no version) would otherwise ship an unordered snapshot the relay can't stale-check, risking a clobber of a concurrent durable write. Skip the live merge instead; the durable write reconciles. * docs(collab-doc): document the accepted concurrent-independent-streams limitation * chore(ci): reconcile api-validation baseline after the staging merge The merge commit auto-merged the baseline at 1000; bump totalRoutes/zodRoutes to 1003 for staging's three new contract-bound routes (nonZodRoutes still 0). * test(files): update storage-accounting assertion to the mergeEditIntoLiveFileDoc options object * fix(collab-doc): trace the full yjs/tiptap external stack into the file-doc route bundles The seed/merge/persist internal routes run the collab-doc converter (markdown <-> Yjs via headless TipTap) server-side. Those deps are serverExternalPackages, and the standalone tracer only force-included jsdom — it does NOT follow yjs's ESM subpath imports of lib0 (lib0/logging, ...), so Docker/standalone builds shipped node_modules without them and the seed route 500'd (Cannot find module 'lib0/logging'). That left every collaborative document unseeded and permanently read-only on deployed envs. Force yjs, lib0, y-protocols, and @tiptap into the trace for all three routes. * fix(collab-doc): copy the full yjs/lib0 stack into the app image The seed/merge/persist routes run the converter (markdown <-> Yjs) server-side. yjs is a serverExternalPackage and the Next standalone tracer copies lib0 only partially — it drops the ESM subpath file lib0/logging.js that yjs.mjs imports via lib0's exports map, so the seed 500s ('Cannot find module lib0/logging') and every collaborative doc is stuck read-only. Verified in the running dev container: /app/node_modules/lib0 had 37/38 files, logging.js missing. outputFileTracingIncludes can't fix it — its globs resolve against apps/sim, but these deps hoist to the monorepo-root node_modules, so the glob matches nothing (my prior next.config attempt was a no-op; reverted). Instead COPY the complete lib0/yjs/y-protocols from the deps stage in the runner, overwriting the partial trace — the same pattern already used for isolated-vm. * feat(files): stream copilot edits into the collaborative doc smoothly (#6122) * feat(files): stream copilot edits into the collaborative doc smoothly - apply the agent stream client-side into the live Yjs binding as minimal updateYFragment diffs (like main's setContent, but incremental) so it renders smoothly AND broadcasts to every peer via CRDT — a collaborator on /files sees the stream for free - gate the apply on collabReady so diffs never land on an unseeded doc; keep the read-only placeholder visible until the seed swaps in - run streamed ops under a dedicated tx origin so they stay out of the user's undo stack - delete the throttled server-side streaming merge and the baseVersion ordering machinery it needed (relay + notify + session contract); the durable final write still reconciles open editors and seeds late joiners * fix(files): apply agent stream as a true CRDT peer + guard base-less snapshots Review round 1 (Greptile P1s): - apply the stream against a private shadow replica (seeded from the live doc at stream start) and relay only the agent's own delta into the shared doc, so a concurrent peer edit to a region the agent snapshot didn't include is no longer reverted (previously the whole-body reconcile deleted it) - gate append snapshots on "must extend the base": a base-less append fragment (emitted before the base loads) can no longer reconcile the seeded doc to a wipe; patch still legitimately replaces a mid-region - gate the apply on collabReady so diffs never land on an unseeded doc; keep the placeholder visible until the seed swaps in - plumb streamOperation through the preview surfaces to drive the append gate - add a peer-edit-preservation test (fails under whole-body reconcile) and refresh the undo-isolation + broadcast tests for the session API * fix(files): destroy the agent shadow deterministically on settle Cursor round 1 (Low): endAgentStream ran inside runOffRender, whose microtask is dropped when a rapid follow-up stream bumps the run token — leaking the shadow Y.Doc. Split it out into an unguarded microtask queued after the (droppable) final apply, so the shadow is always destroyed. * fix(files): agent stream frames skip the relay's durable persist Cursor round 1 (High): client-applied stream frames broadcast over the sync channel, so the relay stamped a socket origin and ran schedulePersist — durably writing partial agent content mid-stream, attributed to the watching user (the old server-merge applied with no origin and never did). Restore that behavior: - new FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST wire tag; the provider tags AGENT_STREAM_ORIGIN updates with it (normal user edits stay SYNC) - the relay applies it under an AgentSyncOrigin (carries the socket id for broadcast exclusion, but is not a plain string) so originSocketId() is null → no edited/schedulePersist/lastEditorUserId; excludeSocketId() still excludes the sender, and the update still publishes to the stream so peers converge - the copilot's final edit_content write remains the authoritative durable persist - tests: relay applies+fans-out but never persists a SYNC_NO_PERSIST frame (verified it fails if applied as a socket edit); provider tags agent edits * fix(files): open the stream shadow at start + private extend baseline Cursor round 2: - High (settle skips apply without session): the stream shadow is now opened on the first ready frame, BEFORE the extend gate — so an `update` rewrite (whose every frame is gated out until settle) and a stream that finishes before seed still get a session, and settle applies the final body via the reused-or-on-demand shadow instead of leaving the doc stale until the durable reconcile. - Medium (peer edits stall the stream): the extend gate now reads a private `lastStreamedBodyRef` (the agent's own last frame), snapshotted at stream start, not `lastSyncedBodyRef` which `onUpdate` clobbers on peer edits — so a collaborator typing can't make the growing snapshot stop prefixing the shown body and freeze it. - Medium (multi-replica over-persist): pre-existing, documented "safe over-persist" (a peer task tails the frame as REDIS_ORIGIN and marks edited) — refreshed the stale comment to describe the SYNC_NO_PERSIST source; copilot's edit_content write remains the authoritative durable persist. * fix(files): fail-close base-less previews + operation-based stream hold Cursor/Greptile round 3 (High + Medium) — remove the fragile string-prefix "extend gate", which was the root of both findings: - Server: `buildFilePreviewText` now fails closed for an `append` whose base content hasn't loaded (returns undefined, like patch/update), so a base-less fragment never reaches the client. This eliminates the base-less wipe at settle (Greptile P1) at the source; an empty file (existingContent === '') still previews normally. - Client: the collab streaming tick no longer string-prefixes the raw preview against the editor's canonical markdown (the '*' vs '-' / emphasis mismatch that froze every append frame — Cursor). The mid-stream hold is now purely operation-based: `update` waits for settle; append/patch/create apply each frame via the (peer-safe) shadow reconcile. lastStreamedBodyRef is now a plain dedup guard, not a prefix baseline. Keeps the shadow, durable write, and SYNC_NO_PERSIST unchanged. * fix(files): elect a single agent-stream writer across tabs Cursor round 4 (High): with the stream applied client-side, two tabs/windows on the same chat could each derive streamingContent (the reconnect/resume path re-consumes preview events) and each independently insert the stream under a different Yjs clientID, duplicating content until the durable reconcile. Fix — single-writer election via the file-doc awareness (new agent-stream-leader): - a client applying an agent stream announces `agentApplying` on its own awareness - only the leader (min clientID among announcers) applies mid-stream AND at settle; a non-leader renders the leader's ops via Yjs and does not apply (a non-leader applying the final body would re-insert the whole doc as a duplicate) - re-checked each frame, so it converges to one writer the moment awareness propagates; the sub-frame startup race is reconciled by the durable write - single-client (the common case) is unaffected: it is the only announcer, so it always leads * fix(files): gate the settle apply locally, not on a settle-time re-election Cursor round 5 (High): the settle recomputed leadership from live awareness and the leader cleared its announcement immediately, so a straggler peer that settled afterward became the sole announcer, self-elected, and applied finalBody through its base-seeded shadow — re-inserting the whole doc as a duplicate. Fix: gate the settle apply on a LOCAL didApplyStreamRef (set only when this client actually applied a mid-stream frame — i.e. it was the mid-stream leader whose shadow is up to date), not on a settle-time re-election. A client that never applied (non-leader, a held `update`, or a pre-seed stream) skips the final apply and converges via Yjs + the durable write. The mid-stream leader election (isAgentStreamLeader) is unchanged, so exactly one client's didApplyStreamRef is ever true. * fix(files): open the agent-stream shadow lazily on lead (no stale handoff) Greptile round 6 (P1): the leader race — (a) a mid-stream leadership handoff could apply from a stale pre-stream shadow, and (b) two tabs starting the same stream before awareness converges could both lead briefly. - (a) fixed: the shadow is now opened LAZILY in the tick, only when this client actually leads, seeded from the CURRENT doc — so a handoff successor diffs against the prior leader's ops (never a stale base) and a non-leader builds no shadow at all. Announce candidacy via a dedicated ref (decoupled from the shadow); settle still gates the final apply on didApplyStreamRef (leader-only). - (b) the pure startup race is inherent to eventually-consistent election. It is now the only residual: bounded to two tabs starting the SAME stream within the awareness-propagation window, transient (converges in a frame or two), and never persisted (SYNC_NO_PERSIST + the durable edit_content reconcile). Resumes are sequential, so the common multi-tab case elects cleanly. Documented inline; a server-granted lease would close it fully but at a round-trip cost on the common single-tab path, which isn't worth it. * fix(files): idempotent settle apply (update lands client-side; no straggler dup) Cursor round 6 (Medium): a lone client's `update` never applied client-side — held mid-stream, then skipped by the didApplyStreamRef settle gate — so the rewrite depended entirely on the durable merge (stale if delayed/failed). Root cause was over-correcting round 5. Now that the shadow is opened lazily in the tick (current-seeded), the round-5 base-shadow duplication is already gone, so didApplyStreamRef is unnecessary. Replaced it: settle applies the final body via `agentStreamSessionRef.current ?? beginAgentStream(editor)` — the leader reuses its up-to-date shadow (last throttled frame), while a client that never applied (non-leader, held `update`, pre-seed) opens a FRESH current-seeded shadow. Reconciling current->final is idempotent: a straggler that settles after another wrote the final reconciles to a noop. So a lone `update` applies at settle (no wait on the merge), and there's still no settle-time election or base-shadow dup. * fix(files): broadcast agent frames to the whole room (same-socket siblings) Cursor round 7 (Medium): SYNC_NO_PERSIST frames applied under an origin carrying the sender socket id, and excludeSocketId dropped that whole socket from the relay fan-out. A second FileDocProvider on the same socket (chat preview + Files editor) then missed all mid-stream ops and stayed stale until the durable reconcile — a regression from the old no-origin server merge, which reached both. Fix: the agent origin is now a plain AGENT_SYNC_ORIGIN symbol, and agent frames broadcast to the WHOLE room (no socket excluded), matching the old behavior — so a same-socket sibling provider stays live; the emitting provider no-ops on its own echo (the ops are already applied locally). originSocketId still returns null for the symbol, so it keeps skipping edited/schedulePersist. Removed excludeSocketId and the socket-carrying origin object. Updated the relay test to assert the whole-room broadcast (verified it fails if the sender is excluded). * fix(files): tag agent stream frames no-persist across replicas A peer task tailing an agent-streamed preview frame previously applied it as REDIS_ORIGIN, marking the seeded room edited and making a transient startup-race duplicate eligible for that task's last-disconnect flush. Mark agent frames with a stream field so peers apply them as REDIS_AGENT_ORIGIN, excluded from the edited/persist gate. The copilot's durable edit_content write stays the sole authority over file bytes. * fix(files): reseed agent shadow on lead regain + agent-only compaction Two multi-writer edge cases surfaced in review: - rich-markdown-editor: a client that led, lost leadership, then regained it reused its stale shadow (which never saw the interim leader's ops), re-emitting ops for content already present. Tear the shadow down when a client observes it is not the leader, so a regain rebuilds fresh from the current doc. - file-doc-store: compaction always stamped its snapshot REDIS_SNAPSHOT_ORIGIN (marks peers edited). A long agent-only stream crossing the threshold could fold preview content into a persist-eligible snapshot. Track whether a room integrated any real edit and stamp an agent-only snapshot REDIS_AGENT_ORIGIN so it stays no-persist. Both covered by falsification-verified tests. * fix(files): close realEdited data-loss race + elect a settle writer Independent audit surfaced two real gaps: - file-doc-store: realEdited was latched AFTER appendUpdate's awaits, but the edit already sits in room.doc synchronously. A concurrent agent-frame compaction could read realEdited=false, snapshot that real content, and stamp it a no-persist agent frame — a lost edit. Latch it synchronously (same tick as the doc mutation) before any await. Deterministic falsifiable test added. - rich-markdown-editor: at settle every tab applied the final body, and a non-leader's local microtask runs before the leader's final propagates, so both insert the tail (Yjs keeps both) -> duplicated tail. Elect a single settle writer (reliable — awareness is long converged by settle), reading leadership before clearing the announcement. Corrects the overclaiming idempotency comment and the handoff pick-up comment. Adds a y-tiptap internals upgrade-guardrail test. * fix(files): own presence per client id, not one-per-socket The shared workspace socket hosts one collaborative provider per mounted view, so the chat file preview and the standalone Files editor for the same file each bind their own Yjs client id over ONE socket. The relay owned a single client id per socket, so the later JOIN overwrote the earlier and dropped its awareness — which silently broke the single-writer agent-stream election (a peer stopped seeing the streaming provider's announcement and could self-elect, duplicating streamed text for the whole stream). Track ownership per (socket, client id): a socket owns a set of client ids; the awareness gate accepts a frame only if every id it carries is owned; cleanup drops all of a socket's ids; the roster stays one-entry-per-session. Reclaim and the same-user reconnect path evict just the reclaimed id, dropping the old socket only if it empties. Falsification-verified test added. * fix(files): make streamed file-preview accumulation replay-safe Guard deriveFilePreviewSession against re-delivered/replayed content events: apply a delta/snapshot only when previewVersion strictly advances, so a client re-render or stream replay can't double-append the tail (the duplicated-content bug) or regress on an older snapshot. * fix(files): fix new-file collab streaming latch and agent-edit duplication - Latch collab readiness so a new file's post-seed `synced` flap can no longer re-gate agent streaming (the stream previously showed only the seed and the rest appeared only on reload) - Relay defers the durable edit_content merge to an actively-streaming client: the client shadow stream and the server merge were both writing the same content into the live doc, duplicating it when the server ran ahead - Render the collaborator caret bar out of flow so a peer caret never nudges the surrounding text by ~1px - Remove dead code: unused FileDocMessageType alias, unnecessary LiveFileDocMergeOrder export Covered by tests: readiness latch (flap/offline/latch cases), relay merge deferral (single- and multi-replica), plus verified-failing guards. * test(copilot): fix loadWorkspaceFileTextForPreview mock to return { text } not a bare string The adapter reads previewBase.text to seed an append/patch base; the mock returned a bare '' so previewBase.text was undefined, making a base-less append fail closed (no file_preview_content). My PR's fail-close change exposed the wrong-shaped mock. --------- Co-authored-by: mzxchandra <129460234+mzxchandra@users.noreply.github.com>	2026-07-31 18:48:12 -07:00
Bill LeoutsakosandBill Leoutsakos	ecf1d7dbbb	refactor Instagram integration architecture (#6143) ... Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-07-31 18:07:30 -07:00
Waleed	19b0312a04	feat(managed-agent): add session lifecycle operations (#6140) ... * feat(managed-agent): add session lifecycle operations Adds an operation selector to the Claude Managed Agents block, backed by nine new tools alongside the existing run-session behavior: - create session (non-blocking, seeds initial_events) - send message to an existing session - get session (surfaces tool calls awaiting approval) - list events - update session (title/metadata) - interrupt session - respond to tool confirmation (allow/deny) - archive session - delete session Run Session stays the default, so blocks saved before the selector existed keep their exact behavior and field layout. * fix(managed-agent): address review findings on session lifecycle ops - List Events kept the OLDEST slice when capped, dropping the agent's most recent reply. Paging is exhaustive again and the cap now keeps the newest N after ordering, with a `truncated` flag so callers know it is a tail. - listPaginated returned whole pages past maxItems; it now trims to the exact cap. - A whitespace-only title passed the update guard and would have cleared an existing session title. Blank is now treated as not provided. - Interrupt had no request timeout and could hang; bounded at 15s while still honoring the workflow signal. - Custom-tool gates were surfaced by Get Session but could not be answered, since they need user.custom_tool_result rather than a confirmation. Adds a Respond To Custom Tool operation and a `kind` on each pending gate so a workflow routes to the right one. - Docs rendered a raw ${DEFAULT_EVENT_LIMIT} placeholder for the default. * fix(managed-agent): correct truncation flag, gate lookup, custom tool result - `truncated` was true whenever the history size equalled the limit, even though nothing was dropped. Event reads now report the untrimmed total and the flag compares against that. - Pending-gate enrichment capped its read, which keeps the OLDEST events in page order — the opposite of where blocking gates live. It now filters to the ids being looked up as pages arrive, which is both correct regardless of page order and bounded by the id count. Paging continues on the raw page so a fully-filtered page is not mistaken for the end of the list. - Respond To Custom Tool applied one result to every id, so multiple pending tools would all receive the same output. It now answers a single call per invocation. * fix(managed-agent): stop fractional event limits reading unbounded A limit below 1 passed the positivity check and then floored to 0, which made `slice(-0)` hand back the ENTIRE history flagged as complete — the opposite of the requested bound. The limit is now floored before it is validated, so anything that does not resolve to a positive integer falls back to the default. Also hardened the library: a zero or negative cap short-circuits to an empty result instead of falling through to `slice(-0)`, so no future caller can hit the same trap. * fix(managed-agent): stop gate lookup scanning the full tool history The id filter keeps the collected array tiny, so `maxItems` never trips and the walk continued to the end of a session's tool history even after every blocking id had been found. `listPaginated` now takes a `stopWhen` predicate and the gate lookup ends as soon as it has all the ids it came for. Also makes a blocked-but-unnamed session observable: when a session reports `requires_action` with no blocking event ids, `requiresAction` stays true — reporting false would tell a workflow the session is fine while it is parked indefinitely — and the dead end is logged and documented instead. * fix(managed-agent): floor event cap and make metadata clearing explicit - A `maxItems` between 0 and 1 slipped past the zero guard and became `slice(-0)` — the whole history — because slice truncates its index toward zero. The cap is now floored at the library boundary, so no caller can hit it whatever they pass. - Update Session documented full metadata replacement but could not express a clear: an empty map normalizes to "absent". Inferring the clear from emptiness would be worse, since an untouched table is also empty and would wipe metadata on every title-only update. Adds an explicit `clearMetadata` instead, and corrects the parameter's documentation. * test(managed-agent): pin the HTTP shape of every session endpoint Method, URL, and beta header for all 11 calls, plus the SSE accept header, the separate memory-store beta (combining the two is a documented 400), and content-type only on requests that carry a body. These are the details types cannot catch and that break silently when a path is "tidied".	2026-07-31 17:59:20 -07:00
Justin Blumencranz	a5ca810cfe	fix(chat): drop unparsable special-tag payloads instead of dumping raw JSON (#6142) ... * fix(chat): drop unparsable special-tag payloads instead of dumping raw JSON * fix(chat): suppress broken payloads mid-stream; reserve marker rescan for mispaired quotes * improvement(chat): require key-value colon evidence before dropping an unparsable tag body	2026-07-31 17:44:17 -07:00
Siddharth Ganesan	88f054e185	fix(auto-model): fix auto model (#6144)	2026-07-31 17:11:49 -07:00
Theodore Li	4ef7bbad53	fix(db): index large-value workflow_id FKs so workflow deletes don't seq-scan (#6136)	2026-07-31 19:48:52 -04:00
Vikhyath Mondreti	b77b9ee491	improvement(workspace): remove tip + simplify modal header text (#6141)	2026-07-31 16:41:24 -07:00
Theodore LiandClaude Opus 5	c5cc6ce26c	feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137) ... * feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 19:23:24 -04:00
Waleed	0bcf64a505	perf(bundle): stop three routes shipping the block and tool registries (#6138) ... The public /integrations page downloaded, parsed and executed 19.54 MB of JavaScript, 15.52 MB of which was the tool registry — 4,351 tool configs, to render a catalog of names and icons. It arrived as `<script async>` in the initial HTML, so the browser fetched and ran it on load. Measured on a production build, chunks referenced by the page's own HTML: /integrations 19.54 MB -> 3.55 MB across 41 chunks (-82%) No chunk over 1 MB remains on that route. credit-usage also loses its copy. Three separate import edges, all the same shape: a module mixing pure helpers with registry-backed ones, so importing the pure half dragged in all 282 block configs and the tool registry behind them. - `lib/integrations/index.ts` built POPULAR_WORKFLOWS at module scope via `getAllBlockMeta()` and re-exported registry functions. The landing grid is a client component, so importing anything from that barrel inherited the whole graph. POPULAR_WORKFLOWS moves to its own module — its one real consumer is a server page — and the re-export goes, which .claude/rules/sim-imports.md already forbids and which nothing imported through the barrel. - `blocks/icon-color.ts` mixed pure contrast maths (`getTileIconColorClass`, `isLightTileColor`, used by the landing page) with `getBareIconStyle`, which reads `getAllBlocks()`. Split by dependency into `blocks/brand-icon-style.ts`; all six of its consumers are under app/workspace/** where the registry is already legitimately present. - `credit-usage-view.tsx` imported a date formatter from the logs feature's `utils.ts`, which also exports registry-backed badge components. `formatDateShort` has three consumers across three features, so it moves to `lib/core/utils/date-display.ts` per the repo's utils-location rule. Not addressed: account/settings/[section] still reaches the registry through workspace-permissions-provider -> socket-provider, one of the four known client edges from the module-graph audit rather than a stray import. It is also loaded via `dynamic()`, so it is likely lazy rather than on first paint. That belongs with the registry split, not here. workspace/* keeps its copy, which is correct — the editor needs tool params and outputs.	2026-07-31 16:04:07 -07:00
Bill LeoutsakosandBill Leoutsakos	c0e7ea9421	fix(pi): surface provider errors from agent events (#6133) ... Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-07-31 14:26:24 -07:00
Theodore Li	648dd345fa	feat(notifications): email on schedule auto-disable and 100% usage limit (#6038) ... * feat(notifications): email on schedule auto-disable and 100% usage limit * fix(notifications): keep partial recipients when one lookup fails	2026-07-31 16:34:12 -04:00
Vikhyath Mondreti	422b4919f1	chore(trigger): upgrade version (#6127)	2026-07-31 13:20:30 -07:00
Theodore Li	b4f027eece	fix(billing): point self-hosted upgrade CTAs at the hosted app (#6003) ... * fix(billing): point self-hosted upgrade CTAs at the hosted app Self-hosted Chat bills against the sim.ai account behind COPILOT_API_KEY, but its 402 upgrade card linked to local billing settings that a self-hosted deployment does not have. Point those CTAs at the hosted app instead, and drop the local workspace-role gate that could hide the CTA from the only person able to act on it. Adds /upgrade, an account-scoped entry for callers that cannot know a workspace id. It delegates to /workspace?redirect=upgrade rather than re-deriving workspace resolution, inheriting local recency, stale-session recovery, and the no-workspace creation policy. * fix(billing): keep the upgrade intent through workspace creation A first-time visitor to /upgrade has no workspace to resolve, so /workspace creates one — and then hardcoded a redirect to home, silently dropping the upgrade intent. Route both exits through one destination helper so the created workspace lands on the plan picker with its reason intact.	2026-07-31 16:08:02 -04:00
Waleed	e98715da01	fix(realtime): noindex the socket server's 404 responses (#6129) ... * fix(realtime): noindex the socket server's 404 responses The sockets.* hostnames are served by this server and return a plain JSON 404, which Google Search Console reports as crawl errors. Mark unmatched routes noindex so crawlers drop the hostnames instead of retrying them. * fix(realtime): noindex every response, not just the 404 Setting the header only on the 404 fallback covered the one response that crawlers already drop on status code alone, while /health — the sole route returning 200 with a body, and so the only indexable surface on the socket hostnames — stayed uncovered, with a test pinning it that way. Set it once on the handler instead. Node merges setHeader values into writeHead and no branch sets X-Robots-Tag, so it reaches every response.	2026-07-31 12:55:48 -07:00
Bill LeoutsakosandBill Leoutsakos	c0b19da782	feat(pi): install Bun in cloud sandboxes (#6123) ... Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-07-31 12:45:13 -07:00
Waleed	5c29d833af	improvement(ui): drop full-content hover tooltips (#6128) ... * improvement(ui): drop full-content hover tooltips - remove the chunk-content tooltip in the KB chunks table (it forced the full chunk body on every truncated row) - remove the sub-block value tooltip on collapsed workflow blocks (whole prompts/code/JSON on hover) - replace the native `title` on subflow and note blocks with the clip-gated OverflowSpan - clip-gate the KB documents tags cell so it stops firing on fully visible tags - drop the dead tooltip on the resource header root title, which can never truncate * fix(workflow-renderer): default the optional note name for OverflowSpan	2026-07-31 12:43:26 -07:00
Waleed	ed23330f88	feat(knowledge): opt-in hybrid lexical + vector retrieval for KB search (#6124) ... * feat(knowledge): hybrid lexical + vector retrieval for KB search KB search ranked purely on pgvector cosine distance, which retrieves exact tokens (error codes, ticket keys, identifiers, rare product names) poorly. Add a full-text leg over the already-present generated `embedding.content_tsv` column and its GIN index — no migration, no re-indexing — and fuse it with the vector leg by reciprocal rank. Both legs run concurrently and share the same visibility and tag-filter predicates; the lexical leg is best-effort and falls back to vector-only on failure. Hybrid is the default for every caller. `searchMode: 'vector'` on the internal and v1 contracts (and an advanced Retrieval Mode dropdown on the Knowledge block) restores the previous behavior. Both search routes now share one `executeKnowledgeSearch` dispatch instead of duplicating the three-branch retrieval logic. * change(knowledge): make vector the default search mode, hybrid opt-in Every existing caller — workflow block, v1 API, copilot, guardrail RAG — keeps its current ranking. Hybrid retrieval is now requested explicitly via `searchMode: 'hybrid'`. Also routes the copilot knowledge tool through the shared `executeKnowledgeSearch` dispatch so all four callers share one retrieval path, and documents `searchMode` on the public v1 search endpoint in the OpenAPI spec. * docs(knowledge): document the hybrid retrieval mode Regenerates the knowledge integration reference for the new searchMode tool param, and adds a Retrieval Mode section to the knowledge base workflow guide explaining when hybrid beats vector-only. * fix(knowledge): stop rank fusion from starving the lexical leg Rank n in one leg always ties rank n in the other, so ordering the fused list by score alone let whichever leg was scored first take every tied slot. At topK=1 that meant a hybrid search returned exactly the vector-only result and discarded the exact keyword match the mode exists to recover. Selection now orders by score and drains each tie group round-robin, taking from whichever leg has contributed fewest rows so far. The lexical leg is passed first so it wins a total tie, since a chunk the vector leg ranked below its distance threshold is the case hybrid was opted into for. * fix(knowledge): credit a shared hit to every leg that returned it Attributing a row found by both legs to a single leg left the round-robin owing the other leg a slot it had already been served. With a shared rank-1 hit and topK 2, that evicted the lexical-only row — the exact match hybrid was enabled to recover — in favor of the vector-only one. A shared row satisfied every leg that returned it, so every one of them is now charged for it. Tie-breaking prefers the candidate whose least-served leg has been served least, which also removes the arbitrary best-rank attribution. * fix(knowledge): reject a whitespace-only copilot query explicitly The shared dispatch treats a whitespace-only query as absent and throws when no tag filters accompany it, where the previous vector-only call would have embedded the blank string and searched. Tighten the existing guard so the tool returns its normal message instead. * fix(knowledge): fan the keyword leg out per knowledge base The vector leg caps candidates per base once getQueryStrategy sets useParallel, but the keyword leg always ran one global query with a single LIMIT. Searching several bases at once let whichever one ranks strongest lexically consume every slot, so an exact-token hit in a smaller base never reached fusion — the case hybrid exists to serve. The keyword leg now uses the same strategy: per-base queries under the same parallel limit, re-ranked globally on a selected ts_rank_cd. Both legs draw candidates the same way, so fusion combines rankings over the same pool. * perf(knowledge): stop the keyword leg detoasting every match's vector Selecting the cosine distance in the ranking query made Postgres detoast the 1536-dimension embedding and compute a distance for every full-text match before the LIMIT applied, so cost tracked how common the query term was rather than topK. On a 20k-chunk base with a term matching every row that was 61,055 buffer hits against 1,030 for the same query without the projection. Rank on ids and ts_rank_cd alone, then hydrate only the rows that survive the limit. Same results, and the worst case drops to ~27ms end to end.	2026-07-31 12:42:09 -07:00
Waleed	898a10d2d2	improvement(files): match CSV/XLSX preview tables to the markdown table chrome (#6125) ... * improvement(files): match CSV/XLSX preview tables to the markdown table chrome Tables in the file viewer looked different depending on the file: CSV and XLSX previews rendered their own chrome (rounded outer frame, --surface-2 header, 13px body / 12px header, --text-secondary cells) while markdown files rendered tables through the rich markdown editor (full cell borders on --divider, --surface-4 header, 14px text). Extract the markdown table chrome into document-table.css and style both surfaces from it. The editor stylesheet keeps only its own concerns (fixed layout for column resizing, prose block margin, cell paragraph reset); DataTable keeps only its edit affordances. * fix(files): wrap unbreakable cell values in preview tables like markdown does The markdown prose root sets overflow-wrap: anywhere; the preview root did not, so with whitespace-nowrap gone a long URL or hash in a CSV cell would overflow instead of breaking.	2026-07-31 12:27:16 -07:00
Waleed	48aeac218c	fix(uploads): set Content-Type once on presigned PUTs; document x-goog-meta-folderid for GCS CORS (#6121) ... * fix(uploads): set Content-Type once on presigned PUTs; document x-goog-meta-folderid in the GCS CORS example XMLHttpRequest.setRequestHeader appends on repeated calls (values join with a comma), and GCS is the only provider whose signed uploadHeaders include Content-Type — so single-shot GCS uploads sent 'x, x', which fails V4 signature verification with 403 (headers canonicalize to a comma-separated value that must match what was signed; multipart part PUTs are unaffected since part URLs don't sign Content-Type). The client now sets its default Content-Type only when the server's signed headers don't already carry one, with regression tests for both paths. Also adds x-goog-meta-folderid to the documented GCS CORS responseHeader list — workspace uploads now sign a folderId metadata header, and GCS CORS matches preflight request headers against that list exactly (no wildcards), so the missing entry blocked browser uploads into folders. * chore(uploads): drop inline comment	2026-07-30 22:22:56 -07:00
Vikhyath Mondreti	413784eceb	improvement(function): secrets access dropdown (#6118)	2026-07-30 20:40:20 -07:00
Vikhyath MondretiandClaude	04a8c0ac8f	improvement(admin): update defaults for better UX (#6112) ... * improvement(admin): uupdate defaults for better UX * refactor: address review nits on the admin/invitation lock work Correct the attach lock-order comment. The order matches admin move, and what makes it mandatory is invitation acceptance: it holds `workspace-invitations:<id>` while waiting for the workspace row, so row-locking first (as this did) deadlocks against it. The previous ownership-transfer justification did not hold — that path takes the organization lock before its workspace rows too, so the two agree on order rather than inverting. Drop `cancelInvitation`. The `revokeInvitationAsAdmin` extraction left it with no callers, and an unlocked, unauthorized `status = 'cancelled'` flip sitting next to the fenced replacement is easy to reach for by mistake. Drop the unused `executor` parameters from `hasWorkspaceAdminAccess` and `isOrganizationAdminOrOwner`. No caller threads a transaction through either, and the former goes back to delegating to `checkWorkspaceAccess` instead of re-deriving the same permission itself. Import `chunkArray` from `@sim/utils/helpers` everywhere and remove the re-export from `batch-delete.ts`, so the symbol has one source rather than a non-barrel shim plus the package. Restore the bounded attachability check in `addDashboardOrganizationMember`: scope the query to the selected ids instead of listing every attachable workspace and scanning that array per selection. Resolve the credential-creation permission through `getEffectiveWorkspacePermission` rather than a second copy of the org-admin derivation ladder, so the rule cannot drift from the shared resolver. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(invitations): drop dead code left by the revocation extraction `revokeInvitationWorkspaceGrant` lost its only caller when the DELETE route moved to `revokeInvitationAsAdmin`, leaving a locked wrapper nothing invoked. Remove it and fold its documentation into `revokeInvitationWorkspaceGrantTx`, which direct grants and scoped revocation still call. The grant-revocation test now drives the transactional form directly, so the sibling-grant and final-grant-cancels behaviour it covers stays under test. `isSameOrgMember` has had no caller since before this branch — direct grant resolves membership through `getUserOrganization` inside its own transaction — so it and its tests go too. `getWorkspaceMembership` is no longer imported outside its module now that credential creation reads `getCredentialCreationWorkspaceContext`; make it module-private rather than leave it on the public surface. Co-Authored-By: Claude <noreply@anthropic.com> * refactor: remove four uncalled billing and large-value helpers Each was checked by hand across every file type, including barrel re-exports and string references, rather than taken from a static analyzer. `isUserMemberOfOrganization` has no reference anywhere. `reapplyPaidOrgJoinBillingForExistingMember` only ever ran from two lock-ordering tests. The transaction-enlisted form it delegated to is what the subscription webhooks call and what those tests actually assert on, so they now drive it directly. The assertions are unchanged: the wrapper contributed a transaction, an organization lock and a membership existence check, none of which appear in the recorded operations. `replaceLargeValueReferences` and `replaceLargeValueReferencesWithClient` are both thin wrappers over `replaceLargeValueReferenceKeysWithClient`, which execution logging, human-in-the-loop resume and the trace backfill all still call. The single test covering a wrapper now composes the key collection itself and targets that live helper. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 20:08:53 -07:00
Waleed	c553de14e8	perf(tests): mock the tool registry globally, drop the dead deps optimizer (#6117) ... * perf(tests): mock the tool registry globally, drop the dead deps optimizer The test suite spent far more time importing modules than running them: on the full suite, `import` was 1,399s aggregate against 88s of actual tests. Per-file import cost showed exactly where it came from — lib/core, which touches no registry, runs at 0.09s/file, while every area that reaches the tool registry runs 12x-79x that (blocks 7.12s/file, providers 3.65, executor 3.07, tools 2.05, app/api 1.05). The tool registry is 4,351 entries pulling ~5,907 modules, and almost nothing under test needs the real thing. `@/blocks/registry` was already globally mocked for this reason; this does the same for `@/tools/registry`. Full suite, same commit, same machine: baseline Duration 166.21s (transform 141.80s, import 1399.17s) after Duration 91.56s (transform 61.98s, import 617.08s) 45% faster, import -56%, transform -56%. Identical results either way: 1252 files / 16873 tests pass, plus one failure that reproduces on unmodified staging (cloud-review-tools.test.ts cannot find `rg` from its spawned python3 locally; CI installs ripgrep and it passes there). Four test files genuinely assert tool registration or tool params, so they opt out with `vi.unmock('@/tools/registry')` rather than being weakened or deleted — outlook, azure_devops, and the two search-replace suites. No coverage is lost. Also removes `deps.optimizer.web`, which was dead config: it only applies to client environments (jsdom/happy-dom) and 985 of 1,219 files declare `@vitest-environment node`. Measured both ways to be sure — removing it is a no-op (19.33s -> 19.23s), and switching it to the correct `ssr` side with an include list for the heavy provider SDKs was also a no-op (19.19s). The cost is first-party module graph, which the optimizer does not touch, so the honest move is to delete it rather than leave config that reads as if it does something. Adds the missing `getBlockRegistry` accessor to the existing `@/blocks/registry` mock. #6083 renamed that export and the mock was never updated, so any test reaching those three consumers would have hit "getBlockRegistry is not a function". Nothing exercises them today. Not done, deliberately: `isolate: false` is ~20% faster but leaks state between files and broke two doc-servable tests on the first run. * docs(tests): explain why the search-replace registry opt-outs are load-bearing Review read the missing direct import of @/tools/registry as evidence the vi.unmock calls were no-ops. They are not: the dependency is transitive — the search-replace planner resolves tool input params through real subblock configs — and removing both opt-outs fails 8 tests across the two suites. Comment now says that, so the next reader does not delete them.	2026-07-30 19:58:59 -07:00
Vikhyath Mondreti	e443a97ffb	improvement(security): isolated-vm env construction (v8s escape case) (#6116) ... * improvement(security): isolated-vm env construction (v8s escape case) * upgrade main node to v24	2026-07-30 19:30:54 -07:00
3 people	632041ce43	feat(library): What Is an AI Agent? Definition, How It Works, and Examples (#6115) ... * feat(library): What Is an AI Agent? Definition, How It Works, and Examples * chore(library): generate cover for the AI agent post --------- Co-authored-by: Sim Pi Agent <pi@sim.ai> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-30 19:23:54 -07:00
Waleed	e79bf81ebe	fix(tables): refuse scale suffixes and resolve the lone-separator ambiguity (#6111) ... * fix(tables): refuse scale suffixes and resolve the lone-separator ambiguity A verification pass over the previous commit found that my own fix opened a new hole of the class it closed. Widening the currency marker to 1-3 letters made scale suffixes parse: `1.2 M` read as 1.2, so a column of `1.2 M` / `3.4 M` — an ordinary spreadsheet paste — converted cleanly and rewrote every cell a millionfold too small. Before the widening those were rejected and the data was safe. Now refused explicitly, while `kr` and `zł` still parse despite starting with the same letters. Stripping the marker also newly routed formatted zero-decimal amounts into the lone-separator branch, where a single dot was always decimal: `1.235 ¥` read as 1.235 rather than 1235. That is a wrong number where there used to be a refusal, which is the worse failure. A lone separator followed by three digits is now resolved by two signals — a marker means a formatter produced it, and formatters group; a currency carrying three decimals (KWD, TND) reads them as decimals. `coerce` passes the column's code, so the parser can ask. Bare typed input keeps the decimal reading. Also: the unchanged-type throw fell through to a 500 for every type except currency, whose message happened to contain the word; it now maps to 400. And the last three comments describing the removed two-transaction architecture are gone — the previous commit claimed they were and two survived. Documented, and failing closed rather than guessing: markers written flush against the digits (`Rp12,00`) stay rejected, because a letter touching a digit is the only thing distinguishing a currency marker from a part number, and reading `SKU400` as 400 invents a value where refusing merely inconveniences. * fix(tables): fit the whole column-type list in the New column dropdown Adding currency made the list nine items (295px with its separator and padding), past the shared dropdown's 240px cap — so Select and Workflow sat below a scrollbar. Sized on this dropdown rather than on the shared `DropdownMenuContent`, which every other menu in the app relies on. Also gave the sidebar's type combobox headroom, since it gained the same entry and was within 4px of its own cap. * fix(tables): read a lone separator against the column's currency Review of the previous commit found the marker signal alone was too coarse. The evidence: a currency with decimal places ALWAYS formats with both separators — `1.234,56 €`, `$1,234.56`, `1.234,000 KWD`. Only a zero-decimal currency emits a lone one (`1.235 ¥`). So a marker does not imply grouping; the currency's own decimal places decide, and a three-decimal currency's trailing three digits are decimals however the value arrived. That matters most for CSV import, which passed no currency at all: a KWD column importing `0,500` read as five hundred rather than a half. The column's code is now forwarded, since nothing in a CSV carries a marker to fall back on. A bare typed `1.234` still reads as decimals. * fix(tables): let the separator decide a lone-separator amount, not the marker Greptile blocked on `$1.234` reading as 1234, and offered the alternative of stating the input contract explicitly. Looking for a precedent settled it against me: a spreadsheet with a USD-formatted cell stores a typed `1.234` as 1.234 and displays $1.23. Reinterpreting it as one thousand two hundred thirty-four is a thousandfold surprise, and my defense of it was wrong. The rule now keys on the separator, tempered by what the currency can express: - A dot is the decimal point in the notation most people type, so it stays a decimal — except for a currency with no decimal places, where `1.235 ¥` cannot be a fraction of a yen and is the one lone-separator form a formatter actually emits. - A comma is grouping by convention (`1,500` is fifteen hundred), except for a three-decimal currency, where `0,500` is a half. A currency with one or two decimal places always formats with both separators, so no formatter output is decided here at all. This also drops the `hadMarker` signal entirely — once the separator and the currency answer the question, where the string came from stops mattering. * fix(tables): check conversions against the target column, not a rebuilt stub Bugbot flagged that the conversion gate calls `isValueCompatibleWithType`, which hand-builds a column from loose arguments and never sets `currencyCode` — while the coercion a few lines later reads the real one. The reported impact does not occur. Whether an amount parses does not depend on the currency, only which number it yields, so the gate and the coercion always agree on accept/reject; and the value written back is already coerced against the real column, so no cell is stored wrong. I checked this rather than argued it: across nine currency codes and every separator shape in the parser's repertoire, acceptance never diverged. The stub is still worth removing. It is correct only because of an invariant nobody wrote down, and `convertedColumn` — sitting right there, whose own comment already says it exists so the scan reads the same option set and currency — is what the gate should have used all along. So the gate now takes the target column itself. The loose-argument form moves into the test file as local marshalling, leaving production with only the whole-column shape, which cannot silently drop a key a future type adds. No behavior change, so no test changes: a currency-divergence regression test could not fail today, and pinning an invariant this removes the need for would only block a legitimate future change. * fix(tables): vet a currency marker against an allowlist, not a scale denylist Bugbot found that `SCALE_SUFFIX` misses `mio`, `mrd`, `bio`, `tsd`, `mln` and `md`. Each is one to three letters, so each matched the currency-marker pattern and was stripped: `1,2 mio` read as 1.2, a millionfold too small, which is the same failure `1.2 M` had. Adding the six words would leave the next six. The denylist is the defect — it has to enumerate every magnitude abbreviation in every language, and each one it misses silently rescales a value rather than refusing it. So the check is inverted. A bare letter token is stripped only when it names a currency: an ISO 4217 code the runtime can enumerate, or one of the common non-ISO markers people type (`kr`, `zł`, `Kč`). Anything else is left in place, fails the amount-shape check, and is refused. `12 units` and `12 pcs` now fall out of the same rule the magnitude words do, without naming either. A marker carrying a currency symbol skips the check — no magnitude abbreviation contains one — so `R$`, `CHF`, `1 234,56 kr` are untouched. This rests on ISO codes and magnitude words being disjoint, which is true for all 162 codes the runtime knows, and is now pinned by a test so a future collision fails loudly instead of silently. `SCALE_SUFFIX` survives only for a runtime without `Intl.supportedValuesOf`, which cannot enumerate codes and so keeps the old permissive behaviour; rejecting every letter marker there would break `USD 12.50`. It no longer has to be complete, since every modern runtime rejects by non-recognition. Verified the new tests fail when the vetting is reverted. * fix(tables): group a lone dot only when a zero-decimal amount came formatted Greptile found that a bare `1.235` typed into a JPY column stored 1235. The zero-decimal branch read the dot as grouping without asking where the value came from, so typing, pasting, a tool write, or an import all inflated a thousandfold in silence. The rule was inverted relative to its own evidence. Grouping was justified as "the one lone-separator form a formatter emits" — and a formatter always emits its marker too. The marker is exactly what separates formatter output from someone typing, and I had removed it as redundant two commits ago. So a lone dot groups only for a zero-decimal currency AND only when a marker came with it. `1.235 ¥` and `JPY 1.235` still read as 1235; a bare `1.235` now stores 1.235 and displays `¥1` — wrong in a way the writer can see and correct, rather than a silent thousandfold error. This does not walk back the earlier fix. For a currency with one or two decimal places a formatter always emits BOTH separators, so a lone dot there never came from one and stays a decimal whether or not a marker is present — `$1.234` is still 1.234. The marker only carries information in the zero-decimal case, which is the only case that now consults it. The two overlapping lone-separator tests are replaced by three that each state one rule. Verified the new one fails when the marker requirement is reverted.	2026-07-30 18:53:16 -07:00
Theodore Li	ef007507e7	fix(demo): exempt landing routes from COEP so the Cal.com booker loads on soft nav (#6110)	2026-07-30 21:25:39 -04:00
Waleed	9e9dc54d7e	fix(emcn): stop the tab strip scrolling vertically by a pixel (#6113) ... * fix(emcn): stop the tab strip scrolling vertically by a pixel The active tab extends one pixel past the strip so it covers the bottom border and reads as joined to the panel below. That pixel came from `-mb-px` on the tab itself, which meant it overflowed the row containing it — and that row is a scroll container, because `overflow-x: auto` computes a visible `overflow-y` to `auto` as well. The strip was therefore scrollable on the y axis by exactly one pixel, which is enough for a trackpad to nudge the tabs out of view. Moving the overlap onto the row keeps the tabs flush inside it, so the scroll container has nothing to scroll, while the row itself still hangs the pixel over the border. The strip is unchanged visually. Measured in a real renderer with the component's own classes: the row was scrollHeight 30 against clientHeight 29, and is now 30 against 30. A first attempt to reproduce it with hand-written inline styles showed no overflow at all — the real class output was needed to see it. Shared by the browser and terminal panels, so both stop scrolling. * test(emcn): pin the invariant, not the class placement Review pointed out the guard read source strings rather than rendered geometry, so it could pass while the overflow returned. Correct, and worth being precise about the compromise: this package's vitest runs in `node` with no browser mode, and the repo's only Playwright harness drives Electron against static HTML fixtures, so nothing here can lay out a React tree against the compiled CSS. Adding that capability is real infrastructure, not a line in this fix. What the guard can do is pin the invariant instead of two particular class strings: no negative bottom margin anywhere inside the scrolling row. That now catches the regression wherever a descendant reintroduces it, rather than only on the tab button it came from — verified against both shapes. The rendered behaviour stays measured rather than asserted, and the numbers are recorded next to the guard.	2026-07-30 18:23:06 -07:00
Theodore Li	7293b67a54	feat(custom-blocks): log and bill child runs in the publisher's workspace (#6023) ... * feat(custom-blocks): log and bill child runs in the publisher's workspace * fix(custom-blocks): sanitize every boundary failure and classify it for consumers * fix(custom-blocks): surface a cancelled child as cancelled, not a generic failure * fix(custom-blocks): share one large-value id list so nested blocks propagate * fix(custom-blocks): add a durable cancel backstop to the child bridge * fix(tools): carry Sim's own status through the tool-response boundary * fix(custom-blocks): stop forwarding the publisher's personal quota to consumers * fix(custom-blocks): correlate agent-tool runs to the real invoking execution * fix(custom-blocks): plumb the invoking execution id and abort signal to agent tools * fix(agent): forward the execution id through the provider payload * fix(executor): stop adopting an upstream target's HTTP status as our own * fix(custom-blocks): drain child log finalization when the parent is cancelled * fix(custom-blocks): require curated outputs instead of exposing the whole result * fix(custom-blocks): track the whole child run, not just its finalization	2026-07-30 21:20:58 -04:00
Waleed	64bcfead34	fix(desktop): clear the traffic lights on every full-viewport surface, and enumerate them in CI (#6109) ... * fix(desktop): clear the traffic lights on every auth-shell surface Only /login reserved the macOS traffic-light lane, so signup drew its logo underneath the lights — and so did reset-password, sso, verify, the CLI auth handoff, and the invite pages. The pre-paint script marks the lane on every desktop route, so any surface that did not reserve it overlapped. Ownership moves to `AuthShell`, which is the single source of truth for the frame all of these wear, and it now reserves unconditionally. Per-route gating was the wrong shape rather than merely incomplete: `/invite/[id]` is a dynamic segment, so no route list could have covered it. `supportsDesktopTitleBar` therefore drops its pathname argument — the caller mounting the controller is the signal, and only `AuthShell` mounts it. Workspace routes never render it and keep their existing `WorkspaceChrome`-owned listener, so the two never contend for the attribute. Off the desktop shell `--desktop-title-bar-height` is `0px`, so the reservation and the drag strip collapse to nothing and `.desktop-title-bar-page` is exactly the `min-h-screen` these surfaces had before — web is unchanged. With the prop gone the client auth layout was a bare passthrough, so the route layout renders the shell directly and the passthrough is deleted. Measured in the Electron renderer over CDP across /signup, /login, /reset-password and /cli/auth: lane 40px, logo top 56px, zero overflow on each. /invite/[id] redirects to login when signed out and was not measured directly. The surface audit gains a `stripComments` helper that every negative assertion runs through. These files document the shapes they avoid, so a bare `not.toContain` was matching the prose explaining the fix and failing on correct code. * fix(desktop): cover the remaining traffic-light overlaps, and enumerate them in CI Fixing signup by hand would have been the fourth time this bug was found by a person hitting it. The audit now enumerates instead of listing what to inspect: it walks every `.tsx` outside workspace chrome, flags each full-viewport root, and fails unless that root either composes `.desktop-title-bar-page` or appears in an allowlist with a written reason. A brand-new page that fills the viewport fails on arrival — verified by adding one, and by reverting each fix below. Running it found three more surfaces already overlapping: - `/oauth-error` is Better Auth's `onAPIError.errorURL`, which is precisely where desktop OAuth failures land, so the one page a user sees when sign-in breaks drew its content under the lights. - `/f/[token]` public file view, same shell family, same origin, reachable in the window. - The signup and reset-password Suspense fallbacks are viewport-tall *inside* the lane-reserving shell, so the page overflowed by the lane while the split chunk loaded. A placeholder needs no viewport height. Four surfaces are allowlisted with reasons: the two landing shells (the desktop shell boots to /login or a workspace and has no path to marketing routes), the dev-only playground, and the embedded resume interface. Measured over CDP: /oauth-error reserves 40px with zero overflow. * fix(desktop): cover the shells behind the allowlist, and make the guard fail Greptile was right on both counts, and the first one is worse than reported. `LogoShell` was allowlisted as "marketing chrome, not reachable in the desktop shell". That claim was simply false: it is the frame for `not-found`, the interfaces shell, the desktop handoff shell, and the public-file access gates — so the password, email, and SSO gates for `/f/[token]` all still drew under the traffic lights. The allowlist existed to make risk visible and instead hid four surfaces behind one unverified sentence. It now carries two entries, both checked: the landing shell (every consumer lives under `app/(landing)/`) and the playground (calls `notFound()` unless `NEXT_PUBLIC_ENABLE_PLAYGROUND` is set). The lane's two halves also travelled separately, so `/oauth-error` and the public-file view reserved the space without the drag strip — clearing the lights but leaving the window with no title bar on those pages. `DesktopTitleBarLane` now ships both together and the audit enforces the pairing. Both new checks were unfailable when first written, and mutation testing is the only reason that surfaced: - the pairing check matched `DesktopTitleBarLane` anywhere in the file, so the import line satisfied it after the JSX was deleted; - the coverage check matched `LogoShell` anywhere, so a shell's own definition file self-certified as covered. Both now match JSX usage (`/<LogoShell\b/`). Reverting either fix, and deleting either half of the lane, now fails. Measured over CDP: /oauth-error and the LogoShell-based 404 each reserve 40px, carry the drag strip, and overflow by zero, with the logo at 56px. * test(desktop): strip comments at read time so positive assertions can fail Cursor caught the mirror of a trap this file already documents. `stripComments` was applied to negative assertions only, so a positive like `toContain('desktop-title-bar-page')` still ran on raw source — and `AuthShell`'s TSDoc names that class, so deleting it from the markup left the assertion passing on a broken lane reservation. Stripping now happens in `read`, so every audit constant is comment-free and no assertion in either direction can match prose. Verified: deleting the class from the markup while leaving the TSDoc intact now fails two tests, where it previously failed none. * fix(desktop): audit workspace routes too, and cover the three that bypass the chrome Greptile was right that the blanket `workspace/` exclusion was load-bearing in the wrong direction. It assumed every workspace route reaches the lane through `WorkspaceChrome`, and three do not: the workspace landing route, the access-denied early return, and the workflow error boundary. The exclusion is gone, so workspace files are audited like everything else and `WorkspaceChrome` joins the lane-aware shells — the normal route passes through its layout, and the three exceptions had to be dealt with on their merits: - the landing route (status card and spinner) and access-denied now reserve the lane; - the error boundary is allowlisted, verified: it renders `<Sidebar>`, which already owns the workspace lane and its drag region, so padding that root would double it. Their content was centred, so the lights were never covering text — the real gap was that none of them rendered a drag strip, leaving the window immovable on those screens. The guard's granularity is per file, not per JSX root: `workspace/page.tsx` holds two full-viewport roots and still passes if only one reserves the lane. Verified by mutation and documented rather than papered over — catching it needs an AST pass, and the check's job is to stop a whole surface being forgotten, which is how every instance of this bug has actually shipped. * fix(desktop): teach the audit about nesting, and stop the resume skeleton double-reserving Four findings, all correct, and the first is a bug this PR introduced. The resume loading skeleton reserved the lane while already rendering inside `(interfaces)/layout.tsx` -> `InterfacesShell` -> `LogoShell`, which reserves it too. Two lots of padding, two drag strips, two controllers. It came from adding the lane there before `LogoShell` became lane-aware and never reconciling the two. The skeleton now reserves nothing and is no longer viewport-tall either — nesting a viewport-tall root inside a viewport-tall shell overflowed even before this PR. The public-file header pinned `sticky top-0`, which parks it inside the reserved lane and under the lights. It now sticks below the lane, inert on web where the variable is `0px`. Both audit gaps were real: - The check was file-local, so it could not see the doubling above. It now resolves ancestor layouts: a root counts as covered when it reserves OR sits inside a layout that does, and reserving on both levels is its own failure. That also stops the check demanding a second reservation from chat and the workspace overlays, which correctly inherit theirs. - Detection only matched `min-h-screen`/`h-screen`, so `fixed inset-0` roots never entered it. Now included. With nesting understood, that addition resolved to a single genuinely uncovered file rather than the ten it flagged beforehand. Two allowlist entries added, both reasoned rather than assumed: the landing prefix (dozens of files, one justification), and the desktop update gate — it centres its content, and under `hiddenInset` macOS draws the lights above the web contents, so web UI cannot cover them. This bug class is app chrome sitting under the lights, never the reverse. Verified by mutation: reintroducing the double reservation fails the new check. * test(desktop): do not credit inherited coverage across a layout's early return Ancestor resolution is static, so it credits any file under a layout that mentions a lane-aware shell. That is wrong when the layout returns the surface *instead of* its chrome: `workspace/[workspaceId]/layout.tsx` returns `<WorkspaceAccessDenied />` at the top and only reaches `<WorkspaceChrome>` far below, so at runtime the denied page has no chrome at all. The page does reserve the lane today, but a regression would have read as inherited and passed. `SessionExpired` is deliberately not listed: it renders as a sibling within the chrome tree, so its inherited coverage is real. The distinction is which side of the early return the surface sits on, not which directory it lives in. Verified by reverting the access-denied page exactly as described — it now fails. * test(desktop): count the lane class itself as a viewport claim Cursor caught the audit failing to watch exactly the files this PR converted. Detection keyed on `min-h-screen`/`h-screen`/`fixed inset-0`, but converting a surface to `.desktop-title-bar-page` removes those tokens — the class supplies `min-height: 100vh` itself. So `/oauth-error`, the public-file view and `AuthShell` dropped out of the check entirely, and a nested class-only reservation could ship green. That also means the doubled-reservation check had never actually fired. The mutation I used to "verify" it removed the lane component as well, so the pairing check caught it and the doubled check was never exercised. It now fires on Cursor's exact scenario: a nested class-only reservation, correctly paired, inside a lane-aware shell. Pulling those files back in exposed a second-order bug: a shell's own definition file sits under the layout that renders it, so ancestor resolution called `AuthShell` nested inside itself. Shell definitions are excluded from inheritance. One limit stays, documented rather than papered over: a root is in scope because of how it claims the viewport, so deleting the reservation outright drops the file from the check. That regression is loud, not silent — the surface stops being full height. Closing it properly means treating every route entry point as a window root, which pulls in seven account/organization/selfhost pages needing individual assessment. Worth doing separately; allowlisting them on assumptions is the mistake that produced the `LogoShell` hole. * fix(desktop): a fixed root escapes ancestor padding — chat drew under the lights Cursor's sharpest catch, and the audit was actively hiding the bug rather than missing it. `position: fixed` resolves against the viewport, not the parent, so a lane-aware shell's `padding-top` never moves it. The chat surfaces sit inside `LogoShell` and still painted at viewport top, under the traffic lights, while the check reported them covered — and adding the correct reservation would then have tripped the nested-reservation check, so the audit pushed toward the wrong answer. Roots matching `fixed inset-0` no longer inherit coverage and are exempt from the doubled check. That reclassified seven surfaces, each decided on evidence: - chat, its loading boundary, the loading state and the voice interface are full-window roots at `z-[100]` with their own top chrome — all now reserve; - the file viewer wraps a full-bleed `<iframe>` whose content starts at viewport top — now reserves; - session-expired centres its content with nothing in the lane, and the search modal's `fixed inset-0` is only its scrim (the panel sits at `top-[15%]`) — both allowlisted with that reasoning rather than an assumption. Verified by reverting chat to a bare `fixed inset-0` inside `LogoShell`: it is flagged now and was called covered before. * fix(desktop): stop the lane controller clobbering a mode another owner set A regression I introduced last round. `DesktopTitleBarController` seeded `inset` unconditionally on mount, before its own `getState()` resolved. That was harmless while only `AuthShell` mounted it — but giving the file viewer a lane put a controller inside workspace for the first time, exactly where this PR's own comments say `WorkspaceChrome` owns the mode. Opening a file during native fullscreen therefore snapped the traffic-light lane back on and jumped the content, and left it wrong permanently if `getState()` rejected, since the rejection is swallowed. It now seeds only when no owner has established a mode. The pre-paint script sets the marker before first paint and `WorkspaceChrome` maintains it, so the unconditional write was never the thing making the lane correct — it was only ever able to make it wrong. Adds a controller test covering the case directly: mount during `fullscreen` with a `getState` that never settles, and the mode survives. Verified it fails against the previous behaviour. * test(desktop): assert the lane on LogoShell directly, not via the sweep Cursor found the one shell the enumeration could go silent on. `LogoShell` sits under the `app/(landing)/` prefix allowlist, and this PR replaced its `min-h-screen` with `desktop-title-bar-page` — so stripping the reservation also strips its last viewport token and the sweep stops watching it. Doubly exempt, and it is the shell behind not-found, the interfaces shell (chat, resume), the desktop handoff and the public-file gates. `AuthShell` already had a dedicated assertion; `LogoShell` now has the same. Verified by stripping its reservation: caught now, silent before. This is the documented enumeration limit made concrete rather than a new class of problem — a root leaves the sweep when it stops claiming the viewport. The general fix is still to treat route entry points as window roots, which pulls in seven unassessed pages and belongs in its own change. * docs(desktop): record why access-denied reserving is not a double reservation Review read `WorkspaceHostProvider` as rendering the denied page inside the chrome. It is the other way round: the provider wraps `<WorkspaceChrome>` in the layout, and its 403 branch returns the denied page instead of its children, so the chrome never mounts. The server-side early return does the same thing higher up. Both paths therefore bypass the chrome and the page's own reservation is the only one. Recording that next to the entry, since the file lives under a workspace path where inherited coverage is the norm and the reading is an easy one to repeat.	2026-07-30 17:55:16 -07:00
Waleed	ee0157df4b	feat(tables): add currency column type on a new column-type registry (#6106) ... * feat(tables): add currency column type on a new column-type registry Adds a `currency` column type, and consolidates the per-type knowledge it would otherwise have been scattered across. **Currency.** Stores a plain number and carries an ISO 4217 `currencyCode` as display metadata. That split is what keeps it cheap: filtering, sorting, uniqueness and CSV export all reuse the numeric paths unchanged, changing a column's currency rewrites no rows, and the public row output stays a number rather than a locale-formatted string consumers would have to reparse. Input accepts the shapes an amount actually arrives in — `$1,234.56`, `1 234,56 €`, `(12.00)` — so pastes, CSV imports and tool writes land as numbers instead of being nulled. **The registry.** Adding this type initially required edits in ~40 places: 32 switch arms under `lib/table`, ~26 UI branches, two hand-maintained icon maps, and a coercion implementation duplicated four times. Every one of those failed silently when missed — a missing `jsonbCastForType` arm compares numbers as text; a missing compatibility arm blocks all conversions. `lib/table/column-types/` now holds one file per type carrying its label, icon, badge colour, storage cast, filter operators, coercion, validation, compatibility and formatting. `Record<ColumnType, …>` on both registries is the completeness gate: adding a type to the union is a compile error naming exactly the two files to fill in, and the interface then requires every field. The 32 switch arms are down to 3. Two duplicates collapse as a consequence: - The client no longer mirrors the server's select id-resolution. Those helpers lived in `validation.ts`, which imports drizzle, so anything reaching them became server-only and the grid hand-rolled its own copy. Extracting them to `select-options.ts` lets both sides share one implementation, so the optimistic cache can no longer disagree with what gets persisted. - The two icon maps become one registry read. It also fixes a live inconsistency it surfaced: currency got a numeric keypad in the grid's inline editor but a plain text field in the row modal. Behaviour-neutral by construction: all 1046 tests in the touched areas pass unchanged, with no test edits. * test(tables): guard the column-type registry's invariants Property tests for the registry itself rather than any one type: entries key by their own id, COLUMN_TYPES stays derived, an unknown type degrades to string instead of throwing, only opaque-id types restrict filter operators, only configuration-free types are CSV-inferable, and every type that can reject a draft has a message to show. Plus the metadata-ownership matrix, which pins the generic ownership check to the same answers the hardcoded per-type rules gave. These target the registry's silent-failure class — a wrong jsonbCast or a stray operator whitelist used to be invisible until a filter failed in SQL. Both are verified to fail under mutation. * fix(tables): read exponent-form amounts and reject bad currency PATCHes up front Two P1s from review. Scientific notation lost magnitude. `String()` emits exponent form past 1e21, so a stored amount round-trips through the editor as `1e+21` — and the sanitizer treated the `e` as decoration to strip, reading it back as 121. An untouched cell silently lost 19 orders of magnitude on its next edit. Exponent form is now taken at face value, but only when the string is wholly a numeric literal once symbols are removed, so `12 EUR` (whose `E` survives the strip) still parses through the separator path. A failed currency PATCH left a partial rename. `renameColumn` commits in its own transaction before the currency write, so a `currencyCode` the service would reject — an unsupported code, or any code on a non-currency column — errored only after the rename had stuck. Both are now caught before the first write, matching the guard the route already applies to unique-on-select for exactly this reason. * refactor(tables): finish the registry migration and drop the dead config Audit pass over every consumer, closing the gaps the first cut left. Functional gap: the copilot agent had no currency support at all — it could create a currency column with no code and could never re-denominate one. `add_column` and `update_column` now accept `currencyCode`, with the same up-front validation and the same code-only routing as the HTTP routes. Config that consumers were still restating, now read from the registry: - `supportsUnique` replaces the unique-on-select guard stated in three places (service, both column routes, the copilot tool). - `editor === 'toggle'` replaces seven `type === 'boolean'` checks in the grid and expanded popover, all of which meant the same thing. - `defaultMetadata` replaces the per-type stamping in `addTableColumn` and `updateColumnType`. - `sampleValue` replaces the per-type example values in the LLM prompt scaffolding. - `storesOpaqueIds` replaces the select filter in the find-row matcher. Dead config removed: `getTypeBadgeVariant` had zero callers (already dead on staging), and it was the only reader of `badgeVariant` — so the field, its union, and all seven values went with it. `inferFromCsv` was read by nothing but a comment; CSV inference is an ordered heuristic a boolean cannot express, so it is gone too and `InferredCsvColumnType` is no longer exported. Fixes a latent crash found on the way: unique-constraint checking normalized a cell keyed on its RUNTIME type but reconstructed it keyed on the column's DECLARED type, so a unique `date` column stored a bare `2024-01-01` and then threw `SyntaxError` parsing it back. Both directions now go through JSON unconditionally. Pre-existing, unrelated to currency. Adds the `/add-column-type` skill and a Tables section in CLAUDE.md/AGENTS.md pointing at it, so the next type is one file plus two registry entries. * fix(tables): run the column PATCH guards ahead of the rename, not after it Greptile was right and my previous reply was wrong. The guards were added in the right shape but the wrong place — below `renameColumn`, which is the first write and commits in its own transaction. A PATCH combining a rename with an invalid currency therefore still committed the rename and then returned 400, exactly the counterexample reported. Moved the column lookup and all three pre-flight guards above every write. This also closes the same latent hole for the pre-existing unique-on-select guard, which sat in the same position. Adds route tests that assert `renameColumn` was never called on each rejection path, and that a valid combined rename + currency change still targets the new name. Verified to fail against the previous ordering. * fix(tables): make the retype gate and the write path share one parser A simplify pass over the registry found two real defects and several places the abstraction was being worked around. Silent data loss on conversion. `isCompatibleWith` was hand-written per type and had already drifted from `coerce`, despite the interface promising they could not: `boolean` accepted '1'/'0'/0/1 in the gate but only 'true'/'false' in the write path, so converting a column holding "1" reported zero incompatible rows and then nulled every one of them. `date` drifted the other way. `isCompatibleWith` is now optional and defaults to `coerce(...).ok`, so the two are the same code; only `select` overrides, because its rules are about the column (cleared-vs-required, cardinality) not the value. `isColumnType` used `in`, which matches inherited keys — `isColumnType('toString')` was true and `columnTypeById('toString')` returned `Function.prototype.toString`, which the validator would then call `.validateDefinition()` on. Now `Object.hasOwn`. `defaultMetadata` only ran on the currency arm of a retype, so a future type would get its defaults on create but silently not on conversion. It now runs for every non-select target, carrying forward only metadata the TARGET type declares it owns — a currency→text conversion no longer strands a currencyCode. The index doc claimed the registry is kept out of the `@/lib/table` barrel so 44 server modules don't pull `@sim/emcn/icons`. That was false: `constants.ts` re-exported `COLUMN_TYPES` from the icon-carrying `registry.ts`, and the barrel re-exports `constants`. `COLUMN_TYPES` now lives in the icon-free `types.ts`; verified with an import tracer that both are icon-free again. Also: 5 no-op `validateDefinition`s and 4 duplicated formatters collapsed into registry defaults; `CURRENCY_OPTIONS` was an eager module-load IIFE costing ~8ms of ICU work on every table API route for a list only the config sidebar reads, now built on first call; and the skill's validation grep claimed 'should return nothing' when it returns 8 legitimate hits — it now explains how to tell a leak from a genuine special case. * fix(tables): reject a non-leading sign so dates don't parse as amounts Found by Cursor Bugbot. `parseCurrencyInput` dropped every `-` as decoration, so an ISO date's hyphens vanished and its digit groups joined: `2024-01-01` read as 20240101. With the gate now sharing the write path's parser, a date → currency conversion reported zero incompatible rows and silently turned every cell into a huge number. A sign is only meaningful at the front; an interior one means the string is not a single amount. Leading signs, accounting parentheses, symbols, ISO codes, grouping separators, and exponent form all still parse — covered by the existing cases plus new ones, verified to fail without the fix. * fix(tables): use getErrorMessage in the columns route test mock `check:utils` bans the inline `e instanceof Error ? e.message : fallback` form; the mock for `rootErrorMessage` used it. * fix(tables): rename the column last so a failed write leaves it untouched Greptile's remaining concern: the pre-flight guards read a schema snapshot, so a column-type change landing concurrently can still make a later write fail — and with the rename running first, that failure returned an error with the rename already committed. Guards cannot close that window; each write is its own locked transaction and only the write itself sees the authoritative state. Ordering can. The rename is the one write that is purely cosmetic, so it now runs last: a failed typed write leaves the column entirely untouched, and a failed rename leaves the typed change applied under the old name — the recoverable half. The typed writes target the column's current name, since no rename has happened yet. Tests cover both directions: a typed write rejected mid-flight must not rename, and a successful one must rename strictly after. Verified to fail under the previous ordering. * fix(tables): write back coerced values on every conversion Round 4 findings, all real. A conversion is allowed exactly when the target type's `coerce` accepts the value — and `coerce` frequently TRANSFORMS it. Only `select` and `currency` wrote the transformed value back, so a conversion to any other transforming type left the cell holding its old bytes under the new type. Converting a number column to `date` accepted epoch values, stored them unchanged, and then `(data->>'col')::timestamptz` failed on EVERY query against that column. I opened this myself by defaulting `isCompatibleWith` to `coerce(...).ok`. Fixed at the class rather than the instance: the compatibility scan now records whatever `coerce` produced whenever it differs from what is stored, and one generic write-back applies it. That subsumes the currency-specific migration entirely, so it and its helpers are gone. `select` keeps its own id↔name migrations, which are not coerce-expressible in the outbound direction. The post-conversion column definition is built once, before the scan, so the coercion reads the same metadata the stored value is later validated against. Exponent parsing was ambiguous when followed by text: `1e5 EUR` read as 15. An `e` with a digit on both sides is an exponent marker, so if the string is not a clean numeric literal it is refused rather than guessed — the digit on both sides is what keeps the `E` inside `12 EUR` parsing normally. A failed rename could still leave a typed change committed. The one rename failure a caller can cause — a name already taken — is now rejected up front, leaving only the concurrent-collision race, which no pre-flight check can close without spanning all writes in one transaction. * fix(tables): stop a blank cell blocking an optional type conversion Found by Cursor Bugbot. `''` is incompatible with every numeric type, and the compatibility scan counted it as a hard blocker regardless of whether the target was optional — so a text column with a single empty cell could not be converted to a number at all, and the error said 'to a required ...' either way. An unreadable-but-empty cell is not a conversion failure. The write path already turns an unreadable value into null on an optional column, so the conversion now does the same and records null for it. A required target still reports it, which the existing guard above already does with the message that actually fits. Also pins the two intentional divergences from the pre-registry behavior. A differential run of the registry against the pre-refactor implementations (55 values x 7 column shapes) found ZERO coercion differences and exactly two compatibility differences, both deliberate: boolean now rejects the '1'/'0' conversions the old gate accepted and then nulled, and date now accepts the epoch numbers its write path always accepted. Tests pin both so neither can be silently reverted or widened. * fix(tables): refuse conversions that would invent or destroy values Final adversarial scan found two data-corrupting conversions, both opened by defaulting the retype gate to the write path's parser. number → date destroyed every value. `date.coerce` reads a number as epoch milliseconds, which is right for one deliberate write and catastrophic applied to a whole column: 1, 5, 42 became three timestamps in January 1970, and a Unix-seconds column landed in 1970 rather than the year it meant. Irreversible. `date` now overrides the gate to reject numbers, restoring the pre-refactor behavior, and the contract states the rule the override obeys: a gate may be STRICTER than `coerce`, never looser. Stricter refuses a bulk conversion while single writes still work; looser is the direction that corrupts. string → currency invented values. The parser stripped every non-digit and joined what was left, so `01/02/2024` read as 1022024, `Room 101` as 101, and `0.1.2` as 12 — a column of SKUs or phone numbers converted with zero reported incompatibilities. What remains after removing symbols, spacing and an ISO code must now be only digits and separators, and grouping must be well-formed (a first group of 1-3 digits, the rest exactly 3). Every legitimate form still parses, including all the locale variants. Also generifies the last three metadata leaks: `buildConvertedColumn` strips and carries back by iterating the key list rather than naming keys (naming them meant a future type's metadata rode onto a target that rejects it, failing that column's validation on every later write), `normalizeColumn` forwards metadata through a shared `typeMetadataOf`, and `filterOperatorsFor` moved onto the definition — it was a per-type branch inside the registry's own accessor, the one thing the registry exists to forbid. Skill corrected: it claimed COLUMN_TYPES derives from the registry (backwards), promised exactly two compile errors (four once a type owns metadata), used a grep that missed half the real branches, and never mentioned `import.ts`'s second coercion path, whose silent default arm is the costliest miss available. Differential re-run vs the pre-refactor implementations: 0 coercion differences, 1 intentional compatibility difference (boolean no longer accepts the 0/1 conversions the old gate accepted and then nulled). * fix(tables): let the row modal accept formatted amounts again Found by Cursor Bugbot. I unified the row modal's input type with the grid's `inputMode` last round, but in the wrong direction: mapping `inputMode: 'decimal'` to `<input type="number">` made the modal reject $1,234.56, 1.234,56 and (12.00) — the exact formats `parseCurrencyInput` exists to accept, and which the grid's inline editor takes fine. A native number input and a numeric keypad are different things. Types whose parser accepts formatted text now say so, and get a text field with `inputMode='decimal'` — the shape the grid already uses. A plain number keeps the native input, its spinner, and its validation. * fix(tables): fold a rename into the write it accompanies Closes the last partial-update window, properly rather than by pre-checking around it. A rename is metadata-only — `renameColumn`'s own comment says so: rows, metadata, and workflow-group refs all key on the stable column id, so it is a pure schema write. Nothing forced it to be its own transaction. Running it separately is what created the window: whichever half committed first survived a failure in the other, and no pre-flight guard can close a concurrent collision because only the write itself sees authoritative state. The four column writes now accept an optional `newName` and apply it through one shared `applyPendingRename`, which validates the name shape and checks the collision against the very schema snapshot that write is landing in. A combined request rides the rename on whichever write runs last, so both halves commit together or neither does — a concurrent claim on the name now aborts the whole transaction instead of leaving the other change applied. The routes also address every write by the column's stable id rather than its name, so folding a rename into one write cannot break the next one's lookup. A rename with nothing to ride on still runs standalone. What remains partial is a type write followed by a failing constraints write — two independently locked transactions, pre-existing, and untouched by this PR. * fix(tables): migrate scalar cells when converting a column to select Found by Cursor Bugbot. `resolveSelectOptionId` stringifies a number or boolean before matching, so a `number` column whose values equal option NAMES passes the compatibility gate — but `migrateCellsToSelectIds` only rewrote JSONB `string` and `array` cells. Those cells stayed raw numbers inside a select column, where they render as nothing and fail option membership on the next write. `data->>key` yields the text form for every scalar, so the existing lookup already worked; the predicate was simply too narrow. Widened to cover `number` and `boolean`. The outbound migration is unchanged — cells leaving a select column are option ids, always strings or arrays. Pre-existing on staging (both the resolver's scalar handling and the migration SQL predate this branch), but it lives in a file this PR creates. Tests pin the resolver behavior the predicate depends on, so narrowing either one without the other now fails. * fix(tables): validate a retype's unique against the values it writes Validated the last partial-update seam with a focused investigation rather than assuming. The answer was split. `required` is already safe: `updateColumnType` runs the same `countEmptyCells` against the constraint the request is about to set, which is why that check exists. `unique` was not, and the reachable case commits the unrecoverable half. A text column holding "5" and "5.0", PATCHed with {type: number, unique: true}: the conversion succeeds and coerces both to 5, then the separate constraint write finds duplicates and 400s — with the column already numeric and "5.0" irreversibly rewritten. A pre-scan of the raw text finds nothing; the conversion is what manufactures the duplicate. The retype now carries `unique` and checks it after the write-back, against the values it just wrote. Constraint changes on a workflow-output column were the same shape — rejected by the constraint write, after a type change had committed. Now rejected in the route's pre-flight block, before any write. The duplicate scan is extracted and shared between both paths for the same reason `countEmptyCells` is: two copies of one rule is the drift that produced the original required-check bug. Deliberately NOT merging `updateColumnType` and `updateColumnConstraints`. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan, so merging would either force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. With both reachable failures pre-validated, what remains at the seam is concurrent races no in-process check can close. * fix(tables): don't drop a rename when the write it rides on no-ops Found by Cursor Bugbot — a bug I introduced folding the rename in. `updateColumnCurrency` returns early when the code is unchanged, and that return sat ahead of the rename, so PATCH {name, currencyCode} with the column's current code answered 200 with the rename silently discarded. Both early returns now treat a pending rename as work: the currency path only no-ops when the code is unchanged AND no rename is riding along, and the retype path applies a rename-only write when the type is unchanged. `applyPendingRename` signals "nothing to do" by returning the same reference, which is what lets both detect it cleanly. Also extracts `persistColumns` — five sites were repeating the same schema-write-and-return. * fix(tables): make a combined column PATCH a single transaction Finishes the fold-in rather than pre-validating around the seam. A retype now APPLIES the constraints it already validates against — it checks empty cells for `required` and post-conversion duplicates for `unique`, so it was doing the work without persisting the result — and the route skips the separate constraint write when the type changed. A request combining a rename, a retype and constraint changes is now one locked transaction: no half of it can commit while another fails. The separate constraint write remains for requests that do not change type, which is the only case that still needs it. Deliberately still NOT merging the two service functions. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan into memory, so a merged function would force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. Folding the payload in gets atomicity without either cost. * fix(tables): reject a flattened list as an amount; fold constraints into every typed write Two findings from round 11. Multi-select converted to nonsense amounts. `selectValueForConversion` flattens a multi cell to its comma-joined option names, and the parser read that as a formatted number: options 12 and 34 became 12.34, and 100 and 200 became 100200. No real amount puts whitespace after a separator, but a delimited list does — so a separator followed by whitespace is now refused. Every legitimate form still parses, including space-grouped locales. Combined options-or-currency + constraints could still commit partially. Those two writes now carry constraints the same way the retype does, through one shared `applyConstraints` that validates (workflow-output, empty cells for required, supportsUnique and duplicates for unique) and applies them. The separate constraint write now runs only when no typed write does. Three copies of those rules is the drift that produced the original required-check bug, so they live in one place. * fix(tables): validate constraints after the migrations that rewrite cells Self-caught while reviewing my own previous commit, which introduced both. `updateColumnOptions` ran the shared `applyConstraints` BEFORE its cell migrations. Those migrations rewrite stored values — a single<->multi toggle changes the shape, removing an option clears cells — so a `unique` scan read the pre-migration values, passed, and the rewrite could then produce the duplicates the scan was meant to prevent. Moved to after the migrations, which is where `updateColumnType` already had it. The same commit also left the options path running `required`'s empty-cell check twice: once in the shared helper and once in its original inline block, whose comment still described a separate constraint write that no longer runs. Removed the duplicate — one query, one rule, which is the whole point of the shared helper. Also routes the options path through `persistColumns` like the others. * fix(tables): stop inventing amounts from identifiers; fix the copilot retype Adversarial pass over the final state, seven real findings. Two destroyed data. The copilot `update_column` still used the two-transaction pattern the HTTP routes were fixed for: `unique` was never forwarded to the typed write, so a retype+unique committed the conversion and then failed the constraint — the same irrecoverable half. It now rides the typed write, and the separate constraint write only runs when no typed write did. And the parser's three-letter strip removed ANY three letters, not an ISO code: `SKU400` parsed as 400, `ABC1234` as 1234. Converting a column of part numbers to currency rewrote every cell with an invented value — while the comment two lines above claimed a SKU was exactly what it prevented. The rule is now that a letter touching a digit means identifier, not amount; a currency marker is always separated by a space or a symbol. That same change fixed a class the review surfaced: the pinned currencies could not parse their own conventional notation. `R$ 1.234,56`, `1 234,56 kr`, `1234,56 zł`, `CHF 1’234.56` and Indian lakh grouping (`₹12,34,567.89`) all work now — these are what Intl emits, so a paste from a spreadsheet was being rejected. `updateColumnConstraints` was a fourth copy of the constraint rules the shared helper exists to unify, and had already drifted: it hardcoded `type === 'select'` where the helper asks the registry, so a future type declaring `supportsUnique: false` would have been ignored on that path. It now uses the helper. `updateColumnType`'s unchanged-type early return silently discarded every field except the rename. Callers gate on the type changing, but from a read taken before the lock — so a concurrent change could land there with real work pending and answer success. It now throws. Also: `UpdateColumnCurrencyData` was missing `required`, which only compiled because the routes pass it through a spread; a missing column returns 404 instead of a 400 reading "of type undefined"; and the comments describing the old two-transaction architecture are gone. Verified NOT a bug: CSV export of a currency column writes the raw number, so export/import round-trips losslessly. * fix(tables): read the negative and RTL forms Intl actually emits An Intl sweep across 24 locales found two forms the parser rejected, both from an ordinary spreadsheet paste. `Intl` emits U+2212 MINUS SIGN rather than the ASCII hyphen for negatives in several locales, so `−12,50 kr` read as null instead of -12.5. And it wraps RTL-locale output in invisible bidi control marks, so `‏1,234.56 ‏₪` carried characters that are not part of the amount. Both are now normalized away. 24 locales x 6 amounts now round-trip, up from 99/100 when the sweep started — and the test generates them from `Intl` rather than listing them by hand, so a parser change cannot quietly regress a locale nobody remembered to write down. Locales that format with their own numeral systems (Arabic-Indic) are still rejected, and now say so in the docstring. That is a safe failure — null rather than a wrong value — and supporting them is a wider decision than this type, since it would also touch `number`, display, and sorting.	2026-07-30 17:21:17 -07:00
Vikhyath MondretiandClaude	7798e83489	feat(function): custom sandboxes (#6071) ... * feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 16:33:35 -07:00
Vikhyath MondretiandClaude	0e4f0e733f	fix(invitations): report every reason a batch of invites failed (#6093) ... * fix(invitations): report every reason a batch of invites failed The invite modal collapsed a multi-failure batch to `N invitations failed.` plus the first reason and discarded the rest. The batch endpoint rejects per email and each reason names its own address, so inviting several people who are all ineligible for the chosen membership — the common multi-failure case, since External requires every invitee to already be on a paid plan — named one of them and hid the others. The failed addresses are reseeded into the field as ordinary chips carrying no error state, so nothing else on screen identified them either. `buildInviteFailureMessage` now lists the distinct reasons, collapses identical ones, and reports the denominator ("2 of 5 invitations could not be sent") the way add-people-modal already did for the same partial -failure shape. Past three reasons the tail is counted rather than listed, so a large batch cannot bury the modal in near-identical sentences. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): name the membership field for the org it acts on The dropdown grants organization standing — Member and Admin join the org and take a seat, External does not — but the label said only "Membership", which reads as membership of the workspace being invited to, the thing the field directly above it already controls. The EE access-control modal has its own "Membership" field for permission-group membership, so the bare word was ambiguous across surfaces too. Sentence case matches the sibling "Workspace access" in the same form. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 14:40:19 -07:00