mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
06506bbd385aa052d14e9b5014fc083045ce34d9
35
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
06506bbd38 |
improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path (#6149)
* improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path - Cache the agent-stream ProseMirror↔Yjs binding metadata per session and reuse it across streamed frames instead of rebuilding it (O(doc)) every frame; matches how y-tiptap's own binding maintains the mapping in place. Safe because the shadow doc only ever sees the agent's own reconciles. - Default createFileResponse to a private, no-cache policy so auth-gated bytes are never stored in a shared cache; genuinely-public serve routes opt into public caching explicitly. - Serve content-addressed (key=) embedded images with an immutable private cache to avoid re-downloading them on every doc re-open; fileId= embeds and public shares keep revalidating (the underlying key can change / a share can be revoked). - Drop the dead conflict.version field from the collab-doc persist result and remove the wasted getWorkspaceFile re-read on the conflict path (the relay treats missing and conflict identically and never read version). - Decorate-sort the files/folders lists (compute each sort key once) and index the move-menu subtree build (O(N) vs O(N^2)); ordering is preserved exactly. * fix(files): serve inline images with no-cache so deletion/authorization is enforced per request Embedded images are authenticated content whose backing file can be deleted or have its access revoked at any time. Both key= and fileId= embeds serve private, no-cache, must-revalidate, so every request re-runs the server-side deletion/authorization check instead of serving a possibly-stale image from cache. |
||
|
Waleedandmzxchandra
|
10bfb5d139 |
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on |
||
|
Waleed
|
e98715da01 |
fix(realtime): noindex the socket server's 404 responses (#6129)
* fix(realtime): noindex the socket server's 404 responses The sockets.* hostnames are served by this server and return a plain JSON 404, which Google Search Console reports as crawl errors. Mark unmatched routes noindex so crawlers drop the hostnames instead of retrying them. * fix(realtime): noindex every response, not just the 404 Setting the header only on the 404 fallback covered the one response that crawlers already drop on status code alone, while /health — the sole route returning 200 with a body, and so the only indexable surface on the socket hostnames — stayed uncovered, with a test pinning it that way. Set it once on the handler instead. Node merges setHeader values into writeHead and no branch sets X-Robots-Tag, so it reaches every response. |
||
|
Waleed
|
02311cafb9 |
perf(db): stop scanning every workspace file on workflow archive, cover the billing period aggregate (#6015)
* perf(files): stop loading every workspace file to archive a workflow's backing files `cleanupWorkflowAliasBacking` runs on every workflow delete. It loaded every `context='workspace'` row for the workspace — active and soft-deleted, all columns — then discarded >99% of them in JS to find the handful of `.changelogs/<workflowId>.md` and `.plans/<workflowId>/**` rows it owns. In production this was the single worst query at 37.83% of total database runtime: p50 6s, p95 9s, max 13.1s, 641 calls/day, 34.1M rows read. It is index-served, so the cost is not a missing index — it is 59,061 rows and ~630MB of cold buffers read to archive a few files. A file's `folderPath` is derived solely from its `folderId`, so matching on folder membership is equivalent to the path comparison it replaces. The load and JS filter become one targeted UPDATE keyed on a handful of folder ids. Folders that are soft-deleted are still included when resolving which files a workflow owns: path resolution ignores `deletedAt`, so a live file parented to an archived folder previously matched and must continue to. Also: - Add `getWorkspaceShares`, replacing an id-list `IN` clause that grew with the file count (59,061 elements on the worst workspace) with one indexed lookup on `workspace_id`. Callers read the map by id, so a superset is equivalent. - Drop `all` from the client-reachable file scope enum. It drops the `deleted_at` predicate and so cannot use the partial index serving the other two. No client requests it; server callers reach that scope directly. - Set `fetch_types: false` on the app and realtime pools. postgres.js otherwise runs a blocking `pg_catalog.pg_type` roundtrip before each new connection's first query — 95,722 of them per day. It builds array parsers only; Drizzle already parses this schema's two `text[]` columns itself. * perf(nav): add route loading boundaries and cover the billing period aggregate Dynamic routes prefetch only down to the nearest loading boundary, and the server stops prefetching at the first one. With no loading.tsx anywhere in the workspace tree and a dynamic layout, Link prefetch was yielding almost nothing and every navigation waited on a full server round trip with no feedback. Add loading.tsx only where the fallback is provably what renders today, so perceived speed improves without changing what users see: - home, chat/[chatId]: reuse HomeFallback, already each page's own Suspense fallback. - integrations, skills: reuse the tab-header chrome, byte-identical to each page's own Suspense fallback. Deliberately not added to workspace root, settings, or w, whose pages are redirect-only or already self-fallbacking — a boundary there would paint a skeleton that does not match the destination. Billing: - Add usage_log_billing_period_cost_idx, trailing the remaining predicate columns and `cost` so the period aggregates resolve index-only. Confirmed against production: the aggregate currently runs as an Index Scan touching 478,559 buffers because `cost` is absent from the existing index. That index is superseded but left in place; dropping it is a separate migration so a planner regression costs nothing to revert. - Collapse the two aggregates behind /api/billing into one scan using SUM(...) FILTER. Besides halving the work on a nav-path query, it removes a latent inconsistency: as separate statements the two sums could observe different snapshots, making the copilot subset exceed the total. Caching was considered and rejected for the usage aggregate. Its callers include usage enforcement, threshold billing, and overage calculation, where a stale-low read permits overspend and a stale-high read double-charges. * test(files): cover cleanupWorkflowAliasBacking and correct a stale share mock cleanupWorkflowAliasBacking had no test coverage, and the rewrite that replaced its load-everything-then-filter body rests on a subtle equivalence: a file's folderPath is derived solely from its folderId, and path resolution ignores deletedAt. The second half is the easy part to get wrong — a live file parented to an archived folder still resolves to a backing path and must still be archived, so folders are filtered by deletedAt only when choosing which folders to archive, never when deciding which files the workflow owns. These tests pin that distinction: they fail if the archived-folder case is dropped from file ownership, and separately assert that archived folders stay out of the folder update and that unrelated workflows are never touched. Also point the workspace files route test at getWorkspaceShares. Its mock still named getSharesForResources, which the route no longer imports. The suite passed regardless because all three cases exercise the upload path, so the GET listing has no coverage — but the stale name would have handed the first GET test an undefined function. * fix(perf): drop the loading boundaries and correct the usage_log index order Adversarial review found real regressions in both. Route loading boundaries — all four removed: The premise in their TSDoc was wrong. Each page's existing `<Suspense>` exists so nuqs can prerender; `useSearchParams` only suspends during SSR, so on a client navigation those fallbacks never painted. Hoisting them to loading.tsx did not "show the same frame" — it made a previously invisible blank frame visible. For chat, Next keys the Suspense boundary by cache key, so a chatId change mounts a fresh suspended boundary and always commits its fallback. Switching chats would have gone from "previous chat stays on screen" to a blank surface for the whole RSC round trip, on the highest-frequency navigation in the product. Partial prefetch only warms the fallback, so no latency was saved to offset it. The integrations and skills fallbacks were correct for their own pages but also became the fallback for four detail routes that render different chrome, inserting a wrong intermediate frame. Fixing that needs per-child boundaries and new skeleton UI that cannot be verified without a browser, for pages whose only server work is `await params`. Not worth it. Every remaining loading.tsx in this app paints real chrome. A blank one was against the grain, and the measurable win was zero. usage_log index: Column order was wrong. The daily-refresh rollup filters entity type, id and period_start but NOT period_end, so putting period_end fourth ended the usable prefix at column three and left user_id and created_at as in-index filters rather than scan boundaries — turning a ~2.2k-entry bitmap scan into a ~266k-entry scan. Verified in production that period_start functionally determines period_end (13,612 groups, zero with more than one end), so the slot bought no selectivity. user_id and created_at now follow the shared prefix; period_end rides as payload. Also corrected the claim that this supersedes usage_log_billing_entity_period_idx. That index deduplicates to 17 MB across 1.42M entries because it has no high-cardinality key column, which is what keeps prefix-only bitmap scans cheap. It is retained deliberately, not pending a drop. Harden cleanupWorkflowAliasBacking: gate the UPDATE on the ownership filter list itself. `and()` and `or()` both drop undefined, so a clause that resolved to nothing would have left a WHERE of workspace + context + not-deleted and archived every file in the workspace. Tests now assert no UPDATE is issued when the workflow owns nothing, plus the filename and context predicates. Note fetch_types also disables array serializers, not just parsers; documented. * docs(db): correct the fetch_types constraint note after differential testing Ran both settings against a real Postgres with the schema's actual text[] columns. Drizzle-typed selects and .returning() are byte-identical either way; only a raw db.execute projecting an array column differs, yielding the wire form. The previous note also claimed a raw JS-array bind fails because the serializers come from the same catalog fetch. It does fail — but under both settings, because Drizzle expands an array into a row constructor before postgres.js ever sees it. That is unrelated to this flag, so the claim is removed rather than left implying a constraint this change introduces. * chore(db): generate the drizzle snapshot for the usage_log index migration 0271 was hand-written, so drizzle-kit's state never learned about the new index — the next `generate` would have re-emitted it as a fresh migration against an already-migrated database. Ran `drizzle-kit generate` to produce the snapshot, then restored the CONCURRENTLY form: drizzle emits a plain CREATE INDEX, which takes an ACCESS EXCLUSIVE lock and would block writes on a 4M-row table for the duration of the build. The generated column order matched the hand-written SQL exactly, which also confirms schema.ts and the migration agree. `generate` is now a no-op, and check:migrations still passes. |
||
|
Vikhyath MondretiandClaude Fable 5
|
f43b52c569 |
fix(realtime): evict revoked collaborators from live workflow rooms (#5917)
* fix(realtime): evict revoked collaborators from live workflow rooms via periodic read-access re-validation * fix * fix(realtime): close join/eviction race and make sweep cleanups independent Re-authorize immediately before socket.join so an in-flight join cannot reverse a sweep eviction, and run the sweep's best-effort cleanups independently so a room-state failure cannot skip the presence broadcast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): single-flight role resolution and retry failed eviction cleanup Coalesce concurrent role resolutions per (user, workflow) so a slow stale read can never overwrite a recorded revocation, and defer failed eviction room-state cleanups into a per-sweep retry queue so collaborators are not left with a stale presence entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): scope room removal to the target workflow and detect swallowed cleanup failures Honor the workflowIdHint as the target room in both room managers so removing a stale room cannot clobber the mapping of a room the socket has since moved to, and confirm eviction cleanup via the returned workflowId plus an unswallowed mapping read so Redis failures actually defer into the retry queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): treat unconfirmed removals as failures and refresh role cache on fresh verify Treat any null removal result as a failed cleanup (the sweep always passes the target room, so null only means failure — including with expired mapping keys), move the same-room rejoin guard to a synchronous check immediately before the removal, and record verifyWorkflowAccess's fresh decision into the role cache so a re-granted user is not blocked by a stale cached revocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): prefer a mid-flight recorded role decision over the in-flight query result If a fresh authoritative read (join-time verify) records a decision while a single-flighted resolution's query is in flight, keep the recorded decision instead of overwriting it with the potentially stale result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): isolate the security scan from the Redis cleanup lane Run the revocation scan (local sockets + DB only) and the best-effort room-state cleanup as independently-guarded lanes so a hanging Redis command can stall only presence cleanup, never revocation enforcement. Evictions now enqueue cleanup instead of awaiting it, and the scan no longer reads presence for a fallback role. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): bound authorization waits in the revocation scan Race each socket's authorization check against a per-socket timeout and cap the whole pass with a budget below the sweep interval, so a hanging DB query skips that socket for the pass (never evicting on uncertainty) instead of wedging the scan lane and starving subsequent ticks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): round-robin the revocation scan so hung checks cannot starve later sockets Resume each scan pass after the last target the previous pass processed, so a fixed prefix of hanging authorization checks can never repeatedly consume the pass budget and leave sockets behind it unexamined. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d24bc7eccb |
feat(agent-stream): thinking and tool streaming (#5671)
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
fac8ec0c51 |
fix(workflow-edges): enforce edge/block validation server-side, not just client-side (#5571)
* fix(workflow-edges): enforce edge/block validation server-side, not just client-side
Dragging a connection that creates a cycle correctly refused to render
client-side, but the cyclic edge was still queued for realtime persistence
and written to the DB, so it reappeared after refresh. Root cause: cycle
detection (and several other edge/block rules) only lived in the client
Zustand store and was never enforced by the realtime persistence layer,
which is the actual source of truth on reload.
- Move wouldCreateCycle, edge scope-boundary, annotation-only-block,
duplicate-edge, and block-name-conflict checks into @sim/workflow-types
so the client store, the collaborative queueing layer, and
apps/realtime's DB write path all share one implementation
- Wire these into apps/realtime/database/operations.ts's edge-add and
block-rename handlers as the authoritative gate
- Client-side behavior is unchanged (same call sites, same error messages,
same rule ordering) — verified via existing + new test coverage
* fix(workflow-edges): address review findings on realtime edge validation
- Select triggerMode when fetching blocks for edge-add validation —
isKnownWorkflowTriggerBlock checked block.triggerMode but the column
was never fetched from the DB, so trigger-mode blocks could still
receive an incoming edge (Cursor Bugbot)
- Make filterUniqueWorkflowEdges incremental, so two duplicate edges
within the same BATCH_ADD_EDGES payload are also deduped instead of
both surviving (Greptile)
* fix(workflow-edges): normalize empty-string handles in duplicate-edge check
filterUniqueWorkflowEdges compared handles with ??, so a `sourceHandle: ''`
edge wasn't recognized as a duplicate of an existing null-handle edge —
even though both get persisted as the same null value at insert time
(edge.sourceHandle || null). Falsy-coalesce in the comparison so '' and
null/undefined are treated as the same "no handle" state everywhere.
(Greptile)
* improvement(workflow-edges): dedup realtime edge-add validation, reuse block-name-conflict helper
/simplify pass on the already-merged-quality PR before sign-off:
- Extract filterEdgesForPersist in apps/realtime/src/database/operations.ts:
the single-edge ADD and batch BATCH_ADD_EDGES handlers hand-inlined the
same six-step validation pipeline (missing block, protected target,
annotation-only, trigger-target, scope boundary, duplicate, cycle) and
each independently re-fetched blocksById/existingEdgesForCycleCheck. Two
copies of one rule in the same file was exactly the drift risk this PR
otherwise closes across client/server. One shared helper now backs both.
Net -63 lines despite the new shared function.
- Fix a real bug this surfaced: droppedCounts was keyed by the free-text,
block-id-bearing scope-boundary message, so it could never aggregate
across edges/runs. Now keyed by a stable 'scope boundary' reason.
- use-collaborative-workflow.ts's collaborativeUpdateBlockName still
hand-rolled the empty/reserved/duplicate block-name-conflict checks this
PR centralized as getWorkflowBlockNameConflict (already adopted by
store.ts). Switched it to the shared helper, which also fixes a latent
check-order mismatch between the two (this pre-check ran reserved before
duplicate; the store's real gate — after this PR's own store.ts change —
runs duplicate before reserved, so they could disagree on which toast a
name that was both reserved and duplicate would surface).
* docs(workflow-edges): document the per-workflow write-serialization invariant
No behavior change. Address Greptile P1 on filterEdgesForPersist ('concurrent
duplicate writes can persist without a per-workflow write guard') by
documenting, at the actual mechanism, why the concern doesn't apply here:
persistWorkflowOperation's leading 'UPDATE workflow SET updatedAt ... WHERE
id = workflowId' already takes a row lock that serializes every operation
(including edge adds) for a given workflowId — a second concurrent call
blocks on that UPDATE until the first transaction commits or rolls back, so
the validate-then-insert sequence in filterEdgesForPersist can never
interleave across two writers on the same workflow.
Verified empirically, not just by reading: ran two concurrent transactions
against a throwaway local Postgres against the exact statement shape used
here (UPDATE the parent row, sleep to simulate the read/validate window,
insert, commit). The second transaction's UPDATE blocked for the full
duration of the first's transaction and only proceeded once the first
committed — confirming the row lock, not any application-level guard,
already provides the serialization Greptile flagged as missing.
Added a comment at the lock site (not a second, redundant advisory lock)
so a future change can't silently break this invariant by making the
UPDATE conditional/skippable as a perceived no-op optimization.
* fix(workflow-edges): validate edges in BATCH_ADD_BLOCKS before persisting
Real gap Cursor's PR summary flagged ('BATCH_ADD_BLOCKS edge inserts... not
fully covered by the new server pipeline'), confirmed by reading the code:
this handler bulk-inserted the edges from a block-paste/duplicate/import
payload directly into workflowEdges with zero validation — no missing-block,
protected-target, annotation-only, trigger-target, scope-boundary,
duplicate, or cycle check. A client sending edges through this operation
instead of BATCH_ADD_EDGES could bypass every rule this PR otherwise
enforces server-side, exactly the class of gap the PR exists to close.
Routes it through the same filterEdgesForPersist used by the other two
edge-add handlers. Runs after the block insert in this same handler, so the
shared helper's blocksById lookup also sees the blocks this same batch just
inserted (a transaction observes its own prior writes).
|
||
|
Waleed
|
b117758eaa |
fix(agent): scope nested tool canonical-mode overrides by instance, not type (#5534)
Two tool entries of the same type inside an Agent block's tool-input array
(e.g. two Table tools) shared a single canonical-mode override keyed by
${toolType}:${canonicalId}, so switching basic/advanced mode on one field
silently switched it on every other instance of the same tool type -
including at execution time, where the wrong basic/advanced value could be
resolved for the second tool.
Rescope the override key to the tool's position in its tool-input array
(${toolIndex}:${canonicalId}) instead of its type, and thread that index
through every consumer: the editor (read + write), execution
(agent-handler/providers), search-index, and fork/promote remapping.
|
||
|
Waleed
|
5db62b8bdc |
feat(observability): extend audit log, PostHog, and storage metering coverage (#5269)
* feat(observability): extend audit log, PostHog, and storage metering coverage Instruments previously-uncaptured resources and actions across the three observability layers (audit log, PostHog product analytics, usage metering): - Exfiltration audit: file downloads (workspace/public-share/v1), table, workflow, and workspace exports. - Credential access audited at the token-issuance boundary (success-only). - Full revenue trail: invoice paid/failed, overage billed, disputes, credit fulfillment, subscription lifecycle, plan/seat changes. - Session/login lifecycle audit via Better Auth hooks (login, blocked sign-in, logout, session revoke, account delete). - v1/admin programmatic surface and copilot tool handlers instrumented. - Dead audit/PostHog constants wired (lock/unlock, table, custom tool, etc.). - Storage metering extended to KB documents and copilot files. - PostHog person identify + workspace/organization group hygiene. Audit package hardened to null the actor FK for system actors (admin-api) and adds an awaitable recordAuditNow for pre-delete hooks. All instrumentation is fire-and-forget and never blocks or breaks the primary operation. * fix(observability): audit file/credential egress only on success Address Cursor Bugbot review: - GET OAuth token: emit CREDENTIAL_ACCESSED/credential_used after refreshTokenIfNeeded succeeds (matches POST), not before. - File export: audit on each actual success exit via a shared helper — including the non-markdown serve redirect (previously unaudited) and after the zip is generated (previously before asset fetch). * fix(audit): record null actor for anonymous public-share downloads Address Greptile P1: setting actorId to the file owner made every anonymous external download read as a self-download, undermining the exfiltration trail. recordAudit now accepts a null actor; the public content/inline routes record actorId=null with the owner in metadata.sharedByUserId and rely on ip/user-agent for the forensic trail. The misleading owner-attributed file_downloaded PostHog event is dropped on these anonymous paths. * fix(observability): audit admin exports after zip; tidy comments - Admin workflow/workspace ZIP exports now audit only after the archive is built (via a local helper), so a zip/build failure no longer logs an export. - Remove redundant 'success-only placement' inline comments and tighten the remaining design-rationale notes (export helper now TSDoc). * fix(billing): complete the chargeback + overage financial trail Address Cursor review: - handleDisputeClosed now records CHARGE_DISPUTE_CLOSED for every closed dispute (won/lost/warning_closed), unblocking only on favorable outcomes. dispute.status in the metadata distinguishes the outcome, so lost chargebacks are no longer missing from the trail. - Threshold overage now emits OVERAGE_BILLED + overage_billed even when credits fully cover the overage (settledVia: 'credits' vs 'stripe'), so credit-settled overages are audited instead of silently returning null. * fix(storage): decrement copilot quota before deleting metadata Address Greptile P1: deleting the metadata row before the decrement meant a decrement failure left the quota permanently inflated with no record to retry from. Decrement first; only remove the metadata row once it succeeds. * fix(observability): atomic copilot storage release; signup-blocked action - Copilot file delete now releases storage via a single transaction (releaseDeletedFileStorage): the soft-delete is the idempotency claim and the decrement shares the transaction, so neither a partial failure (inflated counter) nor a retry (double-decrement) can desync the quota. Resolves the conflicting Greptile/Cursor ordering findings. - Policy-blocked sign-ups now record USER_SIGNUP_BLOCKED instead of USER_SIGNIN_BLOCKED, so account-lifecycle events aren't mislabeled. * fix(storage): meter copilot ingest centrally for path symmetry Address Cursor review: only uploadCopilotFile incremented storage, but copilot files also enter via the generic upload route and presigned uploads — all of which persist metadata through insertFileMetadata. Move the increment into insertFileMetadata (scoped to context='copilot', on genuine insert/restore) so every ingest path is symmetric with the delete-time decrement, and drop the now redundant per-path increment in uploadCopilotFile. Other contexts are metered by their own managers and remain unaffected. * fix(audit): skip WORKFLOW_EXPORTED when admin export is empty Address Cursor review: when every requested workflow fails to load, the admin export still returned 200 but recorded a successful export of zero workflows. Guard auditExport so an empty result records nothing. * fix(storage): settle copilot accounting before deleting the blob Address Cursor review: the blob was removed before releaseDeletedFileStorage, so a release failure left the counter inflated and the metadata active with the blob gone. Now the atomic soft-delete + decrement runs first and the blob is deleted only if it succeeds, so a failure leaves the file fully intact and retryable. * fix(storage): decrement KB document storage atomically with deletion Address Cursor review: hardDeleteDocuments deleted the rows then decremented best-effort, so a decrement failure left billed storage inflated with no row to reconcile. Resolve each owner's subscription up front, then decrement inside the same transaction that deletes the embeddings/documents (decrementStorageUsageInTx, also now shared by releaseDeletedFileStorage), so the counter and the content commit or roll back together. Connector docs remain excluded. * fix(audit): don't treat email verification as a login Address Cursor review: /verify-email could emit USER_LOGIN when verification mints or refreshes a session, mislabeling (or double-counting) a non-sign-in as a login. Restrict isLoginPath to genuine sign-in entrypoints. * fix(audit): null actor FK when the user lookup throws Address Greptile P1: the catch branch left the original actorId, so a system actor like 'admin-api' (or a since-deleted user) would FK-violate the insert and lose the row when the existence lookup errored. Mirror the not-found branch — null the FK with a readable label — so the audit row always persists. * revert(audit): drop session/account-lifecycle auth instrumentation Remove the Better Auth login/logout/session-revoke/account-delete/blocked-signin audit hooks from auth.ts — they touch sensitive auth paths and are noisy. Also removes the now-unused taxonomy (USER_LOGIN/_FAILED, USER_SIGNIN_BLOCKED, USER_SIGNUP_BLOCKED, USER_LOGOUT, SESSION_REVOKED, ACCOUNT_DELETED, ACCOUNT_EMAIL_CHANGED actions; SESSION/USER resource types) and the awaitable recordAuditNow helper that only the pre-delete hook used. auth.ts is back to the staging baseline. * fix(storage): harden copilot+KB delete accounting against read errors and concurrency Final-audit follow-ups: - deleteCopilotFile: a failed metadata *read* (vs a genuine missing row) now blocks the blob delete too, so a transient read error can't leave an active row un-decremented with the blob gone. - hardDeleteDocuments: drive the per-user decrement from the delete's returning() (the rows this tx actually removed), so two concurrent deletes of the same ids can't both decrement. * chore(observability): drop two unused definitions Final-audit cleanup: remove the orphaned knowledge_base_searched PostHog event (never wired; KB search analytics already flow through the OpenTelemetry channel) and the redundant AuditAction.SUBSCRIPTION_UPDATED (subscription/plan changes are audited via ORG_PLAN_CONVERTED). Every remaining new action/event has a real emit site. * fix(knowledge): key hard-delete result off rows actually deleted Address Cursor review: hardDeleteDocuments returned existingIds.length (the requested count) and cleaned storage for the full pre-tx set, even though the decrement was driven by the rows the transaction actually deleted. Under a concurrent delete that claimed some ids first, that overstated the result and re-touched storage for rows this call didn't delete. Drive the storage cleanup, log, and return value from deletedDocs (the returning() rows) so all four are consistent. * fix(storage): gate copilot quota on all ingest paths; tidy inline comments - Copilot uploads via the generic /api/files/upload route and presigned URLs now run checkStorageQuota before writing (matching uploadCopilotFile), so no copilot ingest path can grow usage past the plan limit. The central increment stays in insertFileMetadata. - Trim/remove redundant inline comments across the diff; keep only concise notes on non-obvious decisions (left pre-existing comments untouched). * fix(analytics): omit empty workspace_id on workspace-less file downloads Address Greptile P1: the generic key-based /api/files/download route and the no-workspace markdown-export path emitted file_downloaded with workspace_id:'' creating a phantom '' bucket in PostHog. Make workspace_id optional on the event and omit it when there is no workspace (workspace-scoped routes still pass it). * fix(analytics): omit empty workspace_id/workflow_id on copilot_chat_sent Final-sweep nit: copilot_chat_sent sent '' for workspace_id/workflow_id in the agent (workspace-less) branch, same phantom-bucket issue as file_downloaded. Make both properties optional and omit them when absent. * fix(analytics): clear stale org PostHog group on personal workspaces Address Cursor review: switching from a team workspace to a personal one left the previous organization group set, so later events kept rolling up under it. When the active workspace has no organizationId, resetGroups() to drop the stale org group, then re-apply the workspace group. * fix: address review — export audit timing, copilot delete signal, group reset - Table export (Cursor MED): audit fires before streaming begins, not after controller.close(), so a mid-stream failure still records the partial export. - deleteCopilotFile (Greptile P1): throw when storage accounting can't be settled instead of silently returning, so callers can detect the file was not deleted. - workspace-scope-sync (Cursor MED): only resetGroups() once workspace metadata is loaded (activeWorkspace present), so a team workspace doesn't transiently lose its org group while organizationId is still null during load. * refactor(observability): final line-justification cleanup Address final audit flags: - Table import: move the 'columns added' audit OUT of the import transaction into a post-commit auditTableColumnsAdded() helper called by the three tx-owning callers, so a mid-import row-batch rollback no longer logs a false 'added N columns' (matches the PR's success-only discipline). - Omit empty-string analytics dimensions on workflow_lock_toggled (workspace_id) and organization_created (name), consistent with file_downloaded/copilot_chat_sent. - Restore an unrelated capacity-check comment removed incidentally. - Move copilot_chat_sent emit above the traceparent comment so the comment sits with the code it documents. * fix(table): attribute import column audit to the importing user Address Cursor review: addImportColumns (async createColumns import path) now threads the importing userId into auditTableColumnsAdded instead of falling back to table.createdBy, so column additions are attributed to the actual member who ran the import rather than the table creator. * feat(observability): drop copilot from storage accounting Per design decision: copilot files are working/conversational artifacts, so gating their materialization on storage quota would fail an agent operation mid-flow when a user is over limit, and metering them would inflate usage enough to block KB/workspace uploads indirectly. Remove copilot quota gates + copilot ingest metering entirely (revert copilot-file-manager, metadata, and the upload route's copilot branch to baseline) and drop the now-unused releaseDeletedFileStorage. KB document metering + quota enforcement (the deliberate, persistent storage path) is unchanged. * fix(analytics): switch workspace + org PostHog groups together Address Cursor review: gate the group-sync effect on workspace metadata being loaded (activeWorkspace present) so the workspace and organization groups always update atomically. Acting during the load window paired the new workspace group with the previous workspace's org group; until metadata loads, events stay consistently attributed to the previous workspace. * fix(table): audit async export at authorization, not job completion Address Cursor review: async exports only emitted TABLE_EXPORTED when the background job reached 'ready', so an authorized export whose job later failed or was abandoned left no audit trail — inconsistent with the sync export route, which audits before streaming. Move the audit + analytics to the async route's authorization point (after the job is claimed/dispatched) and remove it from the runner. Drop the now-unused userId from TableExportPayload. * fix(billing): never let payment_failed instrumentation skip user blocking Address Greptile P1: the payment_failed audit hoisted an unguarded isSubscriptionOrgScoped DB read (for entity_type) directly before the attempt-count user-blocking block. A transient failure of that read would throw out of the handler and skip blocking. Wrap the whole audit/analytics block in try/catch (best-effort), and let the blocking compute its own isSubscriptionOrgScoped as it did originally — instrumentation can no longer abort payment processing. * chore(observability): trim verbose inline comments to concise notes * fix(billing): wrap new dispute/enterprise/subscription instrumentation in Stripe webhook idempotency recordAudit/captureServerEvent calls added for charge disputes, enterprise subscription provisioning, and free->paid subscription creation ran unconditionally with no idempotency guard, unlike their sibling handlers in the same files. Stripe redelivers webhooks at-least-once, so a retry would double-record the audit row and PostHog event even though the underlying DB writes were already idempotent. * fix(observability): tag org-scoped audit metadata with organizationId, guard concurrent table delete The org-scoped self-service audit-log endpoint matches org-level rows via metadata.organizationId (or resourceType=organization). Six recordAudit call sites (subscription create/cancel, admin credit issuance, threshold overage billing, charge disputes, credit purchase, invoice payment succeeded/failed) tagged org-scoped events with a differently-named key (referenceId/entityId/targetOrgId), making them invisible to org admins querying their own audit trail despite being stored in the DB. Add the missing organizationId key everywhere the pattern was missed. Also guard deleteTable's archive UPDATE with isNull(archivedAt) so a concurrent duplicate delete request is a no-op instead of re-archiving and re-firing a duplicate TABLE_DELETED audit row. Adds test coverage for the ORG_MEMBER_ADDED audit/analytics emission in acceptInvitation, which previously had none. * fix(test): queue resolveBillingActorId's owner lookup in payment-failure email test handleInvoicePaymentFailed's new payment_failed audit instrumentation resolves the billing actor via an extra db.select before sendPaymentFailureEmails runs. The test's fixed select-response queue didn't account for it, so the org-admin lookup consumed the wrong queued row and the assertion saw zero email sends. Production behavior is unaffected — each query is independent; this was a mock-queue ordering issue only. * fix(billing): don't let a post-commit usage-limit sync failure suppress the seat audit Cursor Bugbot flagged: reconcileOrganizationSeats committed the seat change and Stripe outbox enqueue in a transaction, then called syncSubscriptionUsageLimits outside it before recording the audit/ PostHog events. A thrown sync left a genuinely-changed seat count with no ORG_SEAT_PROVISIONED/DEPROVISIONED trail. Wrap the sync in its own try/catch (log + continue) so the events always fire for a committed change, matching the fire-and-forget instrumentation pattern used elsewhere in this PR. * fix(billing): guard the new subscription-created instrumentation block Greptile P1: the org-scope check I added for the audit-metadata fix (isSubscriptionOrgScoped) was a raw DB call with no error guard, unlike resolveSubscriptionActorId next to it. Since it ran inside the idempotency lambda with an unconditional rethrow, a transient DB error would abort and retry the whole webhook after the free -> paid usage reset had already committed. Wrap the actor/org-scope resolution + audit + analytics block in try/catch, matching the same guarded instrumentation pattern already used in handleInvoicePaymentFailed. |
||
|
Waleed
|
d078c84ee6 |
chore(typescript): upgrade to TypeScript 7 (native Go compiler) (#5521)
* chore(typescript): upgrade to TypeScript 7 (native Go compiler) Bumps typescript to ^7.0.2 across every workspace package. Full bun run type-check/lint/build/test all pass; apps/sim's type-check (the one needing an 8GB heap bump) drops from ~55s to ~7s wall time. Migration fixes required by TS7's stricter defaults: - baseUrl removed: drop it from 5 tsconfigs (paths already resolved relative to tsconfig dir, so behavior is unchanged) and prefix the one bare (non-relative) paths entry each in apps/sim and apps/realtime with './' - moduleResolution=node10 removed: switch packages/cli and packages/ts-sdk to "bundler", matching the rest of the monorepo - types now defaults to [] instead of auto-including every @types/* package: add "types": ["node"] to the shared base tsconfig (this is fundamentally a Node monorepo, so this restores prior behavior in one place instead of duplicating it per-package), add explicit @types/node deps to packages that now rely on it transitively via @sim/db/@sim/logger, and add "declare module '*.css'" to the two packages with plain (non-module) CSS side-effect imports that TS7's stricter checker now flags - packages/logger's isomorphic `typeof window` check no longer needs DOM lib in every consumer: replaced with `'window' in globalThis` - packages/testing and apps/realtime's fetch/DOM mocks need DOM lib where they're compiled, since they model the browser Fetch API - the `typescript` npm package no longer exports the classic Compiler API from its main entry (moved to unstable/ast subpaths); apps/sim's Function-block route used it at runtime to strip import statements from user code, so that one call site now uses Microsoft's official transition package, @typescript/typescript6 - Next.js 16.2.6's own TypeScript-detection heuristic hardcodes a path TS7 no longer ships, and its auto-install fallback assumes npm/pnpm; added @typescript/native-preview as a devDependency to apps/sim and apps/docs so Next detects a valid native compiler instead of trying (and failing) to auto-install one Not merging yet: TS 7.0.2 published today and is still inside this repo's bunfig.toml minimumReleaseAge (7-day) supply-chain gate, so `bun install` will fail for everyone until 2026-07-15. Opening this now to get it through review; hold the actual merge until then. * fix(typescript): address Greptile review findings on TS7 upgrade - packages/logger: 'window' in globalThis treats a shim that leaves globalThis.window explicitly undefined as browser-only, silently dropping production server logs. Restore the original typeof !== 'undefined' semantics via an inline cast instead, so it stays correct without requiring DOM lib in every consumer. - packages/ts-sdk, packages/cli: both are tsc-built, published as Node ESM (package.json "type": "module" with an "exports" map). "moduleResolution": "bundler" is too permissive for that target - it accepts import patterns (e.g. extensionless relative imports) that Node's actual ESM resolver rejects at runtime. Switch both to "module"/"moduleResolution": "nodenext", the correct pairing for a published Node ESM package. Verified real tsc builds (not just --noEmit) still succeed for both. * chore(bunfig): temporarily disable minimumReleaseAge gate for TS7 install TS 7.0.2 published today, still inside the 7-day gate. Lowering to 0 to unblock this merge; will restore to 604800 in an immediate follow-up commit right after merging. |
||
|
Theodore Li
|
0613cebfab |
feat(db): resolve DATABASE_URL per role (DATABASE_URL_<ROLE> with fallback) (#5276)
* feat(db): resolve DATABASE_URL per role (DATABASE_URL_<ROLE> with fallback) * fix(db): pin realtime process to SIM_DB_ROLE=realtime so both pools share the role Without it, the realtime process left SIM_DB_ROLE unset: the shared @sim/db client defaulted role to 'web' (web pool profile + DATABASE_URL_WEB) while socketDb used 'realtime', so the two pools diverged after cutover. Set it at the process level (bootstrap + dev/start scripts), mirroring DB_APP_NAME, so the shared client and socketDb both resolve the realtime profile and URL. |
||
|
Theodore Li
|
845a6276d9 |
perf(db): per-role Postgres connection-pool profiles (#5232)
* perf(db): drive Postgres pool size + application_name from per-role profiles Replace ad-hoc DB_APP_NAME sizing with a per-role profile map keyed by SIM_DB_ROLE (web/trigger/realtime), defaulting to web. Trigger machines open a small pool instead of 15 to avoid PgBouncer connection exhaustion. Also size realtime's separate socketDb pool down to 10. * fix(db): throw on invalid SIM_DB_ROLE instead of silently using web pools * fix(db): use Object.hasOwn for SIM_DB_ROLE validation to avoid prototype keys |
||
|
Theodore Li
|
a68d38ae75 |
feat(db): attribute Postgres connections by runtime via application_name (#5211)
* feat(db): attribute Postgres connections by runtime via application_name * improvement(db): label migration-runner connection sim-migrate; trim DB_APP_NAME comment * fix(db): label realtime's shared @sim/db connections sim-realtime too The realtime process uses both its own socketDb pool and the shared @sim/db client (handlers, preflight, permissions). Only socketDb was labeled, so the shared client defaulted to sim-app, mislabeling much of realtime's DB traffic. Set DB_APP_NAME=sim-realtime at the process level (bootstrap before the dynamic @/index import for prod; dev/start scripts for local) so both clients report it. |
||
|
Theodore Li
|
e1c3c7f6c9 |
feat(secrets): ingest env secrets at container runtime instead of fanning into ECS taskdef (#5189)
* feat(secrets): ingest env secrets at container runtime instead of fanning into ECS taskdef The app/socket ECS taskdefs were ~42KB, ~93% of which was the secrets[] array: 268 pointer entries each restating the full ~78-char secret ARN, marching toward the 64KB taskdef limit and growing ~150 bytes per hosted key added. The secret blob itself is only ~18KB/268 keys. Move secret delivery to container boot: new @sim/runtime-secrets loadRuntimeSecrets() reads SIM_ENV_SECRET_ID, fetches the combined secret once, and hydrates process.env (no-clobber, no-op when unset, fail-fast). Bootstrap entrypoints for app + realtime await it before importing the real server (env-flags reads env at module load). The app bootstrap is bun-bundled in the Dockerfile builder stage since it runs outside the Next standalone bundle; realtime keeps full node_modules and runs the TS entry. Backward-compatible: with the current fan-out taskdef the loader no-ops and the app reads the injected env vars unchanged. The matching infra change (empty secrets[] + SIM_ENV_SECRET_ID) ships separately, after this image is live. * fix(runtime-secrets): address review feedback - Move the binary-secret guard outside the retry loop (sendWithRetry) so a missing SecretString throws immediately instead of burning 3 attempts + backoff. - Bound each Secrets Manager request with AbortSignal.timeout(5s) so a stalled response can't hang boot indefinitely. - Drop the redundant @aws-sdk/client-secrets-manager pin from apps/realtime; it resolves transitively via @sim/runtime-secrets. - Add a test for the non-retriable binary-secret path. |
||
|
Vikhyath Mondreti
|
91f9dfdaec |
improvement(governance): derived access (#5134)
* improvement(governance): org-ws-credential roles clarity * revert isHosted * improvement(credentials): code cleanup * address comments * make kb cascade delete on user hard delete * revert env flags * chore(db): drop local 0242 migration to regenerate after merging staging Our 0242 collides with staging's 0242. Remove it (and its snapshot + journal entry) so the KB-cascade migration can be regenerated with the correct number on top of the merged staging migrations. * chore(db): regenerate kb→workspace cascade migration as 0243 Regenerated via drizzle-kit generate on top of the merged staging migrations (staging took 0242). Re-applied the safety edits: NOT VALID + separate VALIDATE on the FK re-add, and the -- migration-safe note on the DROP. check:migrations passes. * improve copy * update docs |
||
|
Vikhyath Mondreti
|
d14bc78d03 |
fix(realtime): re-check workspace role on mutating socket events (#5080)
* fix(realtime): re-check workspace role on mutating socket events * address comments |
||
|
Waleed
|
ae075f871f |
Revert "fix(realtime): re-validate socket role and evict revoked collaborator…" (#5051)
This reverts commit
|
||
|
Waleed
|
4ab8760ae0 |
fix(realtime): re-validate socket role and evict revoked collaborators (#5050)
Socket.io authorized workflow access only at join and cached the workspace role in presence, so a removed or downgraded collaborator kept live read/ write access until they disconnected. - Re-validate the cached role against the permissions table on mutating events, bounded by a short TTL; refresh or evict on change - Add /api/permissions-updated so the app reconciles active rooms, evicting revoked users (cross-pod) and refreshing downgraded roles - Notify realtime on workspace member removal and permission changes |
||
|
Vikhyath Mondreti
|
f7b40fe4a4 |
fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions (#4975)
* fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions * update docs |
||
|
Vikhyath Mondreti
|
d9f78c0f56 |
improvement(sockets): make offline mode recoverable and stop transient races tripping it (#4980)
* improvement(sockets): make offline mode recoverable and stop transient races tripping it * data persistence issues should trigger offline mode and force refresh * code cleanup |
||
|
Waleed
|
272bad9718 |
feat(realtime): preflight schema-compatibility check on startup (#4940)
* feat(realtime): preflight schema-compatibility check on startup The socket service authorizes every connection with a full-row query against the workflow table. When a deploy ships a realtime image whose compiled schema is ahead of/behind the live DB (e.g. a column dropped by a migration the image predates), that query fails on every request and silently breaks persistence — yet the process stays up and the shallow /health probe keeps returning 200, so the deploy looks healthy while serving nothing. Run one representative workflow query before listen(): a schema mismatch throws, propagates to the entrypoint, and the task exits non-zero and never goes healthy, so CodeDeploy auto-rolls-back instead of shifting traffic onto broken tasks. Schema-class errors (undefined column/table/function) fail fast; connection-class errors retry with backoff so a cold DB at boot does not flap. Runs once at startup, never on the per-probe LB health check, to avoid a DB blip mass- terminating the fleet (cascading failure). * fix(realtime): unwrap cause for schema codes, drop sleep after final attempt - isSchemaMismatch now walks the error.cause chain — drizzle wraps the driver error, so the SQLSTATE often lives on the inner cause, not the outer throw. Without this a wrapped 42703/42P01 was retried 5x and mis-reported as "database unreachable" instead of failing fast. - No longer sleeps after the final failed attempt (~6-10s of dead wait that undermined the fail-fast contract); sleep now only happens between attempts. - Tests: assert sleep is called exactly 4 times on exhaustion, and add a wrapped-cause fail-fast case. |
||
|
+1
|
39418f9615 |
improvement(mothership): v0.2 (#4923)
* CONTRACTS * updates * Prompt caching * Fix regression * updates to prompt caching * Prompt caching trace * VFS updates * Changelog and plan * VFS update * Improvement/mothership (#4775) * ff * System role in cache * Fixes * Add dynamic user info * Improve tool search * improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * New doc setup * Fixes * Fixes * Logs tools * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * Add finalization on error * Fixes * change dev CI to bun run db:push * Updates Please enter the commit message for your changes. Lines starting * Contrat update * improvement(nested): subagents * updates * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Vikhyath Mondreti
|
1933fc4f11 | fix(env): schema treatment of empty string (#4862) | ||
|
Waleed
|
a7b0bd311d |
fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) (#4837)
* fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) - Bump vitest and @vitest/coverage-v8 to ^4.1.0 across all workspaces (only patched release for the critical 'Vitest UI server arbitrary file read/execute' advisory; no 3.x backport exists) - Widen @sim/testing peer range to ^3.0.0 || ^4.0.0 - Migrate constructor mocks to class expressions: vitest 4 uses Reflect.construct for mocks invoked with new, and arrow/function implementations are not constructable (function expressions also get reverted to arrows by biome's useArrowFunction) - Remove deprecated test.poolOptions from apps/sim/vitest.config.ts (options are now top-level in vitest 4) * fix(deps): exclude vulnerable vitest 4.0.x from @sim/testing peer range Tighten the v4 arm of the peer range to >=4.1.0 <5.0.0 so the peer requirement cannot be satisfied by the unpatched 4.0.x builds that GHSA-5xrq-8626-4rwp affects. * fix(testing): make vitest 4 constructor mocks type-check cleanly - logging-session & mcp-oauth mocks: a class passed to mockImplementation has a construct signature that isn't assignable to its (...args) => any parameter, failing tsc. Use named function declarations instead (constructable via Reflect.construct, assignable to mockImplementation, and not rewritten to arrows by biome's useArrowFunction). - database.mock.ts: vitest 4's generic vi.fn typings no longer break the self-referential cycle on the transaction callback's tx param; loosen tx and annotate the callback's return type to resolve the implicit-any errors. * test(isolated-vm): de-flake queue-capacity scheduler tests The 'queue is full' and 'per-owner queued limit' tests relied on 'await sleep(1)' to assume the first request had reached the queue before submitting the overflow request. The first request only enqueues after an async spawn-failure chain (acquireWorker -> spawn exit -> resolve null -> enqueue), which isn't guaranteed within 1ms under CI load — the overflow request then found an empty queue and hit the 200ms queue-wait timeout instead of the capacity rejection. Replace the wall-clock barrier with a deterministic, event-driven one: hold the single global concurrency slot (IVM_MAX_CONCURRENT=1) with an active worker and await an explicit 'dispatched' signal (fired when the worker receives its execute message, after the scheduler counts it active). The follow-up requests then deterministically hit the synchronous enqueue path. Also drops the queue-wait timeout from 200ms to 50ms, so the tests run faster. |
||
|
Waleed
|
8d7bbbc670 |
chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting - Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files - Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors) - Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities - Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time - Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access - Add check:utils script to package.json * chore(utils): replace deepClone wrapper with structuredClone built-in deepClone() was a one-line wrapper around structuredClone(), which is universally available in Node 17+ and all modern browsers. Removing the abstraction reduces indirection and means contributors don't need to learn a project-specific name for a well-known built-in. - Remove deepClone from packages/utils/src/object.ts and index.ts - Replace all 17 call sites with structuredClone() directly - Update check:utils script suggestion text - Update CLAUDE.md and global.md docs * fix(utils): add missing biome noRestrictedImports rule and correct truncate docs - Add noRestrictedImports to biome.json under style — bans nanoid and uuid package imports at lint time (crypto.randomUUID/randomBytes are caught by the check:utils grep script which handles global property access) - Correct truncate() TSDoc and parameter name: sliceLength makes it clear that total output length is sliceLength + suffix.length, matching the behavior all callers were already written to expect * fix(utils): add missing getErrorMessage imports at 4 call sites The sweep agents added getErrorMessage calls without the corresponding import in 4 files, causing test failures. Added the missing imports. * fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue - Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for internal package imports; inline the jitter crypto call directly - Add missing getErrorMessage imports to 32 files where the sweep added calls without the corresponding import (caught by type-check and test runs) - Remove accidental getErrorMessage import from crowdstrike/query/route.ts which has its own domain-specific getErrorMessage for parsing CrowdStrike's JSON error format - Fix use-sub-block-value.ts type error from structuredClone narrowing: add 'as T' cast at emitValue callsite (safe — valueCopy is always a structural copy of newValue) * fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage The catch block was calling the local getErrorMessage function which parses CrowdStrike API JSON responses, not JavaScript Error objects. Use toError(error).message to correctly extract the message from a caught value in this context. |
||
|
Waleed
|
b5dba82ac9 |
improvement(db): reduce connection saturation and egress hotspots (#4594)
* improvement(db): reduce connection saturation and egress hotspots * fix(vfs): preserve native content type in copilot SQL projection * fix(vfs): guard jsonb_array_elements against non-array contentBlocks |
||
|
Vikhyath Mondreti
|
43f53bb7d3 |
feat(execution): payload size bottlenecks with lazy execution value hydration, safer materialization, and batched parallel execution (#4560)
* improvement(resolver): lazy resolution for underlying fields greater than 10MB * progress * feat(parallel): batching * codegen to allow inline substitution * address comments * ui inconsistencies * cleanup redundant code * address more comments * address comments * replace helper * fix tests |
||
|
Vikhyath Mondreti
|
6cb779601a |
feat(search-replace): search & replace, cut, deploy modal ui flicker (#4507)
* feat(search): workflow search and replace * fix alignment * fix hidden fields bug * fix loops/parallel badge case * resource resolver * add cut * update docs * address comments * make source code for func blocks dispay resolved code instead * fix match issue * fix padding |
||
|
Theodore LiandClaude Opus 4.7
|
98f8e854eb |
improvement(tables): extract TablesDetail wrapper, ship trigger followups (#4476)
* ui improvements * Update status pils, make checkbox column sticky * add Run workflow to context menu * Refactor dispatching logic * fix checkbox width to be smaller if csv is small * Add drag behavior for workflows, stop workflow on multi select * fix z index of checkbox to left, add view workflow button * Switch to emcn buttons for Add inputs * Split up workflow sidebar from column sidebar, refactor cells * Lint and add auto run toggle * fix column reordering, add action bar * Create and use emcn square * Reconcile post-merge: drop positionMap, use rowId-based selection Staging refactored Tables UI to decouple from DB position (gutter from array index, checkedRows keyed by rowId, no PositionGapRows). Bring HEAD's action-bar / context-menu helpers in line: contextMenuRowIds, selectedRowIds, actionBarRowIds now key off row.id and walk `rows` directly. Drop the maxPosition / positionMap derived state. Collapse COLUMN_SIDEBAR_WIDTH_CSS to a numeric COLUMN_SIDEBAR_WIDTH used by both the sidebar shell and the table's reserved padding-right. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(table): backfill remapped workflow outputs from execution logs When a workflow column is re-pointed to a different (blockId, path), populate its existing rows with the new output's value pulled from saved execution logs instead of leaving them empty until the next run. Rows where the new mapping has no logged value clear (matching the previous behavior for those rows), but rows where the workflow already has the new output's value surface immediately. Refactor backfillAddedGroupOutputs into a generalized backfillGroupOutputsFromLogs helper with an `overwrite` flag — used in both the added-outputs path (preserves hand-edited values) and the new remapped path (overwrites since the new mapping is the source of truth). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): update column type when remapping workflow output A remap that changes the output's leaf type (string → number, json → boolean, etc.) was leaving the column's declared type stale. The clear- then-backfill flow then failed schema validation on every row, so the backfill silently aborted and the column stayed empty. Resolve the new leaf type via flattenWorkflowOutputs + columnTypeForLeaf for each mappingUpdate, and patch schema.columns[i].type before the schema write. The clear-tx then backfill ordering now works end-to-end across type changes. If the workflow or its target output can't be resolved (workflow deleted, block removed), fall back to leaving the column type alone — the backfill will skip rows whose picked value doesn't match, same as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): stringify objects instead of "[object Object]" in cells If a column's declared type lags its row data (e.g. a workflow column mid-remap, where the schema cache hasn't refetched yet but the row data already has the new mapping's value), formatValueForInput and the cell-render text variant fell through to String(value) and rendered "[object Object]". JSON-stringify objects in both spots so the transient skew shows the actual data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): drop extra left border on workflow group meta header The meta cell had border-r/b/l while regular headers have only border-r/b. With border-separate tables, that extra 1px left border shifted the meta cell's content one pixel right of the columns below it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): align workflow meta header without dropping its left border Restore border-l and pull the cell back -1px with -ml-px so the visible left border overlaps the previous cell's right border instead of adding 1px to the meta cell's box. Content lines up with the columns below. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): draw meta header left border via ::before pseudo Adding border-l to the meta cell shifted its content right by 1px because table-fixed + border-separate honors the border inside the colspan'd cell's width budget. -ml-px doesn't work on <th>. Render the visible left edge via a ::before at left: -1px instead — paints over the prior cell's right border without consuming any of the meta cell's content area. Content lines up with the columns below. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): add missing barrels, drop doubled-path imports Match the convention used by logs/components: every component folder exposes its public API via index.ts so consumers import from the folder name, not from its internal filenames. - New barrels: column-config-sidebar/, workflow-sidebar/, table-action-bar/, table/cells/, table/headers/. - Rename table-filter/index.tsx → index.ts (barrel is not a component). - Top-level components/index.ts re-exports every sibling folder so external consumers have one import path. - Replace `from '../foo/foo'` doubled paths in table.tsx with the shorter barrel-anchored form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): introduce TablesDetail wrapper as thin passthrough Phase 1 step 0 of the wrapper extraction (see plan okay-lets-make-a-shimmying-trinket.md). page.tsx now renders TablesDetail, which today is a passthrough to <Table>. Subsequent commits lift surface state out of <Table> into this wrapper one piece at a time. The mothership chat path (<Table embedded>) is untouched — <Table> stays exportable as a lower-level component for embedded contexts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift slideout panel state into TablesDetail wrapper The three right-edge slideout panels (column config, workflow config, execution details) move out of <Table> into the wrapper. The wrapper owns a single useReducer that encodes the at-most-one-open invariant as a discriminated union — opening any one panel automatically closes the others. <Table> emits open requests via three new callback props. Also extract <ExecutionDetailsSidebar> from inline-in-table.tsx to its own folder so the wrapper can compose it cleanly. Update the embedded mothership callsite (resource-content.tsx) to render <TablesDetail embedded> instead of <Table embedded>. Phase 1 step 1 of the wrapper extraction. <Table> shrinks from 3849 → 3787 lines; <TablesDetail> grows from 19 → 145 lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift delete-table modal + mutation into wrapper The delete-table confirmation modal and `useDeleteTable` mutation move out of <Table> into TablesDetail. <Table> exposes a new `onRequestDeleteTable` callback fired by the page-header Delete action. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift CSV import dialog into wrapper ImportCsvDialog moves out of <Table>. Grid exposes `onRequestImportCsv` fired by the page-header menu item; wrapper owns the open state and renders the dialog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift RowModal (edit + delete) into wrapper Both RowModal instances move out of <Table> into the wrapper. Grid emits `onOpenRowModal(row)` (Space key) and `onRequestDeleteRows(snapshots)` (context menu). Post-delete cleanup (push undo, clear selection) needs grid-internal state, so the grid populates an `afterDeleteRowsSinkRef` callback that the wrapper's modal `onSuccess` invokes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift delete-columns modal into wrapper The destructive delete-columns confirmation modal moves into the wrapper. Grid emits `onRequestDeleteColumns(names)`; the cascade itself (per-column mutation, undo push, columnOrder + columnWidths cleanup) stays in the grid as a sink the wrapper invokes on confirm — too grid-internal to lift cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift run/stop mutations + TableActionBar to wrapper useRunGroup and useCancelTableRuns move out of <Table> into the wrapper, along with the <TableActionBar> render. Grid receives onRunGroup, onRunRows, onStopRow, onStopRows, onStopAll, and cancelRunsPending as props — used by the per-row gutter Play/Stop, the workflow-group meta-cell run menu, and the right-click context menu's Run/Stop on selection items. Action-bar selection state (actionBarRowIds, runningInActionBar, hasWorkflowColumns) is derived from grid-internal state, so the grid emits a `SelectionSnapshot` via `onSelectionChange` from a useEffect. Wrapper uses the snapshot to drive the floating <TableActionBar>. Phase 2 step 1 of the wrapper extraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift queryOptions to wrapper queryOptions (filter + sort) moves out of <Table> into the wrapper, making it a single source of truth that drives one useTable call. The wrapper passes the bundle down to the grid; sort/filter handlers in the grid call onQueryOptionsChange. Eliminates the previous double-useTable pattern (one for the grid's filtered/sorted view, one in the wrapper's hardcoded null/null query for sidebar metadata). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift page header (breadcrumbs/options/filter) to wrapper Phase 3 of the wrapper extraction. The full page-header surface moves out of <Table>: - ResourceHeader (breadcrumbs, table-rename UI, headerActions, createTrigger) - ResourceOptionsBar (sort + filter toggle) - TableFilter (filter panel — wrapper owns filterOpen state) - RunStatusControl (in the leading actions when runs are active) useRenameTable + useInlineRename for the breadcrumb name move to the wrapper. The grid populates pushTableRenameUndoSinkRef so the rename is still part of the grid's undo stack. Extract NewColumnDropdown and RunStatusControl from inline-in-table.tsx to their own folders so the wrapper composes them cleanly without reaching into the grid's internals. Hoist generateColumnName from grid-internal useCallback to a shared util so both the page-header and inline-header NewColumnDropdowns use the same logic. After this lift <Table> is the data grid only — no page surface, no modals, no slideouts, no breadcrumbs. The selection snapshot now includes totalRunning so the wrapper can render the page-header RunStatusControl from outside the grid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tables): cleanup pass on TablesDetail wrapper extraction Six-pass cleanup against the wrapper extraction diff: - Effects: add content-compare bailout to onSelectionChange emit so unchanged snapshots don't churn wrapper re-renders. - Memos: drop unnecessary activeSortState memo, fold into sortConfig. - Callbacks: remove ~10 useCallbacks with no observed reference (sidebars not memoized, modals not memoized, inline arrows on non-memoized children); keep the ones that feed into <DataRow>/<RunStatusControl>/ <ResourceHeader> (memoized) or grid-side useCallback deps. - Dead props: drop onQueryOptionsChange/onRequestDeleteTable/ onRequestImportCsv from <Table> — the page-header lift made them unused but the props weren't removed. - React Query: drop redundant tableWorkflowGroupsRef (created when onRunRows was useCallback-wrapped; after callback cleanup it can read the query data directly). - emcn: normalize Loader sizing to h-[14px] w-[14px] to match the codebase convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): re-seed columnOrder when columns change server-side The metadata-seed effect short-circuited after the first seed, so any later schema change (e.g. adding a workflow output column) couldn't push the new column into local columnOrder. The new column would then fall into the "remaining" bucket of `displayColumns` and render at the end of the table — until the user refreshed and the grid re-mounted with the now-current metadata. Drop the `metadataSeededRef.current` short-circuit from the early return so the effect can also reach the after-first-load re-seed branch, which already does the right thing (only re-seeds when the set of columns changes, leaves pure-reorder cases alone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): rename wrapper to <Table>, grid to <TableGrid> Match the naming convention used elsewhere in the workspace (workflow.tsx → <Workflow>, base.tsx → <Base>, logs.tsx → <Logs>). - tables-detail.tsx → table.tsx (exports <Table>) - components/table/ → components/table-grid/ (exports <TableGrid>) - components/table-grid/table.tsx → table-grid.tsx - Drop <ExecutionDetailsSidebar> — was a 3-line passthrough (executionId → useLogByExecutionId → <LogDetails>); inline directly into table.tsx where it's used. - Flatten components/run-status-control/ folder to a single components/run-status-control.tsx file. 25-line single-use component with no internal subdirs — folder was overhead. Matches knowledge's max-badge.tsx precedent. Net: 1 wrapper rename + grid rename + 2 folder collapses, all imports updated. The mothership chat callsite updates from <TablesDetail embedded> to <Table embedded>. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): don't show "Waiting" for autoRun=false workflow groups A workflow group with autoRun=false never fires from the scheduler — the cell stays empty until the user clicks Run manually. Treating empty cells as "Waiting" misleads the user into thinking the group will auto-fire once deps are filled, which it won't. Skip autoRun=false groups when computing the per-row waiting labels so their cells render the empty-dash instead of the Waiting pill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(copilot): regenerate tool catalog from copilot dev (#247) Pulls in the workflow_group operations on user_table: add_workflow_group / update_workflow_group / delete_workflow_group / add_workflow_group_output / delete_workflow_group_output / run_workflow_group, plus the autoRun / blockId / dependencies / groupId parameters and a tightened mapping description for import_file. Also picks up biome import-order fixes from `bun run lint`. * improvement(table): action bar in mothership + per-execution mode Three related improvements to the table action bar: 1. Reposition from `position: fixed` to `position: absolute` inside the table's container. Fixed-positioning anchored to the viewport, which centered the bar across the whole window instead of the table panel — wrong in mothership embedded view, where the table sits in the right half. Absolute scopes the bar to the table's bounds. 2. Show the bar for single-execution highlights — when the user selects one workflow-output cell, or 1 row × N cols all within the same workflow group. The bar enters per-execution mode with Run / Stop / View execution buttons targeting that one cell or group. 3. Skip View execution for cancelled cells. A cancelled cell may have been cancelled before the worker ever picked the job up, so its executionId can't be relied on. Tighten the gate everywhere (context menu + action bar) to only `completed` / `error` / `running`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): backfill on add_workflow_group_output, don't re-run addWorkflowGroupOutput (the one-shot single-output add path used by the copilot user_table tool) was calling triggerWorkflowGroupRun({ mode: 'all' }) after appending the output — that re-fired the workflow on every row. Trace a307ed8fd5fe2d931aa84dedab5a60f0 shows ~75 workflow-group-cell jobs enqueued in the seconds after a single add_workflow_group_output call. Replace with backfillGroupOutputsFromLogs (overwrite: false), the same flow updateWorkflowGroup uses when receiving newOutputColumns. Reads each row's saved trace spans and writes the new output's value back — no compute beyond a JSONB write per row, no double-billing the user for runs they didn't ask for. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): drop sql.raw quote-escaping in column-name interpolation Six call sites in lib/table/service.ts built JSON-key string literals at runtime via `sql.raw(\`'\${name.replace(/'/g, "''")}'\`)` for use with PostgreSQL's `data->'key'` / `data->>'key'` operators. Practically safe (NAME_PATTERN gates column names to alphanumeric+underscore at insert time) but a smelly pattern that breaks the moment validation loosens. Both `data->` and `data->>` accept a parameterized text value as the key, so the `sql.raw` is unnecessary. Replace each with a normal `${name}::text` binding. No behavior change; eliminates the manual quote-escaping surface. Affected sites: renameColumn (the data-rewrite UPDATE), upsertRow's match filter, updateColumnType's IS-NOT-NULL gate, updateColumnConstraints' required-check + unique-duplicate-check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(copilot-tool): forward autoRun + mappingUpdates on update_workflow_group The sim-side service and contracts already accept both fields, but the copilot tool's update_workflow_group handler was dropping them on the floor. Now `args.autoRun` (toggle the persisted auto-fire flag) and `args.mappingUpdates` (per-output (blockId, path) swap) get forwarded through to updateWorkflowGroup. Pairs with the upcoming copilot-side change that exposes these in the tool catalog JSON / Go handler / prompting (see copilot branch redo-workflow-tools). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): keep gutter border visible when hovering Run-row button The per-row Run button sat flush against the row-gutter cell's right border. Its hover background (rounded-rect surface-2) painted over the border line for the 20px height of the button, making the gutter divider appear to disappear at the hovered row. Add mr-px to the button so the hover bg stops 1px short of the cell's right edge, leaving the divider intact. * fix(table): unify auto-fire and manual run paths in scheduler scheduleWorkflowGroupRuns now owns eligibility, autoRun semantics, dep evaluation, and enqueue for both paths. Auto-fire callers omit opts; manual callers (triggerWorkflowGroupRun) pass { groupId, isManualRun: true } to bypass the autoRun=false skip and (for autoRun=false groups) the dep check. Per-row /run-workflow-group route delegates to triggerWorkflowGroupRun with rowIds=[rowId]. Single server-side path for both manual entry points. Also: optimisticallyScheduleNewlyEligibleGroups skips autoRun=false groups so editing a row's data doesn't phantom-mark autoRun=false output cells as Queued. * fix(table): render empty cells as blank, not em-dash Empty cells (any column type) showed an em-dash placeholder. Drop it so empty cells render blank — matches what the user expects when nothing's there. * fix(table): per-row Run fires autoRun=false groups regardless of deps handleRunRow filtered out every group whose deps weren't satisfied, which silently dropped autoRun=false groups (since their deps usually aren't satisfied — that's the whole point of autoRun=false). Click Run row, the autoRun=false group's cells stayed empty. Mirror the scheduler's semantics: autoRun=false bypasses the dep check, autoRun=true still requires deps. * fix ui shape * improvement(table): collapse run ops into run_column, derive action-bar buttons from selection The action bar now reflects what's actually selected: - Selection-driven scope (cells the user highlighted, not their full rows) - Play visible when there's anything empty/failed; Refresh when there's anything completed; both for mixed - run_cell / run_row deleted; everything funnels through run_column - Per-row gutter Play, right-click "Run workflows on N rows", and column-header menu all share the canonical run path - Shared RunMode type from the contract; cleanup pass via /simplify (readExecution / isExecInFlight reuse, runScope helper, flat onViewExecution prop) * chore(copilot): regen tool catalog after dropping run_cell / run_row + dependencies.workflowGroups Mirror the copilot-side catalog change so the generated TS catalog matches the deployed copilot tool surface. * fix(table): atomic per-key writes for executions, plus run-op race fixes The executions blob on user_table_rows was read-modify-written wholesale on every update. Concurrent writers (a column edit and a manual-retry stamp, two pickup calls, a cancel and a cascade) each computed a merge from their own snapshot, and the last writer clobbered keys it never touched — producing stuck "queued" cells, vanished stamps, and stale completed exec records reappearing after retries. Fixes: - updateRow / batchUpdateRows now apply executionsPatch via a SQL jsonb merge expression. Each writer only mutates the keys it explicitly patches; other keys are preserved. Eliminates the cross-key clobber. - writeWorkflowGroupState bypasses the stale-worker guard for `queued` (new scheduler stamp) and `cancelled` (authoritative cancel) writes — those ARE the new authority for the cell. Previously the new run's stamp was being rejected by the same guard meant to block the OLD worker's writes. - skipScheduler flag on UpdateRowData / BatchUpdateByIdData lets the cancel path and runWorkflowGroupsInternal opt out of the implicit auto-fire pass (cancel was waking up siblings; manual-run was racing its own scheduler). - CELL_CONTENT pinned to h-[22px] so status badges don't grow rows. * chore(table): remove table-row sockets, both sides Tables don't use realtime sockets in prod — strip the dead path so we stop paying the per-row HTTP forward + socket emit on every cell write. Polling on running execs already covers reconciliation. Sim side: - service.ts: drop notifyTableRowUpdated/Deleted, notifyTableDeleted, the postRealtimeBridge helper, and all callsites. - hooks/queries/tables.ts: drop the socket subscription block in useTableRows; poll-on-running stays. Remove useEffect / useSocket imports. - app/.../tables/[tableId]/hooks/use-table.ts: drop the merge-on-event useEffect and unused imports. - app/workspace/providers/socket-provider.tsx: drop joinTable/leaveTable, onTableRowUpdated/Deleted/onTableDeleted, currentTableId state, related events + types. Realtime side: - handlers/tables.ts deleted; index.ts no longer wires it. - routes/http.ts: drop /api/table-row-updated, /api/table-row-deleted, /api/table-deleted endpoints. - rooms/{memory,redis}-manager.ts: drop emitToTable, handleTableRowUpdated/ Deleted, handleTableDeleted, related imports. - rooms/types.ts: drop method declarations, TableRowUpdatedPayload type, tableRoomName helper. - middleware/permissions.ts: drop unused verifyTableAccess. Bonus from parallel work: - cell-content typewriter trigger refinement. * fix(table): clearing a workflow output cell also clears its exec record When the user wipes a workflow output column value, the auto-fire reactor needs to be re-armed for that group. Previously, a stale cancelled / error exec record blocked the eligibility predicate (gate at line 79 hard-rejects those statuses on auto-fire) and the cell stayed stuck in its old terminal state — visible as "Cancelled" cells that wouldn't re-run no matter what. Both updateRow and batchUpdateRows now derive an `executionsPatch[gid] = null` for any output column the patch sets to empty. The data clear and the exec clear ride the same SQL transaction, so the row never lands in a stale- status-with-empty-data state. Symmetric to how `completed` already worked via `areOutputsFilled` in the predicate — clearing the cell wins over the prior exec status, regardless of what that status was. (Also revert typewriter-trigger experiment from a parallel session that was in-progress on this branch.) * fix(table): waiting state, optimistic UX, schema-mutation polling, exec cleanup A bundle of small UX + correctness fixes around workflow-cell run state. cell-render.tsx - In-flight (queued/running/pending) now wins over the existing value, so re-runs surface immediately instead of looking like nothing happened until the worker writes the new value. - "Waiting on X" wins over a stale `cancelled` / `error` exec when deps are unmet — clearing a dep now reads as actionable instead of stuck. useRunColumn (hooks/queries/tables.ts) - onSettled now cancels in-flight polls before invalidating. Stops a poll that landed mid-mutation from clobbering the optimistic state with stale data, which produced the queued → cancelled → queued flicker. addWorkflowGroup / updateWorkflowGroup (autoRun toggle on) - Awaits scheduleRunsForTable instead of fire-and-forget. The route returned before the queued exec stamps committed, so the post-mutation refetch saw no in-flight cells and polling never started — cells looked stuck even though the server eventually stamped them. deleteColumn / deleteColumns - Strip orphaned executions[gid] keys when deleting a column orphans its parent group. Without this, stale running/queued exec records lingered on every row forever and inflated the page-header "N running" counter even on tables with no actually-running cells. UI - Action-bar leading label: "Selected N workflow cell(s)". - Context menu: Run / Refresh items mirror the action bar's Play / Refresh split, gated on the same selection-status flags so both surfaces show the actions that match the current state. * refactor(table): consolidate exec-status helpers + fix N-running counter Cleanup pass on the recent table changes — pulls duplicated predicates and SQL snippets into shared helpers and fixes one drift bug along the way. - isExecInFlight: now single export from lib/table/deps.ts. Removed the duplicate in components/table-grid/utils.ts. Used by isGroupEligible (server eligibility) and runningByRowId (client counter). - isOptimisticInFlight: kept local to hooks/queries/tables.ts — renamed from isInFlight to disambiguate from the stricter isExecInFlight. The two predicates differ on `pending` without a jobId: optimistic patches and poll-trigger want the broader version, eligibility wants the strict one. - areOutputsFilled: single export from lib/table/deps.ts, dropped duplicate from workflow-columns.ts. - classifyExecStatusMix: shared row × group walker in table-grid/utils.ts. Replaces two copies of the same loop in table-grid.tsx (selectionStats + contextMenuStats). Both surfaces now have the same short-circuit semantics, including the seen-all-selected-rows early break that contextMenuStats was missing. - stripGroupExecutions: SQL helper in service.ts. Replaces three copies of the `UPDATE user_table_rows SET executions = executions - $gid::text` pattern across deleteColumn / deleteColumns / deleteWorkflowGroup. Drift bug: - runningByRowId / totalRunning counted only `running` and `queued`. Every other in-flight check in the codebase treats post-stamp `pending` as in-flight too, so the page-header "N running" badge briefly dropped to 0 between scheduler stamp and worker pickup. Now uses isExecInFlight. * fix(table): address pr review (drop dead workflowNameById prop, reset didDragRef on dragend, align sidebar width) * fix(table): scope post-clear schedule to targeted groups, forward mode Multi-group manual runs (Run row, gutter Play, action-bar Play across mixed completed + cancelled cells) re-fired completed-and-filled siblings. runWorkflowGroupsInternal cleared only the groups it filtered, then called scheduleRunsForRows with isManualRun: true and no group / mode filter — so the post-clear pass walked every group on the table with default mode 'all', and any autoRun=true completed sibling whose deps were satisfied got queued again. Scope the post-clear call to targetGroups and forward mode. * fix(table): meta-cell drag-leave flicker guard + plumb unique on create * fix(table): strip sibling deps when removing workflow output via updateWorkflowGroup deleteWorkflowGroup already stripped removed-column deps from sibling groups, but updateWorkflowGroup (the path the UI takes when deleting one output of a multi-output group) didn't — schema validation then rejected the update with 'Group X depends on missing column Y'. * improvement(table): debug logs at every cascade decision branch * improvement(table): parallelize queued-stamp writes within concurrency-cap chunks * Simplify stripping column names * fix lint, ci --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Theodore Li
|
31cfb74dc2 |
feat(table): add workflow execution column type (#4338)
* Add table triggers for columns and row added * Add async batching job for running column * Add ui improvements, stop mechanism * Use trigger dev for workflow runs * Use unified column sidebar for table * Add socket waits for tables, multi column workflow support * change back to cell based trigger jobs * Reuse code, add view log inline in table view * reorganize db to treat each column separately * adjust column naming strategy * Column ui improvements * fix live update on table * Change new column behavior * fixed errored workfows not showing as stopped * Column sidebar improvements * fix table column swapping behavior * fix bugs * add prompting, fix lint * fix ui stuff * flip feature flag * Add zod contracts fix initial auto-run of columns * ui improvements * Use db filter to query unran rows * Use live workflow run * Change wording for deleting workflow column * Update tools * Add tool to run selected rows * Add mothership tools * adjust col width * Restore ff * fix drizzle migration * fix test |
||
|
Vikhyath Mondreti
|
af859cd508 |
feat(workflows): lock/duplicate improvements for workflows (#4387)
* feat(workflows): lock/duplicate improvements * fix duplicate var remap bug * address comments * remove dead vars * fix tests * address comments * code cleanup * address comments * address comments * minor change * remove dead code |
||
|
Vikhyath Mondreti
|
b8959eb20d |
improvement(repo): zod based client-server boundary (#4355)
* improvement(repo): centralized zod contracts (#4336) * improvement(repo): zod schema contracts * type checks * fix(notion): correctly register tool (#4337) * fix func blokc * more improvements * fix tests * type check * remove v3 refs * minor type improvements * address comments * update jira contract * remove validateJsonBody * improvement(repo): consolidation of boundary helpers + better unknown usage (#4352) * improvement(repo): consolidation of boundary helpers + better unknown usage * address comments * improve file transfer error messaging * fix docs listing schema drift * fix inocrrect type casting * address council comments * remove prefix |
||
|
Waleed
|
ccb5f1e690 | fix(db): revert statement_timeout startup options breaking pooled connections (#4284) | ||
|
Theodore Li
|
c22ac38ab0 | Set statement timeout of 90 seconds (#4276) | ||
|
|
5f0f0edd63 |
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |