mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
9299cee58dba31bfb9ef0ba2a8cc0a6418bc71df
191
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
28abbc94c6 |
perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) (#6151)
* perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) `turbopackFileSystemCacheForDev` has been `false` since #5408 — a landing-page homepage redesign whose description covers hero cards, feature-card aspect ratios, eyebrow chips and a voice-input button color, and never mentions Turbopack, caching, or dev performance. It was collateral, not a decision, and it overrode the Next default (true since v16.1). It is not the flag #6078/#6080 measured. That A/B was `...ForBuild` and its conclusion stands — the build cache is a 3.2x regression and stays off. The two flags look alike and are opposite decisions; both are now commented as such. Measured on `/workspace/[workspaceId]/w`, n=3 per arm, SIGINT between runs: cache OFF 31.4s / 30.1s / 31.9s RSS 9.0-9.8 GB cache ON 5.6s / 5.6s / 5.5s RSS 4.4-5.1 GB 5.4x faster restarts, ~2x less resident memory. Cold compile against an empty cache is unchanged (~32s either way) — the cache only pays back on restart, which is the loop that actually hurts. The cache is unbounded on disk: the abandoned one on this machine had reached 78 GB across 1,848 SST files, and a stale cache is slower to read back, so left alone it erodes the win it exists to provide. `prune-turbopack-cache.ts` runs on `predev` and drops it past a cap (default 20 GB, `SIM_TURBOPACK_CACHE_MAX_GB` to override); `bun run dev:cache:prune` forces it. It never blocks `next dev` on a maintenance failure. Adds a `dev-performance` skill recording the cost model, the reference numbers, and the benchmarking method — including that stopping the server with `kill -9` mid-cache-write discards the cache and makes this exact win read as no win. * improvement(dev): chain the cache prune into dev scripts instead of a predev hook Review read the root `bun run dev` path as bypassing the `predev` hook and so never capping the newly-enabled cache. Turbo does fire `pre*` hooks — verified live, the run prints the prune before `next dev` — but the concern is fair in that the guarantee rested on package-manager lifecycle semantics that are invisible at the call site. Chaining it explicitly removes the question entirely: every `dev` variant now runs `bun run dev:cache:cap && …`, which holds on any invocation path, is visible in the command itself, and drops the three duplicated `predev:*` entries for one shared script. Verified on both paths — direct `bun run dev` and root `turbo run dev`, the latter printing: sim:dev: $ bun run dev:cache:cap && next dev --port 3000 sim:dev: $ bun run ../../scripts/prune-turbopack-cache.ts * docs(dev): document cache-corruption recovery, the cost of enabling the cache Stress-tested the failure mode rather than assuming it: deliberately corrupting an SST block makes Turbopack abort with a FATAL panic — it does not self-heal. FATAL: An unexpected Turbopack error occurred. Cache corruption detected: checksum mismatch in block 4 of 00000221.sst `bun run dev:cache:prune` and restart fixes it; verified the canvas serves 200 again afterwards. Documented in the skill and in the script's header, since the symptom is a hard crash and the remedy is not guessable. This is the honest cost of turning the cache on. It is worth paying — a 5.4x faster restart against a rare, loud, single-command failure — but it should be written down rather than discovered. Worth distinguishing from the adjacent case: an ordinary hard kill does *not* corrupt the cache. Turbopack discards a partially-written cache and rebuilds it silently, which is exactly why a `kill -9`-based benchmark reads as "no cache win" (noted in the benchmarking section). * refactor(dev): drop the dev-performance skill, keep its findings at the code A whole skill was too much for what this is. The parts that are load-bearing — why the two lookalike cache flags are opposite decisions, the measured numbers, the corruption remedy, and the benchmarking trap — now live in the config and script they describe, where someone changing the flag actually reads them. The trap is the piece worth keeping: `next dev` compiles on demand so startup time is meaningless, and stopping the server with `kill -9` makes Turbopack discard a partially-written cache and rebuild silently — which reads as 'the cache does nothing' and is how this flag stayed wrong for a month. Dropped rather than relocated: generic advice that was not specific to this repo (antivirus, Docker-on-macOS, orphaned processes) and a measured no-op (`optimizePackageImports` for lucide-react changed nothing, 31.6s vs 31.7s). * docs(dev): record the measured cost and concurrency behaviour of cache pruning Stress-tested the maintenance path rather than assuming it is free. Cost: the size walk is ~30ms on a real cache and ~85ms at 2,000 files — under 2% of a 4.2s warm restart, and invisible against a cold one. It runs before every dev start, so it needed to be cheap; it is. Concurrency: pruning while a dev server is live (which happens when a second server is started from the same checkout) does not crash it. The running server keeps its in-memory state and kept serving HTTP 200 with zero panics. It does stop persisting for the rest of that session, so its next start is cold once — verified recovering at 23.4s then 4.5s. Worth writing down because the directory silently never reappears mid-session, which looks like a bug if you go looking. The cap is a backstop, not routine: a normal session sits at 1-2 GB against a 20 GB default. * fix(dev): cap every app's Turbopack cache, not just apps/sim `apps/docs` is a Next app too (`next dev --port 3001`) and overrides nothing, so it uses the Next default where the dev filesystem cache is on. It already had an uncapped 1.1 GB cache here, and the root `bun run dev` (`turbo run dev`) starts it — so a teammate using the documented command was accumulating a cache nothing would ever prune. The script now resolves its target from the working directory instead of hardcoding `apps/sim`, and each app chains its own cap. Per-app rather than one sweep on purpose: a single pass would let one app's dev start delete a cache another app is holding open, which costs that session its persistence. Verified both: `apps/sim` and `apps/docs` each report and cap their own 1.1 GB cache, and both dev servers start clean (`Ready in 299ms` / `229ms`, docs serving). * refactor(dev): drop dev:cache:prune in favour of the existing dev:clean `dev:cache:prune` duplicated `dev:clean`, which already existed in `apps/sim` and does strictly more (`rm -rf .next/dev/cache` covers the Turbopack cache plus the fetch and image caches). Two commands for one job is worse than one, and the docs pointed at the newer, narrower of the two. Removes it from both apps and gives `apps/docs` the `dev:clean` that `apps/sim` already had, so the recovery command is the same everywhere. `dev:cache:cap` stays — it is the chained step, used by more than one dev variant, and naming it keeps the relative script path out of each command. Verified `dev:clean` is a real remedy: corrupt a cache block, run it, restart — canvas serves 200 with no panic. Also corrects an overstatement. A damaged cache does not *always* abort Turbopack; whether it panics depends on whether the damaged region is read, so it is not reliably reproducible. Both notes now say "can abort" and give the same remedy either way. |
||
|
Waleedandmzxchandra
|
10bfb5d139 |
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on |
||
|
Theodore LiandClaude Opus 5
|
c5cc6ce26c |
feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137)
* feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vikhyath MondretiandClaude
|
7798e83489 |
feat(function): custom sandboxes (#6071)
* feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Theodore Li
|
32293f4b55 | feat(tables): typed predicate filter grammar, cursor pagination, and the v2 table surface (#6067) | ||
|
Vikhyath MondretiandClaude
|
11c0d3b75d |
feat(organizations): sweep a joiner's owned workspaces into the org on join, disclose it at accept, and add external workspace invites (#5918)
* feat(invites): explicit external members
* update docs
* fix(organizations): atomic admin workspace sweep, removal-impact status in dialog, and preview-unavailable disclosure
Review round 1: the v1 admin add-member now commits membership and the
workspace sweep in one transaction; the remove-member dialog holds confirm
while the credential-impact check loads and shows a caution when it fails;
a failed join preview flags joinPreviewUnavailable so the accept screen
falls back to a generic migration notice. Also aligns the invite test's
react-query mock and repairs two pre-existing docs type errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): close the concurrent-workspace escape in the join sweep
Personal workspace creation now serializes with organization joins on the
user's billing-identity lock and re-verifies membership inside its
transaction; both join paths (invite acceptance and the v1 admin add)
re-read the owned-workspace set under that lock after the member insert
and roll the whole join back when it diverged from the advisory-lock plan,
so a workspace created mid-join can never land outside the organization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): fail stale-grant member joins and re-resolve the creation race client-side
A member-role org acceptance whose grants all turned stale now rolls back
with workspace-not-found instead of stranding a workspace-less member, and
the workspace resolver treats the creation-vs-join 409 as a signal to
re-resolve (the user is authenticated with org workspaces) rather than
falling into the login path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): mirror the stale-grant gate in the join preview
The accept-screen preview now returns no-join for a member-role org
invite whose grants all left the stamped organization, matching the
acceptance-side rollback so the disclosure never promises a migration
that acceptance would refuse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(workspaces): survive the join race on lazy default creation and gate removal on live impact data
The workspace list GET now re-lists (returning the join sweep's
workspaces) when lazy default creation loses the race to an organization
join instead of failing with a 500, and the remove-member dialog gates
its confirm on isFetching so a background refetch can never let an admin
confirm against a stale credential-impact list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): surface the accept conflict message and refresh workspace caches post-accept
The accept route now carries the human-readable message alongside the
machine-readable error kind (the client prefers it for server-error), and
a successful accept invalidates workspace queries so the swept workspaces
appear immediately instead of after the stale window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(billing): sweep archived workspaces in Pro-to-Team conversion and org creation
Every attach call site now passes includeArchived so the archived escape
hatch is closed uniformly — join-attach, admin move, subscription-driven
org provisioning, and manual org creation all sweep archived personal
workspaces into the organization.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): reject acceptance when the sweep set differs from the disclosed set
The join preview now carries the workspace ids it disclosed, the accept
screen echoes them back as a disclosure token, and acceptance rolls back
with disclosure-outdated (409) whenever the set it would sweep no longer
matches — a workspace created after the preview rendered can never move
without the user seeing the refreshed notice first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): send the disclosure token for no-join previews too
A preview that predicted no join still tells the user nothing moves — the
empty disclosed set is now echoed on accept, so a join that becomes
possible between preview and accept (left another org, billing turned
usable, grants un-staled) conflicts with disclosure-outdated instead of
sweeping workspaces without a rendered notice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): gate all-stale member joins before disclosure and always refetch removal impact
The all-stale check for member-role org invites now runs before any
mutation and before the disclosure comparison, so an invite whose grants
all left the org fails with workspace-not-found instead of trapping
owners of personal workspaces in a disclosure-outdated retry loop. The
removal-impact query drops its stale window (staleTime 0): every dialog
open refetches while the confirm is held on isFetching.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(organizations): keep-external on org recovery and label archived move candidates
Org creation/recovery now uses the keep-external collaborator policy
(matching Pro-to-Team conversion) so different-org collaborators on
archived workspaces cannot abort it with a conflict, and the admin
workspace-move search and preflight expose an archived flag so internal
tooling can label archived targets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(invitations): guard the reverse disclosure direction
A will-join notice whose acceptance downgrades to no-join (stale
escalation denial, concurrent other-org membership) now fails with
disclosure-outdated instead of silently succeeding as an external grant —
the disclosure token binds the outcome in both directions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* address comments
* fix
* fix(invitations): one invite surface, coalesced grants, coherent seat model
Consolidate the two invite modals into a single surface and fix the
semantics the split had been hiding.
Invite flow:
- One InviteModal for all three entry points (workspace header, workspace
settings, organization settings), with a workspace multi-select and an
explicit Membership choice that states the seat consequence.
- Coalesce grants instead of 500ing. A partial unique index allows one
pending invitation per (email, organization), so inviting someone to a
second workspace raised a raw 23505. New workspaces now merge into the
pending invitation (with a retry for the concurrent-insert race) and the
invitee gets one link covering everything.
- External collaborators require their own paid plan, checked at invite
time and re-checked on accept, since the invitation lives for 7 days.
Imposed externality is exempt in both places: an invitee already in
another organization is forced external regardless of the inviter's
choice, so the plan gate must not apply to them.
- Every invitation must grant at least one workspace, so accepting always
lands somewhere. Enforced at creation for all roles.
- Revocation is grant-scoped. Since one invitation can span workspaces,
revoking from a workspace's member list withdraws only that grant and
cancels the invitation just when the last one goes. Whole-invitation
revocation now requires authority over all of it rather than admin on
any single granted workspace.
- The accept screen names every granted workspace and states whether the
invitee joins as a member, an admin, or an external collaborator, and
whether that uses a seat.
Seats:
- One rule for the pending-invitation predicate, seat capacity, and the
derived figures; the three counting sites now share it.
- Team seats are elastic (subscription.seats tracks the member count), so
treating that as a cap reported negative headroom on any outstanding
invite. Available seats are clamped and gates branch on whether the plan
actually has a fixed cap.
- POST /api/v1/admin/organizations/[id]/members could never succeed on
Team: it validated N members against N seats. It now skips the cap for
elastic plans, matching invitation acceptance, and reconciles seats
after a committed add.
Also surfaces the External label on the workspace Teammates list, which
already received the flag and dropped it, and removes dead code: the
grantless organization-invite route and contract, three unreferenced
invitation helpers, and the unreachable
ensureUserInOrganization/addUserToOrganization/validateMembershipAddition
cluster.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): close the two accept-disclosure gaps Bugbot found
Membership notice ignored the join preview. `buildMembershipNotice` keyed off
the invitation's sent `membershipIntent`, but acceptance resolves an internal
invite to external when the invitee already belongs to another organization, or
when the granted workspace changed organizations after the invite went out. The
screen therefore promised "you'll join as a member, which uses one of their
seats" to people who would neither join nor consume a seat. It now keys on the
preview's `willJoinOrganization` — the same signal the migration notice already
used — and falls back to the sent intent only when no preview could be computed.
In-app accept skipped the disclosure entirely. `useAcceptMyInvitation` posted an
empty body, so `disclosedWorkspaceIds` was absent and the server's consent guard
was skipped, and the pending-invitations modal never showed which owned
workspaces would move. Accepting from the workspace switcher (including the
desktop path) could silently sweep personal workspaces into the organization,
bypassing the consent model this PR adds on /invite. The list endpoint now
returns each invitation's join preview, the modal renders the same
membership/migration disclosure as /invite, and accept echoes
`disclosedWorkspaceIds` so the guard applies on both paths.
Both notices moved into lib/invitations/disclosure-copy.ts and are consumed by
/invite and the modal, so the two accept surfaces cannot drift into disclosing
different outcomes for the same invitation — which is how this gap arose.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): carry the membership outcome in the disclosure token
Empty disclosure skipped membership consent. The token was only the
workspace-id list, so a no-join preview and a will-join preview for someone who
owns nothing both echoed `[]`. Neither guard could tell those apart: the forward
check compares sweep sets, and the reverse check required a non-empty disclosed
set. An invitee who left their other organization between preview and accept
would therefore be silently made a seat-consuming member after being told they
would stay external, and the mirror case could silently demote a promised join.
The accept body now also carries `disclosedWillJoinOrganization`, compared
against the resolved outcome before any write, so consent covers the membership
decision and not just the migration. Both accept surfaces send it.
This was widened by the previous commit: keying the membership notice on the
preview made the screen promise a join outcome the token never verified.
In-app accept errors lacked copy. `getInvitationErrorMessage` omitted
`external-requires-paid-plan`, `disclosure-outdated`, and
`workspace-not-found`, so those failures fell through to the generic "may have
expired" fallback. `disclosure-outdated` became newly reachable in-app the moment
that path started sending the token, so the gap arrived with the fix for it.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): compare the join disclosure against new-membership creation
The membership consent guard compared the disclosed outcome against
`shouldJoinOrganization`, which stays true for an invitee who already belongs to
the target organization — the invitation's intent is still internal. The join
preview reports no-join for exactly that case, because nothing changes for them.
Every such acceptance therefore failed `disclosure-outdated`, and the retry
re-rendered the same preview, so the invitation became permanently unacceptable.
The guard now compares against whether acceptance creates a NEW membership
(`shouldJoinOrganization && !alreadyMemberOfTargetOrganization`), which is what
the disclosure actually promises and what the preview reports. The
already-a-member predicate is hoisted and shared with the join block below so
the guard and the billing path cannot disagree about it.
Regression test asserts a pre-existing member accepts with a no-join disclosure;
it fails with `disclosure-outdated` against the previous comparison.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(invitations): tell an existing member their standing is unchanged
The join preview reported the same no-join shape for two different outcomes: an
external collaborator, and an invitee who already belongs to the organization
acceptance lands in. `buildMembershipNotice` rendered both as "you'll join as an
external collaborator ... everything you own stays yours", which is wrong for an
existing member — they stay an internal member and simply gain the granted
workspaces.
The preview now reports `alreadyMemberOfOrganization` for that case (a
membership in a DIFFERENCE organization is still the external path, since
acceptance downgrades), and the notice states that standing is unchanged. This is
the same conflation behind the accept loop fixed in
|
||
|
Theodore Li
|
985f9e6172 |
feat(tables): saved views with filter, sort, and column presets (#5961)
* feat(tables): saved views with filter, sort, and column presets * fix(tables): views own column layout, preserve deep-linked sort, seed update cache * fix(tables): merge view layout writes server-side, keep layout on save-from-All * fix(tables): send view saves as a merge patch so concurrent writes can't clobber * fix(tables): persist explicit All selection, prune view state for deleted columns * fix(tables): key-order-stable dirty check, reset layout when switching to All * fix(tables): return 404 when patching a missing view * improvement(tables): order the bar filter, sort, columns and drop the hidden count * fix(tables): record newly created columns in the active view's order * fix(tables): don't write layout as a reader, clear dead sort, reset dead view id * feat(tables): add New view to the views menu, starting from All * chore(api): bump route-count baseline to 981 after staging merge * fix(tables): guard default demotion, use the applied filter for dirty/save * fix(tables): clear state when the active view is deleted externally * revert(tables): drop the ineffective rows-query gate for view resolution * feat(tables): enable saved views in the embedded mothership table * fix(tables): resolve the views flag on the chat route too * improvement(tables): right-align the embedded run/stop control * fix(tables): live layout on new views, prune stored config, order cache writes * fix(tables): live layout on save-as-view, reset it per view, ignore inherited param when embedded * fix(tables): route undo/redo column-layout writes to the active view * fix(tables): scope layout undo to the view that recorded it Layout is view-owned, but UndoEntry carried no owner, so the persistence sink — rebound every render to the active view — decided the target at undo time rather than at record time. Reordering in view A then undoing from view B wrote A's column order into B and left A unchanged. Entries are now stamped with the active view id, and the three layout-bearing action types (create-column, delete-column, reorder-columns) are pruned from both stacks when the active view changes. Row and schema actions are table-scoped and survive the switch untouched. Inert when views are disabled: the id is always null, so nothing is ever pruned. * fix(tables): send only the layout keys an action changes Pin, reorder, and insert-column each shipped a full columnWidths snapshot they never modified. Both sinks merge at top-level key granularity, so an earlier-issued patch landing after a concurrent resize replaced the newer width map with its stale copy. Resize, auto-resize, and delete-column still send columnWidths — those genuinely change it. * fix(tables): don't strand layout writes while views load, keep schema undos across views Two more instances of layout being written without a known owner. The sink was left unbound until a view resolved, so a resize/reorder/pin (or the column-append effect) during the views fetch fell through to the table's shared metadata — corrupting All for a table about to adopt a view, and losing the edit to the re-seed. The sink is now bound while the query is in flight and suppresses the write. An error counts as settled, so a failed views fetch falls back to All instead of suppressing layout writes for the session. Pruning was also too broad: create-column and delete-column are table-scoped schema ops that merely have a layout side-effect, so dropping them on a view switch made a deleted column unrecoverable. Only reorder-columns is purely layout and still prunes; the other two survive and have just their layout half suppressed at replay when the recorded view isn't active. * fix(tables): flush layout buffered during load when the owner settles to All Suppressing the write while the views query was in flight stopped All from being corrupted, but nothing resolved the buffer afterwards, so a resize during load looked applied and vanished on refresh. Settling on All re-seeds nothing (viewLayoutKey never changed), so the gesture is still on screen and is now persisted to shared metadata. Adopting a view re-seeds the grid from that view and has already replaced the gesture on screen, so the buffer is dropped to match. * fix(tables): resolve undo layout ownership at write time, not dispatch time The layout writes for column create/delete happen in mutation success callbacks, and persistLayoutRef is rebound every render. Resolving entryOwnsLayout up front meant the guard could still hold from before a view switch while the sink it guarded already pointed at the destination, writing the recorded view's layout into whichever view was now active. Now a function, so the guard and the sink are read at the same moment: switch away mid-mutation and the schema half still lands while the layout half is dropped, matching the rule that undo only ever affects the view on screen. * fix(tables): read layout from the grid instead of mirroring it The wrapper kept two shadow copies of the grid's column layout — liveLayoutRef and pendingLayoutRef — and all three of this round's findings were that mirror going stale: - liveLayoutRef was only cleared on activeView.id change, so widths buffered before the views query settled survived into All, where layout writes bypass the mirror entirely. Both create paths spread it last, so a saved view stored those snapped-back widths. - currentViewConfig memoized a spread of that ref, and mutating a ref doesn't re-run a memo, so Save as view sent the pre-gesture layout. - The flush effect keyed on activeView, but adoption writes the view id through the URL, so for one render the query had settled while activeView was still null — flushing to All in exactly the case that had to drop. The grid owns this state, so it now publishes a reader through a sink ref and the wrapper asks at the moment it needs a value. Nothing to keep in sync, so nothing to go stale. Only whether an unowned change happened is tracked, and the resolve effect — which is what actually picks the owner — decides to persist or drop. * fix(tables): flush unowned layout when the views fetch fails The resolve effect is the only caller of resolvePendingLayout and gated on isSuccess, which never becomes true on a query error — so layout touched during the load window was never persisted on the error path, even though the table had already settled to All and later writes worked. The error branch now flushes to shared metadata, matching the rest of the error path's fall-back-to-All behavior. * fix(tables): gate the on-screen layout restore by ownership, not just the persist The undo success callbacks applied the recorded view's order/widths/pinning to the grid unconditionally and only gated the PATCH, so switching views before a column create/delete mutation resolved left the destination displaying the origin view's layout until switched away and back. Each callback now checks ownership where its layout work begins: three are purely layout and return at the top; delete-column undo restores cell data first (row data, runs everywhere) and gates only the layout block below it. In a non-owning view the restored column still appears via the grid's append effect — at the end, leaving that view's layout untouched. * fix(tables): route all layout writes through one owner-aware sink, reconcile order at seed Two holes closed: The sink binding toggled on activeView, leaving a render-frame gap after the views query settled but before the resolve effect adopted a default — writes in that gap fell through to shared metadata. The sink is now always bound while views are enabled and handlePersistLayout is the single router, reading the owner at call time: unresolved buffers, a view patches the view, All writes metadata. The error branch stamps the owner so post-error writes stop buffering. The append effect only fires when the schema changes, so a column that arrived while another source owned the layout rendered via the displayColumns fallback but was never written into the adopted owner's stored order until a drag happened to heal it. Seeding now reconciles the incoming order against the schema and persists the appended tail through the current sink. * fix(tables): capture the layout owner when a schema action is dispatched Insert-column and the delete chain persist layout from mutation callbacks through updateMetadataRef, which always targets the current sink — so a view switch mid-flight wrote the origin view's order/widths/pins into the destination. Undo got this guard already; the live paths never did. Both now capture viewLayoutKey at dispatch and compare at the callback. On a mismatch the layout work is skipped: the destination re-seeded its own layout, the new column lands there via the append effect when the refetch arrives, and the deleted column's dangling keys are pruned on read. pushUndo also takes the captured owner as an override — stamped at callback time it would record the destination, letting a later undo apply the origin's layout to it. * fix(tables): resolve views on list availability, not query success A failed background refetch flips isError while the cached list stays usable, and every view mutation invalidates the views query — so one blip made the resolve effect treat views as terminally failed and stop applying switches until the next successful refetch. The axis is now whether a list exists: error with no list ever fetched settles to All; error with a cached list resolves normally against the cache; owner is unknown only while the initial fetch is in flight. * ci: raise Build App timeout to 25 minutes The build outgrew the 15-minute cap over the last two days of merges: 10m02 ( |
||
|
mzxchandraandWaleed Latif
|
6ff6255900 |
feat(outlook): add Microsoft Graph calendar operations (#6041)
* feat(outlook): add Calendars.ReadWrite and MailboxSettings.Read scopes
Extend the shared outlook OAuth service with delegated Graph calendar scopes so
one Microsoft connection covers mail and calendar. Existing connected users must
reconnect to be granted the new scopes (noted in a code comment). Adds human-readable
scope descriptions and test assertions.
* feat(outlook): add Microsoft Graph calendar tools
Six calendar operations against graph.microsoft.com/v1.0: list events (calendarView
with nextLink paging), get, create, update (partial PATCH), delete, and respond to
invites. Shared calendar-utils handles Graph's offset-less dateTime+timeZone shape,
attendee normalization, and event flattening. All tools carry the Microsoft Graph
error extractor and 429/backoff retry (honors Retry-After) for the mailbox concurrency
limit. Registered in the tool registry and barrel.
* feat(outlook): surface calendar operations in the Outlook block
Add six calendar operations to the Outlook block operation dropdown with their
conditional subBlocks, tool wiring, inputs, and event-shaped outputs. Mail operations
are unchanged and backward compatible.
* fix(outlook): address calendar review findings
- Validate list-events pageToken origin with assertGraphNextPageUrl (matches the
onedrive/microsoft_ad Graph paging guard) so a workflow-supplied URL can't receive
the Outlook bearer token.
- All-day create/update now normalize both bounds to midnight and force an exclusive
end day (buildAllDayRange), instead of sending a zero-length same-midnight window
that Graph rejects.
- Drop the stale 'suggest meeting times' claim from the block longDescription.
- Remove the unused MailboxSettings.Read scope (least privilege; no tool reads it).
* feat(outlook): add calendar picker and harden calendar tools
Validation pass over the new Microsoft Graph calendar operations against the
v1.0 API docs, plus the calendar selection the tools were missing.
- Add an `outlook.calendars` picker (GET /me/calendars) with basic selector +
advanced manual ID, wired through the `calendarId` canonical param. List and
create now target `/me/calendars/{id}/...`; get/update/delete/respond keep
using `/me/events/{id}` since event IDs are mailbox-unique.
- Fix all-day update rejecting when only one bound is supplied — both bounds are
now normalized to midnight with an exclusive end day.
- Fix "Send Response to Organizer" reading as OFF while Graph's default is to
notify; it is now a dropdown defaulting to Yes, and the param is always sent.
- Add timestamp wandConfig to the four calendar datetime fields and a list
wandConfig to attendees.
- Centralize Graph URL construction in calendar-utils; guard maxResults against
non-numeric input and trim the calendarView time-window bounds.
- Add three calendar templates and four calendar skills to OutlookBlockMeta.
- Regenerate integration docs.
* fix(outlook): scope online-meeting comments to what Graph documents
onlineMeetingProvider is optional and defaults to unknown; the docs state that
setting isOnlineMeeting alone initializes onlineMeeting. They do not document
Graph substituting the calendar's defaultOnlineMeetingProvider, so the comments
now claim only that, plus the real reason not to pin teamsForBusiness (mailboxes
that disallow it via allowedOnlineMeetingProviders).
* fix(outlook): allow calendar paging without re-supplying the time window
Cursor Bugbot round: startDateTime/endDateTime were required tool params, so a
paging call carrying only pageToken failed validateToolParameters before the
request was built — even though the url builder short-circuits on pageToken and
ignores both bounds. Relax them to optional and enforce the real invariant
(pageToken OR both bounds) in the url builder, matching
tools/sharepoint/list_sites.ts. Block subblocks stay required, so the normal
editor flow is unchanged.
Also correct the calendar_respond comment: Graph documents exactly two 400
conditions for accept/decline, both on proposedNewTime, which we never send.
A non-empty comment alongside sendResponse=false is valid.
* feat(outlook): add Calendars.ReadWrite.Shared for shared calendars
The calendar picker lists /me/calendars, which can include calendars other
users have shared with or delegated to the account. Calendars.ReadWrite covers
only the user's own calendars, so selecting a shared team calendar would 403 on
both read and write.
Calendars.ReadWrite is kept alongside it, not replaced: Graph documents it as
the sole accepted permission for creating and updating events and for
accept/tentativelyAccept/decline (all list "Higher: Not available"), so the
.Shared scope does not subsume it.
Added now rather than later because this PR already forces existing Outlook
users to re-consent for Calendars.ReadWrite; deferring would cost them a second
reconnect.
* fix(outlook): treat date-only bounds as all-day and guard partial all-day updates
Cursor round 2:
- Date-only bounds no longer produce a zero-length window. The param docs invited
a date like 2025-06-03 for an all-day event, but buildAllDayRange only ran when
isAllDay was explicitly true, so date-only input built a 00:00->00:00 timed
window that Graph rejects. A date-only bound carries no time, so the only
coherent reading is all-day; create/update now promote on that shape and the
descriptions state it.
- Converting an event to all-day with no bounds now fails with an actionable
message instead of a Graph 400. Graph requires all-day events to have midnight
start and end in the same zone, and those cannot be derived from a partial
PATCH against an event whose existing bounds are timed.
* docs(outlook): note that the calendar window fields are ignored when paging
* fix(outlook): don't promote a lone date-only bound to all-day on update
Regression from the previous round's date-only promotion. A single date-only
startDateTime or endDateTime satisfied "all provided bounds are date-only", so
the tool promoted to all-day and derived the missing side from the supplied one
— turning a partial reschedule of a timed or multi-day event into a one-day
all-day event and dropping the original other bound.
Implicit promotion now requires BOTH bounds to be date-only, matching
calendar_create_event. A lone date-only bound is ambiguous (convert to all-day,
or just move that edge?) and a PATCH cannot read the event's existing bounds to
disambiguate, so it stays on the timed path and leaves the other side untouched.
Deriving a missing bound remains allowed when isAllDay is set explicitly, since
that is stated intent rather than a guess.
Also pins the explicit-isAllDay-false + date-only override with a regression
test: the block always sends isAllDay for create, so an untouched switch arrives
as false, and the data shape has to win or the fix would be unreachable from the
UI.
* revert(outlook): drop Calendars.ReadWrite.Shared from the outlook provider
Reverts the scope I added two rounds ago. It was the wrong call.
This provider is shared by work/school AND personal Outlook accounts, and the
.Shared calendar scopes are not confirmed supported for personal Microsoft
accounts. Requesting one risks failing consent for personal users — which would
take mail access down with it, breaking functionality that works today. The PR
already documents this exact reasoning as why findMeetingTimes was excluded, and
that decision was made against a live personal mailbox.
The evidence I added it on was a summarized read of the permissions reference
claiming MSA support; a targeted follow-up could not confirm it for
Calendars.ReadWrite.Shared specifically. Given the asymmetry — broken consent
for all personal users vs. a shared-calendar feature gap — least privilege wins.
Calendar operations therefore target calendars the account owns. The calendarId
param descriptions now say a calendar shared by another user may return 403, and
the scope list carries a comment explaining why .Shared must not be re-added.
* fix(outlook): make retried event creates duplicate-safe via transactionId
The retry config opts POSTs in via retryIdempotentOnly:false, but the executor's
isRetryableFailure covers 429 AND 500-599 — not just the throttle the comment
justified. A 5xx returned after Graph had already committed a create would be
retried and produce a duplicate calendar event.
create_event now sends a transactionId, which Graph documents for exactly this:
it discards a repeat POST carrying an id it has already seen. The request body is
built once per execution (formatRequestParams runs before the attempt loop), so
the id is stable across retries of a call and unique between calls.
The retry comment now describes what actually retries and why each non-idempotent
method is safe: PATCH replays the same partial body as a no-op, and respond is
state-idempotent though a post-commit retry can send the organizer a second
notification — accepted deliberately, since Graph exposes no transactionId for
accept/decline and failing outright under throttling is worse.
* fix(outlook): tighten date-only detection and align online-meeting copy
Final validation pass over the calendar integration.
- isDateOnly matched "contains no T", so a space-separated datetime
(2025-06-03 10:00) counted as date-only: the time was discarded and the value
built as '2025-06-03 10:00T00:00:00', which Graph rejects. It now matches
YYYY-MM-DD strictly, and buildGraphEventDateTime normalizes the space form to
ISO rather than mangling it, so a natural input works instead of 400ing.
- The isOnlineMeeting param descriptions still claimed Graph 'uses the mailbox
default provider' — the same unverified mechanism already removed from the code
comments. They now state only what the docs and the author's live testing
support: the join URL depends on the providers the mailbox allows, and stays
null on personal accounts.
- Adds blocks/blocks/outlook.test.ts following the repo's per-block test
convention: every calendar operation resolves to a registered tool in
tools.access, supplies all required tool params, emits no params the tool
cannot accept, maps one-to-one onto the calendar tools, and the calendarId
canonical group and sendResponse default are pinned.
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
|
||
|
|
1d64b92b41 |
feat(desktop): desktop app (#5998)
* top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai> |
||
|
Vikhyath Mondreti
|
c809845b99 |
improvement(self-host): enterprise features enabling (#6028)
* improvement(self-host): enterprise features enabling * chore(helm): bump chart to 1.3.0 for the enterprise self-host values values.yaml gained the ENTERPRISE_ENABLED switch and INSTANCE_ORG_* keys, and the feature-flag envDefaults moved from "false" to empty so the master switch can resolve them. Additive and backward compatible, so a minor bump. * fix(self-host): address review findings on instance org and org delete Drop the per-process instance-org id cache. It went stale once the organization was deleted through the Admin API, and clearing it from the delete handler would only heal the replica that served that request. The lookup runs on the signup path against a single-row table, so re-reading costs nothing and keeps every replica self-correcting. Scope the org-delete subscription conflict to entitled statuses. Matching any row regardless of status let a canceled subscription — which bills nobody — permanently block deletion. * fix(admin): block org delete on any live subscription, not just entitled ones ENTITLED_SUBSCRIPTION_STATUSES excludes trialing, so a trial — which grants no entitlement but is a live Stripe subscription that will convert — slipped past the delete guard and could be stranded against a removed organization id. Adds TERMINAL_SUBSCRIPTION_STATUSES and inverts the predicate: block unless the row is finished. Expressed as the terminal set so a status Stripe adds later defaults to blocking, which is the safe direction for a destructive operation. * fix(self-host): resolve SSO and access-control in the UI, not the raw env var Nine client consumers still read NEXT_PUBLIC_SSO_ENABLED / NEXT_PUBLIC_ACCESS_CONTROL_ENABLED directly while the server gates and settings nav had moved to the resolver. With only ENTERPRISE_ENABLED set that produced dead ends: the SSO settings section appeared but ssoClient() was never registered and no login button rendered, and the Access Control section appeared but its page reported "not entitled". Points every consumer at isSsoEnabled / isAccessControlEnabled so visibility and capability come from one place. * fix(admin): validate retention workspace targets on the Admin API too retentionOverrides and per-workspace PII rules both name a workspace, and neither field is a foreign key. The settings UI rejected ids belonging to another organization; the Admin API did not, so the two paths could persist different data for the same org. Extracts the check as getForeignWorkspaceTargetsReason and points both routes at it, so they cannot drift apart again. * fix(self-host): close three review findings on admin routes and cleanup Make org delete atomic. detachOrganizationWorkspaces committed on its own, so a failed delete left workspaces detached and re-billed while the organization, its members, and its settings survived. Adds a Tx variant so both commit together. Gate the admin session-policy PATCH on entitlement, matching the settings UI. Without it the stored policy was inert — getSessionPolicy resolves to no-op when the feature is off, so the one eager clamp would be undone on the next refresh. Stop emitting plan-wide housekeeping when billing is off. It is keyed to the hosted free-tier 30-day window, the same default the per-workspace pass deliberately refuses to apply off-hosted. * fix(admin): gate whitelabel on entitlement and emit detach audits post-commit The Admin whitelabel PATCH skipped the entitlement check the settings UI runs, so an admin key could set branding the product had not granted the org. detachOrganizationWorkspacesTx also wrote its audit rows inside the caller's transaction, contradicting its own doc comment — a rolled-back delete would have left audit history describing detachments that never happened. It now returns the rows and callers emit them after commit. * fix(self-host): refuse instance-org resolution when the slug is ambiguous organization.slug has no unique constraint, and the lookup took the first of however many matched. The choice is unordered, so two replicas could resolve different organizations and split new signups between them. Resolution is now three-state. Ambiguity is distinct from absence, so it both declines to adopt an arbitrary organization and declines to provision another one on top of the duplicates. |
||
|
Waleed
|
cb3611bad2 |
feat(folders): add resource pinning and generalize the folders contract (#6014)
- Adds per-user pinning for workflows, files, knowledge bases, and tables: new `pinned_item` table, `/api/pinned-items` routes, React Query hooks, and a shared `PinButton` wired into Tables, Knowledge, and Files with pinned-first ordering - Adds the generic `folder` table with an idempotent, replay-safe, collision-aware backfill from `workflow_folder` and `workspace_file_folders`; no cutover yet, folder reads still go through a documented legacy adapter - Moves `folderSchema` to the generic vocabulary (`resourceType`, `deletedAt`) and drops the unused `color`/`isExpanded` |
||
|
Waleed
|
ca13cc9c13 |
feat(api): add workflow export and import endpoints to the public v1 API (#5999)
* feat(api): add workflow export and import endpoints to the public v1 API
Adds GET /api/v1/workflows/[id]/export and POST /api/v1/workflows/import.
The export envelope is accepted verbatim by import, so workflows round-trip
between workspaces over the public API.
Unlike the admin export, the public export is secret-sanitized: stored
credentials and password fields are redacted while {{ENV_VAR}} references
and block positions are preserved. Import regenerates block, edge, loop and
parallel ids and de-duplicates the workflow name against the target folder.
Also moves parseWorkflowVariables out of the admin types module into
lib/workflows/variables/parse.ts so the public route does not import from
the admin namespace.
* fix(api): make workflow import atomic and clarify what export redacts
Writes the imported graph and its variables in a single transaction and
deletes the shell workflow row on any failure, so a caller that receives an
error is never left with a partially imported workflow. Previously a throw
from the variables update returned 500 while leaving the workflow behind
with an empty variables map.
Also narrows the export route's sanitization claim: workflow variables are
emitted as stored, matching GET /api/v1/workflows/[id] and the in-app
export. They are plaintext configuration readable at the same permission
level this route requires; secrets belong in environment variables, which
travel as unresolved references.
* fix(api): close import defects found in audit and share one write pipeline
Security:
- Escape block names before interpolating them into a RegExp in
updateValueReferences. Names reach it straight from imported workflow JSON
and normalizeWorkflowBlockName preserves regex metacharacters, so a name
like `a*a*a*a*b` compiled to a catastrophically backtracking pattern. A
sub-kilobyte body blocked the event loop for 50s and grew exponentially.
Also skip rename-to-itself, which is the entire map on the import path, so
the scan no longer runs at all there.
- Validate folder ownership before folder lock state, so a locked folder in
another workspace can no longer be distinguished from a missing one.
Correctness:
- Gate the imported graph on workflowStateSchema, the same schema the
canonical PUT /api/workflows/[id]/state path enforces. Without it a valid
201 could persist a block field of the wrong type, which then threw on
every subsequent read and left a workflow nothing could open.
- Guard the compensating delete so a failed rollback logs the orphaned id
instead of vanishing into a generic 500.
- Validate variable `type` against the enum and build the record on a
null-prototype object, so a `__proto__` key no longer silently drops the
variable.
- Bound payload-derived names and descriptions to the same limits the
contract declares for the explicit overrides.
- Return the description as stored rather than coercing '' to null, matching
GET /api/v1/workflows/[id].
Shared code, so the two write paths cannot drift:
- Extract prepareWorkflowStateForPersistence and use it from both
PUT /api/workflows/[id]/state and the v1 import route: agent-tool
sanitization, block backfill, dangling-edge removal, and loop/parallel
recomputation now have one implementation.
- Persist inline custom tools on import, which the canonical path already did.
- Move variable normalization into lib/workflows/variables and repoint the
admin importer at it, removing the last duplicate.
Docs:
- OpenAPI: oneOf -> anyOf on the import body. WorkflowExport matches any
object, so every valid object payload matched two branches and failed
validation under any spec-driven validator. Document 423 and the loss of
workspace-scoped bindings on export.
Tests: prepare-state unit tests and a real export -> import round trip with
no mocks of the sanitizer or parser, covering loop/parallel children and the
regex-metacharacter payload.
* fix(api): cap import names inside the bound and align the three import paths
- `truncate` appends its suffix after slicing, so capping at the contract
limit produced 203/2003-character values — past the very bound the cap
exists to enforce, and into the headroom reserved for dedup suffixes.
Reserve the ellipsis inside the limit.
- Match `extractWorkflowName`'s candidate order (state.metadata.name before
workflow.name) and trim, so the v1 API and the in-app importer resolve the
same name for the same payload. Previously a hand-authored payload carrying
both could yield two different names.
- Run the admin importer through prepareWorkflowStateForPersistence too. It
was writing raw parsed state, so a dangling edge tripped the workflow_edges
foreign key and a block missing its backfilled columns could land
unopenable — the same class this PR just closed on the v1 path.
|
||
|
Waleed
|
66ac015c4c |
fix(library): generate every post cover from one template (#5980)
* fix(library): generate every post cover from one template Three posts shipped an `ogImage` pointing at a file that was never committed, so the library index rendered broken images and their `og:image`, JSON-LD, and sitemap entries all 404'd. Several others were authored without the brand font loaded or with the title clipping off the bottom edge. Covers were hand-made per post with no generator, which is why they drifted. Adds `bun run library:covers`, rendering each cover from the post's frontmatter title using the reference template already encoded in the docs OG route, and regenerates all 20 so the grid is uniform. Line widths come from the font's real advance metrics rather than an average-glyph-width estimate: the template joins words with non-breaking spaces to dodge a Satori space-measurement bug, which leaves hyphens as the only fallback break points, so an under-measured line breaks mid-compound. Also drops six orphaned `cover.png` sources left over from the JPEG compression pass in #5528. * fix(library): re-render covers every run and add a sync check Covers are derived artifacts, so skipping outputs that already exist left an image showing the old title after a post's frontmatter `title` changed. Every run now re-renders from scratch; rendering is deterministic, so an unchanged title re-encodes to identical bytes and a full run stays a no-op in git. Replaces `--force` (now the default) with `--check`, which renders in memory and compares against the committed bytes without writing, so CI can catch both a stale cover and the missing-cover case that caused the original breakage. * fix(library): compare decoded pixels in the cover sync check Byte-equality on the mozjpeg output assumed portable encoder bytes. libvips/mozjpeg does not guarantee that across OS and CPU, so identical input can encode differently on a contributor's machine or a Linux CI runner and fail the check for no real reason — exactly where the check was meant to run. Decodes both images to greyscale and compares mean absolute difference instead, which discards encoder variance while still testing what the check is about. Measured on this cover set: re-encoding an identical render with a deliberately different encoder moves it ~0.26, a one-word title change moves it ~12; the threshold of 2 sits between them with ~8x margin. * fix(library): count redrawn pixels in the cover sync check Averaging the difference diluted a local edit across all 810,000 pixels. Changing a title's "2026" to "2027" moved the mean by 0.42 — under the tolerance that absorbed encoder noise — so the check passed a cover still showing the old year. Counts pixels that moved more than 48 greyscale levels instead. Measured on this cover set, that one-character edit redraws 2,559 pixels while three deliberately different encodes of an identical render (quality 60/70 without mozjpeg, quality 95 with) redraw none, so the count separates real drift from encoder variance in both directions. * fix(library): parse frontmatter with gray-matter and split oversized tokens Two issues in the cover generator, neither reachable from a current title. The hand-rolled frontmatter regex could disagree with `gray-matter`, which is what renders the page and its `og:title`. On a double-quoted escape or a block scalar the cover would have rendered a title the page never shows, with `--check` calling it in sync. Uses `gray-matter` directly so there is one parser. `wrapTitleLines` only breaks between space-separated words, so a token wider than the title box on its own stayed on an overflowing line, and the non-breaking spaces left Satori no recourse but to break it at a hyphen — the mid-compound break this layout exists to prevent. Oversized tokens now split here, at hyphens first and per-character only for something like a URL, and a font size is accepted only if every line measures within the box. All 20 covers re-render byte-identically, so neither change alters current output. |
||
|
Waleed
|
6d48444525 |
fix(docs): render native block icons instead of the two-letter fallback (#5981)
* fix(docs): render native block icons instead of the two-letter fallback
The Table and Logs pages (and every other native resource block) showed a
two-letter text fallback because the generated icon map never contained them.
Four separate causes in scripts/generate-docs.ts:
- The icon-map allowlist had drifted behind NATIVE_RESOURCE_BLOCK_TYPES, the
set the docs writer uses. Key the exception off that set so the map cannot
fall behind the pages that consume it.
- extractIconNameFromContent only matched identifiers ending in `Icon`, so
Logs (`icon: Library`) resolved to nothing. Match any identifier, excluding
bare JS literals.
- The map imported everything from `@/components/icons`, so an icon sourced
from `@sim/emcn/icons` could not resolve. Imports are now grouped by the
module each icon is actually imported from.
- Trigger-only pages (slack_app, twilio) and hand-written pages (a2a) had no
entry at all. Seed provider icons from the trigger definitions.
Also fixes three regen bugs found while verifying the output:
- A comment reading "this becomes `hideFromToolbar: true`" in slack.ts was
matched as the property itself, so a clean regen dropped Slack from the
integrations catalog and reduced slack.mdx to a 29-line stub. Property
probes now run against comment-stripped source.
- 16 hand-written *-service-account guides were unregistered, so the stale-doc
cleanup deleted them on every regen. Registered them, and cleanup now refuses
to delete any page holding MANUAL-CONTENT (this also restores the intros on
file.mdx and twilio.mdx).
- Trigger outputs referenced as a constant (`outputs: SLACK_TRIGGER_OUTPUTS`)
resolved to nothing, dropping whole Output tables. Constants and sibling
modules now resolve, which also restores 319 lines on clickup.mdx.
Removes the language selector from the docs navbar.
Regenerated docs are included; remaining content deltas are tool-definition
drift since the last regen.
* improvement(docs): drop the preview-gated slack_app page, document managed_agent
- slack_oauth is reachable only through the preview-gated slack_v2 block, so
documenting it published an unreleased surface under its own slack_app page.
Triggers whose every hosting block sets `preview: true` are now excluded from
the docs and the icon map. Triggers no block claims are untouched, so
standalone webhook providers keep their pages.
- Adds the MANUAL-CONTENT intro to managed_agent.mdx, matching the other
integration pages. Verified it survives a regen.
* fix(docs): stop truncating quoted descriptions, tighten the cleanup guard
Review findings from round 1.
- parseSubBlockObject read string properties with a single `['"]…[^'"]+…['"]`
character class, which ends the match at the first quote of either kind. Any
description holding an apostrophe inside a double-quoted string was cut
mid-word ("Your app", "Found in your Zoom app"). Matches the opening quote to
its own closing quote now, reusing the alternation the tool-description
extractor already used. Restores full text across calendly, gmail,
google_sheets, hubspot, intercom, whatsapp, and zoom.
- The stale-doc cleanup guard tested for a bare `MANUAL-CONTENT-START`
substring, so a stray or unterminated marker would pin a stale page that has
nothing recoverable. It now gates on what extractManualContent actually
returns.
|
||
|
Bill LeoutsakosandBill Leoutsakos
|
bd61603701 |
feat(tiktok): unhide integration (#5978)
* feat: unhide TikTok integration * test: remove TikTok visibility assertion --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> |
||
|
Theodore Li
|
1a4bfe4c59 |
fix(setup,compose): bundle Redis, fix socket reconnect, and harden the setup wizard (#5964)
* feat(compose,setup): bundle Redis, always configure it, fix lifecycle detection
Compose shipped no redis service at all — REDIS_URL was ${REDIS_URL:-} in both app and realtime, so every self-hosted stack ran without it. Storage silently falls back to PostgreSQL, but the pub/sub channels (live Chat task-status, table events) have no fallback, so live updates never arrived.
- compose (prod + local): add a redis:7-alpine service with a healthcheck, default REDIS_URL to redis://redis:6379, and make app/realtime depend on it being healthy. Not published to the host — only the containers need it, and binding 6379 would collide with a local Redis. An external REDIS_URL in root .env still overrides. Deliberately not written into root .env: doctor pings REDIS_URL from the host, and a compose-internal hostname would fail that probe the same way DATABASE_URL would.
- dev mode: configure Redis in quick too. Quick uses a new non-interactive ensureRedis (adopt whatever answers, else start the managed container, warn only if Docker is unavailable); custom keeps the ladder, with corrected copy — the old prompt claimed Redis was only for multi-replica.
- lifecycle: detect compose stacks via 'docker compose ls' instead of probing '-f <file> ps' in the working directory. Compose derives the project name from the directory it was started in, so the old probe found a stack only when run from the checkout that launched it (a globally linked sim never could) and listed the same stack once per candidate file. compose ls reports the real project and its config file, so one stack yields one install from anywhere; non-Sim projects are filtered by compose filename. Every compose op now runs in that stack's directory.
- lifecycle: distinguish 'Docker unreachable' from 'nothing installed'. With the daemon down, status reported containers as 'absent' and suggested re-running setup; it now says Docker is down and marks state unknown.
* fix(setup): don't start managed Postgres with a password the volume will ignore
POSTGRES_PASSWORD only applies when initdb runs on an empty data directory. The sim-postgres-data volume outlives its container (sim down keeps it, docker rm keeps it, and the wizard's own recreate path keeps it), and inspectManagedContainer recovers the password from the *container*, not the volume — so once the container is gone the password is unrecoverable.
Setup then generated a fresh password and ran against the initialized volume. Postgres kept its original password and rejected every connection with 'password authentication failed for user postgres', which surfaced as a misleading 'container did not become healthy'.
Detect an already-bootstrapped volume (PG_VERSION present) before choosing a password, and ask: supply the existing password, or delete the volume and start fresh (double-confirmed, since that destroys data). Refusing both fails with the exact docker volume rm command instead of looping.
* improvement(setup): default to Docker Compose and sharpen the run-mode copy
Compose was listed first but only preselected when Docker happened to be running — with Docker stopped the cursor sat on 'Local dev', steering people toward a source checkout when they wanted to run Sim. Compose mode calls ensureDocker(true), which offers to start Docker Desktop, so a stopped daemon is no reason to change the default.
Also tightens the hints to say what each mode is for: run bundled Sim (fastest way to start), work on Sim itself, test a production-style k8s deploy.
* fix(compose): point the browser socket at :3002 so it stops reconnecting
The stack publishes the app on 3000 and realtime on 3002 with no reverse proxy between them, but NEXT_PUBLIC_SOCKET_URL defaulted to empty — which tells the browser client to use the page origin. :3000/socket.io answers 308 (a Next redirect), not a Socket.IO handshake, so the client failed and retried forever. Default it to http://localhost:3002; a proxied deployment overrides it (or sets it empty to use the page origin).
Also give COPILOT_API_KEY and SIM_AGENT_API_URL empty defaults so every compose command stops printing 'variable is not set' warnings. The app already falls back to the prod copilot backend when SIM_AGENT_API_URL is blank.
* feat(setup): pass SIM_AGENT_API_URL through, and warn on a half-set mothership
Sim devs testing against a non-prod mothership export SIM_CLI_AUTH_ORIGIN so the Chat key is minted there, but nothing carried the matching backend URL into the install — the app kept defaulting to prod copilot, which rejects a staging key with 'Invalid API key'.
Persist SIM_AGENT_API_URL when it is exported, so later docker compose up / dev runs stay on that backend instead of reverting to prod once the shell is gone:
SIM_CLI_AUTH_ORIGIN=https://www.staging.sim.ai \
SIM_AGENT_API_URL=https://www.staging.copilot.sim.ai \
bun run setup
Setting only the auth origin is the trap, so that combination warns. Neither set is the self-hoster default and stays silent — no prompts, no flags.
* fix(setup): survive a vanished port owner, and stop flagging our own containers
Two failures from one compose re-run:
- 'Kill it for me' crashed setup with 'kill() failed: ESRCH: No such process'. The owner list is an lsof snapshot, so the process can exit before the signal lands — which is the outcome we wanted, not an error. ESRCH now counts as freed, EPERM warns that it must be stopped by hand, and anything else warns; the loop re-probes either way instead of aborting a setup that had already written .env.
- Compose mode demanded 3000/3002 be free even when this stack was the one holding them, so re-running setup against a running install reported its own realtime container as a blocker and offered to kill Docker's listener. 'docker compose up -d' reconciles its own containers, so skip the check when the project already has some. A foreign process is still caught, and a foreign container still surfaces as a bind error from compose.
* fix(csp,setup): permit the socket origin the client actually uses; encode DSN passwords
Review round on #5964:
- The socket reconnect was a CSP bug, not a URL bug. getSocketUrl() already falls back to localhost:3002 for a localhost page, but generateRuntimeCSP gated that same fallback on isDev — and compose runs NODE_ENV=production, so connect-src omitted ws://localhost:3002 and the browser blocked the handshake. Key the fallback on the app URL being localhost instead, mirroring getSocketUrl. Revert the compose NEXT_PUBLIC_SOCKET_URL default: an explicit value suppresses the page-origin fallback that reverse-proxied self-hosts depend on, and ':-' treats empty as unset so the documented escape hatch could not work either. LOCALHOST_HOSTNAMES is duplicated locally because csp.ts is loaded by next.config.ts before @/ aliases resolve.
- Percent-encode the password when building the Postgres DSN. A user-supplied password containing @ : / # does not merely re-parse to the wrong host — it fails to parse as a URL at all, so a correct password surfaced as a connection failure.
- Tell 'Postgres rejected this password' apart from 'Postgres never started'. On the keep-the-volume path a wrong password left a healthy server and the old generic 'container did not become healthy' error, which is the confusion this change set exists to remove.
Adds a CSP regression test for the unset-socket-URL production case; verified it fails against the previous condition.
* improvement(setup): make k8s mode end somewhere usable, and show install progress
Two things made k8s mode the least satisfying path.
The services are ClusterIP, so a successful install left nothing on :3000 — 'Sim is ready' was true about the cluster and useless to the user, who had to notice and run a port-forward by hand. Compose opens a browser and dev offers to start the server; k8s now offers the forward the same way and runs it in the foreground so Ctrl-C ends it. Realtime gets its own forward (kubectl takes one resource per invocation) or the editor socket fails; it is a child in the same process group, so the terminal's Ctrl-C reaches it, and it is killed explicitly when the app forward exits.
'helm --wait' then blocked for minutes with a single static spinner, so a slow image pull looked identical to a wedged install. Run helm asynchronously and poll the cluster, so the spinner reports '3/3 pods ready · 1 starting'. CronJob-owned pods are excluded: the chart schedules a lot of them (36 on a running cluster here) and they finish as Completed, which would swamp the count and make readiness jitter for reasons unrelated to the install. Restarting pods are surfaced too — a cold cluster restarts realtime while Postgres comes up, and a silent spinner made that look like nothing was happening.
* fix(setup): identify Sim compose projects by content, not filename
Cursor (High): composeInstalls treated any project whose config basename was docker-compose.prod.yml or docker-compose.local.yml as a Sim install. Those names are common, and sim reset runs 'compose down -v' — so a stranger's stack could have had its volumes destroyed.
I introduced that reach. The previous ROOT-scoped '-f' probe was implicitly safe because it could only ever see the project in this checkout; switching to a global 'compose ls' to find stacks started elsewhere means projects must be identified by content instead. Read the config file Docker recorded and require a Sim marker (the published app image, or the app Dockerfile this repo builds), so both the prod and local variants match while an unrelated file with the same name does not. An unreadable or since-deleted file is left unmanaged rather than assumed ours.
Verified against a decoy nginx compose file using our exact filename: ignored, while both real Sim compose files still match.
* fix(setup): scope the compose port skip to published ports; print both k8s forwards
Review round on #5964:
- ensureComposePortsFree skipped conflict handling whenever the project had any container running, so leftover db/redis (which publish neither app port) waved through a foreign process on :3000 — it then surfaced as a raw compose bind error instead of the prompt. Read the host ports the project actually publishes and skip only those; the remaining ports still get the full check. Reading from the containers rather than the file matters because what counts is what is bound right now.
- The post-install note and the skip path documented only the app forward, while offerPortForward runs two. Skipping the prompt or copying the printed command left the editor's socket dead — the exact failure this change set exists to fix. Both commands now come from one forwardCommands() helper, so what is printed and what is run cannot drift.
* fix(setup): one source for the k8s forwards, and surface a dead realtime forward
Third round on the same theme, so fix it at the root rather than at another call site.
- lifecycle's k8sReachHints (used by sim start/restart) still restated an app-only forward, recreating the dead editor socket the setup path had just been fixed for. forwardCommands is now exported and consumed there, so every place that tells a user how to reach a ClusterIP release derives it from one definition.
- The realtime forward was spawned with stdio ignored and never checked, so a busy :3002 or a missing service killed it silently while the app forward kept running — indistinguishable from success until the editor won't connect. Keep its stderr, warn on an exit we did not ask for, and stay quiet on the intentional kill.
* fix(setup): pin the compose project on every lifecycle op
composeInstalls records the real project name from 'compose ls' and status and the destructive confirms print it, but every op ran 'compose -f <file>' with only cwd set — so Compose re-derived the project from that directory. The derived name is frequently not the recorded one: a directory is lowercased and stripped of dots (Sim.Demo_Test derives simdemo_test), and an explicit -p or COMPOSE_PROJECT_NAME at creation diverges outright. stop/down/reset could therefore act on a different project than the one named in the confirm, and reset runs 'down -v'.
Route every op through composeArgs(), which pins '-p <recorded project>'. cwd stays, since the file's own relative paths still resolve against it. Verified with a stack started as -p pinned-name from a directory deriving simdemo_test: the old form found 0 of its containers, the pinned form finds them.
* fix(setup): warn on both halves of a mothership mismatch
mothershipOverride warned only when SIM_CLI_AUTH_ORIGIN was set without SIM_AGENT_API_URL, while its own copy said to set both or neither. The reverse is the same failure mirrored: with only SIM_AGENT_API_URL set, the Chat key is still minted against the default prod auth origin and then validated against the override, which rejects it — silently, which is exactly what this helper exists to prevent.
Warn on either asymmetry, and read the default origin from one constant shared with the handoff so the message can't claim an origin the code no longer uses.
* fix(setup): warn about a half-set mothership before minting the key
mothershipOverride ran two steps after promptCopilotKey, so a half-set override minted a key against one environment, stored it, and only then warned that the other environment would reject it. Worse on a re-run: promptCopilotKey offers to keep an existing COPILOT_API_KEY and defaults to yes, so the bad key survives.
Move the override ahead of the key prompt in both compose and dev, so the warning arrives while it can still change the outcome — the user can abort and set the missing half before anything is minted. Nothing in the override depends on the key, so the order is free.
|
||
|
Theodore Li
|
19c3b6f47d |
feat(setup): setup wizard with browser-based Chat key handoff (#5911)
* feat(setup): setup wizard with browser-based Chat key handoff
Adds `bun run setup` and `bun run doctor` for local installs, and replaces
the wizard's paste-your-Chat-key step with a browser handoff that never puts
the key in a URL.
* improvement(setup): drop the paste-a-key fallback, simplify consent copy
The browser handoff is now the only path — the wizard waits on a spinner
instead of racing a paste prompt. Consent card leads with "Connect your
terminal" and moves the match-the-code disclaimer into the description.
* fix(setup): pin kube context, keep secrets out of argv, validate reused keys
Review findings from #5911:
- helm/kubectl now run against the validated context instead of the ambient one
- helm values are piped on stdin rather than passed as --set arguments
- ENCRYPTION_KEY/API_ENCRYPTION_KEY are checked for the 64-hex format the app
requires, not just length, so an unusable key is replaced rather than kept
- the managed Redis container's published port is read back instead of assumed
* refactor(copilot): one module for Chat API key operations
list/generate/delete each repeated the same /api/validate-key envelope in
their route. They now share callValidateKey in lib/copilot/server/api-keys.ts,
which also keeps the display masking server-side so the full key can only ever
leave at creation.
* improvement(setup): reuse shared helpers, parallelize probes, drop dead code
- PKCE verifier/state/pairing code now use generateSecureToken, generateRandomHex
and generateShortId instead of hand-rolled randomBytes; the pairing loop's
modulo was unbiased only because 256 % 32 == 0
- new sha256Base64Url in @sim/security/hash so both sides of the PKCE exchange
derive the challenge from one implementation
- isUsableSecret moved beside SECRET_KEYS so setup and doctor apply the same
rule; doctor previously passed a key setup would replace
- isTruthy narrowed to true/1, matching the app it claims to mirror — it accepted
yes/on, so a flag could read on in doctor and off in the app
- checkLive runs its five probes concurrently (~17s serial worst case)
- detection overlaps the banner animation instead of queueing behind it
- glyph.fail/glyph.warn at 13 sites that bypassed the constant; removed unused
prompter exports, a dead ENV_PATHS re-export, and an unused export keyword
* fix(setup): make doctor understand the compose env layout
Compose writes a single root .env (what docker-compose reads via env_file) but
the checks required the three per-app files, so a successful compose install was
followed by doctor printing three failures and exiting 1 — and the whole
coherence catalog was skipped because it keyed off apps/sim/.env existing.
Layout is now derived from what's on disk and every check consults it: file and
schema checks iterate the layout's targets, consistency reports skip when
there's only one file to mirror, and coherence/live read the layout's primary
file. The wizard's existing-config detection counts root for the same reason —
a compose install used to read as unconfigured and re-run from scratch.
* feat(cli-auth): device-authorization poll flow, drop the loopback listener
The CLI no longer binds a local port. It generates a request id + poll secret,
opens /cli/auth, and polls /api/cli/auth/poll over TLS while the user approves
in the browser — so the flow works over SSH and inside containers, where the
browser and terminal don't share a machine.
- approve stores the approval keyed by request id (session-authed, userId from
the session only); poll verifies the secret before an atomic claim, so an
observer of the semi-public request id can neither mint nor cancel it
- pairing code stays as the anti-phishing compare; no key ever crosses the
browser; done page just confirms
- removes the loopback listener, /token exchange, buildCliHandoffUrl, and
validateCliCallbackUrl (+ its tests) — nothing hands a key to a URL anymore
* fix(setup): reuse an existing managed Postgres container instead of colliding
A running sim-postgres fell through to `docker run --name sim-postgres` and died
on the name conflict; a stopped one failed with "no DATABASE_URL to reach it"
because the generated password only lived in the env files a fresh clone lacks.
Both facts are recoverable from Docker: the ladder now reads the published port
and password back via `docker inspect` and reuses the container (starting it if
stopped). A container that won't answer prompts before recreating, and never
drops the data volume silently.
* improvement(setup): audience-first run-mode hints
Each run mode now names who it's for — compose for self-hosting/evaluating, dev
for contributing to Sim, k8s for rehearsing a production deploy — with the live
detection state (Docker/kube/VM) appended.
* fix(cli-auth): retry a failed mint, port container/port fixes to Redis + k8s
Review findings from #5911:
- poll now reserves the mint with an atomic NX lock instead of deleting the
approval up front, so a failed mint (e.g. mothership blip) is retried by the
next poll instead of forcing a fresh browser approval; the lock still prevents
a double-mint and its TTL frees the slot if the caller dies
- setup reuses/recreates an unhealthy managed sim-redis instead of colliding on
the name (Redis has no data volume, so it removes and recreates without a prompt)
- k8s failure-path hints carry --context, matching the success-path hints, so a
changed ambient context can't send diagnostics to the wrong cluster
- compose port-free waits for a killed port to actually release before
re-checking; SIGKILL is async, so the immediate re-check re-saw the port
* fix(setup): harden mint cleanup, Windows browser, container detection, helm cwd
Review findings from #5911:
- a post-mint completeApproval failure no longer routes into releaseMint — the
mint lock now outlives the approval (shared TTL), so a cleanup blip can't leave
a re-mintable window and orphan a key; cleanup is best-effort after the key ships
- compose doctor --fix writes the feature-flag twin to the layout's primary env
(root .env on a compose install), not always apps/sim/.env
- Windows opens the browser via `cmd /c start "" <url>` — `start` is a shell
builtin, so spawning it directly ENOENT'd and the handoff never opened
- managed-container detection filters loosely and pins the exact name in code;
Docker's `name=^x$` anchor matches the internal `/x` form and often missed,
skipping the reuse branch
- the shared helm/kind run helper pins cwd to the repo root, matching helm test,
so `helm upgrade --install ./helm/sim` works from any working directory
* feat(chat-keys): standalone manage page, drop from settings nav, refresh README
- Add /account/settings/chat-keys — a linkable page to view, create, and revoke Chat API keys
- Remove Chat keys from the settings sidebar (account + unified nav) and its render branches
- README: replace Docker Compose + Manual Setup with the bun run setup wizard; drop the manual COPILOT_API_KEY step, point to the manage page
* fix(setup): per-key reason in the secret-replacement warning
Cursor: the warn hardcoded '64-character hex key', but only ENCRYPTION_KEY/API_ENCRYPTION_KEY require that — BETTER_AUTH_SECRET/INTERNAL_API_SECRET only need length >= 32. Use the existing secretRequirement(key) helper so each replaced key reports its actual requirement.
* fix(setup): compose doctor schema, cross-platform binary detection, quoted context hints
- Doctor: for the compose (root) env layout, require only the secrets compose has no interpolation default for (BETTER_AUTH_SECRET/ENCRYPTION_KEY/INTERNAL_API_SECRET). DATABASE_URL/BETTER_AUTH_URL/NEXT_PUBLIC_APP_URL come from docker-compose ${VAR:-default}, so a healthy compose install no longer fails doctor.
- Binary detection: use Bun.which instead of which (which is absent on Windows), so kubectl/helm/kind/docker resolve cross-platform.
- k8s diagnostic hints: POSIX-quote the kube-context so a context with whitespace/metacharacters can't break or inject into a copied command.
* fix(setup): quote kube-context in the helm uninstall tear-down hint too
The tear-down hint used --kube-context ${context} raw while the sibling kubectl hints already used shq(); a context with whitespace/metacharacters could break or inject into the copied command. All copyable k8s hints now go through shq(context).
* feat(setup): sim lifecycle CLI — start/stop/status/logs/down/reset
Turn the setup entry into a 'sim' command umbrella so there's one place to run everything, not scattered docker/bun commands. Adds a global bin (bun link) + a bun run sim fallback.
- Detects how you're running (compose file / managed dev containers / helm release) from disk + docker/helm state — no persisted mode. Ambiguous installs prompt.
- start/stop/restart/logs work per mode; down removes containers (volumes kept); reset archives .env + wipes managed data; both destructive verbs confirm first.
- status shows detected mode, container states, and app/realtime health.
- Wizard outro + README now point at the sim commands and the one-time bun link.
* feat(setup): 'bun run sim' is the primary entry; bare invocation prints help
- Lead usage/wizard-outro/README with 'bun run sim <cmd>' (works with zero PATH setup); global bare 'sim' via bun link is an optional upgrade, with the ~/.bun/bin PATH caveat spelled out (Homebrew's bun omits it).
- Bare 'sim' now prints help instead of launching the wizard; the wizard is 'sim setup'. The 'setup' npm script passes the keyword so 'bun run setup' is unchanged.
* fix(setup): quote the auth URL for cmd /c start on Windows
Cursor (High): cmd re-parses the command line and treats & in the query string as a command separator, so cmd /c start opened a URL truncated at the first &, breaking the key flow on win32 (the handoff URL always has request/challenge/pairing). Quote the URL and pass args verbatim so & stays literal.
* fix(setup): verify kube-context is really local; lengthen CLI handoff wait
- k8s: a context named like a local cluster (kind-*, docker-desktop) can actually point at a remote API server. Verify the server host is loopback/docker-internal before defaulting the 'use this context?' confirm to yes; otherwise warn and default to no, so generated secrets can't ship to a remote cluster on a blind Enter.
- cli-auth: bump the device-flow wait from 3 to 15 minutes so first-time users have time to sign up, wait for the email OTP, and approve before the terminal stops polling. The server-side approval record keeps its own short TTL, so a longer client wait only costs cheap rate-limited polls.
* fix(setup): only manage k8s lifecycle on a verified-local context
Greptile: sim down/reset used the ambient kube-context, so switching context after setup could uninstall a same-named sim-dev release from the wrong cluster. Gate k8sInstall on the same locality check the wizard uses (API server is loopback/docker-internal) via a shared isLocalKubeContext helper — the wizard only ever deploys locally, so a remote current-context is never treated as a Sim install.
* fix(setup): doctor skips placeholder secrets when seeding; reset names its target
- checks: the missing-file autofix copied shared keys from apps/sim/.env whenever truthy, including .env.example placeholders — doctor --fix could seed unusable secrets into realtime/db env files. Skip placeholders, matching autofixForMissing.
- lifecycle: reset now names the exact install (k8s context / compose file / dev containers) in its confirm, so a destructive reset can't silently hit the wrong same-named install after a context switch (down already names the context).
* fix(cli-auth): size the poll rate limit to the poll cadence; honor Retry-After
The poll route used the default public-IP bucket (10 burst, 5/min) but the CLI polls every 2s (30/min), so it 429'd within ~20s — worse behind a slow dev cold-compile. Give the endpoint a bucket matched to its cadence (60 burst, 60/min); it's not a brute-force surface (unknown request id returns pending, minting needs the 256-bit verifier). Also make the CLI honor Retry-After and back off on 429 so a shared-NAT per-IP limit degrades gracefully instead of hammering.
* fix(setup): check ports before starting the dev server, not just compose
Local dev auto-start spawned bun run dev:full with no port check, so it silently started a server that couldn't bind when 3000/3002 were already taken (e.g. another worktree's dev server). Extract compose's port-conflict resolver into a shared ensurePortsFree(ports) and run it before the dev start too — kill/recheck/leave, same as compose. Leaving the ports skips the auto-start with guidance instead of failing; compose still treats it as fatal.
* fix(setup): verify the kube cluster is reachable, not just local
A kubeconfig context can outlive its cluster — a kind cluster gets deleted or its Docker container stops (Docker/machine restart), but the context entry remains, pointing at a dead API-server port. The wizard checked the context looked local and handed it to helm, which failed with 'cluster unreachable'.
Add a clusterReachable() liveness probe: only offer the current context when it actually answers; if a local context is dead, fall through to the kind path. There, if kind still knows 'sim' but it's stopped, start its node containers and wait for the API; if it's gone, create fresh. Either way the user gets a working cluster instead of a cryptic helm failure.
* fix(helm): point appVersion at published image tags (v-prefixed, current)
The chart's appVersion was "0.6.73", but CI publishes GHCR tags with a v prefix (its release-commit regex captures v0.7.45). Since sim.image defaults every image tag to Chart.AppVersion, a default helm install requested ghcr.io/simstudioai/{simstudio,realtime,migrations}:0.6.73 — a tag that has never existed — so app and realtime sat in ImagePullBackOff and helm --wait failed with 'progress deadline exceeded'. Any self-hoster installing with default values hit this, not just the setup wizard.
Set appVersion to v0.7.45 (latest release on main; all three images verified present on ghcr) and bump the chart version to 1.1.1. Verified with helm lint, helm template (all images render as v0.7.45), and a live helm upgrade on a kind cluster where the new pods pull successfully while the old 0.6.73 pods remain in ImagePullBackOff.
* Revert "fix(helm): point appVersion at published image tags (v-prefixed, current)"
This reverts commit
|
||
|
Vikhyath Mondreti
|
fe184d3695 |
improvement(whatsapp): validate + improve integration skill for file inputs/outputs (#5942)
* improvement(whatsapp): validate + improve integration skill for file inputs/outputs * fix lint * add whatsapp subblock migration |
||
|
Vikhyath Mondreti
|
17d77795b4 |
feat(providers): prompt caching capability + usage-based cache pricing (#5922)
* improvement(providers): validation pass, and stream tool loop improvements * remove deploy options correctly * fix * feat(providers): prompt caching capability and usage-based cache pricing Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic. * fix(providers): route OpenAI and Gemini block cost through cache-aware pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained. * fix(streaming): redact tool payloads on selected outputs in public chat Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole. * refactor(providers): drop unreachable billing fallbacks Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total. |
||
|
Waleed
|
513292f17b |
feat(sso): DNS domain verification gating org SSO registration (#5909)
* feat(sso): DNS domain verification gating org SSO registration
Add org-scoped domain ownership verification (DNS TXT challenge) as the
security precondition for configuring SSO. Closes the first-come domain-claim
vuln where any org could wire another company's domain to its own IdP.
- New sso_domain table + migration 0266; existing org SSO domains are
grandfathered as verified so live tenants are unaffected
- Verified-domains settings UI (enterprise-gated) with add/verify/remove
- Register route now requires a verified domain for org-scoped registration;
personal SSO and already-grandfathered domains are unaffected
- Self-host register script writes the verified sso_domain row directly, so
script-driven registration stays backwards compatible
* fix(sso): harden domain verification against concurrency + fix CI lint
Addresses review findings on state invariants under concurrent/failed writes:
- Add unique index on (organization_id, domain) so concurrent claims can't
create duplicate pending rows; POST re-reads and stays idempotent on conflict
- Verify flips the row only if it's still the exact pending challenge checked
(guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique
index violation to 409 instead of an unhandled 500
- Wrap the self-host script's provider write + verified-domain upsert in a
transaction so a failed ownership write can't leave a provider committed
- Format 0266 snapshot/journal with biome (fixes @sim/db lint:check)
* fix(sso): re-check domain verification before provider write (TOCTOU)
The register gate checked the verified sso_domain row only at handler entry,
then ran OIDC discovery before writing the provider. A verified row removed
during that window could still complete registration. Extract the check into a
closure and call it both as an entry fast-fail and authoritatively right before
registerSSOProvider, alongside the existing domain-conflict re-check.
* fix(sso): stop rotating verification token on idempotent re-add
Re-adding a pending domain rotated its verification token, which invalidated a
TXT record the admin may have already published and — under two concurrent
re-adds — could return a token the racing write had already superseded, so the
admin's DNS record would never verify. Return the existing row unchanged
instead; the pending token is always shown in the UI, so it is never lost.
* fix(sso): close register TOCTOU with compensating delete + harden edges
Audit-driven hardening:
- Close the residual register TOCTOU: registerSSOProvider is create-only (throws
if the providerId exists), so a compensating delete after the write is provably
safe — it can only remove the just-created row. If verification was revoked
during the write, roll the provider back and 403.
- Verify is now idempotent under concurrency: a same-org row already flipped to
verified by a racing request returns 200, not a confusing 409.
- Grandfather backfill + self-host script now match normalizeSSODomain's dominant
transforms (lower + trim + strip leading wildcard) so a non-canonical legacy
domain can't miss the runtime gate's lookup. Prod backfill result is unchanged.
- Cleanup: drop dead default export, align card radius to sibling convention.
* fix(sso): redact domain tokens from non-admins + fix script stale-update
Round-5 review findings:
- GET /domains redacted the pending TXT verification token (a management
secret) to any org member. Now only owner/admins read it; members see the
list and status without it. Non-Enterprise orgs get an empty list (entitlement
flag only), never the domains/tokens.
- Self-host script decided update-vs-insert from a read taken OUTSIDE the
transaction; a provider deleted mid-flight made the UPDATE match zero rows
silently while the verified-domain upsert still committed (orphaned domain).
The decision now happens inside the transaction from the UPDATE's row count.
* docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains
Verified domains currently only gate SSO configuration. Remove the
forward-looking references to enforcing SSO and auto-joining members (deferred
to a later release) from the docs, settings copy, nav description, and schema
comment so we don't promise unshipped features.
* fix(sso): guard rollback to new providers only + Enterprise-gate domain removal
Round-6 review findings:
- The compensating provider rollback now only fires when the provider did not
exist before this request (providerExistedBefore). registerSSOProvider is
create-only today so reaching the rollback already implies a fresh create, but
this makes the safety local and future-proof: if Better Auth ever allowed
updating an existing provider, a revoked-verification rollback must not delete
that pre-existing row.
- DELETE /domains now requires an Enterprise plan like add/list/verify, so all
domain mutations share one entitlement (the UI already hides removal from
non-Enterprise orgs). Adds a delete-route test.
* fix(sso): roll back the SSO provider by row id, not logical keys
The compensating rollback deleted by (providerId, orgId). providerId is unique,
so if this request's row were deleted and recreated by a concurrent registration
in the narrow window before the rollback, the logical-key delete would remove
that other request's provider. Delete by the primary-key id registerSSOProvider
returns instead, so only the exact row this request created is ever removed.
* chore(sso): final-review polish — trim script read, unify copy, doc migration edge
Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the
new logic):
- Self-host script: narrow the pre-transaction existence read to select({ id })
instead of SELECT * (it only feeds a log line now).
- Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere
409 wording ("is already verified by another organization") across routes.
- p-3 shorthand on the domain row card.
- Document the migration's rare two-orgs-share-a-domain grandfather behavior
(login unaffected; validated no such duplicates in prod).
* fix(sso): apply attribute mapping + make SSO edit work; drop dead guard
Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW,
script-registered with the default mapping, so neither change affects it):
- Attribute mapping was passed at the top level of the register payload, which
Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it
so custom mappings actually apply. (Default mapping is unchanged, so existing
logins are unaffected.)
- Editing an SSO provider was broken: registerSSOProvider is create-only and
threw on the existing providerId → generic 500. Route now detects a provider
the caller already owns and updates it via Better Auth's updateSSOProvider, and
surfaces Better Auth's own error status/message instead of a blanket 500.
Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes
by the created row's primary-key id and register is create-only) and the earlier
final-review polish (script read, unified copy, migration edge note).
Smoke-test SSO login + edit on staging before merge (auth-path change).
* fix(sso): require null org on personal-mode provider lookups (gate bypass)
The personal branch of both provider-ownership lookups keyed on
(providerId, userId) without requiring organizationId IS NULL. Because org
providers store userId = their creator and providerId is globally unique, an org
admin could send a personal-mode request (no orgId) — which skips the membership
check and the domain-verification gate — yet still match, and then via the new
update path move, their org's provider to an unverified domain. Add
isNull(organizationId) to the personal branch of both clauses so it can only
match a genuinely personal provider, matching the route's own isOwnedByCaller.
Found by an adversarial review of the update path added in 394bda9f7.
* fix(sso): script updates the observed provider by id, not providerId
Inside the registration transaction the script updated WHERE providerId — the
logical key. If the observed provider was deregistered and a replacement created
with the same providerId before the transaction ran, that update would clobber
the replacement's config and ownership. Update the specific observed row by its
primary-key id instead; if it's gone we insert, which fails cleanly on the
providerId unique constraint rather than overwriting the replacement.
* fix(sso): script upserts provider via delete-then-insert (no unique constraint)
sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate
duplicates, so the previous "update by id, else insert" could create a duplicate
provider when the observed row was deregistered and replaced before the
transaction — the fallback insert would succeed. Delete every row for the
providerId then insert exactly one, inside the transaction, so the providerId
ends up as exactly this config atomically. Linked accounts key on the providerId
string (not the row id), so existing logins are unaffected.
* fix(sso): guard compensating-delete row id so rollback can't silently no-op
* chore(sso): regenerate migration as 0268 after merging staging
Staging landed migrations 0266/0267, colliding with our 0266. Removed our
migration, merged staging, and regenerated cleanly with drizzle-kit as
0268_sso_domain_verification (identical sso_domain table + indexes), then
re-appended the grandfather backfill. api-validation baseline reconciled to 973
(staging 970 + our 3 domain routes). Also make the register-route test's
registerSSOProvider mock return an id so the guarded compensating delete runs.
* refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate
The self-host script canonicalized SSO domains with a minimal inline transform
(lower+trim+wildcard) that diverged from the app's full normalizeSSODomain
(protocol, port, path, trailing dot, email local part) — equivalent spellings
could store a different ownership key than the runtime gate looks up. Move
normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register
route, the domain-claim route, and the script all use the identical canonicalizer.
The script now skips the verified-domain record when SSO_DOMAIN isn't a valid
registrable domain instead of storing a malformed key.
|
||
|
|
6dcc65be89 |
feat(skills): add skill editors (#5705)
* feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com> |
||
|
|
d24bc7eccb |
feat(agent-stream): thinking and tool streaming (#5671)
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
2b5a92a3c8 |
feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862)
* feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name) |
||
|
WaleedandMarcus Chandra
|
6f33a9485b |
feat(workflows): IDE-style reference viewer for workflows (#5854)
* feat(workflows): add IDE-style reference viewer for workflows
Adds a "Show references" viewer so you can see how workflows connect:
which workflows call a given workflow ("Used by") and which workflows it
calls ("Uses"), rendered as recursive, clickable trees.
- Opened via Cmd/Ctrl+click on a sidebar workflow row and a "Show
references" context-menu item.
- Resolves references through both the workflow / workflow_input blocks
(reusing isWorkflowBlockType) and published custom blocks
(custom_block_* -> source workflow), scoped to the workspace.
- Builds the whole workspace reference graph once from live workflow_blocks
state; cycle-safe DFS marks A->B->A loops as (cycle) leaves.
- Contract-bound GET /api/workflows/[id]/references with workspace-level
authz; React Query hook gated to fetch only when the modal opens.
- Unit tests for the pure graph/tree logic (cycles, self-refs, dangling
drop, custom-block + workflow_input resolution) and route tests
(401/400/403/200).
* fix(workflows): correct reference resolution for active mode, cycles, cache, and graph size
Addresses review findings on the reference viewer:
- Resolve the workflow-block child via resolveActiveCanonicalValue (the
shared SOT) instead of basic-first `||`, so an advanced-mode block whose
old basic workflowId value lingers resolves to the active manual value.
- Keep self-references (A -> A) and render them as a cycle leaf instead of
dropping the edge, matching the cycle-safe viewer's purpose.
- Set the references query staleTime to 0 so reopening the always-mounted
modal refetches live editor state instead of serving a stale cached graph.
- Bound converging paths: a node already expanded elsewhere in the tree is
emitted once more as a plain leaf (edge stays visible) rather than
re-expanded, so a densely reconverging graph can't grow exponentially.
* improvement(workflows): align reference viewer auth, coverage, and UI with platform conventions
- authorize via authorizeWorkflowByWorkspacePermission and derive the
workspace server-side (404/403 semantics; drops the client-supplied
workspaceId query param from the contract, hook, and modal)
- add workflow-tool call edges: workflow_input tools inside tool-input
sub-blocks now appear in both trees; non-call selector shapes stay
deliberately excluded (documented against remap-internal-ids)
- restore native cmd/ctrl+click open-in-new-tab on sidebar workflow rows;
references stay reachable from the context menu
- mount ReferencesModal on demand per row, deleting the prevIsOpen reset,
the enabled knob, and the staleTime-0 workaround (now 30s)
- align the tree with design tokens (--text-icon, --surface-hover, px-4
text gutter) and drop the hardcoded brand hex
- escape LIKE wildcards in the custom_block_ prefix match; import
MAX_CALL_CHAIN_DEPTH instead of mirroring it; remove dead fallbacks,
the duplicate not-found scan, and the redundant custom-block row map
* improvement(workflows): final polish on the reference viewer
- drop the vestigial isOpen prop (conditional mount owns visibility)
- unify on the emcn Workflow icon in tree rows
- inline the static className and derive nodes without an annotation
- remove one restating test comment
* fix(workflows): resolve tool references by active canonical mode and keep the reference cache live
- workflow_input tools inside tool-input now resolve basic/advanced via the
index-scoped canonicalModes override, mirroring execution (Cursor finding)
- staleTime back to 0: no mutation invalidates this key, so a reopen must
background-refetch; on-demand mounting keeps the cached tree painting
instantly (Greptile P1)
- modal header uses the em-dash label-entity convention; tree items carry
aria-level instead of a static aria-selected
* fix(workflows): cover legacy workflow-typed tools and retry depth-truncated expansions
- toolInputCallees matches both workflow tool type spellings via
isWorkflowBlockType and passes the tool's own type as the legacy
canonicalModes fallback, matching providers/utils resolution
- a depth-capped expansion no longer poisons the expanded set, so a
shallower path re-expands the node in full (Cursor finding)
- the allowed-but-workspaceless auth branch now returns 403, not the
authz result's 200
- tests: legacy tool type + per-tool index-scope isolation, diamond
re-expansion with a real subtree, depth ceiling, shallow-path retry
---------
Co-authored-by: Marcus Chandra <mzxchandra@gmail.com>
|
||
|
Waleed
|
93fbf584a2 |
feat(chat): soft-delete sidebar chats with restore from Recently Deleted (#5830)
* feat(chat): soft-delete sidebar chats with restore from Recently Deleted * fix(chat): review round 1 — restore workspace authz, purge recheck, archived-list invalidation * test(chat): update SSE handler assertions for workspaceLists invalidation * fix(chat): bump updatedAt on restore, recheck retention cutoff in task cleanup * fix(chat): drop explicit feedback delete in task cleanup — chat FK cascade covers it * test(chat): cover restore route; guard legacy copilot delete from hard-deleting mothership chats * chore: revert unintended bun.lock drift from worktree install * fix(cleanup): recheck workflow archive cutoff on delete; export deletedAt in chat drain |
||
|
Waleed
|
dd0e736d52 |
feat(managed-agents): add Claude Managed Agents workflow block (#5778)
* feat(managed-agents): add Claude Managed Agents workflow block
* improvement(managed-agents): complete session inputs/outputs; trim block templates
- add memory instructions + file mount_path inputs; bound metadata (16 pairs)
- surface cumulative token usage (inputTokens/outputTokens) as outputs
- validate the full session-create schema against live docs
- remove BlockMeta templates
* improvement(managed-agents): select a Claude Platform credential instead of BYOK
- register Claude Platform as a token-paste service-account credential (descriptor + validator)
- add no-OAuth OAUTH_PROVIDERS entry; generalize the shared credential picker with a 'service-account' kind
- block: oauth-input credential picker + dependsOn dropdowns; list route resolves the key server-side (audit-logged)
- run via directExecution with the executor-injected key; drop the internal run route
- remove the interim claude-platform BYOK provider
* fix(managed-agents): harden reconnect loop; fix credential-picker regression
- drive completion off terminal events + authoritative session status (drop the fragile busy-clock)
- drain full event history so a long session's tail is never cut off
- skip idless events in catch-up; require an id before replying to custom_tool_use
- gate the shared credential-selector service-account lookup on credentialKind and use the non-throwing helper (was crashing multi-service OAuth pickers)
- audit-log the list route's credential access; refresh stale tool docs
* fix(managed-agents): address review round 2
- don't complete on idle status while a requires_action is still outstanding
- vaults: combobox → dropdown so multiSelect actually attaches multiple vaults
- auto-select a freshly-pasted service-account credential (onCreated through the connect modal)
* fix(managed-agents): propagate cancellation, fix reconnect ordering, classify advanced fields
- Thread the executor abort signal into directExecution (additive ToolConfig
change) so a cancelled workflow stops the session immediately; best-effort
user.interrupt releases the Anthropic session past cancel/wall-clock cap
- Recompute the requires_action pending state from the chronological history
so an older agent.message recovered on catch-up can't clear a newer pause
- Retry a custom-tool error reply that failed to send instead of stranding the
session (mark the event seen only once handled)
- Mark optional fields (vaults, memory, files, metadata) mode: advanced
- Title the session with the workflow id for Claude-console traceability
* fix(managed-agents): stop leaked sessions on all give-up paths; harden event ordering and normalizers
- Interrupt on sendUserMessage failure and on the reconnect-cap exit, matching
the abort/wall-clock paths, so no give-up path leaves a session running
- Order the events list by processed_at (list page order isn't guaranteed
chronological) so catch-up accumulates text and reads the latest lifecycle
event correctly regardless of API sort
- Don't reset reconnect backoff on a failing custom-tool reply (stays unseen
for retry) — prevents a no-delay reconnect storm
- Treat only end_turn as a complete status_idle event; an unspecified idle
defers to the sawActivity-gated status check (no empty completion pre-turn)
- normalizeFiles parses a JSON-stringified table instead of dropping it;
normalizeStringList returns [] on malformed JSON; metadata keeps scalar values
- Declare the injected accessToken as a hidden param for convention parity
* fix(managed-agents): interrupt the session on a mid-run stream/API failure
The non-abort error catch path returned without stopping the session, so a
mid-run network/API failure could leave the Anthropic session running against
the workspace key. Interrupt on that path too — completing the invariant that
every give-up exit (abort, cap, send failure, reconnect cap, stream error)
stops the session. Still fails fast; no retry added.
* fix(managed-agents): skip idless stream previews; resolve service-account provider icons
- Skip idless events in the live SSE handler (mirroring catch-up). event_start/
event_delta previews carry no id, are never deduped, and final text always
arrives as a persisted id-bearing agent.message — so appending previews could
double the block's content output
- Map serviceAccountProviderId to its base provider in PROVIDER_ID_TO_BASE_PROVIDER
so parseProvider resolves 'claude-platform-service-account' to 'claude-platform'
(a two-segment base the hyphen split can't recover), fixing the credential-row
icon falling back to the generic external-link glyph
* fix(managed-agents): keep idless terminals and preserve live pending state
Refine the round-6 idless-event fix, which was too broad:
- Process idless events again (revert the blanket stream skip) so an idless
session.status_idle(end_turn)/session.error delivered only on the live stream
still registers as terminal instead of reconnecting to a timeout. Only the
agent.message TEXT append is now id-gated, so preview text still can't double
- When catch-up history has no lifecycle event, restore the live-observed
requires_action state instead of trusting a stale older agent.message that
cleared it — prevents a false completion with partial output while a tool
result is still pending
* fix(managed-agents): route memory via metadata on self-hosted environments
Live API testing revealed self-hosted environments reject the `resources`
array with a 400 ("resources are not supported with self-hosted
environments"), so the prior universal-resources[] payload would have failed
any self-hosted session that attached a memory store or files.
Restore env-type-aware routing: resolve the environment's config.type via a new
getEnvironmentType() before session create, and for self_hosted send the memory
store through metadata.memory_store_ids/memory_access (the worker consumes it)
and drop file attachments. Cloud environments keep resources[]. Verified end-to-
end against the live Managed Agents API (cloud resources[] 200, self-hosted
metadata 200, self-hosted resources[] 400).
* feat(managed-agents): environment-type selector; hide cloud-only fields on self-hosted
Collapse what #5769 split into two blocks into one, natively:
- Add an Environment type selector (Cloud / Self-hosted) that filters the
environment list to the matching type and gates cloud-only fields
- Memory store, memory access/instructions, and files are cloud-only (self-
hosted rejects the resources[] attach — verified live, 400) and are now hidden
on self-hosted instead of silently dropped. A self-hosted worker that uses a
memory store reads its id from a Metadata key the author sets explicitly
- Expose each environment's config.type on the list options so the picker can
filter by mode; pass the selected type as a routing hint (server still re-
resolves the authoritative type via getEnvironmentType)
* fix(managed-agents): track requires_action by processed_at, not history position
Persisted history can lag the live stream, so the last lifecycle event in the
history array may be OLDER than a requires_action the stream already observed.
Deriving the pending state from history position (findLastLifecycleEvent) could
then clear a newer pause and let an idle snapshot complete a still-waiting
session with partial text.
Track requires_action from the NEWEST lifecycle event by processed_at across
both the live stream and catch-up, so an older/lagging event can never override
a newer pause. Removes the pendingBeforeCatchup snapshot and history-position
recompute. New test covers lagging history holding only an older running event.
* fix(managed-agents): treat missing processed_at as oldest, not newest
A lifecycle event without processed_at mapped to +Infinity, poisoning the
high-water mark: once seen, no later timestamped event could update the pending
state (at >= Infinity always false), stranding a pause or clearing one wrongly.
Map missing/unparseable processed_at to -Infinity so an untimestamped lifecycle
event can never outrank a timestamped one in either direction — it neither
blocks later real events nor clears a timestamped requires_action. Persisted
lifecycle events always carry processed_at; this is purely defensive. New test
covers a stray untimestamped running not clearing a timestamped pause.
|
||
|
Waleed
|
7e6aeb2b7a |
feat(chat): favicon external links with secure link-preview tooltips (#5734)
* feat(chat): favicon external links with secure link-preview tooltips * fix(link-preview): address review findings - allow http fetches to match advertised http(s) link support - fix meta content regex to handle apostrophes and either quote delimiter - hash Redis cache keys so sensitive URLs are not stored verbatim - add per-user rate limit to the outbound-fetching route - render siteName-only previews instead of falling back to the URL * fix(link-preview): redact full URLs from failure logs * improvement(link-preview): render-time preview fetch, cheerio parsing, cleanup pass - fetch previews when links render (emcn tooltip shows instantly — hover prefetch had no delay to race); tooltip reads the warmed cache, eliminating the URL-then-preview flash - parse OG metadata with cheerio (already used server-side) instead of hand-rolled regexes + entity decoding, fixing double-decode and quote-handling classes - drop the no-longer-needed prefetch hook; remove dead side prop on Tooltip.Content - extract ExternalLink to a sibling module per component-size guidelines; fix TSDoc placement * fix(link-preview): https-only previews and full-document parsing - drop allowHttp: plain-http fetches would reach the URL validator's self-host loopback exception; previews are now explicitly https-only on both server (early null) and client (query never fires for http) - parse the full capped document instead of truncating at the first <body> substring, which could match inside head scripts/comments and drop metadata * improvement(chat): render mailto links as plain text |
||
|
Vikhyath Mondreti
|
86e6e1d26c |
feat(forking): excluded workflows (#5727)
* feat(forking): excluded workflows * improve sync preview |
||
|
|
f03b4337fa |
feat(mothership): mixture of models, search agent, persistent subagents, fork chat, inline questions (mothership v0.8) (#5410)
* feat(scout): add scout agent * fix(contracts): update contracts to include scout agent * feat(copilot): search agent (research+scout merge) + read-only table/KB tool handlers Mirrors mothership dev f90f9b05: - regenerated tool-catalog/tool-schemas mirrors (search trigger replaces research + scout; QueryUserTable / SearchKnowledgeBase entries) - queryUserTableServerTool / searchKnowledgeBaseServerTool: read-only wrappers delegating to the full user_table / knowledge_base handlers with hard operation allowlists (and outputPath export rejection on query_user_table) - display maps: 'search' agent label/title/icon added; research + scout entries retained so historical transcripts keep rendering - Search.id replaces Research.id in LONG_RUNNING_TOOL_IDS (it inherits research's long crawls) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(copilot): run_code compute-only handler; docs lint fix Mirrors mothership dev db60da94: run_code is the compute-only variant of function_execute for the search agent — same sandbox and inputs, no outputs.files / outputTable, so it cannot create or overwrite workspace resources. Wrapper handler hard-rejects the write vectors and delegates to executeFunctionExecute; run_code is deliberately absent from OUTPUT_PATH_TOOLS and the table output post-processor, so the name gating blocks writes even for leaked args. Added to LONG_RUNNING_TOOL_IDS, display title/icon maps, and the regenerated catalog/schema mirrors. Also removes two ineffective biome suppression comments in the docs workflow-preview (the rule doesn't fire in the docs app config). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(copilot): failed tool calls must surface their error in terminal data A failed handler result that carried a defined-but-empty output (the app-tool executor's 'Tool not found' ships output: {}) won the priority race in getToolCallTerminalData, so the resume payload's data — the only thing the model reads — was a bare {} with the error text dropped. The search agent retried run_code 20+ times blind against a stale server because every failure rendered as empty instead of 'Tool not found'. Failed calls now always carry error in their terminal data: merged into object outputs, wrapped alongside non-object outputs, preserved when the output already has an error field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(chat): render inline question tags from the agent in chat * fix(chat): let inert multi-step questions browse all prompts * improvement(chat): guard question answer formatting against sparse arrays * chore(copilot): drop user_memory from generated contracts and tool display Companion to mothership 8ae32e97 (user_memory tool removed — the feature no longer exists). Regenerates the mothership contract mirrors via generate-mship-contracts.ts, which also picks up the pending telemetry contract additions (gen_ai.agent.name labels, llm.client.context_tokens, llm.client.compactions, llm.request.compaction_trigger, llm.compaction.pause, gen_ai.usage.context_tokens), and removes the user_memory display title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): answered question card becomes the user turn; two select types only UI ordering: answering a question card no longer echoes a duplicate user bubble. The combined answer still goes on the wire as a user message, but the chat pairs it back to its card (strict 'Prompt — Answer' match, now uniform for single questions too) and renders the card as the answered recap — the card IS the user turn, and the next assistant message streams below it. The pairing is derived from the transcript, so live and reloaded renders are identical; a dismissed card followed by an unrelated typed message does not match and renders normally. Messages ending with a question card also drop the copy/thumbs actions row — the card is an input surface, not a reactable assistant turn. Question types are now single_select and multi_select only: text is removed (the free-text 'Something else' row covers it) and confirm collapses into single_select with Yes/No options. multi_select rows toggle with a check and the free-text row's arrow submits the step; answers are comma-joined labels plus any typed entry. Agent-supplied catch-all options ('Other', 'Something else', 'None of the above') are stripped at parse — the card always provides its own free-text row; a question left with no real options is invalid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): question cards are single_select only Removes multi_select (and its toggle/check UI). The card is one shape: pick one option or type into the always-present 'Something else' row. Catch-all stripping and the transcript pairing/recap behavior are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): bring back multi_select question cards Re-adds multi_select with a reworked interaction: option rows carry real checkboxes (emcn Checkbox chrome) instead of numbers and arrows, an option-styled Submit row confirms the step, and the "Something else" row reads as a plain option until clicked — then it becomes the focused text box, auto-checks, and can be unchecked without losing the typed text (blur with nothing typed reverts it). single_select behavior, catch-all stripping, and the transcript pairing/recap format are unchanged; multi_select answers are the checked labels comma-joined. * chore(copilot): regenerate mothership contract mirror (chat blob span attrs) * chore(copilot): regenerate mothership contract mirror (chat blob metrics) * feat(secrets): make output of generate api key a secret * feat(cli): add mkdir, mv, cp to mship tool set * feat(fork-chat): add fork chat to mothership * fix(fork-chat): fix messageid handling in fork chat * feat(credentials): agent-initiated oauth credential reconnect (#5488) * feat(credentials): agent-initiated oauth credential reconnect * fix(credentials): address reconnect review findings * improvement(credentials): log when connect draft name lookups degrade * fix(conflicts): remove migration * fix(conflicts): fix conflicts * fix(fork-chat): add migrations back * fix(ci): fix lint * fix(ci): fix bad import * fix(vfs): fix 500 char limit in vfs for skills and custom tools * feat(copilot): gate user skills to explicit slash-attach (#5536) Stop the mothership from adopting a workspace user-skill on its own: - Remove the load_user_skill tool and its three payload callers (chat payload, mothership execute route, inbox executor); delete lib/mothership/skills.ts + its test. Skills no longer autoload as the agent's own instructions. - Rename the workspace "## Skills" inventory to "## Agent Block Skills — NOT FOR YOU" with a one-line guardrail so a skill's description (e.g. "respond like a pirate") is not treated as an instruction. Skills reach the model as behavior only via explicit /-attach. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(lots-of-things): lots of things * feat(subagents): add persistent subagents * fix(copilot): let edit_workflow set knowledge-base tag filters, and stop it clearing them (#5546) * fix(copilot): persist KB tag subblocks as JSON strings from edit_workflow The edit_workflow tool normalizes array-with-id subblocks (via normalizeArrayWithIds) but only re-stringifies the keys listed in JSON_STRING_SUBBLOCK_KEYS. `tagFilters` (knowledge-tag-filters) and `documentTags` (document-tag-entry) were missing, so agent-authored tag filters were stored as raw JSON arrays while those UI components read their value with JSON.parse (expecting a string). The result: an agent edit to a Knowledge block's tag filter persisted correctly but rendered as an empty filter in the editor (JSON.parse on an array throws -> []). - Add `tagFilters` and `documentTags` to JSON_STRING_SUBBLOCK_KEYS so edit_workflow stores them in the same shape the UI writes. - Make both components' parsers tolerate an already-parsed array on read, self-healing values already persisted in the broken (array) shape. Search execution was unaffected (parseTagFilters accepts arrays), so the value was never lost — only the editor render and round-trip were broken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): expose KB tag definitions in VFS meta.json Surface each knowledge base's defined tags (displayName -> tagSlot) inline in its meta.json via serializeKBMeta, loaded in one batched query (loadKbTagDefinitions), so the agent can bind a knowledge-tag filter to a real tag slot instead of guessing a tag name it cannot otherwise see. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stringify KB tag subblocks on the nested-node edit path The nested-node merge path normalized array-with-id subblocks but never re-serialized the JSON_STRING_SUBBLOCK_KEYS, so editing a block nested in a loop/parallel container still persisted tagFilters/documentTags (and conditions/routes) as raw arrays -- the exact shape the subblock components cannot JSON.parse. Route all four write paths through a single normalizeSubblockValue helper so the normalize and re-stringify steps cannot drift apart again, and extract the duplicated string-or-array read logic into parseJsonArrayValue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): tighten subblock serialization helpers Derive KbTagDefinitionSummary from the canonical TagDefinition instead of restating its fields, make parseJsonArrayValue generic so callers drop their `as T[]` casts, and unexport the three builders helpers that no longer have consumers outside the module now that normalizeSubblockValue fronts them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stop stripping tagFilters/documentTags from the agent's workflow view sanitizeForCopilot dropped `tagFilters` and `documentTags` from the workflow state the agent reads (workflows/{name}/state.json), while edit_workflow is allowed to write both. The field was therefore write-only: on a follow-up edit the agent read back an absent field, concluded no filter was set, and cleared the user's tag filter. The redaction was introduced for workflow *export* (#1628) and is already enforced there by sanitizeWorkflowForSharing's key list. The duplicate in the copilot-only sanitizeSubBlocks was redundant for export and destructive for the agent. Removes it and pins the contract with a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): reject malformed KB tag values instead of clearing the filter `knowledge-tag-filters` and `document-tag-entry` had no arm in the `edit_workflow` input validator, so they fell through to the pass-through default. Any non-array value the agent supplied -- a double-encoded JSON string, an object, an unparseable string -- reached `normalizeSubblockValue`, where `normalizeArrayWithIds` coerces unparseable input to `[]`. The write path then persisted `"[]"` over the tag filter the user had configured. `condition-input` and `router-input` already guard against exactly this and return an actionable error to the model. Extend that arm to cover the two KB subblock types. It keys on subblock type, so the unrelated `tagFilters` short-input on the Algolia block is unaffected. `null`/`undefined` and empty arrays still clear the field, so intentional clears keep working. Also wrap `loadKbTagDefinitions` in try/catch. Tag definitions are an optional meta.json enrichment, but the query ran inside the top-level `Promise.all`, so a transient failure would reject the entire workspace VFS materialize and leave the agent unable to read any file. Now it degrades to a meta.json without tag definitions, matching the sibling materializers. Adds regression tests for both, plus the first tests for `parseJsonArrayValue`, the helper that keeps pre-fix raw-array rows readable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): collapse duplicate JSON-array parsing in edit-workflow builders `normalizeArrayWithIds` and `normalizeConditionRouterIds` each hand-rolled the same "accept a raw array or the JSON string these subblocks persist" parse. Extract `parseJsonArray`, which returns null when the value is neither, so each caller keeps its own distinct fallback: `[]` for the former, the untouched original value for the latter. Behavior-preserving. An empty array is truthy, so `[]` and `"[]"` still parse through rather than hitting either fallback. `validation.ts` has a third copy, but `builders.ts` already imports from it, so sharing the helper across the two would introduce an import cycle. Left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): specify tag name and legal operators in KB meta.json `tagDefinitions` exposed `displayName`, but a `tagFilters` entry must carry the key `tagName`. An entry written with `displayName` passes validation and persists, then filters nothing -- a silent failure. Rename the field at the serializer boundary; the DB column is untouched. Also emit the operators legal for each tag's `fieldType`, reusing `getOperatorsForFieldType`. `between` is valid for number and date but not for text or boolean, and the agent has no way to infer that. An unrecognized fieldType yields an empty list rather than throwing. Still unspecified, and deliberately out of scope: a filter entry's value key is `tagValue` (but `value` on documentTags), and `between` needs `valueTo`. Those describe the subblock entry shape, not the knowledge base, so meta.json is the wrong place for them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): pass a nullish subblock clear through instead of serializing "[]" `validateValueForSubBlockType` accepts null as an explicit clear, but `normalizeSubblockValue` then ran it through `normalizeArrayWithIds`, which coerces any non-array to `[]`, and persisted the string "[]". No data is lost either way -- "[]" and an absent field both mean "no filters". But it left the field present when the caller asked for it to be unset, so `sanitizeForCopilot` showed the agent an empty filter rather than an absent one, contradicting the absent-means-unset invariant the sanitizer documents. It also made Algolia's `if (params.tagFilters)` see a set value, since "[]" is truthy. An explicitly empty array still serializes to "[]" -- clearing with a value is distinct from clearing by omission. Reported by Cursor Bugbot on #5546. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(changes): huge changes * fix(subagents): lanes * fix(mship): transcript stuff * fix(subagents): thinking lanes * fix(superagent): fix superagent tools and checkpoints * fix(scope): scope subagent tools * fix(lint): fix lint * chore(db): regenerate workspace_files.message_id migration as 0260 on staging base * fix(superagent): fix superagent integration tools * improvement(questions): make something else a placeholder * chore(copilot): regenerate mothership contract mirror after staging rebase * feat(mship): add external mcps to mship * fix(ci): fix dev build * fix(stream): show thinking text * fix(ci): force redeploy * fix(mothership): keep chat forks outside workspace storage billing Preserve the product invariant that Mothership chat files are not charged as workspace file storage after the billing storage merge. * fix(uploads): restore listWorkspaceFiles throwOnError option dropped in rebase * fix(subagent-streaming): remove italics * fix(mothership): treat subagent lanes closed by subagent_end as settled so the between-steps thinking indicator isn't suppressed * fix(ui): thinking loader and rool names * fix(ui): add file * fix(thinking): show thinking during subagents * fix(chat): drop dead thinking-channel ternary after lanes skip thinking blocks * fix(thinking): remove thinking text * improvement(function execute): add timeout to function execute and stop showing text in subagents * fix(subagents): hide thinking text * fix(ff): move ff to go * improvement(superagent): nuke superagent * feat(main-agent): superagent into main agent * chore(db): regenerate workspace_files.message_id migration as 0262 on staging base * fix(credentials): restore reconnect params on shared createConnectDraft * fix(migrations): rebase with staging --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu> Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
68c4f28b77 |
feat(clickup): webhook triggers, hierarchy selectors, and Docs knowledge-base connector (#5708)
* feat(clickup): webhook triggers with auto-managed subscriptions + hierarchy selectors * feat(clickup): KB connector (Docs v3), selector-route hardening, subblock migrations, registry-check regex fix * test(clickup): webhook provider handler tests + redact create-response secret in error logs * fix(clickup-connector): trim final page to maxDocs cap with precise listingCapped semantics * chore(clickup): lint formatting * fix(clickup): restore list-op location requiredness, split listSpaceId migration target, surface failed webhook rollback * fix(clickup): clickup.lists selector accepts listSpaceId context like clickup.folders * fix(clickup): integer-only maxDocs and location filters, depth-aware doc headings, most-specific-location hints |
||
|
Waleed
|
3a632936ab |
feat(clickup): ClickUp integration — 23 tools, OAuth + API-token auth, attachment upload (#5702)
* feat(clickup): add ClickUp integration with OAuth + API-token auth, 23 tools, block, and attachment upload
- 23 tools covering tasks (create/get/update/delete/list/search), comments
(create/get/update/delete), attachment upload, tags, members, custom
fields, and the workspace/space/folder/list hierarchy
- OAuth provider wiring (authorization-code flow, non-expiring tokens) plus
clickup-service-account token-paste credential (personal pk_ API tokens),
with a shared clickupAuthorizationHeader helper (pk_ tokens sent bare,
OAuth tokens as Bearer)
- File upload follows the internal-route pattern: contract-validated
/api/tools/clickup/upload-attachment builds the multipart form and
returns UserFiles
- ClickUp block with per-operation subBlocks, canonical file param,
BlockMeta templates/skills, and gradient brand icon
- Generated integration docs page + hand-written service-account guide
* fix(clickup): apply validation-audit fixes across tools, block, and upload route
- Map documented task fields that were dropped: markdown_description,
subtasks, watchers, custom_fields, time_spent, folder, space — making
the include_subtasks / include_markdown_description options observable
- Expand verified filters: assignees/tags/due-date ranges on get_tasks and
search_tasks, include_closed on search_tasks; add due_date_time /
start_date_time flags and update-task assignee add/remove
- Guard update_comment against an empty body and require comment text in
the block; prefer markdown_content over content on create_list and make
markdown reachable for lists in the UI
- Drop the unverified 'required' field from custom-field outputs; read
both err and error keys from ClickUp error bodies; correct notify_all
wording
- Upload route: 100MB size cap, shared attachment mapper with full
documented response fields (version, thumbnails), base-URL constant
* fix(docs): restore clickup-service-account guide and shield it from doc generation
The generator prunes integration pages it does not derive from blocks;
add the hand-written ClickUp API-token guide to
HANDWRITTEN_INTEGRATION_DOCS so regeneration cannot delete it.
* fix(clickup): address review findings — dedupe catalog entries, config-time list parent validation, upload memory cap, unique icon gradient ids
- Remove duplicated clickup entries in docs meta.json and integrations.json
introduced by a double docs regeneration
- Add a Location dropdown for Get Lists / Create List so the folder ID or
space ID is conditionally required at configuration time instead of
failing at run time
- Pass the 100MB cap into downloadServableFileFromStorage so oversized
files abort during download instead of after full buffering
- Use useId()-derived SVG gradient ids for ClickUpIcon in both icon files
* chore(clickup): format integrations.json entry per biome
* improvement(clickup): final validation-pass refinements across tools and block
- create_task: add doc-backed sprint points param (parity with update)
- get_tasks/search_tasks: expose include_markdown_description
- update_task legacy numeric priority in list responses mapped instead of
dropped; create_comment omits absent response fields instead of
emitting sentinel ''/0 values
- order_by only sent when explicitly chosen (Default sentinel); comment
text no longer UI-required for update_comment (resolve-only and
assignee-only updates are valid per the tool contract, which still
rejects an empty body)
- add_tag_to_task sends no request body per docs; upload tool tolerates
non-JSON error responses
* fix(clickup): tolerate nested user wrapper in member mapping
The task/list member endpoints document a flat member object; accept the
workspace-members-style nested { user: {...} } wrapper as well so both
shapes map correctly.
* fix(clickup): map size-limit errors from download/compile to a 400 upload-size response
downloadServableFileFromStorage enforces maxBytes on both the raw download
and the resolved (compiled) artifact via PayloadSizeLimitError; catch it in
the route so oversized content returns the intended 400 instead of
bubbling to the generic 500 handler.
* feat(clickup): add custom field values, checklists, and time tracking (15 tools, 38 total)
- Set/remove custom field values on tasks (PUT/DELETE /task/{id}/field/{field_id});
block value input parses JSON for structured field types, plain values pass through
- Checklist CRUD: create/rename/reorder/delete checklists and create/update/
delete checklist items (assign, resolve, nest), mapped from the documented
{checklist} response shape
- Time tracking: list entries in a date range (assignee/location filters,
task-tag and location-name includes), create/update/delete entries, start/
stop timers, and read the currently running timer; entries mapped from the
documented data envelope with negative-duration running semantics
- Block gains 15 operations with conditionally-required fields, timestamp
wand configs, tri-state billable/resolved dropdowns, and a single-location
filter selector matching the API's one-location-filter rule
* fix(clickup): new-tools audit fixes — POST for set custom field value, tolerant time-entry envelopes, richer mappings
- Set Custom Field Value uses POST per the live reference OpenAPI (the
llms mirror shows PUT; the reference console spec is authoritative)
- delete_time_entry maps the documented array envelope; create_time_entry
tolerates both data-wrapped and flat echo bodies
- Time entries surface task_tags and task_location so the include switches
are observable; checklists carry date_created
- Custom field value input parses any JSON literal (numbers, booleans,
arrays, objects) and passes plain text through
- Update Time Entry supports duration edits; single-assignee time ops get
their own field so a comma-separated list can't silently NaN out
* fix(clickup): send explicit date-time flags whenever a date is set
The due/start date-time switches previously only transmitted true; a
timed date could never be flipped back to date-only. The flag is now sent
as an explicit boolean whenever the corresponding date is provided and
omitted otherwise.
* improvement(clickup): final per-tool audit polish — checklist item children, tolerant comment date
- Checklist items surface the documented children array of nested item IDs
- create_comment tolerates a string-typed date in the response
* fix(clickup): reject empty update_task bodies with a clear local error, matching sibling update tools
|
||
|
Vikhyath Mondreti
|
54b35a4f0e |
improvement(deployments): bugfixes for run-block, airtable + external sub management (#5680)
* improvement(webhooks): external subscription management
* ui/ux
* remove test file
* fix tests
* address comments
* address comments
* update to grain v2 api
* improvement(grain): hide auto-registered webhook URL on v2 triggers
* Revert "improvement(grain): hide auto-registered webhook URL on v2 triggers"
This reverts commit
|
||
|
|
e2ea49ea7e |
feat(instagram): add Instagram integration (#5568)
* feat(instagram): add Instagram Login OAuth, tools, and block * feat(instagram): add Gmail-style media uploads for publish ops Resolve UserFiles to Meta-fetchable presigned HTTPS URLs (600s TTL) via internal publish routes, and fix OAuth scope storage plus connect-draft wiring so Instagram Login publishing is testable end-to-end. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(instagram): simplify messaging tools to direct requests, clean up types * fix(instagram): parallelize carousel child polling, enforce 2-10 items, extend poll window and insights periods, use canonical user_id in OAuth callback * fix(instagram): resolve user id from user_id only, accept numeric user_id Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): use form/query params for publish and comment endpoints, request message timestamps explicitly Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): normalize Graph ID outputs to strings so downstream .trim() calls are safe Co-authored-by: Cursor <cursoragent@cursor.com> * style(instagram): use brand gradient tile for the block icon Match the official Instagram look by filling the tile with the orange–pink–purple radial gradient so the white camera glyph sits on a full-bleed brand background. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): tighten publish defaults and cloud-storage upload UX Default Reel share-to-feed to Yes, drop unused media fields params, share publish transform helpers, and warn when cloud storage is missing for Meta-fetchable uploads. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): fail closed when cloud storage status is unknown Treat loading/error as blocked for requiresCloudStorage uploads, show the warning once the check finishes, and disable selecting local workspace files Meta cannot fetch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): proactively refresh long-lived tokens before expiry Meta only allows refreshing still-valid Instagram tokens, so refresh within 14 days of expiry (after the 24h age gate) instead of waiting until after accessTokenExpiresAt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(oauth): restore TikTok clientIdParamName JSDoc after merge The staging merge dropped the opening /** on ProviderAuthConfig.clientIdParamName, which broke TypeScript parse in CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(api): Zod-contract storage-status and ratchet validation baseline Wire /api/files/storage-status through a shared route contract so the strict API validation audit stays at zero non-Zod routes after the new Instagram cloud-storage check. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): match Gmail advanced media placeholders Drop public-URL paste hints from advanced fields and the cloud-storage banner so the UI mirrors Gmail attachments. Co-authored-by: Cursor <cursoragent@cursor.com> * code review + hide from toolbar * address comments * fix(instagram): drop hidden Instagram from OAuth catalog pin test Instagram is hideFromToolbar so it is excluded from integrations.json; the pinned slug map must not expect it until the block is visible again. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): validate client ID before creating connect draft Avoid orphan pending credential drafts when INSTAGRAM_CLIENT_ID is missing, matching the Shopify authorize ordering. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Theodore LiandClaude Opus 4.8
|
66d1e61beb |
feat(skills): canonicalize skills to a single source with generated .claude/.cursor projections (#5609)
* improvement(cleanup-skill): parallelize analysis, apply fixes sequentially * improvement(cleanup-skill): add comment-reduction pass; mirror 6 missing skills into .claude/commands * fix(cleanup-skill): substitute parsed scope into analysis passes instead of literal <scope> * fix(cleanup-skill): parse fix token anywhere; preserve pass labels through convergence for ordered apply * fix(cleanup-skill): apply Step 1 proposals content-anchored, re-derive when a prior pass invalidated the snippet * fix(babysit-skill): correct garbled --reverse explanation across all three copies * fix(skills): propagate parallel cleanup to cursor/agents copies; disambiguate babysit /ship refs in claude copy * fix(skills): port url-state + comment passes to cursor/agents; clarify converge pass-label ordering * feat(skills): canonicalize skills under .agents/skills with generated .claude/.cursor projections Establish .agents/skills/<name>/SKILL.md as the single source of truth (latest content reconciled per skill from the three drifted copies), and generate the .claude/commands and .cursor/commands projections from it via scripts/sync-skills.ts. Adds skills:sync/skills:check, a CI gate, a pre-commit regen hook, and CONTRIBUTING docs. Structurally fixes prior drift (e.g. abbreviated .claude ship -> full ship). * fix(skills): strip leaked XML tags from skill tails; clarify lint:check has no per-file target Removes stray </content>/</invoke> markup that leaked into add-block, add-connector, add-hosted-key canonical skills, and reword the cleanup skill's lint step to note bun run lint:check runs repo-wide via turbo (no per-path API). Projections regenerated via skills:sync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjefwescJoHZ6zcc3C17FR * fix(add-block-skill): restore unknown-output stop in Final Validation Re-add the "if any tool outputs are still unknown, tell the user instead of guessing block outputs" step that was dropped when Final Validation step 5 became the BlockMeta template check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjefwescJoHZ6zcc3C17FR --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
04535ee33b |
feat(buffer): add Buffer integration with posts, channels, and ideas (#5637)
* feat(buffer): add Buffer integration with posts, channels, and ideas * fix(buffer): return clear tool error when account lookup yields no account * fix(buffer): default schedulingType server-side so basic-mode blocks never fail validation * fix(buffer): probe Content-Type for extensionless media URLs so videos are not sent as images * feat(buffer): add get_ideas and get_idea_groups tools, harden media URL classification * fix(buffer): guard missing post on PostActionSuccess responses |
||
|
Vikhyath Mondreti
|
ef7c8e24b2 |
feat(platform): settings permissions, admin, billing attribution (#5545)
* fix(invites): preserve active organization for external access Keep organization activation server-owned so failed membership checks cannot clear a valid session context. * feat(admin, billing, settings): cleanup settings visibility, billing actor resolution, new admin routes * address comments * chore(db): reset pending migrations before staging merge Remove locally generated migrations so they can be regenerated against the latest staging schema without preserving stale snapshots or numbering. * regen migrations * address comments * chore(db): reset generated migrations before staging merge Remove this branch's generated migrations so they can be regenerated against the latest staging schema with fresh numbering. * upgrade global work * fix lint * address comments * legacy callbacks correctness * address comments * update * guardrail attribution |
||
|
Theodore Li
|
7962236719 |
improvement(custom-blocks): hardened delete with usage count + per-input required option (#5575)
* improvement(custom-blocks): usage visibility + type-to-confirm delete * feat(custom-blocks): per-input required option * improvement(custom-blocks): replace usage tab with delete-confirmation usage count * fix(custom-blocks): escape LIKE wildcards in usage scan + fresh count on delete modal * fix(custom-blocks): explicit ESCAPE clause on usage-scan LIKE prefilter |
||
|
Theodore LiandClaude Opus 4.8
|
f4d47ed826 |
feat(slack): reusable custom bot credentials, slack_v2 block (preview), redesigned trigger (#5323)
* feat(slack): enable assistant-agent tools via assistant:write scope Add assistant:write, app_mentions:read, and im:history to the Slack bot OAuth scopes so the Set Assistant Status / Title / Suggested Prompts tools (assistant.threads.*) work with users' existing Slack credentials — no new app or credentials required. Restore the action_assistant trigger capability (scope assistant:write) in the manifest generator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpeT8J5yVCrrNQB9Hzm9uS * Add slack trigger * fix channel picker in slack trigger * improvement(slack-trigger): reorder app type, gate account to sim mode, add channel-id input * fix(slack-trigger): drop unmapped events from filter, resolve oauth token for reaction text + file downloads * fix(slack-trigger): empty operation selection fires nothing; resolve token via credential owner not execution actor * fix(slack-trigger): ignore message edit/delete/system subtypes; prefer channel picker over stale manual ids * feat(slack-trigger): single-event model with contextual filters and full event catalog * fix(slack-trigger): apply event/channel/bot filters on custom-app path too * fix(slack-trigger): don't drop edit/delete events when channel_type is absent * feat(slack): reusable custom bot credentials, slack_v2 block, interactivity triggers - Custom bot as a workspace service-account credential (set up once, shared ingest URL /api/webhooks/slack/custom/{credentialId}, reused across triggers and actions) - slack_v2 action block: credential-based Custom Bot auth alongside Sim OAuth; v1 hidden from toolbar - Interaction triggers (block_actions / view_submission) with optional action/callback id filter; settings.interactivity in generated manifests - Setup wizard: name + description, full permissions by default with ChipDropdown customization; reconnect mode rotates secrets in place - Centralized service-account token resolution (unknown provider fails loudly) - Shared Slack webhook fan-out dispatcher for native + custom ingest routes * chore(api-validation): bump route baseline to 924 after staging merge * feat(slack): preview-gate slack_v2 and the custom-bot credential surfaces slack_v2 (block + hosted slack_oauth trigger) ships preview: true — hidden from all discovery until revealed via block-visibility AppConfig or PREVIEW_BLOCKS. v1 stays toolbar-visible with the legacy slack_webhook trigger until v2 GAs. The integrations-page custom-bot setup surface rides the same flag via isHiddenUnder(slack_v2); placed instances, existing credentials, and ingest/execution paths are never gated. * fix(slack): v1 keeps slack_webhook trigger subblocks; handle object-form event channels - v1 spread had been swapped to slack_oauth's trigger subblocks (shared with v2), leaving its slack_webhook deploy path without signing-secret config (Bugbot high). v1 now carries the legacy trigger set again; v2 swaps them for slack_oauth's. - resolveSlackEventChannel reads channel.id for channel_created/channel_rename payloads, so channel filters no longer drop every rename event. * fix(slack): default absent appType to custom at deploy; deactivate custom-bot webhooks on credential delete - appType is hidden and seeded 'custom' by value(), which only covers editor-created blocks; defaultValue now persists it via buildProviderConfig and the deploy fallback flips to custom (the only exposed mode this ship) - deleting a slack-custom-bot credential now also deactivates provider='slack' webhooks routed by that credential id, not just native slack_app rows * fix(slack): resolve credential owner for deploy-time team_id lookup A teammate deploying a trigger wired to a shared Slack credential isn't the credential owner; refreshAccessTokenIfNeeded only loads tokens for the owning user. Resolve the account owner first, mirroring the runtime formatInput path. * chore(slack): reconcile staging merge - nullable webhook.path coalesced at correlation/payload/tiktok boundaries - slack dispatch delegates to staging's dispatchResolvedWebhookTarget (shared preprocess/deployment/filter/enqueue lifecycle), keeping the skip-reason diagnostics; route tests reworked around that seam - api-validation route baseline 924 -> 926 * fix(slack): workspace-scope bot credentials at deploy; recreate webhooks on routing transitions - a bot credential id is semi-public (embedded in Slack Request URLs), so the custom deploy branch now rejects credentials outside the workflow's workspace - needsRecreation also compares path/routingKey, so a row from an older routing model can't survive redeploy as a stale delivery surface * test(slack): pin fail-closed behavior for empty/missing event selection * fix(slack): 409 on custom-bot name collision instead of silently returning the existing credential The service-account dedupe matches on displayName, which defaults to the Slack team name — shared by every bot in that workspace. A second unnamed bot create returned the first credential as success, orphaning the new id already pasted into the Slack Request URL. Same-id replays stay idempotent; different-id collisions now fail loudly so the wizard prompts for a distinct name. * fix(slack): reconnect surfaces Atlassian error codes and persists name/description edits - PUT credential route now returns the Atlassian provider code (providerErrorCode -> code) so reconnect failures map to specific token/domain messages, matching create - Google/Atlassian reconnect send + seed displayName/description (parity with Slack); edits are no longer silently discarded, and empty fields don't clobber existing values * fix(slack): require bot name; propagate rotated bot_user_id to webhooks on reconnect - the setup wizard now requires a bot name (canAdvance), so the credential name, manifest app name, and uniqueness key all use the user's choice instead of the shared Slack team-name fallback that collided for a second bot in one workspace - reconnect that changes the bot user id (recreated Slack app) now updates the bot_user_id cached in each bound webhook's providerConfig, so reaction self-drop keeps working instead of letting the bot's own reactions re-enter --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
5d3809a0e3 |
feat(tiktok): add tiktok trigger, block (#5504)
* feat(tiktok): add TikTok integration Adds TikTok as a full OAuth-based integration: provider registration (with TikTok's comma-separated scope and client_key requirements), 9 tools covering profile info, video listing/querying, creator info, direct video/photo posting (URL or file upload), inbox drafts, and post status polling, plus the TikTok block, icon, and generated docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): add avatarFile output to Get User Info Adds a file-typed avatarFile output (sourced from the largest available avatar URL) alongside the existing string avatar fields, so the profile picture can be materialized as a UserFile and chained into file-consuming blocks (e.g. attached to an email), per PR review feedback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): lower upload memory cap, drop redundant avatar string outputs Cap the file-upload video buffer at 250MB instead of TikTok's 4GB ceiling — relaying that much through this server's memory per request isn't safe under concurrent load, and larger files can still go through the PULL_FROM_URL path, which never buffers on our server. Also drop the now-redundant avatarUrl/avatarUrl100/avatarLargeUrl string outputs from Get User Info in favor of the file-typed avatarFile output alone, since the feature is unreleased and the raw URL is still reachable via avatarFile.url. Cover image URLs on List/Query Videos are confirmed to be signed, expiring TikTok CDN links; left as strings (no file-output conversion path exists for fields nested inside array items) but documented the expiry behavior more clearly. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ci): bump API validation route-count baseline for TikTok publish-video route The TikTok integration adds one new Zod-backed internal API route (app/api/tools/tiktok/publish-video), which trips the route-count ratchet in check-api-validation-contracts.ts. Bumping totalRoutes and zodRoutes from 917 to 918 (nonZodRoutes stays 0) to acknowledge the new route is properly validated. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): drop unused avatar_url_100 from default user fields After removing the avatar string outputs, avatar_url_100 was still requested from TikTok's user info endpoint but never surfaced anywhere. Removed it from the default field list and the field descriptions, and noted that avatar_url/avatar_large_url feed the avatarFile output. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop returning raw 'credential' subBlock id from tools.config.params The block's params function built a local `credential` variable from params.oauthCredential and returned it under the key `credential` in every switch case. That literal token is the raw subBlock id, which is deleted after canonical transformation into `oauthCredential` — the blocks.test.ts canonical-param-validation suite flags any params function that still references it. It was also redundant: oauthCredential is already part of the base resolved inputs, which the executor merges into the tool call before config.params overrides are applied, so the OAuth token resolution (which reads contextParams.oauthCredential) worked regardless. Removed the explicit credential plumbing, matching the convention already used by other OAuth blocks like dropbox.ts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): send empty JSON body on Query Creator Info POST query_creator_info had no request.body function, and formatRequestParams() only attaches a body when tool.request.body is defined at all — so despite sending Content-Type: application/json, the request went out with no body whatsoever. Added body: () => ({}), matching the convention already used by other parameterless-POST tools in this codebase (Google Vault, Supabase, Square, Gmail, etc.). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop dropping valid zero values in optional numeric fields cursor, photoCoverIndex, and videoCoverTimestampMs all used a truthy check (params.x && {...}) to decide whether to include an optional numeric override, which drops a legitimate 0 (first page has no cursor issue aside, photoCoverIndex 0 is TikTok's own default cover photo, and timestamp 0 is a valid first-frame cover). Switched to explicit undefined/empty-string checks, matching the !== undefined convention the underlying tools already use. In today's resolution pipeline these fields always arrive as strings (even chained block references get stringified by the template resolver), and a non-empty string like "0" is truthy, so this wasn't actively broken end-to-end - but it was relying on that subtlety rather than being correct by construction, and was inconsistent with the tools' own undefined checks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): accept newline-separated video IDs in Query Videos videoIds is a long-input (multiline textarea), the same widget used for the newline-separated photoImages field on this block, but its parser only split on commas. Entering one ID per line - the natural pattern for a multiline field, and the one already used elsewhere on this block - produced a single concatenated garbage string instead of an array, so TikTok's query would fail or return nothing. Now splits on commas or newlines, and updated the placeholder/description to reflect both formats. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(tiktok): add app-level webhook ingress and triggers * fix(tiktok): only count actually queued webhook executions Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): bump API validation baseline for staging merge Co-authored-by: Cursor <cursoragent@cursor.com> * cleanup code * fix type issues * misc code cleanup * remove photos and add upload for videos * move shared video output properties to types.ts so docs generation resolves them Co-authored-by: Cursor <cursoragent@cursor.com> * hide TikTok from toolbar and docs until the integration is ready to ship Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): ratchet API validation baseline to 924 after staging merge Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Theodore Li
|
5b7513f15d |
feat(blocks): add block visibility gating (preview blocks + AppConfig reveals) (#5526)
* fix(deps): install xlsx from @e965/xlsx npm mirror The dependency was pinned to a direct tarball on cdn.sheetjs.com, which now returns 403 (Cloudflare bot-challenge) to automated clients, breaking bun install in CI. npm's own xlsx is frozen at 0.18.5, so switch to the @e965/xlsx mirror which republishes the identical 0.20.3 CDN build to the npm registry. No code changes needed — all imports use bare 'xlsx'. * feat(blocks): add block visibility gating (preview blocks + AppConfig reveals) * fix(blocks): reset visibility to fail-closed empty state on workspace switch * fix(blocks): carry kill-switch entries across workspace-switch visibility resets * chore(deps): revert stray local xlsx-mirror commit (keep staging's pinned source) * chore(skills): rename gate-block skill to add-block-preview |
||
|
Waleed
|
6e87d74a93 |
feat(jupyter): add Jupyter integration (contents, kernels, sessions) (#5527)
* feat(jupyter): add Jupyter integration (contents, kernels, sessions) - 16 tools covering Contents, Kernels, Kernelspecs, and Sessions REST APIs - File upload/download via UserFile, following the Box upload pattern - Block with operation dropdown, token auth, and 8 catalog templates - Registered tools + block, generated docs, bumped API validation baseline * fix(jupyter): address Greptile review — SSRF guard, upload path ambiguity, silent notebook fallback - Route uploads through validateUrlWithDNS + secureFetchWithPinnedIP (matches Grafana/1Password pattern) instead of a raw fetch to the user-supplied server URL - Replace the upload path/trailing-slash heuristic with an unambiguous directory + filename split - create_file no longer silently writes an empty notebook when notebook content is malformed JSON — it now errors clearly * fix(jupyter): request content=1 when listing directory contents Without it, Jupyter Server returns directory metadata with content: null, so jupyter_list_contents always reported an empty items array. * fix(jupyter): reject path-traversal segments in Jupyter content paths encodeJupyterPath now rejects '.'/'..' segments across the whole path (shared by all 16 tools, not just upload); the upload route returns a clean 400 when it's hit. * fix(jupyter): close remaining path-traversal and redirect-credential gaps - extract the traversal check out of encodeJupyterPath into a shared assertion, and apply it to body-only path fields (rename newPath, copy copyFromPath, session path) that never flowed through URL encoding and so skipped the check - pass stripAuthOnRedirect to the upload route's secureFetchWithPinnedIP call so a malicious Jupyter server can't redirect the PUT to another origin and receive the caller's token * fix(jupyter): also reject percent-encoded traversal segments A segment like %2e%2e wouldn't match the literal '..' check. Now decodes each segment before comparing, in addition to the literal check, so an already-encoded traversal attempt is caught too. * fix(jupyter): route all 15 remaining tools through an internal proxy for HTTP/private-host support and no redirects The generic external tool executor blocks plain-HTTP and non-localhost private-IP hosts by default, so every non-upload Jupyter operation could fail against typical self-hosted setups (LAN IP, docker hostname, or even literal localhost on a hosted deployment) even though the upload route worked via its own internal route. Added /api/tools/jupyter/proxy (DNS-pinned, allowHttp, maxRedirects: 0) that mirrors the upstream Jupyter response verbatim, matching the established pattern for self-hosted-arbitrary-host integrations (Grafana, 1Password) instead of the generic executor path. Each tool's request block now posts to the proxy instead of building a direct external URL; transformResponse and outputs are unchanged since the proxy response mirrors upstream status/body exactly. Also switches the upload route from stripAuthOnRedirect to maxRedirects: 0 — stronger, since it stops the uploaded file body (not just the token) from ever reaching a redirect target. * fix(jupyter): validate proxy path at the trust boundary, reject path separators in upload filename - The proxy route now independently validates the incoming path field for traversal segments instead of only relying on tool-side validation before the request reaches it — the route is a shared internal boundary, not something only our own tool code can call - The upload route's fileName can come from an advanced override or the legacy fileContent path and could itself contain '/' or '\', silently nesting the upload deeper than the directory param specified. Now rejected outright before joining. * fix(jupyter): decode the whole path before splitting, not per-already-split segment A segment like foo%2f..%2fsecret has no literal slash, so splitting on literal '/' first and decoding each piece in isolation treats it as one opaque segment and never notices the '..' hiding behind the encoded slash. Decode the full path once, then split and check every segment the target server's own single URL-decode pass would see. |
||
|
Vikhyath MondretiandSiddharth Ganesan
|
4a80374c03 |
improvement(forking): unlink, settings page migration, deployed chats, public apis, external mcp servers/tools, special subblocks (#5505)
* bad checkpoint * stash * add unlink, move UI to settings pages * fix edge cases * remove dead code * remove migration 0255 ahead of staging merge (regenerated after) * regenerate workflow_mcp_server enum migration on top of staging (0257) * fix tests * more tests * address comments * consolidate fork migrations into 0257 (enum value + activity metadata indexes) * acquire MCP server locks before reads in attachment reconcile (TOCTOU) * move into ee folder + ui perm gates * use randomInt from @sim/utils/random for chat identifier suffix --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com> |
||
|
Waleed
|
9d34fbea1b |
refactor(utils): consolidate duplicated helpers onto @sim/utils (#5509)
* refactor(utils): consolidate duplicated helpers onto @sim/utils Replaces ~90 hand-rolled reimplementations of error-message extraction, postgres error-code checks, sleep, Math.random, retry/backoff, object filtering/omission, noop, string truncation, date/time formatting, email normalization, and plain-object type guards with the shared @sim/utils exports. Wires check:utils into CI (test-build.yml) so these patterns don't regress. * fix(test): mock @sim/utils/random instead of Math.random in schedule-execute tests The schedules/execute route previously used Math.random() for jitter delay; this consolidation PR switched it to randomInt() from @sim/utils/random, which is backed by crypto.getRandomValues() rather than Math.random(). The route.test.ts spies on Math.random() no longer had any effect, so jitter became real random delay instead of the deterministic 0ms the tests expect, causing intermittent 10s timeouts in CI. * fix(retry): preserve uncapped Retry-After comparison in tools/index.ts parseRetryAfter() caps its return value at 30s by default. tools/index.ts compares the parsed Retry-After against a caller-configured maxDelayMs to decide whether to skip a retry entirely -- capping before that comparison silently defeats the skip check whenever maxDelayMs is configured above 30s, since a Retry-After between 30s and maxDelayMs would incorrectly look "within limits" and get retried instead of skipped (caught by Cursor Bugbot). Added an optional maxMs param (default unchanged) so tools/index.ts can request the raw, uncapped value for its own comparison while backoffWithJitter still clamps the actual sleep duration to maxDelayMs. Added a regression test covering maxDelayMs > 30s. * fix(utils): fall back to Intl-resolved abbreviation for unmapped timezones getTimezoneAbbreviation only covered 9 hardcoded IANA zones and returned the raw IANA string for everything else, degrading schedule descriptions for zones like Europe/Berlin or America/Toronto (caught by Greptile). The deleted local implementation in schedules/utils.ts resolved any valid IANA timezone generically via Intl.DateTimeFormat's short timeZoneName. Restore that as a fallback so only genuinely invalid timezone strings return themselves unchanged. |
||
|
|
8fcce5100c |
fix(aws): align cloudwatch, cloudformation, athena, codepipeline with live API docs (#5483)
* fix(aws): align cloudwatch, cloudformation, athena, codepipeline with live API docs - cloudwatch: fix list_metrics pagination (wasn't draining pages past 500), add MaxRecords cap validation to describe_alarms; add describe_alarm_history, filter_log_events, put_log_group_retention - cloudformation: fix get_template missing TemplateStage param, fix invalid ModuleTag in block metadata; add full stack lifecycle tools (create/update/delete/cancel_update_stack, create/describe/execute_change_set, get_template_summary) - athena: fix missing .trim() on query/named-query ID fields; add delete_named_query, batch_get_query_execution, list_databases, list_table_metadata - codepipeline: fix missing rollbackMetadata field in list_pipeline_executions response; add get_pipeline, list_action_executions, disable/enable_stage_transition * fix(aws): require template on cloudformation change-sets, bump route-count baseline - cloudformation create_change_set now rejects requests missing both templateBody and usePreviousTemplate, matching update_stack (Cursor Bugbot finding) - bump check:api-validation route-count baseline 906->917 to reflect the 19 new fully contract-bound routes added in this PR (0 boundary violations) * fix(aws): address Greptile round-1 review findings - cloudwatch describe_alarm_history: always request both MetricAlarm and CompositeAlarm types, even when alarmName is provided (was silently returning empty history for composite alarms queried by name) - cloudformation: add validateAwsRegion refinement to region field on the 7 new write-path contracts (update/delete/cancel-update-stack, create/describe/execute-change-set, get-template-summary), matching the pattern already used elsewhere - cloudformation: destroy the AWS SDK client in a finally block on the same 7 new write routes, matching the pattern used by every other new route in this PR * fix(aws/cloudformation): add missing region validation and client cleanup to create-stack - Add validateAwsRegion refinement to create-stack contract (completes P2 fix from Greptile review) - Wrap AWS SDK call in try/finally with client.destroy() (completes P2 fix from Greptile review) - Aligns create-stack with the pattern used across all 7 other new CloudFormation routes Co-authored-by: Waleed <waleedlatif1@users.noreply.github.com> * fix(aws): final validation pass — bound missing limits, close consistency gaps - cloudformation create_stack: add missing validateAwsRegion refine and client.destroy() finally block, matching sibling write routes - cloudwatch get_metric_statistics: cap statistics array at AWS's 5-item limit - cloudwatch get_log_events: cap limit at AWS's 10,000-record max - athena batch_get_query_execution: surface engineExecutionTimeInMillis/queryPlanningTimeInMillis/queryQueueTimeInMillis, matching the sibling get_query_execution tool's Statistics mapping * fix(aws/athena): destroy AWS SDK client on the 4 new athena routes Cursor Bugbot finding: batch_get_query_execution, delete_named_query, list_databases, and list_table_metadata created an AthenaClient but never called client.destroy(), unlike every other new route in this PR. Wrapped each in try/finally to match. * fix(aws/athena): add validateAwsRegion refinement to the 4 new contracts Greptile finding: delete_named_query, batch_get_query_execution, list_databases, and list_table_metadata accepted any non-empty string for region, unlike every other new contract in this PR. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Waleed <waleedlatif1@users.noreply.github.com> |
||
|
Waleed
|
b686111082 |
feat(textract): migrate to AWS SDK, add AnalyzeExpense and AnalyzeID (#5456)
* feat(textract): migrate to AWS SDK, add AnalyzeExpense and AnalyzeID - Replace hand-rolled AWS SigV4 signing with @aws-sdk/client-textract, matching sibling AWS integrations (secrets_manager, s3, sts) - Add Analyze Expense operation (invoice/receipt structured extraction) via AnalyzeExpense/StartExpenseAnalysis+GetExpenseAnalysis - Add Analyze Identity Document operation (AnalyzeID) with optional back-of-ID page - Add an operation selector to the textract_v2 block; defaults to the existing Analyze Document behavior for backward compatibility - Add tests for tool body/response mapping and route-level AWS response normalization * fix(textract): forward URL documents and fix ambiguous error status - textract_analyze_expense/textract_analyze_id now fall back to filePath/filePathBack when the document input is a URL string rather than an uploaded file object, so advanced "File reference" URL inputs actually reach the API (Cursor Bugbot) - mapTextractSdkError defaults to 500 (not 400) when the AWS SDK error has no HTTP status, since that implies a server-side/network failure rather than a bad request (Greptile) * fix(textract): stop stale processingMode from hiding ID document fields - Front-document fields (fileUpload/fileReference) are shared across all 3 operations; gate them with a values-aware condition so switching to Analyze Identity Document keeps them visible even if a stale processingMode='async' is left over from a previous operation - S3 URI field now also requires operation !== 'analyze_id', since that operation never supports S3 input * fix(textract): preserve first-page metadata across async pagination pollTextractJob's merge callbacks spread only the latest page, dropping any field (DocumentMetadata, model version) the first page had but a follow-up NextToken page omits. Merge accumulated first so later pages only override fields they actually return. * fix(textract): pass through the real upstream status for filePath fetch failures fetchDocumentBytes hardcoded 400 for any non-OK response from a document URL, masking transient 5xx failures from the document host as client errors and blocking tool-execution retries. Use the actual response status instead. * chore(textract): drop redundant inline comments |
||
|
Vikhyath Mondreti
|
24ebba9acc | chore(credential-sets): cleanup feature (#5460) | ||
|
Waleed
|
fb3f95d5dc |
feat(aws): expand SES/STS/Secrets Manager tool coverage, fix API alignment gaps (#5450)
* feat(aws): expand SES/STS/Secrets Manager tool coverage, fix API alignment gaps
- SES: add suppression list management, email identity CRUD, template
update, configuration set creation, custom verification email (10
new tools); fix silent httpsPolicy drop in create_configuration_set
and unvalidated suppression reason enum in list_suppressed_destinations
- STS: add AssumeRoleWithWebIdentity and AssumeRoleWithSAML (unsigned,
no static credentials required); extend assume_role with
policyArns/tags/transitiveTagKeys session params
- Secrets Manager: add describe_secret, tag_resource, untag_resource,
restore_secret, rotate_secret; fix list_secrets dropping
rotation/version metadata fields; normalize tool versions to 1.0.0
and alphabetize registry entries
All 31 tools verified param-by-param against live AWS API docs across
two independent audit passes.
* fix(aws): address review findings on SES config/identity and Secrets Manager rotation
- ses_create_configuration_set: validate suppressedReasons against
BOUNCE/COMPLAINT enum before calling AWS (was silently reaching AWS
as a generic 500 for bad values); tags now a proper Zod array schema
instead of a string with route-side JSON.parse
- ses_create_email_identity: dkimSigningAttributes and tags now proper
Zod object/array schemas instead of strings with route-side
JSON.parse, matching the pattern used elsewhere (e.g. sts_assume_role
tags, secrets_manager_tag_resource)
- secrets_manager_rotate_secret: reject automaticallyAfterDays and
scheduleExpression when both are supplied — AWS RotationRules
accepts only one
- sts createUnauthenticatedSTSClient: corrected a misleading comment
claiming these calls are fully unsigned; the SDK still falls through
its default credential provider chain
* fix(ses): correct json input types for tags/dkimSigningAttributes
The SES block declared the tags and dkimSigningAttributes block inputs
as 'string' instead of 'json', so the generic block executor never
parsed the JSON code-editor value before forwarding it — workflow runs
sent a raw JSON string where the contract now expects a structured
object/array, failing validation. Also corrected the corresponding
tool param TypeScript types, which were still typed as string | null.
* fix(ses): stop coercing switch string 'false' to true in create_configuration_set
Boolean('false') evaluates to true, so turning off the
reputationMetricsEnabled or sendingEnabled switch sent the opposite of
the user's choice to SES. Match the established === 'true' string
comparison pattern used elsewhere in the codebase.
* fix(sts): stop double-parsing assume_role session tags input
tags was declared as a 'json' block input, so the generic executor
JSON.parse'd it before the switch-case handler ran — but that handler
already converts the raw table-rows array (or a passthrough string)
into the JSON string the sts_assume_role contract expects. Declaring
it 'json' broke that conversion for non-string inputs. Reverted to
'string' so the handler's existing string/array disambiguation runs
on the untouched raw value.
* fix(sts): supply placeholder credentials to the unauthenticated client
createUnauthenticatedSTSClient omitted credentials entirely, so the
SDK's signing middleware fell through the default credential provider
chain and threw CredentialsProviderError before the request was sent
in any environment with no ambient AWS identity — even though
AssumeRoleWithWebIdentity/AssumeRoleWithSAML never check the
signature. Static placeholder credentials skip that resolution
without granting or requiring any real IAM identity.
|
||
|
Theodore Li
|
82eff5435d |
feat(custom-block): deploy a workflow as a reusable org-scoped block (#5407)
* feat(custom-block): deploy a workflow as a reusable org-scoped block * fix(custom-block): reseed deploy form, guard duplicate publish, run child deployed * test(custom-block): isolate custom-block rows fetch in execution-core test * fix(custom-block): allow cross-workspace exec, org-scope authority, keep field ids, hide disabled * feat(custom-block): run child under source owner's identity, workspace, and env * fix(custom-block): bind publish authz to the source workflow's workspace * fix(custom-block): gate edit/delete on source-workspace admin, not org admin * chore(custom-block): rebaseline route count to 887 after staging merge * fix(custom-block): sanitize failure output so it can't leak source workflow internals * fix(custom-block): derive inputs and curated outputs from deployed state, not draft * fix(custom-block): hide disabled blocks from the toolbar palette too * fix(custom-block): bill nested + failed-run hosted cost; expose real inputs to the agent * fix(custom-block): enforce enterprise + flag gate at every consumption path |
||
|
Waleed
|
759dddbf4c |
feat(billing): dedicated Credit usage page with date-range filter and CSV export (#5405)
* fix(billing): apportion per-row credit costs so they sum to the page total Cursor Bugbot (medium): each row rounded its own dollar cost to credits independently while the header total rounded the summed dollars once — over enough rows those two roundings can visibly disagree, the exact "line items don't add up to the total" class of bug apportionCredits was already built to prevent (used by the trace view / cost breakdown). Route now apportions each page's row credits against that page's dollar sum instead of rounding rows independently. Added a test with three sub-cent rows that would each independently round to 0 credits (but sum to 1) to prove the reconciliation holds. * fix(billing): dim stale credit usage rows while a new period loads Cursor Bugbot (medium): keepPreviousData kept the prior period's rows and total on screen while a newly selected period fetched, but the dropdown label updated immediately — so during the transition the displayed numbers were labeled under a period they didn't belong to. Now reads isPlaceholderData (the standard TanStack Query signal for "this data is a stale placeholder, not a fresh fetch for the current key") and dims the list while it's true, matching the same flag already used for this exact purpose in integration-skills-section.tsx. * fix(billing): show "<1 credit" for rows apportioned to 0 Cursor Bugbot (low): with apportioned per-row credits, a row with a real but sub-credit dollarCost can legitimately apportion to 0 credits once a sibling row absorbs the shared rounding remainder — rendering a flat "0 credits" reads as if nothing was charged, inconsistent with formatCreditCost's "<1 credit" wording used elsewhere in billing. Added dollarCost to the wire response (needed to distinguish a genuinely free row from a rounded-to-zero one) and a small formatRowCredits helper that only changes the label, not the underlying creditCost number, so the page-total reconciliation from the prior fix is unaffected. * fix(audit-logs): fix broken Custom range picker, trim time-range presets Custom range silently did nothing: the time-range trigger was a ChipSelect (Radix DropdownMenu, modal by default), and selecting "Custom range" opened the Calendar popover in the same tick the modal menu began its close/focus-lock cleanup, trapping the popover non-interactive. Swapped to ChipCombobox (Radix Popover, non-modal), mirroring the already-working pattern in the main Logs page exactly. Also trimmed the preset list from 11 to 8 entries (dropped Past 30 minutes/12 hours/14 days) so the menu fits without scrolling. * feat(billing): dedicated Credit usage page with date-range filter and CSV export Follow-up to #5391 per team feedback in Slack: move the credit usage list out of the inline Billing section into its own page, redesign rows to show source ("Chat", "Workflow: <name>") instead of a raw model description + badge, and add real date-range filtering and export. - Billing settings now shows a compact glance (30-day total + a "View usage logs" link) instead of the full inline list. - New /settings/billing/credit-usage page (sibling of [section], mirrors the secrets/[credentialId] detail-route pattern) with day presets (Today/7d/30d/All time) plus a working Custom range picker — the same ChipCombobox+Popover+Calendar wiring the audit-logs fix in this branch uses, not the broken ChipSelect pattern. - Rows show the humanized source label, or "Workflow: <name>" for workflow-sourced events (new server-side workflow-name lookup, batched per page). Dropped the redundant badge and raw model description. - CSV export of the currently-filtered logs via a new GET .../usage-logs/export route (mode: 'text' contract, synchronous single-response CSV — the dataset is a bounded per-user ledger, not a workspace-wide export, so no async job queue needed). Query-filter logic (date-range resolution, workflow-name lookup) is shared with the list route via shared.ts rather than duplicated. - period/startDate/endDate live in the URL via a co-located search-params.ts; the list query keeps keepPreviousData + isPlaceholderData dimming during filter transitions, matching the behavior already shipped in #5391. Verified live end-to-end: back link navigation, custom range picker opens and applies, day presets, CSV export downloads and matches the on-screen rows exactly (credits reconcile with the total), compact Billing summary + link. * refactor(billing): move workflow-name enrichment into getUserUsageLogs, dedup helpers /simplify pass over the credit-usage-page branch (4 parallel review angles: reuse, simplification, efficiency, altitude): - getUserUsageLogs now LEFT JOINs workflow and returns workflowName directly (matching lib/logs/list-logs.ts's established pattern), eliminating the route-layer resolveWorkflowNames query that both the list and export routes previously ran independently. - Added includeSummary (default true) to getUserUsageLogs so the export route's cursor loop can skip the cursor-independent SUM/GROUP BY aggregate it never reads — that aggregate was being recomputed on every page of a paginated export for no reason. - Fixed an off-by-one in the export's pagination loop: `<= MAX_EXPORT_ROWS` let it fetch one more full page past the cap only to discard it; `< MAX_EXPORT_ROWS` with a shrinking per-page limit never overshoots. - Deduplicated the SOURCE_LABELS map (was defined identically in both the page and the export route) into a shared, DB-free source-labels.ts both can import. - Export route now builds CSV rows via lib/table/export-format.ts's toCsvRow/formatCsvValue instead of a hand-rolled escaper. - Added formatApportionedCreditCost to conversion.ts so the page's row rendering shares its zero/sub-credit wording with formatCreditCost instead of re-deriving the same three-way branch. - Replaced the generic requireStartDateForCustomPeriod<Schema> contract helper (nontrivial generic bound for a single four-line refine used at two call sites) with a plain shared error-options object. - Removed the credit-usage page's dateRangeAppliedRef guard — a controlled Radix Popover never re-invokes onOpenChange in response to the parent's own setState call, so the guard was defending against a re-entrant close that can't happen. - Added a modal prop to ChipSelect (forwarded to the underlying DropdownMenu, which already supported it) so a future call site that hits the same "modal select traps a same-tick Popover" bug the audit-logs Custom range fix worked around has a real fix available instead of having to swap components again. Re-verified live end-to-end after the refactor: workflow-name resolution, credit reconciliation, and CSV export all still correct. * fix(billing): drop Dollar cost from the CSV export, strip inline comments We only surface credits to the user, not the underlying dollar figure — "Dollar cost" was the one place the export literally displayed a dollar amount (the rest of the codebase uses dollarCost purely as an internal signal to distinguish a sub-credit charge from a genuinely free event, never rendered as a "$" value). * fix(billing): export honors partial custom date range, surfaces truncation Greptile (P1) and Cursor Bugbot independently caught the same bug: handleExport only forwarded startDate/endDate when BOTH were truthy, but the list query and both API contracts treat endDate as optional for a custom period (defaults to now). A user landing on a bookmarked ?period=custom&startDate=... URL would see populated rows and an enabled Export button, then get a 400 on click since the export omitted the required startDate too. Fixed by forwarding each date independently, matching the list query's existing behavior. Also addressed Greptile's other two findings: - The export route now sets X-Export-Truncated so a 5,000-row-capped download is visible to the user (a toast), not just a server log. Reading that header meant switching the trigger from a plain anchor navigation to fetch+blob — an anchor can't inspect the response before the browser commits to the download. - resolveDateRange now throws explicitly when a custom period is missing startDate instead of silencing the null check with `as string`, which would have produced a silent Invalid Date if ever called without prior contract validation. * fix(billing): remove the export's arbitrary row cap, fix a cursor pagination bug it exposed A personal credit ledger doesn't have the same unbounded-growth problem a workspace table does — capping the export at 5,000 rows just meant long-tenured or high-usage accounts (exactly the ones most likely to need a full export to reconcile a billing question) got silently truncated. Replaced the cap with a 50,000-row circuit breaker that should never fire in normal use (logged as an error, not a warning, if it ever does) and bumped the page size from 500 to 1,000 to cut round trips. Removing the cap surfaced a real, pre-existing bug in getUserUsageLogs's cursor pagination: a raw `sql` template embedded a JS Date object directly as a bound parameter, which the postgres driver can't serialize (unlike drizzle's typed gte/lte operators, which already handle Date correctly elsewhere in the same function). It only ever manifested past the first page, which nothing before this export route's tight multi-page loop reliably exercised. Replaced the raw sql template with drizzle's typed lt/eq/or/and operators, matching the pattern already proven correct in this file. Verified live: seeded 6,000 rows (past the old cap) and confirmed the export downloads all of them in one request with credits reconciling exactly against the total. * perf(billing): skip the redundant cursor lookup when the caller already has it The export loop holds the previous page's rows in memory, so its next cursor's createdAt is already known — getUserUsageLogs was still re-resolving it via an extra DB round trip every page regardless. Added an optional cursorCreatedAt to skip that lookup when provided; the list route's existing callers are unaffected since they don't pass it. Verified live: zero cursor-lookup queries fired across a 3,500-row / 4-page export that previously issued one per page. * fix(billing): apportion credits over the whole filtered set, not per page/call Cursor Bugbot caught this: the list route apportioned each page's rows against only that page's own dollar total, while the export apportioned every exported row against the complete set's total. Since apportionment depends on the full set, the same log could show a different creditCost between the list and the export, or even between two pages of the same "Load more" list — and the sum of every loaded row could visibly drift from the "Total" header shown above them once more than one page had loaded. Extracted getUsageCreditsByLogId — a single, shared, whole-filter apportionment lookup both routes now call instead of each computing their own subset locally. The list route calls it once per page request (same cost profile as the summary aggregate it already pays for every page); the export calls it once before its pagination loop, not per page, keeping the round-trip count this session's earlier fix already reduced. Also extracted the condition-building shared by the main query, the summary aggregate, and this new lookup into one buildUsageLogConditions helper, removing a third copy of that logic. Verified live: summed every row across 4 "Load more" pages and confirmed it now matches the reported total exactly (previously could drift), and confirmed the list and the export produce byte-identical credit sequences for the same rows. * fix(billing): make custom-range startDate/endDate nullable, not '' defaulted startDate/endDate had no sensible static default (they're only ever meaningful mid-custom-range), so defaulting them to '' via .withDefault('') meant switching back to a preset left the URL carrying startDate=&endDate= instead of dropping the params entirely. Made them nullable (no .withDefault) instead, matching the identical fields in the main Logs page's own search-params.ts. Verified live — switching from a custom range back to a preset now clears both params from the URL completely. * feat(audit-logs): add CSV export, matching the Credit usage page pattern Adds an Export chip to the top-right of the Audit Logs page (via SettingsPanel's actions slot — the same header mechanism the Credit usage page uses), downloading every audit log matching the current search/type/date filters as CSV. - New GET /api/audit-logs/export route: same session + enterprise admin/owner gating as the existing list route, reuses the shared buildFilterConditions/buildOrgScopeCondition/queryAuditLogs helpers (already using drizzle's typed operators for cursor pagination, not the raw-sql-with-embedded-Date pattern fixed elsewhere this session), and the same fetch+blob+X-Export-Truncated pattern the Credit usage export already established. - Capped at 10,000 rows (not the 50,000 used for a personal credit ledger) — an org's audit trail can genuinely grow much larger than one user's usage history, so this is sized for "a reasonable audit review window," with truncation surfaced via a toast rather than silently dropped. - Bumped the API-validation-contract audit's route-count baseline for the new route. Verified live against a real enterprise org: switched to "All time," exported ~750 real audit log rows, confirmed formatting (quoted descriptions, actor email fallback) and correct filter scoping. * fix(billing): skip wasted credit apportionment on the summary fetch, block export during stale data Cursor Bugbot caught two real issues: 1. The compact Billing summary glance (limit=1) only ever reads summary.totalCredits, but the list route unconditionally ran getUsageCreditsByLogId's whole-filter scan on every call including this one — pure wasted work for a caller that discards the result. Added an includeCredits query flag (default true, using the shared booleanQueryFlagSchema) so useUsageSummary can opt out; the main paginated view keeps it on since it genuinely needs per-row values. 2. Export stayed enabled while useUsageLogs held stale rows via keepPreviousData mid-filter-transition — a user could change the period/range and click Export before the new data loaded, exporting against the new filter while the table still showed the old one. Export is now also disabled while isPlaceholderData is true. * fix(billing): deterministic apportionment order, block audit export during stale data Cursor Bugbot caught two more real issues on the latest push: 1. Same stale-export bug as the earlier Credit usage fix, this time in Audit Logs: Export stayed enabled while useAuditLogs held prior rows via keepPreviousData, so it could export against a just-changed filter while the table still showed the old one. Now also disabled while isPlaceholderData is true. 2. getUsageCreditsByLogId had no ORDER BY before apportionCredits's largest-remainder tie-break, so which row absorbed a tied remainder credit depended on undefined Postgres row order — the same event's displayed credit could flip between calls (list vs. export, or even two successive requests). Added the same `orderBy(desc(createdAt), desc(id))` the main list query already uses, making the tie-break reproducible. Verified live: 3 identically-costed rows produced the same tie-break winner across 3 repeated requests (previously order-dependent). * fix(billing): distinguish a failed summary fetch from zero usage The compact Billing glance only branched on isPending, so once useUsageSummary settled into an error state, totalCredits stayed undefined and formatCreditsLabel(0) rendered "0 credits" — visually identical to genuinely having no usage this period. Now shows the same neutral "—" placeholder for isError as it already does for isPending. * fix(billing): gate the credit-usage page server-side for enterprise accounts Greptile (P1) caught this: hiding the "View usage logs" link on the Billing page for enterprise accounts doesn't stop direct navigation — anyone with the URL (bookmark, shared link, browser history) could still reach the full page and its CSV export, which enterprise accounts were never supposed to see at all (billing is managed out-of-band for them). Added a server-side check in page.tsx before anything renders: resolve the session, look up the highest-priority subscription, and redirect to /settings/billing if it's enterprise — matching how getHighestPrioritySubscription is already used elsewhere for server-side plan checks, rather than relying on a client-side-only conditional the way the Billing page's inline section does. Also fixes loading.tsx: it was a Server Component (no directive) passing a raw icon function reference into the client Chip component, which fails RSC serialization. Added 'use client'. Verified live in a real browser against both an enterprise account (redirects to Billing before any credit-usage content renders) and a non-enterprise account (reaches the page normally). |
||
|
Vikhyath Mondreti
|
b4b666bfbe |
improvement(forking): fork time ux (#5348)
* improvement(forking): fork time ux * add storage quota * address comments * merge latest staging * address comments |