116 Commits

Include renames

Author	SHA1	Message	Date
Waleed	5b7cf99126	feat(folders): add the generic resourceType-driven folder engine (#6037) ... - Replaces the workflow-hardcoded folder orchestration with a config-driven engine: one implementation parameterized by `FOLDER_RESOURCES`, with `lib/workflows/orchestration/folder-lifecycle.ts` reduced from 620 lines to thin wrappers so no existing caller moved - Threads `resourceType` through all four folder routes and the React Query keys; contract widened to `workflow | knowledge_base | table` - Folder locking moved behind a `supportsLocking` capability rather than scattered `resourceType === 'workflow'` checks - Folder restore now matches dependents by the cascade timestamp, so a schedule or webhook archived independently before a folder delete stays archived (deliberate behavior change, documented in the PR) - 25 cascade tests including a fixed-statement-count assertion guarding against N+1 regressions	2026-07-28 19:29:07 -07:00
5 people	1d64b92b41	feat(desktop): desktop app (#5998) ... * top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai>	2026-07-28 19:25:59 -07:00
Vikhyath Mondreti	c809845b99	improvement(self-host): enterprise features enabling (#6028) ... * improvement(self-host): enterprise features enabling * chore(helm): bump chart to 1.3.0 for the enterprise self-host values values.yaml gained the ENTERPRISE_ENABLED switch and INSTANCE_ORG_* keys, and the feature-flag envDefaults moved from "false" to empty so the master switch can resolve them. Additive and backward compatible, so a minor bump. * fix(self-host): address review findings on instance org and org delete Drop the per-process instance-org id cache. It went stale once the organization was deleted through the Admin API, and clearing it from the delete handler would only heal the replica that served that request. The lookup runs on the signup path against a single-row table, so re-reading costs nothing and keeps every replica self-correcting. Scope the org-delete subscription conflict to entitled statuses. Matching any row regardless of status let a canceled subscription — which bills nobody — permanently block deletion. * fix(admin): block org delete on any live subscription, not just entitled ones ENTITLED_SUBSCRIPTION_STATUSES excludes trialing, so a trial — which grants no entitlement but is a live Stripe subscription that will convert — slipped past the delete guard and could be stranded against a removed organization id. Adds TERMINAL_SUBSCRIPTION_STATUSES and inverts the predicate: block unless the row is finished. Expressed as the terminal set so a status Stripe adds later defaults to blocking, which is the safe direction for a destructive operation. * fix(self-host): resolve SSO and access-control in the UI, not the raw env var Nine client consumers still read NEXT_PUBLIC_SSO_ENABLED / NEXT_PUBLIC_ACCESS_CONTROL_ENABLED directly while the server gates and settings nav had moved to the resolver. With only ENTERPRISE_ENABLED set that produced dead ends: the SSO settings section appeared but ssoClient() was never registered and no login button rendered, and the Access Control section appeared but its page reported "not entitled". Points every consumer at isSsoEnabled / isAccessControlEnabled so visibility and capability come from one place. * fix(admin): validate retention workspace targets on the Admin API too retentionOverrides and per-workspace PII rules both name a workspace, and neither field is a foreign key. The settings UI rejected ids belonging to another organization; the Admin API did not, so the two paths could persist different data for the same org. Extracts the check as getForeignWorkspaceTargetsReason and points both routes at it, so they cannot drift apart again. * fix(self-host): close three review findings on admin routes and cleanup Make org delete atomic. detachOrganizationWorkspaces committed on its own, so a failed delete left workspaces detached and re-billed while the organization, its members, and its settings survived. Adds a Tx variant so both commit together. Gate the admin session-policy PATCH on entitlement, matching the settings UI. Without it the stored policy was inert — getSessionPolicy resolves to no-op when the feature is off, so the one eager clamp would be undone on the next refresh. Stop emitting plan-wide housekeeping when billing is off. It is keyed to the hosted free-tier 30-day window, the same default the per-workspace pass deliberately refuses to apply off-hosted. * fix(admin): gate whitelabel on entitlement and emit detach audits post-commit The Admin whitelabel PATCH skipped the entitlement check the settings UI runs, so an admin key could set branding the product had not granted the org. detachOrganizationWorkspacesTx also wrote its audit rows inside the caller's transaction, contradicting its own doc comment — a rolled-back delete would have left audit history describing detachments that never happened. It now returns the rows and callers emit them after commit. * fix(self-host): refuse instance-org resolution when the slug is ambiguous organization.slug has no unique constraint, and the lookup took the first of however many matched. The choice is unordered, so two replicas could resolve different organizations and split new signups between them. Resolution is now three-state. Ambiguity is distinct from absence, so it both declines to adopt an arbitrary organization and declines to provision another one on top of the duplicates.	2026-07-28 19:02:41 -07:00
Waleed	60f2d031b0	feat(folders): move workflow folders onto the generic folder table (#6025) ... - Repoints every workflow-folder read and write from `workflow_folder` to the generic `folder` table backfilled by migration 0272, scoped to `resourceType = 'workflow'`; the legacy table is left in place, unread and unwritten - Adds the resourceType filter to id-keyed lookups too: UUIDs cannot collide, but a caller could pass a file/kb/table folder id as a workflow folderId - Handles the generic table's active sibling-name unique index, which `workflow_folder` never had — 409 on create/rename, auto-suffix on restore and duplicate, mkdir -p reuse on admin import - Scopes the shared soft-delete cleanup job so it cannot hard-delete other resource types' folder rows - Drops the dead createFolderRecord/updateFolderRecord/deleteFolderRecord helpers	2026-07-28 17:43:38 -07:00
Theodore Li	3d72ab3d13	fix(cleanup): bind array params as arrays, not expanded value lists (#6010)	2026-07-28 19:21:22 -04:00
Waleed	cb3611bad2	feat(folders): add resource pinning and generalize the folders contract (#6014) ... - Adds per-user pinning for workflows, files, knowledge bases, and tables: new `pinned_item` table, `/api/pinned-items` routes, React Query hooks, and a shared `PinButton` wired into Tables, Knowledge, and Files with pinned-first ordering - Adds the generic `folder` table with an idempotent, replay-safe, collision-aware backfill from `workflow_folder` and `workspace_file_folders`; no cutover yet, folder reads still go through a documented legacy adapter - Moves `folderSchema` to the generic vocabulary (`resourceType`, `deletedAt`) and drops the unused `color`/`isExpanded`	2026-07-28 14:42:11 -07:00
Waleed	23bc210bf8	fix(seo): noindex non-production sim.ai hosts, redirect indexed 404s (#5988) ... * fix(seo): noindex non-production sim.ai hosts, redirect indexed 404s Non-production deployments (dev.sim.ai, staging.sim.ai) serve the same build as www.sim.ai and were fully crawlable. Send X-Robots-Tag noindex for those hosts, and 301 seven marketing paths that an external SEO audit found returning 404. * fix(seo): parse only the first entry of a comma-joined forwarded host A multi-value x-forwarded-host survives the port split intact, so endsWith('.sim.ai') matched on the trailing entry and could apply noindex to the canonical site. Normalize with the same split(',')[0].trim() the rest of the repo uses for forwarded headers. * fix(seo): drop the /security redirect, complete the urls test mock security.txt advertises /security as its RFC 9116 Policy URI, so a permanent redirect to marketing would mislead that link and shadow a real policy page added later. urlsMock is installed globally and documents itself as carrying every real export, but was missing CANONICAL_SITE_HOST and isNonCanonicalSimHost — any test loading proxy.ts would have called undefined in track().	2026-07-27 16:33:22 -07:00
Theodore Li	ae49070bea	fix(slack): stop requesting the unapproved scopes, gate them behind an opt-in flag (#5977) ... * fix(slack): stop requesting the unapproved mention/assistant/DM scopes * feat(slack): gate the approval-pending scopes behind an opt-in env flag	2026-07-27 14:48:45 -04:00
Theodore Li	0dcbc56ef6	feat(tables): per-table mutation locks (schema/insert/update/delete) (#5960) ... * feat(tables): per-table mutation locks (schema/insert/update/delete) Adds four independent, admin-only locks to a table so a workspace can make it append-only, read-only, or schema-frozen. Enforced at the lib/table service layer rather than the routes, so the API, workflow blocks, and Mothership (which calls the services directly) are all covered by one assert. Violations return 423; changing a lock requires workspace admin and returns 403. * fix(tables): address review — CI snapshot, lock-aware jobs, UI gating - Add the drizzle meta snapshot the hand-written migration was missing, which made CI regenerate the lock columns as a phantom 0271 - Stop a queued or retried delete/update job that starts after its lock is enabled; a run that has already committed pages still finishes, since pages are never rolled back and aborting would leave an uncancelled half-done state - Map TableLockedError to 423 on the internal single-row DELETE (was a 500) - Gate the lock settings UI behind NEXT_PUBLIC_TABLE_LOCKS so it can't open onto a Save that 403s against the server-side flag - Respect the delete lock on column drops in the grid, matching assertColumnDestructive, instead of failing only on click * fix(tables): close lock gaps in async import and undo/redo - Assert insert (and delete for replace) at the start of the import worker. Replace deleted every row before the first insert assert, so an insert-locked table was wiped and then failed, leaving it empty. - Run the delete/update worker start checks through assertRowDelete / assertRowUpdate instead of reading the lock flags directly, so the worker and its enqueue site apply identical rules — including the workflow-column exemption a bulk update was being cancelled despite. - Make undo/redo verb mapping direction-aware for column ops: dropping or retyping a column needs the delete lock clear too, matching assertColumnDestructive, so those steps are skipped instead of stranded. * fix(tables): stop locks over-blocking workflow runs, duplicate, and imports - Run / Re-run / Stop no longer inherit the cell-edit gate: they write only workflow-output columns (which the update lock exempts) and Stop is a cancel - Duplicate needs only the insert lock — it inserts a full copied row, so it stays available on an append-only table - updateWorkflowGroup asserts the destructive rule only when a patch actually drops or remaps output columns; rename / autoRun / mapping edits need just the schema lock - Gate the import-async enqueue on insert (and delete for replace) so a locked table 423s instead of claiming the write-job slot and failing in the worker - Disable Import CSV under an insert lock, and wire the expanded cell editor and the previously-unused stable add-row handler to the lock-aware flags * fix(tables): keep column menu usable and let locks always be cleared - Pass the blocked-delete handler instead of undefined so ColumnOptionsMenu still mounts on a delete-locked table; withholding it hid the entire menu, including Insert column, which the delete lock must not affect - Restore ColumnHeaderMenu readOnly to permission-only: it swaps the header for a static label, which was also disabling pin, column-select and open-config - Assert the schema lock at the import-async enqueue when createColumns is set - Allow a locks PATCH that only clears locks even when the feature flag is off, and keep the settings entry reachable on a locked table, so flipping the kill switch can't strand a table with locks nobody can remove * fix(tables): scope the update-lock exemption and re-check locks per page - Make the workflow-output carve-out opt-in via `computedWrite`, set only by the cell-write path. It was caller-agnostic, so an ordinary API caller could PATCH a workflow-output column on an update-locked table - Re-read the lock before every page in the delete and update runners. The single worker-start check went stale immediately, so enabling a lock could not stop a job already deleting or overwriting rows. Committed pages still stay committed, exactly as with an explicit cancel. Import keeps its one-shot check: a half-imported table needs the deletes the lock now forbids - Route Insert column to the locked-action modal under a schema lock instead of leaving it live (regular header) or hiding the whole menu (group header) * fix(tables): revalidate locks in-transaction and unbreak backfill + partial unlock - Re-assert the lock inside each page-write transaction, under the same advisory lock updateTableLocks takes, so check-then-write is atomic against a lock change. The per-page check alone still left the page-selection await between the assert and the write - Pass computedWrite from the backfill runner: batchUpdateRows is the workflow-output backfill path, so scoping the exemption had made rebuilding outputs 423 on an update-locked table while live cell writes still worked - Gate the feature flag on a lock actually going off->on rather than on an all-false payload. The settings UI always submits all four flags, so with the flag off an admin could not clear one lock while another stayed on - Hoist the lock-kind -> flag map to lib/table/types as TABLE_LOCK_FLAGS * fix(tables): guard import batches in-transaction and split paste by verb - Re-assert the insert lock inside each import batch's insert transaction, under the same advisory lock updateTableLocks takes. Reversing the earlier "let the file finish" call: an admin can lift the lock to clean up a partial import, so honouring it beats letting the rest of the file land - Gate paste per verb. Overwriting existing rows is an update and extending past the last row is a full-row insert, so a paste-append still works on an append-only table while an overwrite explains itself. Refuses the whole paste rather than applying half of it - Space opens the same row editor as double-click, so it now follows the update lock instead of filling in a form that 423s on save * fix(tables): guard the remaining import writes and explain blocked Shift+Enter - Route the replace-mode wipe, the inferred-schema write and createColumns through the same guardBatch transaction as the batch inserts, so a delete or schema lock committed while the file is downloading or being sampled is seen instead of the job-start snapshot - Shift+Enter takes the manual-add path, so it now opens the lock modal like the Add row button instead of returning silently * improvement(tables): surface blocked lock actions as a toast, not a modal - Replace TableLockedModal with a warning toast carrying a "Lock settings" action button for admins. Being told you can't edit shouldn't cost a dismiss click, and the button still routes admins straight to the panel - Dedupe by id so repeated attempts on a locked cell replace one notice instead of stacking a column of them - Move the copy into lock-copy.ts alongside the rest of the lock vocabulary and tighten it for toast length * fix(tables): don't show the update-lock notice to users without write access Double-click checked the update lock before any permission check, so a read-only member on a locked table got "Editing rows is locked" — misleading (the lock isn't why they can't edit) and it swallowed the expanded-cell viewer, which is a legitimate read-only affordance. * improvement(tables): trim the update-lock toast copy * fix(tables): map append-import locks to 423 and stop conflating locks with permissions - The sync append branch returns instead of rethrowing, so the outer catch's mapper never saw a TableLockedError and every lock violation became a 500. Map it in that catch, with a regression test (replace mode already rethrows) - Stop mounting the workflow-group column menu for users without edit access: passing the blocked handlers unconditionally made it appear for read-only members and report a lock even on an unlocked table - Enter/F2 now raises the same lock notice as double-click and Space instead of silently doing nothing - guardBatch returns the freshly-read definition so addTableColumnsWithTx asserts live state; a schema lock cleared mid-import no longer fails the createColumns step. The snapshot pre-asserts now only run as the fallback for callers that pass no revalidator - Append-only no longer labels a table whose schema is also locked, which claimed columns were mutable when they weren't - Import dialog withholds Replace on a delete-locked table and create-column on a schema-locked one, instead of offering a configuration that only 423s * fix(tables): revalidate sync imports under the advisory lock, explain every blocked key - The sync append/replace paths asserted only the request-start snapshot, so a lock committed while the CSV was parsed still let the write through. Both now re-read under the schema advisory lock at the top of their own transaction, taken before acquireRowOrderLock so the order stays advisory -> rows_pos -> definitions - Delete/Backspace, Cmd+D, typeahead and cut raised no notice on an update-locked table; they now explain the lock, and stay silent for users without write access - The multipart import fetch threw a plain Error, dropping the 423 status, so the lock self-heal never ran and the stale detail cache survived. It throws ApiClientError now and both import-into-table hooks call handleTableLockRejection - Force append at submit when the delete lock landed while the dialog was open with Replace already selected * fix(tables): restore the workspace ownership check on copilot row deletes Switching deleteRow/deleteRowsByIds to take a TableDefinition dropped the workspaceId argument that previously scoped the query, and the two branches I added loaded the table without the ownership comparison every other operation in the tool performs — letting a caller delete rows from a table in another workspace. Both now reject a foreign table as not found.	2026-07-25 14:21:05 -04:00
Vikhyath Mondreti	17d77795b4	feat(providers): prompt caching capability + usage-based cache pricing (#5922) ... * improvement(providers): validation pass, and stream tool loop improvements * remove deploy options correctly * fix * feat(providers): prompt caching capability and usage-based cache pricing Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic. * fix(providers): route OpenAI and Gemini block cost through cache-aware pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained. * fix(streaming): redact tool payloads on selected outputs in public chat Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole. * refactor(providers): drop unreachable billing fallbacks Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total.	2026-07-24 15:46:11 -07:00
Waleed	513292f17b	feat(sso): DNS domain verification gating org SSO registration (#5909) ... * feat(sso): DNS domain verification gating org SSO registration Add org-scoped domain ownership verification (DNS TXT challenge) as the security precondition for configuring SSO. Closes the first-come domain-claim vuln where any org could wire another company's domain to its own IdP. - New sso_domain table + migration 0266; existing org SSO domains are grandfathered as verified so live tenants are unaffected - Verified-domains settings UI (enterprise-gated) with add/verify/remove - Register route now requires a verified domain for org-scoped registration; personal SSO and already-grandfathered domains are unaffected - Self-host register script writes the verified sso_domain row directly, so script-driven registration stays backwards compatible * fix(sso): harden domain verification against concurrency + fix CI lint Addresses review findings on state invariants under concurrent/failed writes: - Add unique index on (organization_id, domain) so concurrent claims can't create duplicate pending rows; POST re-reads and stays idempotent on conflict - Verify flips the row only if it's still the exact pending challenge checked (guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique index violation to 409 instead of an unhandled 500 - Wrap the self-host script's provider write + verified-domain upsert in a transaction so a failed ownership write can't leave a provider committed - Format 0266 snapshot/journal with biome (fixes @sim/db lint:check) * fix(sso): re-check domain verification before provider write (TOCTOU) The register gate checked the verified sso_domain row only at handler entry, then ran OIDC discovery before writing the provider. A verified row removed during that window could still complete registration. Extract the check into a closure and call it both as an entry fast-fail and authoritatively right before registerSSOProvider, alongside the existing domain-conflict re-check. * fix(sso): stop rotating verification token on idempotent re-add Re-adding a pending domain rotated its verification token, which invalidated a TXT record the admin may have already published and — under two concurrent re-adds — could return a token the racing write had already superseded, so the admin's DNS record would never verify. Return the existing row unchanged instead; the pending token is always shown in the UI, so it is never lost. * fix(sso): close register TOCTOU with compensating delete + harden edges Audit-driven hardening: - Close the residual register TOCTOU: registerSSOProvider is create-only (throws if the providerId exists), so a compensating delete after the write is provably safe — it can only remove the just-created row. If verification was revoked during the write, roll the provider back and 403. - Verify is now idempotent under concurrency: a same-org row already flipped to verified by a racing request returns 200, not a confusing 409. - Grandfather backfill + self-host script now match normalizeSSODomain's dominant transforms (lower + trim + strip leading wildcard) so a non-canonical legacy domain can't miss the runtime gate's lookup. Prod backfill result is unchanged. - Cleanup: drop dead default export, align card radius to sibling convention. * fix(sso): redact domain tokens from non-admins + fix script stale-update Round-5 review findings: - GET /domains redacted the pending TXT verification token (a management secret) to any org member. Now only owner/admins read it; members see the list and status without it. Non-Enterprise orgs get an empty list (entitlement flag only), never the domains/tokens. - Self-host script decided update-vs-insert from a read taken OUTSIDE the transaction; a provider deleted mid-flight made the UPDATE match zero rows silently while the verified-domain upsert still committed (orphaned domain). The decision now happens inside the transaction from the UPDATE's row count. * docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains Verified domains currently only gate SSO configuration. Remove the forward-looking references to enforcing SSO and auto-joining members (deferred to a later release) from the docs, settings copy, nav description, and schema comment so we don't promise unshipped features. * fix(sso): guard rollback to new providers only + Enterprise-gate domain removal Round-6 review findings: - The compensating provider rollback now only fires when the provider did not exist before this request (providerExistedBefore). registerSSOProvider is create-only today so reaching the rollback already implies a fresh create, but this makes the safety local and future-proof: if Better Auth ever allowed updating an existing provider, a revoked-verification rollback must not delete that pre-existing row. - DELETE /domains now requires an Enterprise plan like add/list/verify, so all domain mutations share one entitlement (the UI already hides removal from non-Enterprise orgs). Adds a delete-route test. * fix(sso): roll back the SSO provider by row id, not logical keys The compensating rollback deleted by (providerId, orgId). providerId is unique, so if this request's row were deleted and recreated by a concurrent registration in the narrow window before the rollback, the logical-key delete would remove that other request's provider. Delete by the primary-key id registerSSOProvider returns instead, so only the exact row this request created is ever removed. * chore(sso): final-review polish — trim script read, unify copy, doc migration edge Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the new logic): - Self-host script: narrow the pre-transaction existence read to select({ id }) instead of SELECT * (it only feeds a log line now). - Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere 409 wording ("is already verified by another organization") across routes. - p-3 shorthand on the domain row card. - Document the migration's rare two-orgs-share-a-domain grandfather behavior (login unaffected; validated no such duplicates in prod). * fix(sso): apply attribute mapping + make SSO edit work; drop dead guard Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW, script-registered with the default mapping, so neither change affects it): - Attribute mapping was passed at the top level of the register payload, which Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it so custom mappings actually apply. (Default mapping is unchanged, so existing logins are unaffected.) - Editing an SSO provider was broken: registerSSOProvider is create-only and threw on the existing providerId → generic 500. Route now detects a provider the caller already owns and updates it via Better Auth's updateSSOProvider, and surfaces Better Auth's own error status/message instead of a blanket 500. Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes by the created row's primary-key id and register is create-only) and the earlier final-review polish (script read, unified copy, migration edge note). Smoke-test SSO login + edit on staging before merge (auth-path change). * fix(sso): require null org on personal-mode provider lookups (gate bypass) The personal branch of both provider-ownership lookups keyed on (providerId, userId) without requiring organizationId IS NULL. Because org providers store userId = their creator and providerId is globally unique, an org admin could send a personal-mode request (no orgId) — which skips the membership check and the domain-verification gate — yet still match, and then via the new update path move, their org's provider to an unverified domain. Add isNull(organizationId) to the personal branch of both clauses so it can only match a genuinely personal provider, matching the route's own isOwnedByCaller. Found by an adversarial review of the update path added in 394bda9f7. * fix(sso): script updates the observed provider by id, not providerId Inside the registration transaction the script updated WHERE providerId — the logical key. If the observed provider was deregistered and a replacement created with the same providerId before the transaction ran, that update would clobber the replacement's config and ownership. Update the specific observed row by its primary-key id instead; if it's gone we insert, which fails cleanly on the providerId unique constraint rather than overwriting the replacement. * fix(sso): script upserts provider via delete-then-insert (no unique constraint) sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate duplicates, so the previous "update by id, else insert" could create a duplicate provider when the observed row was deregistered and replaced before the transaction — the fallback insert would succeed. Delete every row for the providerId then insert exactly one, inside the transaction, so the providerId ends up as exactly this config atomically. Linked accounts key on the providerId string (not the row id), so existing logins are unaffected. * fix(sso): guard compensating-delete row id so rollback can't silently no-op * chore(sso): regenerate migration as 0268 after merging staging Staging landed migrations 0266/0267, colliding with our 0266. Removed our migration, merged staging, and regenerated cleanly with drizzle-kit as 0268_sso_domain_verification (identical sso_domain table + indexes), then re-appended the grandfather backfill. api-validation baseline reconciled to 973 (staging 970 + our 3 domain routes). Also make the register-route test's registerSSOProvider mock return an id so the guarded compensating delete runs. * refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate The self-host script canonicalized SSO domains with a minimal inline transform (lower+trim+wildcard) that diverged from the app's full normalizeSSODomain (protocol, port, path, trailing dot, email local part) — equivalent spellings could store a different ownership key than the runtime gate looks up. Move normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register route, the domain-claim route, and the script all use the identical canonicalizer. The script now skips the verified-domain record when SSO_DOMAIN isn't a valid registrable domain instead of storing a malformed key.	2026-07-24 00:34:16 -07:00
3 people	6dcc65be89	feat(skills): add skill editors (#5705) ... * feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-23 19:52:00 -07:00
4 people	d24bc7eccb	feat(agent-stream): thinking and tool streaming (#5671) ... * feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-23 19:39:03 -07:00
Theodore Li	96c67e9123	feat(sandbox): add Daytona as a manual-flip failover for E2B (#5860) ... * feat(sandbox): add Daytona as a manual-flip failover for E2B E2B was a hard single point of failure: lib/execution/e2b.ts had no retry and no fallback, so a failed Sandbox.create() killed Python function blocks, JS-with-imports, shell, doc generation and the Pi cloud agent outright. Extract a SandboxRunner boundary (lib/execution/remote-sandbox) with an E2B runner and a Daytona runner, selected once per execution by the sandbox-provider-daytona AppConfig flag. Everything above the provider boundary — marker parsing, mount materialization, file export, corruption handling — is unchanged. Selection resolves before create() and never mid-execution, since user code has side effects. Each sandbox kind fails closed when its snapshot id is unset. Notes on the Daytona adapter: - language binds at create(), not per call: Daytona applies it as a sandbox label and silently runs JS through Python if passed to codeRun - Python routes via CodeInterpreter for its {name,value,traceback} error shape, which matches E2B's and keeps formatE2BError's line offsets correct - timeouts convert ms to seconds - the streaming path delivers env via the filesystem API, as SessionExecuteRequest has no env field and secrets must not reach a command line Drops the dead E2BExecutionResult.images field (populated, never consumed). * improvement(sandbox): select the provider by SANDBOX_PROVIDER env var Replaces the boolean sandbox-provider-daytona feature flag with a SANDBOX_PROVIDER env var naming the provider ('e2b' default, or 'daytona'). A boolean doesn't scale to a third adapter; a keyed registry does. - PROVIDERS is a Record<SandboxProviderId, SandboxProvider>, so adding an adapter is one entry plus one id-union member — an unhandled provider is a compile error, not a runtime surprise - resolveProvider() reads env synchronously and throws on an unknown value (fail fast) instead of an async feature-flag lookup - drops the sandbox-provider-daytona flag and SANDBOX_PROVIDER_DAYTONA fallback Verified end-to-end: a Python function block through the running app routes to Daytona (Creating Daytona sandbox, kind: code) with SANDBOX_PROVIDER=daytona. * fix(sandbox): gate remote execution by provider availability, not E2B Addresses the review round on #5860. - Availability was gated on isE2bEnabled / isE2BDocEnabled, so a Daytona-only deployment (E2B_ENABLED unset) had its Python/shell/JS-with-imports and doc paths rejected before the provider-neutral sandbox call could run. Replace both with provider-aware flags (isRemoteSandboxEnabled / isDocSandboxEnabled) derived from the selected SANDBOX_PROVIDER's own credentials + image. E2B behavior is unchanged (the E2B branch mirrors the old definitions exactly). - Make the function-block gate error messages provider-neutral. - Daytona's streaming runCommand (Pi) returned empty stdout/stderr and delivered output only via callbacks, so the Pi cloud flow — which parses markers from stdout and formats errors from stderr — saw nothing. Accumulate the streamed chunks and return them while still forwarding to the callbacks. Renames the env-flag exports (and the @sim/testing mock) to match. Adds a conformance test that the streamed Pi output lands in stdout/stderr. * fix(sandbox): resolve SANDBOX_PROVIDER case-insensitively Addresses the round-2 review on #5860. env-flags lowercased SANDBOX_PROVIDER for the availability gate, but resolveProvider looked up the raw value in a lowercase-keyed map — so 'Daytona' passed the gate then threw Unknown SANDBOX_PROVIDER at create. resolveProvider now normalizes casing identically. * fix(sandbox): use getErrorMessage in build/verify scripts check:utils flagged the inline `error instanceof Error ? error.message : ...` pattern in the two new scripts. Use getErrorMessage from @sim/utils/errors, matching the repo convention the check enforces. * fix(sandbox): fall back to stdout for Daytona failure text Daytona merges both streams into stdout and returns an empty stderr, but the shell-error, base64-export, and URL-mount error builders read only result.stderr — so Daytona failures surfaced a generic 'Process exited with code N' / 'base64 failed' / 'curl exited N' instead of the real command output that the API and agents rely on. Fall back to stdout before the generic message (provider-agnostic: E2B still populates stderr). Strengthens the shell-error conformance test to assert the real output surfaces. * fix(sandbox): enforce timeout on Daytona streaming; drop leftover E2B copy - Daytona's streaming path (Pi) started the command with runAsync:true and then awaited getSessionCommandLogs with no bound, so a hung command never timed out the way E2B's commands.run({ timeoutMs }) does. Race the log stream against the timeout; on expiry return exit 124 with the accumulated output, and the finally's deleteSession terminates the still-running command. - Two user-facing strings still named E2B after the provider-neutral rename (the isolated-vm sandboxPath remediation and the disabled-xlsx message). Made both provider-neutral. Adds a streaming-timeout conformance test. * fix(sandbox): handle orphaned stream promise + preserve error detail Two regressions from the previous timeout fix: - When the timeout won the race, the abandoned getSessionCommandLogs promise would reject on deleteSession with no handler (unhandledRejection). Attach a .catch that records the error and yields an 'error' outcome, so a late rejection is always handled. - The streaming catch dropped the thrown error, so failures before any chunks (env write, executeSessionCommand, missing cmdId) surfaced as empty output. Fall back to getErrorMessage(error) when nothing streamed. Adds conformance tests for the stream-reject and start-throw paths.	2026-07-23 19:19:00 -04:00
Waleed	227cd65f41	fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup (#5874) ... * fix(mcp): bound and retry OAuth start so a transient stall recovers instead of a blank popup Empirically root-caused a blank/stuck authorize popup: the provider (planetscale) and our guarded OAuth fetch are both fast (120/120 legs clean from staging), and /oauth/start uses local AES encryption with no Redis lock in its path — so the intermittent hang is the same transient headers-then-stalled-body class we've documented for CDN-fronted MCP hosts (a per-connection stall a fresh attempt dodges), which /oauth/start had no server-side bound against. - Bound every /oauth/start step with the shared timedStep helper (extracted from the callback route, now used by both) + an entry log, so a stalled step surfaces as a labeled error instead of hanging the request (and the browser popup) to the client's 30s timeout. - Retry mcpAuthGuarded once on a bounded 12s timeout: a fresh attempt gets a fresh connection and recovers from the transient stall automatically (two 12s attempts stay under the client's 30s deadline). McpOauthRedirectRequired (the success signal) and DCR-unsupported errors are never retried. Adds OauthStepTimeoutError + makeTimedStep to the shared oauth barrel and test mocks. * fix(mcp): drop the unsafe OAuth-start retry; fail fast without error-logging success Review fixes on the bound+retry change: - Removed the mcpAuthGuarded auto-retry. timedStep can't cancel the loser, so a lingering first attempt shares this server's OAuth row and could overwrite the retry's PKCE verifier / state after the client already got the second authorize URL, breaking the callback. Recovery is now fail-fast (504) → the user re-clicks, which is a clean fresh flow (fresh connection dodges the transient stall) with no shared-state race. - Catch McpOauthRedirectRequired (the success signal) INSIDE the bounded step and return it as a value, so a successful authorize is no longer error-logged as 'OAuth step failed'. - Tighten step budgets (5s DB x3 + 12s auth = 27s) to stay under the client's 30s /oauth/start deadline. * fix(mcp): route all bounded-step timeouts to the 504 handler Move OauthStepTimeoutError handling to the outer catch so a DB-step timeout (loadServer/getOrCreateOauthRow/loadPreregisteredClient) returns the same fast 504 'try again' as the auth step, not a generic 500. Documents that the fresh retry is race-safe: the callback correlates on the state nonce, so a lingering timed-out attempt overwriting the row's state only yields a clean invalid_state on the user's fresh authorize URL — never silent corruption. * fix(mcp): bound the setOauthRowUser write too so no step escapes the budget The user-stamp write was the one DB op left unbounded on the start path; wrap it in timedStep(DB_STEP_MS) so every step stays inside the sub-30s budget and its timeout routes to the same 504. * fix(mcp): shrink OAuth-start step budgets to fit the 30s client deadline with 4 DB steps Bounding setOauthRowUser added a fourth possible DB step, so 4x5+12=32s exceeded the client's 30s /oauth/start abort. Lower DB steps to 4s and auth to 10s: 4x4+10=26s worst case, leaving margin for middleware/network. Comment corrected.	2026-07-22 20:31:24 -07:00
Waleed	dda602a367	improvement(tests+ci): phase 3 — shared-mock convergence completion and CI runner-minute cuts (#5875) ... * chore(ci): cut redundant runner minutes — dedup promotion-PR test runs, companion-pr-check concurrency, right-size trivial jobs - ci.yml: new dedup-promotion gate skips the pull_request test-build on staging/main-headed promotion PRs only when the merge tree provably equals the head tree (empty base delta over the merge base) AND the push-event run at the same sha passed its test jobs (polled). Fail-open on any error/ timeout/failure, job-level skip only (skipped job reports Success); verified no required status checks are configured on main/staging rulesets. Measured 39 duplicate PR runs / 5.15 days (~227/mo) at ~7.1 min each on 8vcpu (~57 vcpu-min), probe costs ~9 vcpu-min worst case on 2vcpu. - companion-pr-check.yml: per-PR concurrency group with cancel-in-progress so superseded synchronize/edit runs stop; no paths filter (check depends on PR body + cross-repo state, not changed files). - detect-version and check-docs-changes: 4vcpu -> 2vcpu Blacksmith runners (pure shell / depth-2 checkout + path filter only). * improvement(testing): complete stateful shared mocks for env, urls, redis-config, environment-utils Shared mock infrastructure for vitest isolate:false convergence: - packages/testing/src/mocks/env.mock.ts: stateful envMock (live env proxy, setEnv/resetEnvMock, process.env fallback) - packages/testing/src/mocks/urls.mock.ts: complete urlsMock with real-behavior default impls + resetUrlsMock - packages/testing/src/mocks/redis-config.mock.ts: adds getRedisConnectionDefaults + resetRedisConfigMock - packages/testing/src/mocks/environment-utils.mock.ts: new environmentUtilsMock + fns + reset - contract tests: env.mock.test.ts, urls.mock.test.ts, redis-config.mock.test.ts, environment-utils.mock.test.ts - packages/testing/src/mocks/index.ts: barrel exports - apps/sim/vitest.setup.ts: global installs for env, urls, redis, environment/utils - real-module tests unmocked: lib/core/config/env.test.ts, lib/core/config/redis.test.ts, lib/core/utils/urls.test.ts, tools/index.test.ts (urls) - stubEnv/process.env fallout migrated to setEnv: lib/webhooks/providers/{revenuecat,rootly,instantly}.test.ts, app/api/auth/oauth2/authorize/route.test.ts * improvement(tests): drop redundant local mocks in executor/tools/providers and misc dirs (shared-worker readiness) * improvement(tests): drop redundant local mocks in app routes (shared-worker readiness) * improvement(tests): drop redundant local mocks in lib (shared-worker readiness) * fix(ci+testing): live base-tip recheck before dedup skip; prod-aware urls mock fallbacks - the dedup gate re-verifies merge-tree equivalence against the LIVE base tip at decision time, closing the window where the base branch gains real commits during the poll (frozen BASE_SHA check alone was stale) - the urls mock's getBaseUrl protocol prefix and getBaseDomain parse fallback now follow the shared isProd flag, mirroring the real module * fix(ci+testing): fail-closed nojobs fallback in dedup gate; TLS-aware redis defaults mock - the dedup gate no longer infers coverage from overall run conclusion when no 'Test and Build /' jobs match — a renamed or skipped test job now runs the tests instead of skipping them - the shared getRedisConnectionDefaults mock mirrors the real TLS resolution (rediss:// to a raw IP requires REDIS_TLS_SERVERNAME and yields tls.servername) * fix(ci): keep polling while nested test jobs have not appeared yet An in-progress push run lists its reusable-workflow jobs only after the caller starts; nojobs is now terminal (fail closed) only once the run has completed without them.	2026-07-22 19:04:17 -07:00
Waleed	9d8e14ce9f	improvement(tests): converge env-flags mocks onto a complete shared mock (#5871) ... * improvement(tests): converge env-flags mocks onto a complete shared mock * fix(tests): drop the repo's only bare vi.mock automock A bare vi.mock('drizzle-orm') automock colliding with factory mocks of the same module in a shared worker corrupts vitest's mock registry (upstream vitest-dev/vitest#10290 / #10145, reproduced in isolation). The global factory mock already covers this suite. * chore(tests): drop defensive resets in non-mutating suites, merge sequential setEnvFlags calls * fix(tests): run providers/utils cases sequentially over shared env-flags state	2026-07-22 17:31:18 -07:00
Waleed	2b5a92a3c8	feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862) ... * feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name)	2026-07-22 16:42:59 -07:00
WaleedandMarcus Chandra	8cce661a37	feat(api): proxyUrl for residential/custom proxy egress on the API block (#5867) ... * feat(api): add proxyUrl for residential/custom proxy egress on the API block The HTTP/API block egresses from the app runtime's fixed datacenter IPs via secureFetchWithPinnedIP, so targets behind Cloudflare/WAF that block datacenter IPs (e.g. state .gov license portals) return 403/429 even when the identical request works from a browser. There was no way to route a request through a residential/custom proxy. Add an optional `proxyUrl` field (Advanced) to the API block. When set, the request routes through the given http:// proxy so it egresses from that proxy's IP. Security: - validateAndPinProxyUrl resolves the proxy host's DNS and blocks private/reserved/loopback IPs (same SSRF guard as target URLs), then pins the connection by rewriting the host to the resolved IP (creds/port preserved), closing the DNS-rebinding window. - Restricted to the http: proxy scheme (https/socks rejected) so host pinning is safe without breaking TLS-to-proxy SNI. - Target-IP pinning is intentionally bypassed when a proxy is active (the proxy resolves the target); target URL validation still runs. Threaded block field -> http tool param -> formatRequestParams -> executeToolRequest (validate + pin) -> secureFetchWithPinnedIP, which swaps its pinned Node agent for HttpsProxyAgent/HttpProxyAgent (keyed off target protocol) when proxyUrl is set. * docs(api): document the Proxy URL advanced field and steer proxy credentials to env vars * fix(api): reject loopback/private proxy hosts unconditionally, closing the self-hosted rebinding gap * chore(api): tighten proxy-path inline comments --------- Co-authored-by: Marcus Chandra <mzxchandra@gmail.com>	2026-07-22 16:24:31 -07:00
Waleed	52659d4a89	improvement(tests): db-mock migration tranche 2 — copilot, mothership, workspaces, connectors, mcp (#5863) ... * improvement(tests): db-mock migration tranche 2 — copilot, mothership, workspaces, connectors, mcp * fix(testing): drain unconsumed ...Once overrides in resetDbChainMock vi.clearAllMocks clears call history only — a ...Once override queued by a previous test but never consumed survived into the next test. resetDbChainMock now mockReset()s every shared spy and stable wrapper, which restores the original implementation AND drains once-queues.	2026-07-22 15:17:43 -07:00
Waleed	62a8ce4953	improvement(tests): db-mock migration tranche 1 — knowledge, billing/org, workflows/background (#5861) ... * improvement(tests): migrate knowledge, billing/org, and workflows/background suites off private @sim/db factories * improvement(tests): db-mock migration tranche 1 — knowledge, billing/org, workflows/background - migrate 19 suites off private vi.mock('@sim/db') factories onto the shared dbChainMock + queueTableRows API (net ~-1,260 lines of bespoke chain plumbing); resolves the known shared-worker rival pairs (knowledge processing-queue vs api utils; billing polluters; persistence/utils vs schedules/deploy) - add .for() to the mock's limit builder (drizzle .limit(1).for('update')) with a contract test - document the join-table queue fallback footgun on queueTableRows	2026-07-22 14:53:17 -07:00
Waleed	3215a12da9	improvement(testing): consolidate @sim/db mocks into one table-aware chain mock (#5856) ... * improvement(testing): consolidate @sim/db mocks into one table-aware chain mock - back databaseMock and dbChainMock with the SAME db instance so a module bound to either export hits identical chain fns — rival-mock divergence between the two @sim/testing db mocks is structurally impossible now - add queueTableRows(table, rows): FIFO per-table select routing keyed by schema-mock table identity, consumed at where() materialization and resolved by every downstream terminal (limit/orderBy/groupBy/for/joins) - delete createMockDb (duplicate chain implementation, no external users) - migrate the five suites that hand-rolled table routing + databaseMock delegation (billing plan/usage/usage-log, admin dashboard-organizations, workspaces/utils) onto queueTableRows; net -295 lines - add a contract test for the mock itself and a test script to @sim/testing so its tests actually run under turbo * fix(testing): harden table routing — join-table queues, direct-await from, mutation isolation - track the chain's tables as a list (from + joins) so rows queued for a join-only table route correctly; from-table queue checked first - make the from/join builder a lazy thenable so awaiting a select with no where clause resolves queued rows (dequeue at await, never double-consumed) - update/delete/set clear the routing context so a mutation's where() can never consume rows queued for a select - document the left-to-right chain-construction assumption; contract tests for all three behaviors * fix(testing): close routing over each chain's own tables for direct-await builders * refactor(testing): move all chain routing state into per-chain closures - shared dbChainMockFns entries become pure spy/override ports: their default implementation returns a sentinel that chain-local builders replace, while any mock* override on the spy wins verbatim - each select().from() captures its own immutable table list; where(), joins, terminals, and direct awaits all resolve through that closure, so partially-built chains for different tables interleave without cross-talk - no module-level routing state remains * fix(testing): lazy queue consumption at resolution and wrapper restore on reset - each chain holds one lazy rows supplier: the queued set is dequeued only when a default thenable actually resolves, so a chain answered by a per-test terminal override leaves its queued rows for the next chain - resetDbChainMock also mockReset()s the stable db entry-point wrappers so direct overrides on databaseMock.db.* cannot outlive a suite	2026-07-22 14:15:44 -07:00
Waleed	fad9728cbd	improvement(tests): make vitest suites state-safe with auto-unstub of env/global stubs (#5853) ... * improvement(tests): make vitest suites state-safe with auto-unstub of env/global stubs - add unstubEnvs/unstubGlobals to vitest config so vi.stubEnv/vi.stubGlobal are restored after every test - move module-scope fetch/crypto/window stubs into beforeEach across executor, data-drains, and knowledge suites so they hold under auto-unstub (several suites were silently hitting live Datadog/BigQuery/Snowflake/OpenAI/Anthropic endpoints when their module-scope stubs were cleared) - converge billing/workspaces/tools/hooks suites onto the shared @sim/testing mock instances and namespace spies instead of rival file-local vi.mock factories, with explicit beforeEach setup and afterAll restore - give lib/core suites private module instances via vite query-suffix imports where module-level state bakes env at import time - stub auth-client/NEXT_PUBLIC_APP_URL in suites that crashed at collection without a local .env Groundwork for eventually running the suite with isolate: false; no CI behavior change (isolation stays on). * fix(tests): complete databaseMock restore and route selectDistinctOn - restoreGlobalDb now mockReset()s entries whose original implementation was undefined instead of leaving the chain-mock delegation installed - add selectDistinctOn to the setup-level createMockDb and to the dashboard suite's delegated key set so the distinct-on path routes under either binding * fix(tests): run knowledge utils cases sequentially over shared per-test state	2026-07-22 13:15:15 -07:00
Theodore LiandClaude Fable 5	70814bc4a2	feat(db): role-keyed dbFor clients for cleanup and exec workloads (#5583) ... * feat(db): role-keyed dbFor clients for cleanup and exec workloads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * fix(db): keep cleanup-invoked helpers and snapshot reads on their role pools Route markLargeValuesDeleted / pruneLargeValueMetadata (optional dbClient) and chat-cleanup's file collection through the cleanup pool, and getSnapshot through the exec pool, so the cleanup and inline-execution workloads stop borrowing the process-wide pool for these queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * fix(db): dbFor falls back to the process-role URL, not the base URL With DATABASE_URL_WEB/TRIGGER set (as in prod) and the sub-pool URLs unset, falling back to the base URL would silently shift execution-log and cleanup traffic to a different PgBouncer endpoint on deploy. Chain the fallback through the URL the process itself resolved so the rollout stays inert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * feat(db): log which connection each dbFor sub-pool resolved to One line per role at first use: the dedicated DATABASE_URL_<ROLE> when set, otherwise an explicit fallback message naming the process connection it shares — so a missing/typo'd env var is visible at rollout instead of silent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy * feat(db): route pause/resume and large-value metadata persistence to the exec pool Coverage scan follow-up: the paused_executions / resume_queue / workflow_execution_logs transactions in human-in-the-loop-manager.ts and the large-value owner/reference registration writes on the execution path now use dbFor('exec'), matching the completion writes in the execution logger. All are self-contained; billing calls remain outside the moved transactions on the default client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNy9Fzpfc1FdHAzYyb6Ycy --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-22 14:35:42 -04:00
Vikhyath Mondreti	37628f1566	fix(workflow-persistence): make persistMigratedBlocks' updated_at guard millisecond-tolerant (#5790) ... * fix(workflow-persistence): make persistMigratedBlocks' updated_at guard millisecond-tolerant * improvement(sockets): state improvements * address comments	2026-07-21 14:36:11 -07:00
Vikhyath Mondreti	86e6e1d26c	feat(forking): excluded workflows (#5727) ... * feat(forking): excluded workflows * improve sync preview	2026-07-16 20:23:26 -07:00
Vikhyath Mondreti	442eabaf27	improvement(checkout): enforce team/enterprise-only org subscription, block double-covered personal checkouts (#5715) ... * improvement(checkout): enforce team/enterprise-only org subscriptions, block double-covered personal checkouts, and drop renewal-triggered workspace detach * close race with personal pro / org inclusions * address comments * fix(billing): compute org coverage independently of the personal-sub lookup and fail closed on unverifiable plan writes	2026-07-16 19:16:44 -07:00
Vikhyath Mondreti	54b35a4f0e	improvement(deployments): bugfixes for run-block, airtable + external sub management (#5680) ... * improvement(webhooks): external subscription management * ui/ux * remove test file * fix tests * address comments * address comments * update to grain v2 api * improvement(grain): hide auto-registered webhook URL on v2 triggers * Revert "improvement(grain): hide auto-registered webhook URL on v2 triggers" This reverts commit c89660cc3e. * address comments * address comments * address rollback * fix grain v2 * fix more comments	2026-07-15 18:01:53 -07:00
Vikhyath Mondreti	ef7c8e24b2	feat(platform): settings permissions, admin, billing attribution (#5545) ... * fix(invites): preserve active organization for external access Keep organization activation server-owned so failed membership checks cannot clear a valid session context. * feat(admin, billing, settings): cleanup settings visibility, billing actor resolution, new admin routes * address comments * chore(db): reset pending migrations before staging merge Remove locally generated migrations so they can be regenerated against the latest staging schema without preserving stale snapshots or numbering. * regen migrations * address comments * chore(db): reset generated migrations before staging merge Remove this branch's generated migrations so they can be regenerated against the latest staging schema with fresh numbering. * upgrade global work * fix lint * address comments * legacy callbacks correctness * address comments * update * guardrail attribution	2026-07-13 00:53:49 -07:00
Waleed	5db62b8bdc	feat(observability): extend audit log, PostHog, and storage metering coverage (#5269) ... * feat(observability): extend audit log, PostHog, and storage metering coverage Instruments previously-uncaptured resources and actions across the three observability layers (audit log, PostHog product analytics, usage metering): - Exfiltration audit: file downloads (workspace/public-share/v1), table, workflow, and workspace exports. - Credential access audited at the token-issuance boundary (success-only). - Full revenue trail: invoice paid/failed, overage billed, disputes, credit fulfillment, subscription lifecycle, plan/seat changes. - Session/login lifecycle audit via Better Auth hooks (login, blocked sign-in, logout, session revoke, account delete). - v1/admin programmatic surface and copilot tool handlers instrumented. - Dead audit/PostHog constants wired (lock/unlock, table, custom tool, etc.). - Storage metering extended to KB documents and copilot files. - PostHog person identify + workspace/organization group hygiene. Audit package hardened to null the actor FK for system actors (admin-api) and adds an awaitable recordAuditNow for pre-delete hooks. All instrumentation is fire-and-forget and never blocks or breaks the primary operation. * fix(observability): audit file/credential egress only on success Address Cursor Bugbot review: - GET OAuth token: emit CREDENTIAL_ACCESSED/credential_used after refreshTokenIfNeeded succeeds (matches POST), not before. - File export: audit on each actual success exit via a shared helper — including the non-markdown serve redirect (previously unaudited) and after the zip is generated (previously before asset fetch). * fix(audit): record null actor for anonymous public-share downloads Address Greptile P1: setting actorId to the file owner made every anonymous external download read as a self-download, undermining the exfiltration trail. recordAudit now accepts a null actor; the public content/inline routes record actorId=null with the owner in metadata.sharedByUserId and rely on ip/user-agent for the forensic trail. The misleading owner-attributed file_downloaded PostHog event is dropped on these anonymous paths. * fix(observability): audit admin exports after zip; tidy comments - Admin workflow/workspace ZIP exports now audit only after the archive is built (via a local helper), so a zip/build failure no longer logs an export. - Remove redundant 'success-only placement' inline comments and tighten the remaining design-rationale notes (export helper now TSDoc). * fix(billing): complete the chargeback + overage financial trail Address Cursor review: - handleDisputeClosed now records CHARGE_DISPUTE_CLOSED for every closed dispute (won/lost/warning_closed), unblocking only on favorable outcomes. dispute.status in the metadata distinguishes the outcome, so lost chargebacks are no longer missing from the trail. - Threshold overage now emits OVERAGE_BILLED + overage_billed even when credits fully cover the overage (settledVia: 'credits' vs 'stripe'), so credit-settled overages are audited instead of silently returning null. * fix(storage): decrement copilot quota before deleting metadata Address Greptile P1: deleting the metadata row before the decrement meant a decrement failure left the quota permanently inflated with no record to retry from. Decrement first; only remove the metadata row once it succeeds. * fix(observability): atomic copilot storage release; signup-blocked action - Copilot file delete now releases storage via a single transaction (releaseDeletedFileStorage): the soft-delete is the idempotency claim and the decrement shares the transaction, so neither a partial failure (inflated counter) nor a retry (double-decrement) can desync the quota. Resolves the conflicting Greptile/Cursor ordering findings. - Policy-blocked sign-ups now record USER_SIGNUP_BLOCKED instead of USER_SIGNIN_BLOCKED, so account-lifecycle events aren't mislabeled. * fix(storage): meter copilot ingest centrally for path symmetry Address Cursor review: only uploadCopilotFile incremented storage, but copilot files also enter via the generic upload route and presigned uploads — all of which persist metadata through insertFileMetadata. Move the increment into insertFileMetadata (scoped to context='copilot', on genuine insert/restore) so every ingest path is symmetric with the delete-time decrement, and drop the now redundant per-path increment in uploadCopilotFile. Other contexts are metered by their own managers and remain unaffected. * fix(audit): skip WORKFLOW_EXPORTED when admin export is empty Address Cursor review: when every requested workflow fails to load, the admin export still returned 200 but recorded a successful export of zero workflows. Guard auditExport so an empty result records nothing. * fix(storage): settle copilot accounting before deleting the blob Address Cursor review: the blob was removed before releaseDeletedFileStorage, so a release failure left the counter inflated and the metadata active with the blob gone. Now the atomic soft-delete + decrement runs first and the blob is deleted only if it succeeds, so a failure leaves the file fully intact and retryable. * fix(storage): decrement KB document storage atomically with deletion Address Cursor review: hardDeleteDocuments deleted the rows then decremented best-effort, so a decrement failure left billed storage inflated with no row to reconcile. Resolve each owner's subscription up front, then decrement inside the same transaction that deletes the embeddings/documents (decrementStorageUsageInTx, also now shared by releaseDeletedFileStorage), so the counter and the content commit or roll back together. Connector docs remain excluded. * fix(audit): don't treat email verification as a login Address Cursor review: /verify-email could emit USER_LOGIN when verification mints or refreshes a session, mislabeling (or double-counting) a non-sign-in as a login. Restrict isLoginPath to genuine sign-in entrypoints. * fix(audit): null actor FK when the user lookup throws Address Greptile P1: the catch branch left the original actorId, so a system actor like 'admin-api' (or a since-deleted user) would FK-violate the insert and lose the row when the existence lookup errored. Mirror the not-found branch — null the FK with a readable label — so the audit row always persists. * revert(audit): drop session/account-lifecycle auth instrumentation Remove the Better Auth login/logout/session-revoke/account-delete/blocked-signin audit hooks from auth.ts — they touch sensitive auth paths and are noisy. Also removes the now-unused taxonomy (USER_LOGIN/_FAILED, USER_SIGNIN_BLOCKED, USER_SIGNUP_BLOCKED, USER_LOGOUT, SESSION_REVOKED, ACCOUNT_DELETED, ACCOUNT_EMAIL_CHANGED actions; SESSION/USER resource types) and the awaitable recordAuditNow helper that only the pre-delete hook used. auth.ts is back to the staging baseline. * fix(storage): harden copilot+KB delete accounting against read errors and concurrency Final-audit follow-ups: - deleteCopilotFile: a failed metadata *read* (vs a genuine missing row) now blocks the blob delete too, so a transient read error can't leave an active row un-decremented with the blob gone. - hardDeleteDocuments: drive the per-user decrement from the delete's returning() (the rows this tx actually removed), so two concurrent deletes of the same ids can't both decrement. * chore(observability): drop two unused definitions Final-audit cleanup: remove the orphaned knowledge_base_searched PostHog event (never wired; KB search analytics already flow through the OpenTelemetry channel) and the redundant AuditAction.SUBSCRIPTION_UPDATED (subscription/plan changes are audited via ORG_PLAN_CONVERTED). Every remaining new action/event has a real emit site. * fix(knowledge): key hard-delete result off rows actually deleted Address Cursor review: hardDeleteDocuments returned existingIds.length (the requested count) and cleaned storage for the full pre-tx set, even though the decrement was driven by the rows the transaction actually deleted. Under a concurrent delete that claimed some ids first, that overstated the result and re-touched storage for rows this call didn't delete. Drive the storage cleanup, log, and return value from deletedDocs (the returning() rows) so all four are consistent. * fix(storage): gate copilot quota on all ingest paths; tidy inline comments - Copilot uploads via the generic /api/files/upload route and presigned URLs now run checkStorageQuota before writing (matching uploadCopilotFile), so no copilot ingest path can grow usage past the plan limit. The central increment stays in insertFileMetadata. - Trim/remove redundant inline comments across the diff; keep only concise notes on non-obvious decisions (left pre-existing comments untouched). * fix(analytics): omit empty workspace_id on workspace-less file downloads Address Greptile P1: the generic key-based /api/files/download route and the no-workspace markdown-export path emitted file_downloaded with workspace_id:'' creating a phantom '' bucket in PostHog. Make workspace_id optional on the event and omit it when there is no workspace (workspace-scoped routes still pass it). * fix(analytics): omit empty workspace_id/workflow_id on copilot_chat_sent Final-sweep nit: copilot_chat_sent sent '' for workspace_id/workflow_id in the agent (workspace-less) branch, same phantom-bucket issue as file_downloaded. Make both properties optional and omit them when absent. * fix(analytics): clear stale org PostHog group on personal workspaces Address Cursor review: switching from a team workspace to a personal one left the previous organization group set, so later events kept rolling up under it. When the active workspace has no organizationId, resetGroups() to drop the stale org group, then re-apply the workspace group. * fix: address review — export audit timing, copilot delete signal, group reset - Table export (Cursor MED): audit fires before streaming begins, not after controller.close(), so a mid-stream failure still records the partial export. - deleteCopilotFile (Greptile P1): throw when storage accounting can't be settled instead of silently returning, so callers can detect the file was not deleted. - workspace-scope-sync (Cursor MED): only resetGroups() once workspace metadata is loaded (activeWorkspace present), so a team workspace doesn't transiently lose its org group while organizationId is still null during load. * refactor(observability): final line-justification cleanup Address final audit flags: - Table import: move the 'columns added' audit OUT of the import transaction into a post-commit auditTableColumnsAdded() helper called by the three tx-owning callers, so a mid-import row-batch rollback no longer logs a false 'added N columns' (matches the PR's success-only discipline). - Omit empty-string analytics dimensions on workflow_lock_toggled (workspace_id) and organization_created (name), consistent with file_downloaded/copilot_chat_sent. - Restore an unrelated capacity-check comment removed incidentally. - Move copilot_chat_sent emit above the traceparent comment so the comment sits with the code it documents. * fix(table): attribute import column audit to the importing user Address Cursor review: addImportColumns (async createColumns import path) now threads the importing userId into auditTableColumnsAdded instead of falling back to table.createdBy, so column additions are attributed to the actual member who ran the import rather than the table creator. * feat(observability): drop copilot from storage accounting Per design decision: copilot files are working/conversational artifacts, so gating their materialization on storage quota would fail an agent operation mid-flow when a user is over limit, and metering them would inflate usage enough to block KB/workspace uploads indirectly. Remove copilot quota gates + copilot ingest metering entirely (revert copilot-file-manager, metadata, and the upload route's copilot branch to baseline) and drop the now-unused releaseDeletedFileStorage. KB document metering + quota enforcement (the deliberate, persistent storage path) is unchanged. * fix(analytics): switch workspace + org PostHog groups together Address Cursor review: gate the group-sync effect on workspace metadata being loaded (activeWorkspace present) so the workspace and organization groups always update atomically. Acting during the load window paired the new workspace group with the previous workspace's org group; until metadata loads, events stay consistently attributed to the previous workspace. * fix(table): audit async export at authorization, not job completion Address Cursor review: async exports only emitted TABLE_EXPORTED when the background job reached 'ready', so an authorized export whose job later failed or was abandoned left no audit trail — inconsistent with the sync export route, which audits before streaming. Move the audit + analytics to the async route's authorization point (after the job is claimed/dispatched) and remove it from the runner. Drop the now-unused userId from TableExportPayload. * fix(billing): never let payment_failed instrumentation skip user blocking Address Greptile P1: the payment_failed audit hoisted an unguarded isSubscriptionOrgScoped DB read (for entity_type) directly before the attempt-count user-blocking block. A transient failure of that read would throw out of the handler and skip blocking. Wrap the whole audit/analytics block in try/catch (best-effort), and let the blocking compute its own isSubscriptionOrgScoped as it did originally — instrumentation can no longer abort payment processing. * chore(observability): trim verbose inline comments to concise notes * fix(billing): wrap new dispute/enterprise/subscription instrumentation in Stripe webhook idempotency recordAudit/captureServerEvent calls added for charge disputes, enterprise subscription provisioning, and free->paid subscription creation ran unconditionally with no idempotency guard, unlike their sibling handlers in the same files. Stripe redelivers webhooks at-least-once, so a retry would double-record the audit row and PostHog event even though the underlying DB writes were already idempotent. * fix(observability): tag org-scoped audit metadata with organizationId, guard concurrent table delete The org-scoped self-service audit-log endpoint matches org-level rows via metadata.organizationId (or resourceType=organization). Six recordAudit call sites (subscription create/cancel, admin credit issuance, threshold overage billing, charge disputes, credit purchase, invoice payment succeeded/failed) tagged org-scoped events with a differently-named key (referenceId/entityId/targetOrgId), making them invisible to org admins querying their own audit trail despite being stored in the DB. Add the missing organizationId key everywhere the pattern was missed. Also guard deleteTable's archive UPDATE with isNull(archivedAt) so a concurrent duplicate delete request is a no-op instead of re-archiving and re-firing a duplicate TABLE_DELETED audit row. Adds test coverage for the ORG_MEMBER_ADDED audit/analytics emission in acceptInvitation, which previously had none. * fix(test): queue resolveBillingActorId's owner lookup in payment-failure email test handleInvoicePaymentFailed's new payment_failed audit instrumentation resolves the billing actor via an extra db.select before sendPaymentFailureEmails runs. The test's fixed select-response queue didn't account for it, so the org-admin lookup consumed the wrong queued row and the assertion saw zero email sends. Production behavior is unaffected — each query is independent; this was a mock-queue ordering issue only. * fix(billing): don't let a post-commit usage-limit sync failure suppress the seat audit Cursor Bugbot flagged: reconcileOrganizationSeats committed the seat change and Stripe outbox enqueue in a transaction, then called syncSubscriptionUsageLimits outside it before recording the audit/ PostHog events. A thrown sync left a genuinely-changed seat count with no ORG_SEAT_PROVISIONED/DEPROVISIONED trail. Wrap the sync in its own try/catch (log + continue) so the events always fire for a committed change, matching the fire-and-forget instrumentation pattern used elsewhere in this PR. * fix(billing): guard the new subscription-created instrumentation block Greptile P1: the org-scope check I added for the audit-metadata fix (isSubscriptionOrgScoped) was a raw DB call with no error guard, unlike resolveSubscriptionActorId next to it. Since it ran inside the idempotency lambda with an unconditional rethrow, a transient DB error would abort and retry the whole webhook after the free -> paid usage reset had already committed. Wrap the actor/org-scope resolution + audit + analytics block in try/catch, matching the same guarded instrumentation pattern already used in handleInvoicePaymentFailed.	2026-07-08 23:23:59 -07:00
Waleed	d078c84ee6	chore(typescript): upgrade to TypeScript 7 (native Go compiler) (#5521) ... * chore(typescript): upgrade to TypeScript 7 (native Go compiler) Bumps typescript to ^7.0.2 across every workspace package. Full bun run type-check/lint/build/test all pass; apps/sim's type-check (the one needing an 8GB heap bump) drops from ~55s to ~7s wall time. Migration fixes required by TS7's stricter defaults: - baseUrl removed: drop it from 5 tsconfigs (paths already resolved relative to tsconfig dir, so behavior is unchanged) and prefix the one bare (non-relative) paths entry each in apps/sim and apps/realtime with './' - moduleResolution=node10 removed: switch packages/cli and packages/ts-sdk to "bundler", matching the rest of the monorepo - types now defaults to [] instead of auto-including every @types/* package: add "types": ["node"] to the shared base tsconfig (this is fundamentally a Node monorepo, so this restores prior behavior in one place instead of duplicating it per-package), add explicit @types/node deps to packages that now rely on it transitively via @sim/db/@sim/logger, and add "declare module '*.css'" to the two packages with plain (non-module) CSS side-effect imports that TS7's stricter checker now flags - packages/logger's isomorphic `typeof window` check no longer needs DOM lib in every consumer: replaced with `'window' in globalThis` - packages/testing and apps/realtime's fetch/DOM mocks need DOM lib where they're compiled, since they model the browser Fetch API - the `typescript` npm package no longer exports the classic Compiler API from its main entry (moved to unstable/ast subpaths); apps/sim's Function-block route used it at runtime to strip import statements from user code, so that one call site now uses Microsoft's official transition package, @typescript/typescript6 - Next.js 16.2.6's own TypeScript-detection heuristic hardcodes a path TS7 no longer ships, and its auto-install fallback assumes npm/pnpm; added @typescript/native-preview as a devDependency to apps/sim and apps/docs so Next detects a valid native compiler instead of trying (and failing) to auto-install one Not merging yet: TS 7.0.2 published today and is still inside this repo's bunfig.toml minimumReleaseAge (7-day) supply-chain gate, so `bun install` will fail for everyone until 2026-07-15. Opening this now to get it through review; hold the actual merge until then. * fix(typescript): address Greptile review findings on TS7 upgrade - packages/logger: 'window' in globalThis treats a shim that leaves globalThis.window explicitly undefined as browser-only, silently dropping production server logs. Restore the original typeof !== 'undefined' semantics via an inline cast instead, so it stays correct without requiring DOM lib in every consumer. - packages/ts-sdk, packages/cli: both are tsc-built, published as Node ESM (package.json "type": "module" with an "exports" map). "moduleResolution": "bundler" is too permissive for that target - it accepts import patterns (e.g. extensionless relative imports) that Node's actual ESM resolver rejects at runtime. Switch both to "module"/"moduleResolution": "nodenext", the correct pairing for a published Node ESM package. Verified real tsc builds (not just --noEmit) still succeed for both. * chore(bunfig): temporarily disable minimumReleaseAge gate for TS7 install TS 7.0.2 published today, still inside the 7-day gate. Lowering to 0 to unblock this merge; will restore to 604800 in an immediate follow-up commit right after merging.	2026-07-08 16:41:23 -07:00
Vikhyath MondretiandSiddharth Ganesan	4a80374c03	improvement(forking): unlink, settings page migration, deployed chats, public apis, external mcp servers/tools, special subblocks (#5505) ... * bad checkpoint * stash * add unlink, move UI to settings pages * fix edge cases * remove dead code * remove migration 0255 ahead of staging merge (regenerated after) * regenerate workflow_mcp_server enum migration on top of staging (0257) * fix tests * more tests * address comments * consolidate fork migrations into 0257 (enum value + activity metadata indexes) * acquire MCP server locks before reads in attachment reconcile (TOCTOU) * move into ee folder + ui perm gates * use randomInt from @sim/utils/random for chat identifier suffix --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com>	2026-07-08 12:15:28 -07:00
Theodore Li	8c61720d33	chore(tables): remove tables-fractional-ordering feature flag (#5503) ... * chore(tables): remove tables-fractional-ordering feature flag * improvement(tables): drop dead position tracking from paste undo path * improvement(tables): drop dead position from create-row undo records	2026-07-08 13:11:45 -04:00
Waleed	858f7d2c2c	fix(api-block): stop spoofing a browser fingerprint on outbound HTTP requests (#5496) ... * fix(api-block): stop spoofing a browser fingerprint on outbound HTTP requests getDefaultHeaders() sent a Chrome User-Agent, a Referer pointing at Sim's own app, and mismatched Sec-Ch-Ua client hints on every API block request. Atlassian's Jira/Confluence Cloud REST API (and other browser-aware anti-CSRF/bot-defense layers) reject requests carrying a browser User-Agent with 403 "XSRF check failed", even with a valid Basic-auth header and X-Atlassian-Token: no-check set. Default headers now identify honestly as Sim, matching how other HTTP clients (curl, Postman, axios) behave. User-supplied headers still override any default. * fix(testing): sync shared tool-tester mock headers with new defaults createMockHeaders in the shared @sim/testing tool-tester builder still hardcoded the old Chrome UA / Referer / Sec-Ch-Ua fallback values. It was unreachable in current tests but would silently diverge from production getDefaultHeaders() for any future test hitting its fallback path.	2026-07-07 16:58:13 -07:00
Theodore Li	56cd2cfa7c	feat(custom-block): audit-log publish, update, and unpublish events (#5493)	2026-07-07 18:58:17 -04:00
Vikhyath Mondreti	24ebba9acc	chore(credential-sets): cleanup feature (#5460)	2026-07-06 18:49:06 -07:00
Waleed	4b133984c6	fix(mcp): make SSRF-guarded fetch structurally mandatory in OAuth auth() calls (#5419) ... PR #5399 fixed the callback route forgetting to pass fetchFn into the SDK's auth(), but every call site (start + callback routes) still imported the raw SDK auth() directly and had to remember to pass fetchFn: createSsrfGuardedMcpFetch() by hand — the same omission was possible again at any future call site. Add mcpAuthGuarded() in lib/mcp/oauth/auth.ts, a thin wrapper around the SDK's auth() that always defaults fetchFn to the SSRF-guarded fetch (still overridable for tests). Both routes now import mcpAuthGuarded from @/lib/mcp/oauth instead of the raw SDK auth, so omitting the guard is no longer possible by omission.	2026-07-04 15:40:38 -07:00
Theodore Li	69b81a679b	feat(data-retention): granular PII redaction stages (input + block outputs) (#5272) ... * feat(data-retention): granular PII redaction stages (input + block outputs) * fix(data-retention): propagate block-output redaction into child workflows * fix(data-retention): close block-output redaction gaps on streaming + resume * fix(data-retention): drain+mask streamed output, resolve PII policy unconditionally (no fail-open) * test(testing): support leftJoin().where().limit() in shared db mock * fix(data-retention): mask agent/Pi memory writes under block-output redaction * fix(data-retention): guard partial PII stages in GET normalize * fix(data-retention): mask seeded memory messages under block-output redaction * fix(guardrails): fail closed on misaligned Presidio batch responses * fix(data-retention): enabled stage with no entity types redacts all (no fail-open) * fix(data-retention): reject enabled stage with no entity types; empty = off everywhere * docs(data-retention): note resume remask covers inline values only * fix(data-retention): scrub offloaded large-value refs from logs when block-output redaction is off * fix(data-retention): hydrate, mask, and re-store large-value refs in logs (preserve redacted content) * fix(data-retention): always apply logs policy to large-value refs when logs stage is on * perf(data-retention): drop redaction byte ceiling, parallelize chunks (env-tunable), remove request timeouts, sync large-value walk * feat(data-retention): gate granular PII stages behind pii-granular-redaction flag - New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION), layered on pii-redaction, gating the execution-altering input + block-output stages - Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off - UI shows only the Logs stage tab unless the flag is on; clamps active stage - Drop the per-search Select all toggle; add a Deselect all action to the PII section header * docs(pii): describe Presidio as a standalone service, not a sidecar Presidio now runs as its own ECS service (and, in Helm, its own Deployment + Service) reached over the network via PII_URL — not a sidecar in the app task. Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to match, and note the deploy requirement that PII_URL must be reachable. * fix(data-retention): re-mask offloaded large-value refs on resume + don't lock out granular saves - Resume/run-from-block restore now hydrates → masks → re-stores large-value refs in restored blockStates (not just inline strings), so a value offloaded before the block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast. - pii-large-values: add onFailure mode (throw on the execution path, scrub for logs) and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values - Granular flag gate now rejects only NEW off→on granular enablement, so orgs that already configured granular stages can still save retention settings when the flag is off	2026-07-01 21:47:02 -04:00
Waleed	af87de09a7	fix(connectors): allow self-hosted private DB hosts via opt-in flag (#5322) ... * fix(connectors): allow self-hosted private DB hosts via opt-in flag Database/connector tools rejected any host resolving to a private/reserved/ loopback IP, blocking the common self-hosted topology where the DB is reached by a Docker/K8s/Swarm service name. Add an opt-in ALLOW_PRIVATE_DATABASE_HOSTS flag that bypasses the private-host block in validateDatabaseHost while still resolving and pinning DNS. Blocked on the hosted platform regardless of the env var, mirroring DISABLE_AUTH. Fixes #4319 * fix(connectors): pin postgres IP in all ssl modes; strip IPv6 brackets Address review on #5322: - validateDatabaseHost now strips surrounding IPv6 brackets before the localhost/private-IP checks and DNS lookup, so a bracketed loopback like [::1] is classified correctly instead of failing as unresolvable. - PostgreSQL connector always connects to the validated, pinned IP (removed the ssl='preferred' carve-out that passed the original hostname and let the driver re-resolve during connection). Matches the MySQL/MongoDB pin pattern. - Add postgres connector pinning tests and bracketed-IPv6 host tests. * fix(connectors): rename flag to isPrivateDatabaseHostsAllowed; trim comment - Rename env-flag const to satisfy the env-flags 'is' prefix CI check (env var ALLOW_PRIVATE_DATABASE_HOSTS is unchanged). - Tighten the postgres pinning comment to a single line.	2026-07-01 11:19:08 -07:00
Vikhyath Mondreti	c59631698f	chore(deploy): remove deploy as a2a (#5255) ... * chore(deploy): remove a2a * add block	2026-06-28 20:01:24 -07:00
Vikhyath Mondreti	5a8134119a	feat(workspaces): fork + push/pull (#5210) ... * feat(workspaces): fork + push/pull * type fix * fix tests * progress on ux * remove modal section * improve UI of modal * update more ui * make rollback part of the footer * track skipped count correctly * address comments * make it workspace admin level * update skipped count * address more comments * deal with unbounded memory possibility * fix deleted kb article bug * no deployed workflow case * UI/UX cleanup * fix oauth dropdown case * fix oauth selector issue * infra work + activity log * consolidate migration * update modal state * more UI simplification * grammar * update audit report ui * perf improvements * fix tool input scenarios and add dependsOn UI handling * minor comments * fix webhook stability issues + drift detection removal * make dependsOn subblock mapping cleanly stored * fix: harden fork dependent-value mapping (clear stale rows, identity-guard first-sync fallback, perf + cleanup) * address comments * update comment * enforce admin perms for activity api * fix required + dependsOn combo	2026-06-28 13:13:57 -07:00
Vikhyath Mondreti	8b5d746fe2	improvement(access-controls): ui/ux improvements (#5190) ... * improvement(access-controls): ui/ux improvements * remove unused col	2026-06-23 17:35:41 -07:00
Vikhyath Mondreti	91f9dfdaec	improvement(governance): derived access (#5134) ... * improvement(governance): org-ws-credential roles clarity * revert isHosted * improvement(credentials): code cleanup * address comments * make kb cascade delete on user hard delete * revert env flags * chore(db): drop local 0242 migration to regenerate after merging staging Our 0242 collides with staging's 0242. Remove it (and its snapshot + journal entry) so the KB-cascade migration can be regenerated with the correct number on top of the merged staging migrations. * chore(db): regenerate kb→workspace cascade migration as 0243 Regenerated via drizzle-kit generate on top of the merged staging migrations (staging took 0242). Re-applied the safety edits: NOT VALID + separate VALIDATE on the FK re-add, and the -- migration-safe note on the DROP. check:migrations passes. * improve copy * update docs	2026-06-19 12:47:09 -07:00
Theodore Li	f0b3550729	feat(files): public share links for workspace files (#5130) ... * feat(files): public share links for workspace files * improvement(files): drop reserved public_share columns until used; sync audit mock * fix(files): share modal tracks authoritative saved state until toggled * feat(files): per-IP rate limit on public share endpoints * fix(files): address PR review — public CSV OOM, content cache, share FK, soft-delete filter, download anchor * fix(files): disable CSV import action in read-only preview (public share) * refactor(files): drive CSV preview import affordance off readOnly, not disableImport * fix(files): version public viewer caches by file updatedAt so edits aren't stale * fix(files): 409 (not corrupt source) when a shared generated doc has no compiled artifact * feat(files): gate public sharing behind an access-control permission	2026-06-18 21:26:49 -04:00
Vikhyath Mondreti	267e49c69c	improvement(workspaces): auto-add without invite if part of organization (#5132) ... * feat(workspaces): auto-add without invite if part of organization * reverse feature flag hardcoding * address comments * improve ux for org invite modal	2026-06-18 15:48:02 -07:00
Waleed	7d46103d09	chore(deps): remove unused dependencies and harden CI supply chain (#5119) ... * chore(deps): remove unused dependencies and harden CI supply chain Dependency cleanup: - Remove unused deps: papaparse, unified, and 6 unused Radix primitives (alert-dialog, radio-group, scroll-area, separator, toggle, visually-hidden) plus @tanstack/react-query-devtools (all verified zero imports repo-wide) - Consolidate jwt-decode into the existing jose dependency (decodeJwt) - Migrate react-window to @tanstack/react-virtual to drop a redundant virtualization library (terminal, structured-output, code viewer) - Remove the better-auth-harmony plugin and its gating env flag Supply-chain hardening: - SHA-pin every GitHub Action to a full commit SHA with a version comment - Pin CI bun-version to 1.3.13 (was "latest" in the release job) - Raise bun minimumReleaseAge cooldown from 3 to 7 days - Add a non-blocking `bun audit` step in CI - Add a CODEOWNERS gate routing dependency-manifest changes to @simstudioai/deps * chore(deps): remove unused apps/docs dependencies (@tabler/icons-react, dotenv-cli) * style(search-modal): use Send icon for Invite teammates action * feat(search-modal): surface New chat as the top action above Create workflow * feat(search-modal): add Secrets to the pages list	2026-06-17 14:56:43 -07:00
Vikhyath Mondreti	7b4626e547	improvement(perm-groups): allow workspace filter for permission groups (#5070) ... * improvement(perm-groups): allow workspace filter for permission groups * show errors correctly * address comments * address concurrent edit concern * address locks * address comments" * index migration safety * address at route level	2026-06-15 20:52:01 -07:00
Theodore Li	3fe061e3b3	feat(feature-flags): AppConfig-backed gated feature flags (#5059) ... * feat(feature-flags): AppConfig-backed gated feature flags * fix(ci): repoint 'Validate feature flags' step to env-flags.ts after rename * improvement(feature-flags): drop in-code defaults; fallback resolves a per-flag secret, gating is AppConfig-only * improvement(feature-flags): make flag names a closed set so every flag requires a fallback secret * improvement(feature-flags): single FEATURE_FLAGS registry — each entry defines name, description, and fallback in one place * improvement(feature-flags): fallback is the env secret key (keyof typeof env), resolved to a boolean	2026-06-15 21:56:12 -04:00
Vikhyath Mondreti	3a796f083d	improvement(permissions): permission groups scoped to organization level (#5035) ... * improvement(permissions): permission groups scoped to organization level * chore(db): drop 0235 permission-groups migration to regenerate after staging merge * merge latest staging	2026-06-13 17:56:10 -07:00
Theodore LiandClaude Opus 4.8	53fdcab5d6	feat(tables): background jobs (delete/export/backfill on trigger.dev) + tenant-scoped query performance (#4915) ... * feat(tables): paginated background row-delete jobs via table_jobs * fix(tables): address review on async row-delete (filtered count, scoped optimistic clear, Cmd+A select-all, hide delete from tray) * improvement(tables): filter-aware select-all runs, delete-job read mask, keyset index + autovacuum tuning * feat(tables): run import/delete/export/backfill jobs on trigger.dev with in-process fallback * improvement(tables): raise delete page to 10k and export batch to 5k * improvement(tables): raise CSV import batch to 5k rows (param-cap bounded) * feat(tables): surface export jobs in the header tray with progress, cancel, and download * improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray * Revert "improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray" This reverts commit 1ea5871610. * fix(tables): preserve export storage key (NoSuchKey) and unify jobs in one spinner tray * improvement(tables): jobs tray icon reflects aggregate state (spinner/check/alert) * fix(tables): restore jobs tray on the tables list (dropped in staging merge) * improvement(tables): keyset-paginate export row reads (offset paging was O(n^2) over large tables) * perf(tables): keyset pagination for grid infinite scroll Default-order row pages now cursor on (order_key, id) instead of OFFSET — each page is an index seek on tableOrderKeyIdx, where OFFSET re-scans and discards every prior row (O(N²) across deep scrolls and full drains like select-all/export-to-clipboard). Sorted views keep offset paging; the contract refines after+sort as mutually exclusive. v1 public rows API is unchanged (extends the unrefined base, omits after). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): show export in job tray immediately on kickoff The export-jobs query's poll only self-sustains once a running job is already in the cache, so a freshly kicked export stayed invisible until an SSE event or page refresh. Invalidate the tray query on kickoff success so the icon appears right away. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): surface real row/column write errors in toasts Drizzle wraps DB errors in DrizzleQueryError whose message is the failed SQL — the real cause (e.g. the row-limit trigger's RAISE) sits on .cause, so the routes' substring classification never matched and everything fell through to generic 500s ("Failed to insert row"). Add rootErrorMessage (cause-chain unwrap) and a shared rowWriteErrorResponse classifier that consolidates the per-route pattern lists and rewrites the trigger message into a friendly "Row limit exceeded — capped at N rows". Applied across the app and v1 row-write routes and the columns route. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-bound filtered row counts (12.7s -> 0.6s) JSONB filter predicates (->> ILIKE / range casts) are opaque to the planner: it estimates a handful of matches and picks a parallel seq scan over the entire shared user_table_rows relation — every tenant's rows — for the page-0 COUNT(*), so any non-equality filter on a large table cost 10s+ regardless of how few rows matched. Run filtered counts in a transaction with SET LOCAL enable_seqscan = off, forcing the tenant-bounded bitmap plan. Unfiltered counts keep their index-only scan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-bound Cmd+F search and stream its window (75s -> 2s) Same planner trap as the filtered count, compounded: the lateral jsonb_each_text ILIKE is unestimatable, so findRowMatches on a 1M-row table seq-scanned the whole 12M-row shared relation and disk-sorted ~120MB of window input (75s measured). SET LOCAL enable_seqscan=off bounds the scan to the tenant; on the default order, additionally penalizing bitmap/sort/parallel steers the planner onto the already- sorted (table_id, order_key, id) index walk so row_number() streams with no sort at all (2s measured). Flags only penalize plan shapes — a custom sort still sorts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-bound sorted pages and filtered write selections Extends the seqscan fix to every remaining jsonb-predicate path, all measured on a 1M-row table in a 12M-row shared relation: - sorted page query (ORDER BY data->>'col'): 9.7s/page -> 0.76s, and deep pages stop spilling ~130MB sorts to disk - updateRowsByFilter / deleteRowsByFilter row selection: 14.4s -> bounded - delete-job worker selectRowIdPage with a filter: 12.6s/page -> bounded - dispatcher filtered-scope window walk: same shape, same fix Shared withSeqscanOff helper moves to lib/table/planner.ts (service + dispatcher both consume it; dispatcher can't import service). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-scoped containment index (migration 0232) The plain GIN on user_table_rows.data matched @> candidates across every tenant sharing the relation — a hot value in someone else's table inflated everyone's equality filters (1.07M candidates fetched for a 33k-row match, lossy bitmap, 1.1s). Replace it with btree_gin (table_id, data jsonb_path_ops): the tenant intersection happens inside the index and paths are single hashed entries. Rare-equality probe 326ms -> 17ms with zero wasted candidates; unique-constraint checks and upsert conflict lookups ride the same index. The new index is smaller than the one it replaces (529MB vs 694MB on the 12M-row dev relation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-bound unique-constraint checks (3.5s -> <1s per write) The unique check runs lower(data->>'col') = $1 LIMIT 1 on every insert and cell edit touching a unique column. The predicate is unestimatable and a unique (non-conflicting) value never exits early, so the planner seq-scanned all 12.3M shared-relation rows per check — 3.5s measured. Tenant-bound both the single and batch variants; the batch path sets the flag on the caller's transaction when one is supplied (SET LOCAL dies at its commit, and the statements that follow are tenant-scoped writes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf(tables): tenant-bound upsert conflict lookup Same unestimatable data->>key predicate as the unique checks; an insert-path upsert has no existing match so the lookup can't exit early and seq-scans the whole shared relation. The upsert already runs in a transaction — set the planner flag on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tables): consolidate executor types onto planner exports service.ts kept a private DbTransaction alias and two inline typeof db | DbTransaction unions after planner.ts began exporting the canonical DbTransaction/DbExecutor — import those instead. From the /simplify review of the perf series; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tests): drop narrow schema mock override in process-contents test The local vi.mock('@sim/db/schema') stubbed only document/knowledgeBase, but the file's import graph reaches lib/table/service whose module scope now references tableJobs. The global schema mock already covers all of it — rely on it per the testing rules instead of re-mocking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): scope cancels and counts to the filtered selection (review) Addresses the open Bugbot/Greptile findings on filtered select-all: - Filtered runs no longer cancel the whole table: cancelWorkflowGroupRuns takes a filter — it stops only dispatches with that exact filter scope and only in-flight cells on matching rows (semi-join); whole-table and differently-scoped dispatches keep running, their cancelled cells skipped via cancelledAt > requestedAt. - Stop on a filtered select-all sends the filter through cancel-runs (contract + route + mutation) instead of a table-wide cancel. - runColumnBodySchema rejects rowIds + filter together (mirrors deleteTableRowsBodySchema). - Select-all delete clears the selection in onSuccess, not at click, so a failed kickoff restores both rows and selection. - Clipboard copy/cut estimates use the filter-aware total (rowTotal) instead of the whole-table rowCount. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: retrigger CI (Actions dropped the previous push events) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump api-validation route baseline to 807 (staging route + merge) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): release job claim when trigger.dev dispatch fails If tasks.trigger (or its dynamic imports) throws after markTableJobRunning, the ghost running row held the table's one-write-job slot until the stale-job janitor fired (~15-20 min of 409s). All four kickoff routes now release the claim and rethrow; the backfill runner releases and warns (a failed backfill never fails the schema change). Greptile P1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(db): squash 0232 into 0231 (one migration for the PR) Both are branch-only — no environment has applied them through the migration ledger yet — so the tenant-scoped GIN (btree_gin extension, index swap) folds into 0231_table_jobs_and_keyset. Snapshot chain re-pointed; drizzle-kit generate confirms zero drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): context-menu delete label shows the true select-all count Under select-all the context menu counted only the loaded page ("Delete 1000 rows" on a 999k-row table) while the action correctly deletes every matching row via the background job. Delete now gets its own count from the filter-aware total minus deselections; the run-action labels keep the loaded-row count since those actions act on loaded rows only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): context-menu bulk actions act on the full select-all scope Follow-up to the label fix: under select-all the context menu's Run / Re-run / Stop only acted on the loaded page of rows. They now route through the same scopes as the action bar — runs dispatch by filter (whole table when unfiltered), Stop uses the filter-scoped cancel — and all labels share one true count (filter-aware total minus deselections, locale-formatted). Like the action bar, filter-scoped runs ignore deselections (the run API has no exclusion set). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tables): exclusion set for select-all runs and stops Select-all minus deselected rows now means exactly that for every bulk action, not just delete. runColumnBodySchema and cancelTableRunsBodySchema accept excludeRowIds (bounded by MAX_EXCLUDE_ROW_IDS, select-all scope only); the dispatch scope persists it and the dispatcher window walk, eager bulk-clear, pre-run cancel, and filter/table-scoped cancel all skip excluded rows. Client threads exclusions from the selection through the action bar and the grid context menu, including the optimistic stamps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): spare excluded-row dispatches on Stop; no orphan placeholder table Two Bugbot findings on the exclusion work: - Select-all-minus-deselections Stop (no filter) cancelled every active dispatch table-wide, killing row-scoped runs on deselected rows. markActiveDispatchesCancelled now spares dispatches whose scope.rowIds are fully contained in the exclusion set (coalesce(false) keeps table-wide dispatches cancellable). - Create-mode import: a failed trigger.dev dispatch released the job claim but left the just-created placeholder table in the workspace. Archive it on the failure path (no hard-delete surface exists). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): row counts reflect a running delete everywhere A mid-delete refresh resurrected the old counts: the optimistic update stripped cached rows but left page-0 totalCount (footer / select-all label) at the old total, and list/detail counts reported raw row_count including doomed-but-not-yet-deleted rows. - onMutate now sets the active view's totalCount to the kept rows and decrements the cached detail rowCount by the doomed estimate - the kickoff persists that estimate on the job (payload.doomedCount, clamped server-side); getTableById/listTables subtract the not-yet-deleted remainder (doomedCount - rows_processed) while the delete runs, so refetched counts match the read path's delete mask Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * copy(tables): drop background mention from delete confirm Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): clear select-all immediately when a delete kicks off The header checkbox lingered as a minus over the optimistically-emptied grid: rowSelectionCoversAll treats zero rows as not-covered, and the selection clear waited for the kickoff's onSuccess. Clear at click (failed kickoffs visibly restore rows + toast; re-selecting is cheap) and render an empty grid's header checkbox unchecked regardless — a selection over zero rows is vacuous. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): export takes the async path while a delete job runs The sync/async export choice reads rowCount, which is a doomed-estimate- adjusted number during a running delete (and the estimate is client- supplied) — an overstated estimate could route a still-large masked set through the synchronous stream. Mid-delete exports now always run as a job: safe at any size, and exports bypass the one-job-per-table gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(build): stop uploads setup from sweeping the project into route graphs next build (Turbopack) failed with "Two or more assets with different content were emitted to the same output path" on the server-root chunk. Root cause: setup.server.ts's unscoped path.resolve(process.cwd()) made node-file-tracing sweep the entire project — next.config.ts included — into every route graph reaching lib/uploads (the files/upload route and, since the export job, the export-async path). Two producers emitted the swept config into same-named chunks; staging's latest commits made their contents diverge and the names collided. Annotate the path derivation with turbopackIgnore per the NFT warning's own remediation — the build passes and all ~390 "unexpected file in NFT list" warnings disappear. Also inline the releaseJobClaim dynamic imports in the kickoff routes to plain static imports — service is already statically imported there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-12 03:03:11 -04:00