796 Commits

Include renames

Author	SHA1	Message	Date
3 people	18214158b9	fix(sanitization): secret exposure in function and agent trace spans (#6000) ... * fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-01 16:37:08 -07:00
Waleed	28abbc94c6	perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) (#6151) ... * perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts) `turbopackFileSystemCacheForDev` has been `false` since #5408 — a landing-page homepage redesign whose description covers hero cards, feature-card aspect ratios, eyebrow chips and a voice-input button color, and never mentions Turbopack, caching, or dev performance. It was collateral, not a decision, and it overrode the Next default (true since v16.1). It is not the flag #6078/#6080 measured. That A/B was `...ForBuild` and its conclusion stands — the build cache is a 3.2x regression and stays off. The two flags look alike and are opposite decisions; both are now commented as such. Measured on `/workspace/[workspaceId]/w`, n=3 per arm, SIGINT between runs: cache OFF 31.4s / 30.1s / 31.9s RSS 9.0-9.8 GB cache ON 5.6s / 5.6s / 5.5s RSS 4.4-5.1 GB 5.4x faster restarts, ~2x less resident memory. Cold compile against an empty cache is unchanged (~32s either way) — the cache only pays back on restart, which is the loop that actually hurts. The cache is unbounded on disk: the abandoned one on this machine had reached 78 GB across 1,848 SST files, and a stale cache is slower to read back, so left alone it erodes the win it exists to provide. `prune-turbopack-cache.ts` runs on `predev` and drops it past a cap (default 20 GB, `SIM_TURBOPACK_CACHE_MAX_GB` to override); `bun run dev:cache:prune` forces it. It never blocks `next dev` on a maintenance failure. Adds a `dev-performance` skill recording the cost model, the reference numbers, and the benchmarking method — including that stopping the server with `kill -9` mid-cache-write discards the cache and makes this exact win read as no win. * improvement(dev): chain the cache prune into dev scripts instead of a predev hook Review read the root `bun run dev` path as bypassing the `predev` hook and so never capping the newly-enabled cache. Turbo does fire `pre*` hooks — verified live, the run prints the prune before `next dev` — but the concern is fair in that the guarantee rested on package-manager lifecycle semantics that are invisible at the call site. Chaining it explicitly removes the question entirely: every `dev` variant now runs `bun run dev:cache:cap && …`, which holds on any invocation path, is visible in the command itself, and drops the three duplicated `predev:*` entries for one shared script. Verified on both paths — direct `bun run dev` and root `turbo run dev`, the latter printing: sim:dev: $ bun run dev:cache:cap && next dev --port 3000 sim:dev: $ bun run ../../scripts/prune-turbopack-cache.ts * docs(dev): document cache-corruption recovery, the cost of enabling the cache Stress-tested the failure mode rather than assuming it: deliberately corrupting an SST block makes Turbopack abort with a FATAL panic — it does not self-heal. FATAL: An unexpected Turbopack error occurred. Cache corruption detected: checksum mismatch in block 4 of 00000221.sst `bun run dev:cache:prune` and restart fixes it; verified the canvas serves 200 again afterwards. Documented in the skill and in the script's header, since the symptom is a hard crash and the remedy is not guessable. This is the honest cost of turning the cache on. It is worth paying — a 5.4x faster restart against a rare, loud, single-command failure — but it should be written down rather than discovered. Worth distinguishing from the adjacent case: an ordinary hard kill does *not* corrupt the cache. Turbopack discards a partially-written cache and rebuilds it silently, which is exactly why a `kill -9`-based benchmark reads as "no cache win" (noted in the benchmarking section). * refactor(dev): drop the dev-performance skill, keep its findings at the code A whole skill was too much for what this is. The parts that are load-bearing — why the two lookalike cache flags are opposite decisions, the measured numbers, the corruption remedy, and the benchmarking trap — now live in the config and script they describe, where someone changing the flag actually reads them. The trap is the piece worth keeping: `next dev` compiles on demand so startup time is meaningless, and stopping the server with `kill -9` makes Turbopack discard a partially-written cache and rebuild silently — which reads as 'the cache does nothing' and is how this flag stayed wrong for a month. Dropped rather than relocated: generic advice that was not specific to this repo (antivirus, Docker-on-macOS, orphaned processes) and a measured no-op (`optimizePackageImports` for lucide-react changed nothing, 31.6s vs 31.7s). * docs(dev): record the measured cost and concurrency behaviour of cache pruning Stress-tested the maintenance path rather than assuming it is free. Cost: the size walk is ~30ms on a real cache and ~85ms at 2,000 files — under 2% of a 4.2s warm restart, and invisible against a cold one. It runs before every dev start, so it needed to be cheap; it is. Concurrency: pruning while a dev server is live (which happens when a second server is started from the same checkout) does not crash it. The running server keeps its in-memory state and kept serving HTTP 200 with zero panics. It does stop persisting for the rest of that session, so its next start is cold once — verified recovering at 23.4s then 4.5s. Worth writing down because the directory silently never reappears mid-session, which looks like a bug if you go looking. The cap is a backstop, not routine: a normal session sits at 1-2 GB against a 20 GB default. * fix(dev): cap every app's Turbopack cache, not just apps/sim `apps/docs` is a Next app too (`next dev --port 3001`) and overrides nothing, so it uses the Next default where the dev filesystem cache is on. It already had an uncapped 1.1 GB cache here, and the root `bun run dev` (`turbo run dev`) starts it — so a teammate using the documented command was accumulating a cache nothing would ever prune. The script now resolves its target from the working directory instead of hardcoding `apps/sim`, and each app chains its own cap. Per-app rather than one sweep on purpose: a single pass would let one app's dev start delete a cache another app is holding open, which costs that session its persistence. Verified both: `apps/sim` and `apps/docs` each report and cap their own 1.1 GB cache, and both dev servers start clean (`Ready in 299ms` / `229ms`, docs serving). * refactor(dev): drop dev:cache:prune in favour of the existing dev:clean `dev:cache:prune` duplicated `dev:clean`, which already existed in `apps/sim` and does strictly more (`rm -rf .next/dev/cache` covers the Turbopack cache plus the fetch and image caches). Two commands for one job is worse than one, and the docs pointed at the newer, narrower of the two. Removes it from both apps and gives `apps/docs` the `dev:clean` that `apps/sim` already had, so the recovery command is the same everywhere. `dev:cache:cap` stays — it is the chained step, used by more than one dev variant, and naming it keeps the relative script path out of each command. Verified `dev:clean` is a real remedy: corrupt a cache block, run it, restart — canvas serves 200 with no panic. Also corrects an overstatement. A damaged cache does not *always* abort Turbopack; whether it panics depends on whether the damaged region is read, so it is not reliably reproducible. Both notes now say "can abort" and give the same remedy either way.	2026-08-01 11:27:35 -07:00
Waleed	19b0312a04	feat(managed-agent): add session lifecycle operations (#6140) ... * feat(managed-agent): add session lifecycle operations Adds an operation selector to the Claude Managed Agents block, backed by nine new tools alongside the existing run-session behavior: - create session (non-blocking, seeds initial_events) - send message to an existing session - get session (surfaces tool calls awaiting approval) - list events - update session (title/metadata) - interrupt session - respond to tool confirmation (allow/deny) - archive session - delete session Run Session stays the default, so blocks saved before the selector existed keep their exact behavior and field layout. * fix(managed-agent): address review findings on session lifecycle ops - List Events kept the OLDEST slice when capped, dropping the agent's most recent reply. Paging is exhaustive again and the cap now keeps the newest N after ordering, with a `truncated` flag so callers know it is a tail. - listPaginated returned whole pages past maxItems; it now trims to the exact cap. - A whitespace-only title passed the update guard and would have cleared an existing session title. Blank is now treated as not provided. - Interrupt had no request timeout and could hang; bounded at 15s while still honoring the workflow signal. - Custom-tool gates were surfaced by Get Session but could not be answered, since they need user.custom_tool_result rather than a confirmation. Adds a Respond To Custom Tool operation and a `kind` on each pending gate so a workflow routes to the right one. - Docs rendered a raw ${DEFAULT_EVENT_LIMIT} placeholder for the default. * fix(managed-agent): correct truncation flag, gate lookup, custom tool result - `truncated` was true whenever the history size equalled the limit, even though nothing was dropped. Event reads now report the untrimmed total and the flag compares against that. - Pending-gate enrichment capped its read, which keeps the OLDEST events in page order — the opposite of where blocking gates live. It now filters to the ids being looked up as pages arrive, which is both correct regardless of page order and bounded by the id count. Paging continues on the raw page so a fully-filtered page is not mistaken for the end of the list. - Respond To Custom Tool applied one result to every id, so multiple pending tools would all receive the same output. It now answers a single call per invocation. * fix(managed-agent): stop fractional event limits reading unbounded A limit below 1 passed the positivity check and then floored to 0, which made `slice(-0)` hand back the ENTIRE history flagged as complete — the opposite of the requested bound. The limit is now floored before it is validated, so anything that does not resolve to a positive integer falls back to the default. Also hardened the library: a zero or negative cap short-circuits to an empty result instead of falling through to `slice(-0)`, so no future caller can hit the same trap. * fix(managed-agent): stop gate lookup scanning the full tool history The id filter keeps the collected array tiny, so `maxItems` never trips and the walk continued to the end of a session's tool history even after every blocking id had been found. `listPaginated` now takes a `stopWhen` predicate and the gate lookup ends as soon as it has all the ids it came for. Also makes a blocked-but-unnamed session observable: when a session reports `requires_action` with no blocking event ids, `requiresAction` stays true — reporting false would tell a workflow the session is fine while it is parked indefinitely — and the dead end is logged and documented instead. * fix(managed-agent): floor event cap and make metadata clearing explicit - A `maxItems` between 0 and 1 slipped past the zero guard and became `slice(-0)` — the whole history — because slice truncates its index toward zero. The cap is now floored at the library boundary, so no caller can hit it whatever they pass. - Update Session documented full metadata replacement but could not express a clear: an empty map normalizes to "absent". Inferring the clear from emptiness would be worse, since an untouched table is also empty and would wipe metadata on every title-only update. Adds an explicit `clearMetadata` instead, and corrects the parameter's documentation. * test(managed-agent): pin the HTTP shape of every session endpoint Method, URL, and beta header for all 11 calls, plus the SSE accept header, the separate memory-store beta (combining the two is a documented 400), and content-type only on requests that carry a body. These are the details types cannot catch and that break silently when a path is "tidied".	2026-07-31 17:59:20 -07:00
Theodore LiandClaude Opus 5	c5cc6ce26c	feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137) ... * feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 19:23:24 -04:00
Bill LeoutsakosandBill Leoutsakos	c0b19da782	feat(pi): install Bun in cloud sandboxes (#6123) ... Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-07-31 12:45:13 -07:00
Waleed	ed23330f88	feat(knowledge): opt-in hybrid lexical + vector retrieval for KB search (#6124) ... * feat(knowledge): hybrid lexical + vector retrieval for KB search KB search ranked purely on pgvector cosine distance, which retrieves exact tokens (error codes, ticket keys, identifiers, rare product names) poorly. Add a full-text leg over the already-present generated `embedding.content_tsv` column and its GIN index — no migration, no re-indexing — and fuse it with the vector leg by reciprocal rank. Both legs run concurrently and share the same visibility and tag-filter predicates; the lexical leg is best-effort and falls back to vector-only on failure. Hybrid is the default for every caller. `searchMode: 'vector'` on the internal and v1 contracts (and an advanced Retrieval Mode dropdown on the Knowledge block) restores the previous behavior. Both search routes now share one `executeKnowledgeSearch` dispatch instead of duplicating the three-branch retrieval logic. * change(knowledge): make vector the default search mode, hybrid opt-in Every existing caller — workflow block, v1 API, copilot, guardrail RAG — keeps its current ranking. Hybrid retrieval is now requested explicitly via `searchMode: 'hybrid'`. Also routes the copilot knowledge tool through the shared `executeKnowledgeSearch` dispatch so all four callers share one retrieval path, and documents `searchMode` on the public v1 search endpoint in the OpenAPI spec. * docs(knowledge): document the hybrid retrieval mode Regenerates the knowledge integration reference for the new searchMode tool param, and adds a Retrieval Mode section to the knowledge base workflow guide explaining when hybrid beats vector-only. * fix(knowledge): stop rank fusion from starving the lexical leg Rank n in one leg always ties rank n in the other, so ordering the fused list by score alone let whichever leg was scored first take every tied slot. At topK=1 that meant a hybrid search returned exactly the vector-only result and discarded the exact keyword match the mode exists to recover. Selection now orders by score and drains each tie group round-robin, taking from whichever leg has contributed fewest rows so far. The lexical leg is passed first so it wins a total tie, since a chunk the vector leg ranked below its distance threshold is the case hybrid was opted into for. * fix(knowledge): credit a shared hit to every leg that returned it Attributing a row found by both legs to a single leg left the round-robin owing the other leg a slot it had already been served. With a shared rank-1 hit and topK 2, that evicted the lexical-only row — the exact match hybrid was enabled to recover — in favor of the vector-only one. A shared row satisfied every leg that returned it, so every one of them is now charged for it. Tie-breaking prefers the candidate whose least-served leg has been served least, which also removes the arbitrary best-rank attribution. * fix(knowledge): reject a whitespace-only copilot query explicitly The shared dispatch treats a whitespace-only query as absent and throws when no tag filters accompany it, where the previous vector-only call would have embedded the blank string and searched. Tighten the existing guard so the tool returns its normal message instead. * fix(knowledge): fan the keyword leg out per knowledge base The vector leg caps candidates per base once getQueryStrategy sets useParallel, but the keyword leg always ran one global query with a single LIMIT. Searching several bases at once let whichever one ranks strongest lexically consume every slot, so an exact-token hit in a smaller base never reached fusion — the case hybrid exists to serve. The keyword leg now uses the same strategy: per-base queries under the same parallel limit, re-ranked globally on a selected ts_rank_cd. Both legs draw candidates the same way, so fusion combines rankings over the same pool. * perf(knowledge): stop the keyword leg detoasting every match's vector Selecting the cosine distance in the ranking query made Postgres detoast the 1536-dimension embedding and compute a distance for every full-text match before the LIMIT applied, so cost tracked how common the query term was rather than topK. On a 20k-chunk base with a term matching every row that was 61,055 buffer hits against 1,030 for the same query without the projection. Rank on ids and ts_rank_cd alone, then hydrate only the rows that survive the limit. Same results, and the worst case drops to ~27ms end to end.	2026-07-31 12:42:09 -07:00
Waleed	48aeac218c	fix(uploads): set Content-Type once on presigned PUTs; document x-goog-meta-folderid for GCS CORS (#6121) ... * fix(uploads): set Content-Type once on presigned PUTs; document x-goog-meta-folderid in the GCS CORS example XMLHttpRequest.setRequestHeader appends on repeated calls (values join with a comma), and GCS is the only provider whose signed uploadHeaders include Content-Type — so single-shot GCS uploads sent 'x, x', which fails V4 signature verification with 403 (headers canonicalize to a comma-separated value that must match what was signed; multipart part PUTs are unaffected since part URLs don't sign Content-Type). The client now sets its default Content-Type only when the server's signed headers don't already carry one, with regression tests for both paths. Also adds x-goog-meta-folderid to the documented GCS CORS responseHeader list — workspace uploads now sign a folderId metadata header, and GCS CORS matches preflight request headers against that list exactly (no wildcards), so the missing entry blocked browser uploads into folders. * chore(uploads): drop inline comment	2026-07-30 22:22:56 -07:00
Waleed	ee0157df4b	feat(tables): add currency column type on a new column-type registry (#6106) ... * feat(tables): add currency column type on a new column-type registry Adds a `currency` column type, and consolidates the per-type knowledge it would otherwise have been scattered across. **Currency.** Stores a plain number and carries an ISO 4217 `currencyCode` as display metadata. That split is what keeps it cheap: filtering, sorting, uniqueness and CSV export all reuse the numeric paths unchanged, changing a column's currency rewrites no rows, and the public row output stays a number rather than a locale-formatted string consumers would have to reparse. Input accepts the shapes an amount actually arrives in — `$1,234.56`, `1 234,56 €`, `(12.00)` — so pastes, CSV imports and tool writes land as numbers instead of being nulled. **The registry.** Adding this type initially required edits in ~40 places: 32 switch arms under `lib/table`, ~26 UI branches, two hand-maintained icon maps, and a coercion implementation duplicated four times. Every one of those failed silently when missed — a missing `jsonbCastForType` arm compares numbers as text; a missing compatibility arm blocks all conversions. `lib/table/column-types/` now holds one file per type carrying its label, icon, badge colour, storage cast, filter operators, coercion, validation, compatibility and formatting. `Record<ColumnType, …>` on both registries is the completeness gate: adding a type to the union is a compile error naming exactly the two files to fill in, and the interface then requires every field. The 32 switch arms are down to 3. Two duplicates collapse as a consequence: - The client no longer mirrors the server's select id-resolution. Those helpers lived in `validation.ts`, which imports drizzle, so anything reaching them became server-only and the grid hand-rolled its own copy. Extracting them to `select-options.ts` lets both sides share one implementation, so the optimistic cache can no longer disagree with what gets persisted. - The two icon maps become one registry read. It also fixes a live inconsistency it surfaced: currency got a numeric keypad in the grid's inline editor but a plain text field in the row modal. Behaviour-neutral by construction: all 1046 tests in the touched areas pass unchanged, with no test edits. * test(tables): guard the column-type registry's invariants Property tests for the registry itself rather than any one type: entries key by their own id, COLUMN_TYPES stays derived, an unknown type degrades to string instead of throwing, only opaque-id types restrict filter operators, only configuration-free types are CSV-inferable, and every type that can reject a draft has a message to show. Plus the metadata-ownership matrix, which pins the generic ownership check to the same answers the hardcoded per-type rules gave. These target the registry's silent-failure class — a wrong jsonbCast or a stray operator whitelist used to be invisible until a filter failed in SQL. Both are verified to fail under mutation. * fix(tables): read exponent-form amounts and reject bad currency PATCHes up front Two P1s from review. Scientific notation lost magnitude. `String()` emits exponent form past 1e21, so a stored amount round-trips through the editor as `1e+21` — and the sanitizer treated the `e` as decoration to strip, reading it back as 121. An untouched cell silently lost 19 orders of magnitude on its next edit. Exponent form is now taken at face value, but only when the string is wholly a numeric literal once symbols are removed, so `12 EUR` (whose `E` survives the strip) still parses through the separator path. A failed currency PATCH left a partial rename. `renameColumn` commits in its own transaction before the currency write, so a `currencyCode` the service would reject — an unsupported code, or any code on a non-currency column — errored only after the rename had stuck. Both are now caught before the first write, matching the guard the route already applies to unique-on-select for exactly this reason. * refactor(tables): finish the registry migration and drop the dead config Audit pass over every consumer, closing the gaps the first cut left. Functional gap: the copilot agent had no currency support at all — it could create a currency column with no code and could never re-denominate one. `add_column` and `update_column` now accept `currencyCode`, with the same up-front validation and the same code-only routing as the HTTP routes. Config that consumers were still restating, now read from the registry: - `supportsUnique` replaces the unique-on-select guard stated in three places (service, both column routes, the copilot tool). - `editor === 'toggle'` replaces seven `type === 'boolean'` checks in the grid and expanded popover, all of which meant the same thing. - `defaultMetadata` replaces the per-type stamping in `addTableColumn` and `updateColumnType`. - `sampleValue` replaces the per-type example values in the LLM prompt scaffolding. - `storesOpaqueIds` replaces the select filter in the find-row matcher. Dead config removed: `getTypeBadgeVariant` had zero callers (already dead on staging), and it was the only reader of `badgeVariant` — so the field, its union, and all seven values went with it. `inferFromCsv` was read by nothing but a comment; CSV inference is an ordered heuristic a boolean cannot express, so it is gone too and `InferredCsvColumnType` is no longer exported. Fixes a latent crash found on the way: unique-constraint checking normalized a cell keyed on its RUNTIME type but reconstructed it keyed on the column's DECLARED type, so a unique `date` column stored a bare `2024-01-01` and then threw `SyntaxError` parsing it back. Both directions now go through JSON unconditionally. Pre-existing, unrelated to currency. Adds the `/add-column-type` skill and a Tables section in CLAUDE.md/AGENTS.md pointing at it, so the next type is one file plus two registry entries. * fix(tables): run the column PATCH guards ahead of the rename, not after it Greptile was right and my previous reply was wrong. The guards were added in the right shape but the wrong place — below `renameColumn`, which is the first write and commits in its own transaction. A PATCH combining a rename with an invalid currency therefore still committed the rename and then returned 400, exactly the counterexample reported. Moved the column lookup and all three pre-flight guards above every write. This also closes the same latent hole for the pre-existing unique-on-select guard, which sat in the same position. Adds route tests that assert `renameColumn` was never called on each rejection path, and that a valid combined rename + currency change still targets the new name. Verified to fail against the previous ordering. * fix(tables): make the retype gate and the write path share one parser A simplify pass over the registry found two real defects and several places the abstraction was being worked around. Silent data loss on conversion. `isCompatibleWith` was hand-written per type and had already drifted from `coerce`, despite the interface promising they could not: `boolean` accepted '1'/'0'/0/1 in the gate but only 'true'/'false' in the write path, so converting a column holding "1" reported zero incompatible rows and then nulled every one of them. `date` drifted the other way. `isCompatibleWith` is now optional and defaults to `coerce(...).ok`, so the two are the same code; only `select` overrides, because its rules are about the column (cleared-vs-required, cardinality) not the value. `isColumnType` used `in`, which matches inherited keys — `isColumnType('toString')` was true and `columnTypeById('toString')` returned `Function.prototype.toString`, which the validator would then call `.validateDefinition()` on. Now `Object.hasOwn`. `defaultMetadata` only ran on the currency arm of a retype, so a future type would get its defaults on create but silently not on conversion. It now runs for every non-select target, carrying forward only metadata the TARGET type declares it owns — a currency→text conversion no longer strands a currencyCode. The index doc claimed the registry is kept out of the `@/lib/table` barrel so 44 server modules don't pull `@sim/emcn/icons`. That was false: `constants.ts` re-exported `COLUMN_TYPES` from the icon-carrying `registry.ts`, and the barrel re-exports `constants`. `COLUMN_TYPES` now lives in the icon-free `types.ts`; verified with an import tracer that both are icon-free again. Also: 5 no-op `validateDefinition`s and 4 duplicated formatters collapsed into registry defaults; `CURRENCY_OPTIONS` was an eager module-load IIFE costing ~8ms of ICU work on every table API route for a list only the config sidebar reads, now built on first call; and the skill's validation grep claimed 'should return nothing' when it returns 8 legitimate hits — it now explains how to tell a leak from a genuine special case. * fix(tables): reject a non-leading sign so dates don't parse as amounts Found by Cursor Bugbot. `parseCurrencyInput` dropped every `-` as decoration, so an ISO date's hyphens vanished and its digit groups joined: `2024-01-01` read as 20240101. With the gate now sharing the write path's parser, a date → currency conversion reported zero incompatible rows and silently turned every cell into a huge number. A sign is only meaningful at the front; an interior one means the string is not a single amount. Leading signs, accounting parentheses, symbols, ISO codes, grouping separators, and exponent form all still parse — covered by the existing cases plus new ones, verified to fail without the fix. * fix(tables): use getErrorMessage in the columns route test mock `check:utils` bans the inline `e instanceof Error ? e.message : fallback` form; the mock for `rootErrorMessage` used it. * fix(tables): rename the column last so a failed write leaves it untouched Greptile's remaining concern: the pre-flight guards read a schema snapshot, so a column-type change landing concurrently can still make a later write fail — and with the rename running first, that failure returned an error with the rename already committed. Guards cannot close that window; each write is its own locked transaction and only the write itself sees the authoritative state. Ordering can. The rename is the one write that is purely cosmetic, so it now runs last: a failed typed write leaves the column entirely untouched, and a failed rename leaves the typed change applied under the old name — the recoverable half. The typed writes target the column's current name, since no rename has happened yet. Tests cover both directions: a typed write rejected mid-flight must not rename, and a successful one must rename strictly after. Verified to fail under the previous ordering. * fix(tables): write back coerced values on every conversion Round 4 findings, all real. A conversion is allowed exactly when the target type's `coerce` accepts the value — and `coerce` frequently TRANSFORMS it. Only `select` and `currency` wrote the transformed value back, so a conversion to any other transforming type left the cell holding its old bytes under the new type. Converting a number column to `date` accepted epoch values, stored them unchanged, and then `(data->>'col')::timestamptz` failed on EVERY query against that column. I opened this myself by defaulting `isCompatibleWith` to `coerce(...).ok`. Fixed at the class rather than the instance: the compatibility scan now records whatever `coerce` produced whenever it differs from what is stored, and one generic write-back applies it. That subsumes the currency-specific migration entirely, so it and its helpers are gone. `select` keeps its own id↔name migrations, which are not coerce-expressible in the outbound direction. The post-conversion column definition is built once, before the scan, so the coercion reads the same metadata the stored value is later validated against. Exponent parsing was ambiguous when followed by text: `1e5 EUR` read as 15. An `e` with a digit on both sides is an exponent marker, so if the string is not a clean numeric literal it is refused rather than guessed — the digit on both sides is what keeps the `E` inside `12 EUR` parsing normally. A failed rename could still leave a typed change committed. The one rename failure a caller can cause — a name already taken — is now rejected up front, leaving only the concurrent-collision race, which no pre-flight check can close without spanning all writes in one transaction. * fix(tables): stop a blank cell blocking an optional type conversion Found by Cursor Bugbot. `''` is incompatible with every numeric type, and the compatibility scan counted it as a hard blocker regardless of whether the target was optional — so a text column with a single empty cell could not be converted to a number at all, and the error said 'to a required ...' either way. An unreadable-but-empty cell is not a conversion failure. The write path already turns an unreadable value into null on an optional column, so the conversion now does the same and records null for it. A required target still reports it, which the existing guard above already does with the message that actually fits. Also pins the two intentional divergences from the pre-registry behavior. A differential run of the registry against the pre-refactor implementations (55 values x 7 column shapes) found ZERO coercion differences and exactly two compatibility differences, both deliberate: boolean now rejects the '1'/'0' conversions the old gate accepted and then nulled, and date now accepts the epoch numbers its write path always accepted. Tests pin both so neither can be silently reverted or widened. * fix(tables): refuse conversions that would invent or destroy values Final adversarial scan found two data-corrupting conversions, both opened by defaulting the retype gate to the write path's parser. number → date destroyed every value. `date.coerce` reads a number as epoch milliseconds, which is right for one deliberate write and catastrophic applied to a whole column: 1, 5, 42 became three timestamps in January 1970, and a Unix-seconds column landed in 1970 rather than the year it meant. Irreversible. `date` now overrides the gate to reject numbers, restoring the pre-refactor behavior, and the contract states the rule the override obeys: a gate may be STRICTER than `coerce`, never looser. Stricter refuses a bulk conversion while single writes still work; looser is the direction that corrupts. string → currency invented values. The parser stripped every non-digit and joined what was left, so `01/02/2024` read as 1022024, `Room 101` as 101, and `0.1.2` as 12 — a column of SKUs or phone numbers converted with zero reported incompatibilities. What remains after removing symbols, spacing and an ISO code must now be only digits and separators, and grouping must be well-formed (a first group of 1-3 digits, the rest exactly 3). Every legitimate form still parses, including all the locale variants. Also generifies the last three metadata leaks: `buildConvertedColumn` strips and carries back by iterating the key list rather than naming keys (naming them meant a future type's metadata rode onto a target that rejects it, failing that column's validation on every later write), `normalizeColumn` forwards metadata through a shared `typeMetadataOf`, and `filterOperatorsFor` moved onto the definition — it was a per-type branch inside the registry's own accessor, the one thing the registry exists to forbid. Skill corrected: it claimed COLUMN_TYPES derives from the registry (backwards), promised exactly two compile errors (four once a type owns metadata), used a grep that missed half the real branches, and never mentioned `import.ts`'s second coercion path, whose silent default arm is the costliest miss available. Differential re-run vs the pre-refactor implementations: 0 coercion differences, 1 intentional compatibility difference (boolean no longer accepts the 0/1 conversions the old gate accepted and then nulled). * fix(tables): let the row modal accept formatted amounts again Found by Cursor Bugbot. I unified the row modal's input type with the grid's `inputMode` last round, but in the wrong direction: mapping `inputMode: 'decimal'` to `<input type="number">` made the modal reject $1,234.56, 1.234,56 and (12.00) — the exact formats `parseCurrencyInput` exists to accept, and which the grid's inline editor takes fine. A native number input and a numeric keypad are different things. Types whose parser accepts formatted text now say so, and get a text field with `inputMode='decimal'` — the shape the grid already uses. A plain number keeps the native input, its spinner, and its validation. * fix(tables): fold a rename into the write it accompanies Closes the last partial-update window, properly rather than by pre-checking around it. A rename is metadata-only — `renameColumn`'s own comment says so: rows, metadata, and workflow-group refs all key on the stable column id, so it is a pure schema write. Nothing forced it to be its own transaction. Running it separately is what created the window: whichever half committed first survived a failure in the other, and no pre-flight guard can close a concurrent collision because only the write itself sees authoritative state. The four column writes now accept an optional `newName` and apply it through one shared `applyPendingRename`, which validates the name shape and checks the collision against the very schema snapshot that write is landing in. A combined request rides the rename on whichever write runs last, so both halves commit together or neither does — a concurrent claim on the name now aborts the whole transaction instead of leaving the other change applied. The routes also address every write by the column's stable id rather than its name, so folding a rename into one write cannot break the next one's lookup. A rename with nothing to ride on still runs standalone. What remains partial is a type write followed by a failing constraints write — two independently locked transactions, pre-existing, and untouched by this PR. * fix(tables): migrate scalar cells when converting a column to select Found by Cursor Bugbot. `resolveSelectOptionId` stringifies a number or boolean before matching, so a `number` column whose values equal option NAMES passes the compatibility gate — but `migrateCellsToSelectIds` only rewrote JSONB `string` and `array` cells. Those cells stayed raw numbers inside a select column, where they render as nothing and fail option membership on the next write. `data->>key` yields the text form for every scalar, so the existing lookup already worked; the predicate was simply too narrow. Widened to cover `number` and `boolean`. The outbound migration is unchanged — cells leaving a select column are option ids, always strings or arrays. Pre-existing on staging (both the resolver's scalar handling and the migration SQL predate this branch), but it lives in a file this PR creates. Tests pin the resolver behavior the predicate depends on, so narrowing either one without the other now fails. * fix(tables): validate a retype's unique against the values it writes Validated the last partial-update seam with a focused investigation rather than assuming. The answer was split. `required` is already safe: `updateColumnType` runs the same `countEmptyCells` against the constraint the request is about to set, which is why that check exists. `unique` was not, and the reachable case commits the unrecoverable half. A text column holding "5" and "5.0", PATCHed with {type: number, unique: true}: the conversion succeeds and coerces both to 5, then the separate constraint write finds duplicates and 400s — with the column already numeric and "5.0" irreversibly rewritten. A pre-scan of the raw text finds nothing; the conversion is what manufactures the duplicate. The retype now carries `unique` and checks it after the write-back, against the values it just wrote. Constraint changes on a workflow-output column were the same shape — rejected by the constraint write, after a type change had committed. Now rejected in the route's pre-flight block, before any write. The duplicate scan is extracted and shared between both paths for the same reason `countEmptyCells` is: two copies of one rule is the drift that produced the original required-check bug. Deliberately NOT merging `updateColumnType` and `updateColumnConstraints`. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan, so merging would either force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. With both reachable failures pre-validated, what remains at the seam is concurrent races no in-process check can close. * fix(tables): don't drop a rename when the write it rides on no-ops Found by Cursor Bugbot — a bug I introduced folding the rename in. `updateColumnCurrency` returns early when the code is unchanged, and that return sat ahead of the rename, so PATCH {name, currencyCode} with the column's current code answered 200 with the rename silently discarded. Both early returns now treat a pending rename as work: the currency path only no-ops when the code is unchanged AND no rename is riding along, and the retype path applies a rename-only write when the type is unchanged. `applyPendingRename` signals "nothing to do" by returning the same reference, which is what lets both detect it cleanly. Also extracts `persistColumns` — five sites were repeating the same schema-write-and-return. * fix(tables): make a combined column PATCH a single transaction Finishes the fold-in rather than pre-validating around the seam. A retype now APPLIES the constraints it already validates against — it checks empty cells for `required` and post-conversion duplicates for `unique`, so it was doing the work without persisting the result — and the route skips the separate constraint write when the type changed. A request combining a rename, a retype and constraint changes is now one locked transaction: no half of it can commit while another fails. The separate constraint write remains for requests that do not change type, which is the only case that still needs it. Deliberately still NOT merging the two service functions. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan into memory, so a merged function would force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. Folding the payload in gets atomicity without either cost. * fix(tables): reject a flattened list as an amount; fold constraints into every typed write Two findings from round 11. Multi-select converted to nonsense amounts. `selectValueForConversion` flattens a multi cell to its comma-joined option names, and the parser read that as a formatted number: options 12 and 34 became 12.34, and 100 and 200 became 100200. No real amount puts whitespace after a separator, but a delimited list does — so a separator followed by whitespace is now refused. Every legitimate form still parses, including space-grouped locales. Combined options-or-currency + constraints could still commit partially. Those two writes now carry constraints the same way the retype does, through one shared `applyConstraints` that validates (workflow-output, empty cells for required, supportsUnique and duplicates for unique) and applies them. The separate constraint write now runs only when no typed write does. Three copies of those rules is the drift that produced the original required-check bug, so they live in one place. * fix(tables): validate constraints after the migrations that rewrite cells Self-caught while reviewing my own previous commit, which introduced both. `updateColumnOptions` ran the shared `applyConstraints` BEFORE its cell migrations. Those migrations rewrite stored values — a single<->multi toggle changes the shape, removing an option clears cells — so a `unique` scan read the pre-migration values, passed, and the rewrite could then produce the duplicates the scan was meant to prevent. Moved to after the migrations, which is where `updateColumnType` already had it. The same commit also left the options path running `required`'s empty-cell check twice: once in the shared helper and once in its original inline block, whose comment still described a separate constraint write that no longer runs. Removed the duplicate — one query, one rule, which is the whole point of the shared helper. Also routes the options path through `persistColumns` like the others. * fix(tables): stop inventing amounts from identifiers; fix the copilot retype Adversarial pass over the final state, seven real findings. Two destroyed data. The copilot `update_column` still used the two-transaction pattern the HTTP routes were fixed for: `unique` was never forwarded to the typed write, so a retype+unique committed the conversion and then failed the constraint — the same irrecoverable half. It now rides the typed write, and the separate constraint write only runs when no typed write did. And the parser's three-letter strip removed ANY three letters, not an ISO code: `SKU400` parsed as 400, `ABC1234` as 1234. Converting a column of part numbers to currency rewrote every cell with an invented value — while the comment two lines above claimed a SKU was exactly what it prevented. The rule is now that a letter touching a digit means identifier, not amount; a currency marker is always separated by a space or a symbol. That same change fixed a class the review surfaced: the pinned currencies could not parse their own conventional notation. `R$ 1.234,56`, `1 234,56 kr`, `1234,56 zł`, `CHF 1’234.56` and Indian lakh grouping (`₹12,34,567.89`) all work now — these are what Intl emits, so a paste from a spreadsheet was being rejected. `updateColumnConstraints` was a fourth copy of the constraint rules the shared helper exists to unify, and had already drifted: it hardcoded `type === 'select'` where the helper asks the registry, so a future type declaring `supportsUnique: false` would have been ignored on that path. It now uses the helper. `updateColumnType`'s unchanged-type early return silently discarded every field except the rename. Callers gate on the type changing, but from a read taken before the lock — so a concurrent change could land there with real work pending and answer success. It now throws. Also: `UpdateColumnCurrencyData` was missing `required`, which only compiled because the routes pass it through a spread; a missing column returns 404 instead of a 400 reading "of type undefined"; and the comments describing the old two-transaction architecture are gone. Verified NOT a bug: CSV export of a currency column writes the raw number, so export/import round-trips losslessly. * fix(tables): read the negative and RTL forms Intl actually emits An Intl sweep across 24 locales found two forms the parser rejected, both from an ordinary spreadsheet paste. `Intl` emits U+2212 MINUS SIGN rather than the ASCII hyphen for negatives in several locales, so `−12,50 kr` read as null instead of -12.5. And it wraps RTL-locale output in invisible bidi control marks, so `‏1,234.56 ‏₪` carried characters that are not part of the amount. Both are now normalized away. 24 locales x 6 amounts now round-trip, up from 99/100 when the sweep started — and the test generates them from `Intl` rather than listing them by hand, so a parser change cannot quietly regress a locale nobody remembered to write down. Locales that format with their own numeral systems (Arabic-Indic) are still rejected, and now say so in the docstring. That is a safe failure — null rather than a wrong value — and supporting them is a wider decision than this type, since it would also touch `number`, display, and sorting.	2026-07-30 17:21:17 -07:00
Vikhyath MondretiandClaude	7798e83489	feat(function): custom sandboxes (#6071) ... * feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 16:33:35 -07:00
Waleed	811a39ec05	fix(integrations): show family service accounts on every product they authenticate (#6102) ... * fix(integrations): show family service accounts on every product they authenticate An Atlassian API token authenticates Jira, Jira Service Management, and Confluence alike, so it is modeled as an `atlassian` pseudo-provider whose only service is named "Atlassian Service Account". Every credential display surface resolved through `getServiceConfigByProviderId`, which walks OAUTH_PROVIDERS in declaration order — so the credential resolved to that pseudo-service instead of to any product. The result: adding a service account from the Jira page, through a modal titled "Add Jira service account", produced a credential that appeared under neither Jira, JSM, nor Confluence, was titled "Atlassian Service Account" on its detail page, and lost its brand tile and category on the list. The same bug hid a Google service account everywhere except Gmail. - match credentials with `credentialProviderMatchesService`, which accepts a service's OAuth id or its service-account id - add `lib/integrations/credential-display.ts` as the single resolver for catalog join, mark, and copy, replacing three duplicated lookups that keyed the catalog by OAuth service *display name* — the reason the pseudo-service fell off the map - derive "family service account" from the catalog (a service-account id serving >1 integration) rather than hardcoding vendors, so a new integration joining a family needs no edit - title service-account detail pages by credential name, subtitle them with their reach, and state that reach up front on the connect form - keep the service description as the detail subtitle for every non-family credential, unchanged No schema, migration, contract, or persisted value changes; resolution is computed at render time from static config. Coverage for all 22 service-account provider ids is pinned in tests, including that the index and the predicate the Connected list filters on cannot drift apart. * chore(icons): use Atlassian's gradient marks for Jira and Confluence Replaces the flat #1868DB Jira and Confluence marks with Atlassian's gradient versions, matching the Atlassian mark added alongside them. - gradient ids go through `useId()` rather than the source SVGs' static ids, which would collide wherever two of these icons render on one page — the integrations list and the landing loops both do - pads the Atlassian viewBox so its artwork fills ~78% of the box, matching the inset Atlassian ships on the Jira and Confluence marks; without it the mark renders ~30% heavier than its siblings in the same tile Visual-only, but these marks render in ~60 files, so it is split from the credential fix to stay independently revertable. * fix(integrations): route the editor's service-account setup modal through the shared target The workflow editor's credential selector passed the OAuth service's own name and icon straight to ConnectServiceAccountModal, so opening the setup form from a Jira block titled it "Add Jira service account" while the integrations page and the chat — both of which already resolve through `useServiceAccountConnectTarget` — titled the same form "Add Atlassian service account". That is the exact confusion this branch set out to remove, surviving on the one surface that bypassed the shared resolver. * docs(atlassian): correct the service-account setup path and cover all three products The setup section could not be followed. It sent readers to a "Settings → Integrations tab" that does not exist (Integrations is a top-level workspace module) and told them to search the integrations list for "Atlassian Service Account", which matches no catalog entry — the catalog lists Jira, Jira Service Management, and Confluence. The page also described the credential as covering "Jira and Confluence" while listing Jira Service Management scopes, and the product now spells the coverage out in the connect form. - correct the path: Integrations -> Jira/JSM/Confluence -> Add to Sim -> Add service account - name all three products consistently, and state that one service account covers them - match the real button label ("Add service account")	2026-07-30 13:40:40 -07:00
Theodore Li	32293f4b55	feat(tables): typed predicate filter grammar, cursor pagination, and the v2 table surface (#6067)	2026-07-30 00:24:06 -07:00
Vikhyath MondretiandClaude	11c0d3b75d	feat(organizations): sweep a joiner's owned workspaces into the org on join, disclose it at accept, and add external workspace invites (#5918) ... * feat(invites): explicit external members * update docs * fix(organizations): atomic admin workspace sweep, removal-impact status in dialog, and preview-unavailable disclosure Review round 1: the v1 admin add-member now commits membership and the workspace sweep in one transaction; the remove-member dialog holds confirm while the credential-impact check loads and shows a caution when it fails; a failed join preview flags joinPreviewUnavailable so the accept screen falls back to a generic migration notice. Also aligns the invite test's react-query mock and repairs two pre-existing docs type errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): close the concurrent-workspace escape in the join sweep Personal workspace creation now serializes with organization joins on the user's billing-identity lock and re-verifies membership inside its transaction; both join paths (invite acceptance and the v1 admin add) re-read the owned-workspace set under that lock after the member insert and roll the whole join back when it diverged from the advisory-lock plan, so a workspace created mid-join can never land outside the organization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): fail stale-grant member joins and re-resolve the creation race client-side A member-role org acceptance whose grants all turned stale now rolls back with workspace-not-found instead of stranding a workspace-less member, and the workspace resolver treats the creation-vs-join 409 as a signal to re-resolve (the user is authenticated with org workspaces) rather than falling into the login path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): mirror the stale-grant gate in the join preview The accept-screen preview now returns no-join for a member-role org invite whose grants all left the stamped organization, matching the acceptance-side rollback so the disclosure never promises a migration that acceptance would refuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(workspaces): survive the join race on lazy default creation and gate removal on live impact data The workspace list GET now re-lists (returning the join sweep's workspaces) when lazy default creation loses the race to an organization join instead of failing with a 500, and the remove-member dialog gates its confirm on isFetching so a background refetch can never let an admin confirm against a stale credential-impact list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): surface the accept conflict message and refresh workspace caches post-accept The accept route now carries the human-readable message alongside the machine-readable error kind (the client prefers it for server-error), and a successful accept invalidates workspace queries so the swept workspaces appear immediately instead of after the stale window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): sweep archived workspaces in Pro-to-Team conversion and org creation Every attach call site now passes includeArchived so the archived escape hatch is closed uniformly — join-attach, admin move, subscription-driven org provisioning, and manual org creation all sweep archived personal workspaces into the organization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): reject acceptance when the sweep set differs from the disclosed set The join preview now carries the workspace ids it disclosed, the accept screen echoes them back as a disclosure token, and acceptance rolls back with disclosure-outdated (409) whenever the set it would sweep no longer matches — a workspace created after the preview rendered can never move without the user seeing the refreshed notice first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): send the disclosure token for no-join previews too A preview that predicted no join still tells the user nothing moves — the empty disclosed set is now echoed on accept, so a join that becomes possible between preview and accept (left another org, billing turned usable, grants un-staled) conflicts with disclosure-outdated instead of sweeping workspaces without a rendered notice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): gate all-stale member joins before disclosure and always refetch removal impact The all-stale check for member-role org invites now runs before any mutation and before the disclosure comparison, so an invite whose grants all left the org fails with workspace-not-found instead of trapping owners of personal workspaces in a disclosure-outdated retry loop. The removal-impact query drops its stale window (staleTime 0): every dialog open refetches while the confirm is held on isFetching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): keep-external on org recovery and label archived move candidates Org creation/recovery now uses the keep-external collaborator policy (matching Pro-to-Team conversion) so different-org collaborators on archived workspaces cannot abort it with a conflict, and the admin workspace-move search and preflight expose an archived flag so internal tooling can label archived targets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): guard the reverse disclosure direction A will-join notice whose acceptance downgrades to no-join (stale escalation denial, concurrent other-org membership) now fails with disclosure-outdated instead of silently succeeding as an external grant — the disclosure token binds the outcome in both directions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address comments * fix * fix(invitations): one invite surface, coalesced grants, coherent seat model Consolidate the two invite modals into a single surface and fix the semantics the split had been hiding. Invite flow: - One InviteModal for all three entry points (workspace header, workspace settings, organization settings), with a workspace multi-select and an explicit Membership choice that states the seat consequence. - Coalesce grants instead of 500ing. A partial unique index allows one pending invitation per (email, organization), so inviting someone to a second workspace raised a raw 23505. New workspaces now merge into the pending invitation (with a retry for the concurrent-insert race) and the invitee gets one link covering everything. - External collaborators require their own paid plan, checked at invite time and re-checked on accept, since the invitation lives for 7 days. Imposed externality is exempt in both places: an invitee already in another organization is forced external regardless of the inviter's choice, so the plan gate must not apply to them. - Every invitation must grant at least one workspace, so accepting always lands somewhere. Enforced at creation for all roles. - Revocation is grant-scoped. Since one invitation can span workspaces, revoking from a workspace's member list withdraws only that grant and cancels the invitation just when the last one goes. Whole-invitation revocation now requires authority over all of it rather than admin on any single granted workspace. - The accept screen names every granted workspace and states whether the invitee joins as a member, an admin, or an external collaborator, and whether that uses a seat. Seats: - One rule for the pending-invitation predicate, seat capacity, and the derived figures; the three counting sites now share it. - Team seats are elastic (subscription.seats tracks the member count), so treating that as a cap reported negative headroom on any outstanding invite. Available seats are clamped and gates branch on whether the plan actually has a fixed cap. - POST /api/v1/admin/organizations/[id]/members could never succeed on Team: it validated N members against N seats. It now skips the cap for elastic plans, matching invitation acceptance, and reconciles seats after a committed add. Also surfaces the External label on the workspace Teammates list, which already received the flag and dropped it, and removes dead code: the grantless organization-invite route and contract, three unreferenced invitation helpers, and the unreachable ensureUserInOrganization/addUserToOrganization/validateMembershipAddition cluster. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): close the two accept-disclosure gaps Bugbot found Membership notice ignored the join preview. `buildMembershipNotice` keyed off the invitation's sent `membershipIntent`, but acceptance resolves an internal invite to external when the invitee already belongs to another organization, or when the granted workspace changed organizations after the invite went out. The screen therefore promised "you'll join as a member, which uses one of their seats" to people who would neither join nor consume a seat. It now keys on the preview's `willJoinOrganization` — the same signal the migration notice already used — and falls back to the sent intent only when no preview could be computed. In-app accept skipped the disclosure entirely. `useAcceptMyInvitation` posted an empty body, so `disclosedWorkspaceIds` was absent and the server's consent guard was skipped, and the pending-invitations modal never showed which owned workspaces would move. Accepting from the workspace switcher (including the desktop path) could silently sweep personal workspaces into the organization, bypassing the consent model this PR adds on /invite. The list endpoint now returns each invitation's join preview, the modal renders the same membership/migration disclosure as /invite, and accept echoes `disclosedWorkspaceIds` so the guard applies on both paths. Both notices moved into lib/invitations/disclosure-copy.ts and are consumed by /invite and the modal, so the two accept surfaces cannot drift into disclosing different outcomes for the same invitation — which is how this gap arose. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): carry the membership outcome in the disclosure token Empty disclosure skipped membership consent. The token was only the workspace-id list, so a no-join preview and a will-join preview for someone who owns nothing both echoed `[]`. Neither guard could tell those apart: the forward check compares sweep sets, and the reverse check required a non-empty disclosed set. An invitee who left their other organization between preview and accept would therefore be silently made a seat-consuming member after being told they would stay external, and the mirror case could silently demote a promised join. The accept body now also carries `disclosedWillJoinOrganization`, compared against the resolved outcome before any write, so consent covers the membership decision and not just the migration. Both accept surfaces send it. This was widened by the previous commit: keying the membership notice on the preview made the screen promise a join outcome the token never verified. In-app accept errors lacked copy. `getInvitationErrorMessage` omitted `external-requires-paid-plan`, `disclosure-outdated`, and `workspace-not-found`, so those failures fell through to the generic "may have expired" fallback. `disclosure-outdated` became newly reachable in-app the moment that path started sending the token, so the gap arrived with the fix for it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): compare the join disclosure against new-membership creation The membership consent guard compared the disclosed outcome against `shouldJoinOrganization`, which stays true for an invitee who already belongs to the target organization — the invitation's intent is still internal. The join preview reports no-join for exactly that case, because nothing changes for them. Every such acceptance therefore failed `disclosure-outdated`, and the retry re-rendered the same preview, so the invitation became permanently unacceptable. The guard now compares against whether acceptance creates a NEW membership (`shouldJoinOrganization && !alreadyMemberOfTargetOrganization`), which is what the disclosure actually promises and what the preview reports. The already-a-member predicate is hoisted and shared with the join block below so the guard and the billing path cannot disagree about it. Regression test asserts a pre-existing member accepts with a no-join disclosure; it fails with `disclosure-outdated` against the previous comparison. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): tell an existing member their standing is unchanged The join preview reported the same no-join shape for two different outcomes: an external collaborator, and an invitee who already belongs to the organization acceptance lands in. `buildMembershipNotice` rendered both as "you'll join as an external collaborator ... everything you own stays yours", which is wrong for an existing member — they stay an internal member and simply gain the granted workspaces. The preview now reports `alreadyMemberOfOrganization` for that case (a membership in a DIFFERENCE organization is still the external path, since acceptance downgrades), and the notice states that standing is unchanged. This is the same conflation behind the accept loop fixed in 4eb7725f1a, now removed from the shape itself rather than worked around per consumer. Also re-verify the removal-impact disclosure at the moment of confirmation. `isFetching` only holds the confirm button while a request is in flight, so an identity-bound credential the member gained after the fetch settled would break on removal without ever being disclosed. Confirm now refetches and, if the set changed, keeps the dialog open on the refreshed warning instead of proceeding. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): serialize the join-preview reads, revert the removal refetch Two corrections to this branch's own review fixes. The pending-invitations list computed each row's join preview with Promise.all. Every preview issues several queries, and the endpoint is hit whenever the workspace switcher opens, so that held one pooled connection per pending invitation for as long as the slowest one took. The loop is sequential now; the list is a handful of rows, so the latency is not worth the pool pressure. The removal-impact refetch on confirm is reverted. It changed behaviour — a click could silently do nothing — and it did not actually close the window it targeted: the credential set can still change between the refetch and the independent DELETE, because the removal endpoint neither receives nor revalidates the disclosed set. Closing that properly means passing the disclosure to the endpoint and revalidating there, which is a feature rather than a review fix, so the prior behaviour stands until it is done deliberately. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): disclose the seat on a personal-workspace join Both accept surfaces scoped the membership notice on `organizationId` or the preview's `organizationName`. A personal-workspace invite has neither until acceptance runs — it creates the organization by converting the billed owner's Pro to Team — so the seat and membership disclosure was suppressed for exactly the case that creates the membership. They now also scope on the preview's `willJoinOrganization`, which is the authoritative signal. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): make the join preview a discriminated outcome The preview returned one no-join shape for five different results — external intent, already a member, a membership in another organization, a dead-grant rejection, and a billing rejection. Two accumulating booleans could not separate them, and the accept screen rendered the external copy for all of them: it told people whose acceptance would fail with `upgrade-required` or `workspace-not-found` that they were getting workspace access without a seat. It now reports one `outcome`: `will-join` (a seat is taken), `already-member` (only workspace access changes), `external` (never a seat), or `blocked` (acceptance fails, so nothing is promised). `blocked` renders no membership notice — silence is accurate where the external claim was false. The accept button is deliberately left enabled: those cases already fail closed with the correct error, and choosing what to actively tell someone whose organization's payment lapsed is a product decision, not a review fix. This also fixes a live mis-attribution the previous commit's guard introduced. The consent check ran before the gates that produce the real cause, so a blocked invitation returned `disclosure-outdated` — and the retry re-rendered the same preview, leaving the invitee looping with no explanation. The guard now sits after the dead-grant gate, and a disclosed `blocked` skips the comparison so the billing gate below can surface `upgrade-required` instead. The accept body carries `disclosedOutcome` in place of the boolean; the membership comparison is unchanged (`will-join` versus a new membership being created), so no acceptance that previously succeeded now fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): let billing-disabled personal invites be accepted The consent guard derived "a membership will be created" from `shouldJoinOrganization`, which is still true at that point — it is only cleared much later, after provisioning fails to yield a target organization. With billing disabled and no organization on the workspace there is nothing to provision and nothing to join, so the preview correctly reports `external` while the guard computed `will-join`, rejecting every personal and grandfathered workspace invite as `disclosure-outdated`. The retry rendered the same preview, so those invites could not be accepted at all on billing-disabled deployments. The predicate now mirrors the preview's own condition, so the two cannot drift. Regression test asserts acceptance succeeds with billing off; it fails with `disclosure-outdated` against the previous predicate. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): allow External with billing off, mark cross-org org invites blocked The External paid-plan requirement is seat economics — an external collaborator takes no seat, so somebody else must be paying for them. With billing disabled there are no seats and no subscription rows at all, so every account resolves as `free` and choosing External failed at send time and would have failed at accept. Member and Admin still worked, so a self-hosted deployment had no way to grant workspace-only access without an organization join and a workspace sweep. Both the invite-time and accept-time gates now short-circuit when billing is off. A route test asserted that rejection without setting `isBillingEnabled`, which the shared mock defaults to false — it passed only because the gate ignored the flag. It now opts in explicitly, since the rule it covers is billing-only. Separately, the preview reported `external` for any invitee already in a different organization, but acceptance only downgrades a workspace-kind invite with live grants; an organization-kind invite hard-fails with `already-in-organization`. Those now report `blocked`, so the screen stops promising external access that acceptance can never grant. Legacy organization-kind rows still exist and coalescing preserves that kind, so this is reachable. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): require org admin to grant org Admin, and two disclosure gaps Privilege escalation. `createWorkspaceInvitation` stamped organization role `admin` whenever the caller passed `membership: 'admin'`, but authorization only checked workspace admin access — and unifying the invite modals exposed the Admin option to any workspace admin, where it had previously been reachable only from organization settings. A workspace-scoped administrator could therefore invite someone who joins as an organization Admin, gaining admin on every workspace the organization owns plus member and billing management. The inviter must now already hold organization owner/admin, checked server-side because the batch endpoint is reachable without the modal, and the modal no longer offers Admin to anyone else. The preview promised external access without mirroring acceptance's `external-requires-paid-plan` gate, so a free invitee — one who cancelled Pro, or left the organization that forced the external invite — was told they had workspace access and then refused. It now mirrors that gate, including its exemptions (billing on, organization-owned workspace, externality not imposed), and reports `blocked`. The modal's Enterprise seat check counted every non-External email as a seat. The server does not: an existing organization member is granted access directly, and an invitee already in another organization is forced external. The hard block refused batches the API would have accepted, so it is advisory now — per-email failures already come back with reasons. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): restore the invite admin gate, hedge an unknown outcome Consolidating the modals dropped a permission check. The old workspace-header modal derived `canInviteMembers` from `userPermissions.canAdmin` internally and disabled its field and button; the shared modal takes `canInvite` as a prop that defaults to true, and no call site passed it. Non-admins therefore saw a fully enabled invite form and only learned otherwise when the server refused the send. All three entry points now supply it: workspace admin for the header, the existing `canManage` for the workspace Teammates page, and organization owner/admin for organization settings. When the join preview cannot be computed the outcome is unknown, and the callers send no disclosure token — so acceptance runs without the consent guards. The membership notice nevertheless asserted a seat-taking join from the sent intent, which acceptance may resolve to external, already-a-member, or a failure. It is conditional now ("If you're added to X as a member, that uses one of their seats"), so the consequence is still disclosed without being claimed as settled. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-29 20:27:55 -07:00
3 people	94778eb68d	feat(pi): add update PR mode (#6031) ... * feat(pi): add update branch mode * feat(pi): manage pull requests in update mode * docs(pi): clarify closed PR behavior * fix(pi): handle update PR finalization races * fix(pi): accept renamed update PR repositories * fix(pi): clarify shared open PR errors * fix(pi): recheck update PR ambiguity * fix(pi): recreate closed update PRs * fix(pi): follow replacement update PRs * fix(pi): preserve update PR BYOK after staging rebase --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain>	2026-07-29 18:59:03 -07:00
Waleed	911b958dbd	feat(exa): refresh Exa integration against current API, retire dead research endpoint (#6074) ... * feat(exa): refresh Exa integration against current API, retire dead research endpoint Exa's dev-rel team flagged that our integration was written against a retired version of their API. Validated every claim against the live API with a real key. - /research/v1 returns HTTP 410 RESEARCH_RETIRED, so the Research operation was hard-broken in production. Removed it and added an Agent operation on /agent/runs. Saved workflows on the old operation are routed to Agent so they start working again. - Category dropdown sent values Exa no longer recognizes (research_paper, news_article, movie, song, ...). Exa accepts category as an unvalidated soft hint, so these silently stopped steering results rather than erroring. Replaced with the current taxonomy and remapped legacy values. - Live crawl mode defaulted to 'never', silently forcing cache-only results on every search. Removed the default and exposed maxAgeHours, which replaces the deprecated livecrawl. Exa 400s when both are sent, so they are now mutually exclusive. - numResults was capped at 25 in the UI; the API allows 1-100. - Search types refreshed to instant/fast/auto/deep-lite/deep/ deep-reasoning. Legacy neural/keyword still pass through. - Exposed result id so search results can be chained into Get Contents via ids, plus highlightScores, subpages, entities, extras, statuses, requestId, and outputSchema structured output with grounding. - answer text controls cited-source text, not the answer; fixed the description and the dead query field. - Marked findSimilar and the crawl-date filters deprecated. Both still work, so existing workflows are unaffected. - Copilot search-online never requested page content, so every snippet was empty. Now requests highlights. * fix(exa): address review findings on the API refresh - A run already terminal on creation went through a path that never set success=false, so a failed or cancelled run reported as successful. Both the create path and the poll loop now settle through one function. - Routing exa_research to Agent dropped the research output shape, so saved workflows referencing research[0].text resolved to undefined. The agent tool now also emits that legacy shape. - The Agent operation's inputs are conditioned on both exa_agent and exa_research so the serializer keeps carrying a stored research query; it drops any value whose sub-block condition no longer matches. - Dropped the model to effort mapping and the unused ExaResearchParams: the serializer drops values for removed sub-blocks, so model never reached the params function. * fix(exa): keep legacy research model, add subblock migrations, sharpen outputs The subblock ID stability check caught the removed subblocks — that gate exists precisely to stop removals from breaking deployed workflows. - Restore the research model sub-block, scoped to the legacy exa_research operation so it never shows for new workflows but still serializes for saved ones, and restore the model to effort mapping. Removing it lost the configured research depth, silently falling back to effort auto. - Register useAutoprompt and livecrawl in SUBBLOCK_ID_MIGRATIONS as intentional removals. Neither has a value-compatible replacement: livecrawl is a mode string and maxAgeHours a number, so mapping one to the other would send NaN. - Replace vague json output descriptions with their inner field lists. - Drop the separate compat test file; the coverage that guards real regressions now lives in exa.test.ts. * fix(exa): flag empty Get Contents configs in the editor, keep legacy research depth - A saved research workflow with no stored model fell through to the Agent default of auto rather than the standard depth the old Research operation used. Legacy research now always maps to an effort level, defaulting to medium, and the legacy model sub-block carries the same default the old dropdown had. - Get Contents needed both selectors optional so the ids path is reachable, which left an empty config failing only at run time. URLs is now conditionally required, dropping the requirement when result IDs are supplied, so the editor flags the empty case. The exactly-one check in the request body stays as the backstop. * fix(exa): revert conditional required on Get Contents URLs The conditional required callback did not work and introduced a regression. `isFieldRequired` in webhook deploy calls `config.required()` with no arguments, so the callback never saw `ids` and left URLs required — an ids-only block would have been reported as missing a required field on deploy. `collectBlockFieldIssues` skips sub-block required checks whose id matches a tool param, so it never evaluated the callback either. Both selectors go back to optional with the exactly-one check in the request body, which is what the integration rules prescribe for mutually exclusive alternate identifiers. Added a comment recording why a conditional required cannot express this, so it is not reattempted.	2026-07-29 18:18:05 -07:00
Waleed	f0b79c5cc6	chore(deps): upgrade next 16.2.11 -> 16.2.12 (#6077) ... 16.2.12 is the current stable (published 2026-07-25) and its entire changelog is two PRs: a docs backport and vercel/next.js#95831, "Fixes to support TypeScript 7". That second one matters here. `apps/sim` declares `typescript: ^7.0.2` and the lockfile resolves 7.0.2, while 16.2.11 predates any TS7 handling — not even the actionable-error guard (#95837), which was never merged. The upstream symptom is `next build` dying with a silent SIGSEGV during its type-check step, because the legacy TypeScript JS API that Next called is gone in TS7. Builds pass today only because Next detects @typescript/native-preview as the compiler and takes a different path, so we are accidentally-working rather than supported. 16.2.12 adds the `experimental.useTypeScriptCli` backend that makes this configuration official. Zero build-performance content in the patch, so this is not a speed change. Bumps all eight pins in lockstep — next, @next/env and the four @next/swc-* binaries at the root, plus the three app/package copies. The swc binaries must move with next: they are platform-gated optionalDependencies, so a version skew or a gate exclusion leaves them out of bun.lock entirely and `bun install --frozen-lockfile` installs no compiler at all (the #5945 failure). Also re-dates the bunfig.toml gate note, which said to drop the next entries on 2026-07-28 — yesterday. 16.2.12 is inside the 7-day window until 2026-08-01, so following that instruction would have blocked this bump and re-triggered the missing-compiler failure. Re-date on future bumps rather than deleting the entries early.	2026-07-29 18:10:50 -07:00
Vikhyath MondretiandClaude	f78367c4e8	feat(logfire): add Pydantic Logfire block, tools, and docs (#6075) ... Four read-token tools over Logfire's query API: structured span/log search, raw SQL against records/metrics, full-trace fetch by ID, and read-token introspection. Plus the block, icon, registry wiring, and generated docs. Requests go to /v2/query with the region resolved from the token's pylf_v{n}_{region}_ prefix, overridable by an explicit region or a self-hosted host (public HTTPS only, per the tool executor's URL policy). Structured filters are emitted as escaped SQL literals, using DataFusion contains() so %/_ in user input stay literal. Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 18:07:39 -07:00
6 people	25d8019a54	feat(pi): Babysit foundations — shared PR/push extraction, five GitHub tools, sandbox lifetime (#5962) ... * feat(pi): optional multi-provider web search for the coding agent Adds a search provider dropdown (Exa, Serper, Parallel, Firecrawl) to the Pi block, off by default. The selected provider's key comes from the block field or Workspace Settings → BYOK; a Sim-hosted key is never spent, so a missing key fails the run with a setup message instead of quietly billing Sim. Search is available in all three modes. Local Dev and Review Code register a host-side tool that goes through the existing provider tools, while Create PR has no host in the loop and gets a generated Pi extension in the sandbox. Both paths derive their requests from one normalizer and are held together by a parity test, since the sandbox copy cannot import Sim's code. Results are normalized to title, URL, snippet, and publication date, capped per field and per envelope, marked untrusted in the prompt, and limited to 20 searches per run so a tool loop cannot drain the workspace's quota. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pi): drop the banned JSON round-trip from the search parity test `check:utils` bans `JSON.parse(JSON.stringify(...))`. The round-trip was normalizing the host body to its wire form, which buys nothing here: the bodies are plain JSON and `toEqual` already ignores undefined members. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(agents): add reviewed-development skill and the babysit implementation plan Carries the plan and review protocol into the repository so a cloud agent working from the remote can read them. Temporary: the plan is removed before this branch goes for review. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(pi): babysit foundations — shared PR/push extraction, GitHub tools, sandbox lifetime Stage 1 of the Babysit plan (.agents/plans/pi-babysit-mode.plan.md), sections 0, 3, and 7 plus their tests. The Babysit mode itself lands in stage 2. Section 0 — shared extraction and push hardening: - Move PREPARE_SCRIPT, PUSH_SCRIPT, and the finalize path/size constants from cloud-backend.ts into cloud-shared.ts. - Move Review Code's PR snapshot helpers into pi/github-pr.ts, generalized off PiCloudReviewRunParams to a plain PullRequestCoordinates, and split the raw fetch from the "must be open" wrapper so a mode that has to report a closed PR gracefully can build on the raw form. - Harden the one token-bearing command: GIT_CONFIG_NOSYSTEM/GIT_CONFIG_GLOBAL on its env, git by absolute path, and an explicit HEAD:refs/heads/$BRANCH refspec. The clone script now emits a .git/config digest marker as its last line for every mode; only Babysit will verify it. Section 3 — five GitHub tools, registered but not wired into the block dropdown: github_list_review_threads, github_reply_review_thread, github_resolve_review_thread, github_status_check_rollup (GraphQL) and github_job_logs (REST). Plus a nullable repo_full_name on the PR reader's branch parse, declared on its own output rather than the shared BRANCH_REF_OUTPUT. Section 7 — CreateSandboxOptions.lifetimeMs threaded to E2B's timeoutMs, clamped below the one-hour Hobby ceiling and lowerable by PI_SANDBOX_LIFETIME_MS. Daytona is deliberately untouched. The per-command Pi timeout is capped at the lifetime. * fix(tools): correct the rollup CheckRun selection and stop leaking the token on redirect Two defects found in review, both verified against the live GitHub API. GraphQL's CheckRun has no `output` object — that is REST's shape. It exposes `title`, `summary`, and `text` as flat nullable fields, confirmed by introspecting the schema. The old selection made every github_status_check_rollup call fail with "Field 'output' doesn't exist on type 'CheckRun'", delivered as an HTTP 200 errors payload that no fixture-based test could catch. The corrected query was run against a real PR and returns 18 check runs plus a Vercel status context, with title and summary null on every Actions run exactly as the plan predicted. github_job_logs redirects to third-party blob storage, and Sim's tool fetch follows redirects itself rather than through the fetch spec, so it replayed the GitHub token to that host. Tools can now declare `stripAuthOnRedirect`, and the log reader does. Also from review: pin Review Code's "must be open" guard with a test now that it lives in its own function, drop the stream reader in github_job_logs since the executor already hands transformResponse a capped buffer, and correct three doc comments that overstated what they guaranteed. * test(tools): cover the stripAuthOnRedirect plumbing end to end Asserting the flag on the tool config alone would not catch a regression in formatRequestParams or in the executor's call into secureFetchWithPinnedIP, so pin what the fetch layer actually receives, in both the opted-in and the default case. * fix(pi): correct the push-hardening claim, reserve finalize time, tighten isRequired Three review findings, each verified rather than taken on faith. PUSH_SCRIPT's comment claimed GIT_CONFIG_NOSYSTEM and GIT_CONFIG_GLOBAL close config-driven URL rewriting. They do not: reproduced locally on git 2.43, a repository-local url.*.insteadOf still rewrites the push URL and sends the token's userinfo to another host. That is the scope a root agent in the checkout can actually write, and it stays open until a mode verifies the config digest — which is Babysit, per the plan. The comment now says that instead of the opposite. PI_TIMEOUT_MS capped the Pi command at the whole sandbox lifetime, so the sandbox always died first and the stated benefit — a clean timeout instead of an opaque SDK error — could never happen. It now reserves the clone and finalize budgets it shares the sandbox with, leaving the host time to commit and push whatever the agent produced. isRequired is Boolean! on both CheckRun and StatusContext (confirmed by schema introspection), so the nullable parse modelled a value GitHub cannot send and left stage 2 a tri-state to handle. It is required now, and an absent value fails loudly rather than reading as "not required", which would let a failing required check stop blocking the green verdict. Also: a github-pr.test.ts pinning that the raw fetchPrSnapshot does not throw on a closed PR (the entire reason for the wrapper split, previously untested), a cloud-shared.test.ts for the timeout reserve and the digest line, the E2B lifetime ceiling documented next to E2B_PI_TEMPLATE_ID as section 7 asks, and the new registry tests no longer leaving their fake tools registered. * fix(pi): reserve both finalize budgets in the Pi command timeout Create PR dispatches two commands at FINALIZE_TIMEOUT_MS, not one — the commit and the push — so reserving a single budget left the push unbudgeted and the worst case overshot the sandbox lifetime by exactly that amount. Losing the sandbox during the push is the most expensive moment to lose it: the work is committed and unpushed, which is the outcome the reserve exists to prevent. The comment no longer claims more than the arithmetic delivers. What is reserved is each command's timeout ceiling rather than its measured elapsed time, so this is a budget that adds up, not a guarantee. Two other comments described Babysit verifying the config digest in the present tense, when no mode verifies it yet. Also drops the digest-line test: it asserted a string constant contains its own substrings, while cloud-backend.test.ts already pins the property that matters — the marker being the clone's last line, after the remote rewrite. * docs(pi): stop describing Babysit's digest check in the present tense Three comments still read as statements of current behavior: the Create PR push test's note beside the assertion that proves no verification happens, github-pr's module doc naming a second consumer that does not exist yet, and the timeout floor claiming a short-lifetime run was doomed regardless when the reserved ceilings are pessimistic enough that it may well finish. * fix(pi): scope the sandbox lifetime cap to E2B and harden the job-log path Deriving PI_TIMEOUT_MS from the E2B lifetime applied it to every provider, so a Daytona Create PR run lost its ~90-minute agent turn to a ceiling Daytona does not have — it stops on inactivity instead. The reserve now only applies when the provider imposes an absolute lifetime. A configured PI_SANDBOX_LIFETIME_MS below the clone and finalize reserves left no positive remainder for the turn, so E2B could reap the sandbox before the push. Such a value is raised to a floor rather than rejected: a module-scope throw on a config typo would take down every path that imports this, not just Pi. github_job_logs returns its response body verbatim, so unlike its siblings that parse a typed shape, a coordinate carrying URL syntax turned a bearer-authenticated request into a general read. Path segments are now escaped and the job id checked. Also corrects the plan's rollup field path: GraphQL's CheckRun has no output object, and isRequired is Boolean!, so stage 2 needs neither the nested path nor an unknown-required branch. Co-authored-by: Cursor <cursoragent@cursor.com> * Add Pi Babysit mode * fix(pi): wait on required checks before optional failures * fix(pi): preserve Daytona babysit budget * fix(pi): bound babysit round setup * fix(pi): keep babysit sandboxes active * fix(pi): report babysit round state accurately * fix(pi): classify babysit finalize failures * fix(pi): preserve babysit partial state * fix(pi): retain post-push check state * fix(pi): retain pending rereview state * fix(pi): normalize empty sandbox provider * Move Babysit into Create PR * Fix Babysit wait-only budgeting * Wait for reviews before skipped-thread exit * Polish Babysit reviewer field spacing * Fix duplicated Internet Search section from staging merge The staging merge landed the branch's Internet Search section and staging's reviewed replacement side by side, leaving two `### Internet Search` headings and two `#internet-search` anchors. The branch's copy also still claimed a Settings > BYOK fallback for the search key, which staging deliberately removed. Keep staging's section and fold the branch's only new fact — that the Babysit continuation sandbox carries both keys — into its warning callout. * fix(pi): correct Babysit check, budget, and push-guard accuracy Correctness: - The `.github/` push refusal compared raw `git diff --name-only` output, which git C-quotes for non-ASCII paths. `.github/workflows/évil.yml` arrived as `".github/workflows/\303\251vil.yml"` — leading quote included — so neither `.github` nor `.github/` matched and the file pushed. Both name-listing diffs now pin `core.quotePath=false`; Create PR's does too so `changedFiles` reports real names. - `CANCELLED` and `STALE` were counted as non-failing conclusions, so a cancelled required check produced `checksGreen: true` and `stopReason: 'clean'` on a PR branch protection still blocks. Both now fall through to failing, matching every other unknown conclusion. - The per-round check bound counted optional checks, so a repo with a wide optional matrix ended the run at `bounds_exceeded` before a single review thread was addressed. Only required-check overflow is fatal now; the rest trims, required checks first, and reports what was left out. This also makes the prompt's existing slice reachable. - A re-review request that posted nothing still re-armed `requestedAt` with `landed: false`, leaving the loop waiting on a review nobody had asked for — burning the remaining lifetime on a billed idle sandbox before reporting `awaiting_review` on a clean PR. The previous request now stands. - Prompt bounds threw where every other bound in the file trims, ending a busy PR's run on round one after the PR and its review comments were already posted. They now drop trailing entries and note the omission. - `babysitMode` used a strict boolean compare; a `switch` input arriving as the string 'true' silently opened a draft PR and skipped Babysit while the editor showed it enabled. Matches `wait-handler`'s coercion now. - The fork check compared head against the block's typed owner/repo, so a renamed repository — which GitHub serves through a 301 while reporting the canonical name — was reported as a fork. Compares head against base now. - Each round's diff is capped like Create PR's. The cumulative guard measures the net change, so a round that reverts an earlier addition passed it while contributing a full-size diff. - Reviewer mentions must start with `@`. Each entry becomes its own issue comment re-posted every round, so a comma inside one left prose on the PR. Efficiency: - Thread and check reads per poll now run together; neither consumes the other's result and both are paginating loops. - `babysitReviewLandedSince` takes the `latestReview` the caller already fetched instead of re-listing the PR for it. - Actions log reads fan out in small batches rather than one at a time. Cleanup: - `BabysitFinalizeError` was a twin of `BabysitGitHubError`; folded together. - Replaced the hand-inlined copies of `threadsAreClean`. - Dropped `MEMORY_MODES`, a duplicate of `AUTHORING_MODES`, and the `parsePiMode` tombstone for a mode that never shipped. Adds regression tests for the quoted `.github` path, cancelled/stale required checks, and the renamed-repository snapshot. * fix(pi): budget Babysit against the run's real deadline Three fixes that each needed to land a level below where the symptom showed. Execution deadline. Babysit planned its wait loop against `getMaxExecutionTimeout()`, which unconditionally returns the enterprise async ceiling (90 min) with no regard for the run's plan or sync/async mode — the sync ceiling is 300s on free, 3000s on pro. A synchronously triggered run was therefore killed mid-loop with the PR already opened, its review comments already posted, and none of the rounds/stopReason outputs produced. Rather than thread a numeric deadline through ExecutionContext — five entry points that would each have to remember it, and nothing to catch the one that forgot — `createTimeoutAbortController` now records the deadline against the signal it creates, and `getRemainingExecutionMs(signal)` reads it back. The number cannot disagree with the timer that enforces it, because both are established in the one place the timeout exists, and every entry point already hands the executor that signal. `undefined` means unknown, not unlimited: Babysit keeps the old ceiling as its fallback for an untimed run. Job logs. The executor caps a response at 10 MB and throws rather than truncating, so a verbose CI job produced no diagnostic at all — and GitHub Actions reports null title/summary on every check run, leaving the agent a bare URL it has no tool to follow. The tool now sends `Range: bytes=-N` so the storage host returns only the tail. Since a suffix range is a request and not a guarantee, a 200 still takes the local slice. That made the old output dishonest, so the contract changed while the tool is still new and has one consumer: `totalCharacters` (which a ranged read cannot know) is replaced by `totalBytes`, sourced from the `Content-Range` total and null when unreported. A ranged body is trimmed at its first line break, since the byte window cuts mid-line and can split a multi-byte character. Generated docs. `github.mdx` was missing the `repo_full_name` the PR reader now returns, and regenerating deleted the head/base rows instead of adding it: `scripts/generate-docs.ts` only expands a spread at depth 0 of a const, and `PR_BRANCH_REF_OUTPUT` spread inline under `properties`. Restructured into a named properties const in types.ts, next to the shapes it belongs with, which the generator resolves the same way it already resolves BRANCH_REF_OUTPUT. Only the github.mdx hunk is committed. The generator is lossy elsewhere — it drops 126 lines of trigger configuration from jira.mdx — which is pre-existing drift for whoever owns that surface, not this branch. * fix(pi): refuse any Git-quoted path before the Babysit push `core.quotePath=false` stopped Git escaping non-ASCII bytes, but it closed one instance rather than the class. Git still quotes any path it cannot state on a single line — one containing a newline, a double quote, a backslash, or a tab — and such a path arrives with a leading `"`, so the `.github/` prefix test does not match it and the refusal is bypassed exactly as before. Refuse any quoted path outright. Unescaping instead would put a second implementation of Git's quoting rules on the security-relevant side of the push, and these characters have no legitimate place in a source path. * fix(pi): harden the Babysit push against a repository the agent controls Every host-side guard before the push was computed by a git invocation the agent could influence from inside `.git/`, which is never committed and so is covered by neither the config digest nor any content check. - Diff measurement ignored `.git/info/attributes`. One line saying `* -diff` reduced a 500 KB change to `Binary files differ`: verified locally as 119 bytes reported against a real 500,147, so the 200 KB cumulative bound never fired, and the `diff` block output — the only record of what an unattended, review-comment-steered agent pushed — recorded the same nothing. Both diffs now run `--text --no-ext-diff --no-textconv` with an empty `core.attributesFile`. - `rev-list --count` and `merge-base --is-ancestor` answered about replaced history. A `refs/replace/*` mapping makes a five-commit chain report one, while the pushed SHA stays real. `GIT_NO_REPLACE_OBJECTS` is now set for both the prepare and push commands. - The push asserted HEAD's *shape* — right branch, one commit past the pin, a descendant — every part of which survives `commit --amend` to a different tree. So the bounds, the quoted-path refusal, the `.github/` refusal and the reported diff could all describe a commit other than the one pushed. The validated SHA is now passed in and the refspec names it. - The digest comparison resolved `cat`, `sha256sum` and `cut` through `$PATH` in the same script that invokes git absolutely for exactly that reason, and was handed the expected digest in its own environment — so a shim could print it. The utilities are now absolute. - The clone left the token in `.git/config` when the pinned-SHA assertion failed, because `set -e` aborted before `remote set-url`. Reordered. - `changedFiles` reached the block output unscrubbed on the Babysit path, where Create PR already scrubs it. Names are agent-chosen. These reduce exposure; they do not bound it. E2B runs each command as `bash -l`, so a login-shell profile the agent writes executes ahead of the next host command with the token in its environment. Removing the token from the sandbox entirely means pushing host-side through the Git Data API, which the ≤50-file / ≤200 KB bound already makes practical. * fix(pi): correct switch coercion for draft and tidy Babysit reporting - `draft` had the same string-coercion bug that `babysitMode` was fixed for one line above it. A switch arrives as `'true'`/`'false'` when its value came through a variable reference, an API trigger payload, or a legacy serialized workflow, and `inputs.draft !== false` read `'false'` as truthy — opening a draft PR against the user's explicit setting. Both now go through one `isSwitchEnabled` helper that handles either polarity and takes the field's default, because the bug is opposite on each. - `mergePhaseDiffs` joined two separately-capped diffs without re-capping, so the combined output could reach twice MAX_DIFF_BYTES. - The cancellation poller's `logger.warn` was the one message in these files emitted unscrubbed. A Redis poll error is unlikely to carry a run credential, but a uniform invariant is easier to keep than a per-call-site argument. - Renamed `waitWithSandboxKeepalive` to `waitWithSandboxProbe`. E2B's `timeoutMs` counts down from create and is reset only by `Sandbox.setTimeout`, never by running a command, so `true` every four minutes proves liveness and buys no time. The old name invited raising the round wait on the assumption that waits extend the sandbox, which would let E2B reap it mid-wait. - Dropped a `{@link}` to a symbol in another module that was never imported. * docs(tools): record why the Babysit GitHub tools are registry-only The same branch added four user-facing GitHub tools through the full block-exposure recipe (v2 variant, tools.access, dropdown, subBlocks) and five internal ones through none of it. The distinction is deliberate — the five are called by the Pi Babysit handler via executeTool, which resolves against the registry rather than any block's access list — but nothing in CI encodes it, and `check-block-registry.ts` silently skips ids it cannot find. Worth stating because the trap is non-obvious: `GitHubV2Block` builds its access list by appending `_v2` to every entry, so adding one of these to `tools.access` without first adding a v2 variant would point the block at an id that does not exist. * docs(pi): document the clean stop reason and Babysit's fixed bounds The FAQ told readers to inspect `stopReason`, but the reference list never named `clean` — the one value that means the PR actually reached the goal state — and omitted `closed_or_merged`, `fork_pr`, and `check_read_failed`. Also records the bounds that were previously undiscoverable, split by how each one actually behaves: the reviewer-mention limits reject the block before the run starts, the 30-thread limit trims a round, and only the failing-check and cumulative-change limits produce `bounds_exceeded`. Corrects step 6, which claimed Babysit reruns CI. It never does — the push is what re-triggers checks. Co-Authored-By: Claude <noreply@anthropic.com> * docs(tools): correct and widen the registry-only note The previous note contrasted these five with "the user-facing tools added alongside them", implying this branch added both. It did not: the branch never touches blocks/blocks/github.ts, and github_create_pr_review came from #5471, which predates staging. The real contrast is with every user-facing GitHub tool in the registry. Also records the governance consequence, which was the part actually worth writing down: the permission-group deny list is built from tools.access, so an admin cannot deny these from the UI, and the allowedIntegrations gate keys on block type while Babysit calls them with a tool id alone. Co-Authored-By: Claude <noreply@anthropic.com> * fix(pi): size the sandbox to the run's own execution timeout The Pi sandbox lifetime was a global constant while the execution timeout is per-plan and per-mode, varying 18x (a free sync run gets 5 minutes, an async run 90). Every Pi sandbox asked E2B for the same sub-hour ceiling, so a five-minute run whose web process died left a sandbox billing for an hour. PI_SANDBOX_LIFETIME_MS could not close the gap: its floor is 31 minutes. resolvePiRunLifetimeMs lowers the provider ceiling to whatever the run's own deadline leaves, read from the signal that enforces it. Untimed runs and Daytona are unchanged, so no path gets a longer lifetime than before. The turn cap had to move with it. PI_TIMEOUT_MS reserved the clone and both finalize budgets out of the ceiling as a module constant; leaving it there while shrinking the lifetime would re-open the exact bug its docs describe — the sandbox dying first, taking the agent's finished work with it unpushed. It is now resolvePiTimeoutMs(lifetimeMs), and each backend resolves the lifetime once and feeds both, so the two cannot disagree. Two things this surfaced: Babysit had to read context.signal, not the cancellation signal it uses everywhere else. createCancellationSignal returns a fresh controller that only forwards aborts, so the deadline lookup answers "unknown" through it and would have silently left the longest-lived mode on the ceiling. Covered by a test that fails against the wrong signal. The E2B adapter tested lifetimeMs for truthiness, so a run resolving to zero would have had the key dropped and been handed the SDK's five-minute default - longer than it asked for, on the run least entitled to it. Options precede the callback in withPiSandbox so that adding one did not re-indent every caller's sandbox body. Co-Authored-By: Claude <noreply@anthropic.com> * fix(pi): raise the sandbox ceiling to the longest execution we allow The ceiling was pinned just under E2B's one-hour *Hobby* session limit. Sim is on Professional, where the limit is 24 hours, so the cap was enforcing a restriction no plan imposes — and it sat below the 90-minute async execution ceiling, which made the sandbox the binding constraint. A long Babysit run could be handed a 90-minute budget and still lose its sandbox at 59. Derived from getMaxExecutionTimeout rather than given a number of its own, so the sandbox always outlasts the longest run the platform permits and an operator who raises the async timeout does not have to know this file exists. The provider session limit stays as a clamp, so the derivation can never ask E2B for a lifetime it will refuse. Effect: ceiling 59 -> 90 min, and the agent turn it funds 29 -> 60 min, since resolvePiTimeoutMs reserves the clone and both finalize budgets out of it. Runs with a shorter deadline are unaffected — resolvePiRunLifetimeMs already lowers the ceiling to whatever the run itself has left. Co-Authored-By: Claude <noreply@anthropic.com> * docs(pi): describe the deadline-sized sandbox, not a fixed hour Both facts in this paragraph were stale: the lifetime is no longer a single sub-hour constant (it tracks the run's own remaining execution time), and the ceiling was justified by E2B's Hobby limit on an account that is on Professional. Co-Authored-By: Claude <noreply@anthropic.com> * fix(pi): share the sandbox sizing and lift E2B off the base default The two Pi images had drifted on exactly the axis their shared module exists to prevent. Daytona asked for 4 CPU / 8 GB; the E2B template asked for nothing and inherited its base default of 2 vCPU / 512 MB. That is a 16x memory gap between the provider Pi normally runs on and the one it fails over to, so a failover could be OOM-killed doing work that had just succeeded. 512 MB is too small independently of the drift: the Pi CLI is a Node process holding an LLM context, running beside a clone of the user's repository, and Node is OOM-killed rather than degraded at that ceiling — which reaches the user as an opaque agent failure. CPU and memory now come from pi-sandbox-packages.ts alongside the package lists. Disk stays in the Daytona renderer: its 10 GB per-sandbox cap is a hard provider limit with no E2B equivalent, so it is the one dimension where the images legitimately differ. E2B fixes resources at template build time, so this takes effect only when build-pi-e2b-template.ts is re-run — nothing builds these images in CI. Co-Authored-By: Claude <noreply@anthropic.com> * chore(pi): remove internal planning files * chore(pi): remove generated review commands * fix(pi): align babysit toggle visibility --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 16:08:37 -07:00
mzxchandraandWaleed Latif	6ff6255900	feat(outlook): add Microsoft Graph calendar operations (#6041) ... * feat(outlook): add Calendars.ReadWrite and MailboxSettings.Read scopes Extend the shared outlook OAuth service with delegated Graph calendar scopes so one Microsoft connection covers mail and calendar. Existing connected users must reconnect to be granted the new scopes (noted in a code comment). Adds human-readable scope descriptions and test assertions. * feat(outlook): add Microsoft Graph calendar tools Six calendar operations against graph.microsoft.com/v1.0: list events (calendarView with nextLink paging), get, create, update (partial PATCH), delete, and respond to invites. Shared calendar-utils handles Graph's offset-less dateTime+timeZone shape, attendee normalization, and event flattening. All tools carry the Microsoft Graph error extractor and 429/backoff retry (honors Retry-After) for the mailbox concurrency limit. Registered in the tool registry and barrel. * feat(outlook): surface calendar operations in the Outlook block Add six calendar operations to the Outlook block operation dropdown with their conditional subBlocks, tool wiring, inputs, and event-shaped outputs. Mail operations are unchanged and backward compatible. * fix(outlook): address calendar review findings - Validate list-events pageToken origin with assertGraphNextPageUrl (matches the onedrive/microsoft_ad Graph paging guard) so a workflow-supplied URL can't receive the Outlook bearer token. - All-day create/update now normalize both bounds to midnight and force an exclusive end day (buildAllDayRange), instead of sending a zero-length same-midnight window that Graph rejects. - Drop the stale 'suggest meeting times' claim from the block longDescription. - Remove the unused MailboxSettings.Read scope (least privilege; no tool reads it). * feat(outlook): add calendar picker and harden calendar tools Validation pass over the new Microsoft Graph calendar operations against the v1.0 API docs, plus the calendar selection the tools were missing. - Add an `outlook.calendars` picker (GET /me/calendars) with basic selector + advanced manual ID, wired through the `calendarId` canonical param. List and create now target `/me/calendars/{id}/...`; get/update/delete/respond keep using `/me/events/{id}` since event IDs are mailbox-unique. - Fix all-day update rejecting when only one bound is supplied — both bounds are now normalized to midnight with an exclusive end day. - Fix "Send Response to Organizer" reading as OFF while Graph's default is to notify; it is now a dropdown defaulting to Yes, and the param is always sent. - Add timestamp wandConfig to the four calendar datetime fields and a list wandConfig to attendees. - Centralize Graph URL construction in calendar-utils; guard maxResults against non-numeric input and trim the calendarView time-window bounds. - Add three calendar templates and four calendar skills to OutlookBlockMeta. - Regenerate integration docs. * fix(outlook): scope online-meeting comments to what Graph documents onlineMeetingProvider is optional and defaults to unknown; the docs state that setting isOnlineMeeting alone initializes onlineMeeting. They do not document Graph substituting the calendar's defaultOnlineMeetingProvider, so the comments now claim only that, plus the real reason not to pin teamsForBusiness (mailboxes that disallow it via allowedOnlineMeetingProviders). * fix(outlook): allow calendar paging without re-supplying the time window Cursor Bugbot round: startDateTime/endDateTime were required tool params, so a paging call carrying only pageToken failed validateToolParameters before the request was built — even though the url builder short-circuits on pageToken and ignores both bounds. Relax them to optional and enforce the real invariant (pageToken OR both bounds) in the url builder, matching tools/sharepoint/list_sites.ts. Block subblocks stay required, so the normal editor flow is unchanged. Also correct the calendar_respond comment: Graph documents exactly two 400 conditions for accept/decline, both on proposedNewTime, which we never send. A non-empty comment alongside sendResponse=false is valid. * feat(outlook): add Calendars.ReadWrite.Shared for shared calendars The calendar picker lists /me/calendars, which can include calendars other users have shared with or delegated to the account. Calendars.ReadWrite covers only the user's own calendars, so selecting a shared team calendar would 403 on both read and write. Calendars.ReadWrite is kept alongside it, not replaced: Graph documents it as the sole accepted permission for creating and updating events and for accept/tentativelyAccept/decline (all list "Higher: Not available"), so the .Shared scope does not subsume it. Added now rather than later because this PR already forces existing Outlook users to re-consent for Calendars.ReadWrite; deferring would cost them a second reconnect. * fix(outlook): treat date-only bounds as all-day and guard partial all-day updates Cursor round 2: - Date-only bounds no longer produce a zero-length window. The param docs invited a date like 2025-06-03 for an all-day event, but buildAllDayRange only ran when isAllDay was explicitly true, so date-only input built a 00:00->00:00 timed window that Graph rejects. A date-only bound carries no time, so the only coherent reading is all-day; create/update now promote on that shape and the descriptions state it. - Converting an event to all-day with no bounds now fails with an actionable message instead of a Graph 400. Graph requires all-day events to have midnight start and end in the same zone, and those cannot be derived from a partial PATCH against an event whose existing bounds are timed. * docs(outlook): note that the calendar window fields are ignored when paging * fix(outlook): don't promote a lone date-only bound to all-day on update Regression from the previous round's date-only promotion. A single date-only startDateTime or endDateTime satisfied "all provided bounds are date-only", so the tool promoted to all-day and derived the missing side from the supplied one — turning a partial reschedule of a timed or multi-day event into a one-day all-day event and dropping the original other bound. Implicit promotion now requires BOTH bounds to be date-only, matching calendar_create_event. A lone date-only bound is ambiguous (convert to all-day, or just move that edge?) and a PATCH cannot read the event's existing bounds to disambiguate, so it stays on the timed path and leaves the other side untouched. Deriving a missing bound remains allowed when isAllDay is set explicitly, since that is stated intent rather than a guess. Also pins the explicit-isAllDay-false + date-only override with a regression test: the block always sends isAllDay for create, so an untouched switch arrives as false, and the data shape has to win or the fix would be unreachable from the UI. * revert(outlook): drop Calendars.ReadWrite.Shared from the outlook provider Reverts the scope I added two rounds ago. It was the wrong call. This provider is shared by work/school AND personal Outlook accounts, and the .Shared calendar scopes are not confirmed supported for personal Microsoft accounts. Requesting one risks failing consent for personal users — which would take mail access down with it, breaking functionality that works today. The PR already documents this exact reasoning as why findMeetingTimes was excluded, and that decision was made against a live personal mailbox. The evidence I added it on was a summarized read of the permissions reference claiming MSA support; a targeted follow-up could not confirm it for Calendars.ReadWrite.Shared specifically. Given the asymmetry — broken consent for all personal users vs. a shared-calendar feature gap — least privilege wins. Calendar operations therefore target calendars the account owns. The calendarId param descriptions now say a calendar shared by another user may return 403, and the scope list carries a comment explaining why .Shared must not be re-added. * fix(outlook): make retried event creates duplicate-safe via transactionId The retry config opts POSTs in via retryIdempotentOnly:false, but the executor's isRetryableFailure covers 429 AND 500-599 — not just the throttle the comment justified. A 5xx returned after Graph had already committed a create would be retried and produce a duplicate calendar event. create_event now sends a transactionId, which Graph documents for exactly this: it discards a repeat POST carrying an id it has already seen. The request body is built once per execution (formatRequestParams runs before the attempt loop), so the id is stable across retries of a call and unique between calls. The retry comment now describes what actually retries and why each non-idempotent method is safe: PATCH replays the same partial body as a no-op, and respond is state-idempotent though a post-commit retry can send the organizer a second notification — accepted deliberately, since Graph exposes no transactionId for accept/decline and failing outright under throttling is worse. * fix(outlook): tighten date-only detection and align online-meeting copy Final validation pass over the calendar integration. - isDateOnly matched "contains no T", so a space-separated datetime (2025-06-03 10:00) counted as date-only: the time was discarded and the value built as '2025-06-03 10:00T00:00:00', which Graph rejects. It now matches YYYY-MM-DD strictly, and buildGraphEventDateTime normalizes the space form to ISO rather than mangling it, so a natural input works instead of 400ing. - The isOnlineMeeting param descriptions still claimed Graph 'uses the mailbox default provider' — the same unverified mechanism already removed from the code comments. They now state only what the docs and the author's live testing support: the join URL depends on the providers the mailbox allows, and stays null on personal accounts. - Adds blocks/blocks/outlook.test.ts following the repo's per-block test convention: every calendar operation resolves to a registered tool in tools.access, supplies all required tool params, emits no params the tool cannot accept, maps one-to-one onto the calendar tools, and the calendarId canonical group and sendResponse default are pinned. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-29 10:20:13 -07:00
5 people	1d64b92b41	feat(desktop): desktop app (#5998) ... * top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai>	2026-07-28 19:25:59 -07:00
Vikhyath Mondreti	c809845b99	improvement(self-host): enterprise features enabling (#6028) ... * improvement(self-host): enterprise features enabling * chore(helm): bump chart to 1.3.0 for the enterprise self-host values values.yaml gained the ENTERPRISE_ENABLED switch and INSTANCE_ORG_* keys, and the feature-flag envDefaults moved from "false" to empty so the master switch can resolve them. Additive and backward compatible, so a minor bump. * fix(self-host): address review findings on instance org and org delete Drop the per-process instance-org id cache. It went stale once the organization was deleted through the Admin API, and clearing it from the delete handler would only heal the replica that served that request. The lookup runs on the signup path against a single-row table, so re-reading costs nothing and keeps every replica self-correcting. Scope the org-delete subscription conflict to entitled statuses. Matching any row regardless of status let a canceled subscription — which bills nobody — permanently block deletion. * fix(admin): block org delete on any live subscription, not just entitled ones ENTITLED_SUBSCRIPTION_STATUSES excludes trialing, so a trial — which grants no entitlement but is a live Stripe subscription that will convert — slipped past the delete guard and could be stranded against a removed organization id. Adds TERMINAL_SUBSCRIPTION_STATUSES and inverts the predicate: block unless the row is finished. Expressed as the terminal set so a status Stripe adds later defaults to blocking, which is the safe direction for a destructive operation. * fix(self-host): resolve SSO and access-control in the UI, not the raw env var Nine client consumers still read NEXT_PUBLIC_SSO_ENABLED / NEXT_PUBLIC_ACCESS_CONTROL_ENABLED directly while the server gates and settings nav had moved to the resolver. With only ENTERPRISE_ENABLED set that produced dead ends: the SSO settings section appeared but ssoClient() was never registered and no login button rendered, and the Access Control section appeared but its page reported "not entitled". Points every consumer at isSsoEnabled / isAccessControlEnabled so visibility and capability come from one place. * fix(admin): validate retention workspace targets on the Admin API too retentionOverrides and per-workspace PII rules both name a workspace, and neither field is a foreign key. The settings UI rejected ids belonging to another organization; the Admin API did not, so the two paths could persist different data for the same org. Extracts the check as getForeignWorkspaceTargetsReason and points both routes at it, so they cannot drift apart again. * fix(self-host): close three review findings on admin routes and cleanup Make org delete atomic. detachOrganizationWorkspaces committed on its own, so a failed delete left workspaces detached and re-billed while the organization, its members, and its settings survived. Adds a Tx variant so both commit together. Gate the admin session-policy PATCH on entitlement, matching the settings UI. Without it the stored policy was inert — getSessionPolicy resolves to no-op when the feature is off, so the one eager clamp would be undone on the next refresh. Stop emitting plan-wide housekeeping when billing is off. It is keyed to the hosted free-tier 30-day window, the same default the per-workspace pass deliberately refuses to apply off-hosted. * fix(admin): gate whitelabel on entitlement and emit detach audits post-commit The Admin whitelabel PATCH skipped the entitlement check the settings UI runs, so an admin key could set branding the product had not granted the org. detachOrganizationWorkspacesTx also wrote its audit rows inside the caller's transaction, contradicting its own doc comment — a rolled-back delete would have left audit history describing detachments that never happened. It now returns the rows and callers emit them after commit. * fix(self-host): refuse instance-org resolution when the slug is ambiguous organization.slug has no unique constraint, and the lookup took the first of however many matched. The choice is unordered, so two replicas could resolve different organizations and split new signups between them. Resolution is now three-state. Ambiguity is distinct from absence, so it both declines to adopt an arbitrary organization and declines to provision another one on top of the duplicates.	2026-07-28 19:02:41 -07:00
4 people	e8e3d6984c	feat(pi): optional multi-provider web search for the coding agent (#5951) ... * feat(pi): optional multi-provider web search for the coding agent Adds a search provider dropdown (Exa, Serper, Parallel, Firecrawl) to the Pi block, off by default. The selected provider's key comes from the block field or Workspace Settings → BYOK; a Sim-hosted key is never spent, so a missing key fails the run with a setup message instead of quietly billing Sim. Search is available in all three modes. Local Dev and Review Code register a host-side tool that goes through the existing provider tools, while Create PR has no host in the loop and gets a generated Pi extension in the sandbox. Both paths derive their requests from one normalizer and are held together by a parity test, since the sandbox copy cannot import Sim's code. Results are normalized to title, URL, snippet, and publication date, capped per field and per envelope, marked untrusted in the prompt, and limited to 20 searches per run so a tool loop cannot drain the workspace's quota. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pi): drop the banned JSON round-trip from the search parity test `check:utils` bans `JSON.parse(JSON.stringify(...))`. The round-trip was normalizing the host body to its wire form, which buys nothing here: the bodies are plain JSON and `toEqual` already ignores undefined members. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pi): upgrade the E2B SDK so long Pi output streams stop failing Create PR streams the whole Pi run through one Connect server-stream (`commands.run` -> envd `Process.Start`), held open for the full `PI_TIMEOUT_MS`. Mid-stream it could die with: [internal] protocol error: received unsupported compressed output That string is `@connectrpc/connect-web`, not Pi — Pi has no Connect dependency at all. connect's `compressedFlag` is `0b00000001` and gzip's magic first byte is `0x1f`; `0x1f & 0x01 === 1`, so a raw gzip body fed to the envelope reader trips this on byte one. It reads as "the server sent a compressed envelope" but really means "this was never a Connect envelope" — an HTTP-level gzip that was not transparently decompressed. e2b 2.30.0 pinned `@connectrpc/connect-web@2.0.0-rc.3` and drove envd through undici 7 with `allowH2: true`. e2b 2.36.1 moves to stable connect-web 2.1.2 and loads undici 8.8.0 when Node >= 22.19.0 — exactly our engine floor — so the failing path gets a different HTTP stack. The connect-web upgrade alone is not the fix: 2.0.0-rc.3 and 2.1.2 ship a byte-identical `connect-transport.js` (bar the copyright year), and connect-web still has no `acceptCompression` option by design. The undici 8 swap is the part that matters. `@e2b/code-interpreter@2.7.0` only asks for `e2b: ^2.28.0`, so the override pins the floor we actually need. Verified API-compatible: every method we call (`Sandbox.create`, `runCode`, `commands.run`, `files.read/write`, `kill`, `Template`, `defaultBuildLogger`, `waitForTimeout`) has an identical signature across the two versions, and we never touch `SandboxPaginator`, the one type that changed. * fix(pi): correct search normalization edge cases and the budget's stated scope Follow-ups from review of the web-search work. Each fix lands in both the host adapter (`normalize.ts`) and the Create PR sandbox copy (`extension-source.ts`), with the extension test asserting the two produce byte-identical envelopes. - `usableUrl` was the one provider-controlled field not whitespace-bounded: title/snippet/date all go through `collapseWhitespace`, `url` only trimmed. Up to 2048 chars of newlines and control characters could ride into the envelope. Dropped rather than collapsed — `url` must stay byte-exact to stay resolvable, so collapsing would emit a different, still-dead link, and a URL carrying raw whitespace is already malformed under RFC 3986. - `numResults: null` (or `''`, or `[]`) returned 1 result, not the documented default of 5: `Number(null)` is a finite 0, so the clamp floor won rather than the default. Only a real number or a non-blank numeric string now counts as the model having asked for a count. - Envelope truncation was silent. When results were dropped to fit the 50 KB ceiling the model read the short list as the complete answer. It now carries a message saying so, and the message is inside what gets measured so the note cannot push a truncated envelope back over the ceiling. - The budget is per *block execution*, not per workflow run: the counter lives in the tool spec and both adapters build a fresh one per execution, so a Pi block inside a Loop gets the full allowance every iteration. The constant, the agent-facing message, and the docs all claimed "per run". Renamed to `PI_SEARCH_MAX_CALLS_PER_EXECUTION` and corrected the wording rather than tightening the cap, since a shared ceiling would fail late iterations of a legitimate fan-out. - The Search API Key tooltip promised "switching providers clears this field". That clear is driven through the collaborative editor setter, so a workflow imported, forked, or updated via the API keeps the previous provider's key — exactly the case where sending it to a new vendor matters. Docs also gain a warning that Create PR hands both the model key and the search key to the agent as environment variables, which Pi copies into every bash child. That matters most for Settings > BYOK keys: those are workspace-scoped, only admins can manage them, and the API only ever returns them masked — yet anyone who can run a Pi block in Create PR mode can read the raw value. * fix(pi): make the search provider drift guards actually fire The "you cannot add a provider without mirroring it" story rested on two mechanisms that did not hold. Verified by adding a fifth provider to `PI_SEARCH_PROVIDERS` and running the build: it produced only two errors, and every test still passed. - `normalizePiSearchRecords` assigns to `let built` inside its switch rather than returning, so unlike its two siblings a missing case was not a type error — it silently normalized the new provider to zero results. Added an explicit `never` check. - The sandbox copy's `normalizeRecords` used a trailing `else` for Firecrawl, so an unmirrored provider was silently normalized with Firecrawl's field names; `extractRecords` did the same with its `payload.data` tail. Both now test for `firecrawl` explicitly and throw otherwise. - `Record<PiSearchProvider, ...>` on the `TOOLS` and `payloads` fixtures looked like exhaustiveness guards but are inert: `apps/sim/tsconfig.json` excludes `**/*.test.ts`, and vitest transpiles without typechecking. Both suites drive their providers off `Object.keys(fixture)`, so a missing provider was skipped rather than failed. Each suite now asserts its fixture covers the registry. Re-running the same experiment now yields three compile errors plus two test failures naming the missing fixtures. * fix(pi): drop the workspace BYOK fallback for the search key A fallback exists so a key has somewhere to go when the field is unavailable. The Search API Key field is unconditionally available: unlike the model key, whose visibility runs through `shouldRequireApiKeyForModel` and its `isHosted` branch, `getSearchApiKeyCondition` gates only on whether a provider is selected. So the fallback never had a configuration to cover. Removing it also closes an escalation. Workspace BYOK keys are admin-managed and the API only ever returns them masked, yet `resolvePiSearchKey` would resolve one for any member who could run the block — and in Create PR that key is handed to the sandbox as an environment variable, which Pi copies into every bash child. A member could read a credential the product deliberately never shows them. Requiring the key on the block keeps the sandbox exposure to a key its author already holds. Nothing depends on the fallback: it has never shipped. - `resolvePiSearchKey` is now synchronous and returns the key, since there is no lookup left to await. `byokProviderId` leaves the search registry and `PiSearchKeySource` / `PiSearchKeyResolution` are gone — with one source, `keySource` carried no information, and the logging rationale for it (a block field silently shadowing a stored key) no longer exists. - The field is now `required`. Safe alongside its condition: the serializer's required check returns early for fields that are not visible, so a Pi block with search off still validates. Pinned by a test. Docs and the block's tooltip, placeholder, and best practices updated. The Create PR key-exposure callout now explains the missing fallback rather than recommending the block field as a way around it. * docs(pi): import Callout explicitly, as the sibling block docs do `fumadocs-ui/mdx`'s `defaultMdxComponents` already provides `Callout`, so the callout added earlier rendered fine without this — but logs.mdx, credential.mdx, and response.mdx all import it explicitly and pi.mdx was the outlier. Not a build fix: the docs Vercel deployment is failing on staging HEAD as well. * chore(deps): exclude the e2b packages from the release-age gate CI's `bun install --frozen-lockfile` failed on the E2B upgrade: error: No version matching "@e2b/code-interpreter" found for specifier "^2.7.0" (blocked by minimum-release-age: 604800 seconds) This did not reproduce locally because the checkout's bun was 1.2.15, which predates `minimumReleaseAge` support and ignored the gate outright; CI runs the pinned 1.3.13 and enforces it. Excludes only the two packages that are actually too young — @e2b/code-interpreter 2.7.0 (2026-07-23) and e2b 2.36.1 (2026-07-27). The rest of the chain already clears the gate: @connectrpc/connect{,-web} 2.1.2 and @bufbuild/protobuf 2.13.0 and undici 8.8.0 are all older than a week, and `tar` resolves from the lockfile at 7.5.22 without needing an exception (the original CI error named only @e2b/code-interpreter, and `bun install --frozen-lockfile --ignore-scripts` under 1.3.13 now passes locally). The lockfile is regenerated with bun 1.3.13 rather than 1.2.15, which also corrects hoisting the older bun had gotten wrong on the merge commit: the root `lucide-react` hoist moves from 1.23.0 back to 0.511.0 and `@radix-ui/react-slot` from 1.3.0 to 1.2.2, each with the proper scoped entries. Package resolution still differs from staging by exactly the e2b chain and nothing else. Both entries age out on 2026-07-30 and 2026-08-03; drop them then. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-28 16:16:51 -07:00
Waleed	b69fdbd17b	fix(api): give every v1 endpoint quota headers and errors that name the field (#6012) ... * fix(api): give every v1 endpoint quota headers and errors that name the field Three consistency gaps found by probing the live v1 surface end to end. Rate-limit headers were only published by routes built on createApiResponse — workflows, logs and audit-logs. Tables, files and knowledge are rate limited by the same bucket and will return 429, but published no quota on success, so a client discovered the ceiling only by hitting it. Adds a shared rateLimitHeaders() builder, reused by createRateLimitResponse, and attaches it to all 31 success responses on those three families. Missing required fields did not name themselves. .min(1, '...') only fires for a present-but-empty string, so an omitted field fell through to Zod's default "Invalid input: expected string, received undefined". A previous pass fixed the shared id schemas, but 22 of 23 v1 workspaceId declarations bypassed them, so the fix reached almost nothing. Adds requiredFieldSchema(message) and routes every v1 request input through it, preserving each site's existing, more specific wording (for example "workspaceId query parameter is required") instead of flattening them to the generic one. Response schemas are left alone — "required" wording would be wrong there. Validation failures on tables, files and knowledge reported the literal "Validation error" and discarded the schema's message. Adds v1ValidationErrorResponse, which surfaces the first issue while keeping details, and wires it into the 19 parseRequest calls that had no handler. Routes with deliberately specific wording keep theirs. The global default is untouched, since routes outside v1 assert the current string. * fix(api): finish the v1 consistency sweep at call level, not file level Review found three places the first pass missed, all from filters that worked on whole files instead of individual call sites. - GET /api/v1/files/{fileId} returns the file bytes via `new Response`, not a `success: true` JSON body, so the header pass skipped it. The download now carries the same quota headers as the DELETE beside it. - Four parseRequest calls in the table-row routes still reported the generic "Validation error". The first pass skipped any file that already had a handler anywhere in it, which excluded these two files wholesale. The check is now per call site, and no bare call remains. - POST /api/v1/tables takes its body from the shared tables contract, which still used the bare `.min(1)` form, so an omitted workspaceId did not name itself. Converted there and in the other v1-reachable contracts. Scope note: roughly a thousand `.min(1, '... is required')` declarations remain under contracts/tools/**. Those are block and tool definitions rather than the public REST surface, and converting them belongs in its own change. * refactor(api): publish quota headers from one chokepoint, not 32 call sites A quality pass found the previous commit only made the happy path consistent. Those three route families have 117 response sites; 32 got headers. The other 85 are the error paths — 400/403/404/500 — which are exactly the responses a client is deciding whether to retry, and they published no quota at all. `checkRateLimit` now records the bucket snapshot against the request, and `withRouteHandler` attaches the headers next to the `x-request-id` it already sets, on both the success and the unhandled-error branch. Every v1 response carries the quota now, and a new v1 route gets it without remembering to. The carrier is a WeakMap keyed by the request, so it needs no cleanup and routes that never record a snapshot — everything outside v1 — are untouched. This deletes more than it adds: the 32 decorations are gone, and so is the `rateLimit` parameter that had been threaded into `handleBatchInsert` purely so a business-logic helper could decorate its own response. Also from the same pass: - One definition of the header trio. `createApiResponse` had its own copy, so after the last commit there were two; both now build from `buildRateLimitHeaders`. - `v1ValidationErrorResponse` delegates to the shared `validationErrorResponse` instead of hand-rolling the same body, and takes a fallback message, which collapses four route closures that differed only in that string. - 36 sites wrote `requiredFieldSchema('Workspace ID is required')` — the verbatim definition of the exported `workspaceIdSchema`. Using the primitive is the whole point of having it; they now import it. - Dropped TSDoc that had gone stale or contradicted the call sites it advised. * docs(api): reattach the withRouteHandler docblock and drop stale wording The comment pass caught a real casualty of the previous commit: inserting `applyResponseHeaders` put it between `withRouteHandler`'s docblock and the function itself, so the file's most-used export lost its documentation to the new private helper. Reattached, and its header bullet now mentions the rate-limit trio it also emits. Remaining edits are wording only. The WeakMap rationale moved off `RateLimitSnapshot` — three self-evident fields — onto the `snapshots` declaration it actually describes. The record site no longer restates that rationale; it keeps only the part unique to it. And three id-schema docs claimed "same constraint as nonEmptyIdSchema", which stopped being true when that schema was documented as deliberately message-less. * docs(api): document the quota headers on every v1 success response The spec already asserted, in the RateLimited description, that the X-RateLimit-* trio accompanies every authenticated response. Before this branch that was false for tables, files and knowledge; it is true now, but no operation documented it — only 1 of 40 v1 success responses carried the headers. All 40 now reference the shared header components. The shared BadRequest, Forbidden and NotFound components are deliberately left alone: they are also $ref-ed by non-v1 operations that publish no quota, so annotating them there would over-claim. The RateLimited description carries the general rule instead, now stating explicitly that the only responses without the headers are the ones that failed authentication. * fix(api): stop the last v1 validation paths from swallowing the message Bugbot found GET /api/v1/tables/{tableId}/rows still answering with the bare "Validation error". Its handler special-cases malformed filter/sort JSON and then falls back to the shared helper — so the site looked handled to a check that only asked whether a handler existed, which is why the earlier call-level sweep passed over it. Auditing the whole class turned up more of the same shape: - The optional-body parses on deploy and rollback, where a bad `version` lost "version must be a positive integer". - Eleven catch-block `validationErrorResponseFromError` handlers across the table routes, which discard the message of any ZodError thrown deeper. Adds `v1ValidationErrorResponseFromError` as the v1 counterpart for unknown caught values, and routes every remaining v1 validation path through the v1 helpers. No call to the generic helpers survives under app/api/v1 outside admin, which keeps its own error envelope.	2026-07-28 12:56:38 -07:00
Waleed	79b1ed1fa1	fix(api): report the real rate-limit ceiling on every v1 endpoint (#6011) ... X-RateLimit-Limit was the bucket's refill rate while X-RateLimit-Remaining was tokens left in the bucket, and createBucketConfig sets maxTokens = refillRate * burstMultiplier. The two headers described different quantities, so remaining routinely exceeded limit — observed live on a team plan as limit 200 alongside remaining 399 — and any client computing used = limit - remaining got a negative number. Report the bucket capacity, which is what remaining counts down from. This lives in the shared checkRateLimit, so it applies to every v1 endpoint: workflows, logs, tables, files, knowledge, audit-logs and copilot. Also stops publishing rate-limit headers on an authentication failure. That path never reaches the bucket, so the previous placeholder advertised a quota that does not exist and told unauthenticated callers they had been throttled. Separately, the shared id schemas reported Zod's default "expected string, received undefined" when a required field was omitted, because .min(1) only fires for a present-but-empty string. Adding the message to the z.string() constructor makes an omitted workspaceId/organizationId/workflowId/fileId name the field it is complaining about — the first thing an API consumer sees on a malformed request. Documents all three headers in the OpenAPI spec as reusable components, including the burst-capacity semantics and the fact that they are absent on authentication failures. They were previously undocumented. Adds the first tests for the v1 middleware.	2026-07-28 11:42:42 -07:00
Waleed	ca13cc9c13	feat(api): add workflow export and import endpoints to the public v1 API (#5999) ... * feat(api): add workflow export and import endpoints to the public v1 API Adds GET /api/v1/workflows/[id]/export and POST /api/v1/workflows/import. The export envelope is accepted verbatim by import, so workflows round-trip between workspaces over the public API. Unlike the admin export, the public export is secret-sanitized: stored credentials and password fields are redacted while {{ENV_VAR}} references and block positions are preserved. Import regenerates block, edge, loop and parallel ids and de-duplicates the workflow name against the target folder. Also moves parseWorkflowVariables out of the admin types module into lib/workflows/variables/parse.ts so the public route does not import from the admin namespace. * fix(api): make workflow import atomic and clarify what export redacts Writes the imported graph and its variables in a single transaction and deletes the shell workflow row on any failure, so a caller that receives an error is never left with a partially imported workflow. Previously a throw from the variables update returned 500 while leaving the workflow behind with an empty variables map. Also narrows the export route's sanitization claim: workflow variables are emitted as stored, matching GET /api/v1/workflows/[id] and the in-app export. They are plaintext configuration readable at the same permission level this route requires; secrets belong in environment variables, which travel as unresolved references. * fix(api): close import defects found in audit and share one write pipeline Security: - Escape block names before interpolating them into a RegExp in updateValueReferences. Names reach it straight from imported workflow JSON and normalizeWorkflowBlockName preserves regex metacharacters, so a name like `a*a*a*a*b` compiled to a catastrophically backtracking pattern. A sub-kilobyte body blocked the event loop for 50s and grew exponentially. Also skip rename-to-itself, which is the entire map on the import path, so the scan no longer runs at all there. - Validate folder ownership before folder lock state, so a locked folder in another workspace can no longer be distinguished from a missing one. Correctness: - Gate the imported graph on workflowStateSchema, the same schema the canonical PUT /api/workflows/[id]/state path enforces. Without it a valid 201 could persist a block field of the wrong type, which then threw on every subsequent read and left a workflow nothing could open. - Guard the compensating delete so a failed rollback logs the orphaned id instead of vanishing into a generic 500. - Validate variable `type` against the enum and build the record on a null-prototype object, so a `__proto__` key no longer silently drops the variable. - Bound payload-derived names and descriptions to the same limits the contract declares for the explicit overrides. - Return the description as stored rather than coercing '' to null, matching GET /api/v1/workflows/[id]. Shared code, so the two write paths cannot drift: - Extract prepareWorkflowStateForPersistence and use it from both PUT /api/workflows/[id]/state and the v1 import route: agent-tool sanitization, block backfill, dangling-edge removal, and loop/parallel recomputation now have one implementation. - Persist inline custom tools on import, which the canonical path already did. - Move variable normalization into lib/workflows/variables and repoint the admin importer at it, removing the last duplicate. Docs: - OpenAPI: oneOf -> anyOf on the import body. WorkflowExport matches any object, so every valid object payload matched two branches and failed validation under any spec-driven validator. Document 423 and the loss of workspace-scoped bindings on export. Tests: prepare-state unit tests and a real export -> import round trip with no mocks of the sanitizer or parser, covering loop/parallel children and the regex-metacharacter payload. * fix(api): cap import names inside the bound and align the three import paths - `truncate` appends its suffix after slicing, so capping at the contract limit produced 203/2003-character values — past the very bound the cap exists to enforce, and into the headroom reserved for dedup suffixes. Reserve the ellipsis inside the limit. - Match `extractWorkflowName`'s candidate order (state.metadata.name before workflow.name) and trim, so the v1 API and the in-app importer resolve the same name for the same payload. Previously a hand-authored payload carrying both could yield two different names. - Run the admin importer through prepareWorkflowStateForPersistence too. It was writing raw parsed state, so a dangling edge tripped the workflow_edges foreign key and a block missing its backfilled columns could land unopenable — the same class this PR just closed on the v1 path.	2026-07-27 22:50:13 -07:00
WaleedandWaleed Latif	a042f0bd0f	chore(deps): fix OTel version split, drop dead deps, declare emcn peers (#5994) ... * chore(deps): fix OTel version split, drop dead deps, declare emcn peers - Pin @opentelemetry/{resources,sdk-metrics,sdk-trace-base,sdk-trace-node} to exact 2.7.1 so they match sdk-node's pins instead of floating to 2.8.0. The carets meant app code built spans with 2.8.0 and passed them into NodeSDK from 2.7.1, which only worked by duck-typing. - Declare the 13 packages @sim/emcn imports but never declared, as peers mirrored into devDeps. @radix-ui/react-dismissable-layer had no declaration anywhere in the repo and resolved only transitively. - Remove ffmpeg-static: its binary downloads via postinstall, but it is not in trustedDependencies and Docker installs with --ignore-scripts, so the accessSync branch never succeeded and both call sites always fell through to system ffmpeg. - Remove critters + experimental.optimizeCss: Next only loads critters from the Pages Router renderer, and apps/sim is App Router only. - Make simstudio-ts-sdk zero-dependency by dropping node-fetch for native fetch; engines >=18. - Remove unused @vercel/og and postgres from docs, dotenv/inquirer/listr2 from the CLI, and yaml from the root. - Move @aws-sdk/client-appconfig from the root to apps/sim, its only consumer. - Delete the apps/sim overrides block; Bun only honors top-level overrides. - Bump free-email-domains 1.2.25 -> 1.9.70 (4,779 -> 13,059 domains). - Validate object/array variables with JSON.parse instead of JSON5, matching what the executor actually parses. - Swap the changelog GitHub icon off lucide to GithubOutlineIcon, matching the navbar chip on the same page. - Unify @types/node on 24.2.1 and lucide-react on ^0.511.0; bump chalk to 5 and image-size to 2. * fix(deps): complete the OTel pin, restore SDK error detail, revert email list Follow-ups from an independent audit of the previous commit. - Pin @opentelemetry/sdk-node and the three otlp-http exporters to exact 0.217.0. Pinning only their four dependents was self-reversing: sdk-node 0.219.0 requires core 2.8.0 exactly, so the next update would have silently rebuilt the split this PR removes. - Declare @opentelemetry/core (2.7.1). It is imported by lib/copilot/request/go/propagation.ts but resolved only by hoisting, and it is the OTel package with the most version churn in the tree. - Pin @radix-ui/react-dismissable-layer to exact 1.1.13 in @sim/emcn. All five transitive parents pin it exactly; a caret would fork a second copy on 1.1.14, which is the duplicate-context bug the declaration prevents. - Surface error.cause in simstudio-ts-sdk. Native fetch reports network failures as a bare "fetch failed" and puts the reason on cause, so every DNS/TLS/refused error was reaching callers with no diagnostic content. - Revert free-email-domains to 1.2.25. Upstream now merges the free-domain list with two disposable-email blocklists, so 1.9.70 classifies real organization domains as free — UK charities, some companies and universities, and the JP/KR ISP domains APAC SMBs use for business mail. The demo form blocks submission on that check, so a false positive costs the booking entirely. Worth doing deliberately, not inside a deps change. - Lower packages/cli engines to >=18. chalk 5 and commander 11 both accept >=16 and the source uses no Node 20 API, so >=20 only produced EBADENGINE for Node 18 users. --------- Co-authored-by: Waleed Latif <waleed@simstudio.ai>	2026-07-27 19:16:32 -07:00
Waleed	6d48444525	fix(docs): render native block icons instead of the two-letter fallback (#5981) ... * fix(docs): render native block icons instead of the two-letter fallback The Table and Logs pages (and every other native resource block) showed a two-letter text fallback because the generated icon map never contained them. Four separate causes in scripts/generate-docs.ts: - The icon-map allowlist had drifted behind NATIVE_RESOURCE_BLOCK_TYPES, the set the docs writer uses. Key the exception off that set so the map cannot fall behind the pages that consume it. - extractIconNameFromContent only matched identifiers ending in `Icon`, so Logs (`icon: Library`) resolved to nothing. Match any identifier, excluding bare JS literals. - The map imported everything from `@/components/icons`, so an icon sourced from `@sim/emcn/icons` could not resolve. Imports are now grouped by the module each icon is actually imported from. - Trigger-only pages (slack_app, twilio) and hand-written pages (a2a) had no entry at all. Seed provider icons from the trigger definitions. Also fixes three regen bugs found while verifying the output: - A comment reading "this becomes `hideFromToolbar: true`" in slack.ts was matched as the property itself, so a clean regen dropped Slack from the integrations catalog and reduced slack.mdx to a 29-line stub. Property probes now run against comment-stripped source. - 16 hand-written *-service-account guides were unregistered, so the stale-doc cleanup deleted them on every regen. Registered them, and cleanup now refuses to delete any page holding MANUAL-CONTENT (this also restores the intros on file.mdx and twilio.mdx). - Trigger outputs referenced as a constant (`outputs: SLACK_TRIGGER_OUTPUTS`) resolved to nothing, dropping whole Output tables. Constants and sibling modules now resolve, which also restores 319 lines on clickup.mdx. Removes the language selector from the docs navbar. Regenerated docs are included; remaining content deltas are tool-definition drift since the last regen. * improvement(docs): drop the preview-gated slack_app page, document managed_agent - slack_oauth is reachable only through the preview-gated slack_v2 block, so documenting it published an unreleased surface under its own slack_app page. Triggers whose every hosting block sets `preview: true` are now excluded from the docs and the icon map. Triggers no block claims are untouched, so standalone webhook providers keep their pages. - Adds the MANUAL-CONTENT intro to managed_agent.mdx, matching the other integration pages. Verified it survives a regen. * fix(docs): stop truncating quoted descriptions, tighten the cleanup guard Review findings from round 1. - parseSubBlockObject read string properties with a single `['"]…[^'"]+…['"]` character class, which ends the match at the first quote of either kind. Any description holding an apostrophe inside a double-quoted string was cut mid-word ("Your app", "Found in your Zoom app"). Matches the opening quote to its own closing quote now, reusing the alternation the tool-description extractor already used. Restores full text across calendly, gmail, google_sheets, hubspot, intercom, whatsapp, and zoom. - The stale-doc cleanup guard tested for a bare `MANUAL-CONTENT-START` substring, so a stray or unterminated marker would pin a stale page that has nothing recoverable. It now gates on what extractManualContent actually returns.	2026-07-27 14:07:25 -07:00
Bill LeoutsakosandBill Leoutsakos	bd61603701	feat(tiktok): unhide integration (#5978) ... * feat: unhide TikTok integration * test: remove TikTok visibility assertion --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-07-27 11:47:15 -07:00
Vikhyath Mondreti	8a2ae25d78	improvement(agent-streaming): add way to opt in for workflow executions (#5956)	2026-07-24 20:10:06 -07:00
Waleed	919a98d00f	refactor(settings): fold verified domains into SSO, move group-detail state to nuqs, design-system cleanup (#5950) ... * refactor(settings): fold verified domains into SSO and use shared primitives Verified domains only gates SSO, so managing it on a separate page meant discovering the requirement after filling out the whole IdP form and then navigating away mid-setup. Move it into the SSO page as a section above the provider config and drop the standalone page, its nav entry, and both route branches. Align the surfaces with the shared settings primitives rather than bespoke chrome, matching whitelabeling/custom-blocks/access-control: - SSO's local FormField (muted labels) is replaced by the shared SettingRow, so its fields read like every other settings page. SettingRow gains optional `optional` and `error` props to absorb what FormField did — additive, so existing consumers are untouched. - The domains section is built from SettingsSection, SettingRow, SettingsResourceRow, and SettingsEmptyState instead of hand-rolled cards. Also drop the redundant Upload/Change buttons in whitelabeling: the logo and wordmark thumbnails were already clickable, so the button was a second control for the same action. Remove still appears once an image is set. * fix(settings): move group-detail view state to nuqs and clean up design-system drift Access control's group detail kept its tab, three search boxes, and three status filters in useState, so a `?group-id=` link always landed on General and a filter was lost on reload — the parent already puts the group id in the URL. The three tabs never render together, so search and status share one param each rather than carrying three mutually-exclusive keys, and switching tabs resets both. Closing the detail clears all three alongside group-id in one batched write, so nothing lingers on the list URL. Design-system fixes from a cleanup pass over the surfaces this branch touched: - Restore accessible names lost when the whitelabeling Upload buttons were removed. The thumbnail is now the only click target, and it contained just an icon, so it announced as an unlabeled button; the icon-only Remove had the same problem. Both now carry aria-labels reflecting their state. - Use Chip, not the legacy Button, for the domain actions — Button is ~26px against the 30px ChipInput beside it, so "Add domain" sat visibly short. - SettingRow now uses the emcn Info component; a bare svg as Tooltip.Trigger was neither focusable nor nameable. It also stops re-specifying Label's own default styling. - Use the new SettingRow error prop for the group name instead of a hand-rolled error paragraph, which is what the prop was added for. - Hoist the block-category lookup out of a sort comparator, size-* over h/w, name the staleTime constants the rules require, and import RowActionsMenu from its barrel. * fix(settings): alias the old /settings/domains path to SSO Folding verified domains into the SSO page dropped /settings/domains, so bookmarks and shared links 404'd instead of landing where domains now live. Both alias maps already exist for exactly this (organization/'members', subscription/'billing'); add domains -> sso to each. * chore(settings): adopt ChipCopyInput, named staleTime constants, and a11y labels * fix(settings): reset group detail params on open and drop issuer mono styling	2026-07-24 19:11:34 -07:00
Vikhyath Mondreti	fe184d3695	improvement(whatsapp): validate + improve integration skill for file inputs/outputs (#5942) ... * improvement(whatsapp): validate + improve integration skill for file inputs/outputs * fix lint * add whatsapp subblock migration	2026-07-24 16:14:27 -07:00
Vikhyath Mondreti	17d77795b4	feat(providers): prompt caching capability + usage-based cache pricing (#5922) ... * improvement(providers): validation pass, and stream tool loop improvements * remove deploy options correctly * fix * feat(providers): prompt caching capability and usage-based cache pricing Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic. * fix(providers): route OpenAI and Gemini block cost through cache-aware pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained. * fix(streaming): redact tool payloads on selected outputs in public chat Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole. * refactor(providers): drop unreachable billing fallbacks Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total.	2026-07-24 15:46:11 -07:00
Waleed	5a8d21904d	fix(sso): surface DNS verification failures and the provider auto-append gotcha (#5931) ... * fix(sso): surface DNS verification failures and the provider auto-append gotcha Post-merge audit follow-ups for verified domains (#5909): - The host field handed admins the FQDN `_sim-challenge.acme.com`. GoDaddy, Namecheap, Hover and most cPanel panels append the zone to whatever is typed, yielding `_sim-challenge.acme.com.acme.com` — the record looks right in their panel but never verifies, and our 422 tells them to wait 48 hours. Add a hint under the field (and a docs callout) telling those admins to enter just the label. - DNS failures were logged at debug, but production log level is ERROR, so a blocked-egress or SERVFAIL condition was invisible to us and misreported to the admin as "record not found yet". Log infrastructure-class failures at warn with the DNS error code; keep the genuinely-absent codes at debug. - Trim the joined TXT value before comparing: several DNS panels pad the stored string, which otherwise fails an exact match forever. - Lower the resolver to 2s/1 try. c-ares multiplies timeout across servers and retries by ~7x, so the previous 5s/2-try config could block a verify request for ~35s when resolvers are unreachable. - Cover `checkDomainTxtRecord` — the function that decides whether the gate opens had no tests. Adds exact match, chunk-joined value, match among unrelated records, padded value, near-miss, another org's token, absent record, infrastructure failure, and empty-response cases. * fix(sso): log DNS infrastructure failures at error so prod actually surfaces them Production's default minimum log level is ERROR, so the warn introduced in the previous commit was still filtered out — the fault stayed invisible exactly as before. A resolver failure that is not 'record absent' is a genuine infrastructure error, so ERROR is both the visible and the honest severity. * fix(sso): state the zone-removal rule instead of a wrong subdomain hint The hint computed the bare label as the first segment of the challenge host, so for a subdomain like eng.acme.com it advised entering `_sim-challenge` when the host relative to the acme.com zone is `_sim-challenge.eng` — following it would publish the record on the wrong name and verification would never succeed, the exact failure the hint exists to prevent. Deriving the real zone needs the Public Suffix List (acme.co.uk defeats naive label-stripping), so state the rule instead: enter the host with the trailing zone removed. Docs show both the apex and the subdomain form.	2026-07-24 11:32:27 -07:00
Waleed	70313cd2ad	feat(providers): add Claude Opus 5 model (#5925) ... * feat(providers): add Claude Opus 5 model * fix(providers): route Opus 5 through adaptive thinking Opus 5 supports only adaptive thinking (no extended thinking / budget_tokens), same as Opus 4.8/4.7. Add opus-5 to supportsAdaptiveThinking so thinking requests use thinking.type: adaptive instead of falling through to budget_tokens extended thinking, which Opus 4.7+ rejects with a 400. * docs(agent): regenerate agent-stream docs for Opus 5	2026-07-24 10:33:47 -07:00
Waleed	513292f17b	feat(sso): DNS domain verification gating org SSO registration (#5909) ... * feat(sso): DNS domain verification gating org SSO registration Add org-scoped domain ownership verification (DNS TXT challenge) as the security precondition for configuring SSO. Closes the first-come domain-claim vuln where any org could wire another company's domain to its own IdP. - New sso_domain table + migration 0266; existing org SSO domains are grandfathered as verified so live tenants are unaffected - Verified-domains settings UI (enterprise-gated) with add/verify/remove - Register route now requires a verified domain for org-scoped registration; personal SSO and already-grandfathered domains are unaffected - Self-host register script writes the verified sso_domain row directly, so script-driven registration stays backwards compatible * fix(sso): harden domain verification against concurrency + fix CI lint Addresses review findings on state invariants under concurrent/failed writes: - Add unique index on (organization_id, domain) so concurrent claims can't create duplicate pending rows; POST re-reads and stays idempotent on conflict - Verify flips the row only if it's still the exact pending challenge checked (guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique index violation to 409 instead of an unhandled 500 - Wrap the self-host script's provider write + verified-domain upsert in a transaction so a failed ownership write can't leave a provider committed - Format 0266 snapshot/journal with biome (fixes @sim/db lint:check) * fix(sso): re-check domain verification before provider write (TOCTOU) The register gate checked the verified sso_domain row only at handler entry, then ran OIDC discovery before writing the provider. A verified row removed during that window could still complete registration. Extract the check into a closure and call it both as an entry fast-fail and authoritatively right before registerSSOProvider, alongside the existing domain-conflict re-check. * fix(sso): stop rotating verification token on idempotent re-add Re-adding a pending domain rotated its verification token, which invalidated a TXT record the admin may have already published and — under two concurrent re-adds — could return a token the racing write had already superseded, so the admin's DNS record would never verify. Return the existing row unchanged instead; the pending token is always shown in the UI, so it is never lost. * fix(sso): close register TOCTOU with compensating delete + harden edges Audit-driven hardening: - Close the residual register TOCTOU: registerSSOProvider is create-only (throws if the providerId exists), so a compensating delete after the write is provably safe — it can only remove the just-created row. If verification was revoked during the write, roll the provider back and 403. - Verify is now idempotent under concurrency: a same-org row already flipped to verified by a racing request returns 200, not a confusing 409. - Grandfather backfill + self-host script now match normalizeSSODomain's dominant transforms (lower + trim + strip leading wildcard) so a non-canonical legacy domain can't miss the runtime gate's lookup. Prod backfill result is unchanged. - Cleanup: drop dead default export, align card radius to sibling convention. * fix(sso): redact domain tokens from non-admins + fix script stale-update Round-5 review findings: - GET /domains redacted the pending TXT verification token (a management secret) to any org member. Now only owner/admins read it; members see the list and status without it. Non-Enterprise orgs get an empty list (entitlement flag only), never the domains/tokens. - Self-host script decided update-vs-insert from a read taken OUTSIDE the transaction; a provider deleted mid-flight made the UPDATE match zero rows silently while the verified-domain upsert still committed (orphaned domain). The decision now happens inside the transaction from the UPDATE's row count. * docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains Verified domains currently only gate SSO configuration. Remove the forward-looking references to enforcing SSO and auto-joining members (deferred to a later release) from the docs, settings copy, nav description, and schema comment so we don't promise unshipped features. * fix(sso): guard rollback to new providers only + Enterprise-gate domain removal Round-6 review findings: - The compensating provider rollback now only fires when the provider did not exist before this request (providerExistedBefore). registerSSOProvider is create-only today so reaching the rollback already implies a fresh create, but this makes the safety local and future-proof: if Better Auth ever allowed updating an existing provider, a revoked-verification rollback must not delete that pre-existing row. - DELETE /domains now requires an Enterprise plan like add/list/verify, so all domain mutations share one entitlement (the UI already hides removal from non-Enterprise orgs). Adds a delete-route test. * fix(sso): roll back the SSO provider by row id, not logical keys The compensating rollback deleted by (providerId, orgId). providerId is unique, so if this request's row were deleted and recreated by a concurrent registration in the narrow window before the rollback, the logical-key delete would remove that other request's provider. Delete by the primary-key id registerSSOProvider returns instead, so only the exact row this request created is ever removed. * chore(sso): final-review polish — trim script read, unify copy, doc migration edge Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the new logic): - Self-host script: narrow the pre-transaction existence read to select({ id }) instead of SELECT * (it only feeds a log line now). - Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere 409 wording ("is already verified by another organization") across routes. - p-3 shorthand on the domain row card. - Document the migration's rare two-orgs-share-a-domain grandfather behavior (login unaffected; validated no such duplicates in prod). * fix(sso): apply attribute mapping + make SSO edit work; drop dead guard Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW, script-registered with the default mapping, so neither change affects it): - Attribute mapping was passed at the top level of the register payload, which Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it so custom mappings actually apply. (Default mapping is unchanged, so existing logins are unaffected.) - Editing an SSO provider was broken: registerSSOProvider is create-only and threw on the existing providerId → generic 500. Route now detects a provider the caller already owns and updates it via Better Auth's updateSSOProvider, and surfaces Better Auth's own error status/message instead of a blanket 500. Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes by the created row's primary-key id and register is create-only) and the earlier final-review polish (script read, unified copy, migration edge note). Smoke-test SSO login + edit on staging before merge (auth-path change). * fix(sso): require null org on personal-mode provider lookups (gate bypass) The personal branch of both provider-ownership lookups keyed on (providerId, userId) without requiring organizationId IS NULL. Because org providers store userId = their creator and providerId is globally unique, an org admin could send a personal-mode request (no orgId) — which skips the membership check and the domain-verification gate — yet still match, and then via the new update path move, their org's provider to an unverified domain. Add isNull(organizationId) to the personal branch of both clauses so it can only match a genuinely personal provider, matching the route's own isOwnedByCaller. Found by an adversarial review of the update path added in 394bda9f7. * fix(sso): script updates the observed provider by id, not providerId Inside the registration transaction the script updated WHERE providerId — the logical key. If the observed provider was deregistered and a replacement created with the same providerId before the transaction ran, that update would clobber the replacement's config and ownership. Update the specific observed row by its primary-key id instead; if it's gone we insert, which fails cleanly on the providerId unique constraint rather than overwriting the replacement. * fix(sso): script upserts provider via delete-then-insert (no unique constraint) sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate duplicates, so the previous "update by id, else insert" could create a duplicate provider when the observed row was deregistered and replaced before the transaction — the fallback insert would succeed. Delete every row for the providerId then insert exactly one, inside the transaction, so the providerId ends up as exactly this config atomically. Linked accounts key on the providerId string (not the row id), so existing logins are unaffected. * fix(sso): guard compensating-delete row id so rollback can't silently no-op * chore(sso): regenerate migration as 0268 after merging staging Staging landed migrations 0266/0267, colliding with our 0266. Removed our migration, merged staging, and regenerated cleanly with drizzle-kit as 0268_sso_domain_verification (identical sso_domain table + indexes), then re-appended the grandfather backfill. api-validation baseline reconciled to 973 (staging 970 + our 3 domain routes). Also make the register-route test's registerSSOProvider mock return an id so the guarded compensating delete runs. * refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate The self-host script canonicalized SSO domains with a minimal inline transform (lower+trim+wildcard) that diverged from the app's full normalizeSSODomain (protocol, port, path, trailing dot, email local part) — equivalent spellings could store a different ownership key than the runtime gate looks up. Move normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register route, the domain-claim route, and the script all use the identical canonicalizer. The script now skips the verified-domain record when SSO_DOMAIN isn't a valid registrable domain instead of storing a malformed key.	2026-07-24 00:34:16 -07:00
Theodore Li	444c415a0b	improvement(data-retention): docs for overrides + PII redaction, fix wedged saves (#5905) ... * fix(data-retention): clamp sub-day retention values so saves aren't wedged A stored value under 12 hours rounded to '0' days on load and was re-sent as 0, which the contract rejects (min 24) — blocking every save on the page, including unrelated fields. Clamp hours->days into the contract's range on read, and throw instead of emitting 0/NaN on write. * improvement(docs): rewrite data retention for workspace overrides + PII redaction - Document PII redaction (Logs / Workflow input / Block outputs stages, entity types, languages, custom regex patterns) - Replace the stale 'no per-workspace overrides' section with the retention-policies list and override inheritance - Correct log retention (also covers background job logs) and soft deletion (adds Chat conversations, KB documents) - Add PII + override screenshots, refresh the main one	2026-07-24 02:08:12 -04:00
Vikhyath Mondreti	7120cdc901	fix(providers): final regenerated stream must not re-call tools (empty chat answers) (#5915) ... * fix(providers): stop the final regenerated stream from re-calling tools and clobbering the answer After the silent tool loop settles, OpenAI Responses and Gemini re-issue a streaming request purely to stream the answer as prose — but with tools still attached and auto tool choice, a reasoning model can re-decide to call a tool there. Streamed calls are never executed on this path, so the run ends with a dead function call, an empty streamed answer, and the stream callback clobbering the tool loop's settled text with '' (deployed chat rendered {"content": ""}). Force tool_choice 'none' / functionCallingConfig NONE on the regeneration and keep the tool loop's settled answer whenever the stream ends without text. * feat(openai): settled tool chips on the regenerated answer stream The silent Responses tool loop has no live stream while tools run, so opted-in consumers saw no tool chips at all for OpenAI. The loop now records each executed call and prepends settled tool_call_start/end pairs (name + status only) to the agent-events stream ahead of the regenerated answer. Runs without a sink never see these events, so legacy output is unchanged. * fix(providers): apply the regeneration guard fleet-wide Audit of every silent tool loop for the same race fixed for OpenAI/Gemini (final regenerated stream re-calls a tool that is never executed, ending with an empty answer that clobbers the settled text): - anthropic (both implementations): tool_choice {type:'none'} on the regeneration (tools must stay — history carries tool_use blocks) + keep the settled answer when the stream ends without text - groq: was re-applying the ORIGINAL tool_choice, so forced-tool runs re-forced the tool on the regeneration — guaranteed dead call; now 'none' + fallback - deepseek, mistral, cerebras, azure-openai (legacy chat path), openrouter, xai: 'auto' -> 'none' + fallback - bedrock: fallback only — Bedrock's ToolChoice has no 'none' and toolConfig is required when history carries toolUse blocks Already guarded (no change): meta, sakana, nvidia, vllm, litellm, baseten, together, fireworks, kimi, zai, ollama.	2026-07-23 20:20:25 -07:00
3 people	6dcc65be89	feat(skills): add skill editors (#5705) ... * feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-23 19:52:00 -07:00
4 people	d24bc7eccb	feat(agent-stream): thinking and tool streaming (#5671) ... * feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-23 19:39:03 -07:00
Waleed	d48722a04e	fix(helm): correct chart docs, examples, and dead config across the board (#5907) ... * fix(helm): correct chart docs, examples, and dead config across the board Audit-driven accuracy pass over the chart's entire documentation surface, verified by rendering every example against the templates: - migrations run as an init container on the app pod, not a Job — fix the README component list, troubleshooting commands, and sim-helm skill refs; drop the dead migrations-job NetworkPolicy ingress rule - referenced-but-never-created resources: document the GKE ManagedCertificate creation (values-gcp), comment out the key-file Secret mount that stuck all pods in ContainerCreating (values-gcp), enable certManager for the postgres TLS issuerRef (values-production), add the cert-manager cluster-issuer annotation nginx needs (values-azure) - values-external-db: networkPolicy.egress is a list, not a map (the map rendered an invalid manifest); fill schema-failing placeholder host/username - realtime >1 replica requires REDIS_URL (Socket.IO Redis adapter) — default examples to 1 replica with the scaling note, and warn where autoscaling HPAs override replicaCount - pod anti-affinity selectors matched nothing (simstudio vs sim name label) - kubernetes.mdx: install commands were missing required CRON_SECRET and postgresql password (failed at template time), wrong deployment name in port-forward, stale version requirements, unsupported key-remapping claim - remove unimplemented app.secrets.existingSecret.keys from values + schema; fix README PDB default, cronjob list, /metrics caveat, NOTES secret count, Azure-only StorageClass in generic examples, dead SOCKET_SERVER_URL and GOOGLE_CLOUD_* env, ESO apiVersion mismatch, and skill-reference drift - bump chart to 1.0.1 * fix(helm): review round 1 — scoped example egress, in-tab secret note, copilot Job wording - external-db example egress scopes to a placeholder database CIDR instead of to: [] (which allowed every destination on 5432, defeating the isolation the example teaches) - kubernetes.mdx cloud tabs state explicitly that they reuse the variables generated in the Installation block - Copilot migrations really do run as a Helm-hook Job — restore Job wording there (only the app migrations are an init container) * fix(helm): template-sweep fixes — telemetry validity, ESO rollout checksums, dead passwordKey knob - telemetry: memory_limiter gets the required check_interval (collector failed startup validation whenever telemetry.enabled=true); the jaeger exporter was removed from collector-contrib in v0.86 — export to Jaeger via its native OTLP endpoint instead (otlp/jaeger, default port 4317) - app/realtime rollout checksums now hash the ExternalSecret manifest too, mirroring the copilot pattern — with ESO enabled the inline Secret renders empty, so remoteRefs changes never rolled the pods - remove the unimplemented existingSecret.passwordKey knob (values, schema, README, dead helpers): nothing consumed it, and a non-default value silently produced a DATABASE_URL with an unexpandable placeholder; secrets must use the standard POSTGRES_PASSWORD / EXTERNAL_DB_PASSWORD keys - drop the orphaned sim.migrations.labels helper (its only consumer was the dead NetworkPolicy rule removed earlier) - helm test pod image resolves through sim.image so global.imageRegistry mirroring applies; NetworkPolicy realtime-ingress comment reflects actual traffic direction; smoke unittest suite loads the newly referenced external-secret template * chore(helm): bump chart to 1.1.0 with upgrade notes Removing (inert) documented values keys and changing the rollout-checksum inputs is a values-surface change — per SemVer chart conventions that is more than a patch. Adds an Upgrading section documenting the one-time pod roll, the removed no-op keys, and the Jaeger-over-OTLP change. * fix(helm): review round — external-db NP opt-in with real-CIDR-first flow, prod Jaeger OTLP endpoint - external-db example ships networkPolicy disabled so a verbatim install always reaches the database; the scoped egress rule stays as the documented opt-in (set your CIDR first, then enable) - values-production still pointed telemetry.jaeger at the legacy 14250 collector port — now Jaeger's OTLP gRPC endpoint to match the otlp/jaeger exporter * feat(helm): autoscaling.realtime.enabled toggle so examples can scale the app without unsafe realtime replicas Cursor correctly flagged that comment-level warnings didn't stop a verbatim production/external-db install from running the realtime HPA at minReplicas 2 without REDIS_URL (silent cross-pod event loss). Adds an opt-out toggle (default true — existing deployments unchanged): the realtime HPA renders only when autoscaling.realtime.enabled, and the realtime Deployment keeps spec.replicas under its control when the HPA is excluded. The three autoscaling examples set it false with the Redis rationale; README and upgrade notes document the toggle. * fix(helm): review round — whitelabeled realtime HPA opt-out, external-db isolation on with required CIDR in install flow - values-whitelabeled now actually sets autoscaling.realtime.enabled: false (the earlier batch aborted before reaching this file — Cursor caught it) - external-db keeps networkPolicy enabled (no isolation regression); the DB egress CIDR is marked REQUIRED and wired into both documented install commands via --set, so the copy-paste flow sets the real subnet in the same breath as the DB host * fix(helm): use a syntactically valid example CIDR in the external-db install commands An unreplaced <YOUR_DB_CIDR> literal fails Kubernetes CIDR validation and aborts the install; every sibling placeholder in the same command (host, username) installs fine and simply doesn't connect until replaced. The CIDR now behaves the same way: valid example value (10.20.0.0/24), explicitly marked as the operator's database subnet in both commands and the values comment. * fix(helm): remove text after continuation backslashes in external-db install commands A trailing comment after the line-continuation backslash (and equally a comment line spliced mid-command) breaks the shell command when copied — the remaining --set overrides run as separate commands and required-value validation fails. Both documented commands now reconstruct to bash-clean multi-line invocations (verified with bash -n); the CIDR guidance lives in the networkPolicy section comment. * fix(docs): cloud-tab installs are alternatives via helm upgrade --install Following Installation and then a cloud tab ran helm install twice for the same release and failed on the second. The tabs now state they replace the generic install and use helm upgrade --install, which is idempotent and also converts an existing generic install to the cloud values. * fix(docs): honest conversion caveat for cloud-tab upgrades over an existing install The cloud values rename the bundled Postgres database to simstudio, which Postgres only applies at first initialization — an in-place conversion of a generic install would point DATABASE_URL at a nonexistent database. Document the two safe paths: keep the original name via --set, or uninstall + delete PVCs and install fresh. * fix(helm): external-db header install command declares its secrets and includes CRON_SECRET The primary documented command failed the chart's required-value validation (missing app.env.CRON_SECRET with cronjobs default-on) and referenced an undeclared DB_PASSWORD. It now shows the export lines for every variable it uses and sets CRON_SECRET; both commands verified with bash -n. * chore(helm): restore values.schema.json formatting — surgical deletions only The earlier programmatic edit reformatted the whole file (~390 lines of whitespace churn hiding the 12 real deleted lines). Re-applied the removal of the dead keys/passwordKey properties as text-level deletions preserving the original style. * fix(helm+docs): explicit secret exports in external-db header; conversion must reuse original secrets - all five export lines are written out (three were only named in a trailing comment, so a verbatim copy passed empty required values) - the cloud-conversion caveat now leads with reusing the original secret values (helm get values) — a regenerated ENCRYPTION_KEY makes previously encrypted credentials undecryptable	2026-07-23 18:28:11 -07:00
Vikhyath Mondreti	e737901b0b	chore(blocks): rename webhook block (#5900)	2026-07-23 14:14:16 -07:00
Waleed	78fb2c0679	chore(deps): bump next to 16.2.11 to clear security advisories (#5890) ... * chore(deps): bump next to 16.2.11 to clear security advisories Patches SSRF, cache confusion, DoS, and middleware-bypass advisories (GHSA-89xv-2m56-2m9x et al.) affecting next < 16.2.11 across apps/sim, apps/docs, and packages/emcn. Excludes next/@next/env from the minimum-release-age gate until the 7-day window elapses on 2026-07-28. * chore(deps): drop aged-out typescript entries from release-age excludes typescript and @typescript/typescript6 passed the 7-day minimum-release-age gate (aged out 2026-07-15 and 2026-07-13), so their exclusions are no longer needed. Keeps @typescript/native-preview (permanent nightly builds) and the Pi packages (age out 2026-07-24).	2026-07-23 10:47:07 -07:00
Waleed	874e742a47	fix(connectors): purge archived/deleted source items in KB connectors (#5880) ... * fix(confluence): exclude archived pages from KB connector listings so reconciliation purges them * fix(connectors): purge archived/deleted source items across seven more KB connectors The sync engine only purges a knowledge-base document when its source item is absent from a full-sync listing, so any connector that keeps listing archived/trashed/canceled items never drops them. An audit of all 51 connectors found seven with this bug: - asana: list only non-archived projects (the API returns both when `archived` is omitted), so tasks under archived projects stop being re-listed - google-sheets: skip a spreadsheet Drive reports as trashed, which stays readable by id for 30 days before the Sheets call starts 404ing - incidentio: exclude canceled incidents by default (cancelling is incident.io's documented stand-in for deletion), with an explicit opt-in to sync them - outlook: exclude Deleted Items from the all-mail listing, which Graph otherwise includes - servicenow: drop retired knowledge articles, which the Table API returns with no implicit state filter - webflow: drop archived CMS items, which the staged items endpoint always returns and offers no way to filter - youtube: drop playlist entries whose video was deleted or made private, which the API keeps returning as placeholder items Every exclusion keys off an explicit non-current signal and fails open on a missing field or a failed metadata read, since wrongly excluding a live item would hard-delete it. Explicit user filter selections are still honoured verbatim; the new defaults apply only when nothing is configured. Also flag truncated listings as capped in asana, outlook, and servicenow. All three silently cut a listing short at their configured item cap without setting `syncContext.listingCapped`, so reconciliation read the untraversed tail as deleted at the source and hard-deleted it. * fix(asana): honour the pinned-project exception on the task rehydrate path listDocuments deliberately keeps syncing a project the user pinned via the `project` config field even once it is archived, but getDocument ignored sourceConfig and applied the all-parents-archived exclusion unconditionally. For a pinned archived project the listing kept emitting its tasks while every hydration returned null, so new tasks were dropped as empty and already-indexed ones were frozen at their last content. isTaskUnderActiveProject now takes the pinned project gid and keeps any task reachable through it, matching the listing exactly. The unpinned path is unchanged and still fails open on missing/non-boolean archived values. * fix(connectors): key removal on explicit source signals, never on absence Follow-up to the connector purge fixes, from an independent audit. YouTube inferred deletion from absence: a playlist entry whose id was missing from a `videos.list` response was dropped, so a well-formed 200 that returned 49 of 50 requested ids hard-deleted the 50th. Playlist items instead carry a documented `status.privacyStatus`, available as a free part on a call the connector already makes, so the extra `videos.list` request is gone along with its quota-failure and pagination-wedge risks. An item is now excluded only on an explicit `private`; missing, empty, or unrecognized values keep it. ServiceNow read every record through a guard requiring a string `sys_id`, but the listing requests `sysparm_display_value=all`, under which every field — `sys_id` included — comes back as `{display_value, value}`. The guard rejected every record, so the retired-article filter was unreachable and the sys_id object would have leaked into `externalId` and `title` had it not been. Records are now read through the existing `rawValue` normalizer, which accepts both wire shapes, and the fixtures use the shape the API actually returns. Also: resolve the ServiceNow cap ambiguity with `X-Total-Count` so a table that ends exactly on a page boundary is not read as truncated; stop the Google Sheets comment claiming a purge the engine's zero-document guard prevents; and assert the Outlook junk-mail invariant instead of comparing a constant to itself. Document the behavior change: content archived, retired, or trashed at the source is now removed from the knowledge base, and restoring it re-ingests it.	2026-07-22 22:34:48 -07:00
Waleed	2b5a92a3c8	feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862) ... * feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name)	2026-07-22 16:42:59 -07:00
WaleedandMarcus Chandra	8cce661a37	feat(api): proxyUrl for residential/custom proxy egress on the API block (#5867) ... * feat(api): add proxyUrl for residential/custom proxy egress on the API block The HTTP/API block egresses from the app runtime's fixed datacenter IPs via secureFetchWithPinnedIP, so targets behind Cloudflare/WAF that block datacenter IPs (e.g. state .gov license portals) return 403/429 even when the identical request works from a browser. There was no way to route a request through a residential/custom proxy. Add an optional `proxyUrl` field (Advanced) to the API block. When set, the request routes through the given http:// proxy so it egresses from that proxy's IP. Security: - validateAndPinProxyUrl resolves the proxy host's DNS and blocks private/reserved/loopback IPs (same SSRF guard as target URLs), then pins the connection by rewriting the host to the resolved IP (creds/port preserved), closing the DNS-rebinding window. - Restricted to the http: proxy scheme (https/socks rejected) so host pinning is safe without breaking TLS-to-proxy SNI. - Target-IP pinning is intentionally bypassed when a proxy is active (the proxy resolves the target); target URL validation still runs. Threaded block field -> http tool param -> formatRequestParams -> executeToolRequest (validate + pin) -> secureFetchWithPinnedIP, which swaps its pinned Node agent for HttpsProxyAgent/HttpProxyAgent (keyed off target protocol) when proxyUrl is set. * docs(api): document the Proxy URL advanced field and steer proxy credentials to env vars * fix(api): reject loopback/private proxy hosts unconditionally, closing the self-hosted rebinding gap * chore(api): tighten proxy-path inline comments --------- Co-authored-by: Marcus Chandra <mzxchandra@gmail.com>	2026-07-22 16:24:31 -07:00
Theodore Li	e3f9deb65e	fix(slack): allow empty status to clear the assistant status indicator (#5827)	2026-07-21 20:25:04 -04:00
4 people	2c4d2091b8	feat(pi): add code review mode (#5577) ... * feat(pi): add Cloud Code Review mode and rename Cloud to Cloud PR Introduce a third Pi mode that reviews an existing GitHub PR in an E2B sandbox and posts a structured review with optional inline comments. Keep the stored cloud id for backward compatibility, extend github_create_pr_review for inline comments, and harden review submission against stale SHAs and invalid comment payloads. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(pi): cleanup code * address comments * address mor * address comments * update --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-20 16:25:40 -07:00
Waleed	5eafa86992	feat(email): native Gmail API mail provider for GCP self-hosting (#5736) ... * feat(email): native Gmail API mail provider for GCP self-hosting Adds Gmail as a fifth transactional mail provider (Resend → SES → SMTP → ACS → Gmail). GCP has no first-party SES/ACS equivalent, so the native Google path is the Gmail API: a service account with domain-wide delegation impersonates a Workspace sender (GMAIL_SENDER) and posts the raw RFC 822 message (built via nodemailer's MailComposer — full parity incl. attachments, replyTo, unsubscribe headers) to the media-upload messages.send endpoint. The Workspace SMTP relay alternative is documented against the existing SMTP provider. * fix(email): review round 1 + audit hardening for Gmail provider - a 2xx from Gmail with an empty/malformed body no longer surfaces as a send failure (the mailer's fallback chain would deliver the same email twice); covered by a regression test - normalize bare-LF line endings in html/text bodies to CRLF (RFC 822) before composing the raw message - add multi-recipient and text-only test cases	2026-07-17 12:55:21 -07:00
Waleed	f6bb8e6d3e	feat(storage): native Google Cloud Storage support for self-hosting (#5728) ... * feat(storage): native Google Cloud Storage support for self-hosting Adds GCS as a third object-storage backend with full parity with S3 and Azure Blob: uploads, streaming downloads, deletes, head, V4 signed URLs (single + batch), and browser/server multipart uploads via the GCS XML API. Selection precedence is Azure Blob > S3 > GCS > local disk. - new provider client at lib/uploads/providers/gcs (cached singleton, ADC/Workload Identity or inline GCS_CREDENTIALS_JSON auth) - per-context GCS_*_BUCKET_NAME config wired through getStorageConfig - shared getServeStoragePrefix() replaces hardcoded blob/s3 serve paths - docs (object-storage, environment-variables), .env.example, helm values.yaml + values-gcp.yaml storage section * fix(storage): review round 1 — GCS per-context bucket fallback + ETag quote normalization - getGcsConfig falls back to the general bucket for every context (GCS bucket names are globally unique, so the S3-style sim-execution-files literal default would point at an unowned bucket; empty per-context buckets previously made uploads and downloads disagree) - completeGcsMultipartUpload restores quotes on ETags stripped by the shared browser upload client before building the completion XML - docs/.env.example/helm updated for the fallback behavior * fix(storage): review round 2 — route chat authz and execution-URL detection through getStorageConfig - getChatStorageConfig delegates to getStorageConfig('chat') (identical for S3/Azure, picks up the GCS general-bucket fallback instead of reading the raw chat config and rejecting valid chat files) - parse route resolves the execution bucket via getStorageConfig('execution') for all providers, so GCS execution files in the fallback bucket are still recognized as our own objects * fix(storage): validation pass — gcs serve-prefix parity in key parsers + CORS doc fix - extractStorageKey, extractFilename, and extractEmbeddedFileRef now strip the gcs/ serve prefix like s3/ and blob/, so direct-uploaded files on GCS parse, delete, download, and embed correctly (previously only the serve route knew the prefix) - file-download storageProvider union includes 'gcs' - completeGcsMultipartUpload defensively rejects a 200 response carrying an XML error document - docs: CORS example lists concrete x-goog-meta-* header names (GCS matches responseHeader entries exactly; wildcards are only supported for origin)	2026-07-17 00:15:14 -07:00
Vikhyath Mondreti	86e6e1d26c	feat(forking): excluded workflows (#5727) ... * feat(forking): excluded workflows * improve sync preview	2026-07-16 20:23:26 -07:00
Theodore Li	1da346b703	feat(triggers): service-account credentials in the hubspot trigger + token-SA name-collision 409 (#5693)	2026-07-16 19:11:57 -04:00