mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
e4cedcb91f8ea705ccf8a003fde49cdbb5bdf221
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
513292f17b |
feat(sso): DNS domain verification gating org SSO registration (#5909)
* feat(sso): DNS domain verification gating org SSO registration
Add org-scoped domain ownership verification (DNS TXT challenge) as the
security precondition for configuring SSO. Closes the first-come domain-claim
vuln where any org could wire another company's domain to its own IdP.
- New sso_domain table + migration 0266; existing org SSO domains are
grandfathered as verified so live tenants are unaffected
- Verified-domains settings UI (enterprise-gated) with add/verify/remove
- Register route now requires a verified domain for org-scoped registration;
personal SSO and already-grandfathered domains are unaffected
- Self-host register script writes the verified sso_domain row directly, so
script-driven registration stays backwards compatible
* fix(sso): harden domain verification against concurrency + fix CI lint
Addresses review findings on state invariants under concurrent/failed writes:
- Add unique index on (organization_id, domain) so concurrent claims can't
create duplicate pending rows; POST re-reads and stays idempotent on conflict
- Verify flips the row only if it's still the exact pending challenge checked
(guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique
index violation to 409 instead of an unhandled 500
- Wrap the self-host script's provider write + verified-domain upsert in a
transaction so a failed ownership write can't leave a provider committed
- Format 0266 snapshot/journal with biome (fixes @sim/db lint:check)
* fix(sso): re-check domain verification before provider write (TOCTOU)
The register gate checked the verified sso_domain row only at handler entry,
then ran OIDC discovery before writing the provider. A verified row removed
during that window could still complete registration. Extract the check into a
closure and call it both as an entry fast-fail and authoritatively right before
registerSSOProvider, alongside the existing domain-conflict re-check.
* fix(sso): stop rotating verification token on idempotent re-add
Re-adding a pending domain rotated its verification token, which invalidated a
TXT record the admin may have already published and — under two concurrent
re-adds — could return a token the racing write had already superseded, so the
admin's DNS record would never verify. Return the existing row unchanged
instead; the pending token is always shown in the UI, so it is never lost.
* fix(sso): close register TOCTOU with compensating delete + harden edges
Audit-driven hardening:
- Close the residual register TOCTOU: registerSSOProvider is create-only (throws
if the providerId exists), so a compensating delete after the write is provably
safe — it can only remove the just-created row. If verification was revoked
during the write, roll the provider back and 403.
- Verify is now idempotent under concurrency: a same-org row already flipped to
verified by a racing request returns 200, not a confusing 409.
- Grandfather backfill + self-host script now match normalizeSSODomain's dominant
transforms (lower + trim + strip leading wildcard) so a non-canonical legacy
domain can't miss the runtime gate's lookup. Prod backfill result is unchanged.
- Cleanup: drop dead default export, align card radius to sibling convention.
* fix(sso): redact domain tokens from non-admins + fix script stale-update
Round-5 review findings:
- GET /domains redacted the pending TXT verification token (a management
secret) to any org member. Now only owner/admins read it; members see the
list and status without it. Non-Enterprise orgs get an empty list (entitlement
flag only), never the domains/tokens.
- Self-host script decided update-vs-insert from a read taken OUTSIDE the
transaction; a provider deleted mid-flight made the UPDATE match zero rows
silently while the verified-domain upsert still committed (orphaned domain).
The decision now happens inside the transaction from the UPDATE's row count.
* docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains
Verified domains currently only gate SSO configuration. Remove the
forward-looking references to enforcing SSO and auto-joining members (deferred
to a later release) from the docs, settings copy, nav description, and schema
comment so we don't promise unshipped features.
* fix(sso): guard rollback to new providers only + Enterprise-gate domain removal
Round-6 review findings:
- The compensating provider rollback now only fires when the provider did not
exist before this request (providerExistedBefore). registerSSOProvider is
create-only today so reaching the rollback already implies a fresh create, but
this makes the safety local and future-proof: if Better Auth ever allowed
updating an existing provider, a revoked-verification rollback must not delete
that pre-existing row.
- DELETE /domains now requires an Enterprise plan like add/list/verify, so all
domain mutations share one entitlement (the UI already hides removal from
non-Enterprise orgs). Adds a delete-route test.
* fix(sso): roll back the SSO provider by row id, not logical keys
The compensating rollback deleted by (providerId, orgId). providerId is unique,
so if this request's row were deleted and recreated by a concurrent registration
in the narrow window before the rollback, the logical-key delete would remove
that other request's provider. Delete by the primary-key id registerSSOProvider
returns instead, so only the exact row this request created is ever removed.
* chore(sso): final-review polish — trim script read, unify copy, doc migration edge
Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the
new logic):
- Self-host script: narrow the pre-transaction existence read to select({ id })
instead of SELECT * (it only feeds a log line now).
- Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere
409 wording ("is already verified by another organization") across routes.
- p-3 shorthand on the domain row card.
- Document the migration's rare two-orgs-share-a-domain grandfather behavior
(login unaffected; validated no such duplicates in prod).
* fix(sso): apply attribute mapping + make SSO edit work; drop dead guard
Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW,
script-registered with the default mapping, so neither change affects it):
- Attribute mapping was passed at the top level of the register payload, which
Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it
so custom mappings actually apply. (Default mapping is unchanged, so existing
logins are unaffected.)
- Editing an SSO provider was broken: registerSSOProvider is create-only and
threw on the existing providerId → generic 500. Route now detects a provider
the caller already owns and updates it via Better Auth's updateSSOProvider, and
surfaces Better Auth's own error status/message instead of a blanket 500.
Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes
by the created row's primary-key id and register is create-only) and the earlier
final-review polish (script read, unified copy, migration edge note).
Smoke-test SSO login + edit on staging before merge (auth-path change).
* fix(sso): require null org on personal-mode provider lookups (gate bypass)
The personal branch of both provider-ownership lookups keyed on
(providerId, userId) without requiring organizationId IS NULL. Because org
providers store userId = their creator and providerId is globally unique, an org
admin could send a personal-mode request (no orgId) — which skips the membership
check and the domain-verification gate — yet still match, and then via the new
update path move, their org's provider to an unverified domain. Add
isNull(organizationId) to the personal branch of both clauses so it can only
match a genuinely personal provider, matching the route's own isOwnedByCaller.
Found by an adversarial review of the update path added in 394bda9f7.
* fix(sso): script updates the observed provider by id, not providerId
Inside the registration transaction the script updated WHERE providerId — the
logical key. If the observed provider was deregistered and a replacement created
with the same providerId before the transaction ran, that update would clobber
the replacement's config and ownership. Update the specific observed row by its
primary-key id instead; if it's gone we insert, which fails cleanly on the
providerId unique constraint rather than overwriting the replacement.
* fix(sso): script upserts provider via delete-then-insert (no unique constraint)
sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate
duplicates, so the previous "update by id, else insert" could create a duplicate
provider when the observed row was deregistered and replaced before the
transaction — the fallback insert would succeed. Delete every row for the
providerId then insert exactly one, inside the transaction, so the providerId
ends up as exactly this config atomically. Linked accounts key on the providerId
string (not the row id), so existing logins are unaffected.
* fix(sso): guard compensating-delete row id so rollback can't silently no-op
* chore(sso): regenerate migration as 0268 after merging staging
Staging landed migrations 0266/0267, colliding with our 0266. Removed our
migration, merged staging, and regenerated cleanly with drizzle-kit as
0268_sso_domain_verification (identical sso_domain table + indexes), then
re-appended the grandfather backfill. api-validation baseline reconciled to 973
(staging 970 + our 3 domain routes). Also make the register-route test's
registerSSOProvider mock return an id so the guarded compensating delete runs.
* refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate
The self-host script canonicalized SSO domains with a minimal inline transform
(lower+trim+wildcard) that diverged from the app's full normalizeSSODomain
(protocol, port, path, trailing dot, email local part) — equivalent spellings
could store a different ownership key than the runtime gate looks up. Move
normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register
route, the domain-claim route, and the script all use the identical canonicalizer.
The script now skips the verified-domain record when SSO_DOMAIN isn't a valid
registrable domain instead of storing a malformed key.
|
||
|
Waleed
|
d078c84ee6 |
chore(typescript): upgrade to TypeScript 7 (native Go compiler) (#5521)
* chore(typescript): upgrade to TypeScript 7 (native Go compiler) Bumps typescript to ^7.0.2 across every workspace package. Full bun run type-check/lint/build/test all pass; apps/sim's type-check (the one needing an 8GB heap bump) drops from ~55s to ~7s wall time. Migration fixes required by TS7's stricter defaults: - baseUrl removed: drop it from 5 tsconfigs (paths already resolved relative to tsconfig dir, so behavior is unchanged) and prefix the one bare (non-relative) paths entry each in apps/sim and apps/realtime with './' - moduleResolution=node10 removed: switch packages/cli and packages/ts-sdk to "bundler", matching the rest of the monorepo - types now defaults to [] instead of auto-including every @types/* package: add "types": ["node"] to the shared base tsconfig (this is fundamentally a Node monorepo, so this restores prior behavior in one place instead of duplicating it per-package), add explicit @types/node deps to packages that now rely on it transitively via @sim/db/@sim/logger, and add "declare module '*.css'" to the two packages with plain (non-module) CSS side-effect imports that TS7's stricter checker now flags - packages/logger's isomorphic `typeof window` check no longer needs DOM lib in every consumer: replaced with `'window' in globalThis` - packages/testing and apps/realtime's fetch/DOM mocks need DOM lib where they're compiled, since they model the browser Fetch API - the `typescript` npm package no longer exports the classic Compiler API from its main entry (moved to unstable/ast subpaths); apps/sim's Function-block route used it at runtime to strip import statements from user code, so that one call site now uses Microsoft's official transition package, @typescript/typescript6 - Next.js 16.2.6's own TypeScript-detection heuristic hardcodes a path TS7 no longer ships, and its auto-install fallback assumes npm/pnpm; added @typescript/native-preview as a devDependency to apps/sim and apps/docs so Next detects a valid native compiler instead of trying (and failing) to auto-install one Not merging yet: TS 7.0.2 published today and is still inside this repo's bunfig.toml minimumReleaseAge (7-day) supply-chain gate, so `bun install` will fail for everyone until 2026-07-15. Opening this now to get it through review; hold the actual merge until then. * fix(typescript): address Greptile review findings on TS7 upgrade - packages/logger: 'window' in globalThis treats a shim that leaves globalThis.window explicitly undefined as browser-only, silently dropping production server logs. Restore the original typeof !== 'undefined' semantics via an inline cast instead, so it stays correct without requiring DOM lib in every consumer. - packages/ts-sdk, packages/cli: both are tsc-built, published as Node ESM (package.json "type": "module" with an "exports" map). "moduleResolution": "bundler" is too permissive for that target - it accepts import patterns (e.g. extensionless relative imports) that Node's actual ESM resolver rejects at runtime. Switch both to "module"/"moduleResolution": "nodenext", the correct pairing for a published Node ESM package. Verified real tsc builds (not just --noEmit) still succeed for both. * chore(bunfig): temporarily disable minimumReleaseAge gate for TS7 install TS 7.0.2 published today, still inside the 7-day gate. Lowering to 0 to unblock this merge; will restore to 604800 in an immediate follow-up commit right after merging. |
||
|
Waleed
|
9d34fbea1b |
refactor(utils): consolidate duplicated helpers onto @sim/utils (#5509)
* refactor(utils): consolidate duplicated helpers onto @sim/utils Replaces ~90 hand-rolled reimplementations of error-message extraction, postgres error-code checks, sleep, Math.random, retry/backoff, object filtering/omission, noop, string truncation, date/time formatting, email normalization, and plain-object type guards with the shared @sim/utils exports. Wires check:utils into CI (test-build.yml) so these patterns don't regress. * fix(test): mock @sim/utils/random instead of Math.random in schedule-execute tests The schedules/execute route previously used Math.random() for jitter delay; this consolidation PR switched it to randomInt() from @sim/utils/random, which is backed by crypto.getRandomValues() rather than Math.random(). The route.test.ts spies on Math.random() no longer had any effect, so jitter became real random delay instead of the deterministic 0ms the tests expect, causing intermittent 10s timeouts in CI. * fix(retry): preserve uncapped Retry-After comparison in tools/index.ts parseRetryAfter() caps its return value at 30s by default. tools/index.ts compares the parsed Retry-After against a caller-configured maxDelayMs to decide whether to skip a retry entirely -- capping before that comparison silently defeats the skip check whenever maxDelayMs is configured above 30s, since a Retry-After between 30s and maxDelayMs would incorrectly look "within limits" and get retried instead of skipped (caught by Cursor Bugbot). Added an optional maxMs param (default unchanged) so tools/index.ts can request the raw, uncapped value for its own comparison while backoffWithJitter still clamps the actual sleep duration to maxDelayMs. Added a regression test covering maxDelayMs > 30s. * fix(utils): fall back to Intl-resolved abbreviation for unmapped timezones getTimezoneAbbreviation only covered 9 hardcoded IANA zones and returned the raw IANA string for everything else, degrading schedule descriptions for zones like Europe/Berlin or America/Toronto (caught by Greptile). The deleted local implementation in schedules/utils.ts resolved any valid IANA timezone generically via Intl.DateTimeFormat's short timeZoneName. Restore that as a fallback so only genuinely invalid timezone strings return themselves unchanged. |
||
|
Theodore Li
|
12fb4a9db1 |
feat(db): auto-apply tracked script data migrations in db:migrate (#5497)
* feat(db): auto-apply tracked script data migrations in db:migrate * fix(db): reset session lock_timeout before script migrations, guard journal insert * improvement(db): re-verify advisory-lock session before script migrations |
||
|
Waleed
|
48752c6024 |
fix(media-embed): remove ReDoS-prone regexes in host-gated providers (#5305)
* fix(media-embed): remove ReDoS-prone regexes in host-gated providers Replace the unbounded '.*' patterns flagged by CodeQL (js/polynomial-redos) in the YouTube, Facebook, and Giphy branches with bounded extraction off the parsed URL (pathname / searchParams). Eliminates the O(n^2) backtracking a crafted valid-host URL could trigger, with no change to matched links. * test(media-embed): lock youtu.be trailing-slash + edge parity Use the first path segment for youtu.be ids so a trailing slash still resolves (matching the previous regex), and cover extra-query-param, si-param, embed-query, and short-id cases. * fix(media-embed): dispatch YouTube id by path shape; drop inline comments - Resolve id from the /embed/ path segment before the ?v= query param so a valid embed URL with a spurious v param still embeds (was returning null) - Remove non-TSDoc inline comments from the module and its test |
||
|
Waleed
|
ca0a7ff0c2 |
feat(rich-markdown-editor): live media embeds + shared embed detection util (#5290)
* feat(rich-markdown-editor): live media embeds + shared embed detection util - Extract getEmbedInfo/EmbedInfo into pure @sim/utils/media-embed (carries the PR #5288 dropbox host-validation hardening); repoint the note block to it - Add LinkEmbed: a ProseMirror widget-decoration plugin that renders media players (YouTube, Vimeo, Spotify, Dropbox, …) beneath standalone links in the rich markdown editor, in both editing and read-only surfaces. The document stays a plain markdown link, so markdown round-trips stay lossless - Gate embeds behind an opt-in flag (on for the file editor, off for modal fields) - Polish the knowledge chunk editor to the file editor's centered reading frame while keeping it plaintext for exact embedding fidelity * fix(media-embed): gate provider detection on parsed hostname Validate each platform against the URL's parsed host before extracting, so a look-alike host (youtube.com.evil.com) or a provider domain in the path (evil.com/youtube.com/...) can no longer render a trusted-looking embed. Dropbox is no longer a special case — all providers share the hostMatches gate. Also consolidates the five Spotify branches and orders Twitch clip before channel. * fix(rich-markdown-editor): unique widget key per duplicate embed URL Key embed widgets by source + per-source occurrence index so two standalone links to the same URL render as two distinct players instead of collapsing into one, while keeping the key stable across unrelated edits (no iframe reload). * refactor(media-embed): tighten comments and drop a redundant guard - Drop the redundant paragraph type-check in getStandaloneLinkHref (the caller already filters to paragraphs) and rename the param for clarity - Remove an inline comment and a TSDoc sentence that restated logic documented elsewhere |
||
|
Waleed
|
0673e3c0f7 |
refactor(sim): consolidate record guards + pure utils into @sim/utils (#5061)
* feat(utils): add record guards and pure helpers to @sim/utils Add isRecordLike (loose, non-prototype-checked record guard) and sortObjectKeysDeep; relocate isPlainRecord (strict) and normalizeEmail into @sim/utils so they are reusable across apps and packages. Unit tests cover the loose-vs-strict distinction, deep key sorting, and email normalization. * refactor(sim): consolidate record guards and normalize helpers onto @sim/utils Replace ~55 re-implemented loose record guards with the canonical @sim/utils isRecordLike (and one strict site with isPlainRecord), and dedupe three normalize clusters: sortObjectKeysDeep (sanitization + copilot builders), normalizeToken (salesforce + servicenow triggers), and normalizeEmail. Array-allowing guards and domain-specific normalizers are intentionally left untouched. Pure refactor — identical predicates and transforms, no behavior change. |
||
|
|
0075ab9cf6 |
improvement(platform): remove tour, simplify sidebar/header, drop loading skeletons (#4354)
* improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Theodore LiandClaude Opus 4.8
|
aed44024d4 |
fix(tables): surface real error causes on cell-execution failures (diagnostics) (#4868)
* fix(tables): retry transient DB/Redis failures in cell execution and surface error causes Workflow-group-cell runs intermittently failed on trivial DB reads/writes under heavy fan-out, stranding cells in `running`. Investigation showed the PlanetScale and ElastiCache backends were healthy at the time — the failures are transient connection-level faults that the cell (maxAttempts: 1) had no tolerance for, and the real cause was never logged (Drizzle wraps it as "Failed query: ..." and the driver cause lives in error.cause). Resilience: - Add retryTransient (lib/table/retry-transient.ts): retries only transient infra errors (reuses isRetryableInfrastructureError; adds an ioredis command-timeout match) with jittered backoff, then rethrows. Fail-fast for everything else. - Wrap the cell's getTableById/getRowById reads, the terminal write (cell-write updateRow — idempotent via the executionId guard), and the Redis cascade-lock acquire. Diagnostics: - Add describeError (lib/core/errors/retryable-infrastructure.ts): walks the .cause chain and always returns the underlying driver cause (code/errno/ syscall + causeChain), including for unclassified errors like AbortError. - Log `cause` + a `retryable` flag (and aborted/timedOut in the cell's main catch) across the cell + finalization error paths, mirroring the existing schedule-execution pattern. Logging-only; no behavior change. This lets the next recurrence reveal the real cause and whether the retry applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): address review feedback on cell retry resilience - retryTransient: re-check the abort signal after the backoff sleep so a cancellation during sleep stops the next attempt (don't run/return work for an already-cancelled task). - isRetryableRedisError: walk the .cause chain (mirroring the infra classifier) so wrapped Redis timeouts are recognized; drop "Connection is in subscriber mode" — that's a connection-state programming error, not a transient drop, and would just fail identically every retry. - cascade-lock: stop wrapping acquireLock in retryTransient. acquireLock is a non-idempotent SET NX, so retrying after a timed-out-but-applied first SET returns false (key already ours) and yields a false `contended` that skips the cascade. A transient Redis blip here just fails the run before pickup (no stranded cell); the dispatcher re-drives it. - Tests: cause-chain Redis match, subscriber-mode exclusion, abort-during-sleep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): drop out-of-scope abort/timeout fields from cell catch The main catch logged `aborted`/`timedOut` from `abortSignal`/`timeoutController`, but those are declared inside the outer try block (the inner try around executeWorkflow is try/finally, so this catch belongs to the outer try) and are not in scope in the catch — `next build`'s type-check failed with "Cannot find name 'abortSignal'". Local incremental `tsc --noEmit` had skipped the file and falsely passed; the Cursor/Greptile reviewers flagged this correctly. Removed the two fields. Abort/timeout is still surfaced via `cause: describeError(err)` (an aborted run shows `name: 'AbortError'` / the timeout message), so no diagnostic signal is lost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(tables): drop in-process retry, keep cause diagnostics only In-process retry is the wrong layer for this path: the cell task is maxAttempts:1 by design, retrying on a possibly-degraded worker may not help, and it masks the very transient-failure signal we're trying to capture before we understand the root cause. Removed retryTransient entirely (file + all wrapping in cell-write, the cascade reads, and the lock acquire) and kept only the diagnostic logging. - Deleted lib/table/retry-transient.ts (+ test); cell-write and the cascade reads call getTableById/getRowById/updateRow directly again, fail-fast. - Kept describeError + `cause`/`retryable` fields across the cell + finalization catch blocks; the cell-path `retryable` flag now sources from isRetryableInfrastructureError (the canonical classifier) for consistency. Diagnostics-first: surface the real driver cause on the next recurrence, then decide the actual fix (e.g. task-level maxAttempts, or addressing the worker- side cause) from evidence rather than a speculative in-process retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): log error cause on scheduled-execution failure paths The scheduled-job failure paths logged the raw error (.message/stack only) — its `.cause` (the real driver error behind a Drizzle "Failed query: ..." wrapper) was never recorded, and the classified-only `describeRetryableInfrastructureError` returns undefined for unrecognized errors. A real failed run (same incident window as the cell failures) failed in `applyScheduleUpdate` with exactly this unrecorded cause. Added `cause: describeError(error)` (always-on, walks the cause chain) to the applyScheduleUpdate catch, the early-failure catch, and the unhandled-error catch — passed as a second arg so the existing message+stack still emit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(errors): move describeError to @sim/utils/errors `describeError` is a general-purpose error/cause-chain helper — it didn't belong in `lib/core/errors/retryable-infrastructure.ts` (that module is specifically about classifying retryable infra errors, and the name read wrong for a generic diagnostic). Moved it to `@sim/utils/errors` alongside `toError`/ `getErrorMessage`/`getPostgresErrorCode`, with its own cycle-safe cause walk. - Added describeError + DescribedError + tests to packages/utils/src/errors.ts. - Reverted the describeError addition from retryable-infrastructure.ts (it keeps only isRetryableInfrastructureError / describeRetryableInfrastructureError, which are accurately named and still used by the schedule retry path). - Re-pointed all consumers (cell, logging-session, pause-persistence, schedule) to import describeError from @sim/utils/errors. The `retryable` classification flag still sources from isRetryableInfrastructureError where used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
a7b0bd311d |
fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) (#4837)
* fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) - Bump vitest and @vitest/coverage-v8 to ^4.1.0 across all workspaces (only patched release for the critical 'Vitest UI server arbitrary file read/execute' advisory; no 3.x backport exists) - Widen @sim/testing peer range to ^3.0.0 || ^4.0.0 - Migrate constructor mocks to class expressions: vitest 4 uses Reflect.construct for mocks invoked with new, and arrow/function implementations are not constructable (function expressions also get reverted to arrows by biome's useArrowFunction) - Remove deprecated test.poolOptions from apps/sim/vitest.config.ts (options are now top-level in vitest 4) * fix(deps): exclude vulnerable vitest 4.0.x from @sim/testing peer range Tighten the v4 arm of the peer range to >=4.1.0 <5.0.0 so the peer requirement cannot be satisfied by the unpatched 4.0.x builds that GHSA-5xrq-8626-4rwp affects. * fix(testing): make vitest 4 constructor mocks type-check cleanly - logging-session & mcp-oauth mocks: a class passed to mockImplementation has a construct signature that isn't assignable to its (...args) => any parameter, failing tsc. Use named function declarations instead (constructable via Reflect.construct, assignable to mockImplementation, and not rewritten to arrows by biome's useArrowFunction). - database.mock.ts: vitest 4's generic vi.fn typings no longer break the self-referential cycle on the transaction callback's tx param; loosen tx and annotate the callback's return type to resolve the implicit-any errors. * test(isolated-vm): de-flake queue-capacity scheduler tests The 'queue is full' and 'per-owner queued limit' tests relied on 'await sleep(1)' to assume the first request had reached the queue before submitting the overflow request. The first request only enqueues after an async spawn-failure chain (acquireWorker -> spawn exit -> resolve null -> enqueue), which isn't guaranteed within 1ms under CI load — the overflow request then found an empty queue and hit the 200ms queue-wait timeout instead of the capacity rejection. Replace the wall-clock barrier with a deterministic, event-driven one: hold the single global concurrency slot (IVM_MAX_CONCURRENT=1) with an active worker and await an explicit 'dispatched' signal (fired when the worker receives its execute message, after the scheduler counts it active). The follow-up requests then deterministically hit the synchronous enqueue path. Also drops the queue-wait timeout from 200ms to 50ms, so the tests run faster. |
||
|
Waleed
|
8d7bbbc670 |
chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting - Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files - Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors) - Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities - Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time - Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access - Add check:utils script to package.json * chore(utils): replace deepClone wrapper with structuredClone built-in deepClone() was a one-line wrapper around structuredClone(), which is universally available in Node 17+ and all modern browsers. Removing the abstraction reduces indirection and means contributors don't need to learn a project-specific name for a well-known built-in. - Remove deepClone from packages/utils/src/object.ts and index.ts - Replace all 17 call sites with structuredClone() directly - Update check:utils script suggestion text - Update CLAUDE.md and global.md docs * fix(utils): add missing biome noRestrictedImports rule and correct truncate docs - Add noRestrictedImports to biome.json under style — bans nanoid and uuid package imports at lint time (crypto.randomUUID/randomBytes are caught by the check:utils grep script which handles global property access) - Correct truncate() TSDoc and parameter name: sliceLength makes it clear that total output length is sliceLength + suffix.length, matching the behavior all callers were already written to expect * fix(utils): add missing getErrorMessage imports at 4 call sites The sweep agents added getErrorMessage calls without the corresponding import in 4 files, causing test failures. Added the missing imports. * fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue - Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for internal package imports; inline the jitter crypto call directly - Add missing getErrorMessage imports to 32 files where the sweep added calls without the corresponding import (caught by type-check and test runs) - Remove accidental getErrorMessage import from crowdstrike/query/route.ts which has its own domain-specific getErrorMessage for parsing CrowdStrike's JSON error format - Fix use-sub-block-value.ts type error from structuredClone narrowing: add 'as T' cast at emitValue callsite (safe — valueCopy is always a structural copy of newValue) * fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage The catch block was calling the local getErrorMessage function which parses CrowdStrike API JSON responses, not JavaScript Error objects. Use toError(error).message to correctly extract the message from a caught value in this context. |
||
|
Waleed
|
0c25fc4ee1 |
fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies (#4369)
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies
- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml
Fixes simstudioai/sim#1243
* fix(auth): address PR review comments
* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)
* fix(docker): include @sim/utils in migrations image
Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.
* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections
The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.
* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema
The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.
* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional
The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:
- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
http://localhost:3002 to empty for consistency with prod.yml; the
in-code localhost fallback handles the dev case identically
Applied across all 6 documentation languages (en/fr/de/ja/es/zh).
* chore: untrack and ignore .claude/scheduled_tasks.lock
|
||
|
Vikhyath Mondreti
|
aee6189d14 |
improvement(access-control): migrate to workspace scope (#4244)
* improvement(access-control): migrate to workspace scope * fix edge cases * update docs * prep merge * regen migrations * address comments * add ws id, user constraint * address more comments * address ui comments * address more comments |
||
|
Waleed
|
b5674d9ed4 |
improvement(codebase): centralize test mocks, extract @sim/utils, remove dead code (#4228)
* improvement(codebase): centralize test mocks, extract @sim/utils, remove dead code * improvement(codebase): apply @sim/utils conventions to staging-introduced files |