mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
7349bf403ff754c8091eeff2aabe2ca5b606ff5c
287
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Theodore Li
|
7349bf403f |
feat(files): password, email-OTP, and SSO auth for public file shares (#5140)
* feat(files): password, email-OTP, and SSO auth for public file shares * fix(files): suppress filename in share previews for email/sso, not just password * fix(files): normalize allow-list emails to lowercase; genericize shared SSO denial message * fix(security): make isEmailAllowed case-insensitive; normalize email at client gates * test(security): cover isEmailAllowed case-insensitive matching * fix(security): bind auth cookie to auth type; password endpoint rejects non-password shares * chore(db): format generated migration meta * fix(files): share upsert validation returns 400 not 500; disabling always succeeds * feat(access-control): org admins can restrict allowed file-share auth types |
||
|
Theodore Li
|
208d135dac |
feat(enrichment): add enrichment details sidebar with cost + provider cascade (#5139)
* feat(enrichment): add enrichment details sidebar with cost + provider cascade * fix(enrichment): address review — persist detail on cancel/skip, exclude not_run from ran count, refetch on panel open * fix(enrichment): keep cascade detail sticky on upsert; mark unattempted providers not_run on abort * fix(enrichment): show Cancelled in details panel for aborted runs |
||
|
Theodore Li
|
9d2a6ef043 |
feat(logs): redact PII from workflow logs via configurable rules (#5136)
* feat(logs): redact PII from workflow logs via configurable rules Enterprise PII redaction for workflow execution logs, configured under Data Retention as org-scoped rules (each rule picks entity types + which workspaces it applies to). Reuses the guardrails Presidio engine in mask mode at the log-persist choke point, with a check-digit-validated VIN recognizer. Also adds per-workspace data-retention-hours overrides. * fix(logs): widen PII entity visibleValues to string[] for strict build typecheck * fix(logs): redact error/trigger/executionState; keep guardrails import lazy - Extend PII redaction to span error/errorMessage/toolCalls and top-level error/completionFailure/trigger/executionState (Bugbot: PII in execution metadata). executionState is safe to redact — resume reads from the separate pausedExecutions table, not the log copy. - Lazy-import validate_pii in pii-redaction so the Python/child_process guardrails module stays out of the static middleware/RSC graph. - Type the org retention mutation to the contract body (optional, non-null). * refactor(logs): drop per-workspace retention override; PII redaction stays org-scoped - Remove the unused per-workspace data-retention-hours override (no UI; superseded by workspace-scoped PII rules). Reverts cleanup-dispatcher to org-only retention, drops resolveEffectiveRetentionHours, the workspace.dataRetentionSettings column + migration, and the workspace data-retention route/contract/hooks. Fixes Bugbot's null-as-unset finding by removing the buggy path entirely; org retention behavior is unchanged. - Stop re-checking isWorkspaceOnEnterprisePlan at persist time (it returns false on transient errors, which would fail-open and leak PII). Enabled rules already imply entitlement; redact whenever rules apply (fail-safe). * fix(logs): redact oversized strings and executionData.environment - Drop the per-string size cap in PII redaction: oversized strings were left unmasked (leak). Nothing is skipped now; large payloads still fail-safe via the total-bytes ceiling + per-chunk timeout (scrub, never leak). - Add executionData.environment (incl. variables) to the redaction set. * refactor(logs): single-scope PII rules with most-specific-wins resolution Each rule now targets one scope — all workspaces (workspaceId: null) or a single workspace — with workspaceId unique across rules. Resolution is most-specific-wins (a workspace's own rule overrides the all rule), not union; an empty specific rule exempts that workspace. Matches Access Control's resolveWorkspaceGroup precedence. UI 'Applies to' becomes a single-select; Add rule disables when all scopes are taken. * feat(logs): default + workspace-overrides UI for PII redaction Reshape the PII redaction settings into a 'Default (all workspaces)' block plus a 'Workspace overrides' list, making the most-specific-wins precedence explicit (overrides replace the default; unlisted workspaces use it). Same data model (workspaceId null = default), UI only. * improvement(logs): clearer default/overrides PII UI Drop the uppercase section labels and the overrides description; gate the Workspace overrides section behind a configured default; use a single Delete action; 'Add redaction' creates the all-workspaces default and disappears once set. * fix(guardrails): handle stdin EPIPE in PII python spawns Attach an 'error' listener to the child's stdin in both runPythonScript (the batch masking hot path) and executePythonPIIDetection. A 256KB chunk can exceed the OS pipe buffer, so if the Python process exits mid-read (OOM/kill) the EPIPE emitted on stdin was unhandled and would crash the Node process. Funnel it into the promise rejection so the fail-safe scrub path handles it gracefully. * fix(logs): redact executionData.correlation The top-level correlation field is copied from pre-redaction trigger data, so webhook/schedule correlation values could persist unredacted. Add it to the redaction set alongside trigger/environment. * fix(logs): enforce unique PII rule scope server-side The contract accepted multiple rules with the same workspaceId (or several null all-rules); resolution is first-match, so duplicates could disagree with the UI. Add a schema refine rejecting duplicate scopes. * fix(logs): re-hydrate data-retention form on org switch The form hydrated once via a boolean ref, so switching the active org left stale retention days + PII rules and saves targeted the new org with old config. Key hydration on orgId so it re-loads per org. |
||
|
Vikhyath Mondreti
|
91f9dfdaec |
improvement(governance): derived access (#5134)
* improvement(governance): org-ws-credential roles clarity * revert isHosted * improvement(credentials): code cleanup * address comments * make kb cascade delete on user hard delete * revert env flags * chore(db): drop local 0242 migration to regenerate after merging staging Our 0242 collides with staging's 0242. Remove it (and its snapshot + journal entry) so the KB-cascade migration can be regenerated with the correct number on top of the merged staging migrations. * chore(db): regenerate kb→workspace cascade migration as 0243 Regenerated via drizzle-kit generate on top of the merged staging migrations (staging took 0242). Re-applied the safety edits: NOT VALID + separate VALIDATE on the FK re-add, and the -- migration-safe note on the DROP. check:migrations passes. * improve copy * update docs |
||
|
Theodore Li
|
f0b3550729 |
feat(files): public share links for workspace files (#5130)
* feat(files): public share links for workspace files * improvement(files): drop reserved public_share columns until used; sync audit mock * fix(files): share modal tracks authoritative saved state until toggled * feat(files): per-IP rate limit on public share endpoints * fix(files): address PR review — public CSV OOM, content cache, share FK, soft-delete filter, download anchor * fix(files): disable CSV import action in read-only preview (public share) * refactor(files): drive CSV preview import affordance off readOnly, not disableImport * fix(files): version public viewer caches by file updatedAt so edits aren't stale * fix(files): 409 (not corrupt source) when a shared generated doc has no compiled artifact * feat(files): gate public sharing behind an access-control permission |
||
|
Vikhyath Mondreti
|
267e49c69c |
improvement(workspaces): auto-add without invite if part of organization (#5132)
* feat(workspaces): auto-add without invite if part of organization * reverse feature flag hardcoding * address comments * improve ux for org invite modal |
||
|
Theodore LiandClaude Fable 5
|
63fdc472c1 |
improvement(block): table empty-state filter/sort builders + upsert conflict-column selection (#5123)
* ci(migrations): fail dev schema push with an actionable error on rename/drop prompt
`drizzle-kit push --force` only suppresses the data-loss confirm, not the
rename-vs-drop disambiguation prompt. That prompt fires whenever a diff both
adds and drops tables/columns at once (e.g. migration 0231 created
sim_trigger_state while dropping the workspace_notification_* tables), and in
CI it crashes with a bare "Interactive prompts require a TTY" stack trace.
Catch that specific failure in the dev push step and emit a GitHub error
annotation explaining the cause and the fix (drop the stale objects on the dev
DB to match schema.ts — the same DROPs the versioned migration already applied
to staging/prod), instead of leaving an opaque trace. Exit status is preserved
either way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* improvement(tables): empty-state filter/sort builders + upsert conflict-column selection
* improvement(tables): throw on ambiguous upsert instead of guessing the conflict column
* Revert "ci(migrations): fail dev schema push with an actionable error on rename/drop prompt"
This reverts commit
|
||
|
Theodore Li
|
597d7eafb5 |
fix(tables): enforce row limits against the current plan, not a frozen per-table cap (#5120)
* fix(tables): enforce row limits against the current plan, not a frozen per-table cap * fix(tables): gate multi-batch CSV create + initial rows against the plan, harden limits cache bound * fix(tables): thread running row count through copilot batchInsertAll capacity check * chore(tables): align tx-variant capacity docstrings * fix(tables): map row-limit errors to 400 in create-from-CSV import * feat(tables): add Upgrade action to the row-limit toast * fix(tables): keep CreateTableData.maxRows so staging callers type-check after merge * improvement(tables): route row-limit Upgrade action to the explore-plans page |
||
|
Waleed
|
7d46103d09 |
chore(deps): remove unused dependencies and harden CI supply chain (#5119)
* chore(deps): remove unused dependencies and harden CI supply chain Dependency cleanup: - Remove unused deps: papaparse, unified, and 6 unused Radix primitives (alert-dialog, radio-group, scroll-area, separator, toggle, visually-hidden) plus @tanstack/react-query-devtools (all verified zero imports repo-wide) - Consolidate jwt-decode into the existing jose dependency (decodeJwt) - Migrate react-window to @tanstack/react-virtual to drop a redundant virtualization library (terminal, structured-output, code viewer) - Remove the better-auth-harmony plugin and its gating env flag Supply-chain hardening: - SHA-pin every GitHub Action to a full commit SHA with a version comment - Pin CI bun-version to 1.3.13 (was "latest" in the release job) - Raise bun minimumReleaseAge cooldown from 3 to 7 days - Add a non-blocking `bun audit` step in CI - Add a CODEOWNERS gate routing dependency-manifest changes to @simstudioai/deps * chore(deps): remove unused apps/docs dependencies (@tabler/icons-react, dotenv-cli) * style(search-modal): use Send icon for Invite teammates action * feat(search-modal): surface New chat as the top action above Create workflow * feat(search-modal): add Secrets to the pages list |
||
|
Theodore Li
|
ea505f0388 |
improvement(tables): versioned CSV snapshot cache for table mounts + parallel multipart uploader (#5108)
* improvement(tables): versioned CSV snapshot cache for table mounts + parallel multipart uploader * chore(db): drop colliding 0239 migration (renumber pending) * chore(db): renumber rows_version migration to 0240 (off staging's 0239) * improvement(tables): mount snapshots by presigned URL so the sandbox fetches directly (raise cap to 500MB) * fix(tables): allow url sandbox entries in the function-execute contract; key snapshot by column shape so schema edits invalidate it * chore(e2b): log sandbox inputs split by url-fetch vs inline write * improvement(tables): order export + snapshot rows by order_key so the CSV matches the grid under fractional ordering |
||
|
Waleed
|
a82b44d36d |
perf(db): logs-list index, drop redundant indexes, replica routing, hot-path write cleanups (#5105)
* perf(db): logs-list index, drop redundant indexes, replica routing, hot-path write cleanups * fix(logs): keep /api/v1/logs on primary db — its permissions join is the auth gate, not replica-safe |
||
|
Vikhyath Mondreti
|
7b4626e547 |
improvement(perm-groups): allow workspace filter for permission groups (#5070)
* improvement(perm-groups): allow workspace filter for permission groups * show errors correctly * address comments * address concurrent edit concern * address locks * address comments" * index migration safety * address at route level |
||
|
Theodore Li
|
3fe061e3b3 |
feat(feature-flags): AppConfig-backed gated feature flags (#5059)
* feat(feature-flags): AppConfig-backed gated feature flags * fix(ci): repoint 'Validate feature flags' step to env-flags.ts after rename * improvement(feature-flags): drop in-code defaults; fallback resolves a per-flag secret, gating is AppConfig-only * improvement(feature-flags): make flag names a closed set so every flag requires a fallback secret * improvement(feature-flags): single FEATURE_FLAGS registry — each entry defines name, description, and fallback in one place * improvement(feature-flags): fallback is the env secret key (keyof typeof env), resolved to a boolean |
||
|
Waleed
|
0673e3c0f7 |
refactor(sim): consolidate record guards + pure utils into @sim/utils (#5061)
* feat(utils): add record guards and pure helpers to @sim/utils Add isRecordLike (loose, non-prototype-checked record guard) and sortObjectKeysDeep; relocate isPlainRecord (strict) and normalizeEmail into @sim/utils so they are reusable across apps and packages. Unit tests cover the loose-vs-strict distinction, deep key sorting, and email normalization. * refactor(sim): consolidate record guards and normalize helpers onto @sim/utils Replace ~55 re-implemented loose record guards with the canonical @sim/utils isRecordLike (and one strict site with isPlainRecord), and dedupe three normalize clusters: sortObjectKeysDeep (sanitization + copilot builders), normalizeToken (salesforce + servicenow triggers), and normalizeEmail. Array-allowing guards and domain-specific normalizers are intentionally left untouched. Pure refactor — identical predicates and transforms, no behavior change. |
||
|
Waleed
|
4ec26a0404 |
feat(scheduled-tasks): minute-granular calendar + user timezone preference (#5038)
* feat(scheduled-tasks): position week/day chips at their exact minute Replace hour-bucketed event rendering with a per-day absolute overlay that places each task chip at timeToOffset(start), so a 5:38 task sits at 5:38 instead of the top of the 5:00 cell. Hour cells become click/gridline-only; the overlay is non-interactive so empty-space clicks still create. Removes the now-obsolete eventsByHour/hourKey/bucketEventsByHour path (month view already used eventsByDay). * feat(settings): user timezone preference for scheduled tasks Add a Timezone preference under Settings → General. Scheduled tasks now run in the user's chosen IANA zone instead of whatever device created them. - settings table gains a nullable `timezone` column (migration 0236); null means "use the browser-detected zone", so existing users are unchanged - contract: validated IANA `timezone` on the settings get/update shapes - useTimezone() resolves the saved zone or the browser fallback; the task modal captures it instead of recomputing the device zone - General settings adds a searchable timezone combobox defaulting to the detected zone - shared timezone util (getBrowserTimezone / getSupportedTimezones) * fix(scheduled-tasks): interpret launch/end times in the account timezone Address review: one-time runs and the end-of-day boundary were resolved in the browser zone, so a task could fire at the wrong instant when the account zone differed from the device. Resolve wall-clock launch/end through the account zone (DST-correct), and evaluate the past-launch guard and the default seed in that zone too — matching how the recurring cron is already evaluated. - timezone util: zonedWallClockToUtc (DST-correct, no library), wallClockNow; getSupportedTimezones falls back to a common set and always includes UTC - recurrenceToScheduleFields takes the timezone and resolves time/endsAt in it - settings timezone Label drops its dangling htmlFor - tests for the zone converter (UTC / +5:30 / DST) and the zoned mappings * feat(scheduled-tasks): Google-Calendar-style side-by-side overlap layout Tasks whose pills would collide now split the column into side-by-side lanes (like Google Calendar) instead of stacking on top of each other; tasks that don't overlap keep the full width. Adds a pure layoutColumn lane-assignment helper (interval clustering + greedy lane packing) with tests. * fix(scheduled-tasks): zone-consistent recurrence/edit + duplicate + loading Address review (zone consistency): - recurrenceToCron derives weekday/day-of-month from a UTC-parsed calendar date so the cron targets the right day regardless of device zone - cronToRecurrence + editSeedFor recover the launch date/time and ends-on date read back in the schedule's zone (zonedWallClock), so editing shows the right values when the account zone differs from the device - defaultLaunch no longer compares browser-local slot days against account-zone "today" Features: - right-click Duplicate: opens a pre-filled create modal from any task (TaskEditSeed now extends a shared TaskPrefill; modal gains a prefill prop) - loading.tsx paints only the header chrome (the page is a calendar, not a table) so it no longer pops table -> calendar; the empty calendar loads tasks in - task context menu drops "See details" (finished tasks open on click) * fix(scheduled-tasks): edit/duplicate use the task's own timezone, not the account one A task created in one zone but edited after the account zone changed (or duplicated) seeded its launch in the task's stored zone while validating and submitting in the current account zone, drifting unchanged run times. TaskPrefill now carries the task's timezone; the modal seeds AND submits in it for edit/duplicate, and only blank creates use the account zone. * fix(scheduled-tasks): duplicating a past one-time task seeds a future launch Audit follow-up: a duplicate of a one-time task whose launch already passed opened with Schedule disabled. It now falls through to the next-hour default so the new task is immediately schedulable. Also clarifies the DST spring-forward note on zonedWallClockToUtc and drops a stray test comment. * fix(scheduled-tasks): clear duplicate pre-fill when starting a fresh create The create modal had two open-sources (isCreateOpen + duplicatePrefill) that weren't coordinated. Starting a header/slot create now clears any duplicate pre-fill (and duplicating closes any open create), so a stale duplicate draft can never bleed into a blank or slot-seeded create. * fix(scheduled-tasks): make create/duplicate/edit modals mutually exclusive Opening any of the three modal flows (blank create, duplicate pre-fill, task edit/record) now closes the other two, so the create modal can never co-exist with the edit modal and no stale state survives a switch. * feat(scheduled-tasks): render calendar in the effective timezone Position each occurrence at its wall-clock time in the task's own timezone, and draw the now-line / today highlight in the viewer's effective zone, so the calendar always shows a task at the local time it was scheduled for — matching the modal. Adds zonedClockDate as the single zone boundary; the default case (account zone == browser zone) is unchanged. * fix(scheduled-tasks): re-sync calendar day frame when timezone resolves When useTimezone() resolves from the browser fallback to the saved account zone after mount, re-derive today (and the focused day, while it is still on today) so the grid frame, now-line, and fetched range stay in agreement. The focused day is preserved across the change once the user has navigated away. * fix(scheduled-tasks): pad view window for timezone slop; re-center scroll on zone change visibleRange now expands the rendered span by a day on each side so an occurrence whose own-zone display day is on screen is never filtered out by the account-zone frame; bucketEventsByDay still places each on its zoned day, dropping any off-screen. The week/day auto-scroll re-centers when the effective timezone resolves. * test(scheduled-tasks): pass timezone to cronToRecurrence ends-on case The second cronToRecurrence call omitted the required timezone, so the recovered end date depended on the test runner's system zone instead of the schedule zone. Pin it to UTC for determinism. * fix(scheduled-tasks): re-seed blank-create launch when timezone resolves useTimezone() starts on the browser fallback, so a blank create's next-top-of-the-hour default (and its past-launch guard) could be seeded in the wrong zone and submitted in the resolved account zone. Re-seed the default when the effective zone changes, unless the user has edited the fields; slot/edit/duplicate seeds are zone-stable and untouched. * fix(scheduled-tasks): DST fall-back resolve, today month-cell default, late-night pill bounds Audit follow-ups: - zonedWallClockToUtc resolves to the self-consistent instant, fixing one-time launches on the autumn fall-back day (were an hour early) while keeping the spring-forward gap rolling forward; adds DST regression tests. - defaultLaunch: today's whole-day (month-cell) click defaults to the next top of the hour like the header action, not a past 9am that disables Save. - DayEvents clips to the day bounds so a late-night pill never spills past the final hour row; now-line sits above event pills. |
||
|
Vikhyath Mondreti
|
3a796f083d |
improvement(permissions): permission groups scoped to organization level (#5035)
* improvement(permissions): permission groups scoped to organization level * chore(db): drop 0235 permission-groups migration to regenerate after staging merge * merge latest staging |
||
|
Emir Karabegandwaleed
|
bcedadf1b1 |
feat(scheduled-tasks): calendar views + persisted, runnable tasks (#4979)
* improvement(resource): simplify table shell, toasts, and loading breadcrumbs - Resource.Table: remove internal sorting (defaultSort/sortValues) and the emptyMessage state — rows render in the order given, chrome always paints - Resource: root is now the positioning context for overlays; consumers (files, tables, knowledge, document) wrap detail views in <Resource> instead of hand-rolled divs - ResourceHeader: root titles no longer truncate during initial layout; LocationFocusVeil gates the portal on mount to fix a hydration mismatch - Toasts: drop the StackDismiss ring and stack countdown — each toast runs its own timer; remove the Mod+E clear-notifications command; align toast typography and icons with chip chrome - Breadcrumbs: use the canonical '…' placeholder while names load - incident.io: fix display name and catalog slug (with redirect) - Add dev:capped / dev:full:capped scripts with a 4GB heap cap * feat(scheduled-tasks): calendar views; rename Mothership to Sim/Chat Add month/time calendar views for scheduled tasks with toolbar, event chips, and a create-task modal, backed by calendar-grid and schedule-events utils (with tests) and a use-calendar hook. Replace the old schedule-modal/context-menu flow. Rename the "Mothership" agent to "Sim" and the chat surface to "Chat" across landing copy, constitution, block metadata, API error messages, and copilot/data-drain internals. Drop unused workspace route layouts. * fix(emcn): force dropdown menus modal inside dialogs so they scroll A non-modal DropdownMenu portals outside an open dialog's react-remove-scroll subtree, so its content cannot be wheel-scrolled (e.g. the time picker in the scheduled-task create modal). ModalContent now marks its subtree via an InsideModal context, and the emcn DropdownMenu root upgrades itself to modal inside dialogs so it mounts its own scroll lock and focus scope; page-level menus keep their consumer-chosen modality. Also stretch the create-task modal's date/time chip controls to full width and drop the dead EDGE_GUTTER constant left behind by the equal-tracks calendar layout. * fix(scheduled-tasks): address review — midnight rollover, stub feedback, smooth Today scroll - useCalendar: today was frozen at mount, so after midnight the isToday column highlight and the current-time indicator stayed on the previous day. today is now state refreshed by a sleep-resilient minute poll that only re-renders when the calendar day actually changes - CreateTaskModal: the stub submit closed silently, reading as false success; it now shows an info toast that the task was not created - ScheduleCalendar: Today presses scroll smoothly as an orientation cue; mount and scope switches keep instant positioning * feat(emcn): view-only field primitives + scheduled-task modals - ChipCopyInput (canonical view-only copy field), ChipTimePicker, ChipModalField type='copy', ChipTextarea viewOnly; new border chip variant and shared chipPrimaryFillTokens - migrate ~40 consumers off disabled inputs and the deleted CopyableValueField; ChipConfirmModal description->text and secondaryActions[] API sweep - scheduled-tasks: rename create-task-modal to task-modal, add task-details-modal + task-context-menu, useScheduledTasks hook - home: extract prompt-editor (usePromptEditor) out of user-input * feat(scheduled-tasks): persist + run tasks via the job-schedule backend Wire the calendar UI to the existing sourceType='job' workflow_schedule backend instead of local component state, so tasks persist and actually run as Sim agent invocations. - schema: add contexts (@-mentions resolved into the run), excludedDates (per-occurrence deletes), and endsAt (recurrence end) to workflow_schedule (migration 0235) - contracts/schedules: expose one-time `time`, contexts, endsAt on create; add the exclude_occurrence action; nullable cron in the create response - orchestration: persist the new fields, honor exclusions + end boundary via a shared computeNextRunAt, add performExcludeOccurrence - execution: forward contexts to /api/mothership/execute and recompute the next run through computeNextRunAt - mothership/execute: accept + resolve contexts like the interactive chat path - frontend: replace the local hook with React Query (create/update/delete + exclude-occurrence), expand recurrences into calendar occurrences, add the recurrence control (frequency + end) and the recurring this/all delete dialog * chore(scheduled-tasks): satisfy biome line-width on user-input imports * fix(scheduled-tasks): log agent-context resolution failures in execute route * fix(scheduled-tasks): keep context double-cast adjacent to its boundary annotation * fix(scheduled-tasks): preserve @-mention contexts on edit; sync editor valueRef on input * fix(scheduled-tasks): footer-wrap modal controls; audit fixes; cleanup - chip-modal: footer secondary cluster now wraps (min-w-0 flex-wrap) with a non-shrinking action cluster, so scheduling controls can never clip Cancel/ primary; recurrence labels compacted so the common case stays one row - schedule-execution: failure path now completes a recurring job when maxRuns/ endsAt/exclusions are exhausted (and a one-time/maxRuns job), mirroring the success path instead of leaving it active with a stale nextRunAt - prompt-editor: commitValue keeps valueRef in lockstep with state on the mention-hook setter paths, completing the stale-ref fix - task-modal: preserve @-mention contexts on edit (seed editor.setContexts); single emptiness source of truth - recurrence-control: preserve prior count when toggling end type; drop a needless useMemo - contracts: reuse scheduleContextSchema for the execute contexts shape * feat(scheduled-tasks): valid future default launch time; test scheduleToTasks mapping * chore(scheduled-tasks): convert added inline comments to TSDoc * simplify(scheduled-tasks): reuse date-fns for launch/end math; drop dead 'running' status + single-use helper --------- Co-authored-by: waleed <walif6@gmail.com> |
||
|
|
51733b8d85 |
fix(db): correct misleading error message when DATABASE_REPLICA_URL is malformed (#5023)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * fix(db): correct misleading error message when DATABASE_REPLICA_URL is malformed The error message said reads fall back to the primary when unset, but the code throws a fatal error instead. The misleading parenthetical contradicted actual behavior and could waste time during incident response when an operator sees this message and expects graceful degradation. --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai> |
||
|
Theodore Li
|
2c75a4ac04 |
fix(tables): per-batch delete-job commits, real trigger.dev retries, post-index ANALYZE guard (#4997)
* fix(tables): per-batch delete-job commits, real trigger.dev retries, post-index ANALYZE guard * fix(tables): resume job progress across retries, rethrow root cause for clean failure messages |
||
|
Theodore LiandClaude Opus 4.8
|
53fdcab5d6 |
feat(tables): background jobs (delete/export/backfill on trigger.dev) + tenant-scoped query performance (#4915)
* feat(tables): paginated background row-delete jobs via table_jobs
* fix(tables): address review on async row-delete (filtered count, scoped optimistic clear, Cmd+A select-all, hide delete from tray)
* improvement(tables): filter-aware select-all runs, delete-job read mask, keyset index + autovacuum tuning
* feat(tables): run import/delete/export/backfill jobs on trigger.dev with in-process fallback
* improvement(tables): raise delete page to 10k and export batch to 5k
* improvement(tables): raise CSV import batch to 5k rows (param-cap bounded)
* feat(tables): surface export jobs in the header tray with progress, cancel, and download
* improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray
* Revert "improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray"
This reverts commit
|
||
|
Vikhyath Mondreti
|
f7b40fe4a4 |
fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions (#4975)
* fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions * update docs |
||
|
Waleed
|
9efb7b45f4 |
feat(byok): support multiple keys per provider with round-robin rotation (#4963)
* feat(byok): support multiple keys per provider with round-robin rotation * fix(byok): address review feedback — serialize cap check, defer encryption, explicit delete guard * fix(byok): drop ON CONFLICT on removed unique index in legacy key migration script * improvement(byok): guard double-submit, Enter-to-save on key field, cap hint in manage modal |
||
|
Waleed
|
284edf0b85 |
improvement(db): opt-in read-replica client + migration runner hardening (#4955)
* improvement(db): add opt-in read-replica client and harden migration runner * fix(db): detect wrapped lock-timeout errors, jittered retries, direct migration DSN support * fix(db): audit fixes — unparameterized SET, primary reads for authz scoping, shared error helper * fix(db): pin migration session — disable max_lifetime recycling, guard backend pid across retries * fix(db): gate pid session guard to direct migration connections (pooler pids legitimately vary) * improvement(billing): route display-only usage aggregations to the read replica via executor threading * chore(db): drop call-site justification comments * chore(db): tighten doc comments * ci(migrations): map optional direct-connection DSN secrets * fix(data-drains): add stability window to time cursors so late-visible rows are never skipped * fix(billing): resolve pagination cursor on primary; replica for member ledger display * fix(billing): usage-limits returns cost and limit from one computation |
||
|
Waleed
|
3cedac8e82 |
fix(security): authz, IDOR, and abuse-prevention fixes (#4944)
* fix(knowledge): require write access for batch chunk operations The PATCH /api/knowledge/[id]/documents/[documentId]/chunks handler performs enable/disable/delete operations but authorized callers with only read-level access (checkDocumentAccess). This let read-only workspace members destroy or disable indexed chunks. Switch to checkDocumentWriteAccess (write/admin required), matching the sibling POST/PUT/DELETE chunk mutation endpoints. * fix(env): restrict decrypted workspace env vars to secret admins GET /api/workspaces/:id/environment returned decrypted workspace environment variables to any member, including read-only collaborators, leaking API tokens, database URLs, and other secrets. Mask workspace variable values for non-admin viewers while preserving the variable names, so editor autocomplete and conflict detection keep working. A value is revealed only when the caller is a credential admin of that key, or — for legacy keys with no per-secret ACL — holds workspace admin permission. This mirrors the per-key edit gating already enforced by PUT/DELETE: if you can administer a secret, you can read it. Personal variables and execution-time resolution are unchanged. * fix(files): block cross-tenant deletion via client-controlled context POST /api/files/delete trusted a client-supplied `context`, letting any authenticated user delete another tenant's file by naming an arbitrary key with `context: "og-images"`. verifyFileAccess() short-circuited the three public contexts (profile-pictures, og-images, workspace-logos) to `true` before any ownership/requireWrite check. - Derive the storage context strictly from the trusted key prefix in the delete route; reject a supplied `context` that disagrees with the key. - Gate the public-context short-circuit to reads only. Destructive ops (requireWrite) now prove ownership via verifyPublicAssetWriteAccess: workspace-logos require write/admin on the bound workspace, profile-pictures require an exact owner match, og-images always deny. Reads of public assets are unchanged. * fix(telegram): verify X-Telegram-Bot-Api-Secret-Token on inbound webhooks Telegram triggers accepted any forged update from anyone who knew the webhook URL path: verifyAuth was a no-op that always returned null, and setWebhook registered no secret_token. Generate a per-webhook secret in createSubscription, register it with Telegram as secret_token, and persist it to providerConfig. verifyAuth now fails closed — rejects when no token is configured, when the X-Telegram-Bot-Api-Secret-Token header is absent, or when it does not match via constant-time safeCompare. * fix(security): pin DNS for Agiloft directExecution and Grafana update tools The Agiloft directExecution tools (read/create/search/update/delete/lock/ saved_search/select/get_choice_line_id/remove_attachment/attachment_info) and the Grafana update_dashboard/update_alert_rule postProcess hooks issued outbound HTTP to a fully user-controlled host (instanceUrl/baseUrl) via the global fetch(), guarded only by the synchronous validateExternalUrl() — which never resolves DNS, so a hostname resolving to an internal/reserved IP passed validation (SSRF). Route all of these through the codebase's standard SSRF-safe path: - Agiloft: moved executeAgiloftRequest into utils.server.ts where the existing pinned helpers live. It now resolves+validates the instance URL once and pins every hop (login, operation, logout) to that IP via secureFetchWithPinnedIP. The 11 tool configs now import it from utils.server; URL builders stay in the client-safe utils.ts. - Grafana: the postProcess POST/PUT now uses validateUrlWithDNS + secureFetchWithPinnedIP, matching the already-pinned initial GET. This completes the Agiloft SSRF pinning started in #4639 (which covered the attach/retrieve API routes) by closing the directExecution path, and extends the same guard to the Grafana update tools. * fix(api): enforce workspace allowPersonalApiKeys policy on v1 surface The external v1 API authenticated API keys without evaluating the per-workspace allowPersonalApiKeys setting, so a personal API key could read and mutate a workspace's resources (workflows, tables, files, knowledge, logs) even when the workspace had explicitly disabled personal keys. The same control is already enforced on the workflow-execution surface. Enforce the policy in checkWorkspaceScope (covering validateWorkspaceAccess too): reject personal keys with 403 when the workspace has allowPersonalApiKeys=false. checkWorkspaceScope becomes async; all v1 route callsites updated to await it. * fix(billing): close usage-cap admission race with atomic reservation The server-side usage-limit gate read already-recorded cost, but cost is only written when an execution finishes. A burst of concurrent executions all observed the same pre-burst usage, all passed the cap, and all ran — collectively spending far past the limit before any cost landed in the ledger (free-tier abuse / hard-cap defeat). manual/chat triggers also skip rate limiting, removing the only throttle. Add an atomic check-then-reserve admission step (Redis Lua) that bounds in-flight, un-costed executions per billing entity by both a per-plan concurrency cap and remaining usage headroom, so recordedUsage + reservedSlots * estimate <= limit always holds. The slot is released at execution completion via LoggingSession (skipped on pause; TTL self-heals crashes). Runs for all trigger types, covering the previously-unthrottled manual/chat paths. Fails open when billing is disabled or Redis is unavailable, matching the rate limiter — a Redis blip can't turn into an execution outage, and the recorded-usage gate still runs. * fix(workflows): validate folderId belongs to workflow's workspace on create/update/reorder Reject a folderId that references a folder in a different workspace (or an archived/non-existent folder) before writing it to workflow.folderId. Previously create, update, and reorder only checked workspace permission on the workflow and the folder's lock status, never that the folder lived in the workflow's own workspace, allowing a dangling cross-workspace folder reference. Adds isFolderInWorkspace/assertFolderInWorkspace + FolderNotFoundError to @sim/workflow-authz (mirroring assertTargetFolderMutable in the duplicate path), enforced in performCreateWorkflow, performUpdateWorkflow, and the reorder route. Invalid folders now return 400. * fix(folders): validate parentId against workspace on create/update/reorder Folder write endpoints accepted a caller-supplied parentId and persisted it without verifying the parent existed in the same workspace, and the create and reorder paths had no cycle guard. A workspace member with write access could reparent a folder to a foreign-workspace folder, a non-existent id, or (via reorder) into a cycle, hiding the folder and its workflows from all members. - performCreateFolder: reject self-parenting and validate the parent exists in the workspace and is not archived (mirrors the duplicate route). - performUpdateFolder: add the same workspace/archived parent check alongside the existing circular-reference guard. - folders/reorder: validate every target parent against the workspace, detect cycles in the resulting parent graph (catches batch cycles), and normalize falsy parentId to null to prevent orphaning. Adds tests for cross-workspace parent rejection and batch-cycle rejection. * chore(knowledge): drop non-TSDoc inline comments from chunks route * fix(webhooks): fail closed when HMAC signing secret is not configured Inbound webhook signature verification failed open for HMAC providers (GitHub, Intercom, Jira, JSM, Confluence, Cal.com, Notion, Greenhouse, Typeform, Fireflies, Circleback): when no signing secret was stored, verifyAuth returned null and the workflow executed on a fully attacker-controlled body. Reject these deliveries with 401 instead, matching the fail-closed Stripe/WhatsApp/Vercel providers. Run provider reachability/verification handshakes (Notion verification_token, Grain/Intercom ping) ahead of auth so the pre-secret setup handshake still completes — those return a canned 200 without executing the workflow, and real event payloads fall through to fail-closed verification. Update the trigger secret-field copy to state the secret is required for deliveries to be accepted (was misleadingly marked optional). * style(files): trim verbose inline comments on delete authorization fix * fix(auth): close account-enumeration oracle on email sign-up The custom before-hook pre-check threw a distinguishing 422/USER_ALREADY_EXISTS for already-registered emails, letting an unauthenticated attacker enumerate accounts — defeating better-auth's own OWASP enumeration protection (active under requireEmailVerification). Remove the pre-check and rely on better-auth's generic duplicate-sign-up response, wiring: - onExistingUserSignUp: notify the real account owner out-of-band, mirroring the privacy-preserving forget-password flow. - customSyntheticUser: include admin (role/banned/banReason/banExpires) and Stripe (stripeCustomerId, billing-gated) user fields so the fake response shape is byte-identical to a real new-user response. Adds an ExistingAccountEmail template + 'existing-account' subject. * style(tools): drop non-TSDoc inline comments from Grafana/Agiloft SSRF tools * chore(api): trim extraneous inline comments in v1 logs/files routes Remove a redundant size annotation and two verbose multi-line materialization comments whose intent is already clear from the code. Load-bearing comments (race-condition and key-translation notes) kept. * fix(billing): exclude table-cell dispatch from admission reservation Table-cell dispatch is row-bounded, async rate-limited, and already surfaces a graceful usage state. Applying the in-flight concurrency reservation there turned its 429 into a hard cell error on a normal >15-concurrent-cell run (only 402 was handled gracefully). Skip the reservation for that surface via a new skipConcurrencyReservation option (the usage-cost cap is still enforced), and tidy the reservation comments to TSDoc. * fix(chat): rate-limit and constant-time password auth for public chats Password-protected public chat (POST /api/chat/[identifier]) had no throttling on the password check and compared with a non-constant-time !==, allowing unlimited brute-force and per-character timing leaks. - Add per-IP rate limiting (10 / 15min) to the password branch of validateChatAuth, mirroring the OTP/SSO endpoints; return 429 with Retry-After. Only explicit unlock attempts consume tokens — message sends carry no password and ride the auth cookie. - Replace password !== decrypted with safeCompare. - Fails open on rate-limiter storage errors; no availability regression. * fix(security): cap JSON request body size and gate public chat endpoint The shared parseJsonBody helper (behind parseRequest, used by nearly every contract route) read request bodies with no size limit, buffering the full body into memory before validation. The unauthenticated public deployed-chat endpoint reached this sink with no admission gate, enabling an anonymous memory-exhaustion DoS. - parseRequest/parseJsonBody now enforce a byte cap via a size-limited stream read (content-length precheck + streamed cap), returning 413. Default is API_MAX_JSON_BODY_BYTES (50 MB), overridable per route via maxBodyBytes. Decoding uses TextDecoder to match request.json() BOM handling. - Public chat POST is wrapped with the admission gate (tryAdmit) and passes an explicit CHAT_MAX_REQUEST_BYTES (20 MB) cap. - Chat body contract gains .max() bounds on input, password, conversationId, file data/name/type, and files array length. - Admin bulk workspace import opts into a higher 100 MB cap to avoid regressing large multi-workflow imports. * fix(chat): rate-limit and constant-time password auth for public chats Password-protected public chat (POST /api/chat/[identifier]) had no throttling on the password check and compared with a non-constant-time !==, allowing unlimited brute-force and per-character timing leaks. - Add per-IP rate limiting (10 / 15min) to the password branch of validateChatAuth, mirroring the OTP/SSO endpoints; return 429 with Retry-After. Only explicit unlock attempts consume tokens — message sends carry no password and ride the auth cookie. - Replace password !== decrypted with safeCompare. - Fails open on rate-limiter storage errors; no availability regression. Reinstates the fix reverted by an intervening commit. * fix(billing): never block a lone execution on usage headroom The admission reservation tapered allowed concurrency by remaining usage headroom. With under one credit of headroom left (but not yet over the cap), floor(headroom / estimate) hit zero and rejected even a single, zero-concurrency execution — stricter than the recorded-usage gate, which would have allowed that last run, and with a misleading "too many concurrent executions" message. Floor the headroom term at 1 so a lone execution is governed only by the cost gate; concurrency above the first slot still tapers with headroom. * refactor(env): document workspace env masking, drop inline comments Extract the workspace-env value masking into a TSDoc-documented maskWorkspaceEnvForViewer helper and remove the redundant inline comments from the GET handler and its test. No behavior change. * refactor(env): convert PUT/DELETE authz comments to TSDoc Move the tiered-authorization rationale for the workspace env upsert and delete handlers into TSDoc blocks and drop the inline comments. No behavior change. * fix(telegram): keep legacy webhooks working via Telegram source-IP fallback The secret-token check rejected every webhook registered before secret_token support, breaking live triggers until re-saved. Fall back to verifying the request originates from Telegram's published webhook IP ranges when no secret is configured, so existing triggers keep firing with no re-save or migration while forged updates from arbitrary hosts are still rejected. Webhooks with a registered secret continue to use strict constant-time token verification. * fix(chat): restore constant-time password auth and IP rate limit A billing commit ( |
||
|
Vikhyath MondretiandCursor
|
6abcf82db2 |
feat(workflows): sim trigger, logs v2 block, toolbar renaming (#4941)
* feat(workflows): sim trigger, logs v2 block, toolbar renaming * fix(review): bound rule queries, canonical logs params, watched-workflow SQL scoping Code-review fixes: read the canonical workflowIds param in logs_v2 (the serializer deletes the source pair ids), aggregate failure-rate in the DB and switch rule windows to the indexed startedAt column, clamp rule config to the legacy contract bounds, push no_activity watch scoping into SQL before the LIMIT, fix the generated sim icon-map key, normalize docs wording, and drop dead exports. Co-authored-by: Cursor <cursoragent@cursor.com> * address comments * fix(review): integer rule rounding, success-gated workflow labels, display module hygiene Second-pass review fixes: round integer rule fields so fractional input never reaches SQL LIMIT, gate workflow-name readiness on a successful non-placeholder load in both editor and preview (errored loads mislabeled valid workflows as deleted), lazily read the variables store in preview rows, move the filter-field JSON preview into the shared display module and unexport its single-consumer helpers, and align >= boundary copy (failure rate, error count, cooldown window) with implementation. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: sync lockfile after staging merge Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): keyset-paginate the no_activity subscription scan A fixed LIMIT 500 with no ORDER BY silently starved subscriptions beyond the cap once the global count exceeded it. The poll now pages by webhook id so every subscription is visited each cycle; pagination bounds memory, not total work. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): keyset-paginate the watched-workflow scan The 500-row LIMIT silently and deterministically excluded high-id workflows from no_activity coverage in watch-everything subscriptions on large workspaces. The scan now pages by workflow id, mirroring the subscription scan; per-workflow checks move into a helper so the pagination loop stays flat. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): skip no_activity subscriptions on the execution-completion path no_activity is poller-owned and can never fire from a completed execution, but it passed into the rule branch and cost a pointless cooldown point-read per subscription on the hottest path. Early-continue alongside the workflow_deployed guard. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(sim-trigger): note failure-based alert conditions evaluate on failed runs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(blocks): recategorize Data Enrichment as a core block It's a Sim-native capability (registry enrichments over a managed provider cascade, like Search), not a third-party integration. Moves it to Core Blocks in the toolbar, out of the integrations catalog, and relocates its docs page to blocks/ with the icon-map allowlist keeping the docs card icon. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(blocks): recategorize MySQL, PostgreSQL, SFTP, SMTP, SSH as integrations External-system connectors with host/credential auth belong under Integrations, not Core Blocks — consistent with MongoDB, Redis, ClickHouse, and the other datastore integrations. They already carried integrationType and /tools docsLinks; the regenerated docs pages turn those previously-dangling links into real pages, and the blocks join the integrations catalog and icon maps. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Waleed
|
b2a485e164 |
fix(db): serialize concurrent migrations with a Postgres advisory lock (#4939)
* fix(db): serialize concurrent migrations with a Postgres advisory lock Deployments start N app replicas at once, each with a migration sidecar. drizzle migrate() has no cross-process lock, so all N read __drizzle_migrations, all see the same migration pending, and all apply it concurrently — one wins, the losers run the same DDL against already-mutated state and exit 1 (e.g. DROP TABLE "form" -> table does not exist / TaskFailedToStart). Wrap migrate() in a session-level pg_advisory_lock so runners serialize: the winner migrates, the losers block, then re-read and find nothing pending. Session locks auto-release on disconnect, so a crashed runner never wedges the lock. * fix(db): guard pg_advisory_unlock so it cannot mask a successful migration If the explicit unlock throws (e.g. connection drops in the window after migrate() commits), the exception bubbled to the outer catch and exited 1 — falsely reporting a failed migration to the deploy orchestrator. The session lock auto-releases on disconnect anyway, so swallow and log instead. * refactor(db): move unlock-guard rationale to TSDoc helper |
||
|
+1
|
39418f9615 |
improvement(mothership): v0.2 (#4923)
* CONTRACTS * updates * Prompt caching * Fix regression * updates to prompt caching * Prompt caching trace * VFS updates * Changelog and plan * VFS update * Improvement/mothership (#4775) * ff * System role in cache * Fixes * Add dynamic user info * Improve tool search * improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * New doc setup * Fixes * Fixes * Logs tools * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * Add finalization on error * Fixes * change dev CI to bun run db:push * Updates Please enter the commit message for your changes. Lines starting * Contrat update * improvement(nested): subagents * updates * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Theodore Li
|
f7811f8acc |
feat(tables): stable column ids for metadata-only rename (#4898)
* feat(tables): stable column ids for metadata-only rename * fix(tables): address review — id-key exec clears + waiting labels, name in upsert error, un-gate group output ids * fix(tables): id-correct column undo (rename/create/delete) with id reuse on re-add * refactor(tables): mint column ids via generateId (uuid), drop collision-check plumbing * fix(tables): match column id or name when removing column from optimistic delete cache * fix(tables): resolve column storage id once for delete optimistic strips; biome-format 0228 snapshot * fix(tables): make in-grid find id-native (scan/return JSONB keys as column ids) * fix(tables): translate id→name in CSV/JSON export read path (+regression test) |
||
|
Theodore Li
|
ec256d244f |
fix(tables): compare order_key bytewise (COLLATE "C") to stop insert collation errors (#4908)
* fix(tables): compare order_key bytewise (COLLATE "C") to stop insert collation errors * fix(tables): detect mis-keyed tables by position order, not order_key sort * improvement(tables): make repair-script dry-run lock-free and clearly labeled |
||
|
|
0075ab9cf6 |
improvement(platform): remove tour, simplify sidebar/header, drop loading skeletons (#4354)
* improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Theodore Li
|
24a6086d95 |
feat(tables): fractional order keys for O(log n) row insert/delete (flag-gated, default off) (#4890)
* feat(tables): add order_key column, fractional-indexing util, and ordering flag (off)
* feat(tables): write order_key on insert, flag-gate delete reindex + query ordering, add backfill
Flag off (default) = identical behavior. Single-insert assigns a fractional
order_key; queryRows orders by order_key when the flag is on; deletes skip the
O(N) reindex when on. Per-table-atomic backfill script populates existing rows.
* feat(tables): write order_key on all insert paths (batch, upsert, replace, import, create, copilot)
Completes the always-write-keys prerequisite: every row insert now assigns a
fractional order_key consistent with position order, so the flag can be flipped
safely after backfill. Flag off (default) still = identical behavior.
* feat(tables): insert-by-neighbor-id + orderKey on wire + client order-by-key
Inserts express intent as afterRowId/beforeRowId (O(1) key mint via the
(table_id,order_key,id) index); orderKey is returned on every row; client
reconcile/undo place by orderKey (no neighbor bump) with position fallback.
Flag off = unchanged. 205 table tests pass.
* feat(tables): resolve position-based inserts by key ordinal under the flag
Position-based callers (mothership tool, v1 API, undo fallback, transient old
clients) resolve their insert neighbor by order_key ordinal (OFFSET) when the
flag is on — positions are gappy then, so WHERE position=N would miss. Flag off
keeps the indexed position lookup. The mothership tool itself is unchanged.
* test(tables): flag-on coverage — delete skips reindex, insert mints key + no shift
* fix lint
* chore(db): regenerate order_key migration with default drizzle name
* fix(tables): address review — guard neighbor insert + mutual-exclusion + safe reconcile
- resolveInsertByNeighbor throws when the anchor row is missing (was silently
inserting at the front) and when its order_key is null under the flag.
- insert contract: afterRowId/beforeRowId are mutually exclusive (refine).
- reconcileCreatedRow only key-sorts when every cached row is keyed, so mid-
backfill un-keyed rows aren't yanked to the front.
* fix(kb): restore non-null guard in storage-key filter (unsafe-lint regression)
* refactor(tables): extract maxOrderKey + thread import append key
- Extract maxOrderKey(executor, tableId) helper; replaces three identical
max(order_key) selects (single/batch insert append + import).
- Import: read the append anchor once up front and thread each batch's last
key forward (nextImportStartOrderKey + afterOrderKey) instead of re-scanning
max(order_key) per batch over a growing table — one scan per import, not
one per 1k-row batch.
* fix(tables): keep insert body base omittable for v1 contract
The afterRowId/beforeRowId mutual-exclusion .refine() turned the schema into a
ZodEffects, which Zod forbids .omit() on — v1's insertTableRowBodySchema.omit({
position }) threw at module load (runtime-only; tsc misses it). Split the plain
object base out, apply the shared refine on top, and have v1 omit from the base
then re-apply it.
* fix(tables): chunk backfill order-key writes
A single UPDATE … FROM (VALUES …) over a whole large table overflows the JS call
stack while drizzle assembles the VALUES list (and would blow past Postgres's
65535 bound-param limit at ~32k rows) — large tables failed with 'Maximum call
stack size exceeded'. Write in 1000-row chunks inside the same per-table
transaction so keying stays atomic.
* fix(tables): emit orderKey in insert responses
The single-row and batch insert handlers dropped orderKey from the JSON
response even though the service returns it, so reconcileCreatedRow always fell
back to position-sorting and could place neighbor inserts wrong under the
fractional-ordering flag. Serialize orderKey alongside position.
* fix(tables): restore by orderKey, not position, under fractional flag
A saved position is the gappy column value, but under the flag insert reads
position as a visual rank (OFFSET) — so position-based restore misplaces rows.
- create-row redo now goes through the batch path carrying the saved orderKey
(the single-insert API has no orderKey field); drop the now-unused single
create mutation.
- resolveBatchInsertOrderKeys appends under the flag instead of feeding gappy
positions to resolveInsertOrderKey; positions remain the flag-off path.
* perf(tables): backfill writes 5000 rows/chunk (was 1000)
5x fewer round-trips per table; ~10k bound params stays well under Postgres's
65535 ceiling and far below the single-statement size that overflows the stack.
* fix(tables): drop rowNumber from table trigger payload
position is gappy under the fractional-ordering flag, so rowNumber (= row.position)
no longer reflects a contiguous visual rank. Rather than compute-on-read, remove
it from the trigger payload, output schema, and column-execution input.
Also pin isTablesFractionalOrderingEnabled=false in update-row.test.ts so its
flag-off position-shift assertions are deterministic regardless of local env.
* chore(db): format generated 0226 migration metadata
biome check . flagged the drizzle-generated _journal.json and 0226_snapshot.json;
apply the formatter so packages/db lint:check passes in CI.
* docs(triggers): drop rowNumber from table trigger outputs
rowNumber was removed from the table trigger payload; remove it from the
documented output fields to match.
* test(tables): remove flag-on fractional-ordering unit suite
Flag-on behavior is covered by manual large-table verification; the heavily-
mocked DB-chain suite added little signal.
|
||
|
Waleed
|
5505076222 |
chore(db): drop legacy copilot_chats.messages JSONB column (#4886)
Reads and writes are fully cut over to the normalized copilot_messages table (verified in production: no writes to the column in 24h, recently-active chats have empty JSONB while copilot_messages holds the transcript). Drop the dead column via drizzle migration 0225 and re-type CopilotChatDetailRow.messages as an assembled (non-column) field. Deploy notes: reconcile any chats where the JSONB still leads copilot_messages before applying, and pg_repack copilot_chats afterward to reclaim the ~5.7GB TOAST storage (DROP COLUMN is metadata-only). |
||
|
Theodore LiandClaude Opus 4.8
|
3518b999af |
feat(tables): background import for large CSVs with live progress (#4861)
* feat(tables): background import for large CSVs with live progress * fix(tables): address review — import heartbeat, overlap guard, column/empty validation * fix(tables): guard sync import overlap, scope fileKey to workspace, delete-on-replace after download * fix(tables): stream large CSV imports from storage instead of buffering the whole file * test(tables): fix async-import route tests for workspace-scoped fileKey + name uniquification * fix(tables): append imports start after existing rows; reconcile missed import failures in the tray * fix(tables): delete the uploaded CSV from storage after the import finishes * fix(tables): validate replace before deleting rows; ignore stale replayed import events by importId * fix(tables): bind import worker to its importId (no stale-worker clobber/overlap) and destroy storage stream on failure * feat(tables): byte-based import progress, cancel support, and a start toast that opens the import view * fix(tables): don't emit ready after cancel; honor cancel during the upload phase * improvement(tables): use a stop (square) icon for canceling an active import * fix(tables): make markTableImporting an atomic claim to close the concurrent-import TOCTOU race * improvement(tables): preview CSV import from a slice, drop client row-count warning The import dialog parsed the entire file in the browser to show an exact row count and a row-limit warning. That holds the whole file in memory, blocks the main thread, and hits V8's ~512MB string ceiling — so the dialog capped the effective import size well below what the streaming importer handles. Parse only the first 512KB (headers + sample for the mapping); drop the exact count and the "would exceed the row limit by N" gate. The DB row-count trigger already enforces max_rows server-side, so an over-limit import fails fast during the run with a clear message instead of being blocked by an expensive parse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): gate import ownership every batch and stop canceled imports reappearing - Worker checked run ownership only at the progress cadence (~every 5k rows), so a canceled/superseded import could insert several more batches (incl. the final partial batch) before stopping. Move the updateImportProgress ownership gate to the top of every flush — a run that lost the table stops within one batch. - A list/dialog import canceled mid-upload left the server row `importing` until the in-flight server cancel landed; hydration re-seeded it from useTablesList, so the dismissed import flickered back. Flag the real table id canceled on the mid-upload cancel path, skip re-seeding flagged tables in hydration, and clear the flag once the server import is terminal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(tables): drive import tray by polling derived from server, not SSE Import progress no longer holds an SSE connection per importing table. The tray now derives its importing rows live from the table list (React Query), polled only while an import is in flight; the table detail page keeps its own cell-state SSE for grid refresh. - store holds only client-only state now: optimistic uploads, which terminal completions to surface this session, canceled ids, menu open — no copied importStatus/rowsProcessed. - useWorkspaceImports is the single source: polls via a data-predicate refetchInterval, derives rows, and fires completion toasts on the importing -> terminal transition. - kickoff handlers use startUpload/setUploadPercent/endUpload; the invalidated list refetch surfaces the server row and polling takes over. - removes use-hydrate-import-tray + use-import-progress-tracker (folded in). - trims over-verbose comments across the import paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): ignore superseded-run import events in the detail SSE cache applyImport applied every replayed import payload to the detail cache. The SSE buffer can replay a prior import's terminal event for the same table, stomping a newer in-flight import's UI. Lock to the active run's importId (and ignore a replayed terminal before the id is known), matching the guard the header tracker used to have. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): close sync-import TOCTOU by claiming the atomic import gate The sync import route checked importStatus from a checkAccess snapshot, then parsed/validated/wrote seconds later without taking the atomic claim. A concurrent async kickoff (markTableImporting) could slip into that window and both writers would run together — for replace mode, two delete+insert passes leave the table indeterminate. Claim the same atomic gate (markTableImporting) right before the write and release it in the finally (before the response returns, so a client refetch never sees the transient status). A row-level FOR UPDATE was avoided on purpose: it would invert lock order against the position advisory lock / row-count trigger and risk a deadlock — markTableImporting is the established gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(multipart): keep abort wired after resolve so a mid-upload disconnect tears down the stream readMultipart resolves on the file-part header and hands the caller an un-drained stream, but settle() ran cleanup() and detached the abort listener on that path too. A client disconnect mid-upload then destroyed nothing — busboy never saw EOF, the file stream stalled, and the route's `for await` held a request slot until maxDuration (300s). Re-arm an abort handler scoped to the file stream on resolve, detached when the stream closes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Theodore LiandClaude Opus 4.8
|
aed44024d4 |
fix(tables): surface real error causes on cell-execution failures (diagnostics) (#4868)
* fix(tables): retry transient DB/Redis failures in cell execution and surface error causes Workflow-group-cell runs intermittently failed on trivial DB reads/writes under heavy fan-out, stranding cells in `running`. Investigation showed the PlanetScale and ElastiCache backends were healthy at the time — the failures are transient connection-level faults that the cell (maxAttempts: 1) had no tolerance for, and the real cause was never logged (Drizzle wraps it as "Failed query: ..." and the driver cause lives in error.cause). Resilience: - Add retryTransient (lib/table/retry-transient.ts): retries only transient infra errors (reuses isRetryableInfrastructureError; adds an ioredis command-timeout match) with jittered backoff, then rethrows. Fail-fast for everything else. - Wrap the cell's getTableById/getRowById reads, the terminal write (cell-write updateRow — idempotent via the executionId guard), and the Redis cascade-lock acquire. Diagnostics: - Add describeError (lib/core/errors/retryable-infrastructure.ts): walks the .cause chain and always returns the underlying driver cause (code/errno/ syscall + causeChain), including for unclassified errors like AbortError. - Log `cause` + a `retryable` flag (and aborted/timedOut in the cell's main catch) across the cell + finalization error paths, mirroring the existing schedule-execution pattern. Logging-only; no behavior change. This lets the next recurrence reveal the real cause and whether the retry applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): address review feedback on cell retry resilience - retryTransient: re-check the abort signal after the backoff sleep so a cancellation during sleep stops the next attempt (don't run/return work for an already-cancelled task). - isRetryableRedisError: walk the .cause chain (mirroring the infra classifier) so wrapped Redis timeouts are recognized; drop "Connection is in subscriber mode" — that's a connection-state programming error, not a transient drop, and would just fail identically every retry. - cascade-lock: stop wrapping acquireLock in retryTransient. acquireLock is a non-idempotent SET NX, so retrying after a timed-out-but-applied first SET returns false (key already ours) and yields a false `contended` that skips the cascade. A transient Redis blip here just fails the run before pickup (no stranded cell); the dispatcher re-drives it. - Tests: cause-chain Redis match, subscriber-mode exclusion, abort-during-sleep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): drop out-of-scope abort/timeout fields from cell catch The main catch logged `aborted`/`timedOut` from `abortSignal`/`timeoutController`, but those are declared inside the outer try block (the inner try around executeWorkflow is try/finally, so this catch belongs to the outer try) and are not in scope in the catch — `next build`'s type-check failed with "Cannot find name 'abortSignal'". Local incremental `tsc --noEmit` had skipped the file and falsely passed; the Cursor/Greptile reviewers flagged this correctly. Removed the two fields. Abort/timeout is still surfaced via `cause: describeError(err)` (an aborted run shows `name: 'AbortError'` / the timeout message), so no diagnostic signal is lost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(tables): drop in-process retry, keep cause diagnostics only In-process retry is the wrong layer for this path: the cell task is maxAttempts:1 by design, retrying on a possibly-degraded worker may not help, and it masks the very transient-failure signal we're trying to capture before we understand the root cause. Removed retryTransient entirely (file + all wrapping in cell-write, the cascade reads, and the lock acquire) and kept only the diagnostic logging. - Deleted lib/table/retry-transient.ts (+ test); cell-write and the cascade reads call getTableById/getRowById/updateRow directly again, fail-fast. - Kept describeError + `cause`/`retryable` fields across the cell + finalization catch blocks; the cell-path `retryable` flag now sources from isRetryableInfrastructureError (the canonical classifier) for consistency. Diagnostics-first: surface the real driver cause on the next recurrence, then decide the actual fix (e.g. task-level maxAttempts, or addressing the worker- side cause) from evidence rather than a speculative in-process retry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): log error cause on scheduled-execution failure paths The scheduled-job failure paths logged the raw error (.message/stack only) — its `.cause` (the real driver error behind a Drizzle "Failed query: ..." wrapper) was never recorded, and the classified-only `describeRetryableInfrastructureError` returns undefined for unrecognized errors. A real failed run (same incident window as the cell failures) failed in `applyScheduleUpdate` with exactly this unrecorded cause. Added `cause: describeError(error)` (always-on, walks the cause chain) to the applyScheduleUpdate catch, the early-failure catch, and the unhandled-error catch — passed as a second arg so the existing message+stack still emit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(errors): move describeError to @sim/utils/errors `describeError` is a general-purpose error/cause-chain helper — it didn't belong in `lib/core/errors/retryable-infrastructure.ts` (that module is specifically about classifying retryable infra errors, and the name read wrong for a generic diagnostic). Moved it to `@sim/utils/errors` alongside `toError`/ `getErrorMessage`/`getPostgresErrorCode`, with its own cycle-safe cause walk. - Added describeError + DescribedError + tests to packages/utils/src/errors.ts. - Reverted the describeError addition from retryable-infrastructure.ts (it keeps only isRetryableInfrastructureError / describeRetryableInfrastructureError, which are accurately named and still used by the schedule retry path). - Re-pointed all consumers (cell, logging-session, pause-persistence, schedule) to import describeError from @sim/utils/errors. The `retryable` classification flag still sources from isRetryableInfrastructureError where used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
648a5a117d |
feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage (#4865)
* feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage Add S3_ENDPOINT and S3_FORCE_PATH_STYLE env vars, wired into the shared upload S3 client so Cloudflare R2, MinIO, Backblaze B2, and other S3-compatible stores work for self-hosted file storage. The endpoint is trusted operator config (no SSRF/HTTPS gate). Makes the multipart Location fallback endpoint-aware, extends the S3 client unit tests, and documents the new vars in Helm values, .env.example, and the English self-hosting docs (incl. browser-reachability + CORS guidance). * docs(storage): add RustFS as an S3-compatible provider example * fix(storage): address review feedback and fix env mock for CI - Add envBoolean to the shared env test mock (createEnvMock) so config.ts's forcePathStyle coercion resolves — fixes failing knowledge/utils.test.ts - Declare S3_FORCE_PATH_STYLE as z.string() (every other env var's pattern); it's coerced via envBoolean at the consumption site, avoiding a boolean type that never matches the string process.env value - Log path-style from S3_CONFIG.forcePathStyle (envBoolean) instead of a separate isTruthy call, so the startup log can't disagree with the client - Make buildObjectFallbackUrl honor forcePathStyle: virtual-hosted-style URL (bucket as subdomain) for R2, path-style only when forcePathStyle is set * docs(storage): add backlinks to S3-compatible providers (R2, MinIO, Ceph, B2, RustFS) and backends |
||
|
Waleed
|
3f3efc98c3 | chore(auth): remove deprecated OAuth MCP provider plugin and backing tables (#4847) | ||
|
Waleed
|
a7b0bd311d |
fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) (#4837)
* fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) - Bump vitest and @vitest/coverage-v8 to ^4.1.0 across all workspaces (only patched release for the critical 'Vitest UI server arbitrary file read/execute' advisory; no 3.x backport exists) - Widen @sim/testing peer range to ^3.0.0 || ^4.0.0 - Migrate constructor mocks to class expressions: vitest 4 uses Reflect.construct for mocks invoked with new, and arrow/function implementations are not constructable (function expressions also get reverted to arrows by biome's useArrowFunction) - Remove deprecated test.poolOptions from apps/sim/vitest.config.ts (options are now top-level in vitest 4) * fix(deps): exclude vulnerable vitest 4.0.x from @sim/testing peer range Tighten the v4 arm of the peer range to >=4.1.0 <5.0.0 so the peer requirement cannot be satisfied by the unpatched 4.0.x builds that GHSA-5xrq-8626-4rwp affects. * fix(testing): make vitest 4 constructor mocks type-check cleanly - logging-session & mcp-oauth mocks: a class passed to mockImplementation has a construct signature that isn't assignable to its (...args) => any parameter, failing tsc. Use named function declarations instead (constructable via Reflect.construct, assignable to mockImplementation, and not rewritten to arrows by biome's useArrowFunction). - database.mock.ts: vitest 4's generic vi.fn typings no longer break the self-referential cycle on the transaction callback's tx param; loosen tx and annotate the callback's return type to resolve the implicit-any errors. * test(isolated-vm): de-flake queue-capacity scheduler tests The 'queue is full' and 'per-owner queued limit' tests relied on 'await sleep(1)' to assume the first request had reached the queue before submitting the overflow request. The first request only enqueues after an async spawn-failure chain (acquireWorker -> spawn exit -> resolve null -> enqueue), which isn't guaranteed within 1ms under CI load — the overflow request then found an empty queue and hit the 200ms queue-wait timeout instead of the capacity rejection. Replace the wall-clock barrier with a deterministic, event-driven one: hold the single global concurrency slot (IVM_MAX_CONCURRENT=1) with an active worker and await an explicit 'dispatched' signal (fired when the worker receives its execute message, after the scheduler counts it active). The follow-up requests then deterministically hit the synchronous enqueue path. Also drops the queue-wait timeout from 200ms to 50ms, so the tests run faster. |
||
|
Vikhyath Mondreti
|
9bed841173 |
improvement(kbs): ownership bindings (#4833)
* improvement(kbs): ownership association * address comments * more comments |
||
|
Waleed
|
75d98739eb |
chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809)
Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. |
||
|
Vikhyath Mondreti
|
15ca66f112 | fix(copilot): seq migration (#4804) | ||
|
Vikhyath Mondreti
|
4b0dab4682 |
chore(copilot): deprecate mcp server (#4797)
* chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route |
||
|
Vikhyath MondretiandCursor
|
49673055be |
improvement(logs): object storage backed tracespans (#4787)
* improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
WaleedandClaude Opus 4.8
|
fd77bb4069 |
feat(copilot): add seq ordinal to copilot_messages for order-preserving reads (#4791)
copilot_messages had no column preserving message order: created_at (set from each message's timestamp) ties at millisecond granularity in 58% of chats, and some chats have out-of-order timestamps within their array. The only other tiebreaker, id, is a random UUID — so ORDER BY created_at, id renders same-timestamp user/assistant pairs swapped. This blocks the R+1 read cutover. Add an integer seq = the message's 0-based index within the chat's JSONB array (ground-truth order), backfilled inline in migration 0219 (no script for self-hosters or us). Reads will use ORDER BY seq NULLS LAST, created_at, id at cutover; reads still come from JSONB after this PR. Design: - seq is a tiebreaker, not the sole sort key (concurrent-append/NULL safety). - Nullable now; defer NOT NULL so rolling-deploy old pods don't fail inserts. - replace (update-messages snapshot) overwrites seq = array index (re-densifies after a mid-conversation delete); append preserves existing seq via COALESCE and assigns base+idx from a single MAX(seq) read (never MAX+i in SQL — multi-row batches would collide). The non-atomic read-then-insert window is documented and bounded by the read tiebreak + snapshot re-densify. - Dedupe message ids before insert (87 prod chats carry dup ids; a repeated id in one INSERT...ON CONFLICT would otherwise throw). - Backfill picks first-occurrence per (chat,id), gap-free via ROW_NUMBER; validated on staging data (0-based, contiguous, 0 bad ranges). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Vikhyath MondretiandCursor
|
53eaa600af |
improvement(billing): migrate hot path writes away from user_stats (#4768)
* improvement(billing): migrate hot path writes away from user_stats * fix period start and end sot * address comments * Remove stale billing migration Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
WaleedandClaude Opus 4.7
|
ddc47eb221 |
feat(copilot): add copilot_messages table with dual-write rollout (#4726)
Splits copilot chat messages out of the copilot_chats.messages JSONB column into a dedicated copilot_messages table. JSONB stays canonical during R+0 — every write path dual-writes to the new table best-effort (try/catch + log warn, never throws). Migration 0217 creates the table + indexes and inline-backfills history from JSONB so OSS self-hosters don't need to run a separate script. Write paths covered: - post.ts (user message append) - terminal-state.ts (assistant turn finalize) - update-messages/route.ts (snapshot replace) - inbox/executor.ts (background turn) - fork/route.ts (chat clone) - superuser/import-workflow/route.ts (chat import) Each call threads chatModel + streamId where relevant; ON CONFLICT DO UPDATE preserves existing stream_id / model via COALESCE. For pre-R+1 reconciliation, run: bun apps/sim/scripts/copilot-messages-reconcile.ts [--since='7 days'] Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Waleed
|
34c47f7e93 |
chore(auth): upgrade better-auth 1.3.12 → 1.6.11 (#4766)
* chore(auth): upgrade better-auth 1.3.12 → 1.6.11 * chore(auth): address Greptile review — broaden change-email type + migration newline * fix(auth): correct oneTimeToken expiresIn unit (minutes, not seconds) Better-auth's oneTimeToken expiresIn is in minutes (multiplied by 60_000ms internally). Sim's existing 24*60*60 evaluated to ~60 days of token lifetime instead of the intended 24 hours. Tokens are one-time-use and typically consumed within seconds of generation (Socket.IO handshake), so this tightens an unused security window without affecting UX. |
||
|
Vikhyath Mondreti
|
59792c052f |
improvement(mcp): bound MCP memory and lifecycle concurrency (#4751)
* improvement(mcp): bound MCP memory and lifecycle concurrency * update db mock * address comments * address comments |
||
|
Vikhyath MondretiandCursor
|
81bcdf255a |
improvement(schedules): retries, concurrency limits (#4755)
* improvement(schedu les): retries, concurrency limits * improvement(schedules): remove stale generated migration Drop the pre-merge generated 0213 migration so it can be regenerated after syncing with staging. Co-authored-by: Cursor <cursoragent@cursor.com> * address comments * fix pinned version for lru cache * retryable errs cleanup --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Theodore LiandClaude Opus 4.7
|
92fd17cd15 |
fix(tables): workflow-column run fixes + bounded run-N-rows (#4754)
* fix(db): disable statement_timeout for migrations
* fix(ci): route migration workflow through guarded migrate.ts
* feat(tables): workflow-column run fixes + bounded "run N rows"
- Pass group.autoRun as the add-group dispatch flag so an autoRun=false
column no longer opens a no-op dispatch that flashes the run-count badge.
- Scope the context-menu re-run to the right-clicked workflow cell's group
(cascading to dependents) instead of every group on the row.
- Add an extensible per-dispatch row cap (DispatchLimit { type:'rows', max })
surfaced as "Run 10 / 1,000 empty rows" in the group header; dispatcher
stops after N eligible rows. New limit/processed_count columns on
table_run_dispatches.
- Fix stranded "Queued" cells: the cascade owner now treats a queued marker
(orphan pre-stamp) as a manual run so autoRun=false requested groups are
picked up, and drains late markers before releasing the row lock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(db): regenerate dispatch limit migration on staging chain (0214)
Re-numbers the table_run_dispatches limit/processed_count columns from the
collided 0212 to 0214 after merging staging (which added its own 0212/0213).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tables): lint formatting
* fix(tables): address PR review on dispatch cap + cascade drain
- Don't consume the row cap when batchEnqueueAndWait fails; a transient
failure no longer completes a capped dispatch with zero rows started.
- Outer cascade-drain loop only re-drives a genuine queued marker, not any
eligible group, so an empty-output group can't re-run forever.
- completeDispatch forwards limit on the terminal SSE event.
- Extract shared LIMITED_RUN_PRESETS for the Run-N-rows menu items.
* chore(lint): format generated tool-schemas-v1
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Theodore LiandClaude Opus 4.7
|
65e2fe81c1 |
feat(tables): Add enrichment table column type (#4752)
* feat(tables): native enrichments sidebar + workflow input mapping Add a Clay-style enrichments catalog to the table view and wire per-row input mapping into workflow-backed columns. - New "Enrichments" entry in the New-column dropdown opens a sliding panel listing curated enrichment templates; picking one swaps to the workflow config in-place (no cross-slide) with a back button. - Type the workflow sidebar as manual | enrichment; enrichment hides the launch + add-column-inputs affordances. - Add a "Workflow inputs" advanced panel mapping Start-block input fields to table columns (left-of-workflow columns only), with name-match auto-fill and collapsible input-mapping-style rows. - Persist type + inputMappings on the workflow group (types, contract, route, service, hook) — jsonb, no migration. - Consume inputMappings at run time: when present, feed Start-block fields from the mapped columns; otherwise fall back to name-match spread. - Clean up inputMappings on column rename/delete (stripGroupDeps + renameColumn). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(emcn): extract CollapsibleCard and reuse for input mapping Pull the collapsible field-card markup (surface-4 header + surface-2 body, click/keyboard toggle, truncated title + optional badge) into a shared `CollapsibleCard` emcn component, and use it in the workflow-builder input mapping rows and the table sidebar's input-mapping panel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(tables): code-defined enrichment registry run directly per row Enrichments are now TS configs in apps/sim/enrichments/ (registry, like connectors) that run directly per table row via the existing run/dispatch/ cell-write rails — no workflow execution. - enrichments/{types,registry} + work-email (heuristic) and phone-number (stub). - WorkflowGroup gains enrichmentId; WorkflowGroupOutput gains outputId (workflowId/blockId/path kept required, '' for enrichment groups). - Executor branches on group.type === 'enrichment' → maps inputMappings → enrich() → outputs by outputId → cell-write. Missing required inputs skip (blank cell) instead of erroring. - Sidebar lists the registry; enrichment-config panel maps inputs to columns and creates the enrichment group (no workflow UI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(enrichments): provider fallback cascade + hosted-key usage source Replace each enrichment's single enrich() with an ordered providers[] fallback cascade. Providers are plain data ({ id, label, toolId, buildParams, mapOutput }) so the catalog stays client-safe; the server-only runner (run.ts) calls executeTool per provider, first non-empty result wins, misses/errors fall through, all-miss = blank cell. Wire four enrichments on the hosted-safe providers (Hunter, PDL): - Work Email (fullName, companyDomain): Hunter -> PDL - Phone Number (fullName, companyDomain): PDL - Company Domain (companyName): PDL - Company Info (domain): PDL -> Hunter Person enrichments take a single canonical fullName (Clay-style); Hunter gets first/last via splitName(), PDL takes name directly. Add 'enrichment' to usage_log_source enum (+ migration) so hosted-key tool cost from these per-row calls can be billed to the table owner. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(enrichments): bill hosted-key cost; surface provider errors; abort safety - runEnrichment now returns { result, cost, error }: accumulates hosted-key cost across the cascade, and sets `error` only when every provider that ran errored (auth/rate-limit/outage) vs a clean miss. - Executor records the cost to the table owner (createdBy) via recordUsage (source 'enrichment'); billing failures are logged, never error the cell. - F1: all-providers-errored now writes status 'error' instead of a blank 'completed' cell that looked like "no data found". - F2: re-check the abort signal after the cascade so a cancel mid-tool-call isn't recorded as a completed empty cell. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(tables): present enrichment columns as first-class in the grid - Meta-header shows the enrichment's name + icon (Mail/Phone/Globe/Building2) instead of "Workflow" + a color chip. - Per-column header icon uses the enrichment's icon (via columnSourceInfo) instead of the generic play icon. - Hide "View execution" for enrichment cells in both the row context menu and the action bar (no workflow execution exists to open); also hide the meta-menu "View workflow" item for enrichment groups. - Clicking an enrichment column header now opens the enrichments sidebar in edit mode (pre-filled input mappings, Update via useUpdateWorkflowGroup) instead of the workflow "Configure workflow" sidebar. - Enrichment config lets the user name each output column (editable per-output, deduped defaults) since enrichments can produce multiple columns. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): enrichment columns use type icon; output names editable - Drop the per-column enrichment icon (it duplicated the meta-header icon). Enrichment output columns now render the standard column-type icon (Text, etc.) — the enrichment's icon stays only on the group meta-header. - Make output column names editable in the enrichment config edit mode too; changed names rename their columns via useUpdateColumn (the rename cascades into the group's output refs server-side). Validation excludes the output's own current name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): wrap enrichment catalog descriptions instead of truncating Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): edit enrichment output columns via the plain column editor Edit column on an enrichment output now opens the normal column-config sidebar (rename / type / unique) instead of the workflow 'Configure output column' panel, which showed workflow-only fields and blocked a simple rename. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(copilot): list_enrichments + add_enrichment table tools Let the copilot enumerate the code-defined enrichment registry and add an enrichment column to a table (validating required input mappings against the table's columns), backed by the same workflow-group machinery the UI uses. * fix(enrichments): address PR review feedback - Guard the enrichment cell path on `enrichmentId` so a group typed 'enrichment' without a registry id falls through to the workflow path instead of erroring. - Clear stale output values when skipping a row for missing required inputs, so the auto cascade re-enriches once inputs return (was left completed+filled). - Write a terminal state on abort in the enrichment path (matches the workflow path) so a cancel between run and terminal-write can't leave the cell running. - Edit mode: apply the group update (mappings/deps/auto-run) before column renames so the primary edit lands even if a rename fails. - Disable Save once validation has surfaced a missing required input. - Use the workflowGroupById map instead of O(n) find in the context-menu and action-bar hot paths. * chore(commands): add /add-enrichment command Guides adding a code-defined table enrichment to the registry, with a required step to verify each provider tool has hosted-key support and chain to /add-hosted-key when it doesn't. * fix(enrichments): address second-pass PR review - updateWorkflowGroup output diff now keys on outputId (falling back to blockId::path) so enrichment outputs — which share empty blockId/path — no longer collapse to one key and drop sibling columns. - Enrichment terminal write now clears output columns absent from the result, so a partial/empty re-run doesn't leave stale values. - Editing a group whose enrichment was removed from the registry shows an explanatory panel instead of silently falling through to the new-enrichment catalog. * feat(tables): show "Not found" badge for empty completed enrichment cells An enrichment that runs to completion but matches nothing now renders a gray "Not found" badge (like the Queued/Waiting cell states) instead of a blank cell, so a real miss is distinguishable from an unrun cell. Scoped to enrichment output columns; an empty string no longer counts as a value. * fix(enrichments): don't re-run completed no-match enrichments on auto cascade A completed enrichment with empty outputs is a real no-match result, not an unfinished run. Eligibility now treats an enrichment's completed status as terminal (regardless of output fill), so the auto cascade stops re-invoking billable provider calls on every no-match row each dispatch. Input changes still clear the exec entry, so genuine re-runs are unaffected; manual Run all still re-runs. * fix(enrichments): treat provider 404 as no-match, not a cell error Providers like People Data Labs signal 'no record found' with HTTP 404, which executeTool surfaces as a failed ToolResponse (status on output.status). The cascade now treats a 404 as a clean miss — falls through to the next provider and lets the cell render 'Not found' — instead of marking the cell errored. Auth/rate-limit/5xx still propagate as real errors. * fix(tools): surface HTTP status on error ToolResponse output executeTool's catch handled Error instances in its first branch and only extracted status/statusText/data for non-Error object throws — so HTTP errors (thrown as Error instances carrying .status) lost their status on the returned output. Surface it for Error instances too, so callers can branch on the status (e.g. the enrichment cascade treating a provider 404 as a no-match). * fix lint * Revert ff --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |