mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
f9b09ecfd4f2ba8174fa9b7fa4e89dd6815d76cb
85
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
5db62b8bdc |
feat(observability): extend audit log, PostHog, and storage metering coverage (#5269)
* feat(observability): extend audit log, PostHog, and storage metering coverage Instruments previously-uncaptured resources and actions across the three observability layers (audit log, PostHog product analytics, usage metering): - Exfiltration audit: file downloads (workspace/public-share/v1), table, workflow, and workspace exports. - Credential access audited at the token-issuance boundary (success-only). - Full revenue trail: invoice paid/failed, overage billed, disputes, credit fulfillment, subscription lifecycle, plan/seat changes. - Session/login lifecycle audit via Better Auth hooks (login, blocked sign-in, logout, session revoke, account delete). - v1/admin programmatic surface and copilot tool handlers instrumented. - Dead audit/PostHog constants wired (lock/unlock, table, custom tool, etc.). - Storage metering extended to KB documents and copilot files. - PostHog person identify + workspace/organization group hygiene. Audit package hardened to null the actor FK for system actors (admin-api) and adds an awaitable recordAuditNow for pre-delete hooks. All instrumentation is fire-and-forget and never blocks or breaks the primary operation. * fix(observability): audit file/credential egress only on success Address Cursor Bugbot review: - GET OAuth token: emit CREDENTIAL_ACCESSED/credential_used after refreshTokenIfNeeded succeeds (matches POST), not before. - File export: audit on each actual success exit via a shared helper — including the non-markdown serve redirect (previously unaudited) and after the zip is generated (previously before asset fetch). * fix(audit): record null actor for anonymous public-share downloads Address Greptile P1: setting actorId to the file owner made every anonymous external download read as a self-download, undermining the exfiltration trail. recordAudit now accepts a null actor; the public content/inline routes record actorId=null with the owner in metadata.sharedByUserId and rely on ip/user-agent for the forensic trail. The misleading owner-attributed file_downloaded PostHog event is dropped on these anonymous paths. * fix(observability): audit admin exports after zip; tidy comments - Admin workflow/workspace ZIP exports now audit only after the archive is built (via a local helper), so a zip/build failure no longer logs an export. - Remove redundant 'success-only placement' inline comments and tighten the remaining design-rationale notes (export helper now TSDoc). * fix(billing): complete the chargeback + overage financial trail Address Cursor review: - handleDisputeClosed now records CHARGE_DISPUTE_CLOSED for every closed dispute (won/lost/warning_closed), unblocking only on favorable outcomes. dispute.status in the metadata distinguishes the outcome, so lost chargebacks are no longer missing from the trail. - Threshold overage now emits OVERAGE_BILLED + overage_billed even when credits fully cover the overage (settledVia: 'credits' vs 'stripe'), so credit-settled overages are audited instead of silently returning null. * fix(storage): decrement copilot quota before deleting metadata Address Greptile P1: deleting the metadata row before the decrement meant a decrement failure left the quota permanently inflated with no record to retry from. Decrement first; only remove the metadata row once it succeeds. * fix(observability): atomic copilot storage release; signup-blocked action - Copilot file delete now releases storage via a single transaction (releaseDeletedFileStorage): the soft-delete is the idempotency claim and the decrement shares the transaction, so neither a partial failure (inflated counter) nor a retry (double-decrement) can desync the quota. Resolves the conflicting Greptile/Cursor ordering findings. - Policy-blocked sign-ups now record USER_SIGNUP_BLOCKED instead of USER_SIGNIN_BLOCKED, so account-lifecycle events aren't mislabeled. * fix(storage): meter copilot ingest centrally for path symmetry Address Cursor review: only uploadCopilotFile incremented storage, but copilot files also enter via the generic upload route and presigned uploads — all of which persist metadata through insertFileMetadata. Move the increment into insertFileMetadata (scoped to context='copilot', on genuine insert/restore) so every ingest path is symmetric with the delete-time decrement, and drop the now redundant per-path increment in uploadCopilotFile. Other contexts are metered by their own managers and remain unaffected. * fix(audit): skip WORKFLOW_EXPORTED when admin export is empty Address Cursor review: when every requested workflow fails to load, the admin export still returned 200 but recorded a successful export of zero workflows. Guard auditExport so an empty result records nothing. * fix(storage): settle copilot accounting before deleting the blob Address Cursor review: the blob was removed before releaseDeletedFileStorage, so a release failure left the counter inflated and the metadata active with the blob gone. Now the atomic soft-delete + decrement runs first and the blob is deleted only if it succeeds, so a failure leaves the file fully intact and retryable. * fix(storage): decrement KB document storage atomically with deletion Address Cursor review: hardDeleteDocuments deleted the rows then decremented best-effort, so a decrement failure left billed storage inflated with no row to reconcile. Resolve each owner's subscription up front, then decrement inside the same transaction that deletes the embeddings/documents (decrementStorageUsageInTx, also now shared by releaseDeletedFileStorage), so the counter and the content commit or roll back together. Connector docs remain excluded. * fix(audit): don't treat email verification as a login Address Cursor review: /verify-email could emit USER_LOGIN when verification mints or refreshes a session, mislabeling (or double-counting) a non-sign-in as a login. Restrict isLoginPath to genuine sign-in entrypoints. * fix(audit): null actor FK when the user lookup throws Address Greptile P1: the catch branch left the original actorId, so a system actor like 'admin-api' (or a since-deleted user) would FK-violate the insert and lose the row when the existence lookup errored. Mirror the not-found branch — null the FK with a readable label — so the audit row always persists. * revert(audit): drop session/account-lifecycle auth instrumentation Remove the Better Auth login/logout/session-revoke/account-delete/blocked-signin audit hooks from auth.ts — they touch sensitive auth paths and are noisy. Also removes the now-unused taxonomy (USER_LOGIN/_FAILED, USER_SIGNIN_BLOCKED, USER_SIGNUP_BLOCKED, USER_LOGOUT, SESSION_REVOKED, ACCOUNT_DELETED, ACCOUNT_EMAIL_CHANGED actions; SESSION/USER resource types) and the awaitable recordAuditNow helper that only the pre-delete hook used. auth.ts is back to the staging baseline. * fix(storage): harden copilot+KB delete accounting against read errors and concurrency Final-audit follow-ups: - deleteCopilotFile: a failed metadata *read* (vs a genuine missing row) now blocks the blob delete too, so a transient read error can't leave an active row un-decremented with the blob gone. - hardDeleteDocuments: drive the per-user decrement from the delete's returning() (the rows this tx actually removed), so two concurrent deletes of the same ids can't both decrement. * chore(observability): drop two unused definitions Final-audit cleanup: remove the orphaned knowledge_base_searched PostHog event (never wired; KB search analytics already flow through the OpenTelemetry channel) and the redundant AuditAction.SUBSCRIPTION_UPDATED (subscription/plan changes are audited via ORG_PLAN_CONVERTED). Every remaining new action/event has a real emit site. * fix(knowledge): key hard-delete result off rows actually deleted Address Cursor review: hardDeleteDocuments returned existingIds.length (the requested count) and cleaned storage for the full pre-tx set, even though the decrement was driven by the rows the transaction actually deleted. Under a concurrent delete that claimed some ids first, that overstated the result and re-touched storage for rows this call didn't delete. Drive the storage cleanup, log, and return value from deletedDocs (the returning() rows) so all four are consistent. * fix(storage): gate copilot quota on all ingest paths; tidy inline comments - Copilot uploads via the generic /api/files/upload route and presigned URLs now run checkStorageQuota before writing (matching uploadCopilotFile), so no copilot ingest path can grow usage past the plan limit. The central increment stays in insertFileMetadata. - Trim/remove redundant inline comments across the diff; keep only concise notes on non-obvious decisions (left pre-existing comments untouched). * fix(analytics): omit empty workspace_id on workspace-less file downloads Address Greptile P1: the generic key-based /api/files/download route and the no-workspace markdown-export path emitted file_downloaded with workspace_id:'' creating a phantom '' bucket in PostHog. Make workspace_id optional on the event and omit it when there is no workspace (workspace-scoped routes still pass it). * fix(analytics): omit empty workspace_id/workflow_id on copilot_chat_sent Final-sweep nit: copilot_chat_sent sent '' for workspace_id/workflow_id in the agent (workspace-less) branch, same phantom-bucket issue as file_downloaded. Make both properties optional and omit them when absent. * fix(analytics): clear stale org PostHog group on personal workspaces Address Cursor review: switching from a team workspace to a personal one left the previous organization group set, so later events kept rolling up under it. When the active workspace has no organizationId, resetGroups() to drop the stale org group, then re-apply the workspace group. * fix: address review — export audit timing, copilot delete signal, group reset - Table export (Cursor MED): audit fires before streaming begins, not after controller.close(), so a mid-stream failure still records the partial export. - deleteCopilotFile (Greptile P1): throw when storage accounting can't be settled instead of silently returning, so callers can detect the file was not deleted. - workspace-scope-sync (Cursor MED): only resetGroups() once workspace metadata is loaded (activeWorkspace present), so a team workspace doesn't transiently lose its org group while organizationId is still null during load. * refactor(observability): final line-justification cleanup Address final audit flags: - Table import: move the 'columns added' audit OUT of the import transaction into a post-commit auditTableColumnsAdded() helper called by the three tx-owning callers, so a mid-import row-batch rollback no longer logs a false 'added N columns' (matches the PR's success-only discipline). - Omit empty-string analytics dimensions on workflow_lock_toggled (workspace_id) and organization_created (name), consistent with file_downloaded/copilot_chat_sent. - Restore an unrelated capacity-check comment removed incidentally. - Move copilot_chat_sent emit above the traceparent comment so the comment sits with the code it documents. * fix(table): attribute import column audit to the importing user Address Cursor review: addImportColumns (async createColumns import path) now threads the importing userId into auditTableColumnsAdded instead of falling back to table.createdBy, so column additions are attributed to the actual member who ran the import rather than the table creator. * feat(observability): drop copilot from storage accounting Per design decision: copilot files are working/conversational artifacts, so gating their materialization on storage quota would fail an agent operation mid-flow when a user is over limit, and metering them would inflate usage enough to block KB/workspace uploads indirectly. Remove copilot quota gates + copilot ingest metering entirely (revert copilot-file-manager, metadata, and the upload route's copilot branch to baseline) and drop the now-unused releaseDeletedFileStorage. KB document metering + quota enforcement (the deliberate, persistent storage path) is unchanged. * fix(analytics): switch workspace + org PostHog groups together Address Cursor review: gate the group-sync effect on workspace metadata being loaded (activeWorkspace present) so the workspace and organization groups always update atomically. Acting during the load window paired the new workspace group with the previous workspace's org group; until metadata loads, events stay consistently attributed to the previous workspace. * fix(table): audit async export at authorization, not job completion Address Cursor review: async exports only emitted TABLE_EXPORTED when the background job reached 'ready', so an authorized export whose job later failed or was abandoned left no audit trail — inconsistent with the sync export route, which audits before streaming. Move the audit + analytics to the async route's authorization point (after the job is claimed/dispatched) and remove it from the runner. Drop the now-unused userId from TableExportPayload. * fix(billing): never let payment_failed instrumentation skip user blocking Address Greptile P1: the payment_failed audit hoisted an unguarded isSubscriptionOrgScoped DB read (for entity_type) directly before the attempt-count user-blocking block. A transient failure of that read would throw out of the handler and skip blocking. Wrap the whole audit/analytics block in try/catch (best-effort), and let the blocking compute its own isSubscriptionOrgScoped as it did originally — instrumentation can no longer abort payment processing. * chore(observability): trim verbose inline comments to concise notes * fix(billing): wrap new dispute/enterprise/subscription instrumentation in Stripe webhook idempotency recordAudit/captureServerEvent calls added for charge disputes, enterprise subscription provisioning, and free->paid subscription creation ran unconditionally with no idempotency guard, unlike their sibling handlers in the same files. Stripe redelivers webhooks at-least-once, so a retry would double-record the audit row and PostHog event even though the underlying DB writes were already idempotent. * fix(observability): tag org-scoped audit metadata with organizationId, guard concurrent table delete The org-scoped self-service audit-log endpoint matches org-level rows via metadata.organizationId (or resourceType=organization). Six recordAudit call sites (subscription create/cancel, admin credit issuance, threshold overage billing, charge disputes, credit purchase, invoice payment succeeded/failed) tagged org-scoped events with a differently-named key (referenceId/entityId/targetOrgId), making them invisible to org admins querying their own audit trail despite being stored in the DB. Add the missing organizationId key everywhere the pattern was missed. Also guard deleteTable's archive UPDATE with isNull(archivedAt) so a concurrent duplicate delete request is a no-op instead of re-archiving and re-firing a duplicate TABLE_DELETED audit row. Adds test coverage for the ORG_MEMBER_ADDED audit/analytics emission in acceptInvitation, which previously had none. * fix(test): queue resolveBillingActorId's owner lookup in payment-failure email test handleInvoicePaymentFailed's new payment_failed audit instrumentation resolves the billing actor via an extra db.select before sendPaymentFailureEmails runs. The test's fixed select-response queue didn't account for it, so the org-admin lookup consumed the wrong queued row and the assertion saw zero email sends. Production behavior is unaffected — each query is independent; this was a mock-queue ordering issue only. * fix(billing): don't let a post-commit usage-limit sync failure suppress the seat audit Cursor Bugbot flagged: reconcileOrganizationSeats committed the seat change and Stripe outbox enqueue in a transaction, then called syncSubscriptionUsageLimits outside it before recording the audit/ PostHog events. A thrown sync left a genuinely-changed seat count with no ORG_SEAT_PROVISIONED/DEPROVISIONED trail. Wrap the sync in its own try/catch (log + continue) so the events always fire for a committed change, matching the fire-and-forget instrumentation pattern used elsewhere in this PR. * fix(billing): guard the new subscription-created instrumentation block Greptile P1: the org-scope check I added for the audit-metadata fix (isSubscriptionOrgScoped) was a raw DB call with no error guard, unlike resolveSubscriptionActorId next to it. Since it ran inside the idempotency lambda with an unconditional rethrow, a transient DB error would abort and retry the whole webhook after the free -> paid usage reset had already committed. Wrap the actor/org-scope resolution + audit + analytics block in try/catch, matching the same guarded instrumentation pattern already used in handleInvoicePaymentFailed. |
||
|
Vikhyath MondretiandSiddharth Ganesan
|
4a80374c03 |
improvement(forking): unlink, settings page migration, deployed chats, public apis, external mcp servers/tools, special subblocks (#5505)
* bad checkpoint * stash * add unlink, move UI to settings pages * fix edge cases * remove dead code * remove migration 0255 ahead of staging merge (regenerated after) * regenerate workflow_mcp_server enum migration on top of staging (0257) * fix tests * more tests * address comments * consolidate fork migrations into 0257 (enum value + activity metadata indexes) * acquire MCP server locks before reads in attachment reconcile (TOCTOU) * move into ee folder + ui perm gates * use randomInt from @sim/utils/random for chat identifier suffix --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com> |
||
|
Theodore Li
|
8c61720d33 |
chore(tables): remove tables-fractional-ordering feature flag (#5503)
* chore(tables): remove tables-fractional-ordering feature flag * improvement(tables): drop dead position tracking from paste undo path * improvement(tables): drop dead position from create-row undo records |
||
|
Waleed
|
858f7d2c2c |
fix(api-block): stop spoofing a browser fingerprint on outbound HTTP requests (#5496)
* fix(api-block): stop spoofing a browser fingerprint on outbound HTTP requests getDefaultHeaders() sent a Chrome User-Agent, a Referer pointing at Sim's own app, and mismatched Sec-Ch-Ua client hints on every API block request. Atlassian's Jira/Confluence Cloud REST API (and other browser-aware anti-CSRF/bot-defense layers) reject requests carrying a browser User-Agent with 403 "XSRF check failed", even with a valid Basic-auth header and X-Atlassian-Token: no-check set. Default headers now identify honestly as Sim, matching how other HTTP clients (curl, Postman, axios) behave. User-supplied headers still override any default. * fix(testing): sync shared tool-tester mock headers with new defaults createMockHeaders in the shared @sim/testing tool-tester builder still hardcoded the old Chrome UA / Referer / Sec-Ch-Ua fallback values. It was unreachable in current tests but would silently diverge from production getDefaultHeaders() for any future test hitting its fallback path. |
||
|
Theodore Li
|
56cd2cfa7c | feat(custom-block): audit-log publish, update, and unpublish events (#5493) | ||
|
Vikhyath Mondreti
|
24ebba9acc | chore(credential-sets): cleanup feature (#5460) | ||
|
Waleed
|
4b133984c6 |
fix(mcp): make SSRF-guarded fetch structurally mandatory in OAuth auth() calls (#5419)
PR #5399 fixed the callback route forgetting to pass fetchFn into the SDK's auth(), but every call site (start + callback routes) still imported the raw SDK auth() directly and had to remember to pass fetchFn: createSsrfGuardedMcpFetch() by hand — the same omission was possible again at any future call site. Add mcpAuthGuarded() in lib/mcp/oauth/auth.ts, a thin wrapper around the SDK's auth() that always defaults fetchFn to the SSRF-guarded fetch (still overridable for tests). Both routes now import mcpAuthGuarded from @/lib/mcp/oauth instead of the raw SDK auth, so omitting the guard is no longer possible by omission. |
||
|
Theodore Li
|
69b81a679b |
feat(data-retention): granular PII redaction stages (input + block outputs) (#5272)
* feat(data-retention): granular PII redaction stages (input + block outputs) * fix(data-retention): propagate block-output redaction into child workflows * fix(data-retention): close block-output redaction gaps on streaming + resume * fix(data-retention): drain+mask streamed output, resolve PII policy unconditionally (no fail-open) * test(testing): support leftJoin().where().limit() in shared db mock * fix(data-retention): mask agent/Pi memory writes under block-output redaction * fix(data-retention): guard partial PII stages in GET normalize * fix(data-retention): mask seeded memory messages under block-output redaction * fix(guardrails): fail closed on misaligned Presidio batch responses * fix(data-retention): enabled stage with no entity types redacts all (no fail-open) * fix(data-retention): reject enabled stage with no entity types; empty = off everywhere * docs(data-retention): note resume remask covers inline values only * fix(data-retention): scrub offloaded large-value refs from logs when block-output redaction is off * fix(data-retention): hydrate, mask, and re-store large-value refs in logs (preserve redacted content) * fix(data-retention): always apply logs policy to large-value refs when logs stage is on * perf(data-retention): drop redaction byte ceiling, parallelize chunks (env-tunable), remove request timeouts, sync large-value walk * feat(data-retention): gate granular PII stages behind pii-granular-redaction flag - New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION), layered on pii-redaction, gating the execution-altering input + block-output stages - Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off - UI shows only the Logs stage tab unless the flag is on; clamps active stage - Drop the per-search Select all toggle; add a Deselect all action to the PII section header * docs(pii): describe Presidio as a standalone service, not a sidecar Presidio now runs as its own ECS service (and, in Helm, its own Deployment + Service) reached over the network via PII_URL — not a sidecar in the app task. Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to match, and note the deploy requirement that PII_URL must be reachable. * fix(data-retention): re-mask offloaded large-value refs on resume + don't lock out granular saves - Resume/run-from-block restore now hydrates → masks → re-stores large-value refs in restored blockStates (not just inline strings), so a value offloaded before the block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast. - pii-large-values: add onFailure mode (throw on the execution path, scrub for logs) and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values - Granular flag gate now rejects only NEW off→on granular enablement, so orgs that already configured granular stages can still save retention settings when the flag is off |
||
|
Waleed
|
af87de09a7 |
fix(connectors): allow self-hosted private DB hosts via opt-in flag (#5322)
* fix(connectors): allow self-hosted private DB hosts via opt-in flag Database/connector tools rejected any host resolving to a private/reserved/ loopback IP, blocking the common self-hosted topology where the DB is reached by a Docker/K8s/Swarm service name. Add an opt-in ALLOW_PRIVATE_DATABASE_HOSTS flag that bypasses the private-host block in validateDatabaseHost while still resolving and pinning DNS. Blocked on the hosted platform regardless of the env var, mirroring DISABLE_AUTH. Fixes #4319 * fix(connectors): pin postgres IP in all ssl modes; strip IPv6 brackets Address review on #5322: - validateDatabaseHost now strips surrounding IPv6 brackets before the localhost/private-IP checks and DNS lookup, so a bracketed loopback like [::1] is classified correctly instead of failing as unresolvable. - PostgreSQL connector always connects to the validated, pinned IP (removed the ssl='preferred' carve-out that passed the original hostname and let the driver re-resolve during connection). Matches the MySQL/MongoDB pin pattern. - Add postgres connector pinning tests and bracketed-IPv6 host tests. * fix(connectors): rename flag to isPrivateDatabaseHostsAllowed; trim comment - Rename env-flag const to satisfy the env-flags 'is' prefix CI check (env var ALLOW_PRIVATE_DATABASE_HOSTS is unchanged). - Tighten the postgres pinning comment to a single line. |
||
|
Vikhyath Mondreti
|
c59631698f |
chore(deploy): remove deploy as a2a (#5255)
* chore(deploy): remove a2a * add block |
||
|
Vikhyath Mondreti
|
5a8134119a |
feat(workspaces): fork + push/pull (#5210)
* feat(workspaces): fork + push/pull * type fix * fix tests * progress on ux * remove modal section * improve UI of modal * update more ui * make rollback part of the footer * track skipped count correctly * address comments * make it workspace admin level * update skipped count * address more comments * deal with unbounded memory possibility * fix deleted kb article bug * no deployed workflow case * UI/UX cleanup * fix oauth dropdown case * fix oauth selector issue * infra work + activity log * consolidate migration * update modal state * more UI simplification * grammar * update audit report ui * perf improvements * fix tool input scenarios and add dependsOn UI handling * minor comments * fix webhook stability issues + drift detection removal * make dependsOn subblock mapping cleanly stored * fix: harden fork dependent-value mapping (clear stale rows, identity-guard first-sync fallback, perf + cleanup) * address comments * update comment * enforce admin perms for activity api * fix required + dependsOn combo |
||
|
Vikhyath Mondreti
|
8b5d746fe2 |
improvement(access-controls): ui/ux improvements (#5190)
* improvement(access-controls): ui/ux improvements * remove unused col |
||
|
Vikhyath Mondreti
|
91f9dfdaec |
improvement(governance): derived access (#5134)
* improvement(governance): org-ws-credential roles clarity * revert isHosted * improvement(credentials): code cleanup * address comments * make kb cascade delete on user hard delete * revert env flags * chore(db): drop local 0242 migration to regenerate after merging staging Our 0242 collides with staging's 0242. Remove it (and its snapshot + journal entry) so the KB-cascade migration can be regenerated with the correct number on top of the merged staging migrations. * chore(db): regenerate kb→workspace cascade migration as 0243 Regenerated via drizzle-kit generate on top of the merged staging migrations (staging took 0242). Re-applied the safety edits: NOT VALID + separate VALIDATE on the FK re-add, and the -- migration-safe note on the DROP. check:migrations passes. * improve copy * update docs |
||
|
Theodore Li
|
f0b3550729 |
feat(files): public share links for workspace files (#5130)
* feat(files): public share links for workspace files * improvement(files): drop reserved public_share columns until used; sync audit mock * fix(files): share modal tracks authoritative saved state until toggled * feat(files): per-IP rate limit on public share endpoints * fix(files): address PR review — public CSV OOM, content cache, share FK, soft-delete filter, download anchor * fix(files): disable CSV import action in read-only preview (public share) * refactor(files): drive CSV preview import affordance off readOnly, not disableImport * fix(files): version public viewer caches by file updatedAt so edits aren't stale * fix(files): 409 (not corrupt source) when a shared generated doc has no compiled artifact * feat(files): gate public sharing behind an access-control permission |
||
|
Vikhyath Mondreti
|
267e49c69c |
improvement(workspaces): auto-add without invite if part of organization (#5132)
* feat(workspaces): auto-add without invite if part of organization * reverse feature flag hardcoding * address comments * improve ux for org invite modal |
||
|
Waleed
|
7d46103d09 |
chore(deps): remove unused dependencies and harden CI supply chain (#5119)
* chore(deps): remove unused dependencies and harden CI supply chain Dependency cleanup: - Remove unused deps: papaparse, unified, and 6 unused Radix primitives (alert-dialog, radio-group, scroll-area, separator, toggle, visually-hidden) plus @tanstack/react-query-devtools (all verified zero imports repo-wide) - Consolidate jwt-decode into the existing jose dependency (decodeJwt) - Migrate react-window to @tanstack/react-virtual to drop a redundant virtualization library (terminal, structured-output, code viewer) - Remove the better-auth-harmony plugin and its gating env flag Supply-chain hardening: - SHA-pin every GitHub Action to a full commit SHA with a version comment - Pin CI bun-version to 1.3.13 (was "latest" in the release job) - Raise bun minimumReleaseAge cooldown from 3 to 7 days - Add a non-blocking `bun audit` step in CI - Add a CODEOWNERS gate routing dependency-manifest changes to @simstudioai/deps * chore(deps): remove unused apps/docs dependencies (@tabler/icons-react, dotenv-cli) * style(search-modal): use Send icon for Invite teammates action * feat(search-modal): surface New chat as the top action above Create workflow * feat(search-modal): add Secrets to the pages list |
||
|
Vikhyath Mondreti
|
7b4626e547 |
improvement(perm-groups): allow workspace filter for permission groups (#5070)
* improvement(perm-groups): allow workspace filter for permission groups * show errors correctly * address comments * address concurrent edit concern * address locks * address comments" * index migration safety * address at route level |
||
|
Theodore Li
|
3fe061e3b3 |
feat(feature-flags): AppConfig-backed gated feature flags (#5059)
* feat(feature-flags): AppConfig-backed gated feature flags * fix(ci): repoint 'Validate feature flags' step to env-flags.ts after rename * improvement(feature-flags): drop in-code defaults; fallback resolves a per-flag secret, gating is AppConfig-only * improvement(feature-flags): make flag names a closed set so every flag requires a fallback secret * improvement(feature-flags): single FEATURE_FLAGS registry — each entry defines name, description, and fallback in one place * improvement(feature-flags): fallback is the env secret key (keyof typeof env), resolved to a boolean |
||
|
Vikhyath Mondreti
|
3a796f083d |
improvement(permissions): permission groups scoped to organization level (#5035)
* improvement(permissions): permission groups scoped to organization level * chore(db): drop 0235 permission-groups migration to regenerate after staging merge * merge latest staging |
||
|
Theodore LiandClaude Opus 4.8
|
53fdcab5d6 |
feat(tables): background jobs (delete/export/backfill on trigger.dev) + tenant-scoped query performance (#4915)
* feat(tables): paginated background row-delete jobs via table_jobs
* fix(tables): address review on async row-delete (filtered count, scoped optimistic clear, Cmd+A select-all, hide delete from tray)
* improvement(tables): filter-aware select-all runs, delete-job read mask, keyset index + autovacuum tuning
* feat(tables): run import/delete/export/backfill jobs on trigger.dev with in-process fallback
* improvement(tables): raise delete page to 10k and export batch to 5k
* improvement(tables): raise CSV import batch to 5k rows (param-cap bounded)
* feat(tables): surface export jobs in the header tray with progress, cancel, and download
* improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray
* Revert "improvement(tables): surface exports as derived tables-scoped toasts instead of the import tray"
This reverts commit
|
||
|
Vikhyath Mondreti
|
f7b40fe4a4 |
fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions (#4975)
* fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions * update docs |
||
|
Waleed
|
284edf0b85 |
improvement(db): opt-in read-replica client + migration runner hardening (#4955)
* improvement(db): add opt-in read-replica client and harden migration runner * fix(db): detect wrapped lock-timeout errors, jittered retries, direct migration DSN support * fix(db): audit fixes — unparameterized SET, primary reads for authz scoping, shared error helper * fix(db): pin migration session — disable max_lifetime recycling, guard backend pid across retries * fix(db): gate pid session guard to direct migration connections (pooler pids legitimately vary) * improvement(billing): route display-only usage aggregations to the read replica via executor threading * chore(db): drop call-site justification comments * chore(db): tighten doc comments * ci(migrations): map optional direct-connection DSN secrets * fix(data-drains): add stability window to time cursors so late-visible rows are never skipped * fix(billing): resolve pagination cursor on primary; replica for member ledger display * fix(billing): usage-limits returns cost and limit from one computation |
||
|
Waleed
|
3cedac8e82 |
fix(security): authz, IDOR, and abuse-prevention fixes (#4944)
* fix(knowledge): require write access for batch chunk operations The PATCH /api/knowledge/[id]/documents/[documentId]/chunks handler performs enable/disable/delete operations but authorized callers with only read-level access (checkDocumentAccess). This let read-only workspace members destroy or disable indexed chunks. Switch to checkDocumentWriteAccess (write/admin required), matching the sibling POST/PUT/DELETE chunk mutation endpoints. * fix(env): restrict decrypted workspace env vars to secret admins GET /api/workspaces/:id/environment returned decrypted workspace environment variables to any member, including read-only collaborators, leaking API tokens, database URLs, and other secrets. Mask workspace variable values for non-admin viewers while preserving the variable names, so editor autocomplete and conflict detection keep working. A value is revealed only when the caller is a credential admin of that key, or — for legacy keys with no per-secret ACL — holds workspace admin permission. This mirrors the per-key edit gating already enforced by PUT/DELETE: if you can administer a secret, you can read it. Personal variables and execution-time resolution are unchanged. * fix(files): block cross-tenant deletion via client-controlled context POST /api/files/delete trusted a client-supplied `context`, letting any authenticated user delete another tenant's file by naming an arbitrary key with `context: "og-images"`. verifyFileAccess() short-circuited the three public contexts (profile-pictures, og-images, workspace-logos) to `true` before any ownership/requireWrite check. - Derive the storage context strictly from the trusted key prefix in the delete route; reject a supplied `context` that disagrees with the key. - Gate the public-context short-circuit to reads only. Destructive ops (requireWrite) now prove ownership via verifyPublicAssetWriteAccess: workspace-logos require write/admin on the bound workspace, profile-pictures require an exact owner match, og-images always deny. Reads of public assets are unchanged. * fix(telegram): verify X-Telegram-Bot-Api-Secret-Token on inbound webhooks Telegram triggers accepted any forged update from anyone who knew the webhook URL path: verifyAuth was a no-op that always returned null, and setWebhook registered no secret_token. Generate a per-webhook secret in createSubscription, register it with Telegram as secret_token, and persist it to providerConfig. verifyAuth now fails closed — rejects when no token is configured, when the X-Telegram-Bot-Api-Secret-Token header is absent, or when it does not match via constant-time safeCompare. * fix(security): pin DNS for Agiloft directExecution and Grafana update tools The Agiloft directExecution tools (read/create/search/update/delete/lock/ saved_search/select/get_choice_line_id/remove_attachment/attachment_info) and the Grafana update_dashboard/update_alert_rule postProcess hooks issued outbound HTTP to a fully user-controlled host (instanceUrl/baseUrl) via the global fetch(), guarded only by the synchronous validateExternalUrl() — which never resolves DNS, so a hostname resolving to an internal/reserved IP passed validation (SSRF). Route all of these through the codebase's standard SSRF-safe path: - Agiloft: moved executeAgiloftRequest into utils.server.ts where the existing pinned helpers live. It now resolves+validates the instance URL once and pins every hop (login, operation, logout) to that IP via secureFetchWithPinnedIP. The 11 tool configs now import it from utils.server; URL builders stay in the client-safe utils.ts. - Grafana: the postProcess POST/PUT now uses validateUrlWithDNS + secureFetchWithPinnedIP, matching the already-pinned initial GET. This completes the Agiloft SSRF pinning started in #4639 (which covered the attach/retrieve API routes) by closing the directExecution path, and extends the same guard to the Grafana update tools. * fix(api): enforce workspace allowPersonalApiKeys policy on v1 surface The external v1 API authenticated API keys without evaluating the per-workspace allowPersonalApiKeys setting, so a personal API key could read and mutate a workspace's resources (workflows, tables, files, knowledge, logs) even when the workspace had explicitly disabled personal keys. The same control is already enforced on the workflow-execution surface. Enforce the policy in checkWorkspaceScope (covering validateWorkspaceAccess too): reject personal keys with 403 when the workspace has allowPersonalApiKeys=false. checkWorkspaceScope becomes async; all v1 route callsites updated to await it. * fix(billing): close usage-cap admission race with atomic reservation The server-side usage-limit gate read already-recorded cost, but cost is only written when an execution finishes. A burst of concurrent executions all observed the same pre-burst usage, all passed the cap, and all ran — collectively spending far past the limit before any cost landed in the ledger (free-tier abuse / hard-cap defeat). manual/chat triggers also skip rate limiting, removing the only throttle. Add an atomic check-then-reserve admission step (Redis Lua) that bounds in-flight, un-costed executions per billing entity by both a per-plan concurrency cap and remaining usage headroom, so recordedUsage + reservedSlots * estimate <= limit always holds. The slot is released at execution completion via LoggingSession (skipped on pause; TTL self-heals crashes). Runs for all trigger types, covering the previously-unthrottled manual/chat paths. Fails open when billing is disabled or Redis is unavailable, matching the rate limiter — a Redis blip can't turn into an execution outage, and the recorded-usage gate still runs. * fix(workflows): validate folderId belongs to workflow's workspace on create/update/reorder Reject a folderId that references a folder in a different workspace (or an archived/non-existent folder) before writing it to workflow.folderId. Previously create, update, and reorder only checked workspace permission on the workflow and the folder's lock status, never that the folder lived in the workflow's own workspace, allowing a dangling cross-workspace folder reference. Adds isFolderInWorkspace/assertFolderInWorkspace + FolderNotFoundError to @sim/workflow-authz (mirroring assertTargetFolderMutable in the duplicate path), enforced in performCreateWorkflow, performUpdateWorkflow, and the reorder route. Invalid folders now return 400. * fix(folders): validate parentId against workspace on create/update/reorder Folder write endpoints accepted a caller-supplied parentId and persisted it without verifying the parent existed in the same workspace, and the create and reorder paths had no cycle guard. A workspace member with write access could reparent a folder to a foreign-workspace folder, a non-existent id, or (via reorder) into a cycle, hiding the folder and its workflows from all members. - performCreateFolder: reject self-parenting and validate the parent exists in the workspace and is not archived (mirrors the duplicate route). - performUpdateFolder: add the same workspace/archived parent check alongside the existing circular-reference guard. - folders/reorder: validate every target parent against the workspace, detect cycles in the resulting parent graph (catches batch cycles), and normalize falsy parentId to null to prevent orphaning. Adds tests for cross-workspace parent rejection and batch-cycle rejection. * chore(knowledge): drop non-TSDoc inline comments from chunks route * fix(webhooks): fail closed when HMAC signing secret is not configured Inbound webhook signature verification failed open for HMAC providers (GitHub, Intercom, Jira, JSM, Confluence, Cal.com, Notion, Greenhouse, Typeform, Fireflies, Circleback): when no signing secret was stored, verifyAuth returned null and the workflow executed on a fully attacker-controlled body. Reject these deliveries with 401 instead, matching the fail-closed Stripe/WhatsApp/Vercel providers. Run provider reachability/verification handshakes (Notion verification_token, Grain/Intercom ping) ahead of auth so the pre-secret setup handshake still completes — those return a canned 200 without executing the workflow, and real event payloads fall through to fail-closed verification. Update the trigger secret-field copy to state the secret is required for deliveries to be accepted (was misleadingly marked optional). * style(files): trim verbose inline comments on delete authorization fix * fix(auth): close account-enumeration oracle on email sign-up The custom before-hook pre-check threw a distinguishing 422/USER_ALREADY_EXISTS for already-registered emails, letting an unauthenticated attacker enumerate accounts — defeating better-auth's own OWASP enumeration protection (active under requireEmailVerification). Remove the pre-check and rely on better-auth's generic duplicate-sign-up response, wiring: - onExistingUserSignUp: notify the real account owner out-of-band, mirroring the privacy-preserving forget-password flow. - customSyntheticUser: include admin (role/banned/banReason/banExpires) and Stripe (stripeCustomerId, billing-gated) user fields so the fake response shape is byte-identical to a real new-user response. Adds an ExistingAccountEmail template + 'existing-account' subject. * style(tools): drop non-TSDoc inline comments from Grafana/Agiloft SSRF tools * chore(api): trim extraneous inline comments in v1 logs/files routes Remove a redundant size annotation and two verbose multi-line materialization comments whose intent is already clear from the code. Load-bearing comments (race-condition and key-translation notes) kept. * fix(billing): exclude table-cell dispatch from admission reservation Table-cell dispatch is row-bounded, async rate-limited, and already surfaces a graceful usage state. Applying the in-flight concurrency reservation there turned its 429 into a hard cell error on a normal >15-concurrent-cell run (only 402 was handled gracefully). Skip the reservation for that surface via a new skipConcurrencyReservation option (the usage-cost cap is still enforced), and tidy the reservation comments to TSDoc. * fix(chat): rate-limit and constant-time password auth for public chats Password-protected public chat (POST /api/chat/[identifier]) had no throttling on the password check and compared with a non-constant-time !==, allowing unlimited brute-force and per-character timing leaks. - Add per-IP rate limiting (10 / 15min) to the password branch of validateChatAuth, mirroring the OTP/SSO endpoints; return 429 with Retry-After. Only explicit unlock attempts consume tokens — message sends carry no password and ride the auth cookie. - Replace password !== decrypted with safeCompare. - Fails open on rate-limiter storage errors; no availability regression. * fix(security): cap JSON request body size and gate public chat endpoint The shared parseJsonBody helper (behind parseRequest, used by nearly every contract route) read request bodies with no size limit, buffering the full body into memory before validation. The unauthenticated public deployed-chat endpoint reached this sink with no admission gate, enabling an anonymous memory-exhaustion DoS. - parseRequest/parseJsonBody now enforce a byte cap via a size-limited stream read (content-length precheck + streamed cap), returning 413. Default is API_MAX_JSON_BODY_BYTES (50 MB), overridable per route via maxBodyBytes. Decoding uses TextDecoder to match request.json() BOM handling. - Public chat POST is wrapped with the admission gate (tryAdmit) and passes an explicit CHAT_MAX_REQUEST_BYTES (20 MB) cap. - Chat body contract gains .max() bounds on input, password, conversationId, file data/name/type, and files array length. - Admin bulk workspace import opts into a higher 100 MB cap to avoid regressing large multi-workflow imports. * fix(chat): rate-limit and constant-time password auth for public chats Password-protected public chat (POST /api/chat/[identifier]) had no throttling on the password check and compared with a non-constant-time !==, allowing unlimited brute-force and per-character timing leaks. - Add per-IP rate limiting (10 / 15min) to the password branch of validateChatAuth, mirroring the OTP/SSO endpoints; return 429 with Retry-After. Only explicit unlock attempts consume tokens — message sends carry no password and ride the auth cookie. - Replace password !== decrypted with safeCompare. - Fails open on rate-limiter storage errors; no availability regression. Reinstates the fix reverted by an intervening commit. * fix(billing): never block a lone execution on usage headroom The admission reservation tapered allowed concurrency by remaining usage headroom. With under one credit of headroom left (but not yet over the cap), floor(headroom / estimate) hit zero and rejected even a single, zero-concurrency execution — stricter than the recorded-usage gate, which would have allowed that last run, and with a misleading "too many concurrent executions" message. Floor the headroom term at 1 so a lone execution is governed only by the cost gate; concurrency above the first slot still tapers with headroom. * refactor(env): document workspace env masking, drop inline comments Extract the workspace-env value masking into a TSDoc-documented maskWorkspaceEnvForViewer helper and remove the redundant inline comments from the GET handler and its test. No behavior change. * refactor(env): convert PUT/DELETE authz comments to TSDoc Move the tiered-authorization rationale for the workspace env upsert and delete handlers into TSDoc blocks and drop the inline comments. No behavior change. * fix(telegram): keep legacy webhooks working via Telegram source-IP fallback The secret-token check rejected every webhook registered before secret_token support, breaking live triggers until re-saved. Fall back to verifying the request originates from Telegram's published webhook IP ranges when no secret is configured, so existing triggers keep firing with no re-save or migration while forged updates from arbitrary hosts are still rejected. Webhooks with a registered secret continue to use strict constant-time token verification. * fix(chat): restore constant-time password auth and IP rate limit A billing commit ( |
||
|
Vikhyath MondretiandCursor
|
6abcf82db2 |
feat(workflows): sim trigger, logs v2 block, toolbar renaming (#4941)
* feat(workflows): sim trigger, logs v2 block, toolbar renaming * fix(review): bound rule queries, canonical logs params, watched-workflow SQL scoping Code-review fixes: read the canonical workflowIds param in logs_v2 (the serializer deletes the source pair ids), aggregate failure-rate in the DB and switch rule windows to the indexed startedAt column, clamp rule config to the legacy contract bounds, push no_activity watch scoping into SQL before the LIMIT, fix the generated sim icon-map key, normalize docs wording, and drop dead exports. Co-authored-by: Cursor <cursoragent@cursor.com> * address comments * fix(review): integer rule rounding, success-gated workflow labels, display module hygiene Second-pass review fixes: round integer rule fields so fractional input never reaches SQL LIMIT, gate workflow-name readiness on a successful non-placeholder load in both editor and preview (errored loads mislabeled valid workflows as deleted), lazily read the variables store in preview rows, move the filter-field JSON preview into the shared display module and unexport its single-consumer helpers, and align >= boundary copy (failure rate, error count, cooldown window) with implementation. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: sync lockfile after staging merge Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): keyset-paginate the no_activity subscription scan A fixed LIMIT 500 with no ORDER BY silently starved subscriptions beyond the cap once the global count exceeded it. The poll now pages by webhook id so every subscription is visited each cycle; pagination bounds memory, not total work. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): keyset-paginate the watched-workflow scan The 500-row LIMIT silently and deterministically excluded high-id workflows from no_activity coverage in watch-everything subscriptions on large workspaces. The scan now pages by workflow id, mirroring the subscription scan; per-workflow checks move into a helper so the pagination loop stays flat. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(workspace-events): skip no_activity subscriptions on the execution-completion path no_activity is poller-owned and can never fire from a completed execution, but it passed into the rule branch and cost a pointless cooldown point-read per subscription on the hottest path. Early-continue alongside the workflow_deployed guard. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(sim-trigger): note failure-based alert conditions evaluate on failed runs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(blocks): recategorize Data Enrichment as a core block It's a Sim-native capability (registry enrichments over a managed provider cascade, like Search), not a third-party integration. Moves it to Core Blocks in the toolbar, out of the integrations catalog, and relocates its docs page to blocks/ with the icon-map allowlist keeping the docs card icon. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(blocks): recategorize MySQL, PostgreSQL, SFTP, SMTP, SSH as integrations External-system connectors with host/credential auth belong under Integrations, not Core Blocks — consistent with MongoDB, Redis, ClickHouse, and the other datastore integrations. They already carried integrationType and /tools docsLinks; the regenerated docs pages turn those previously-dangling links into real pages, and the blocks join the integrations catalog and icon maps. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
+1
|
39418f9615 |
improvement(mothership): v0.2 (#4923)
* CONTRACTS * updates * Prompt caching * Fix regression * updates to prompt caching * Prompt caching trace * VFS updates * Changelog and plan * VFS update * Improvement/mothership (#4775) * ff * System role in cache * Fixes * Add dynamic user info * Improve tool search * improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * New doc setup * Fixes * Fixes * Logs tools * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * Add finalization on error * Fixes * change dev CI to bun run db:push * Updates Please enter the commit message for your changes. Lines starting * Contrat update * improvement(nested): subagents * updates * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Theodore Li
|
f7811f8acc |
feat(tables): stable column ids for metadata-only rename (#4898)
* feat(tables): stable column ids for metadata-only rename * fix(tables): address review — id-key exec clears + waiting labels, name in upsert error, un-gate group output ids * fix(tables): id-correct column undo (rename/create/delete) with id reuse on re-add * refactor(tables): mint column ids via generateId (uuid), drop collision-check plumbing * fix(tables): match column id or name when removing column from optimistic delete cache * fix(tables): resolve column storage id once for delete optimistic strips; biome-format 0228 snapshot * fix(tables): make in-grid find id-native (scan/return JSONB keys as column ids) * fix(tables): translate id→name in CSV/JSON export read path (+regression test) |
||
|
|
0075ab9cf6 |
improvement(platform): remove tour, simplify sidebar/header, drop loading skeletons (#4354)
* improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Waleed
|
648a5a117d |
feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage (#4865)
* feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage Add S3_ENDPOINT and S3_FORCE_PATH_STYLE env vars, wired into the shared upload S3 client so Cloudflare R2, MinIO, Backblaze B2, and other S3-compatible stores work for self-hosted file storage. The endpoint is trusted operator config (no SSRF/HTTPS gate). Makes the multipart Location fallback endpoint-aware, extends the S3 client unit tests, and documents the new vars in Helm values, .env.example, and the English self-hosting docs (incl. browser-reachability + CORS guidance). * docs(storage): add RustFS as an S3-compatible provider example * fix(storage): address review feedback and fix env mock for CI - Add envBoolean to the shared env test mock (createEnvMock) so config.ts's forcePathStyle coercion resolves — fixes failing knowledge/utils.test.ts - Declare S3_FORCE_PATH_STYLE as z.string() (every other env var's pattern); it's coerced via envBoolean at the consumption site, avoiding a boolean type that never matches the string process.env value - Log path-style from S3_CONFIG.forcePathStyle (envBoolean) instead of a separate isTruthy call, so the startup log can't disagree with the client - Make buildObjectFallbackUrl honor forcePathStyle: virtual-hosted-style URL (bucket as subdomain) for R2, path-style only when forcePathStyle is set * docs(storage): add backlinks to S3-compatible providers (R2, MinIO, Ceph, B2, RustFS) and backends |
||
|
Waleed
|
3f3efc98c3 | chore(auth): remove deprecated OAuth MCP provider plugin and backing tables (#4847) | ||
|
Waleed
|
a7b0bd311d |
fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) (#4837)
* fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) - Bump vitest and @vitest/coverage-v8 to ^4.1.0 across all workspaces (only patched release for the critical 'Vitest UI server arbitrary file read/execute' advisory; no 3.x backport exists) - Widen @sim/testing peer range to ^3.0.0 || ^4.0.0 - Migrate constructor mocks to class expressions: vitest 4 uses Reflect.construct for mocks invoked with new, and arrow/function implementations are not constructable (function expressions also get reverted to arrows by biome's useArrowFunction) - Remove deprecated test.poolOptions from apps/sim/vitest.config.ts (options are now top-level in vitest 4) * fix(deps): exclude vulnerable vitest 4.0.x from @sim/testing peer range Tighten the v4 arm of the peer range to >=4.1.0 <5.0.0 so the peer requirement cannot be satisfied by the unpatched 4.0.x builds that GHSA-5xrq-8626-4rwp affects. * fix(testing): make vitest 4 constructor mocks type-check cleanly - logging-session & mcp-oauth mocks: a class passed to mockImplementation has a construct signature that isn't assignable to its (...args) => any parameter, failing tsc. Use named function declarations instead (constructable via Reflect.construct, assignable to mockImplementation, and not rewritten to arrows by biome's useArrowFunction). - database.mock.ts: vitest 4's generic vi.fn typings no longer break the self-referential cycle on the transaction callback's tx param; loosen tx and annotate the callback's return type to resolve the implicit-any errors. * test(isolated-vm): de-flake queue-capacity scheduler tests The 'queue is full' and 'per-owner queued limit' tests relied on 'await sleep(1)' to assume the first request had reached the queue before submitting the overflow request. The first request only enqueues after an async spawn-failure chain (acquireWorker -> spawn exit -> resolve null -> enqueue), which isn't guaranteed within 1ms under CI load — the overflow request then found an empty queue and hit the 200ms queue-wait timeout instead of the capacity rejection. Replace the wall-clock barrier with a deterministic, event-driven one: hold the single global concurrency slot (IVM_MAX_CONCURRENT=1) with an active worker and await an explicit 'dispatched' signal (fired when the worker receives its execute message, after the scheduler counts it active). The follow-up requests then deterministically hit the synchronous enqueue path. Also drops the queue-wait timeout from 200ms to 50ms, so the tests run faster. |
||
|
WaleedandClaude Opus 4.8
|
fd77bb4069 |
feat(copilot): add seq ordinal to copilot_messages for order-preserving reads (#4791)
copilot_messages had no column preserving message order: created_at (set from each message's timestamp) ties at millisecond granularity in 58% of chats, and some chats have out-of-order timestamps within their array. The only other tiebreaker, id, is a random UUID — so ORDER BY created_at, id renders same-timestamp user/assistant pairs swapped. This blocks the R+1 read cutover. Add an integer seq = the message's 0-based index within the chat's JSONB array (ground-truth order), backfilled inline in migration 0219 (no script for self-hosters or us). Reads will use ORDER BY seq NULLS LAST, created_at, id at cutover; reads still come from JSONB after this PR. Design: - seq is a tiebreaker, not the sole sort key (concurrent-append/NULL safety). - Nullable now; defer NOT NULL so rolling-deploy old pods don't fail inserts. - replace (update-messages snapshot) overwrites seq = array index (re-densifies after a mid-conversation delete); append preserves existing seq via COALESCE and assigns base+idx from a single MAX(seq) read (never MAX+i in SQL — multi-row batches would collide). The non-atomic read-then-insert window is documented and bounded by the read tiebreak + snapshot re-densify. - Dedupe message ids before insert (87 prod chats carry dup ids; a repeated id in one INSERT...ON CONFLICT would otherwise throw). - Backfill picks first-occurrence per (chat,id), gap-free via ROW_NUMBER; validated on staging data (0-based, contiguous, 0 bad ranges). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
WaleedandClaude Opus 4.7
|
ddc47eb221 |
feat(copilot): add copilot_messages table with dual-write rollout (#4726)
Splits copilot chat messages out of the copilot_chats.messages JSONB column into a dedicated copilot_messages table. JSONB stays canonical during R+0 — every write path dual-writes to the new table best-effort (try/catch + log warn, never throws). Migration 0217 creates the table + indexes and inline-backfills history from JSONB so OSS self-hosters don't need to run a separate script. Write paths covered: - post.ts (user message append) - terminal-state.ts (assistant turn finalize) - update-messages/route.ts (snapshot replace) - inbox/executor.ts (background turn) - fork/route.ts (chat clone) - superuser/import-workflow/route.ts (chat import) Each call threads chatModel + streamId where relevant; ON CONFLICT DO UPDATE preserves existing stream_id / model via COALESCE. For pre-R+1 reconciliation, run: bun apps/sim/scripts/copilot-messages-reconcile.ts [--since='7 days'] Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Vikhyath Mondreti
|
59792c052f |
improvement(mcp): bound MCP memory and lifecycle concurrency (#4751)
* improvement(mcp): bound MCP memory and lifecycle concurrency * update db mock * address comments * address comments |
||
|
Vikhyath Mondreti
|
209ca5f121 |
fix(large-refs): cleanup based on table read (#4716)
* fix(large-refs): cleanup based on table read * address comments * address comments * bubble up storage ref errors * cleanup code * do not attempt blob deletion for infra outage * cleanup dup helper |
||
|
WaleedandClaude Opus 4.7
|
46db40620f |
feat(mcp): OAuth 2.1 + PKCE for outbound MCP servers (#4441)
* feat(mcp): OAuth 2.1 support for outbound MCP servers
* fix(mcp): tighten OAuth refresh race and session-error detection
Re-load the OAuth row inside withMcpOauthRefreshLock so concurrent
callers observe predecessor-written tokens instead of a stale snapshot
loaded before lock acquisition. Without this, the second caller's
provider held a rotated-out refresh token and the SDK tripped
invalid_grant, forcing reauthorization.
Switch isSessionError to match the SDK's typed StreamableHTTPError
(code 404/400) instead of substring-checking arbitrary error messages,
removing false positives on URLs that happen to contain those digits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(mcp): tighten OAuth callback contract and registration metadata
- Validate callback query params via mcpOauthCallbackContract instead of
raw searchParams.get, matching the rest of the MCP route surface.
- Drop non-RFC-7591 application_type field from dynamic client registration
to avoid rejection by strict authorization servers.
- Collapse the pre-lock OAuth row load in createClient — the row is now
loaded exclusively inside withMcpOauthRefreshLock, removing a redundant
query and a stale-snapshot path.
* fix(mcp): narrow workspaceId before async closure in OAuth createClient
* fix(mcp): return authType from create-server endpoint
The POST /api/mcp/servers handler omitted authType from the success
response, so useCreateMcpServer always saw data.data.authType as
undefined and never triggered the OAuth popup after creating an
OAuth-protected server. Thread authType through performCreateMcpServer
into the response so the client can decide whether to auto-start OAuth.
* fix(mcp): mirror server null normalization in optimistic oauthClientId update
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): revert optimistic oauthClientId to undefined to match McpServer type
The response contract preprocesses null → undefined, so McpServer.oauthClientId
is string | undefined. Using null broke type checking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): tighten OAuth probe signal and clear stale popup interval
- probe: only classify as OAuth on resource_metadata or scope params.
Bare `Bearer error="invalid_token"` is generic and used by API-key servers,
so it must not auto-flip the auth type to OAuth.
- popup hook: clear any existing close-watcher interval before overwriting
when startOauthForServer is invoked twice for the same serverId.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): normalize empty-string oauthClientId at route boundary
Orchestration already converts falsy → null via `|| null` (server-lifecycle.ts),
so the DB was never receiving an empty string. Tightening the route layer to
match the same convention keeps the boundary contract consistent and avoids
relying on downstream normalization.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(canvas): expand MCP tool params into per-row labels on block tile
The MCP Tool block on the workflow canvas previously crammed every selected-
tool parameter into a stringified blob under the `Tool` row. Now, when a tool
is selected, the tile reads the cached `_toolSchema` and emits one labeled
SubBlockRow per parameter (matching the Exa block's per-param layout). Labels
reuse `formatParameterLabel` for parity with the editor panel; values pass
through the existing `getDisplayValue` so booleans/numbers/arrays render
identically to other blocks. Deterministic tile height counts expanded rows
so the tile sizes correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(logs): show MCP icon and strip prefix in trace tool spans
Tool spans for MCP calls were rendering the raw id (e.g.
`mcp-f908f259-planetscale_list_organizations`) with the default blank-
square icon. Now they read just the tool name and render the MCP block's
icon and bgColor, matching how workflow-execute tools render.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(logs): lift near-black trace icon backgrounds for dark-mode contrast
Block bgColors below a small luminance threshold (e.g. the MCP block's
#181C1E) rendered nearly invisible against the dark-mode surface
(--bg: #1b1b1b). Adds a tiny adjustBgForContrast helper that floors each
RGB channel at 0x33 only when luminance is below 30,000, leaving every
branded color above that band untouched. Applied to both the trace tree
row and the detail pane.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(logs): fall back to neutral gray for near-black trace icon bgs
#333333 was still too close to the dark-mode surface to read. For bgs
below the luminance threshold (e.g. the MCP block's #181C1E) we now fall
back to DEFAULT_BLOCK_COLOR (#6b7280) — the same neutral the renderer
uses for blocks with no distinct identity. Clearly visible in both
themes; brighter brand colors still pass through.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(db): drop 0209_mcp_oauth migration ahead of staging merge
Staging shipped 0209_smiling_fixer; the MCP OAuth migration will be
regenerated on top of staging as 0210.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(db): regenerate MCP OAuth migration as 0210
Re-runs drizzle-kit generate on top of staging's 0209_smiling_fixer.
Same schema (mcp_server_oauth table + mcp_servers.auth_type / oauth_*
columns) as the dropped 0209_mcp_oauth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(audit): bump route baseline 748 → 749 after staging merge
The post-merge route count is 749 (this branch's OAuth start/callback
plus staging's new route). I had set the baseline to 748 in the merge
conflict resolution — bumping to match reality so the strict audit
passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: remove source-command skill files committed by accident
These were untracked-then-accidentally-staged in
|
||
|
Theodore LiandClaude Opus 4.7
|
f0311a6f5e |
feat(table): chunked dispatcher + workflow cascade (#4672)
* feat(table): chunked dispatcher for workflow-column runs
Replaces the all-rows-at-once runWorkflowColumn with a row-window dispatcher
backed by a new table_run_dispatches row. Each user click inserts a dispatch
row and triggers a trigger.dev task that crawls the table 20 rows at a time,
re-enqueueing itself between windows. The HTTP/Mothership entrypoints return
{ dispatchId } immediately instead of holding the request open for minutes
on multi-thousand-row dispatches.
- Per-row cancel stamps cancelledAt; the dispatcher skips cells whose
cancelledAt > dispatch.requestedAt so a mid-cascade cancel sticks even
under isManualRun.
- Table-wide cancel marks active dispatches cancelled atomically so the
dispatcher bails on its next iteration.
- New 'dispatch' SSE event variant plumbed; client ignores for v1.
* fix(table): eager bulk clear on column run so cells flip immediately
Run-column with run-mode 'all' wasn't visually flipping rows that already
had data — the cell renderer's "value wins" branch kept showing the prior
output behind the queued/running state. The dispatcher only cleared one
window of rows at a time, so most of the column stayed stale until the
cursor walked to it.
Now:
- Dispatcher's `pending → dispatching` transition runs a single SQL UPDATE
that wipes targeted `data` output columns and `executions[gid]` across
every targeted row (mode-aware: 'incomplete' skips fully-filled rows).
- Per-window clear in `dispatcherStep` is gone — rows are pre-cleared,
the loop only filters cancel tombstones / unmet deps and enqueues.
- Optimistic patch in `useRunColumn` mirrors the bulk clear by nulling
output values in the cached row, so the UI flips queued/running
instantly without waiting for the SSE catch-up.
* fix(table): bulk clear honors in-flight execs under mode: 'incomplete'
The eager bulk clear for mode: 'incomplete' only skipped rows that were
already fully filled, so two overlapping dispatches could race — dispatch B
would nuke executions[gid] on a row dispatch A had just stamped 'queued',
flickering the cell and potentially confusing the worker.
Skip any row whose targeted group is currently queued/running/pending — an
'incomplete' run shouldn't touch what another dispatch is actively working
on. The per-walk 'in-flight' eligibility skip already handles rows that
flip in-flight between the clear and the cursor reaching them.
* refactor(table): dispatcher uses batchTriggerAndWait + tag-based cancel
Switch the per-window cell fan-out from fire-and-forget tasks.trigger to
tasks.batchTriggerAndWait. The dispatcher is now a single long-lived
trigger.dev task that loops dispatcherStep until the table is exhausted;
trigger.dev CRIU-checkpoints the parent during each wait so we don't pay
compute while cells execute. Queue depth is bounded at WINDOW_SIZE per
dispatch — no more flooding trigger.dev with a million queued runs.
- dispatcher.ts builds payloads via the new shared buildPendingRuns helper
and calls tasks.batchTriggerAndWait directly. Pre-stamps each cell to
`queued` (jobId=null) so the UI flips instantly.
- table-run-dispatcher.ts is now a plain while-true loop. No
RUN_BUDGET_MS, no self-re-enqueue, no cold-start tax per window.
Cancel:
- New cancelCellRunsByTags(tags) paginates runs.list + runs.cancel(id).
- cancelWorkflowGroupRuns fires the tag-sweep alongside the per-jobId
queue.cancelJob path (preserved for auto-fire cells that have real
jobIds from single tasks.trigger calls).
- Trigger.dev acks the cancel → batchTriggerAndWait resumes → dispatcher
observes the dispatch-row cancel flag → exits.
Side fixes:
- getAsyncBackendType returns 'trigger-dev' whenever taskContext.isInsideTask
is true, regardless of TRIGGER_DEV_ENABLED env. The preview/dev-sim
worker silently routing cell jobs to DatabaseJobQueue (no poller) is
fixed without any env config change.
- runWorkflowColumn skips the dispatcher entirely when trigger.dev is
disabled, running cells inline via DatabaseJobQueue.runInline. HTTP
response returns dispatchId: null in that mode.
- runColumnContract response schema updated to dispatchId.nullable().
* fix(table): show Stop button on optimistic-pending row cells
isExecInFlight required a jobId for `pending` status, gating it as "real
backend pending" vs "optimistic flag only." The row-gutter Stop button
keyed on this — so a freshly clicked Play sat as `pending` (no jobId) and
the user couldn't cancel it until the server-side `queued` stamp arrived
via SSE. With the dispatcher pre-batch stamping cells as `queued` (not
`pending`) and no per-cell jobIds under batchTriggerAndWait, the gap was
worse.
Drop the jobId requirement. `pending` now counts as in-flight everywhere.
Cancel writes `cancelled` to the cell exec authoritatively whether or not
a real trigger.dev run exists yet — cancelling an optimistic cell means
"don't run this," which is correct.
Also collapse isOptimisticInFlight into isExecInFlight since the two
helpers are now identical.
* refactor(table): loop-in-cell cascade + dispatcher-everywhere routing
Two coupled changes:
1. Cell-task runs the row's full cascade in-process. executeWorkflowGroupCellJob
acquires a Redis lock per (tableId, rowId) with heartbeat (10s/30s TTL),
then loops through eligible workflow groups for the row. One cell-task =
one row's full cascade, not N. Resume worker holds the same lock and
continues the cascade after a HITL resume. Shared withCascadeLock helper
in lib/table/cascade-lock.ts.
2. Every cell-enqueue goes through the dispatcher. The implicit
scheduleRunsForRows reactor in service.ts is removed — 8 callsites
(insertRow, batchInsertRows, upsertRow, updateRowsByFilter,
batchUpdateRows, addWorkflowGroup, updateWorkflowGroup) now fire
runWorkflowColumn with mode: 'incomplete', isManualRun: false. HTTP
routes that call updateRow directly also fire runWorkflowColumn
afterwards. scheduleRunsForTable / scheduleRunsForRowIds deleted;
scheduleRunsForRows demoted to private (only the TRIGGER_DEV_ENABLED=false
fallback uses it). skipScheduler flag dropped from UpdateRowData /
BatchUpdateByIdData — no longer meaningful since there's nothing implicit
to suppress.
Plumbed isManualRun through the dispatch row (new is_manual_run column,
default true) so auto-fire callers honor autoRun: false and don't re-run
completed cells.
Stamp 'pending' (not 'queued', executionId: null) before
batchTriggerAndWait — cell-task writes its own 'queued' on lock acquire.
Small UI polish: row gutter Play button spacing, "Delete workflow" →
"Delete column" label, optimistic-pending cells now show Stop button
(isExecInFlight no longer requires jobId).
* fix(table): SQL cancellation guard allows worker to claim a null-execId cell
The dispatcher's pre-batch `pending` stamp leaves executionId unset so any
cell-task that wins the cascade lock can claim the cell. The cancellation-
guard SQL clause was rejecting these claims because it tested
`executions->gid IS NULL` (whole exec missing) but the pre-stamp leaves
the exec present with executionId=null.
Add a third carve-out: `executions->gid->>'executionId' IS NULL`. Now the
guard reads "write allowed if no exec exists, OR no executionId is set
yet, OR the executionId matches ours."
Symptom: every cell-task's first markWorkflowGroupPickedUp call would log
"SQL guard saw cancelled" and skip, leaving cells stuck at the dispatcher's
pending stamp.
* fix(table): dispatcher cursor starts at -1 so position 0 is included
The dispatcher's row-window SELECT is `position > cursor` for exclusive
lower-bound semantics. With cursor initialized to 0, position-0 rows were
never picked up — every dispatch silently skipped the table's first row.
Start cursor at -1 instead. First window's filter `position > -1` matches
position 0; subsequent iterations advance to `lastPosition` which then
correctly excludes already-processed rows.
* refactor(table): align optimistic UI with new dispatcher; sticky cancel via 'new' mode
Fix 0: new `DispatchMode = 'new'` for auto-fire callsites. Eligibility skips
rows with any prior `executions[gid]` entry — cancelled / errored / completed
cells stay sticky until a manual run. Dispatcher's windowed SELECT pushes
`NOT jsonb_exists_any(...)` to SQL so CSV imports into mostly-attempted
tables don't pay a per-window load+JS-filter. `batchInsertRows` drops its
`rowIds` payload (keeps dispatch scope tiny on big imports).
Fix A/B/D: client optimistic patches now mirror the backend's actual
invariants. `useCreateTableRow.onSuccess` stamps eligible groups via
`optimisticallyScheduleNewlyEligibleGroups` so newly-inserted rows show
`Queued` instantly. `useCancelTableRuns.onMutate` distinguishes optimistic-
only pending (`executionId == null` — strip silently) from real worker
claims (stamp cancelled; SSE will reconcile). Drop `onSettled` invalidation
on `useUpdateTableRow` / `useBatchUpdateTableRows` to kill the
delete-cell flicker.
Fix C: active-dispatches overlay. New `listActiveDispatches` helper,
contract, and `GET /api/table/[tableId]/dispatches` route. `kind:'dispatch'`
SSE events carry scope+cursor+mode on every transition. New
`useActiveDispatches` hook + `resolveCellExec` synthesize a virtual
`pending` exec for cells in an active dispatch's scope ahead of cursor —
queued indicators now survive page refresh during long Run-all dispatches.
`cancelWorkflowGroupRuns` emits `kind:'dispatch',status:'cancelled'`
events so the overlay clears without a refetch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(table): unify trigger.dev and inline dispatcher paths
`runWorkflowColumn` now always inserts a `table_run_dispatches` row and
drives the dispatcher state machine. The trigger.dev / in-process branch
narrows to a single line: trigger.dev fires `tableRunDispatcherTask` (which
calls the new `runDispatcherToCompletion`), the inline path calls the same
helper fire-and-forget. Deletes `scheduleRunsForRows` and
`stampQueuedOrCancel` — the inline-fallback no longer duplicates window
walking, SSE emission, or cancel.
The dispatcher's window-execute call goes through `JobQueueBackend`:
- New `batchEnqueueAndWait` interface method.
- Trigger.dev impl wraps `tasks.batchTriggerAndWait` behind a
`taskContext.isInsideTask` guard (clear error if called from outside a
task).
- Database impl skips `async_jobs` entirely — `Promise.all` over
`options.runner(payload, signal)` per item, with per-cell AbortControllers
tracked by `cancelKey` for cancel.
`cancelInlineRun` moves to the interface as `cancelByKey` so
`cancelWorkflowGroupRuns` no longer reaches into the database backend.
Fix `mode: 'new'` SQL filter:
- `${array}::text[]` interpolated as a tuple-cast which Postgres rejected
("cannot cast type record to text[]") and every inline dispatch silently
failed. Switched to `ARRAY[${sql.join(...)}]::text[]`.
- Predicate was `jsonb_exists_any` ("any one targeted group present"),
which excluded rows that needed at least one group re-run after a
downstream output was deleted. Switched to `jsonb_exists_all` — per-group
JS eligibility handles the rest.
Cascade-loop workflowId bug: `runRowCascadeLoop` was not threading the new
group's `workflowId` when advancing across groups. The cell-task ran the
previous group's workflow against the next group's cell, terminating
`completed` with empty `accumulatedData`. Fixed by tracking
`currentWorkflowId` alongside `currentGroupId` / `currentExecutionId`.
Client optimistic-patch tightening:
- `useRunColumn.onMutate` mirrors server eligibility — skip cells with
unmet deps so unmet rows don't flash Queued and get stuck (no SSE will
arrive for cells the server skipped).
- `resolveCellExec` overlay synthesizes a virtual `pending` only when
`areGroupDepsSatisfied` is true. Rows with unmet deps render Waiting,
matching the dispatcher's actual behavior.
Cleanup from /simplify pass:
- Use `generateShortId(20)` instead of
`generateId().replace(/-/g, '').slice(0, 20)`.
- Inline `batchEnqueueAndWait` no longer allocates synthetic ids
(returned `string[]` is unused).
- Flattened the per-cell `tracked` array — only push entries that
registered controllers, drop the null placeholders.
- Extracted `runDispatcherToCompletion` to share the loop between the
trigger.dev wrapper and the in-process path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(table): backend running counter, dep-aware retrigger, sidebar polish
Counter (Fix 1): top-right "X running" + per-row badge are now
backend-bootstrapped via a count on `user_table_rows.executions ->> 'status'
= 'running'` returned alongside active dispatches. SSE `kind: 'cell'` events
compute a delta from `prev → next` status to keep the cache live; cell
events for rows outside the loaded page slice trigger a run-state refetch.
On `pruned` we invalidate the cache. Counts only worker-claimed `running`
cells — optimistic queued/pending no longer inflate the badge, and rows
outside the loaded page slice are counted too.
Sidebar (Fix 2 + 3a): `Run after` no longer ticks every column by default
for new groups (empty list). Save is disabled with an inline error when
auto-run is on with zero deps. `edit-group` mode anchors the left-of-current
filter to the group's leftmost column, so a workflow can only depend on
columns to its left.
Reorder scrub (Fix 3b): `updateTableMetadata` walks the schema's workflow
groups when `columnOrder` is in the patch and drops any dep whose new
position lands at or after the group's leftmost column (uses the existing
`stripGroupDeps` helper). Metadata + schema updates land atomically.
Server returns ordered columns (Fix 3b cont'd): `getTableById` /
`listTables` now sort `schema.columns` by `metadata.columnOrder` before
returning, via a new `applyColumnOrderToSchema` helper. Every consumer
(grid, sidebar, copilot, mothership) gets one ordered list — the sidebar's
leftmost-group-column anchor now points at the right index.
Dep-aware retrigger (Fix 4): editing a value that a downstream workflow
depends on now re-runs that workflow.
- `deriveExecClearsForDataPatch` returns
`{ executionsPatch, inFlightDownstreamGroups }`. Walks
`schema.workflowGroups[].dependencies.columns` for every column in the
patch, clears terminal-state downstream entries, and reports in-flight
entries.
- `updateRow` calls `cancelWorkflowGroupRuns` + `runWorkflowColumn`
(`mode: 'incomplete' + isManualRun: true`) for in-flight downstream
groups, then always fires `runWorkflowColumn({ mode: 'new' })` for the
cleared groups. Skips both when `executionsPatch` is provided by the
caller — those are cell-task / cancel writes that would otherwise spawn
a recursive flood of dispatches per partial-write.
- `cancelWorkflowGroupRuns(tableId, rowId, { groupIds? })` accepts a
per-group filter so the cancel only touches the affected groups, not
every in-flight cell on the row.
- `pickNextEligibleGroupForRow` now treats a dispatcher pre-stamp
(`pending` + `executionId: null`) as claimable — the cascade-loop is the
real owner. Without this, the dispatcher's pre-stamp of downstream
groups made the cascade-loop see them as "in-flight" and skip them,
stranding `pending` cells forever.
- `optimisticallyScheduleNewlyEligibleGroups` extends the cache patch to
flip dep-touched groups to `pending` regardless of their current status,
matching the server's cancel-then-rerun behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(table): paused workflow cells route through executeResumeJob; render Pending + viewable
Three connected issues with workflows that pause mid-cell (e.g. wait blocks):
1. `/api/resume/poll` (the time-pause auto-resumer) called
`PauseResumeManager.startResumeExecution` directly, bypassing
`executeResumeJob` from `background/resume-execution.ts`. The wrapper is
where the cell-context restoration + cascade-loop continuation lives —
without it, the resumed workflow ran to completion but never wrote the
terminal state back to the table cell. Cell stays `pending` forever
even though the underlying execution finished.
Fix: dynamically import `executeResumeJob` and use it for the
`'starting'` branch. Same primitive the trigger.dev `resumeExecutionTask`
wraps — calling it directly handles both trigger.dev-disabled local dev
and trigger.dev-enabled prod identically.
2. The cell renderer mapped `status: 'pending'` to `kind: 'queued'` (gray
"Queued" badge) regardless of whether the run had started. A HITL-paused
run has `status: 'pending'` + `jobId` prefixed `paused-` + a real
`executionId` — semantically very different from "queued, hasn't run."
Now renders as `pending-upstream` (the existing Pending pill) for
paused-jobId rows.
3. Right-click "View execution" was disabled for `pending` cells (gated to
`completed | error | running`), so users couldn't open the trace for a
paused execution. Paused runs have a viewable trace (the executionId is
real and the log row exists). Both the per-row context menu and the
action-bar derivation now recognize `pending` + `paused-` jobId as a
started run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(table): typewriter reveal for SSE-driven workflow cell values
Workflow-output cells now reveal their text character-by-character when an
SSE update lands, while page reloads and virtualization remounts still paint
the value instantly. A first-render guard inside the new useTypewriter hook
distinguishes hydration from live updates with no plumbing through the cell
tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(table): address bugbot/greptile review feedback
Two P1 issues + one cleanup from the bot reviewers:
1. **Double-dispatch + completed-output wipe.** Both PATCH row routes
(`app/api/table/[tableId]/rows/[rowId]` and
`app/api/v1/tables/[tableId]/rows/[rowId]`) were firing a second
`runWorkflowColumn({ mode: 'incomplete' })` after `updateRow` returns.
`updateRow` already fires `mode: 'new'` internally for user edits, so
the second call created a concurrent dispatch. Worse, the
`mode: 'incomplete'` path's `bulkClearWorkflowGroupCells` wipes ALL
targeted output columns on any row where any one column is empty —
meaning sibling-group completed outputs could be erased. Removed both
route-level calls; auto-dispatch lives entirely in `updateRow`.
2. **`runWorkflowColumn` log-spamming on plain tables.**
`if (targetGroups.length === 0) throw new Error(...)` fired on every
row insert/update for tables without any workflow groups (the
majority). Every caller wraps with `.catch(logger.error)`, so each
PATCH produced an error-level log. Return `{ dispatchId: null }`
silently — manual `runWorkflowColumn` callers pass `groupIds`
explicitly so they can't reach this branch.
3. **`isManualRun` plumbed through dispatch SSE events.** Late-arriving
`kind: 'dispatch'` events for dispatches not in the initial fetch
were hardcoding `isManualRun: false`. Added the field to the event
shape, emit it from `dispatcherStep` (pending → complete, dispatching
transitions) and `markActiveDispatchesCancelled`, and consume it in
the SSE handler with a sensible fallback for legacy emits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(table): row executions sidecar + left-to-right dep retrigger + cancel counter refresh
Split per-row workflow-group execution state out of the user_table_rows.executions
JSONB column into a new table_row_executions sidecar keyed by (row_id, group_id).
Dispatcher filters, "X running" counter, bulk clears, and the cancellation guard
all hit indexed columns instead of walking JSONB. Wire shape unchanged — server
merges sidecar rows back into row.executions on the way out.
Also:
- deriveExecClearsForDataPatch now walks workflowGroups left-to-right with a
propagating dirtied-column set so transitive dep chains (edit col A → group 1
re-runs → group 2 depends on group 1's output → group 2 re-runs) collapse to
a single forward pass.
- useCancelTableRuns.onSettled invalidates the activeDispatches query so the
top-right counter and row gutter Stop button refetch from the server after
any Stop (per-cell, row, or table-wide). countRunningCells is the source of
truth; client no longer needs duplicate state.
Three migrations on this branch (0209 + 0210 + new sidecar) collapsed into one
since the feature is unreleased.
* fix(table): address remaining cursor/greptile review feedback
- Mothership update_row no longer double-dispatches. updateRow already fires
the auto-cascade internally; the second `mode: 'incomplete'` call here
raced with it and could bulk-clear sibling-group outputs.
- SSE dispatch events no longer dropped when the activeDispatches cache is
cold. Seed an empty TableRunState if the initial fetch hasn't landed yet
so the queued overlay doesn't lose the first dispatch event.
- batchUpdateRows now runs cancel+rerun for per-row in-flight downstream
groups, mirroring updateRow. Without this, dep edits in a batch left
running workflows reading stale upstream values.
* fix(table): cancel prior runs, scope batch insert dispatch, recover orphan pre-stamps
Addresses cursor + greptile review feedback on table dispatcher edge cases:
- Manual table-wide Run-all / Run-column now cancels prior active dispatches
AND in-flight cell workers before bulk-clearing. Without this, mode:'all'
deleted running sidecar rows out from under their workers (which kept
writing into the wiped state) and a second Run-all could enqueue overlapping
cells racing on the same rows. Row-scoped manual calls (dep-edit cascade)
are excluded — those already cancel their own scope.
- batchInsertRowsWithTx now scopes its auto-dispatch to the newly-inserted
row ids. Without this, after the sidecar migration the NOT EXISTS filter
matches every existing row (zero sidecar entries), so a CSV import would
walk the entire table dispatching workflow runs on every pre-existing row.
- classifyEligibility carve-out: pending + executionId=null is an orphan
pre-stamp (cascade-lock contention, batchEnqueueAndWait failure, etc.),
treated as claimable so future dispatchers can re-stamp instead of skipping
it as 'in-flight' forever. Matches pickNextEligibleGroupForRow's logic.
- On batchEnqueueAndWait failure, dispatcherStep now sweeps the orphan
pre-stamps it wrote for the failed batch so the cells don't render Queued
forever; the next user action picks them up cleanly.
* fix(table): row-scoped Refresh cancels in-flight; counter includes queued/pending
- runWorkflowColumn now cancels prior in-flight cells for row-scoped manual
runs too (context-menu Refresh on a row subset, action-bar Refresh on
selected rows). Previously only the table-wide path cancelled, so a
row-scoped Refresh would bulk-clear running sidecar rows without aborting
workers. Per-row cancel skips markActiveDispatchesCancelled so unrelated
dispatches keep running.
- countRunningCells now counts all in-flight statuses (queued / running /
pending) instead of just running. The row gutter Run/Stop button reads
this map — with the old behavior, clicking Play during the queued window
would re-enqueue an already-queued cell. SSE applyCell handler updated
to use isExecInFlight so client deltas track the same semantics.
* fix(table): per-row Stop tombstones ahead-of-cursor rows during Run-all
Per-row Stop only cancelled sidecar rows already in flight. A row the
dispatcher hadn't reached yet had no exec record, so Stop was a no-op there
— the dispatcher would later walk to it, classify the group eligible, and
re-fire workflows the user thought they stopped.
cancelWorkflowGroupRuns now, for a per-row cancel, checks active dispatches
whose scope covers the row and writes `cancelled` tombstones (cancelledAt =
now) for the at-risk groups that don't already have a sidecar entry. The
dispatcher's existing `cancelledAt > dispatch.requestedAt` filter then skips
them when the cursor arrives. onConflictDoNothing guards against clobbering
a concurrently-written entry; the active-dispatch check avoids stamping
spurious cancels on idle rows.
* fix(table): seed dispatch overlay on Run; surface batch-enqueue failures as error
- useRunColumn.onSuccess invalidates the activeDispatches query so the
resolveCellExec queued overlay populates immediately for ahead-of-cursor
rows (scrolled-in / refetched), instead of waiting for the first dispatch
SSE. Targeted at activeDispatches only — the rows cache stays owned by
useTableEventStream.
- On batchEnqueueAndWait failure, dispatcherStep now flips the orphan
pre-stamps to a terminal `error` state and emits a cell SSE event, rather
than deleting them. The cursor still advances past the window, but the
dropped cells are now visible (Error pill) instead of silently empty, stay
out of the in-flight set, and re-run on the next manual run.
* fix(table): seed dispatch overlay on Run; surface batch-enqueue failures as error
- useRunColumn.onSuccess invalidates activeDispatches so the resolveCellExec
queued overlay populates immediately for ahead-of-cursor rows instead of
waiting for the first dispatch SSE. Rows cache stays owned by SSE.
- On batchEnqueueAndWait failure, dispatcherStep flips orphan pre-stamps to a
terminal error state (+ cell SSE) instead of deleting them, so the dropped
window is visible (Error pill) rather than silently empty and re-runs on the
next manual run.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Waleed
|
8d7bbbc670 |
chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting - Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files - Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors) - Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities - Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time - Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access - Add check:utils script to package.json * chore(utils): replace deepClone wrapper with structuredClone built-in deepClone() was a one-line wrapper around structuredClone(), which is universally available in Node 17+ and all modern browsers. Removing the abstraction reduces indirection and means contributors don't need to learn a project-specific name for a well-known built-in. - Remove deepClone from packages/utils/src/object.ts and index.ts - Replace all 17 call sites with structuredClone() directly - Update check:utils script suggestion text - Update CLAUDE.md and global.md docs * fix(utils): add missing biome noRestrictedImports rule and correct truncate docs - Add noRestrictedImports to biome.json under style — bans nanoid and uuid package imports at lint time (crypto.randomUUID/randomBytes are caught by the check:utils grep script which handles global property access) - Correct truncate() TSDoc and parameter name: sliceLength makes it clear that total output length is sliceLength + suffix.length, matching the behavior all callers were already written to expect * fix(utils): add missing getErrorMessage imports at 4 call sites The sweep agents added getErrorMessage calls without the corresponding import in 4 files, causing test failures. Added the missing imports. * fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue - Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for internal package imports; inline the jitter crypto call directly - Add missing getErrorMessage imports to 32 files where the sweep added calls without the corresponding import (caught by type-check and test runs) - Remove accidental getErrorMessage import from crowdstrike/query/route.ts which has its own domain-specific getErrorMessage for parsing CrowdStrike's JSON error format - Fix use-sub-block-value.ts type error from structuredClone narrowing: add 'as T' cast at emitValue callsite (safe — valueCopy is always a structural copy of newValue) * fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage The catch block was calling the local getErrorMessage function which parses CrowdStrike API JSON responses, not JavaScript Error objects. Use toError(error).message to correctly extract the message from a caught value in this context. |
||
|
|
c9118e775b |
feat(files): folders, multiselect, vfs update (#4572)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * feat(files): folders + vfs update * address comments * address comments * cleanup unnused code * address comments * perf improvements * address next set * cycle detect * error handling * path improvements * cleanup, best practices * react query best practices: targeted invalidation, optimistic updates, key factory hierarchy - Add workspaceLists(workspaceId) intermediate key level to both workspaceFilesKeys and workspaceFileFolderKeys so invalidation targets only the affected workspace instead of all workspaces - Replace all lists() invalidation calls with workspaceLists(workspaceId) across every mutation (upload, rename, delete, restore, update content, folder mutations) - Add optimistic updates to useRenameWorkspaceFile and useUpdateWorkspaceFileFolder with onMutate snapshot, onError rollback, onSettled reconciliation - Move storage key into the content() factory as optional param so query keys are always built through the factory (useWorkspaceFileContent, useWorkspaceFileBinary) - Fix AnimatePresence wrapping in FilesActionBar so exit animation fires on deselect - Fix ResourceColGroup to use percentage weights instead of pixel widths to prevent horizontal scroll on narrow viewports * add shift-click range selection and selection-aware context menu for files - Extend SelectableConfig.onSelectRow with optional shiftKey param; DataRow captures shiftKey before onCheckedChange fires via a ref so the Radix Checkbox interaction chain stays intact - Implement shift-click range selection in files.tsx using lastSelectedIndexRef; tracks last-selected index in visibleRowIds to compute the range - Reset lastSelectedIndexRef on deselect and select-all - Add selectedCount prop to FileRowContextMenu; hide Open and Rename when multiple items are selected, show "Delete N items" / "Download N items" labels in multi-select mode * add Move submenu to file context menu and fix shift-click anchor update - Add nested Move submenu to FileRowContextMenu using DropdownMenuSub/SubTrigger/SubContent; shows available folders filtered by selection, converts '__root__' -> null for moving to the root level - Add handleContextMenuMove in files.tsx that calls moveItems.mutateAsync directly (no modal) and clears selection on success - Fix shift-click range selection: update lastSelectedIndexRef after range select so chained shift-clicks extend from the new anchor point correctly * fix move submenu: use folder names with tree-ordered indentation instead of stale paths - Compute folder depth from parentId chain client-side (avoids stale server-computed path field) - Tree-order folders so parents appear before their children, sorted by sortOrder then name - Show folder.name instead of folder.path so optimistic renames are reflected immediately - Indent each folder by depth * 12px in the submenu so po/shit renders as 'shit' indented under 'po' - MoveOption gains optional depth field; contextMenuMoveOptions is a separate memo from moveFolderOptions (modal keeps its existing path-label behavior) * fix shift-click anchor drift and remove dead stopPropagation constant - Remove dead stopPropagation const in resource.tsx (replaced by handleSelectRowClick) - Reset lastSelectedIndexRef when visibleRowIds changes so search/filter/folder navigation doesn't leave a stale anchor that produces wrong ranges on the next shift-click - Update lastSelectedIndexRef in handleRowContextMenu when right-clicking resets selection to a single item, so the anchor matches the newly-selected row - Add visibleRowIds to handleRowContextMenu deps (now reads it to compute anchor index) - Remove moveItems.mutateAsync from handleContextMenuMove deps per project convention (.mutateAsync is stable in TanStack v5) * complete workspace files feature: audit logs, posthog events, folder restore, empty state, keyboard shortcuts, storage indicator, breadcrumb rename - Audit + PostHog: wire file_renamed, file_deleted, file_moved, file_bulk_deleted, folder_created, folder_renamed, folder_deleted, folder_moved events to all file/folder API routes - Add AuditAction.FOLDER_UPDATED, FILE_MOVED, FOLDER_MOVED to audit types - Folder restore: server function, contract, API route (POST /files/folders/[folderId]/restore), hook (useRestoreWorkspaceFileFolder), Recently Deleted integration with new File Folders tab - Empty state: contextual emptyMessage passed to <Resource> based on search/filters/folder context - Keyboard shortcuts: Delete/Backspace deletes selection, Escape deselects, Cmd+A selects all (list view only, input-aware guard) - Storage indicator: useStorageInfo drives compact "used / limit" display in file list header via leadingActions - Breadcrumb rename: current folder breadcrumb gains Rename dropdown + inline editing via breadcrumbRename (useInlineRename) - Resource: thread leadingActions prop from ResourceProps to ResourceHeader * cleanup: accessibility, emcn design tokens, react best practices across workspace UI - Add sr-only ModalDescription to dialogs/modals for accessibility - Replace hardcoded colors and z-indices with design token CSS variables - Apply emcn design review fixes across tables, knowledge, logs, settings, workflows * fix audit and posthog: FOLDER_RESTORED action on restore, fire folder_moved event separately from file_moved * sidebar: add Files section with nested folder tree; polish move UX and cleanup - Files section in sidebar shows folder/file tree with expand/collapse, matching Workflows section structure; collapsed sidebar shows flyout menu - Move action bar now uses nested DropdownMenuSub tree instead of flat modal - Context menu and action bar share renderMoveOption from move-options.tsx - FolderInput added to emcn icons barrel; all FolderInput imports migrated - Drag ghost uses CSS vars (--border, --shadow-medium, --z-toast) - Selection pruning converted from useEffect to render-time comparison - Keyboard listener stabilized with handleBulkDeleteRef pattern - toError() used consistently in restore and move route handlers * remove Files section from sidebar * restore Files nav item in sidebar workspace section * fix infinite re-render on files page - revert selection pruning to useEffect * add filefolder resource type for ingesting workspace file folders * export filefolder tree types; add toast feedback for file/folder mutations * regenerate migration as 0208 after rebase onto staging * add workspaceFileFolder to schema mock * add FILE_MOVED, FOLDER_MOVED, FOLDER_UPDATED to audit mock * add filefolder ChatContext kind and wire through schema and resolver * add filefolder to AgentContextType * add filefolder to chat context kind registry; fix resolver to use workspaceFiles table * add .deepsec to gitignore * cleanup: effect, emcn tokens, mutation error handling - Replace selection-pruning useEffect with inline state adjustment during render - Fix drag overlay using invalid --accent HSL token → --brand-secondary; z-50 → z-[var(--z-dropdown)] - Move static inline styles on context menu trigger div to className - Add missing onError toast to useUpdateWorkspaceFileFolder, useRestoreWorkspaceFileFolder, useRestoreWorkspaceFile * lint * fix: remove duplicate handleCopilotStopGeneration from rebase * feat(copilot): folder-aware file context in WORKSPACE.md * feat(copilot): add move operation to file manage API * fix(files): make targetFolder optional in move file contract * perf(files): parallelize buffer fetches, fix N+1 folder queries, stabilize drag useMemo - download route: fan out all fetchWorkspaceFileBuffer calls with Promise.all before zip assembly so 100 files resolve in one round-trip instead of sequentially - getWorkspaceFileFolder: replace per-ancestor SELECTs with a single workspace-wide folder load + buildWorkspaceFileFolderPathMap, making depth irrelevant to query count - ensureWorkspaceFileFolderPath: pre-load all workspace folders in one SELECT before the segment loop; resolve existing segments from an in-memory map; only hit the DB to CREATE missing segments; conflict retry path preserved and also updates the map - files.tsx rowDragDropConfig: move activeDropTargetId into a ref so the useMemo does not recompute on every drag-over event * fix(files): remove files/ path stripping, fix stale path in optimistic update - splitWorkspaceFilePath: remove the unconditional .replace(/^files\//, '') that clobbered paths for files inside a folder literally named "files" - useUpdateWorkspaceFileFolder: when a name update is in flight, recompute the path field for the renamed folder (replace last segment) and propagate the new prefix to all descendant folders so breadcrumbs stay correct during the optimistic window * fix(files): revert broken ref opt, clean 409 on restore, null parentId on orphaned restore - files.tsx: revert the activeDropTargetId ref optimization — the ref doesn't trigger re-renders so the drop-target highlight never updated during drag; activeDropTargetId is back in state and in the rowDragDropConfig deps - restore/route.ts: catch Postgres 23505 unique-constraint violation and return a clean 409 instead of leaking the raw error as 400 - restoreWorkspaceFileFolder: check if the parent folder is still archived before restoring; if it is, restore to root (parentId: null) so the folder is never orphaned under an archived parent * feat(search): show folder path for files in cmd-k modal, strip extraneous comments - FileItem interface with folderPath?: string[] added to search modal utils - MemoizedFileItem component renders folder breadcrumb identically to MemoizedWorkflowItem — truncated path segments on the right with / separators - FilesGroup rewritten as a dedicated memo component (was createIconGroup factory) so it accepts FileItem[] and includes folderPath segments in the search value - searchModalFiles in sidebar splits f.folderPath string into string[] segments - search-modal.tsx typed to FileItem and includes folderPath in filterAndSort - Remove self-explanatory "Phase 1" section label from download route - Remove redundant TSDoc on the unique index in db schema * fix(workspace-files): audit fixes — transaction, status codes, contract refinements, guards * fix(vfs): pass folderPath separately so buildWorkspaceMd groups files correctly * fix(types): narrow unknown fileInput with Record cast after object guard * fix(routes): replace instanceof Error with toError() across new workspace file routes * improvement(files): cleanup pass — remove unnecessary useCallbacks, consolidate emcn icon imports - Remove useCallback from 5 drag-event handlers in DataRow (passed to native <tr> elements, no observer) - Remove stable useCallback fns from 3 useMemo deps arrays in files.tsx (editingId/editValue remain) - Merge all @/components/emcn/icons subpath imports into barrel (files.tsx, action-bar, file-row-context-menu) * fix(files): apply activeSort to folders, reject drop onto current parent folder - visibleFolders now respects activeSort column (name/updated/created) and direction so folder ordering stays consistent with file ordering - isInvalidDropTarget now returns true when all dragged items are already direct children of the target folder, preventing a no-op move mutation * fix breadcrumb * add new tools to rename, create, delete folders * move more ui actions into orchestration dir * address comments * fix params * fix tests * address comments * improve error codes * address comments * address more nits * fix mcp server error code --------- Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: waleed <walif6@gmail.com> |
||
|
WaleedandClaude Opus 4.7
|
d721dc3358 |
feat(enterprise): add data drains for continuous export to S3 / webhook (#4440)
* feat(enterprise): add data drains for continuous export to S3 / webhook * chore(data-drains): regenerate migration on top of staging + bump route baseline Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(data-drains): clarify retention pairing is user-coupled, not enforced Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(data-drains): preserve explicit forcePathStyle=false + reserve x-sim-signature Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(data-drains): drift guard ensures every webhook header is reserved Asserts that any header buildHeaders writes is rejected when reused as a custom signatureHeader. Adding a new metadata header without mirroring it into RESERVED_SIGNATURE_HEADER_NAMES now fails CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Waleed
|
68f66ba3f3 |
fix(credentials): clear stored refs on credential delete to prevent silent cascade orphaning (#4418)
* fix(credentials): clear stored refs on credential delete to prevent silent cascade orphaning * fix(testing): sync auditMock with new CREDENTIAL_RECONNECTED action * improvement(credentials): parallelize independent ref-clearing scans |
||
|
Waleed
|
af55bad491 |
fix(uploads): direct-to-upload workspace files + shared transport (#4407)
* fix(uploads): direct-to-S3 workspace files + shared transport
* chore(testing): centralize posthog and storage-service mocks
* fix(uploads): address PR review — abort propagation, orphan cleanup, error handling
- Throw immediately on AbortError in KB retry loop (no useless 14s backoff)
- Cleanup S3/Blob object on quota or size-cap rejection in registerUploadedWorkspaceFile
- Enforce MAX_WORKSPACE_FILE_SIZE at registration (defense vs presigned PUT lying about size)
- Handle non-OK / non-JSON responses in workspace-files upload paths
* fix(uploads): add Zod contracts for workspace presigned/register routes
* fix(uploads): correct BlobServiceClient type name in headBlobObject
* fix(uploads): address PR review — typo, complete-failure cleanup, double-increment
* fix(uploads): preserve fallback size and reuse existing display name on re-register
* fix(uploads): surface server error message and bypass quota for local-storage fallback
* fix(uploads): align register response schema with UserFile; skip presigned for KB large files
- registerWorkspaceFileResponseSchema now matches the UserFile shape the route actually returns; previous schema required workspace DB-row fields that were never populated, causing requestJson validation to reject successful uploads.
- KB batch presigned fetch now skips files >= LARGE_FILE_THRESHOLD since multipart bypasses the per-file presigned URL anyway.
* fix(uploads): idempotent register skips duplicate audit/posthog; add edge-case tests
- registerUploadedWorkspaceFile now returns { file, created } so the route can skip captureServerEvent and recordAudit on idempotent re-register (existing metadata reused). Previously a re-register fired duplicate analytics + audit log entries.
- Add tests covering: idempotent re-register skips audit/analytics, isNetworkError matches econnreset/timeout/etc keywords, multipart complete failure fires action=abort cleanup.
* fix(uploads): include 50MiB boundary in batch presigned fetch
* fix(uploads): trust HEAD size to prevent quota inflation
The head.size > 0 fallback let a client PUT 0 bytes and register
with an inflated size, debiting quota without storing data. HEAD
on an existing object always returns the true byte count, so trust
it directly — a genuine 0-byte file correctly contributes 0.
* fix(uploads): audit verified file size, not client-supplied
* fix(uploads): handle register retries and name-collision races
Two bugs in registerUploadedWorkspaceFile:
1. Register retry could orphan storage. When a successful response
was lost on the wire and the client retried, the quota check saw
the bytes already counted, failed, and cleanupOrphan deleted the
already-registered storage object — leaving the DB row pointing
to nothing. Fix: check getFileMetadataByKey before quota guard
and short-circuit on existing record.
2. Concurrent same-named uploads could lose data. allocateUniqueWorkspaceFileName
is best-effort; two racing uploads can pass it and both attempt
the same display name. The loser's insert hits 23505, the catch
block called cleanupOrphan, and successfully-uploaded bytes
were deleted. Fix: retry on 23505 with a fresh allocateUniqueWorkspaceFileName,
matching the pattern in uploadWorkspaceFile. Throw FileConflictError
after exhaustion.
* fix(uploads): retry transient DirectUploadErrors at outer KB level
The KB outer retry only triggered on isNetworkError, missing
transient 5xx from S3/Azure (DirectUploadError code
DIRECT_UPLOAD_ERROR or MULTIPART_ERROR). Adds isTransientUploadError
and retries on it, restoring resilience for small-file presigned
PUTs against flaky cloud storage.
* fix(uploads): only retry transient 5xx, not deterministic 4xx
DirectUploadError now carries the HTTP status. isTransientUploadError
gates on 5xx so callers don't loop on 400/403/404 (e.g., malformed
request, expired signature). Multipart per-part retry also short-circuits
on 4xx — same reasoning.
* refactor(uploads): collapse getFileContentType into resolveFileType
The two helpers differed only in whether application/octet-stream
falls back to the extension map. Add an option flag to resolveFileType
and keep getFileContentType as a thin wrapper for direct-PUT callers
that need to preserve the exact browser-reported content-type.
* chore(uploads): trim verbose comments
Drop inline comments that restate code ("Use the full storageKey as fileName"),
collapse a multi-line block comment into a tighter TSDoc on the existence
check, and prune verbose vitest file headers — describe blocks already
document what's tested.
* fix(uploads): regenerate fileId per insert retry; require cloud storage for register
* fix(uploads): cap formdata fallback at 100MB; drop unused size param
* fix(uploads): abort multipart on get-part-urls failure; retry register on transient errors
* fix(uploads): drop vestigial size field from register contract
* fix(uploads): abort multipart on complete-fetch throw
* fix(uploads): set kb presignedEndpoint fallback; race-safe blob HEAD
* fix(uploads): include ?type=knowledge-base on kb presigned fallback
* fix(uploads): remove abort listener on xhr timeout
* fix(uploads): add timeout/abort to kb api fallback upload
|
||
|
Waleed
|
879dab9f19 |
feat(table): make plan table limits configurable via env vars (#4406)
* feat(table): make plan table limits configurable via env vars * fix(table): coerce env table limits to number for skipValidation env * improvement(env): extract envNumber helper for numeric env coercion * improvement(knowledge): use envNumber helper for KB_CONFIG_* env reads * fix(testing): add envNumber to env mock factory * fix(env): allow zero in envNumber for max-throughput configs * fix(env): add min option to envNumber for strict-positive configs |
||
|
Vikhyath Mondreti
|
af859cd508 |
feat(workflows): lock/duplicate improvements for workflows (#4387)
* feat(workflows): lock/duplicate improvements * fix duplicate var remap bug * address comments * remove dead vars * fix tests * address comments * code cleanup * address comments * address comments * minor change * remove dead code |
||
|
WaleedandClaude Opus 4.7
|
e2b3ae43e3 |
fix(terminal): correct error/cancel block status in logs panel (#4372)
* fix(terminal): correct error/cancel block status in logs panel Three bugs in the workflow editor's terminal/logs panel where block status diverged from the engine's truth on error paths: 1. **Errored block shown as "canceled"** — when the SSE 'add' mode produced a duplicate entry on block error and `cancelRunningEntries` then swept the original placeholder. 2. **Upstream blocks stuck on "Running"** — terminal events arrived before the engine's last block events under reconnect/timeout, so the live panel never received the per-block terminal state. 3. **Phantom "Run Error" pseudo-row** — the failing block rendered as "canceled" while a synthetic row carried the real error text. Fixes: - **Fix B** (`addConsoleErrorEntry`): when a running placeholder exists for `(blockId, executionId)`, route through `updateConsoleErrorEntry` instead of creating a second entry. Aligns 'add' mode with the existing 'update' mode behavior. - **Fix C** (`reconcileFinalBlockLogs`): terminal SSE events now carry `finalBlockLogs` (server-authoritative snapshot). On execution:error / execution:cancelled, reconcile any still-running entries with their server-side terminal state. Recovers correctness on network drop, server timeout/abort, and reconnect-resume paths where individual block:* events may not have reached the client. - **Fix D** (`addExecutionErrorConsoleEntry`): cross-check `useTerminalConsoleStore` for entries with `error` set scoped to the executionId before emitting the synthetic "Run Error" row. Suppresses the phantom row when the failing block already carries the message. - **Signature refactor**: `handleExecutionErrorConsole` / `handleExecutionCancelledConsole` now take a typed `ExecutionConsoleDeps` object instead of stacking positional deps — matches the existing `createBlockEventHandlers(config, deps)` precedent in the same file. Tests: - 12 tests in `workflow-execution-utils.test.ts` covering Fix B/C/D and the deps-object signature refactor. - Centralized terminal-console store mock in `@sim/testing` so future tests can stub the store without per-file boilerplate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(terminal): wire copilot cancellation to finalBlockLogs reconciliation Address Greptile review: - `executeWorkflowWithFullLogging`'s `onExecutionCancelled` was `() => {}` and silently dropped the `finalBlockLogs` payload, so Bug 2's "upstream blocks stuck on Running" fix did not fire on copilot-initiated cancellations. Wire it through `handleExecutionCancelledConsole` to match the SSE-route `onExecutionCancelled` path. - Test for the `blockType !== 'error'` filter used a different `executionId` than the seeded entry, so the executionId scope rejected the entry before the blockType predicate ran. Align executionIds so the test actually exercises the filter. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(terminal): pass durationMs in reconnect cancellation handler Reconnect-resume `onExecutionCancelled` was forwarding `finalBlockLogs` but not `data?.duration`, so the "Run Cancelled" entry rendered with a 0ms duration. Match the other two `handleExecutionCancelledConsole` callsites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Waleed
|
0c25fc4ee1 |
fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies (#4369)
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies
- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml
Fixes simstudioai/sim#1243
* fix(auth): address PR review comments
* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)
* fix(docker): include @sim/utils in migrations image
Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.
* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections
The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.
* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema
The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.
* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional
The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:
- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
http://localhost:3002 to empty for consistency with prod.yml; the
in-code localhost fallback handles the dev case identically
Applied across all 6 documentation languages (en/fr/de/ja/es/zh).
* chore: untrack and ignore .claude/scheduled_tasks.lock
|
||
|
Vikhyath Mondreti
|
b8959eb20d |
improvement(repo): zod based client-server boundary (#4355)
* improvement(repo): centralized zod contracts (#4336) * improvement(repo): zod schema contracts * type checks * fix(notion): correctly register tool (#4337) * fix func blokc * more improvements * fix tests * type check * remove v3 refs * minor type improvements * address comments * update jira contract * remove validateJsonBody * improvement(repo): consolidation of boundary helpers + better unknown usage (#4352) * improvement(repo): consolidation of boundary helpers + better unknown usage * address comments * improve file transfer error messaging * fix docs listing schema drift * fix inocrrect type casting * address council comments * remove prefix |
||
|
Waleed
|
36742740aa |
fix(cleanup): batch orphaned snapshot deletes to avoid slow-query spike (#4348)
* fix(cleanup): batch orphaned snapshot deletes to avoid slow-query spike * fix(cleanup): recheck orphan condition in delete to close TOCTOU gap |
||
|
Vikhyath Mondreti
|
2e3de9ac8a |
feat(governance): external workspace users from outside org (#4313)
* feat(governance): external workspace users from outside org * update docs * address comments * edge case improvements * remove unused fallback * address comments * add outbox for seat reduction * fix edge case with org join after invite * add server side batch invites for workspace * use zod schema for route |
||
|
Theodore Li
|
04f1d015f3 | fix(mothership): Use heartbeat mechanism for chat locks (#4286) | ||
|
|
5f0f0edd63 |
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |