mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-24 01:23:02 +08:00
c95aa879f7f08de05c3813c17bade3ea3bf8bf20
245
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 WaleedandClaude Opus 4.8
|
fd77bb4069 |
feat(copilot): add seq ordinal to copilot_messages for order-preserving reads (#4791)
copilot_messages had no column preserving message order: created_at (set from each message's timestamp) ties at millisecond granularity in 58% of chats, and some chats have out-of-order timestamps within their array. The only other tiebreaker, id, is a random UUID — so ORDER BY created_at, id renders same-timestamp user/assistant pairs swapped. This blocks the R+1 read cutover. Add an integer seq = the message's 0-based index within the chat's JSONB array (ground-truth order), backfilled inline in migration 0219 (no script for self-hosters or us). Reads will use ORDER BY seq NULLS LAST, created_at, id at cutover; reads still come from JSONB after this PR. Design: - seq is a tiebreaker, not the sole sort key (concurrent-append/NULL safety). - Nullable now; defer NOT NULL so rolling-deploy old pods don't fail inserts. - replace (update-messages snapshot) overwrites seq = array index (re-densifies after a mid-conversation delete); append preserves existing seq via COALESCE and assigns base+idx from a single MAX(seq) read (never MAX+i in SQL — multi-row batches would collide). The non-atomic read-then-insert window is documented and bounded by the read tiebreak + snapshot re-densify. - Dedupe message ids before insert (87 prod chats carry dup ids; a repeated id in one INSERT...ON CONFLICT would otherwise throw). - Backfill picks first-occurrence per (chat,id), gap-free via ROW_NUMBER; validated on staging data (0-based, contiguous, 0 bad ranges). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Vikhyath MondretiandCursor
|
53eaa600af |
improvement(billing): migrate hot path writes away from user_stats (#4768)
* improvement(billing): migrate hot path writes away from user_stats * fix period start and end sot * address comments * Remove stale billing migration Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
WaleedandClaude Opus 4.7
|
ddc47eb221 |
feat(copilot): add copilot_messages table with dual-write rollout (#4726)
Splits copilot chat messages out of the copilot_chats.messages JSONB column into a dedicated copilot_messages table. JSONB stays canonical during R+0 — every write path dual-writes to the new table best-effort (try/catch + log warn, never throws). Migration 0217 creates the table + indexes and inline-backfills history from JSONB so OSS self-hosters don't need to run a separate script. Write paths covered: - post.ts (user message append) - terminal-state.ts (assistant turn finalize) - update-messages/route.ts (snapshot replace) - inbox/executor.ts (background turn) - fork/route.ts (chat clone) - superuser/import-workflow/route.ts (chat import) Each call threads chatModel + streamId where relevant; ON CONFLICT DO UPDATE preserves existing stream_id / model via COALESCE. For pre-R+1 reconciliation, run: bun apps/sim/scripts/copilot-messages-reconcile.ts [--since='7 days'] Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Waleed
|
34c47f7e93 |
chore(auth): upgrade better-auth 1.3.12 → 1.6.11 (#4766)
* chore(auth): upgrade better-auth 1.3.12 → 1.6.11 * chore(auth): address Greptile review — broaden change-email type + migration newline * fix(auth): correct oneTimeToken expiresIn unit (minutes, not seconds) Better-auth's oneTimeToken expiresIn is in minutes (multiplied by 60_000ms internally). Sim's existing 24*60*60 evaluated to ~60 days of token lifetime instead of the intended 24 hours. Tokens are one-time-use and typically consumed within seconds of generation (Socket.IO handshake), so this tightens an unused security window without affecting UX. |
||
|
Vikhyath Mondreti
|
59792c052f |
improvement(mcp): bound MCP memory and lifecycle concurrency (#4751)
* improvement(mcp): bound MCP memory and lifecycle concurrency * update db mock * address comments * address comments |
||
|
Vikhyath MondretiandCursor
|
81bcdf255a |
improvement(schedules): retries, concurrency limits (#4755)
* improvement(schedu les): retries, concurrency limits * improvement(schedules): remove stale generated migration Drop the pre-merge generated 0213 migration so it can be regenerated after syncing with staging. Co-authored-by: Cursor <cursoragent@cursor.com> * address comments * fix pinned version for lru cache * retryable errs cleanup --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Theodore LiandClaude Opus 4.7
|
92fd17cd15 |
fix(tables): workflow-column run fixes + bounded run-N-rows (#4754)
* fix(db): disable statement_timeout for migrations
* fix(ci): route migration workflow through guarded migrate.ts
* feat(tables): workflow-column run fixes + bounded "run N rows"
- Pass group.autoRun as the add-group dispatch flag so an autoRun=false
column no longer opens a no-op dispatch that flashes the run-count badge.
- Scope the context-menu re-run to the right-clicked workflow cell's group
(cascading to dependents) instead of every group on the row.
- Add an extensible per-dispatch row cap (DispatchLimit { type:'rows', max })
surfaced as "Run 10 / 1,000 empty rows" in the group header; dispatcher
stops after N eligible rows. New limit/processed_count columns on
table_run_dispatches.
- Fix stranded "Queued" cells: the cascade owner now treats a queued marker
(orphan pre-stamp) as a manual run so autoRun=false requested groups are
picked up, and drains late markers before releasing the row lock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(db): regenerate dispatch limit migration on staging chain (0214)
Re-numbers the table_run_dispatches limit/processed_count columns from the
collided 0212 to 0214 after merging staging (which added its own 0212/0213).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tables): lint formatting
* fix(tables): address PR review on dispatch cap + cascade drain
- Don't consume the row cap when batchEnqueueAndWait fails; a transient
failure no longer completes a capped dispatch with zero rows started.
- Outer cascade-drain loop only re-drives a genuine queued marker, not any
eligible group, so an empty-output group can't re-run forever.
- completeDispatch forwards limit on the terminal SSE event.
- Extract shared LIMITED_RUN_PRESETS for the Run-N-rows menu items.
* chore(lint): format generated tool-schemas-v1
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Theodore LiandClaude Opus 4.7
|
65e2fe81c1 |
feat(tables): Add enrichment table column type (#4752)
* feat(tables): native enrichments sidebar + workflow input mapping Add a Clay-style enrichments catalog to the table view and wire per-row input mapping into workflow-backed columns. - New "Enrichments" entry in the New-column dropdown opens a sliding panel listing curated enrichment templates; picking one swaps to the workflow config in-place (no cross-slide) with a back button. - Type the workflow sidebar as manual | enrichment; enrichment hides the launch + add-column-inputs affordances. - Add a "Workflow inputs" advanced panel mapping Start-block input fields to table columns (left-of-workflow columns only), with name-match auto-fill and collapsible input-mapping-style rows. - Persist type + inputMappings on the workflow group (types, contract, route, service, hook) — jsonb, no migration. - Consume inputMappings at run time: when present, feed Start-block fields from the mapped columns; otherwise fall back to name-match spread. - Clean up inputMappings on column rename/delete (stripGroupDeps + renameColumn). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(emcn): extract CollapsibleCard and reuse for input mapping Pull the collapsible field-card markup (surface-4 header + surface-2 body, click/keyboard toggle, truncated title + optional badge) into a shared `CollapsibleCard` emcn component, and use it in the workflow-builder input mapping rows and the table sidebar's input-mapping panel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(tables): code-defined enrichment registry run directly per row Enrichments are now TS configs in apps/sim/enrichments/ (registry, like connectors) that run directly per table row via the existing run/dispatch/ cell-write rails — no workflow execution. - enrichments/{types,registry} + work-email (heuristic) and phone-number (stub). - WorkflowGroup gains enrichmentId; WorkflowGroupOutput gains outputId (workflowId/blockId/path kept required, '' for enrichment groups). - Executor branches on group.type === 'enrichment' → maps inputMappings → enrich() → outputs by outputId → cell-write. Missing required inputs skip (blank cell) instead of erroring. - Sidebar lists the registry; enrichment-config panel maps inputs to columns and creates the enrichment group (no workflow UI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(enrichments): provider fallback cascade + hosted-key usage source Replace each enrichment's single enrich() with an ordered providers[] fallback cascade. Providers are plain data ({ id, label, toolId, buildParams, mapOutput }) so the catalog stays client-safe; the server-only runner (run.ts) calls executeTool per provider, first non-empty result wins, misses/errors fall through, all-miss = blank cell. Wire four enrichments on the hosted-safe providers (Hunter, PDL): - Work Email (fullName, companyDomain): Hunter -> PDL - Phone Number (fullName, companyDomain): PDL - Company Domain (companyName): PDL - Company Info (domain): PDL -> Hunter Person enrichments take a single canonical fullName (Clay-style); Hunter gets first/last via splitName(), PDL takes name directly. Add 'enrichment' to usage_log_source enum (+ migration) so hosted-key tool cost from these per-row calls can be billed to the table owner. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(enrichments): bill hosted-key cost; surface provider errors; abort safety - runEnrichment now returns { result, cost, error }: accumulates hosted-key cost across the cascade, and sets `error` only when every provider that ran errored (auth/rate-limit/outage) vs a clean miss. - Executor records the cost to the table owner (createdBy) via recordUsage (source 'enrichment'); billing failures are logged, never error the cell. - F1: all-providers-errored now writes status 'error' instead of a blank 'completed' cell that looked like "no data found". - F2: re-check the abort signal after the cascade so a cancel mid-tool-call isn't recorded as a completed empty cell. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(tables): present enrichment columns as first-class in the grid - Meta-header shows the enrichment's name + icon (Mail/Phone/Globe/Building2) instead of "Workflow" + a color chip. - Per-column header icon uses the enrichment's icon (via columnSourceInfo) instead of the generic play icon. - Hide "View execution" for enrichment cells in both the row context menu and the action bar (no workflow execution exists to open); also hide the meta-menu "View workflow" item for enrichment groups. - Clicking an enrichment column header now opens the enrichments sidebar in edit mode (pre-filled input mappings, Update via useUpdateWorkflowGroup) instead of the workflow "Configure workflow" sidebar. - Enrichment config lets the user name each output column (editable per-output, deduped defaults) since enrichments can produce multiple columns. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): enrichment columns use type icon; output names editable - Drop the per-column enrichment icon (it duplicated the meta-header icon). Enrichment output columns now render the standard column-type icon (Text, etc.) — the enrichment's icon stays only on the group meta-header. - Make output column names editable in the enrichment config edit mode too; changed names rename their columns via useUpdateColumn (the rename cascades into the group's output refs server-side). Validation excludes the output's own current name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): wrap enrichment catalog descriptions instead of truncating Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tables): edit enrichment output columns via the plain column editor Edit column on an enrichment output now opens the normal column-config sidebar (rename / type / unique) instead of the workflow 'Configure output column' panel, which showed workflow-only fields and blocked a simple rename. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(copilot): list_enrichments + add_enrichment table tools Let the copilot enumerate the code-defined enrichment registry and add an enrichment column to a table (validating required input mappings against the table's columns), backed by the same workflow-group machinery the UI uses. * fix(enrichments): address PR review feedback - Guard the enrichment cell path on `enrichmentId` so a group typed 'enrichment' without a registry id falls through to the workflow path instead of erroring. - Clear stale output values when skipping a row for missing required inputs, so the auto cascade re-enriches once inputs return (was left completed+filled). - Write a terminal state on abort in the enrichment path (matches the workflow path) so a cancel between run and terminal-write can't leave the cell running. - Edit mode: apply the group update (mappings/deps/auto-run) before column renames so the primary edit lands even if a rename fails. - Disable Save once validation has surfaced a missing required input. - Use the workflowGroupById map instead of O(n) find in the context-menu and action-bar hot paths. * chore(commands): add /add-enrichment command Guides adding a code-defined table enrichment to the registry, with a required step to verify each provider tool has hosted-key support and chain to /add-hosted-key when it doesn't. * fix(enrichments): address second-pass PR review - updateWorkflowGroup output diff now keys on outputId (falling back to blockId::path) so enrichment outputs — which share empty blockId/path — no longer collapse to one key and drop sibling columns. - Enrichment terminal write now clears output columns absent from the result, so a partial/empty re-run doesn't leave stale values. - Editing a group whose enrichment was removed from the registry shows an explanatory panel instead of silently falling through to the new-enrichment catalog. * feat(tables): show "Not found" badge for empty completed enrichment cells An enrichment that runs to completion but matches nothing now renders a gray "Not found" badge (like the Queued/Waiting cell states) instead of a blank cell, so a real miss is distinguishable from an unrun cell. Scoped to enrichment output columns; an empty string no longer counts as a value. * fix(enrichments): don't re-run completed no-match enrichments on auto cascade A completed enrichment with empty outputs is a real no-match result, not an unfinished run. Eligibility now treats an enrichment's completed status as terminal (regardless of output fill), so the auto cascade stops re-invoking billable provider calls on every no-match row each dispatch. Input changes still clear the exec entry, so genuine re-runs are unaffected; manual Run all still re-runs. * fix(enrichments): treat provider 404 as no-match, not a cell error Providers like People Data Labs signal 'no record found' with HTTP 404, which executeTool surfaces as a failed ToolResponse (status on output.status). The cascade now treats a 404 as a clean miss — falls through to the next provider and lets the cell render 'Not found' — instead of marking the cell errored. Auth/rate-limit/5xx still propagate as real errors. * fix(tools): surface HTTP status on error ToolResponse output executeTool's catch handled Error instances in its first branch and only extracted status/statusText/data for non-Error object throws — so HTTP errors (thrown as Error instances carrying .status) lost their status on the returned output. Surface it for Error instances too, so callers can branch on the status (e.g. the enrichment cascade treating a provider 404 as a no-match). * fix lint * Revert ff --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Vikhyath Mondreti
|
209ca5f121 |
fix(large-refs): cleanup based on table read (#4716)
* fix(large-refs): cleanup based on table read * address comments * address comments * bubble up storage ref errors * cleanup code * do not attempt blob deletion for infra outage * cleanup dup helper |
||
|
Theodore Li
|
786c6f0607 |
fix(db): disable statement_timeout for migrations (#4714)
* fix(db): disable statement_timeout for migrations * fix(ci): route migration workflow through guarded migrate.ts |
||
|
Waleed
|
3f7698c66b |
perf(db): reduce read/write fanout across hot paths (#4704)
* perf(db): reduce read/write fanout across hot paths * fix(templates): include name/workflowId/status/tags in DELETE projection * fix(db): address review feedback (joinedAt backfill, mcp delete error, chatDeploy projection, sql newline) * fix(webhooks): include NULL failedCount in markWebhookSuccess guard |
||
|
WaleedandClaude Opus 4.7
|
46db40620f |
feat(mcp): OAuth 2.1 + PKCE for outbound MCP servers (#4441)
* feat(mcp): OAuth 2.1 support for outbound MCP servers
* fix(mcp): tighten OAuth refresh race and session-error detection
Re-load the OAuth row inside withMcpOauthRefreshLock so concurrent
callers observe predecessor-written tokens instead of a stale snapshot
loaded before lock acquisition. Without this, the second caller's
provider held a rotated-out refresh token and the SDK tripped
invalid_grant, forcing reauthorization.
Switch isSessionError to match the SDK's typed StreamableHTTPError
(code 404/400) instead of substring-checking arbitrary error messages,
removing false positives on URLs that happen to contain those digits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(mcp): tighten OAuth callback contract and registration metadata
- Validate callback query params via mcpOauthCallbackContract instead of
raw searchParams.get, matching the rest of the MCP route surface.
- Drop non-RFC-7591 application_type field from dynamic client registration
to avoid rejection by strict authorization servers.
- Collapse the pre-lock OAuth row load in createClient — the row is now
loaded exclusively inside withMcpOauthRefreshLock, removing a redundant
query and a stale-snapshot path.
* fix(mcp): narrow workspaceId before async closure in OAuth createClient
* fix(mcp): return authType from create-server endpoint
The POST /api/mcp/servers handler omitted authType from the success
response, so useCreateMcpServer always saw data.data.authType as
undefined and never triggered the OAuth popup after creating an
OAuth-protected server. Thread authType through performCreateMcpServer
into the response so the client can decide whether to auto-start OAuth.
* fix(mcp): mirror server null normalization in optimistic oauthClientId update
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): revert optimistic oauthClientId to undefined to match McpServer type
The response contract preprocesses null → undefined, so McpServer.oauthClientId
is string | undefined. Using null broke type checking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): tighten OAuth probe signal and clear stale popup interval
- probe: only classify as OAuth on resource_metadata or scope params.
Bare `Bearer error="invalid_token"` is generic and used by API-key servers,
so it must not auto-flip the auth type to OAuth.
- popup hook: clear any existing close-watcher interval before overwriting
when startOauthForServer is invoked twice for the same serverId.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): normalize empty-string oauthClientId at route boundary
Orchestration already converts falsy → null via `|| null` (server-lifecycle.ts),
so the DB was never receiving an empty string. Tightening the route layer to
match the same convention keeps the boundary contract consistent and avoids
relying on downstream normalization.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(canvas): expand MCP tool params into per-row labels on block tile
The MCP Tool block on the workflow canvas previously crammed every selected-
tool parameter into a stringified blob under the `Tool` row. Now, when a tool
is selected, the tile reads the cached `_toolSchema` and emits one labeled
SubBlockRow per parameter (matching the Exa block's per-param layout). Labels
reuse `formatParameterLabel` for parity with the editor panel; values pass
through the existing `getDisplayValue` so booleans/numbers/arrays render
identically to other blocks. Deterministic tile height counts expanded rows
so the tile sizes correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(logs): show MCP icon and strip prefix in trace tool spans
Tool spans for MCP calls were rendering the raw id (e.g.
`mcp-f908f259-planetscale_list_organizations`) with the default blank-
square icon. Now they read just the tool name and render the MCP block's
icon and bgColor, matching how workflow-execute tools render.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(logs): lift near-black trace icon backgrounds for dark-mode contrast
Block bgColors below a small luminance threshold (e.g. the MCP block's
#181C1E) rendered nearly invisible against the dark-mode surface
(--bg: #1b1b1b). Adds a tiny adjustBgForContrast helper that floors each
RGB channel at 0x33 only when luminance is below 30,000, leaving every
branded color above that band untouched. Applied to both the trace tree
row and the detail pane.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(logs): fall back to neutral gray for near-black trace icon bgs
#333333 was still too close to the dark-mode surface to read. For bgs
below the luminance threshold (e.g. the MCP block's #181C1E) we now fall
back to DEFAULT_BLOCK_COLOR (#6b7280) — the same neutral the renderer
uses for blocks with no distinct identity. Clearly visible in both
themes; brighter brand colors still pass through.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(db): drop 0209_mcp_oauth migration ahead of staging merge
Staging shipped 0209_smiling_fixer; the MCP OAuth migration will be
regenerated on top of staging as 0210.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(db): regenerate MCP OAuth migration as 0210
Re-runs drizzle-kit generate on top of staging's 0209_smiling_fixer.
Same schema (mcp_server_oauth table + mcp_servers.auth_type / oauth_*
columns) as the dropped 0209_mcp_oauth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(audit): bump route baseline 748 → 749 after staging merge
The post-merge route count is 749 (this branch's OAuth start/callback
plus staging's new route). I had set the baseline to 748 in the merge
conflict resolution — bumping to match reality so the strict audit
passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: remove source-command skill files committed by accident
These were untracked-then-accidentally-staged in
|
||
|
Theodore LiandClaude Opus 4.7
|
f0311a6f5e |
feat(table): chunked dispatcher + workflow cascade (#4672)
* feat(table): chunked dispatcher for workflow-column runs
Replaces the all-rows-at-once runWorkflowColumn with a row-window dispatcher
backed by a new table_run_dispatches row. Each user click inserts a dispatch
row and triggers a trigger.dev task that crawls the table 20 rows at a time,
re-enqueueing itself between windows. The HTTP/Mothership entrypoints return
{ dispatchId } immediately instead of holding the request open for minutes
on multi-thousand-row dispatches.
- Per-row cancel stamps cancelledAt; the dispatcher skips cells whose
cancelledAt > dispatch.requestedAt so a mid-cascade cancel sticks even
under isManualRun.
- Table-wide cancel marks active dispatches cancelled atomically so the
dispatcher bails on its next iteration.
- New 'dispatch' SSE event variant plumbed; client ignores for v1.
* fix(table): eager bulk clear on column run so cells flip immediately
Run-column with run-mode 'all' wasn't visually flipping rows that already
had data — the cell renderer's "value wins" branch kept showing the prior
output behind the queued/running state. The dispatcher only cleared one
window of rows at a time, so most of the column stayed stale until the
cursor walked to it.
Now:
- Dispatcher's `pending → dispatching` transition runs a single SQL UPDATE
that wipes targeted `data` output columns and `executions[gid]` across
every targeted row (mode-aware: 'incomplete' skips fully-filled rows).
- Per-window clear in `dispatcherStep` is gone — rows are pre-cleared,
the loop only filters cancel tombstones / unmet deps and enqueues.
- Optimistic patch in `useRunColumn` mirrors the bulk clear by nulling
output values in the cached row, so the UI flips queued/running
instantly without waiting for the SSE catch-up.
* fix(table): bulk clear honors in-flight execs under mode: 'incomplete'
The eager bulk clear for mode: 'incomplete' only skipped rows that were
already fully filled, so two overlapping dispatches could race — dispatch B
would nuke executions[gid] on a row dispatch A had just stamped 'queued',
flickering the cell and potentially confusing the worker.
Skip any row whose targeted group is currently queued/running/pending — an
'incomplete' run shouldn't touch what another dispatch is actively working
on. The per-walk 'in-flight' eligibility skip already handles rows that
flip in-flight between the clear and the cursor reaching them.
* refactor(table): dispatcher uses batchTriggerAndWait + tag-based cancel
Switch the per-window cell fan-out from fire-and-forget tasks.trigger to
tasks.batchTriggerAndWait. The dispatcher is now a single long-lived
trigger.dev task that loops dispatcherStep until the table is exhausted;
trigger.dev CRIU-checkpoints the parent during each wait so we don't pay
compute while cells execute. Queue depth is bounded at WINDOW_SIZE per
dispatch — no more flooding trigger.dev with a million queued runs.
- dispatcher.ts builds payloads via the new shared buildPendingRuns helper
and calls tasks.batchTriggerAndWait directly. Pre-stamps each cell to
`queued` (jobId=null) so the UI flips instantly.
- table-run-dispatcher.ts is now a plain while-true loop. No
RUN_BUDGET_MS, no self-re-enqueue, no cold-start tax per window.
Cancel:
- New cancelCellRunsByTags(tags) paginates runs.list + runs.cancel(id).
- cancelWorkflowGroupRuns fires the tag-sweep alongside the per-jobId
queue.cancelJob path (preserved for auto-fire cells that have real
jobIds from single tasks.trigger calls).
- Trigger.dev acks the cancel → batchTriggerAndWait resumes → dispatcher
observes the dispatch-row cancel flag → exits.
Side fixes:
- getAsyncBackendType returns 'trigger-dev' whenever taskContext.isInsideTask
is true, regardless of TRIGGER_DEV_ENABLED env. The preview/dev-sim
worker silently routing cell jobs to DatabaseJobQueue (no poller) is
fixed without any env config change.
- runWorkflowColumn skips the dispatcher entirely when trigger.dev is
disabled, running cells inline via DatabaseJobQueue.runInline. HTTP
response returns dispatchId: null in that mode.
- runColumnContract response schema updated to dispatchId.nullable().
* fix(table): show Stop button on optimistic-pending row cells
isExecInFlight required a jobId for `pending` status, gating it as "real
backend pending" vs "optimistic flag only." The row-gutter Stop button
keyed on this — so a freshly clicked Play sat as `pending` (no jobId) and
the user couldn't cancel it until the server-side `queued` stamp arrived
via SSE. With the dispatcher pre-batch stamping cells as `queued` (not
`pending`) and no per-cell jobIds under batchTriggerAndWait, the gap was
worse.
Drop the jobId requirement. `pending` now counts as in-flight everywhere.
Cancel writes `cancelled` to the cell exec authoritatively whether or not
a real trigger.dev run exists yet — cancelling an optimistic cell means
"don't run this," which is correct.
Also collapse isOptimisticInFlight into isExecInFlight since the two
helpers are now identical.
* refactor(table): loop-in-cell cascade + dispatcher-everywhere routing
Two coupled changes:
1. Cell-task runs the row's full cascade in-process. executeWorkflowGroupCellJob
acquires a Redis lock per (tableId, rowId) with heartbeat (10s/30s TTL),
then loops through eligible workflow groups for the row. One cell-task =
one row's full cascade, not N. Resume worker holds the same lock and
continues the cascade after a HITL resume. Shared withCascadeLock helper
in lib/table/cascade-lock.ts.
2. Every cell-enqueue goes through the dispatcher. The implicit
scheduleRunsForRows reactor in service.ts is removed — 8 callsites
(insertRow, batchInsertRows, upsertRow, updateRowsByFilter,
batchUpdateRows, addWorkflowGroup, updateWorkflowGroup) now fire
runWorkflowColumn with mode: 'incomplete', isManualRun: false. HTTP
routes that call updateRow directly also fire runWorkflowColumn
afterwards. scheduleRunsForTable / scheduleRunsForRowIds deleted;
scheduleRunsForRows demoted to private (only the TRIGGER_DEV_ENABLED=false
fallback uses it). skipScheduler flag dropped from UpdateRowData /
BatchUpdateByIdData — no longer meaningful since there's nothing implicit
to suppress.
Plumbed isManualRun through the dispatch row (new is_manual_run column,
default true) so auto-fire callers honor autoRun: false and don't re-run
completed cells.
Stamp 'pending' (not 'queued', executionId: null) before
batchTriggerAndWait — cell-task writes its own 'queued' on lock acquire.
Small UI polish: row gutter Play button spacing, "Delete workflow" →
"Delete column" label, optimistic-pending cells now show Stop button
(isExecInFlight no longer requires jobId).
* fix(table): SQL cancellation guard allows worker to claim a null-execId cell
The dispatcher's pre-batch `pending` stamp leaves executionId unset so any
cell-task that wins the cascade lock can claim the cell. The cancellation-
guard SQL clause was rejecting these claims because it tested
`executions->gid IS NULL` (whole exec missing) but the pre-stamp leaves
the exec present with executionId=null.
Add a third carve-out: `executions->gid->>'executionId' IS NULL`. Now the
guard reads "write allowed if no exec exists, OR no executionId is set
yet, OR the executionId matches ours."
Symptom: every cell-task's first markWorkflowGroupPickedUp call would log
"SQL guard saw cancelled" and skip, leaving cells stuck at the dispatcher's
pending stamp.
* fix(table): dispatcher cursor starts at -1 so position 0 is included
The dispatcher's row-window SELECT is `position > cursor` for exclusive
lower-bound semantics. With cursor initialized to 0, position-0 rows were
never picked up — every dispatch silently skipped the table's first row.
Start cursor at -1 instead. First window's filter `position > -1` matches
position 0; subsequent iterations advance to `lastPosition` which then
correctly excludes already-processed rows.
* refactor(table): align optimistic UI with new dispatcher; sticky cancel via 'new' mode
Fix 0: new `DispatchMode = 'new'` for auto-fire callsites. Eligibility skips
rows with any prior `executions[gid]` entry — cancelled / errored / completed
cells stay sticky until a manual run. Dispatcher's windowed SELECT pushes
`NOT jsonb_exists_any(...)` to SQL so CSV imports into mostly-attempted
tables don't pay a per-window load+JS-filter. `batchInsertRows` drops its
`rowIds` payload (keeps dispatch scope tiny on big imports).
Fix A/B/D: client optimistic patches now mirror the backend's actual
invariants. `useCreateTableRow.onSuccess` stamps eligible groups via
`optimisticallyScheduleNewlyEligibleGroups` so newly-inserted rows show
`Queued` instantly. `useCancelTableRuns.onMutate` distinguishes optimistic-
only pending (`executionId == null` — strip silently) from real worker
claims (stamp cancelled; SSE will reconcile). Drop `onSettled` invalidation
on `useUpdateTableRow` / `useBatchUpdateTableRows` to kill the
delete-cell flicker.
Fix C: active-dispatches overlay. New `listActiveDispatches` helper,
contract, and `GET /api/table/[tableId]/dispatches` route. `kind:'dispatch'`
SSE events carry scope+cursor+mode on every transition. New
`useActiveDispatches` hook + `resolveCellExec` synthesize a virtual
`pending` exec for cells in an active dispatch's scope ahead of cursor —
queued indicators now survive page refresh during long Run-all dispatches.
`cancelWorkflowGroupRuns` emits `kind:'dispatch',status:'cancelled'`
events so the overlay clears without a refetch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(table): unify trigger.dev and inline dispatcher paths
`runWorkflowColumn` now always inserts a `table_run_dispatches` row and
drives the dispatcher state machine. The trigger.dev / in-process branch
narrows to a single line: trigger.dev fires `tableRunDispatcherTask` (which
calls the new `runDispatcherToCompletion`), the inline path calls the same
helper fire-and-forget. Deletes `scheduleRunsForRows` and
`stampQueuedOrCancel` — the inline-fallback no longer duplicates window
walking, SSE emission, or cancel.
The dispatcher's window-execute call goes through `JobQueueBackend`:
- New `batchEnqueueAndWait` interface method.
- Trigger.dev impl wraps `tasks.batchTriggerAndWait` behind a
`taskContext.isInsideTask` guard (clear error if called from outside a
task).
- Database impl skips `async_jobs` entirely — `Promise.all` over
`options.runner(payload, signal)` per item, with per-cell AbortControllers
tracked by `cancelKey` for cancel.
`cancelInlineRun` moves to the interface as `cancelByKey` so
`cancelWorkflowGroupRuns` no longer reaches into the database backend.
Fix `mode: 'new'` SQL filter:
- `${array}::text[]` interpolated as a tuple-cast which Postgres rejected
("cannot cast type record to text[]") and every inline dispatch silently
failed. Switched to `ARRAY[${sql.join(...)}]::text[]`.
- Predicate was `jsonb_exists_any` ("any one targeted group present"),
which excluded rows that needed at least one group re-run after a
downstream output was deleted. Switched to `jsonb_exists_all` — per-group
JS eligibility handles the rest.
Cascade-loop workflowId bug: `runRowCascadeLoop` was not threading the new
group's `workflowId` when advancing across groups. The cell-task ran the
previous group's workflow against the next group's cell, terminating
`completed` with empty `accumulatedData`. Fixed by tracking
`currentWorkflowId` alongside `currentGroupId` / `currentExecutionId`.
Client optimistic-patch tightening:
- `useRunColumn.onMutate` mirrors server eligibility — skip cells with
unmet deps so unmet rows don't flash Queued and get stuck (no SSE will
arrive for cells the server skipped).
- `resolveCellExec` overlay synthesizes a virtual `pending` only when
`areGroupDepsSatisfied` is true. Rows with unmet deps render Waiting,
matching the dispatcher's actual behavior.
Cleanup from /simplify pass:
- Use `generateShortId(20)` instead of
`generateId().replace(/-/g, '').slice(0, 20)`.
- Inline `batchEnqueueAndWait` no longer allocates synthetic ids
(returned `string[]` is unused).
- Flattened the per-cell `tracked` array — only push entries that
registered controllers, drop the null placeholders.
- Extracted `runDispatcherToCompletion` to share the loop between the
trigger.dev wrapper and the in-process path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(table): backend running counter, dep-aware retrigger, sidebar polish
Counter (Fix 1): top-right "X running" + per-row badge are now
backend-bootstrapped via a count on `user_table_rows.executions ->> 'status'
= 'running'` returned alongside active dispatches. SSE `kind: 'cell'` events
compute a delta from `prev → next` status to keep the cache live; cell
events for rows outside the loaded page slice trigger a run-state refetch.
On `pruned` we invalidate the cache. Counts only worker-claimed `running`
cells — optimistic queued/pending no longer inflate the badge, and rows
outside the loaded page slice are counted too.
Sidebar (Fix 2 + 3a): `Run after` no longer ticks every column by default
for new groups (empty list). Save is disabled with an inline error when
auto-run is on with zero deps. `edit-group` mode anchors the left-of-current
filter to the group's leftmost column, so a workflow can only depend on
columns to its left.
Reorder scrub (Fix 3b): `updateTableMetadata` walks the schema's workflow
groups when `columnOrder` is in the patch and drops any dep whose new
position lands at or after the group's leftmost column (uses the existing
`stripGroupDeps` helper). Metadata + schema updates land atomically.
Server returns ordered columns (Fix 3b cont'd): `getTableById` /
`listTables` now sort `schema.columns` by `metadata.columnOrder` before
returning, via a new `applyColumnOrderToSchema` helper. Every consumer
(grid, sidebar, copilot, mothership) gets one ordered list — the sidebar's
leftmost-group-column anchor now points at the right index.
Dep-aware retrigger (Fix 4): editing a value that a downstream workflow
depends on now re-runs that workflow.
- `deriveExecClearsForDataPatch` returns
`{ executionsPatch, inFlightDownstreamGroups }`. Walks
`schema.workflowGroups[].dependencies.columns` for every column in the
patch, clears terminal-state downstream entries, and reports in-flight
entries.
- `updateRow` calls `cancelWorkflowGroupRuns` + `runWorkflowColumn`
(`mode: 'incomplete' + isManualRun: true`) for in-flight downstream
groups, then always fires `runWorkflowColumn({ mode: 'new' })` for the
cleared groups. Skips both when `executionsPatch` is provided by the
caller — those are cell-task / cancel writes that would otherwise spawn
a recursive flood of dispatches per partial-write.
- `cancelWorkflowGroupRuns(tableId, rowId, { groupIds? })` accepts a
per-group filter so the cancel only touches the affected groups, not
every in-flight cell on the row.
- `pickNextEligibleGroupForRow` now treats a dispatcher pre-stamp
(`pending` + `executionId: null`) as claimable — the cascade-loop is the
real owner. Without this, the dispatcher's pre-stamp of downstream
groups made the cascade-loop see them as "in-flight" and skip them,
stranding `pending` cells forever.
- `optimisticallyScheduleNewlyEligibleGroups` extends the cache patch to
flip dep-touched groups to `pending` regardless of their current status,
matching the server's cancel-then-rerun behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(table): paused workflow cells route through executeResumeJob; render Pending + viewable
Three connected issues with workflows that pause mid-cell (e.g. wait blocks):
1. `/api/resume/poll` (the time-pause auto-resumer) called
`PauseResumeManager.startResumeExecution` directly, bypassing
`executeResumeJob` from `background/resume-execution.ts`. The wrapper is
where the cell-context restoration + cascade-loop continuation lives —
without it, the resumed workflow ran to completion but never wrote the
terminal state back to the table cell. Cell stays `pending` forever
even though the underlying execution finished.
Fix: dynamically import `executeResumeJob` and use it for the
`'starting'` branch. Same primitive the trigger.dev `resumeExecutionTask`
wraps — calling it directly handles both trigger.dev-disabled local dev
and trigger.dev-enabled prod identically.
2. The cell renderer mapped `status: 'pending'` to `kind: 'queued'` (gray
"Queued" badge) regardless of whether the run had started. A HITL-paused
run has `status: 'pending'` + `jobId` prefixed `paused-` + a real
`executionId` — semantically very different from "queued, hasn't run."
Now renders as `pending-upstream` (the existing Pending pill) for
paused-jobId rows.
3. Right-click "View execution" was disabled for `pending` cells (gated to
`completed | error | running`), so users couldn't open the trace for a
paused execution. Paused runs have a viewable trace (the executionId is
real and the log row exists). Both the per-row context menu and the
action-bar derivation now recognize `pending` + `paused-` jobId as a
started run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(table): typewriter reveal for SSE-driven workflow cell values
Workflow-output cells now reveal their text character-by-character when an
SSE update lands, while page reloads and virtualization remounts still paint
the value instantly. A first-render guard inside the new useTypewriter hook
distinguishes hydration from live updates with no plumbing through the cell
tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(table): address bugbot/greptile review feedback
Two P1 issues + one cleanup from the bot reviewers:
1. **Double-dispatch + completed-output wipe.** Both PATCH row routes
(`app/api/table/[tableId]/rows/[rowId]` and
`app/api/v1/tables/[tableId]/rows/[rowId]`) were firing a second
`runWorkflowColumn({ mode: 'incomplete' })` after `updateRow` returns.
`updateRow` already fires `mode: 'new'` internally for user edits, so
the second call created a concurrent dispatch. Worse, the
`mode: 'incomplete'` path's `bulkClearWorkflowGroupCells` wipes ALL
targeted output columns on any row where any one column is empty —
meaning sibling-group completed outputs could be erased. Removed both
route-level calls; auto-dispatch lives entirely in `updateRow`.
2. **`runWorkflowColumn` log-spamming on plain tables.**
`if (targetGroups.length === 0) throw new Error(...)` fired on every
row insert/update for tables without any workflow groups (the
majority). Every caller wraps with `.catch(logger.error)`, so each
PATCH produced an error-level log. Return `{ dispatchId: null }`
silently — manual `runWorkflowColumn` callers pass `groupIds`
explicitly so they can't reach this branch.
3. **`isManualRun` plumbed through dispatch SSE events.** Late-arriving
`kind: 'dispatch'` events for dispatches not in the initial fetch
were hardcoding `isManualRun: false`. Added the field to the event
shape, emit it from `dispatcherStep` (pending → complete, dispatching
transitions) and `markActiveDispatchesCancelled`, and consume it in
the SSE handler with a sensible fallback for legacy emits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(table): row executions sidecar + left-to-right dep retrigger + cancel counter refresh
Split per-row workflow-group execution state out of the user_table_rows.executions
JSONB column into a new table_row_executions sidecar keyed by (row_id, group_id).
Dispatcher filters, "X running" counter, bulk clears, and the cancellation guard
all hit indexed columns instead of walking JSONB. Wire shape unchanged — server
merges sidecar rows back into row.executions on the way out.
Also:
- deriveExecClearsForDataPatch now walks workflowGroups left-to-right with a
propagating dirtied-column set so transitive dep chains (edit col A → group 1
re-runs → group 2 depends on group 1's output → group 2 re-runs) collapse to
a single forward pass.
- useCancelTableRuns.onSettled invalidates the activeDispatches query so the
top-right counter and row gutter Stop button refetch from the server after
any Stop (per-cell, row, or table-wide). countRunningCells is the source of
truth; client no longer needs duplicate state.
Three migrations on this branch (0209 + 0210 + new sidecar) collapsed into one
since the feature is unreleased.
* fix(table): address remaining cursor/greptile review feedback
- Mothership update_row no longer double-dispatches. updateRow already fires
the auto-cascade internally; the second `mode: 'incomplete'` call here
raced with it and could bulk-clear sibling-group outputs.
- SSE dispatch events no longer dropped when the activeDispatches cache is
cold. Seed an empty TableRunState if the initial fetch hasn't landed yet
so the queued overlay doesn't lose the first dispatch event.
- batchUpdateRows now runs cancel+rerun for per-row in-flight downstream
groups, mirroring updateRow. Without this, dep edits in a batch left
running workflows reading stale upstream values.
* fix(table): cancel prior runs, scope batch insert dispatch, recover orphan pre-stamps
Addresses cursor + greptile review feedback on table dispatcher edge cases:
- Manual table-wide Run-all / Run-column now cancels prior active dispatches
AND in-flight cell workers before bulk-clearing. Without this, mode:'all'
deleted running sidecar rows out from under their workers (which kept
writing into the wiped state) and a second Run-all could enqueue overlapping
cells racing on the same rows. Row-scoped manual calls (dep-edit cascade)
are excluded — those already cancel their own scope.
- batchInsertRowsWithTx now scopes its auto-dispatch to the newly-inserted
row ids. Without this, after the sidecar migration the NOT EXISTS filter
matches every existing row (zero sidecar entries), so a CSV import would
walk the entire table dispatching workflow runs on every pre-existing row.
- classifyEligibility carve-out: pending + executionId=null is an orphan
pre-stamp (cascade-lock contention, batchEnqueueAndWait failure, etc.),
treated as claimable so future dispatchers can re-stamp instead of skipping
it as 'in-flight' forever. Matches pickNextEligibleGroupForRow's logic.
- On batchEnqueueAndWait failure, dispatcherStep now sweeps the orphan
pre-stamps it wrote for the failed batch so the cells don't render Queued
forever; the next user action picks them up cleanly.
* fix(table): row-scoped Refresh cancels in-flight; counter includes queued/pending
- runWorkflowColumn now cancels prior in-flight cells for row-scoped manual
runs too (context-menu Refresh on a row subset, action-bar Refresh on
selected rows). Previously only the table-wide path cancelled, so a
row-scoped Refresh would bulk-clear running sidecar rows without aborting
workers. Per-row cancel skips markActiveDispatchesCancelled so unrelated
dispatches keep running.
- countRunningCells now counts all in-flight statuses (queued / running /
pending) instead of just running. The row gutter Run/Stop button reads
this map — with the old behavior, clicking Play during the queued window
would re-enqueue an already-queued cell. SSE applyCell handler updated
to use isExecInFlight so client deltas track the same semantics.
* fix(table): per-row Stop tombstones ahead-of-cursor rows during Run-all
Per-row Stop only cancelled sidecar rows already in flight. A row the
dispatcher hadn't reached yet had no exec record, so Stop was a no-op there
— the dispatcher would later walk to it, classify the group eligible, and
re-fire workflows the user thought they stopped.
cancelWorkflowGroupRuns now, for a per-row cancel, checks active dispatches
whose scope covers the row and writes `cancelled` tombstones (cancelledAt =
now) for the at-risk groups that don't already have a sidecar entry. The
dispatcher's existing `cancelledAt > dispatch.requestedAt` filter then skips
them when the cursor arrives. onConflictDoNothing guards against clobbering
a concurrently-written entry; the active-dispatch check avoids stamping
spurious cancels on idle rows.
* fix(table): seed dispatch overlay on Run; surface batch-enqueue failures as error
- useRunColumn.onSuccess invalidates the activeDispatches query so the
resolveCellExec queued overlay populates immediately for ahead-of-cursor
rows (scrolled-in / refetched), instead of waiting for the first dispatch
SSE. Targeted at activeDispatches only — the rows cache stays owned by
useTableEventStream.
- On batchEnqueueAndWait failure, dispatcherStep now flips the orphan
pre-stamps to a terminal `error` state and emits a cell SSE event, rather
than deleting them. The cursor still advances past the window, but the
dropped cells are now visible (Error pill) instead of silently empty, stay
out of the in-flight set, and re-run on the next manual run.
* fix(table): seed dispatch overlay on Run; surface batch-enqueue failures as error
- useRunColumn.onSuccess invalidates activeDispatches so the resolveCellExec
queued overlay populates immediately for ahead-of-cursor rows instead of
waiting for the first dispatch SSE. Rows cache stays owned by SSE.
- On batchEnqueueAndWait failure, dispatcherStep flips orphan pre-stamps to a
terminal error state (+ cell SSE) instead of deleting them, so the dropped
window is visible (Error pill) rather than silently empty and re-runs on the
next manual run.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Siddharth Ganesan
|
67e18f747d | fix(migrations): remove duplicate column add (#4632) | ||
|
Waleed
|
8d7bbbc670 |
chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting - Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files - Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors) - Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities - Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time - Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access - Add check:utils script to package.json * chore(utils): replace deepClone wrapper with structuredClone built-in deepClone() was a one-line wrapper around structuredClone(), which is universally available in Node 17+ and all modern browsers. Removing the abstraction reduces indirection and means contributors don't need to learn a project-specific name for a well-known built-in. - Remove deepClone from packages/utils/src/object.ts and index.ts - Replace all 17 call sites with structuredClone() directly - Update check:utils script suggestion text - Update CLAUDE.md and global.md docs * fix(utils): add missing biome noRestrictedImports rule and correct truncate docs - Add noRestrictedImports to biome.json under style — bans nanoid and uuid package imports at lint time (crypto.randomUUID/randomBytes are caught by the check:utils grep script which handles global property access) - Correct truncate() TSDoc and parameter name: sliceLength makes it clear that total output length is sliceLength + suffix.length, matching the behavior all callers were already written to expect * fix(utils): add missing getErrorMessage imports at 4 call sites The sweep agents added getErrorMessage calls without the corresponding import in 4 files, causing test failures. Added the missing imports. * fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue - Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for internal package imports; inline the jitter crypto call directly - Add missing getErrorMessage imports to 32 files where the sweep added calls without the corresponding import (caught by type-check and test runs) - Remove accidental getErrorMessage import from crowdstrike/query/route.ts which has its own domain-specific getErrorMessage for parsing CrowdStrike's JSON error format - Fix use-sub-block-value.ts type error from structuredClone narrowing: add 'as T' cast at emitValue callsite (safe — valueCopy is always a structural copy of newValue) * fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage The catch block was calling the local getErrorMessage function which parses CrowdStrike API JSON responses, not JavaScript Error objects. Use toError(error).message to correctly extract the message from a caught value in this context. |
||
|
|
c9118e775b |
feat(files): folders, multiselect, vfs update (#4572)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * feat(files): folders + vfs update * address comments * address comments * cleanup unnused code * address comments * perf improvements * address next set * cycle detect * error handling * path improvements * cleanup, best practices * react query best practices: targeted invalidation, optimistic updates, key factory hierarchy - Add workspaceLists(workspaceId) intermediate key level to both workspaceFilesKeys and workspaceFileFolderKeys so invalidation targets only the affected workspace instead of all workspaces - Replace all lists() invalidation calls with workspaceLists(workspaceId) across every mutation (upload, rename, delete, restore, update content, folder mutations) - Add optimistic updates to useRenameWorkspaceFile and useUpdateWorkspaceFileFolder with onMutate snapshot, onError rollback, onSettled reconciliation - Move storage key into the content() factory as optional param so query keys are always built through the factory (useWorkspaceFileContent, useWorkspaceFileBinary) - Fix AnimatePresence wrapping in FilesActionBar so exit animation fires on deselect - Fix ResourceColGroup to use percentage weights instead of pixel widths to prevent horizontal scroll on narrow viewports * add shift-click range selection and selection-aware context menu for files - Extend SelectableConfig.onSelectRow with optional shiftKey param; DataRow captures shiftKey before onCheckedChange fires via a ref so the Radix Checkbox interaction chain stays intact - Implement shift-click range selection in files.tsx using lastSelectedIndexRef; tracks last-selected index in visibleRowIds to compute the range - Reset lastSelectedIndexRef on deselect and select-all - Add selectedCount prop to FileRowContextMenu; hide Open and Rename when multiple items are selected, show "Delete N items" / "Download N items" labels in multi-select mode * add Move submenu to file context menu and fix shift-click anchor update - Add nested Move submenu to FileRowContextMenu using DropdownMenuSub/SubTrigger/SubContent; shows available folders filtered by selection, converts '__root__' -> null for moving to the root level - Add handleContextMenuMove in files.tsx that calls moveItems.mutateAsync directly (no modal) and clears selection on success - Fix shift-click range selection: update lastSelectedIndexRef after range select so chained shift-clicks extend from the new anchor point correctly * fix move submenu: use folder names with tree-ordered indentation instead of stale paths - Compute folder depth from parentId chain client-side (avoids stale server-computed path field) - Tree-order folders so parents appear before their children, sorted by sortOrder then name - Show folder.name instead of folder.path so optimistic renames are reflected immediately - Indent each folder by depth * 12px in the submenu so po/shit renders as 'shit' indented under 'po' - MoveOption gains optional depth field; contextMenuMoveOptions is a separate memo from moveFolderOptions (modal keeps its existing path-label behavior) * fix shift-click anchor drift and remove dead stopPropagation constant - Remove dead stopPropagation const in resource.tsx (replaced by handleSelectRowClick) - Reset lastSelectedIndexRef when visibleRowIds changes so search/filter/folder navigation doesn't leave a stale anchor that produces wrong ranges on the next shift-click - Update lastSelectedIndexRef in handleRowContextMenu when right-clicking resets selection to a single item, so the anchor matches the newly-selected row - Add visibleRowIds to handleRowContextMenu deps (now reads it to compute anchor index) - Remove moveItems.mutateAsync from handleContextMenuMove deps per project convention (.mutateAsync is stable in TanStack v5) * complete workspace files feature: audit logs, posthog events, folder restore, empty state, keyboard shortcuts, storage indicator, breadcrumb rename - Audit + PostHog: wire file_renamed, file_deleted, file_moved, file_bulk_deleted, folder_created, folder_renamed, folder_deleted, folder_moved events to all file/folder API routes - Add AuditAction.FOLDER_UPDATED, FILE_MOVED, FOLDER_MOVED to audit types - Folder restore: server function, contract, API route (POST /files/folders/[folderId]/restore), hook (useRestoreWorkspaceFileFolder), Recently Deleted integration with new File Folders tab - Empty state: contextual emptyMessage passed to <Resource> based on search/filters/folder context - Keyboard shortcuts: Delete/Backspace deletes selection, Escape deselects, Cmd+A selects all (list view only, input-aware guard) - Storage indicator: useStorageInfo drives compact "used / limit" display in file list header via leadingActions - Breadcrumb rename: current folder breadcrumb gains Rename dropdown + inline editing via breadcrumbRename (useInlineRename) - Resource: thread leadingActions prop from ResourceProps to ResourceHeader * cleanup: accessibility, emcn design tokens, react best practices across workspace UI - Add sr-only ModalDescription to dialogs/modals for accessibility - Replace hardcoded colors and z-indices with design token CSS variables - Apply emcn design review fixes across tables, knowledge, logs, settings, workflows * fix audit and posthog: FOLDER_RESTORED action on restore, fire folder_moved event separately from file_moved * sidebar: add Files section with nested folder tree; polish move UX and cleanup - Files section in sidebar shows folder/file tree with expand/collapse, matching Workflows section structure; collapsed sidebar shows flyout menu - Move action bar now uses nested DropdownMenuSub tree instead of flat modal - Context menu and action bar share renderMoveOption from move-options.tsx - FolderInput added to emcn icons barrel; all FolderInput imports migrated - Drag ghost uses CSS vars (--border, --shadow-medium, --z-toast) - Selection pruning converted from useEffect to render-time comparison - Keyboard listener stabilized with handleBulkDeleteRef pattern - toError() used consistently in restore and move route handlers * remove Files section from sidebar * restore Files nav item in sidebar workspace section * fix infinite re-render on files page - revert selection pruning to useEffect * add filefolder resource type for ingesting workspace file folders * export filefolder tree types; add toast feedback for file/folder mutations * regenerate migration as 0208 after rebase onto staging * add workspaceFileFolder to schema mock * add FILE_MOVED, FOLDER_MOVED, FOLDER_UPDATED to audit mock * add filefolder ChatContext kind and wire through schema and resolver * add filefolder to AgentContextType * add filefolder to chat context kind registry; fix resolver to use workspaceFiles table * add .deepsec to gitignore * cleanup: effect, emcn tokens, mutation error handling - Replace selection-pruning useEffect with inline state adjustment during render - Fix drag overlay using invalid --accent HSL token → --brand-secondary; z-50 → z-[var(--z-dropdown)] - Move static inline styles on context menu trigger div to className - Add missing onError toast to useUpdateWorkspaceFileFolder, useRestoreWorkspaceFileFolder, useRestoreWorkspaceFile * lint * fix: remove duplicate handleCopilotStopGeneration from rebase * feat(copilot): folder-aware file context in WORKSPACE.md * feat(copilot): add move operation to file manage API * fix(files): make targetFolder optional in move file contract * perf(files): parallelize buffer fetches, fix N+1 folder queries, stabilize drag useMemo - download route: fan out all fetchWorkspaceFileBuffer calls with Promise.all before zip assembly so 100 files resolve in one round-trip instead of sequentially - getWorkspaceFileFolder: replace per-ancestor SELECTs with a single workspace-wide folder load + buildWorkspaceFileFolderPathMap, making depth irrelevant to query count - ensureWorkspaceFileFolderPath: pre-load all workspace folders in one SELECT before the segment loop; resolve existing segments from an in-memory map; only hit the DB to CREATE missing segments; conflict retry path preserved and also updates the map - files.tsx rowDragDropConfig: move activeDropTargetId into a ref so the useMemo does not recompute on every drag-over event * fix(files): remove files/ path stripping, fix stale path in optimistic update - splitWorkspaceFilePath: remove the unconditional .replace(/^files\//, '') that clobbered paths for files inside a folder literally named "files" - useUpdateWorkspaceFileFolder: when a name update is in flight, recompute the path field for the renamed folder (replace last segment) and propagate the new prefix to all descendant folders so breadcrumbs stay correct during the optimistic window * fix(files): revert broken ref opt, clean 409 on restore, null parentId on orphaned restore - files.tsx: revert the activeDropTargetId ref optimization — the ref doesn't trigger re-renders so the drop-target highlight never updated during drag; activeDropTargetId is back in state and in the rowDragDropConfig deps - restore/route.ts: catch Postgres 23505 unique-constraint violation and return a clean 409 instead of leaking the raw error as 400 - restoreWorkspaceFileFolder: check if the parent folder is still archived before restoring; if it is, restore to root (parentId: null) so the folder is never orphaned under an archived parent * feat(search): show folder path for files in cmd-k modal, strip extraneous comments - FileItem interface with folderPath?: string[] added to search modal utils - MemoizedFileItem component renders folder breadcrumb identically to MemoizedWorkflowItem — truncated path segments on the right with / separators - FilesGroup rewritten as a dedicated memo component (was createIconGroup factory) so it accepts FileItem[] and includes folderPath segments in the search value - searchModalFiles in sidebar splits f.folderPath string into string[] segments - search-modal.tsx typed to FileItem and includes folderPath in filterAndSort - Remove self-explanatory "Phase 1" section label from download route - Remove redundant TSDoc on the unique index in db schema * fix(workspace-files): audit fixes — transaction, status codes, contract refinements, guards * fix(vfs): pass folderPath separately so buildWorkspaceMd groups files correctly * fix(types): narrow unknown fileInput with Record cast after object guard * fix(routes): replace instanceof Error with toError() across new workspace file routes * improvement(files): cleanup pass — remove unnecessary useCallbacks, consolidate emcn icon imports - Remove useCallback from 5 drag-event handlers in DataRow (passed to native <tr> elements, no observer) - Remove stable useCallback fns from 3 useMemo deps arrays in files.tsx (editingId/editValue remain) - Merge all @/components/emcn/icons subpath imports into barrel (files.tsx, action-bar, file-row-context-menu) * fix(files): apply activeSort to folders, reject drop onto current parent folder - visibleFolders now respects activeSort column (name/updated/created) and direction so folder ordering stays consistent with file ordering - isInvalidDropTarget now returns true when all dragged items are already direct children of the target folder, preventing a no-op move mutation * fix breadcrumb * add new tools to rename, create, delete folders * move more ui actions into orchestration dir * address comments * fix params * fix tests * address comments * improve error codes * address comments * address more nits * fix mcp server error code --------- Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: waleed <walif6@gmail.com> |
||
|
Theodore Li
|
b1a87d531c |
Revert "improvement(db): add session statement/lock timeouts; simplify KB doc tx (#4593)" (#4599)
This reverts commit
|
||
|
|
4295a5c855 |
improvement(db): add session statement/lock timeouts; simplify KB doc tx (#4593)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * improvement(db): add session statement/lock timeouts; simplify KB doc tx * fix(knowledge): close soft-delete TOCTOU on KB document insert Fix the race the bots flagged: KB delete is soft (`deletedAt = now`) so the FK can't catch a concurrent KB delete between the existence check and the document insert. - Add `insertDocumentsIfKbAlive` helper that gates the insert on `EXISTS(SELECT 1 FROM knowledge_base WHERE id=$kb AND deleted_at IS NULL)` in the same statement via INSERT...SELECT...WHERE EXISTS. Atomic at the MVCC snapshot — no transaction, no row lock. - Use jsonb_to_recordset to declare column types once, avoiding per-param casts for nullable columns. - Wire into both `createDocumentRecords` (bulk) and `createSingleDocument`. - Keep the upfront KB existence check as a fast-path early-out for the common case; the atomic insert is the race guard. --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> |
||
|
Waleed
|
b5dba82ac9 |
improvement(db): reduce connection saturation and egress hotspots (#4594)
* improvement(db): reduce connection saturation and egress hotspots * fix(vfs): preserve native content type in copilot SQL projection * fix(vfs): guard jsonb_array_elements against non-array contentBlocks |
||
|
Waleed
|
1ed3a4ec6d |
feat(mothership): pin tasks to keep them at the top of the sidebar (#4582)
* feat(mothership): pin tasks to keep them at the top of the sidebar * fix(sidebar): address PR review feedback for pin tasks * fix(posthog): register task_pinned and task_unpinned events * fix(tasks): insert new optimistic tasks below pinned partition |
||
|
Vikhyath Mondreti
|
43f53bb7d3 |
feat(execution): payload size bottlenecks with lazy execution value hydration, safer materialization, and batched parallel execution (#4560)
* improvement(resolver): lazy resolution for underlying fields greater than 10MB * progress * feat(parallel): batching * codegen to allow inline substitution * address comments * ui inconsistencies * cleanup redundant code * address more comments * address comments * replace helper * fix tests |
||
|
Waleed
|
39f74aa353 |
feat(data-drains): add GCS, Azure Blob, BigQuery, Snowflake, and Datadog destinations (#4552)
* feat(data-drains): add GCS, Azure Blob, BigQuery, Snowflake, and Datadog destinations
* fix(data-drains): address PR review comments
* fix(data-drains): extract sleepUntilAborted, honor abort across all destinations
* fix(data-drains): widen BigQuery projectId max and dedupe parseServiceAccount
* fix(data-drains): tighten GCS bucket contract and expose Azure endpointSuffix
* improvement(data-drains): extract normalizePrefix and buildObjectKey to shared utils
* fix(data-drains): retry BigQuery network errors; tighten Azure accountKey contract
- BigQuery insertAll now wraps the fetch in try/catch inside the retry loop so DNS failures, socket resets, and timeouts are retried with backoff instead of propagating immediately.
- Align azureBlobCredentialsBodySchema with the runtime schema (min 64 / max 120 / base64 regex) so obviously invalid keys are rejected at the API boundary rather than at drain-run time.
* improvement(data-drains): consolidate parseRetryAfter; add Datadog NDJSON line context
- Extract a single parseRetryAfter helper (capped at 30s, returns number | null) into lib/data-drains/destinations/utils.ts and remove the five local copies in bigquery, datadog, gcs, snowflake, and webhook.
- Datadog parseNdjson now wraps JSON.parse in try/catch and surfaces the failing line index, matching BigQuery's parser.
* fix(data-drains): correct Datadog size guard and Snowflake VARIANT limit
- Datadog payload guard now checks the uncompressed size against the 5 MB limit and the wire size against the 6 MB compressed limit, so gzip cannot smuggle an oversized body past the client-side check.
- Snowflake VARIANT limit is 16 MiB (16,777,216 bytes), not 16,000,000 bytes — small payloads between 16 MB and 16 MiB were being rejected unnecessarily.
- Drop the unused apiKey field on Datadog PostInput; the key is already embedded in the prepared request headers.
* improvement(data-drains): consolidate backoffWithJitter into shared utils
Datadog, GCS, and webhook each had byte-identical backoff helpers (BASE 500ms, MAX 30s, jitter ±20%, Retry-After floor). Lift the helper into lib/data-drains/destinations/utils.ts alongside parseRetryAfter and sleepUntilAborted, and drop the per-file copies and their BASE_BACKOFF_MS/MAX_BACKOFF_MS constants.
* fix(data-drains): align destinations with live provider specs
Audited every destination against live AWS/GCS/Azure/BigQuery/Snowflake/
Datadog/webhook docs and applied spec-correctness fixes:
- S3: reserved bucket prefix amzn-s3-demo-, suffixes --x-s3/--table-s3;
metadata byte formula excludes x-amz-meta- prefix per AWS spec
- GCS: reject -./.- adjacency; UTF-8 prefix cap; forbid .well-known/
acme-challenge/ prefix; ASCII-only x-goog-meta-* enforcement
- BigQuery: insertId is 128 chars (not bytes); split DATASET_RE (ASCII)
and TABLE_RE (Unicode L/M/N + connectors); UTF-8 byte cap on tableId
- Snowflake: disambiguate org-account vs legacy locator account formats;
requestId+retry=true for idempotent retries; server-side timeout=600;
default column DATA uppercase to match unquoted canonical form
- Azure: endpoint suffix allowlist (4 sovereign clouds); accountKey
length(88) base64
- Webhook: url max(2048); CRLF/NUL rejection on bearer/secret/sig header
* fix(data-drains): address PR review on snowflake poll + shared NDJSON parsing
- snowflake pollStatement: per-attempt timeout via AbortSignal.any, retry on 429/5xx with Retry-After + jitter
- bigquery parseNdjson error messages now 1-indexed
- consolidate parseNdjson variants into shared parseNdjsonLines/parseNdjsonObjects in utils
* fix(data-drains): per-attempt fetch timeouts in gcs/bigquery, snowflake poll double-sleep
- gcs.fetchWithRetry + bigquery.postInsertAll now use AbortSignal.any with a per-attempt timeout so a hung TCP connection cannot stall the drain
- snowflake.pollStatement skips the next interval sleep when it just slept for retry backoff
* fix(data-drains): bigquery probe timeout + jittered retries, align Snowflake column default UI/docs
- bigquery test() probe now uses AbortSignal.any + per-attempt timeout
- bigquery insertAll retry switches to backoffWithJitter for thundering-herd avoidance
- Snowflake column placeholder + docs say DATA (uppercase) to match the code default
* fix(data-drains): mirror webhook signingSecret min length in form gate
isComplete now requires signingSecret >= 32 to match the contract/runtime
schema so the Save button can't enable on a value that will fail server-side.
* fix(data-drains): validate JSON client-side for Snowflake before binding
Switch Snowflake to parseNdjsonObjects so malformed rows are caught locally
with 1-indexed line numbers instead of failing the whole INSERT server-side.
Re-stringify each parsed object before binding to PARSE_JSON(?).
Drop the now-unused parseNdjsonLines helper.
* fix(data-drains): cross-cutting audit pass against live provider docs
- Azure: bound retryOptions on BlobServiceClient (SDK default tryTimeoutInMs is per-try unbounded; cap at 30s x 5 tries)
- Webhook contract: mirror runtime — signingSecret.max(512), bearerToken.max(4096) + CRLF/NUL refine, signatureHeader charset + CRLF/NUL refine
- S3 (lib + contract): reject bucket names with dash adjacent to dot; require https:// endpoint at the schema layer
- Snowflake: bind original NDJSON line bytes (re-stringifying a JSON.parse'd value loses bigint precision beyond 2^53-1); check pollStatement 200 body for the SQL error envelope (sqlState/code)
- Datadog: entry builder writes defaults first then user attrs then forced ddtags/message so user rows can't clobber routing fields; validate config.tags as comma-separated key:value pairs
- registry.tsx: tighten isComplete predicates to mirror contract minimums (GCS bucket >= 3, Azure containerName >= 3 / accountKey === 88, BigQuery projectId >= 6, Snowflake account >= 3)
* fix(data-drains): force ddsource/service overrides on Datadog entries
Previous fix placed ddsource/service before ...attrs, leaving them clobberable
by a user row field. Per Datadog docs, service + ddsource pick the processing
pipeline, so a drain's routing config must not be overridable per-row. Spread
attrs first, then force all four reserved fields (ddsource, service, ddtags,
message).
* fix(data-drains): preserve row-distinguishing index when BigQuery insertId overflows
Truncating from the left dropped the index suffix, so any overflow would
collapse all rows in a chunk to the same insertId and BigQuery would silently
dedupe them. Path is unreachable today (UUIDs keep raw ~85 chars), but the
overflow branch is now correct: hash the prefix, keep the index intact.
* fix(data-drains): refresh GCS token per retry, tighten Azure key regex
- gcs: rebuild Authorization header per attempt via buildHeaders so token
refresh from google-auth-library kicks in if a 5xx retry crosses the
hour-long token lifetime
- azure_blob: pin account-key regex to {0,2} trailing '=' (base64 of 64
bytes = exactly 88 chars with up to two '=' pad chars)
* fix(data-drains): address bugbot review of 6336948f6
- gcs: allow 1-char dot-separated bucket components (e.g. "a.bucket")
to match GCS naming rules — overall name is 3-63 (or up to 222 with
dots), but per-component minimum is 1 per Google's spec
- bigquery: drain the 401 response body before re-issuing the request
with a refreshed token so undici can return the socket to the
keep-alive pool
- snowflake: hoist getJwt() above the perAttempt timer in
executeStatement so JWT signing doesn't eat the network budget
(matches the order already used in pollStatement)
* fix(data-drains): allow org-account Snowflake identifier with region suffix
The account validation rejected `<orgname>-<acctname>.<region>.<cloud>`
because `ACCOUNT_LOCATOR_RE`'s first segment forbade hyphens, while
`ACCOUNT_ORG_RE` forbade dots. `normalizeAccountForJwt` already handles
this composite form. Widen the first segment of `ACCOUNT_LOCATOR_RE` to
allow hyphens so the boundary contract and the runtime schema accept
what the JWT layer was already designed to process.
* fix(data-drains): drain retryable response bodies in datadog/gcs loops
Mirrors the bigquery 401 fix. Without consuming the body before
sleeping, undici can't return the socket to the keep-alive pool, so
each retry leaks a TCP connection instead of reusing it.
* fix(data-drains): drain snowflake poll bodies on 202 and retryable status
Mirrors the bigquery/datadog/gcs drains. Long async statements can poll
many times against the same connection; without consuming the body
undici can't return the socket to the keep-alive pool, so each iteration
leaks a connection until GC.
* fix(data-drains): consume success bodies; check Snowflake sqlState on 200
- gcs: drain the body on success paths so undici can return the socket
to the keep-alive pool
- snowflake: drain the body on synchronous 200 OK and run the same
sqlState envelope check pollStatement already does — otherwise a
statement-level failure that completes synchronously would silently
return success
* fix(data-drains): drain datadog and bigquery probe success bodies
Same undici keep-alive issue as the prior fixes: postWithRetries
returned the Response on success without draining (callers only read
headers); the BigQuery `test()` probe returned without consuming the
body. Both now drain before returning.
* chore(data-drains): regenerate enum migration as 0206 after staging rebase
* fix(data-drains): cap snowflake poll retries; tighten datadog tags min length
|
||
|
Siddharth GanesanandTheodore Li
|
0b2cfaf7f7 |
feat(mothership): add superuser env selection (#4558)
* feat(table): live cell updates via SSE + per-table event buffer
Replaces the polling-based row refetch with a push-based SSE stream that
patches the React Query cache directly as cell-state events arrive.
Architecture:
- New per-table event buffer in apps/sim/lib/table/events.ts. Redis sorted-set
with monotonic eventId, 1h TTL, 5000-event cap, in-memory fallback. Modeled
after apps/sim/lib/execution/event-buffer.ts but stripped of complexity
tables don't need (no per-execution lifecycle, no id-batching, no write
queue serialization). ~150 lines instead of 700.
- writeWorkflowGroupState appends a fat event after each successful 'wrote'.
Status transitions carry executionId + jobId; terminal/partial transitions
also include the new output values inline so the client can patch row data
without a follow-up refetch.
- New SSE route at /api/table/[tableId]/events/stream?from=<lastEventId>.
Replays from buffer on connect, polls at 500ms (mirrors workflow execution
stream), heartbeat every 15s, signals 'pruned' if the caller fell off the
back of the buffer.
- Client hook useTableEventStream subscribes via EventSource. Reconnect-resume
with last-seen eventId. On 'pruned', invalidates the rows query and resumes
from the new earliest. Cache patches walk every cached query under
rowsRoot(tableId) so filter/sort variants all stay live.
- Removes refetchInterval from useTableRows and the per-page polling effect
from useInfiniteTableRows. React Query's refetchOnWindowFocus +
refetchOnReconnect cover the durability gap if any push is dropped.
Out of scope:
- Bulk-cancel events (cancellation path is being redesigned separately).
- Generalizing the workflow event-buffer module to a shared primitive (defer
until a third use case appears; for now the table buffer is the simpler
cousin of the workflow one).
* fix(table): drop run-mutation refetch so SSE patches aren't overwritten
useRunColumn.onSettled was canceling in-flight queries and invalidating the
rows query — leftover behavior from the polling era. With the SSE stream
now keeping the cache live via incremental patches, this refetch races the
stream and snaps the cache back to whatever DB shows at the refetch moment,
which can lag the just-arrived queued/running events. Cells appeared stuck
on the optimistic 'pending' even though the SSE was delivering the real
transitions.
* chore(table): simplify SSE plumbing — reuse helpers, drop dead polling code
- Reuse snapshotAndMutateRows for SSE cache patches instead of reimplementing
the page-walk + cache-shape detection. Adds a {cancelInFlight: false} opt
for the SSE caller (mutations still cancel as before).
- Drop client-side type duplication in use-table-event-stream — import
TableEvent and TableEventEntry from lib/table/events directly.
- Drop the now-dead mergePagePreservingIdentity + rowEqual from tables.ts;
their only caller was the polling effect that was removed earlier.
- Drop the defensive try/catch around appendTableEvent in cell-write — the
function is documented as never-throwing (returns null on failure).
- Combine INCR + ZADD into one Lua eval in events.ts. Halves Redis RTT per
cell-write. Lua returns the new eventId; the script splices it into the
pre-built entry JSON.
- Trim refs to plain let bindings inside the effect; trim stale
comments referencing the old polling implementation.
* fix(table): address PR review on SSE buffer
- TTL-expiry silent miss: when all keys expire, hgetall(meta) returns empty
so earliestEventId is undefined and the prune branch was skipped. Reconnect
with non-zero afterEventId now checks the seq counter — its absence (TTL
expired) signals pruned so the client refetches. Memory fallback mirrors.
- Unbounded ZRANGEBYSCORE: cap reads at TABLE_EVENT_READ_CHUNK = 500 events
per call. The route's 500ms poll loop drains chunks across ticks instead of
flushing 5000 entries (multi-MB) in one tick after a long disconnect.
- Pruned handler closes EventSource client-side: server-side close was firing
onerror and routing through the 500ms backoff path. Now we close
proactively, reset the reconnect attempt counter, and reconnect immediately
from the new earliest.
* Cross env copilot
* Force deploy
* Run migration
* Updates
* Fix migration
* Redeploy
* Make dev db push
* restore old migs
* Cross env copilot
* Add custom tools, skills, mcps to mothership
* Update migration
* Fix migs
* UPdate
* Fix types
* Fix
---------
Co-authored-by: Theodore Li <theo@sim.ai>
|
||
|
WaleedandCursor
|
3ed8615b5d |
chore(cleanup): react-doctor dead code elimination, landing + docs overhaul, component modernization (#4544)
* fix(react-doctor): remove unused export types, useEffect clearTimeout missing, a11y fixes * fix(react-doctor): strip unused export types from contracts, copilot, stores, and components Remove export keyword from type/interface declarations confirmed to have zero importers across lib/api/contracts/tools/aws/, lib/api/contracts/*.ts, lib/copilot/generated/, stores/workflows/workflow/types.ts, ee/access-control, ee/data-retention, lib/logs/types.ts, and app/workspace component files. TypeScript and API validation both pass clean. Reduces unused-types count from 394 → 181 and fully eliminates the ✗ critical dead-code categories (exports, types, files now show as ⚠ warnings not ✗ errors). * docs improvements * fix(react-doctor): delete 50 unused files (dead barrels, unreachable components, stale utilities) Remove confirmed-unused barrel index.ts files across stores/, connectors/, executor/, lib/, and app/workspace/ that had zero importers. Also delete unreachable components (chat-history-skeleton, trace-spans, logs-list, template-profile, enterprise landing sections), stale utilities (buffered-stream, blob-to-data-url, queued-workflow-execution, compute-edit-sequence), and obsolete generated/contract files. TypeScript passes clean. * remove dead code * cleanup * more * fix(blog): restore DiffControlsDemo for v0-5 blog post * fix: restore ContactButton for enterprise blog post, export WIKIPEDIA_PAGE_CONTENT_OUTPUT_PROPERTIES * fix(react-doc): restore stripped exports and remove server-only dependency * chore(deps): update lockfile after removing server-only * added back some exports * docs * more * fix type issues * tc * fix docs search route Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tag-dropdown): add missing isEqual import from es-toolkit --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Vikhyath Mondreti
|
6cb779601a |
feat(search-replace): search & replace, cut, deploy modal ui flicker (#4507)
* feat(search): workflow search and replace * fix alignment * fix hidden fields bug * fix loops/parallel badge case * resource resolver * add cut * update docs * address comments * make source code for func blocks dispay resolved code instead * fix match issue * fix padding |
||
|
Vikhyath Mondreti
|
1e5cc392d5 |
improvement(deployment): solve multiple client side races, and deployed state management issues (#4502)
* improvement(deploy): state transitions * more fixes * improvements and shift to outbox policy with eager call * address comments * ux improvement * address comments * address bugbot |
||
|
Vikhyath Mondreti
|
09425553b3 |
fix(sanitize): repair malformed subblock states (#4490)
* fix(sanitize): repair malformed subblock states * address comments * address comment |
||
|
WaleedandClaude Opus 4.7
|
d721dc3358 |
feat(enterprise): add data drains for continuous export to S3 / webhook (#4440)
* feat(enterprise): add data drains for continuous export to S3 / webhook * chore(data-drains): regenerate migration on top of staging + bump route baseline Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(data-drains): clarify retention pairing is user-coupled, not enforced Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(data-drains): preserve explicit forcePathStyle=false + reserve x-sim-signature Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(data-drains): drift guard ensures every webhook header is reserved Asserts that any header buildHeaders writes is rejected when reused as a custom signatureHeader. Adding a new metadata header without mirroring it into RESERVED_SIGNATURE_HEADER_NAMES now fails CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
09f4c94b3c |
feat(block): Allow wait block to wait up to 30 days (#4331)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * feat(block): Allow wait block to wait up to 30 days * restore ff * Filter out waits from hitl endpoints * Use correct count, filtering out wait blocks * improvement(wait): tighten poll route and pause-manager helpers - Parallelize per-row dispatch with Promise.all - Add status='paused' guard on nextResumeAt rewrite to prevent clobbering concurrent resumes - Extract computeEarliestResumeAt + PauseResumeManager.setNextResumeAt helpers - Use canonical PausePoint type in poll route (drop StoredPausePoint) - Narrow UNIT_TO_MS via as const + WaitUnit guard - Bump LOCK_TTL_SECONDS above route maxDuration - Clearer error when allowedPauseKinds rejects a resume --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> |
||
|
Theodore Li
|
c09e0a04bd |
fix(copilot): disambiguate VFS upload paths to prevent stale-row reads (#4454)
* fix(copilot): disambiguate VFS upload paths to prevent stale-row reads |
||
|
Waleed
|
68f66ba3f3 |
fix(credentials): clear stored refs on credential delete to prevent silent cascade orphaning (#4418)
* fix(credentials): clear stored refs on credential delete to prevent silent cascade orphaning * fix(testing): sync auditMock with new CREDENTIAL_RECONNECTED action * improvement(credentials): parallelize independent ref-clearing scans |
||
|
Waleed
|
af55bad491 |
fix(uploads): direct-to-upload workspace files + shared transport (#4407)
* fix(uploads): direct-to-S3 workspace files + shared transport
* chore(testing): centralize posthog and storage-service mocks
* fix(uploads): address PR review — abort propagation, orphan cleanup, error handling
- Throw immediately on AbortError in KB retry loop (no useless 14s backoff)
- Cleanup S3/Blob object on quota or size-cap rejection in registerUploadedWorkspaceFile
- Enforce MAX_WORKSPACE_FILE_SIZE at registration (defense vs presigned PUT lying about size)
- Handle non-OK / non-JSON responses in workspace-files upload paths
* fix(uploads): add Zod contracts for workspace presigned/register routes
* fix(uploads): correct BlobServiceClient type name in headBlobObject
* fix(uploads): address PR review — typo, complete-failure cleanup, double-increment
* fix(uploads): preserve fallback size and reuse existing display name on re-register
* fix(uploads): surface server error message and bypass quota for local-storage fallback
* fix(uploads): align register response schema with UserFile; skip presigned for KB large files
- registerWorkspaceFileResponseSchema now matches the UserFile shape the route actually returns; previous schema required workspace DB-row fields that were never populated, causing requestJson validation to reject successful uploads.
- KB batch presigned fetch now skips files >= LARGE_FILE_THRESHOLD since multipart bypasses the per-file presigned URL anyway.
* fix(uploads): idempotent register skips duplicate audit/posthog; add edge-case tests
- registerUploadedWorkspaceFile now returns { file, created } so the route can skip captureServerEvent and recordAudit on idempotent re-register (existing metadata reused). Previously a re-register fired duplicate analytics + audit log entries.
- Add tests covering: idempotent re-register skips audit/analytics, isNetworkError matches econnreset/timeout/etc keywords, multipart complete failure fires action=abort cleanup.
* fix(uploads): include 50MiB boundary in batch presigned fetch
* fix(uploads): trust HEAD size to prevent quota inflation
The head.size > 0 fallback let a client PUT 0 bytes and register
with an inflated size, debiting quota without storing data. HEAD
on an existing object always returns the true byte count, so trust
it directly — a genuine 0-byte file correctly contributes 0.
* fix(uploads): audit verified file size, not client-supplied
* fix(uploads): handle register retries and name-collision races
Two bugs in registerUploadedWorkspaceFile:
1. Register retry could orphan storage. When a successful response
was lost on the wire and the client retried, the quota check saw
the bytes already counted, failed, and cleanupOrphan deleted the
already-registered storage object — leaving the DB row pointing
to nothing. Fix: check getFileMetadataByKey before quota guard
and short-circuit on existing record.
2. Concurrent same-named uploads could lose data. allocateUniqueWorkspaceFileName
is best-effort; two racing uploads can pass it and both attempt
the same display name. The loser's insert hits 23505, the catch
block called cleanupOrphan, and successfully-uploaded bytes
were deleted. Fix: retry on 23505 with a fresh allocateUniqueWorkspaceFileName,
matching the pattern in uploadWorkspaceFile. Throw FileConflictError
after exhaustion.
* fix(uploads): retry transient DirectUploadErrors at outer KB level
The KB outer retry only triggered on isNetworkError, missing
transient 5xx from S3/Azure (DirectUploadError code
DIRECT_UPLOAD_ERROR or MULTIPART_ERROR). Adds isTransientUploadError
and retries on it, restoring resilience for small-file presigned
PUTs against flaky cloud storage.
* fix(uploads): only retry transient 5xx, not deterministic 4xx
DirectUploadError now carries the HTTP status. isTransientUploadError
gates on 5xx so callers don't loop on 400/403/404 (e.g., malformed
request, expired signature). Multipart per-part retry also short-circuits
on 4xx — same reasoning.
* refactor(uploads): collapse getFileContentType into resolveFileType
The two helpers differed only in whether application/octet-stream
falls back to the extension map. Add an option flag to resolveFileType
and keep getFileContentType as a thin wrapper for direct-PUT callers
that need to preserve the exact browser-reported content-type.
* chore(uploads): trim verbose comments
Drop inline comments that restate code ("Use the full storageKey as fileName"),
collapse a multi-line block comment into a tighter TSDoc on the existence
check, and prune verbose vitest file headers — describe blocks already
document what's tested.
* fix(uploads): regenerate fileId per insert retry; require cloud storage for register
* fix(uploads): cap formdata fallback at 100MB; drop unused size param
* fix(uploads): abort multipart on get-part-urls failure; retry register on transient errors
* fix(uploads): drop vestigial size field from register contract
* fix(uploads): abort multipart on complete-fetch throw
* fix(uploads): set kb presignedEndpoint fallback; race-safe blob HEAD
* fix(uploads): include ?type=knowledge-base on kb presigned fallback
* fix(uploads): remove abort listener on xhr timeout
* fix(uploads): add timeout/abort to kb api fallback upload
|
||
|
Theodore Li
|
31cfb74dc2 |
feat(table): add workflow execution column type (#4338)
* Add table triggers for columns and row added * Add async batching job for running column * Add ui improvements, stop mechanism * Use trigger dev for workflow runs * Use unified column sidebar for table * Add socket waits for tables, multi column workflow support * change back to cell based trigger jobs * Reuse code, add view log inline in table view * reorganize db to treat each column separately * adjust column naming strategy * Column ui improvements * fix live update on table * Change new column behavior * fixed errored workfows not showing as stopped * Column sidebar improvements * fix table column swapping behavior * fix bugs * add prompting, fix lint * fix ui stuff * flip feature flag * Add zod contracts fix initial auto-run of columns * ui improvements * Use db filter to query unran rows * Use live workflow run * Change wording for deleting workflow column * Update tools * Add tool to run selected rows * Add mothership tools * adjust col width * Restore ff * fix drizzle migration * fix test |
||
|
Waleed
|
879dab9f19 |
feat(table): make plan table limits configurable via env vars (#4406)
* feat(table): make plan table limits configurable via env vars * fix(table): coerce env table limits to number for skipValidation env * improvement(env): extract envNumber helper for numeric env coercion * improvement(knowledge): use envNumber helper for KB_CONFIG_* env reads * fix(testing): add envNumber to env mock factory * fix(env): allow zero in envNumber for max-throughput configs * fix(env): add min option to envNumber for strict-positive configs |
||
|
Vikhyath Mondreti
|
af859cd508 |
feat(workflows): lock/duplicate improvements for workflows (#4387)
* feat(workflows): lock/duplicate improvements * fix duplicate var remap bug * address comments * remove dead vars * fix tests * address comments * code cleanup * address comments * address comments * minor change * remove dead code |
||
|
WaleedandClaude Opus 4.7
|
e2b3ae43e3 |
fix(terminal): correct error/cancel block status in logs panel (#4372)
* fix(terminal): correct error/cancel block status in logs panel Three bugs in the workflow editor's terminal/logs panel where block status diverged from the engine's truth on error paths: 1. **Errored block shown as "canceled"** — when the SSE 'add' mode produced a duplicate entry on block error and `cancelRunningEntries` then swept the original placeholder. 2. **Upstream blocks stuck on "Running"** — terminal events arrived before the engine's last block events under reconnect/timeout, so the live panel never received the per-block terminal state. 3. **Phantom "Run Error" pseudo-row** — the failing block rendered as "canceled" while a synthetic row carried the real error text. Fixes: - **Fix B** (`addConsoleErrorEntry`): when a running placeholder exists for `(blockId, executionId)`, route through `updateConsoleErrorEntry` instead of creating a second entry. Aligns 'add' mode with the existing 'update' mode behavior. - **Fix C** (`reconcileFinalBlockLogs`): terminal SSE events now carry `finalBlockLogs` (server-authoritative snapshot). On execution:error / execution:cancelled, reconcile any still-running entries with their server-side terminal state. Recovers correctness on network drop, server timeout/abort, and reconnect-resume paths where individual block:* events may not have reached the client. - **Fix D** (`addExecutionErrorConsoleEntry`): cross-check `useTerminalConsoleStore` for entries with `error` set scoped to the executionId before emitting the synthetic "Run Error" row. Suppresses the phantom row when the failing block already carries the message. - **Signature refactor**: `handleExecutionErrorConsole` / `handleExecutionCancelledConsole` now take a typed `ExecutionConsoleDeps` object instead of stacking positional deps — matches the existing `createBlockEventHandlers(config, deps)` precedent in the same file. Tests: - 12 tests in `workflow-execution-utils.test.ts` covering Fix B/C/D and the deps-object signature refactor. - Centralized terminal-console store mock in `@sim/testing` so future tests can stub the store without per-file boilerplate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(terminal): wire copilot cancellation to finalBlockLogs reconciliation Address Greptile review: - `executeWorkflowWithFullLogging`'s `onExecutionCancelled` was `() => {}` and silently dropped the `finalBlockLogs` payload, so Bug 2's "upstream blocks stuck on Running" fix did not fire on copilot-initiated cancellations. Wire it through `handleExecutionCancelledConsole` to match the SSE-route `onExecutionCancelled` path. - Test for the `blockType !== 'error'` filter used a different `executionId` than the seeded entry, so the executionId scope rejected the entry before the blockType predicate ran. Align executionIds so the test actually exercises the filter. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(terminal): pass durationMs in reconnect cancellation handler Reconnect-resume `onExecutionCancelled` was forwarding `finalBlockLogs` but not `data?.duration`, so the "Run Cancelled" entry rendered with a 0ms duration. Match the other two `handleExecutionCancelledConsole` callsites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Waleed
|
0c25fc4ee1 |
fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies (#4369)
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies
- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml
Fixes simstudioai/sim#1243
* fix(auth): address PR review comments
* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)
* fix(docker): include @sim/utils in migrations image
Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.
* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections
The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.
* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema
The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.
* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional
The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:
- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
http://localhost:3002 to empty for consistency with prod.yml; the
in-code localhost fallback handles the dev case identically
Applied across all 6 documentation languages (en/fr/de/ja/es/zh).
* chore: untrack and ignore .claude/scheduled_tasks.lock
|
||
|
Vikhyath Mondreti
|
b8959eb20d |
improvement(repo): zod based client-server boundary (#4355)
* improvement(repo): centralized zod contracts (#4336) * improvement(repo): zod schema contracts * type checks * fix(notion): correctly register tool (#4337) * fix func blokc * more improvements * fix tests * type check * remove v3 refs * minor type improvements * address comments * update jira contract * remove validateJsonBody * improvement(repo): consolidation of boundary helpers + better unknown usage (#4352) * improvement(repo): consolidation of boundary helpers + better unknown usage * address comments * improve file transfer error messaging * fix docs listing schema drift * fix inocrrect type casting * address council comments * remove prefix |
||
|
Waleed
|
36742740aa |
fix(cleanup): batch orphaned snapshot deletes to avoid slow-query spike (#4348)
* fix(cleanup): batch orphaned snapshot deletes to avoid slow-query spike * fix(cleanup): recheck orphan condition in delete to close TOCTOU gap |
||
|
Vikhyath Mondreti
|
2e3de9ac8a |
feat(governance): external workspace users from outside org (#4313)
* feat(governance): external workspace users from outside org * update docs * address comments * edge case improvements * remove unused fallback * address comments * add outbox for seat reduction * fix edge case with org join after invite * add server side batch invites for workspace * use zod schema for route |
||
|
WaleedandClaude Opus 4.7
|
60b80ec172 |
improvement(tables): race-free row-count trigger + scoped tx timeouts (#4289)
* improvement(tables): race-free row-count trigger + scoped tx timeouts * fix(tables): close upsert race + serialize replaceTableRows Two concurrency bugs flagged by review: 1. `upsertRow` insert path: removing FOR UPDATE broke serialization between the initial existing-row SELECT and the INSERT. Two concurrent upserts on the same conflict target could both find no match, then both insert, producing a duplicate that bypasses the app-level unique check. Fixed by re-checking for the matching row *after* acquiring the per-table advisory lock; if a racing tx already inserted, switch to UPDATE. 2. `replaceTableRows`: under READ COMMITTED each tx's DELETE uses its own MVCC snapshot, so two concurrent replaces could each DELETE only the rows visible at their start, then both INSERT, leaving the table with the union of both row sets. Fixed by acquiring the per-table advisory lock at the start of the tx to serialize replaces against each other and against auto-position inserts. Also updated a stale docstring on `upsertRow` that still referenced the removed FOR UPDATE. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(tables): serialize explicit-position inserts with advisory lock The `(table_id, position)` index is non-unique. Concurrent explicit- position inserts at the same slot can both observe the slot empty, both skip the shift, then each INSERT — producing duplicate `(table_id, position)` rows. Take the per-table advisory lock in the explicit-position branches of `insertRow` and `batchInsertRows` (the auto-position branches already do this). Updated the test that asserted the explicit path skipped the lock, and added coverage for `batchInsertRows` with explicit positions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(tables): dedupe upsert UPDATE path + extract nextAutoPosition Two pure cleanups on the round-1 changes: 1. Extract `nextAutoPosition(trx, tableId)` — the `SELECT coalesce(max( position), -1) + 1` pattern was repeated in three call sites (`insertRow` auto branch, `batchInsertRows` auto branch, `upsertRow` insert branch). One helper, same behavior. 2. Consolidate `upsertRow` update path. The initial-SELECT match and the post-lock re-select match previously had two literal duplicates of the same UPDATE + return block. Resolve `matchedRowId` first, then run one UPDATE branch. Lock is still only acquired when we don't find a match on the first pass. No behavior change. 98/98 table unit tests unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Theodore Li
|
04f1d015f3 | fix(mothership): Use heartbeat mechanism for chat locks (#4286) | ||
|
Waleed
|
ccb5f1e690 | fix(db): revert statement_timeout startup options breaking pooled connections (#4284) | ||
|
Theodore Li
|
c22ac38ab0 | Set statement timeout of 90 seconds (#4276) | ||
|
WaleedandClaude Opus 4.7
|
0ae19dab85 |
feat(files): default sort by updated and add updated sort option (#4279)
* feat(files): default sort by updated and add updated sort option * feat(files): show Last Updated column Matches the visible-column pattern already used on Knowledge and Tables so users can see the value they're sorting by. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Theodore Li
|
65972f2fa3 |
fix(retention): switch data retention to be org-level (#4270)
* fix(retention): switch data retention to be org-level * fix lint * cleanup mothership ran logs * fix cleanup dispatcher * fix ui flash for data retention settings * fix lint * remove raw sql string interprolation |
||
|
|
5f0f0edd63 |
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Theodore Li
|
8ce56fe1f2 |
fix(auth): add api key auth via sha256 hash lookup (#4266)
* fix(auth): add api key auth via sha256 hash lookup * Remove promise all logic * Restore feature flag * fix feature flag * Combine auth and hash gate |
||
|
Theodore Li
|
d927d8bdff |
fix(db): raise db pool size (#4263)
* fix(db): raise db pool size * Raise socket connections * bump up connection size even more |
||
|
Vikhyath Mondreti
|
41a1b50ace | improvement(migrations): log better errors (#4260) |