mirror of
https://github.com/simstudioai/sim.git
synced 2026-11-05 00:02:28 +08:00
9664e7ec17331864f260bd50589aff53675b4e22
40
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vikhyath Mondreti
|
2da885582c |
test(realtime): wait for the idle read the streak test depends on (#6673)
The guard failed intermittently in CI — `expected 1870 to be less than 1000`, which is the carried-streak backoff, meaning the streak was never reset before the phase that measures it. The middle phase cleared the fault and then waited a FIXED 1000ms for an idle read to land. It usually did. But the pending backoff from the two failures before it runs 400–600ms then 800–1200ms, so the next read is due anywhere up to ~1805ms — and the phase ends at 1800ms. When both jitters drew high the read arrived after the fault had already been re-armed, so it failed instead of succeeding, the streak survived at two, and the measurement caught the third backoff (1600–2400ms) rather than the first. Waiting for a duration where the thing being waited for is an event is the bug. Each phase now waits for its own event: two failed reads to build the streak, then a read that actually RETURNS to clear it. Also measure failure-to-failure rather than read-to-read. A successful read can land in the instant after the fault is re-armed, and as the first sample it would make the gap ~5ms — passing for the wrong reason, the same false-pass shape review caught in this test last round. 20 consecutive runs green; still fails on the un-fixed reader every time (1753ms, 1688ms, 1881ms, 1701ms, 1837ms against the 1200ms bound). |
||
|
Vikhyath Mondreti
|
bed25e2a86 |
fix(realtime): keep the file-doc store reconnecting instead of dying quietly (#6661)
* fix(realtime): keep the file-doc store reconnecting instead of dying quietly A relay that lost Redis for longer than its retry budget did not degrade — it went silently split-brain and stayed that way. The reconnect strategy returned an `Error` after ten attempts, which tells node-redis to give up and CLOSE the client, and a closed client rejects every command with "The client is closed" for the rest of the process's life. From that point the task kept serving clients while its rooms stopped receiving other tasks' updates, its own edits stopped reaching the shared stream (also the crash buffer between persists), and seeds, locks and the persist If-Match token all failed. The tail loop then treated that as a transient read error — `running` is only false during shutdown, so it retried every 500ms forever, one warning per attempt. A tab left open overnight produced thousands of identical lines, which is how the actual failure stayed invisible. - Never stop reconnecting. This process holds live documents whose only convergence path is that connection, so a connection it can rebuild is always worth rebuilding. Same capped backoff, now via the shared `backoffWithJitter`, and no error return. - Back the reader off after a failed read (500ms → 10s) instead of retrying at the read cadence, re-open a client that was CLOSED — node-redis reconnects a dropped client, never a closed one — and log the first failure of a streak then one in twenty, carrying the streak length, so an outage stays visible without burying itself. Pinned by a test that models a closed connection: six read attempts in three seconds before, about three after, and proof the reader is re-opened rather than abandoned. * fix(realtime): end the reader's failure streak on an idle read, not a busy one Review findings, both accurate. The streak reset sat after the entries were applied, so a blocking read that timed out with nothing new — the idle steady state — skipped it via `continue`. A healed outage's count therefore survived through normal polling, and the next unrelated blip opened at the backoff cap: minutes of avoidable split-brain, and a log line claiming a failure count it never earned. The streak now ends on the read RETURNING, which is what proves the connection works. Also: the new test built a raw `setTimeout` promise instead of the shared `sleep`, which CLAUDE.md calls out by name. * test(realtime): assert the retry delay, not a count inside a window The streak-reset guard could pass on the very regression it exists to catch. It counted read attempts inside a 1900ms window, and the jittered delay for a carried streak is 1600–2400ms — so whenever jitter landed below about 0.95, a second read fell inside the window and the assertion held even though the idle reads had never cleared `failures`. A single falsification run happened to draw a long delay, which is exactly how a guard like this goes quiet. Assert the delay itself instead. The first retry after a reset is 500ms ±20% (400–600ms); carried over it is the third, 2000ms ±20% (1600–2400ms). Those ranges are disjoint, so the check no longer depends on which jitter is drawn: against the old placement it now fails every time (measured 2160ms, 1925ms, 2046ms against the 1000ms bound). |
||
|
Waleed
|
6de8ba2504 |
fix(v2): close the correctness gaps an end-to-end audit found (#6655)
* fix(v2): stop a third-party tool description from 500ing MCP discovery
`v2McpToolInputSchema` declared `description: z.string().optional()` inside a
`.catchall(z.unknown())` object, and a declared key beats the catchall. The MCP
SDK's own `ToolSchema.inputSchema` does not declare `description` at all, so any
value — including the JSON `null` a Python server emits for an absent one —
passes its validation and reaches Sim unchecked. The builder's outbound `.parse()`
then threw, and the discovery error policy correctly declines to classify a
Sim-side schema defect, so the endpoint that completes MCP onboarding answered a
bare 500. The key is dropped and left to the catchall; `type`, `properties`, and
`required` stay pinned because the SDK enforces those at least as tightly.
Also in the v2 resources family:
- The single-resource query schemas for MCP servers, skills, custom tools, and
secrets are now `.strict()`, matching every list in the same family. A mistyped
flag was silently ignored behind a 200.
- `openapi/resources.ts` re-derived `RESOURCE_ERRORS` and
`RESOURCE_CONFLICT_ERRORS` inline in 21 of 22 operations. They now import the
shared constants; the generated spec is unchanged, which is the point.
- The internal MCP refresh route stamped `updatedAt` alongside `lastToolsRefresh`.
`updatedAt` means "configuration last changed" and is a public keyset sort, so
a refresh moved rows out from under an in-flight page. `updateServerStatus`
already held that invariant; the route now matches it.
- The discovery cooldown is a typed `McpServerCooldownError` rather than a
substring search for `cooldown`. `McpConnectionError` interpolates the server's
display name into its message, so a server named after the word was reported as
a transient cooldown when its connection had genuinely failed.
* fix(v2): close correctness gaps in the workflows deployment surface
Deploy and rollback bodies were plain objects, so a misspelled key was
stripped rather than rejected. On rollback that is silent misbehavior:
an omitted `version` legitimately means "reactivate the preceding
version", so `{"versoin": 5}` rolled back somewhere else and answered
200. Both v2 bodies, the run-read query, and the versions cursor are now
strict.
Deployment versions are an `integer` column, but the path param, the
versions cursor, and the v1 body each bounded it differently or not at
all — an out-of-range value overflowed the comparison into an
unclassifiable 500. One exported bound now covers all three.
Resume admission raised bare `Error`s for a stale contextId or an
already-resumed run, which the resume surfaces could not classify and
reported as 500. They now use the sibling `ResumeAdmissionError` already
in that file, carrying 404/409/400 and whether an automatic retry can
clear the refusal.
Docs corrections: rollback publishes the 409 its webhook-path conflict
already produces; deploy/undeploy/rollback reject a workspace key with
403, not the concealed 404 they documented; the workflows OpenAPI module
imports the shared error sets instead of re-deriving them; import and
the folder ops explain their folder-tree 413. The export route is marked
`headSafe: false` so a HEAD probe stops filing a WORKFLOW_EXPORTED audit
event for an export that never happened. `runId` is one bounded schema
across the run and log resources.
* fix(v2): conceal knowledge upload existence, tighten knowledge/files bounds
Security: the four knowledge document-upload routes rendered a bare upload
error policy with no resource concealment, while every sibling knowledge route
uses one. Because the use case resolves the knowledge-base context before
workspace authorization, the unconcealed 403 told any valid API-key holder that
a knowledge base exists in a workspace it cannot reach — the exact signal
GET /api/v2/knowledge/{id} withholds by answering 404 either way. All four now
use the composed concealing policy, which also renders the 415/402/413 the
route-local renderer already handled; that duplicate renderer is deleted.
Contracts:
- POST /knowledge/search is strict. It was the only non-strict v2 request body,
so a mis-cased rerankerEnabled or topK returned 200 with the key stripped,
changing what the caller was billed and silently disabling reranking.
- The document list takes limit, cursor, and search from the shared v2 schemas.
search was an unbounded, empty-accepting v1 string, so ?search= answered 200
with a full page here and 400 on GET /knowledge, and the term reached an
unindexed filename LIKE scan with no ceiling.
- The 16 non-strict single-field workspace query slices across both families are
strict, matching GET /knowledge/{id}/tags.
- GET /audit-logs takes workspaceIdSchema instead of a bare string (?workspaceId=
was forwarded as a filter and returned zero rows) and the shared run-window
bounds for startDate/endDate.
Documentation:
- listAuditLogs drops the 404 it has no code path to emit.
- upsertFileShare describes its workspace-key refusal as the 403 it renders;
the operation denies the key by principal kind, which the concealment policy
does not rewrite.
- The 12 body-reading knowledge and files operations publish the 413 their
pre-validation body read raises, and the file list publishes the folder-tree
413 its now-capped path index raises.
Correctness: queryWorkspaceFilePage loads its folder path index under
MAX_FOLDERS_PER_WORKSPACE like the workflow, table, and knowledge lists. An
uncapped index does not fail on truncation, so a real folder outside the read
rows resolved to undefined and answered "Folder not found".
* fix(v2): publish the reachable 413 on body-carrying resources ops
`parseRequest` buffers a JSON body through `parseJsonBody` under
`DEFAULT_MAX_JSON_BODY_BYTES` before any schema runs, and the v2 builders supply
`V2_PARSE_DEFAULTS.payloadTooLargeResponse`, so every operation whose contract
declares a body already answers 413 above the cap. The resources family
published it on none of them. A status a caller cannot see in the spec is a
status they will not handle.
Adds `RESOURCE_BODY_ERRORS` and `RESOURCE_CONFLICT_BODY_ERRORS` to the shared
sets and applies them to the seven affected operations: createMcpServer,
updateMcpServer, createSkill, updateSkill, createCustomTool, updateCustomTool,
and setSecret. All seven are `defineV2JsonRoute` handlers on non-GET methods
with no `parseOptions` override, so the 413 is genuinely reachable on each. The
new sets are opt-in rather than folded into the base sets precisely because
reachability is not automatic — an operation with no body, or one whose payload
reaches it through an uncapped path, would be publishing a response that can
never arrive.
A sweep test pins the invariant across the resources, billing, and logs
documents. It is one-directional by construction: several bodyless operations
publish 413 for their own folder-tree and render ceilings, so the converse would
flag correct documentation.
Also completes the shared-constant consolidation started in
|
||
|
Vikhyath MondretiandClaude Opus 5
|
e8d278b6e4 |
fix(files): stop the collaborative editor rewriting and reflowing a document on open (#6652)
* fix(files): stop the collaborative editor rewriting and reflowing a document on open Opening a file rewrote it. Binding an editor to a seeded document emits a Yjs update of its own — ProseMirror appends an empty paragraph to any doc that does not end in one — which the relay saw as a real edit and persisted. Every open therefore uploaded the file under a FRESH storage key and deleted the old one, 404ing the page's own in-flight content read, bumping "Last Updated" just from viewing, and churning a blob per open. Worse, a trailing blank line cannot serialize, so the file never recorded that paragraph and nothing reconciled the two: each client that seeded without seeing another's contribution stacked one more. A real document reached 18 against the placeholder's 1 — measured as the pane growing several hundred pixels the instant the live editor took over. - Seed and merge through the editor's own normal form (`editorNormalForm`), so binding is a no-op and `canonicalizeYDoc` collapses an accumulated run back to one. Placed at the collab boundary, not in `parseMarkdownToDoc`: only the CRDT has to agree with the editor — every other consumer of the parse renders through a real editor that normalizes itself. - Skip a persist whose projection already matches the durable bytes. Byte length is the free reject, so the compare read only happens when a no-op write is actually on the table. - Revoke collaborative readiness on a fatal join. The sticky `syncedOnce` latch outlived the document: after a readiness timeout the provider drops `synced` so the gate closes, but the latch re-opened it on the offline fallback's seed flag — handing back an EDITABLE editor on a document the provider had abandoned, with client autosave gated off because collaboration is nominally on. Keystrokes went nowhere and vanished on reload, with no error shown. - Recover from a superseded storage key instead of stranding the reader: a 404 re-resolves the file record, so the read re-keys onto the current object. And do not focus-refetch durable bytes while the relay owns durability. - Prefetch the workspace file list in the layout, where the sidebar already reads it. `HydrationBoundary` defers an already-seen query to an effect that SSR never runs, so a page-level prefetch of that key could not reach the server render — the file route rendered a spinner and disagreed with the client about the header's markup (a hydration mismatch). - Load the document font with `display: block`. A swap repaints prose in metric-adjusted Arial first, so paragraphs re-wrap when the real face lands. Also: a detail-route `loading.tsx` (the segment was inheriting the list chrome), `normalize.ts` renamed to `field.ts` now that it holds only the field constant, and the duplicate `COLLAB_DOC_FIELD` in the streaming path folded into it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(files): serve a whole document on join, and stop persistence deadlocking Opening a file right after an edit replayed it — the block you had just moved moved again, in front of you. A room rebuilds itself from the file's Redis stream one entry at a time, into the same Y.Doc that fans every update out to its room, and the join attached the socket before that finished and before the server seed. So a client was never sent the document; it was sent the document's history, and it watched that replay. The new join-readiness test reproduces it exactly — ['AAA', 'BBBAAA'] where one state was due — and fails without the fix. Underneath it, persistence had deadlocked. The If-Match token is a REMEMBERED timestamp: held in the room, which dies with it, and in a cluster key written fire-and-forget. A relay that exits in the moments after a successful write comes back holding a version older than the file's, every persist then fails the CAS, and because a conflict neither writes nor advances the token, that document can never be persisted again. Measured on a live file: token 1786594348911 against a content version of 1786594418409, with 103 unpersisted entries still in the stream. The durable markdown froze there — and the editor's placeholder is built from it, so every reload painted the pre-edit document and the live one corrected it on screen. - Assemble a room before attaching a client to it. The join awaits the stream catch-up and the seed, so the first sync IS the finished document, in one message. The seed is memoized on the room, so concurrent joins wait for the same one instead of the second being served an empty doc; a task that loses the seed lock PULLS the winner's seed from the stream rather than waiting for the tailer to push it, which is what left a freshly uploaded file read-only until its readiness deadline lapsed. - Drop the client-side quiet-frame gate that was standing in for this. It was unsound in both directions: it delayed a document that was already correct, and it opened mid-flight anyway whenever updates arrived more than a frame apart, which is what a cross-region Redis and a long room history produce. - Await the cluster version write after a successful persist. It is the only record that survives a teardown, and one round trip after a blob write is not a cost worth a wedged document. - On a version conflict, ask the CONTENT, not the clock. If the file still holds the bytes this document last projected — the tag written with every persist — then nothing out-of-band exists to protect, so re-sync the token and write. Any other bytes and the conflict stands exactly as before. - Actually run the stale-storage-key recovery. It was requested from inside the failing read's own queryFn, where react-query drops it, so nothing re-resolved the record and the reader sat on a dead key showing "Failed to load file content" until something unrelated refetched. It now runs off that cycle and cancels a read already in flight, which could only hand back the dead key. While the record is re-resolving the surface reports loading, not failure. - One document per file, for its whole life. A document rebuilt from markdown is a DIFFERENT document to Yjs — its items carry new client ids — so a client still holding the old one merges the file into itself, twice, on both sides. The seed now stores what it builds (until that row existed, a file opened but never edited was rebuilt on every open) and resumes it with a CRDT diff when the markdown moved on out-of-band. A document also carries an identity the join ack names, so a tab that outlived its room is refused rather than merged. - Let a browser keep an embedded image. The inline route sent no-cache with no validator, so every open re-downloaded the whole image — measured at ~1 MB per open of a real document, with the image area blank until it landed. A `key` names one storage object and a content write never rewrites one, so those bytes are immutable; a `fileId` names the file, whose bytes move, so that form still revalidates. * fix(files): name every collaborative document, and only cache what the URL names Two findings from review, both real. A document stored before identities existed is returned by the seed's fast path on every open, and that path never named one — so those files could never acquire an identity, and the join-ack guard could never fire for them. That is the population most likely to have a tab that outlived its room, which is the case the guard exists for. The fast path now names an unnamed document and stores it, once: minting without storing would name it differently on every open and the guard would start refusing clients that hold the very same document. The inline route marked a response immutable whenever the caller passed a key, but a key is resolved to a FILE and the file's current key is what gets streamed. A content write landing between those two reads would serve the new bytes under a URL naming the old object — and cached for a year, that is wrong forever. The flag is now what it always meant: the URL names the exact object that was streamed. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
245ad1bd46 |
feat(workflows): new workflow block card, progress indicator, colors, dsl for natural language preview, retry configs (#6458)
* improvement(workflow): refine canvas interactions and rendering
* fix(workflow): keep outputs on the right, focus newly created blocks
Connection anchors: an output now always leaves a card from the right.
The cursor swell lets a drag start on any edge, but the left side is the
input, so anchoring an outgoing edge there drew a line out of the input
port and read as a second input. `normalizeCursorSourceHandleId` resolves
every drag to the right anchor, `normalizePositionedSourceHandleId`
collapses `source-left` alongside the legacy vertical anchors (so data
from the API, an older client, or a stale save self-heals on load), and
only the right-side source anchor is mounted.
Drops in `onConnectEnd` are always source -> target. The branch that
reversed the edge for a drag starting on an input could never run: the
`target` handle is `isConnectableStart={false}` and the positioned side
anchors are `isConnectable={false}`, so React Flow never reports an input
as a drag origin. Removed it and its now-unused imports.
A newly created block is centered once its node mounts and is measured,
so a card added from a drag-release, the block menu, or the toolbar is
never left off-screen or under the editor panel.
The editor panel's block icon uses the same type accent as the card's
badge instead of the block's legacy `bgColor`, which had left the panel
on the old per-integration brand colours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(workflow): floor header-only card height, adopt brand tag palette
The Start card intermittently collapsed after load, squashing the
action-menu tab so its icon row sat over the card.
`.workflow-drag-handle` is the host the border renderer measures, and both
it and the header row took their height from `blockHeight && blockHeight >
0`. `blockHeight` comes from the deterministic-dimensions pass and is
already floored at MIN_PAINTED_HEIGHT (48), but it is absent on the first
frames — and with no floor the host collapsed to its natural content
height (25.5px for a header-only trigger, exactly the title's line box).
The border builds its perimeter from `host.offsetHeight`, so that window
painted a sub-floor card: too little straight edge remained on the
vertical runs for the action-menu tab, which collapsed into the corner
arcs. Whether you saw it depended purely on whether the dimension publish
had landed, which is why it reproduced on one workflow and not another.
Floor all three: the host, the header row (so `items-center` centres the
title and type tag rather than pinning them to the top), and the border's
own `offsetHeight` read.
Also raise ACTION_MENU_CONTENT_READY_THRESHOLD to 0.9. At 0.8 the 24px
icon row was revealed while the swell had only reached 22.4px of its 28px
— shorter than the row it contains. Secondary to the above, but a real
overflow window on its own. The test now pins the ratio rather than the
constant.
Tag palette moves to fixed brand values (hex, not derived oklch) with two
inks — #F8F8F8 on dark fills, #1A1A1A on light. Tones are renamed to match
what they render. `green` (2.55:1) and `orange` (3.15:1) sit under WCAG AA
against their paired ink; both are deliberate brand decisions and are
documented in the component.
Deploy and Run take two new Button variants rather than className
overrides, so `tertiary` stays green everywhere else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* improvement(workflow): polish workflow canvas interactions
* fix(workflow): restyle loop drop target outline
* fix(workflow): shorten human block catalog label
* fix(workflow): canonicalize realtime edge handles
* code cleanup
* sizing fixes
* improvement(blocks): sentencify every block
* change tebse
* improvement(workflow): land notes UI and execution progress, consolidate duplicates
Ports the notes canvas editing and execution progress-indicator work, then
removes the parallel paths it arrived with so each concern has one owner.
Fixes found while consolidating:
- Note height was measured while the card was expanded to NOTE_EXPANDED_WIDTH,
where text re-wraps shorter, and published as the node's compact height. The
collapse then animated to a height never measured at the compact width.
- The edge pulse glow filter used the default objectBoundingBox units, so a
straight horizontal edge — what an auto-laid-out chain produces — resolved the
filter region to zero height and stopped the glow rendering entirely.
- A subflow's inner Start pill still read isNodeSelected while its border read
usesSelectedVisuals, so the two disagreed during execution.
- The Run/Stop button's disabled prop gated only Run while its handler cancelled
unconditionally, offering a Stop the cancel route answers with 403.
Consolidated:
- One note editor. The view's built-in textarea was unreachable in production
(the app always injects the markdown editor) and was kept alive only by tests
asserting against it; renderContentEditor is now required.
- onBlur/onCancel collapse to onEndEditing — content persists per keystroke, so
there was never a draft for a cancel path to discard.
- DEFAULT_NOTE_COLOR, the note height bounds, the note content reader and the
card width each had two or three definitions; each now has one.
- Removed with zero consumers: graphite/graphiteSubtle button variants,
data-subflow-selected, inputPlaceholderClassName, an effect that could never
fire, and getNoteColorOption's unreachable fallbacks.
Restores the role='status' announcement the progress rewrite dropped, and hardens
isNoteColor against inherited Object keys.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement(workflow): reuse the platform markdown editor in notes
Notes carried their own TipTap wiring — a second markdown editor that
reimplemented, more thinly, what `RichMarkdownField` already does for the skill
modal, skill fields and the deploy version description. It is now a ~20 line
skin: the Note supplies its type scale and per-colour selection tint, and the
field supplies the extension set, frontmatter held out-of-band, the round-trip
safety gate and its raw-source fallback, and markdown paste.
`RichMarkdownField` gains two additive props, both defaulting to today's
behaviour so the file editor is untouched: `surface` ('field' | 'bare') and
`proseClassName`. All three existing consumers pass an explicit `minHeight` and
no `surface`, so they take the original path unchanged.
Exiting the note editor moved to the card, because the editor's `/` and `@`
menus consume Escape to close themselves and ProseMirror checks `editorProps`
before plugin handlers — intercepting it inside the editor would have broken
both menus. The card now honours Escape only when nothing already consumed it,
which also let `onEndEditing` leave the injection contract.
The note editor is lazy now, matching every other consumer: it was pulling
TipTap and the full extension set into the canvas's initial chunk.
Also:
- One `areRunFromBlockDependenciesSatisfied`. The ActionBar, the canvas context
menu and the run-from-block handler each carried a byte-identical copy, and
the handler expressed the snapshot requirement differently, so the affordance
and the action could disagree. Each copy also re-scanned `edges` once per
incoming edge, on every ActionBar on the canvas.
- Reduced motion is one `usePrefersReducedMotion` in @sim/emcn rather than a
sixth ad-hoc `matchMedia`. The edge pulse now stops rendering instead of
hiding: `motion-reduce:hidden` is `display: none`, which left four SMIL
timelines running per edge.
- The pulse glow bleed covers the canvas minimum zoom. The strokes are
`non-scaling-stroke`, so the 6px tail spans 3/zoom user units — 30 at 0.1.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement(workflow): port canvas styling from workflow-updates
Ports the 14 styling commits your colleague added since the last sync, leaving
the ~68 staging PRs on that branch alone — those are platform/core work, not
this. Cherry-picked individually rather than merged so each conflict was small
enough to reason about.
What came in:
- Core block colors unify behind a two-level map: block type -> semantic role
-> accent, replacing the flat per-type table. Adds `purple` and `content`
tones to ChipTag, and a shared `WorkflowTypeIcon` that replaces the
hand-rolled ChipTag + accent lookup at each discovery surface.
- Native triggers take semantic colors; the deployments block moves to the
shared Rocket icon and drops its now-unused `iconColor`.
- Running-state polish: loader artwork and position, stop hover in dark mode,
the loader blended into the execution swell, and tooltips suppressed for
actions that are hidden mid-run.
- The toolbar drag preview clones the rendered icon container instead of
rebuilding a bgColor tile, so it matches what the canvas paints.
- The sidebar shows route-derived workspace identity instead of a skeleton
while the full record loads.
Conflict resolutions worth knowing:
- The running-loader artwork went through the shared `Loader` and back to the
custom SVG on their branch; the second commit is the intent, so that is what
landed — keeping our `role='status'` announcement layered on top.
- Two commits carried the lucide-react -> in-house icon migration along with
them. That migration is a staging change we have not taken, so our imports
stayed on lucide: adopting it in two files would leave the icon set split
across the app.
- `getMappedWorkflowTypeAccent` referenced a constant their refactor removed.
It had no consumers left once the search modal moved to `WorkflowTypeIcon`,
and their branch deletes it too, so it is gone here.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(workflow): make the subflow Start swell a real connection source
Dragging an edge into whitespace opens the add-block picker, but starting that
drag from a loop/parallel Start pill did nothing: the pill's border swell was
visual-only. Regular blocks and the container's own exit mint a draggable
cursor handle from their swell; the pill rendered only its invisible 14px
static strip, so grabbing the glowing affordance started no connection at all.
Everything downstream already worked and was nearly unreachable:
- the drop hit-test skips subflow containers, so a release inside the loop
opens the picker there (z 2000, above containers)
- handleToolbarDrop parents the new block into the container at the drop point
- it already carries the exact boundary rule for this source: a container
start handle only wires to a child of that container
The pill now runs the same cursor-handle machinery as the container view, with
one deliberate difference: its temporary handle carries the branch-cursor form
of the start id. The plain cursor id normalizes by block type — for a
container that is `loop-end-source`/`parallel-end-source`, the exit — so a
swell drag from Start would have persisted as an edge leaving the container.
The branch form passes `loop-start-source`/`parallel-start-source` through
normalization verbatim on both the picker and direct-connect paths; a test
pins that contract.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(notes): stop the field's prose classes recoloring bare-surface editing
Opening a note for editing shifted the text and turned it black: the
ProseMirror root unconditionally carried `rich-markdown-prose
rich-markdown-field-prose`, which pin the field's own ink and type ramp —
`--text-primary` at 15px/25px, then 14px/22px — overriding the card's
`text-current` at 14px/20px the moment the editor mounted.
`surface='bare'` means the host owns typography (the Note card mirrors its
rendered view via `proseClassName`), so on that surface the root now carries no
shared prose classes. The field surface is untouched. Edit mode inherits the
note colour's ink — including the caret — and sits on the same metrics as the
read view.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(workflow): give the in-flight connection line contrast inside containers
The drag line was drawn but camouflaged: its default stroke was the
resting-edge grey (#e0e0e0), which disappears against a loop body's opaque
`--surface-3` fill (~1.1:1) — so dragging an edge inside any container, nested
included, showed nothing. The z-order was never the problem; the connection
line layer already sits above every node.
The default token is now `--text-muted`, one value with contrast on every
canvas surface, still lighter than the `selected` variant so the variant
hierarchy holds. No per-surface special-casing.
Resting edges inside containers share the same camouflage (`--workflow-edge` on
`--surface-3`) — left alone deliberately: recoloring placed edges is a design
decision, not a bug fix.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(notes): match edit mode to the read view, and land the caret where clicked
Three defects, all from the read and edit views being built independently.
1. Blocks jumped up ~12px on entering edit mode. Streamdown wraps its output in
a container carrying `space-y-4` plus first/last margin resets, which outrank
the per-element margins in NOTE_COMPONENTS — so that wrapper, not those
margins, is what the read view actually paints. The editor had no equivalent.
The rhythm is now named (NOTE_MARKDOWN_FLOW), passed to Streamdown
explicitly so a dependency upgrade cannot move the read view out from under
the editor, and mirrored on the ProseMirror root. Tailwind's JIT only sees
literal strings so the mirror cannot be composed from the constant; a test
pins the two together instead, and fails if either side drifts.
2. The caret was barely visible: it inherited the note's 75%-opacity ink. The
palette owns per-colour chrome, so it now names the caret alongside the
selection tint.
3. The caret always landed at the document end. The read view sits under a
full-bleed overlay that must swallow the click to enter editing, so the
point never reached the editor and `autofocus: 'end'` was all that was left.
The view now forwards that point and the field resolves it through
`posAtCoords` on create — after the DOM is laid out, which `autofocus`
cannot wait for. Keyboard activation carries no point and still lands at the
end.
`autoFocusAt` is additive on the shared field and defaults to null, so the file
editor and the other three consumers are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(workflow): stop edges rendering behind top-level subflows
Containers are z-indexed by nesting depth, so a top-level subflow is 0. Edges
derived their z from their parent container — `+1`, or 0 with no parent — so a
root-level edge landed on exactly the same z as a root-level subflow. Equal
z-index falls back to DOM order, and React Flow paints the nodes layer after
the edges layer, so the container's opaque body won: any edge crossing a
top-level loop or parallel was drawn behind it, in-flight or persisted.
Edges now sit in their own band above the whole container scale and below
cards, keeping both the deeper-container-wins ordering and the rule that a line
always passes behind card chrome. This is why the edge became visible only once
a block was dropped: the new block is selected, and an edge inside the
container was already `containerZ + 1`, clear of the tie.
The in-flight connection line is declared in the same scale rather than
inheriting React Flow's stylesheet default of 1001, which is both below a
selected container child and outside the scale this file owns. Its stroke moves
to `--text-secondary`, the token the canvas already uses for an active edge —
the previous `--workflow-edge` grey is ~1.1:1 against a subflow body.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(notes): paste and drop images through the workspace-file pipeline
Traced the file editor's image path end to end and reused it verbatim rather
than minting a note-specific source: insertImages -> useUploadWorkspaceFile ->
POST /api/workspaces/{id}/files/presigned -> direct-to-S3 PUT -> workspace_files
row -> the editor persists the workspace-scoped
/api/workspaces/{id}/files/inline URL, which the serve route authorizes by
workspace membership and the embedded-image-ref machinery already recognizes
for share rewriting and referenced-by-doc tracking.
The shared field gains an optional `uploadImage(file) -> {url, alt} | null`
capability. With it, image paste/drop uploads sequentially and inserts each at
the evolving position, mirroring the file editor's flow, with a bail if the
editor unmounts mid-upload; without it, the existing swallow-guard on file
drops is unchanged, so the skill modal, skill fields and version-description
consumers behave exactly as before. The upload mutation owns its own toasts.
The note host wires the capability with folderId null, so note images land in
the workspace Files root — visible, manageable and deletable there like any
other upload. The note read view renders images through its Streamdown
components map with the card's own sizing.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement for notes, subflows
* fix(uploads): surface the server's message when a multipart upload is refused
A file over the 50MB direct-PUT threshold goes through multipart initiate, which
is where the storage quota is enforced — but the client threw away the response
body and reported `Failed to initiate multipart upload: Payload Too Large`. That
is the string the upload mutation puts in its toast, and it names neither which
limit was hit nor by how much, so the one place that answer surfaces didn't have
it.
It now prefers `errorBody.error` exactly as `getPresignedUploadInfo` already does
on the single-PUT path, and passes the body through as the error's details.
Control flow is unchanged: still throws, still `MULTIPART_ERROR`, and the
cloud-storage-absent branch above still claims its 400 first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(notes): restore GFM in the note read view
Streamdown's `remarkPlugins` prop REPLACES its default plugin list rather than
extending it, and remark-gfm is one of those defaults. The note passed
`[remarkBreaks]` — so the read view silently lost every GFM construct: task
lists, tables, strikethrough and autolinks.
The editor writes all of them (it has TaskList, TableKit and Strike), so a note
round-tripped through editing came back as raw source the moment editing closed:
`- [x] HELLO` rendered as a disc bullet followed by the literal text `[x] HELLO`.
`NOTE_COMPONENTS` has carried table/thead/tbody/tr/th/td entries this whole time
that could never fire.
Restoring the plugin is only half of it: remark-gfm marks a checklist
`contains-task-list` and emits a native checkbox, which under the note's generic
`ul` styling renders a checkbox sitting behind a disc bullet — the same defect
the editor had before the chrome/typography split. The read view now drops the
marker and indent for a task list, lays the row out as a flex line, and styles
the checkbox to match `.rich-markdown-nodes input[type="checkbox"]` declaration
for declaration, tick clip-path included, so the two views agree either side of a
click.
`remark-gfm` is now a declared dependency of the renderer package rather than one
borrowed transitively from streamdown.
Five tests cover the GFM surface — checked/unchecked boxes, no literal `[x]`, the
marker only dropped for checklists, tables, strikethrough — and four go red with
the plugin removed.
Checked the other three Streamdown call sites (Chat, the chat interface renderer,
the changelog): none override `remarkPlugins`, so none were affected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(canvas): drop dead markers and share the tile-brightness maths
Review pass over the branch against staging.
Dead code removed:
- `tileIconColorClass` in the renderer package — never called; only its
`isLightTileColor` sibling is.
- `data-connection-selector-search-frost`, `data-workflow-cursor-edge` and
`data-workflow-cursor-source-side` — written on three elements, read by no
stylesheet, selector or test.
- `CHIP_TARGET_SELECTOR_TYPES`, `MAX_CHIPS` and `chipPriority` were exported from
`canvas-rows.ts` but only used inside it.
Consolidated the one real divergence: the renderer package carried a hand-copied
mirror of the app's perceived-brightness maths, because it may not import app
code. The copy had already drifted — it dropped the `white`/`black` keyword
handling, so a block shipping `bgColor: 'white'` would render a white
`currentColor` icon on a white tile on the canvas while every other surface drew
it black. No block ships one today, which is exactly why nothing caught it. The
function now lives in `@sim/utils/color` and both sides import it; only the
0.75 threshold stays local to each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(canvas): share the z-scale, fix the preview's edge layering, and re-home strays
The preview canvas carried its own z numbers and had the collision the editor
canvas was fixed for: containers at nesting depth, top-level cards at an implicit
0, and edges at 0/5/10 by execution status — so a default edge tied with a
top-level subflow and painted behind it, while a success edge painted over
unselected cards.
The scale now lives once, in `@sim/workflow-renderer/canvas-layers`, and both
canvases read it. The preview keeps its status ordering, expressed inside the
shared edge band rather than as a second set of magic numbers.
Placement and duplication:
- `perceivedBrightness` moved to `@sim/utils/color`, with its unit test, and its
consumers import it directly. It had been re-exported through
`lib/colors/brightness.ts`, and the renderer package kept a hand-copy.
- `filterAcyclicEdges`/`wouldCreateCycle` were pass-through wrappers in the
workflow store's utils over the real implementations in `@sim/workflow-types`.
Deleted; the three consumers import the source.
- `lib/ui/glass-surface.ts` was a one-constant, one-consumer app-wide module, and
its consumer then aliased it a second time. Collapsed into the navbar shell.
- `nested-subflow-node` was set on nested container nodes in both canvases with no
stylesheet, selector or test behind it.
`packages/workflow-renderer` now has its own vitest config, so the four mount
tests for its components live with the components instead of in
`apps/sim/lib/workflows/**`. That immediately earned its keep: `apps/sim`
excludes test files from type-check, and once these were checked, tsc found three
`SubflowNodeView` renders being handed a `renderContentEditor` prop it does not
accept — a copy-paste from the note cases that had been silently ignored.
Verified: type-check 23/23, 21,143 app tests + 49 renderer + 147 utils, biome
clean, all 23 audits pass (`check:bare-icons` imported the moved helper and was
repointed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix field noun bug + notes
* fix(notes): let a note service the canvas actions the panel editor cannot
The panel editor clears any note put in front of it and renders nothing, but the
block menu still routed Rename and Open Editor through it.
Rename latched the editor's rename state onto the note — `handleStartRename`
reads the store directly, so it saw the id the menu had just set — and nothing
reset it when the clear ran. `handleSaveRename` writes to `renamingBlockIdRef`,
so the header went on showing a rename field over whatever was selected next and
saved that name to the note. Open Editor was a plain no-op that opened an empty
pane.
Rename now goes to the card, which expands and opens its own title — the same
menu-to-card routing Add Image already used, so both events now live in one
`lib/workflows/notes/canvas-requests.ts` and `add-image.ts` keeps only its
markdown concern. Open Editor is hidden for notes.
The panel editor also drops any rename whose block stops being the selected one.
That is belt-and-braces for notes now, but it closes the same hole for ordinary
blocks, where only the input's blur ended a rename and blur only fires if it held
focus. A rename interrupted that way is now discarded rather than left pending.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(canvas): author sentences for Snowflake and Mintlify, repoint Instagram's
Staging's two new integrations shipped without a `canvasPresentation`, so their
39 operations painted the field rows the rest of the canvas has stopped using.
The Instagram break is the more interesting one: staging renamed the insight
metrics subblock `metrics` -> `insightMetrics` while this branch was adding
sentences that named `metrics`. Both hunks merged cleanly — the union check
reports the file as an exact union — and the result was two clauses pointing at
a field that no longer exists, which resolves to nothing with no error and no
log. Only `check:canvas-sentences` sees it.
Two Snowflake sentences say something the block does not do, so they anchor
elsewhere: `taskName` filters `list_task_runs`/`get_task_run` rather than keying
them, and `table` filters `introspect_schema` — blank means "every one", not "not
filled in yet", and a core chip would have claimed otherwise.
Coverage is back to 4727/4727 operations across 321/321 blocks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(canvas): keep the block-type tag naming its type after a rename
The header tag dropped its label whenever the block's title already said the
same word, so the same block read two ways depending on nothing the user did
deliberately: a freshly dropped Wait showed a bare icon, and its second copy —
auto-named "Wait 2" — showed "Wait". The tag looked like a badge that appeared
on rename rather than a fixed part of the header.
It now always names the type, which is what loop and parallel containers already
do with their own tag, so every card on the canvas reads the same way.
`blockName` was only ever read for that comparison, so the prop is gone rather
than left behind for a future reader to wonder about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): compare edge handles by port, not by spelling
Two places answer "does this need redeploying?" and they load their sides
differently. The client diffs the live store against `/api/workflows/[id]/deployed`;
the server diffs the normalized tables against the version's raw jsonb. Only
some of those paths run handles through `loadWorkflowFromNormalizedTables`, so a
snapshot holding a side-anchored id (`source-right`) met a canonical one
(`source`) on the other side and the set comparison read it as every edge being
removed and re-added.
Each answer therefore differed, and they arrive on separate query timelines: the
button reads the client's, the modal badge reads the server's, so the state
flipped between Live and "Update deployment" with whichever query landed last
until both settled.
`normalizeEdge` now canonicalizes both handles, so the comparison cannot tell
two spellings of one port apart no matter how its inputs were loaded. The
existing normalization in `materializeDeploymentState` stays — that path also
feeds React Flow, which needs the handle it mounts to match.
The preview's error port had the mirror problem: it rendered for every
non-trigger block regardless of `errorEnabled`, so a card with no error row grew
a red knob anyway. It now gates the way the editor canvas does, keeping the port
mounted when an error edge already leaves it so React Flow cannot drop that edge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): stop counting the error flag twice in change detection
`errorEnabled` has two homes. It persists inside the block's `data` jsonb — the
realtime server `jsonb_set`s it there, and load mirrors it back onto the block as
a field — so it reached the diff twice, and only some paths populate the copy.
`setBlockErrorEnabled` writes the mirror alone, so right after toggling the port
the live block said `errorEnabled: true` with `data.errorEnabled: false`, while
the snapshot the deploy had just taken from the tables said true in both. The
diff read the stale `data` and reported the workflow as changed the instant it
finished deploying — then a state refetch rehydrated the block and it agreed
again. That is the flip between Live and "Update deployment": the button and the
modal read two different queries, so each landing swapped the answer. A block
created in-session had the same shape from the other side, its `data` carrying no
key at all against a persisted `false`.
Excluded from `normalizeBlockData` alongside the other fields that are duplicated
out of the block's own state. The block field is still compared on its own, with
`!!`, so absent and `false` agree and turning the flag on is still a change.
Fixing the store to write both homes was the other option and is not taken:
nothing reads the in-memory `data.errorEnabled` (save and load both let the block
field win), so it would add a second copy that only the diff could see — which is
the shape of this bug, not its fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(blocks): give the error-output flag a column instead of two homes
`errorEnabled` had no column, so it persisted inside the block's `data` jsonb
and was mirrored onto the block as a field on load. Every writer had to route
its `data` through `withPersistedErrorEnabled` or silently drop the toggle, the
realtime op `jsonb_set`, and change detection saw the same value twice — which
is what made the deploy badge flip between Live and "Update deployment" after
toggling the port.
Its siblings — `enabled`, `horizontal_handles`, `advanced_mode`, `trigger_mode`,
`locked` — are all boolean columns; `data` is for React Flow and subflow state.
The flag belongs with them, so it now has `error_enabled` and one home. The
shuttle helper, its `BlockData` mirror, the store's fallback read, and the
comparison exclusion the duplication forced are all gone.
Backwards compatibility, since released versions draw the error port with no
toggle in front of it: a block already wired to an error edge HAS the output on,
because there was no other way to draw that edge. That rule is now stated in
three places and none may be narrowed to read the flag alone —
- the migration backfills `error_enabled` from the edges, so live rows are true
before any new code reads them;
- `materializeDeploymentState` derives it for a version's frozen jsonb, which the
migration cannot reach — otherwise every workflow deployed before the toggle
would ask to be redeployed once;
- `workflow-block.tsx` keeps it at render time for states that reach the canvas
through neither (imports, copilot edits), where unmounting the port would make
React Flow drop the edge leaving it.
The migration also moves any `data.errorEnabled` a developer created on this
branch onto the column and strips the key; both statements match zero rows in
production, where it never shipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(canvas): realign the Snowflake and Dynatrace sentences with staging's blocks
Both breaks are the class the union check cannot see: separate hunks of the same
file merged cleanly, and the result names fields that no longer exist. A sentence
that does resolves to nothing, with no throw and no log.
Snowflake's rewrite (#6474) moved database, schema, table, warehouse and
procedure onto canonical selector pairs, so seven clauses anchored on ids that
are gone. Each now names both members of its pair, which is also what keeps the
card readable for someone working in advanced mode. Its nine new operations have
sentences.
Dynatrace (#6463) scoped the mute reason to the operations that mute, because
unmuting accepts exactly one — so the two unmute sentences were asking for a
field their card no longer shows. They drop the clause.
Coverage is 4736/4736 operations across 321/321 blocks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): drop the block-data error flag from the workflow contract
Left behind by the consolidation: the flag no longer lives in `data`, and a
schema that still declares it there invites the mirror back through the wire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): compare edge handles by port, so a falsy one cannot read as changed
`loadWorkflowFromNormalizedTables` now runs handles through the canonicalizer,
which falsy-coalesces — so an edge persisted with `sourceHandle: ''` loads as no
handle at all. The server diffs that against the deployment version's raw jsonb,
which still has `''`, and the set comparison reads one edge as removed and
another added. Every workflow holding such an edge would ask to be redeployed
the moment this ships, for nothing. Two write paths use `?? null` rather than
`|| null`, so `''` is reachable.
Canonicalized inside `normalizeEdge` rather than at either call site: the two
sides are loaded by different paths and only some of them normalize, so the
comparison has to be unable to tell two spellings of one port apart however its
inputs arrived.
This is the change reverted in
|
||
|
Waleed
|
e5d2f7d80a |
fix(realtime): stop read-only members persisting block positions (#6174)
* fix(realtime): stop read-only members persisting block positions The socket operation ACL granted the `read` role block.updatePosition and blocks.batchUpdatePositions on the premise that they were ephemeral cursor sync. They are not: both are followed by persistWorkflowOperation, which UPDATEs workflow_blocks.positionX/positionY and bumps workflow.updatedAt. A member holding only `read` on a workspace could therefore permanently rewrite the coordinates of every block of every workflow in it — a write across the read/write boundary. Live cursors ride their own `cursor-update` event, and the smooth-drag broadcast is the UNCOMMITTED position path, which returns before persisting and never consults the role table — so the read role needs no grant at all. * test(realtime): assert the role ACL against production, not a fixture The shared ROLE_ALLOWED_OPERATIONS fixture still listed the two position operations for the read role, and three tests compared that fixture against itself — so they certified whatever it said, including the grants this PR removes. They now assert checkRolePermission over the protocol's complete operation list (the fixture's copy omits subblock/variable/admin-only ops), plus one test that pins the fixture to the production ACL so the two cannot drift apart again. |
||
|
Waleed
|
5686b7bb83 |
fix(realtime): enforce room access continuously, not only at join (#6170)
* fix(realtime): enforce room access continuously, not only at join File-doc, table, and workspace-list rooms authorized once at JOIN and never again, so a member whose workspace access was revoked or downgraded kept live collaborative write access — including durable Yjs document writes — for the whole lifetime of an already-open socket. The access-revalidation sweep explicitly skipped every non-workflow room. - sweep every room type, authorizing each against its own resource - share one membership policy (ROOM_MEMBERSHIP_ACTIONS) between the join check and the sweep, so a file-doc room keeps requiring write in both - gate file-doc document frames and table cell selections on the cached permission, evicting on a confirmed loss of access - re-check the cached decision before a join commits, so a join that authorized just before a revocation cannot re-enter the room - never let a join's own cache write clobber a revocation recorded mid-flight - surface room-access-revoked to clients; the file-doc editor falls back to read-only instead of accepting keystrokes that go nowhere * refactor(realtime): one shared eviction path for revoked room access The sweep and both per-frame gates each open-coded emit + leave + local-state cleanup. Route them all through evictSocketFromRoom so they cannot diverge on what eviction means; workflow keeps its historical access-revoked payload. * fix(realtime): re-check access before workspace-list room joins too The workspace-files / workspace-tables joins committed straight from their authorize result, so a join that authorized just before a revocation could put the socket back in a room the sweep had already evicted it from. Mirrors the guard the file-doc and table joins already had. * fix(realtime): order role-cache writes by read start, not write time Two authorizations can start in one order and finish in the other, so the decision written last can come from the older read. A join that authorized before a revocation but returned after the sweep's denial would bury it, handing the socket another full cache TTL of access. Every writer now takes a monotonic ticket before it queries and yields only to a later-started read. * chore(realtime): drop the test-only unguarded role-cache writer Tests can express the same setup with commitRoomPermission + a read ticket, so the cache has exactly one write path and no export without a production caller. * fix(realtime): keep handler-initiated table eviction retryable Evicting leaves the Socket.IO room synchronously, which is also how the sweep discovers work — so a presence removal failing in the per-frame path could never be retried and left a ghost collaborator until disconnect. Failed (or unconfirmed) removals now hand off to the sweep's existing cleanup lane instead of a second retry loop. * fix(realtime): re-resolve access at join commit instead of peeking the cache The pre-commit recheck peeked the role cache, which reports an EXPIRED entry as unknown and fails open — so a join stalled longer than the cache TTL could re-enter a room the sweep had already evicted it from, including a file-doc room where the next cold-cache frame is accepted as a durable write. All three joins now re-resolve the way the workflow join always has; it is normally a cache hit, since the join's own authorize just warmed it. * fix(realtime): keep the join generation guard after the access re-check The access re-resolve added in the previous commit sat AFTER the generation / superseded guard in the table and workspace-list joins, so a leave or a newer join landing during that await no longer cancelled the stale join — it would go on to leave the room the client had switched to and commit the abandoned one. The guard is now the last thing before the commit in all three handlers, as it already was for file-doc and workflow. * test(realtime): use the shared sleep helper in the new join tests check:utils bans the inline new Promise(setTimeout) form; the two stalled-join tests were the only new offenders. * fix(realtime): leave the prior table room only once the join is certain A table switch left the previous room before the access re-check ran, so a denial there aborted the join and left the client in no table room at all — silently dropped from one it may still be allowed to occupy. The leave now happens after the re-check, matching the file-doc and workspace-list joins. * fix(realtime): close the table join window between re-check and commit Moving the prior-room leave after the access re-check left Redis awaits between that check and socket.join, and superseded() only watches the join generation — so a sweep revocation landing in that window could still put a revoked socket back in the room. A synchronous cache peek immediately before the commit closes it without reintroducing the await; the authoritative resolve moments earlier wrote a fresh entry, so a differing read IS the revocation being guarded. |
||
|
Waleed
|
06506bbd38 |
improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path (#6149)
* improvement(files): cache stream binding meta, tighten file cache-control, prune dead persist path - Cache the agent-stream ProseMirror↔Yjs binding metadata per session and reuse it across streamed frames instead of rebuilding it (O(doc)) every frame; matches how y-tiptap's own binding maintains the mapping in place. Safe because the shadow doc only ever sees the agent's own reconciles. - Default createFileResponse to a private, no-cache policy so auth-gated bytes are never stored in a shared cache; genuinely-public serve routes opt into public caching explicitly. - Serve content-addressed (key=) embedded images with an immutable private cache to avoid re-downloading them on every doc re-open; fileId= embeds and public shares keep revalidating (the underlying key can change / a share can be revoked). - Drop the dead conflict.version field from the collab-doc persist result and remove the wasted getWorkspaceFile re-read on the conflict path (the relay treats missing and conflict identically and never read version). - Decorate-sort the files/folders lists (compute each sort key once) and index the move-menu subtree build (O(N) vs O(N^2)); ordering is preserved exactly. * fix(files): serve inline images with no-cache so deletion/authorization is enforced per request Embedded images are authenticated content whose backing file can be deleted or have its access revoked at any time. Both key= and fileId= embeds serve private, no-cache, must-revalidate, so every request re-runs the server-side deletion/authorization check instead of serving a possibly-stale image from cache. |
||
|
Waleedandmzxchandra
|
10bfb5d139 |
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on |
||
|
Waleed
|
e98715da01 |
fix(realtime): noindex the socket server's 404 responses (#6129)
* fix(realtime): noindex the socket server's 404 responses The sockets.* hostnames are served by this server and return a plain JSON 404, which Google Search Console reports as crawl errors. Mark unmatched routes noindex so crawlers drop the hostnames instead of retrying them. * fix(realtime): noindex every response, not just the 404 Setting the header only on the 404 fallback covered the one response that crawlers already drop on status code alone, while /health — the sole route returning 200 with a body, and so the only indexable surface on the socket hostnames — stayed uncovered, with a test pinning it that way. Set it once on the handler instead. Node merges setHeader values into writeHead and no branch sets X-Robots-Tag, so it reaches every response. |
||
|
Waleed
|
02311cafb9 |
perf(db): stop scanning every workspace file on workflow archive, cover the billing period aggregate (#6015)
* perf(files): stop loading every workspace file to archive a workflow's backing files `cleanupWorkflowAliasBacking` runs on every workflow delete. It loaded every `context='workspace'` row for the workspace — active and soft-deleted, all columns — then discarded >99% of them in JS to find the handful of `.changelogs/<workflowId>.md` and `.plans/<workflowId>/**` rows it owns. In production this was the single worst query at 37.83% of total database runtime: p50 6s, p95 9s, max 13.1s, 641 calls/day, 34.1M rows read. It is index-served, so the cost is not a missing index — it is 59,061 rows and ~630MB of cold buffers read to archive a few files. A file's `folderPath` is derived solely from its `folderId`, so matching on folder membership is equivalent to the path comparison it replaces. The load and JS filter become one targeted UPDATE keyed on a handful of folder ids. Folders that are soft-deleted are still included when resolving which files a workflow owns: path resolution ignores `deletedAt`, so a live file parented to an archived folder previously matched and must continue to. Also: - Add `getWorkspaceShares`, replacing an id-list `IN` clause that grew with the file count (59,061 elements on the worst workspace) with one indexed lookup on `workspace_id`. Callers read the map by id, so a superset is equivalent. - Drop `all` from the client-reachable file scope enum. It drops the `deleted_at` predicate and so cannot use the partial index serving the other two. No client requests it; server callers reach that scope directly. - Set `fetch_types: false` on the app and realtime pools. postgres.js otherwise runs a blocking `pg_catalog.pg_type` roundtrip before each new connection's first query — 95,722 of them per day. It builds array parsers only; Drizzle already parses this schema's two `text[]` columns itself. * perf(nav): add route loading boundaries and cover the billing period aggregate Dynamic routes prefetch only down to the nearest loading boundary, and the server stops prefetching at the first one. With no loading.tsx anywhere in the workspace tree and a dynamic layout, Link prefetch was yielding almost nothing and every navigation waited on a full server round trip with no feedback. Add loading.tsx only where the fallback is provably what renders today, so perceived speed improves without changing what users see: - home, chat/[chatId]: reuse HomeFallback, already each page's own Suspense fallback. - integrations, skills: reuse the tab-header chrome, byte-identical to each page's own Suspense fallback. Deliberately not added to workspace root, settings, or w, whose pages are redirect-only or already self-fallbacking — a boundary there would paint a skeleton that does not match the destination. Billing: - Add usage_log_billing_period_cost_idx, trailing the remaining predicate columns and `cost` so the period aggregates resolve index-only. Confirmed against production: the aggregate currently runs as an Index Scan touching 478,559 buffers because `cost` is absent from the existing index. That index is superseded but left in place; dropping it is a separate migration so a planner regression costs nothing to revert. - Collapse the two aggregates behind /api/billing into one scan using SUM(...) FILTER. Besides halving the work on a nav-path query, it removes a latent inconsistency: as separate statements the two sums could observe different snapshots, making the copilot subset exceed the total. Caching was considered and rejected for the usage aggregate. Its callers include usage enforcement, threshold billing, and overage calculation, where a stale-low read permits overspend and a stale-high read double-charges. * test(files): cover cleanupWorkflowAliasBacking and correct a stale share mock cleanupWorkflowAliasBacking had no test coverage, and the rewrite that replaced its load-everything-then-filter body rests on a subtle equivalence: a file's folderPath is derived solely from its folderId, and path resolution ignores deletedAt. The second half is the easy part to get wrong — a live file parented to an archived folder still resolves to a backing path and must still be archived, so folders are filtered by deletedAt only when choosing which folders to archive, never when deciding which files the workflow owns. These tests pin that distinction: they fail if the archived-folder case is dropped from file ownership, and separately assert that archived folders stay out of the folder update and that unrelated workflows are never touched. Also point the workspace files route test at getWorkspaceShares. Its mock still named getSharesForResources, which the route no longer imports. The suite passed regardless because all three cases exercise the upload path, so the GET listing has no coverage — but the stale name would have handed the first GET test an undefined function. * fix(perf): drop the loading boundaries and correct the usage_log index order Adversarial review found real regressions in both. Route loading boundaries — all four removed: The premise in their TSDoc was wrong. Each page's existing `<Suspense>` exists so nuqs can prerender; `useSearchParams` only suspends during SSR, so on a client navigation those fallbacks never painted. Hoisting them to loading.tsx did not "show the same frame" — it made a previously invisible blank frame visible. For chat, Next keys the Suspense boundary by cache key, so a chatId change mounts a fresh suspended boundary and always commits its fallback. Switching chats would have gone from "previous chat stays on screen" to a blank surface for the whole RSC round trip, on the highest-frequency navigation in the product. Partial prefetch only warms the fallback, so no latency was saved to offset it. The integrations and skills fallbacks were correct for their own pages but also became the fallback for four detail routes that render different chrome, inserting a wrong intermediate frame. Fixing that needs per-child boundaries and new skeleton UI that cannot be verified without a browser, for pages whose only server work is `await params`. Not worth it. Every remaining loading.tsx in this app paints real chrome. A blank one was against the grain, and the measurable win was zero. usage_log index: Column order was wrong. The daily-refresh rollup filters entity type, id and period_start but NOT period_end, so putting period_end fourth ended the usable prefix at column three and left user_id and created_at as in-index filters rather than scan boundaries — turning a ~2.2k-entry bitmap scan into a ~266k-entry scan. Verified in production that period_start functionally determines period_end (13,612 groups, zero with more than one end), so the slot bought no selectivity. user_id and created_at now follow the shared prefix; period_end rides as payload. Also corrected the claim that this supersedes usage_log_billing_entity_period_idx. That index deduplicates to 17 MB across 1.42M entries because it has no high-cardinality key column, which is what keeps prefix-only bitmap scans cheap. It is retained deliberately, not pending a drop. Harden cleanupWorkflowAliasBacking: gate the UPDATE on the ownership filter list itself. `and()` and `or()` both drop undefined, so a clause that resolved to nothing would have left a WHERE of workspace + context + not-deleted and archived every file in the workspace. Tests now assert no UPDATE is issued when the workflow owns nothing, plus the filename and context predicates. Note fetch_types also disables array serializers, not just parsers; documented. * docs(db): correct the fetch_types constraint note after differential testing Ran both settings against a real Postgres with the schema's actual text[] columns. Drizzle-typed selects and .returning() are byte-identical either way; only a raw db.execute projecting an array column differs, yielding the wire form. The previous note also claimed a raw JS-array bind fails because the serializers come from the same catalog fetch. It does fail — but under both settings, because Drizzle expands an array into a row constructor before postgres.js ever sees it. That is unrelated to this flag, so the claim is removed rather than left implying a constraint this change introduces. * chore(db): generate the drizzle snapshot for the usage_log index migration 0271 was hand-written, so drizzle-kit's state never learned about the new index — the next `generate` would have re-emitted it as a fresh migration against an already-migrated database. Ran `drizzle-kit generate` to produce the snapshot, then restored the CONCURRENTLY form: drizzle emits a plain CREATE INDEX, which takes an ACCESS EXCLUSIVE lock and would block writes on a 4M-row table for the duration of the build. The generated column order matched the hand-written SQL exactly, which also confirms schema.ts and the migration agree. `generate` is now a no-op, and check:migrations still passes. |
||
|
Vikhyath MondretiandClaude Fable 5
|
f43b52c569 |
fix(realtime): evict revoked collaborators from live workflow rooms (#5917)
* fix(realtime): evict revoked collaborators from live workflow rooms via periodic read-access re-validation * fix * fix(realtime): close join/eviction race and make sweep cleanups independent Re-authorize immediately before socket.join so an in-flight join cannot reverse a sweep eviction, and run the sweep's best-effort cleanups independently so a room-state failure cannot skip the presence broadcast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): single-flight role resolution and retry failed eviction cleanup Coalesce concurrent role resolutions per (user, workflow) so a slow stale read can never overwrite a recorded revocation, and defer failed eviction room-state cleanups into a per-sweep retry queue so collaborators are not left with a stale presence entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): scope room removal to the target workflow and detect swallowed cleanup failures Honor the workflowIdHint as the target room in both room managers so removing a stale room cannot clobber the mapping of a room the socket has since moved to, and confirm eviction cleanup via the returned workflowId plus an unswallowed mapping read so Redis failures actually defer into the retry queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): treat unconfirmed removals as failures and refresh role cache on fresh verify Treat any null removal result as a failed cleanup (the sweep always passes the target room, so null only means failure — including with expired mapping keys), move the same-room rejoin guard to a synchronous check immediately before the removal, and record verifyWorkflowAccess's fresh decision into the role cache so a re-granted user is not blocked by a stale cached revocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): prefer a mid-flight recorded role decision over the in-flight query result If a fresh authoritative read (join-time verify) records a decision while a single-flighted resolution's query is in flight, keep the recorded decision instead of overwriting it with the potentially stale result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): isolate the security scan from the Redis cleanup lane Run the revocation scan (local sockets + DB only) and the best-effort room-state cleanup as independently-guarded lanes so a hanging Redis command can stall only presence cleanup, never revocation enforcement. Evictions now enqueue cleanup instead of awaiting it, and the scan no longer reads presence for a fallback role. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): bound authorization waits in the revocation scan Race each socket's authorization check against a per-socket timeout and cap the whole pass with a budget below the sweep interval, so a hanging DB query skips that socket for the pass (never evicting on uncertainty) instead of wedging the scan lane and starving subsequent ticks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(realtime): round-robin the revocation scan so hung checks cannot starve later sockets Resume each scan pass after the last target the previous pass processed, so a fixed prefix of hanging authorization checks can never repeatedly consume the pass budget and leave sockets behind it unexamined. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d24bc7eccb |
feat(agent-stream): thinking and tool streaming (#5671)
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
fac8ec0c51 |
fix(workflow-edges): enforce edge/block validation server-side, not just client-side (#5571)
* fix(workflow-edges): enforce edge/block validation server-side, not just client-side
Dragging a connection that creates a cycle correctly refused to render
client-side, but the cyclic edge was still queued for realtime persistence
and written to the DB, so it reappeared after refresh. Root cause: cycle
detection (and several other edge/block rules) only lived in the client
Zustand store and was never enforced by the realtime persistence layer,
which is the actual source of truth on reload.
- Move wouldCreateCycle, edge scope-boundary, annotation-only-block,
duplicate-edge, and block-name-conflict checks into @sim/workflow-types
so the client store, the collaborative queueing layer, and
apps/realtime's DB write path all share one implementation
- Wire these into apps/realtime/database/operations.ts's edge-add and
block-rename handlers as the authoritative gate
- Client-side behavior is unchanged (same call sites, same error messages,
same rule ordering) — verified via existing + new test coverage
* fix(workflow-edges): address review findings on realtime edge validation
- Select triggerMode when fetching blocks for edge-add validation —
isKnownWorkflowTriggerBlock checked block.triggerMode but the column
was never fetched from the DB, so trigger-mode blocks could still
receive an incoming edge (Cursor Bugbot)
- Make filterUniqueWorkflowEdges incremental, so two duplicate edges
within the same BATCH_ADD_EDGES payload are also deduped instead of
both surviving (Greptile)
* fix(workflow-edges): normalize empty-string handles in duplicate-edge check
filterUniqueWorkflowEdges compared handles with ??, so a `sourceHandle: ''`
edge wasn't recognized as a duplicate of an existing null-handle edge —
even though both get persisted as the same null value at insert time
(edge.sourceHandle || null). Falsy-coalesce in the comparison so '' and
null/undefined are treated as the same "no handle" state everywhere.
(Greptile)
* improvement(workflow-edges): dedup realtime edge-add validation, reuse block-name-conflict helper
/simplify pass on the already-merged-quality PR before sign-off:
- Extract filterEdgesForPersist in apps/realtime/src/database/operations.ts:
the single-edge ADD and batch BATCH_ADD_EDGES handlers hand-inlined the
same six-step validation pipeline (missing block, protected target,
annotation-only, trigger-target, scope boundary, duplicate, cycle) and
each independently re-fetched blocksById/existingEdgesForCycleCheck. Two
copies of one rule in the same file was exactly the drift risk this PR
otherwise closes across client/server. One shared helper now backs both.
Net -63 lines despite the new shared function.
- Fix a real bug this surfaced: droppedCounts was keyed by the free-text,
block-id-bearing scope-boundary message, so it could never aggregate
across edges/runs. Now keyed by a stable 'scope boundary' reason.
- use-collaborative-workflow.ts's collaborativeUpdateBlockName still
hand-rolled the empty/reserved/duplicate block-name-conflict checks this
PR centralized as getWorkflowBlockNameConflict (already adopted by
store.ts). Switched it to the shared helper, which also fixes a latent
check-order mismatch between the two (this pre-check ran reserved before
duplicate; the store's real gate — after this PR's own store.ts change —
runs duplicate before reserved, so they could disagree on which toast a
name that was both reserved and duplicate would surface).
* docs(workflow-edges): document the per-workflow write-serialization invariant
No behavior change. Address Greptile P1 on filterEdgesForPersist ('concurrent
duplicate writes can persist without a per-workflow write guard') by
documenting, at the actual mechanism, why the concern doesn't apply here:
persistWorkflowOperation's leading 'UPDATE workflow SET updatedAt ... WHERE
id = workflowId' already takes a row lock that serializes every operation
(including edge adds) for a given workflowId — a second concurrent call
blocks on that UPDATE until the first transaction commits or rolls back, so
the validate-then-insert sequence in filterEdgesForPersist can never
interleave across two writers on the same workflow.
Verified empirically, not just by reading: ran two concurrent transactions
against a throwaway local Postgres against the exact statement shape used
here (UPDATE the parent row, sleep to simulate the read/validate window,
insert, commit). The second transaction's UPDATE blocked for the full
duration of the first's transaction and only proceeded once the first
committed — confirming the row lock, not any application-level guard,
already provides the serialization Greptile flagged as missing.
Added a comment at the lock site (not a second, redundant advisory lock)
so a future change can't silently break this invariant by making the
UPDATE conditional/skippable as a perceived no-op optimization.
* fix(workflow-edges): validate edges in BATCH_ADD_BLOCKS before persisting
Real gap Cursor's PR summary flagged ('BATCH_ADD_BLOCKS edge inserts... not
fully covered by the new server pipeline'), confirmed by reading the code:
this handler bulk-inserted the edges from a block-paste/duplicate/import
payload directly into workflowEdges with zero validation — no missing-block,
protected-target, annotation-only, trigger-target, scope-boundary,
duplicate, or cycle check. A client sending edges through this operation
instead of BATCH_ADD_EDGES could bypass every rule this PR otherwise
enforces server-side, exactly the class of gap the PR exists to close.
Routes it through the same filterEdgesForPersist used by the other two
edge-add handlers. Runs after the block insert in this same handler, so the
shared helper's blocksById lookup also sees the blocks this same batch just
inserted (a transaction observes its own prior writes).
|
||
|
Waleed
|
b117758eaa |
fix(agent): scope nested tool canonical-mode overrides by instance, not type (#5534)
Two tool entries of the same type inside an Agent block's tool-input array
(e.g. two Table tools) shared a single canonical-mode override keyed by
${toolType}:${canonicalId}, so switching basic/advanced mode on one field
silently switched it on every other instance of the same tool type -
including at execution time, where the wrong basic/advanced value could be
resolved for the second tool.
Rescope the override key to the tool's position in its tool-input array
(${toolIndex}:${canonicalId}) instead of its type, and thread that index
through every consumer: the editor (read + write), execution
(agent-handler/providers), search-index, and fork/promote remapping.
|
||
|
Waleed
|
5db62b8bdc |
feat(observability): extend audit log, PostHog, and storage metering coverage (#5269)
* feat(observability): extend audit log, PostHog, and storage metering coverage Instruments previously-uncaptured resources and actions across the three observability layers (audit log, PostHog product analytics, usage metering): - Exfiltration audit: file downloads (workspace/public-share/v1), table, workflow, and workspace exports. - Credential access audited at the token-issuance boundary (success-only). - Full revenue trail: invoice paid/failed, overage billed, disputes, credit fulfillment, subscription lifecycle, plan/seat changes. - Session/login lifecycle audit via Better Auth hooks (login, blocked sign-in, logout, session revoke, account delete). - v1/admin programmatic surface and copilot tool handlers instrumented. - Dead audit/PostHog constants wired (lock/unlock, table, custom tool, etc.). - Storage metering extended to KB documents and copilot files. - PostHog person identify + workspace/organization group hygiene. Audit package hardened to null the actor FK for system actors (admin-api) and adds an awaitable recordAuditNow for pre-delete hooks. All instrumentation is fire-and-forget and never blocks or breaks the primary operation. * fix(observability): audit file/credential egress only on success Address Cursor Bugbot review: - GET OAuth token: emit CREDENTIAL_ACCESSED/credential_used after refreshTokenIfNeeded succeeds (matches POST), not before. - File export: audit on each actual success exit via a shared helper — including the non-markdown serve redirect (previously unaudited) and after the zip is generated (previously before asset fetch). * fix(audit): record null actor for anonymous public-share downloads Address Greptile P1: setting actorId to the file owner made every anonymous external download read as a self-download, undermining the exfiltration trail. recordAudit now accepts a null actor; the public content/inline routes record actorId=null with the owner in metadata.sharedByUserId and rely on ip/user-agent for the forensic trail. The misleading owner-attributed file_downloaded PostHog event is dropped on these anonymous paths. * fix(observability): audit admin exports after zip; tidy comments - Admin workflow/workspace ZIP exports now audit only after the archive is built (via a local helper), so a zip/build failure no longer logs an export. - Remove redundant 'success-only placement' inline comments and tighten the remaining design-rationale notes (export helper now TSDoc). * fix(billing): complete the chargeback + overage financial trail Address Cursor review: - handleDisputeClosed now records CHARGE_DISPUTE_CLOSED for every closed dispute (won/lost/warning_closed), unblocking only on favorable outcomes. dispute.status in the metadata distinguishes the outcome, so lost chargebacks are no longer missing from the trail. - Threshold overage now emits OVERAGE_BILLED + overage_billed even when credits fully cover the overage (settledVia: 'credits' vs 'stripe'), so credit-settled overages are audited instead of silently returning null. * fix(storage): decrement copilot quota before deleting metadata Address Greptile P1: deleting the metadata row before the decrement meant a decrement failure left the quota permanently inflated with no record to retry from. Decrement first; only remove the metadata row once it succeeds. * fix(observability): atomic copilot storage release; signup-blocked action - Copilot file delete now releases storage via a single transaction (releaseDeletedFileStorage): the soft-delete is the idempotency claim and the decrement shares the transaction, so neither a partial failure (inflated counter) nor a retry (double-decrement) can desync the quota. Resolves the conflicting Greptile/Cursor ordering findings. - Policy-blocked sign-ups now record USER_SIGNUP_BLOCKED instead of USER_SIGNIN_BLOCKED, so account-lifecycle events aren't mislabeled. * fix(storage): meter copilot ingest centrally for path symmetry Address Cursor review: only uploadCopilotFile incremented storage, but copilot files also enter via the generic upload route and presigned uploads — all of which persist metadata through insertFileMetadata. Move the increment into insertFileMetadata (scoped to context='copilot', on genuine insert/restore) so every ingest path is symmetric with the delete-time decrement, and drop the now redundant per-path increment in uploadCopilotFile. Other contexts are metered by their own managers and remain unaffected. * fix(audit): skip WORKFLOW_EXPORTED when admin export is empty Address Cursor review: when every requested workflow fails to load, the admin export still returned 200 but recorded a successful export of zero workflows. Guard auditExport so an empty result records nothing. * fix(storage): settle copilot accounting before deleting the blob Address Cursor review: the blob was removed before releaseDeletedFileStorage, so a release failure left the counter inflated and the metadata active with the blob gone. Now the atomic soft-delete + decrement runs first and the blob is deleted only if it succeeds, so a failure leaves the file fully intact and retryable. * fix(storage): decrement KB document storage atomically with deletion Address Cursor review: hardDeleteDocuments deleted the rows then decremented best-effort, so a decrement failure left billed storage inflated with no row to reconcile. Resolve each owner's subscription up front, then decrement inside the same transaction that deletes the embeddings/documents (decrementStorageUsageInTx, also now shared by releaseDeletedFileStorage), so the counter and the content commit or roll back together. Connector docs remain excluded. * fix(audit): don't treat email verification as a login Address Cursor review: /verify-email could emit USER_LOGIN when verification mints or refreshes a session, mislabeling (or double-counting) a non-sign-in as a login. Restrict isLoginPath to genuine sign-in entrypoints. * fix(audit): null actor FK when the user lookup throws Address Greptile P1: the catch branch left the original actorId, so a system actor like 'admin-api' (or a since-deleted user) would FK-violate the insert and lose the row when the existence lookup errored. Mirror the not-found branch — null the FK with a readable label — so the audit row always persists. * revert(audit): drop session/account-lifecycle auth instrumentation Remove the Better Auth login/logout/session-revoke/account-delete/blocked-signin audit hooks from auth.ts — they touch sensitive auth paths and are noisy. Also removes the now-unused taxonomy (USER_LOGIN/_FAILED, USER_SIGNIN_BLOCKED, USER_SIGNUP_BLOCKED, USER_LOGOUT, SESSION_REVOKED, ACCOUNT_DELETED, ACCOUNT_EMAIL_CHANGED actions; SESSION/USER resource types) and the awaitable recordAuditNow helper that only the pre-delete hook used. auth.ts is back to the staging baseline. * fix(storage): harden copilot+KB delete accounting against read errors and concurrency Final-audit follow-ups: - deleteCopilotFile: a failed metadata *read* (vs a genuine missing row) now blocks the blob delete too, so a transient read error can't leave an active row un-decremented with the blob gone. - hardDeleteDocuments: drive the per-user decrement from the delete's returning() (the rows this tx actually removed), so two concurrent deletes of the same ids can't both decrement. * chore(observability): drop two unused definitions Final-audit cleanup: remove the orphaned knowledge_base_searched PostHog event (never wired; KB search analytics already flow through the OpenTelemetry channel) and the redundant AuditAction.SUBSCRIPTION_UPDATED (subscription/plan changes are audited via ORG_PLAN_CONVERTED). Every remaining new action/event has a real emit site. * fix(knowledge): key hard-delete result off rows actually deleted Address Cursor review: hardDeleteDocuments returned existingIds.length (the requested count) and cleaned storage for the full pre-tx set, even though the decrement was driven by the rows the transaction actually deleted. Under a concurrent delete that claimed some ids first, that overstated the result and re-touched storage for rows this call didn't delete. Drive the storage cleanup, log, and return value from deletedDocs (the returning() rows) so all four are consistent. * fix(storage): gate copilot quota on all ingest paths; tidy inline comments - Copilot uploads via the generic /api/files/upload route and presigned URLs now run checkStorageQuota before writing (matching uploadCopilotFile), so no copilot ingest path can grow usage past the plan limit. The central increment stays in insertFileMetadata. - Trim/remove redundant inline comments across the diff; keep only concise notes on non-obvious decisions (left pre-existing comments untouched). * fix(analytics): omit empty workspace_id on workspace-less file downloads Address Greptile P1: the generic key-based /api/files/download route and the no-workspace markdown-export path emitted file_downloaded with workspace_id:'' creating a phantom '' bucket in PostHog. Make workspace_id optional on the event and omit it when there is no workspace (workspace-scoped routes still pass it). * fix(analytics): omit empty workspace_id/workflow_id on copilot_chat_sent Final-sweep nit: copilot_chat_sent sent '' for workspace_id/workflow_id in the agent (workspace-less) branch, same phantom-bucket issue as file_downloaded. Make both properties optional and omit them when absent. * fix(analytics): clear stale org PostHog group on personal workspaces Address Cursor review: switching from a team workspace to a personal one left the previous organization group set, so later events kept rolling up under it. When the active workspace has no organizationId, resetGroups() to drop the stale org group, then re-apply the workspace group. * fix: address review — export audit timing, copilot delete signal, group reset - Table export (Cursor MED): audit fires before streaming begins, not after controller.close(), so a mid-stream failure still records the partial export. - deleteCopilotFile (Greptile P1): throw when storage accounting can't be settled instead of silently returning, so callers can detect the file was not deleted. - workspace-scope-sync (Cursor MED): only resetGroups() once workspace metadata is loaded (activeWorkspace present), so a team workspace doesn't transiently lose its org group while organizationId is still null during load. * refactor(observability): final line-justification cleanup Address final audit flags: - Table import: move the 'columns added' audit OUT of the import transaction into a post-commit auditTableColumnsAdded() helper called by the three tx-owning callers, so a mid-import row-batch rollback no longer logs a false 'added N columns' (matches the PR's success-only discipline). - Omit empty-string analytics dimensions on workflow_lock_toggled (workspace_id) and organization_created (name), consistent with file_downloaded/copilot_chat_sent. - Restore an unrelated capacity-check comment removed incidentally. - Move copilot_chat_sent emit above the traceparent comment so the comment sits with the code it documents. * fix(table): attribute import column audit to the importing user Address Cursor review: addImportColumns (async createColumns import path) now threads the importing userId into auditTableColumnsAdded instead of falling back to table.createdBy, so column additions are attributed to the actual member who ran the import rather than the table creator. * feat(observability): drop copilot from storage accounting Per design decision: copilot files are working/conversational artifacts, so gating their materialization on storage quota would fail an agent operation mid-flow when a user is over limit, and metering them would inflate usage enough to block KB/workspace uploads indirectly. Remove copilot quota gates + copilot ingest metering entirely (revert copilot-file-manager, metadata, and the upload route's copilot branch to baseline) and drop the now-unused releaseDeletedFileStorage. KB document metering + quota enforcement (the deliberate, persistent storage path) is unchanged. * fix(analytics): switch workspace + org PostHog groups together Address Cursor review: gate the group-sync effect on workspace metadata being loaded (activeWorkspace present) so the workspace and organization groups always update atomically. Acting during the load window paired the new workspace group with the previous workspace's org group; until metadata loads, events stay consistently attributed to the previous workspace. * fix(table): audit async export at authorization, not job completion Address Cursor review: async exports only emitted TABLE_EXPORTED when the background job reached 'ready', so an authorized export whose job later failed or was abandoned left no audit trail — inconsistent with the sync export route, which audits before streaming. Move the audit + analytics to the async route's authorization point (after the job is claimed/dispatched) and remove it from the runner. Drop the now-unused userId from TableExportPayload. * fix(billing): never let payment_failed instrumentation skip user blocking Address Greptile P1: the payment_failed audit hoisted an unguarded isSubscriptionOrgScoped DB read (for entity_type) directly before the attempt-count user-blocking block. A transient failure of that read would throw out of the handler and skip blocking. Wrap the whole audit/analytics block in try/catch (best-effort), and let the blocking compute its own isSubscriptionOrgScoped as it did originally — instrumentation can no longer abort payment processing. * chore(observability): trim verbose inline comments to concise notes * fix(billing): wrap new dispute/enterprise/subscription instrumentation in Stripe webhook idempotency recordAudit/captureServerEvent calls added for charge disputes, enterprise subscription provisioning, and free->paid subscription creation ran unconditionally with no idempotency guard, unlike their sibling handlers in the same files. Stripe redelivers webhooks at-least-once, so a retry would double-record the audit row and PostHog event even though the underlying DB writes were already idempotent. * fix(observability): tag org-scoped audit metadata with organizationId, guard concurrent table delete The org-scoped self-service audit-log endpoint matches org-level rows via metadata.organizationId (or resourceType=organization). Six recordAudit call sites (subscription create/cancel, admin credit issuance, threshold overage billing, charge disputes, credit purchase, invoice payment succeeded/failed) tagged org-scoped events with a differently-named key (referenceId/entityId/targetOrgId), making them invisible to org admins querying their own audit trail despite being stored in the DB. Add the missing organizationId key everywhere the pattern was missed. Also guard deleteTable's archive UPDATE with isNull(archivedAt) so a concurrent duplicate delete request is a no-op instead of re-archiving and re-firing a duplicate TABLE_DELETED audit row. Adds test coverage for the ORG_MEMBER_ADDED audit/analytics emission in acceptInvitation, which previously had none. * fix(test): queue resolveBillingActorId's owner lookup in payment-failure email test handleInvoicePaymentFailed's new payment_failed audit instrumentation resolves the billing actor via an extra db.select before sendPaymentFailureEmails runs. The test's fixed select-response queue didn't account for it, so the org-admin lookup consumed the wrong queued row and the assertion saw zero email sends. Production behavior is unaffected — each query is independent; this was a mock-queue ordering issue only. * fix(billing): don't let a post-commit usage-limit sync failure suppress the seat audit Cursor Bugbot flagged: reconcileOrganizationSeats committed the seat change and Stripe outbox enqueue in a transaction, then called syncSubscriptionUsageLimits outside it before recording the audit/ PostHog events. A thrown sync left a genuinely-changed seat count with no ORG_SEAT_PROVISIONED/DEPROVISIONED trail. Wrap the sync in its own try/catch (log + continue) so the events always fire for a committed change, matching the fire-and-forget instrumentation pattern used elsewhere in this PR. * fix(billing): guard the new subscription-created instrumentation block Greptile P1: the org-scope check I added for the audit-metadata fix (isSubscriptionOrgScoped) was a raw DB call with no error guard, unlike resolveSubscriptionActorId next to it. Since it ran inside the idempotency lambda with an unconditional rethrow, a transient DB error would abort and retry the whole webhook after the free -> paid usage reset had already committed. Wrap the actor/org-scope resolution + audit + analytics block in try/catch, matching the same guarded instrumentation pattern already used in handleInvoicePaymentFailed. |
||
|
Theodore Li
|
0613cebfab |
feat(db): resolve DATABASE_URL per role (DATABASE_URL_<ROLE> with fallback) (#5276)
* feat(db): resolve DATABASE_URL per role (DATABASE_URL_<ROLE> with fallback) * fix(db): pin realtime process to SIM_DB_ROLE=realtime so both pools share the role Without it, the realtime process left SIM_DB_ROLE unset: the shared @sim/db client defaulted role to 'web' (web pool profile + DATABASE_URL_WEB) while socketDb used 'realtime', so the two pools diverged after cutover. Set it at the process level (bootstrap + dev/start scripts), mirroring DB_APP_NAME, so the shared client and socketDb both resolve the realtime profile and URL. |
||
|
Theodore Li
|
845a6276d9 |
perf(db): per-role Postgres connection-pool profiles (#5232)
* perf(db): drive Postgres pool size + application_name from per-role profiles Replace ad-hoc DB_APP_NAME sizing with a per-role profile map keyed by SIM_DB_ROLE (web/trigger/realtime), defaulting to web. Trigger machines open a small pool instead of 15 to avoid PgBouncer connection exhaustion. Also size realtime's separate socketDb pool down to 10. * fix(db): throw on invalid SIM_DB_ROLE instead of silently using web pools * fix(db): use Object.hasOwn for SIM_DB_ROLE validation to avoid prototype keys |
||
|
Theodore Li
|
a68d38ae75 |
feat(db): attribute Postgres connections by runtime via application_name (#5211)
* feat(db): attribute Postgres connections by runtime via application_name * improvement(db): label migration-runner connection sim-migrate; trim DB_APP_NAME comment * fix(db): label realtime's shared @sim/db connections sim-realtime too The realtime process uses both its own socketDb pool and the shared @sim/db client (handlers, preflight, permissions). Only socketDb was labeled, so the shared client defaulted to sim-app, mislabeling much of realtime's DB traffic. Set DB_APP_NAME=sim-realtime at the process level (bootstrap before the dynamic @/index import for prod; dev/start scripts for local) so both clients report it. |
||
|
Theodore Li
|
e1c3c7f6c9 |
feat(secrets): ingest env secrets at container runtime instead of fanning into ECS taskdef (#5189)
* feat(secrets): ingest env secrets at container runtime instead of fanning into ECS taskdef The app/socket ECS taskdefs were ~42KB, ~93% of which was the secrets[] array: 268 pointer entries each restating the full ~78-char secret ARN, marching toward the 64KB taskdef limit and growing ~150 bytes per hosted key added. The secret blob itself is only ~18KB/268 keys. Move secret delivery to container boot: new @sim/runtime-secrets loadRuntimeSecrets() reads SIM_ENV_SECRET_ID, fetches the combined secret once, and hydrates process.env (no-clobber, no-op when unset, fail-fast). Bootstrap entrypoints for app + realtime await it before importing the real server (env-flags reads env at module load). The app bootstrap is bun-bundled in the Dockerfile builder stage since it runs outside the Next standalone bundle; realtime keeps full node_modules and runs the TS entry. Backward-compatible: with the current fan-out taskdef the loader no-ops and the app reads the injected env vars unchanged. The matching infra change (empty secrets[] + SIM_ENV_SECRET_ID) ships separately, after this image is live. * fix(runtime-secrets): address review feedback - Move the binary-secret guard outside the retry loop (sendWithRetry) so a missing SecretString throws immediately instead of burning 3 attempts + backoff. - Bound each Secrets Manager request with AbortSignal.timeout(5s) so a stalled response can't hang boot indefinitely. - Drop the redundant @aws-sdk/client-secrets-manager pin from apps/realtime; it resolves transitively via @sim/runtime-secrets. - Add a test for the non-retriable binary-secret path. |
||
|
Vikhyath Mondreti
|
91f9dfdaec |
improvement(governance): derived access (#5134)
* improvement(governance): org-ws-credential roles clarity * revert isHosted * improvement(credentials): code cleanup * address comments * make kb cascade delete on user hard delete * revert env flags * chore(db): drop local 0242 migration to regenerate after merging staging Our 0242 collides with staging's 0242. Remove it (and its snapshot + journal entry) so the KB-cascade migration can be regenerated with the correct number on top of the merged staging migrations. * chore(db): regenerate kb→workspace cascade migration as 0243 Regenerated via drizzle-kit generate on top of the merged staging migrations (staging took 0242). Re-applied the safety edits: NOT VALID + separate VALIDATE on the FK re-add, and the -- migration-safe note on the DROP. check:migrations passes. * improve copy * update docs |
||
|
Vikhyath Mondreti
|
d14bc78d03 |
fix(realtime): re-check workspace role on mutating socket events (#5080)
* fix(realtime): re-check workspace role on mutating socket events * address comments |
||
|
Waleed
|
ae075f871f |
Revert "fix(realtime): re-validate socket role and evict revoked collaborator…" (#5051)
This reverts commit
|
||
|
Waleed
|
4ab8760ae0 |
fix(realtime): re-validate socket role and evict revoked collaborators (#5050)
Socket.io authorized workflow access only at join and cached the workspace role in presence, so a removed or downgraded collaborator kept live read/ write access until they disconnected. - Re-validate the cached role against the permissions table on mutating events, bounded by a short TTL; refresh or evict on change - Add /api/permissions-updated so the app reconciles active rooms, evicting revoked users (cross-pod) and refreshing downgraded roles - Notify realtime on workspace member removal and permission changes |
||
|
Vikhyath Mondreti
|
f7b40fe4a4 |
fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions (#4975)
* fix(db-part-1): eliminate pool self-deadlock from nested checkouts inside transactions * update docs |
||
|
Vikhyath Mondreti
|
d9f78c0f56 |
improvement(sockets): make offline mode recoverable and stop transient races tripping it (#4980)
* improvement(sockets): make offline mode recoverable and stop transient races tripping it * data persistence issues should trigger offline mode and force refresh * code cleanup |
||
|
Waleed
|
272bad9718 |
feat(realtime): preflight schema-compatibility check on startup (#4940)
* feat(realtime): preflight schema-compatibility check on startup The socket service authorizes every connection with a full-row query against the workflow table. When a deploy ships a realtime image whose compiled schema is ahead of/behind the live DB (e.g. a column dropped by a migration the image predates), that query fails on every request and silently breaks persistence — yet the process stays up and the shallow /health probe keeps returning 200, so the deploy looks healthy while serving nothing. Run one representative workflow query before listen(): a schema mismatch throws, propagates to the entrypoint, and the task exits non-zero and never goes healthy, so CodeDeploy auto-rolls-back instead of shifting traffic onto broken tasks. Schema-class errors (undefined column/table/function) fail fast; connection-class errors retry with backoff so a cold DB at boot does not flap. Runs once at startup, never on the per-probe LB health check, to avoid a DB blip mass- terminating the fleet (cascading failure). * fix(realtime): unwrap cause for schema codes, drop sleep after final attempt - isSchemaMismatch now walks the error.cause chain — drizzle wraps the driver error, so the SQLSTATE often lives on the inner cause, not the outer throw. Without this a wrapped 42703/42P01 was retried 5x and mis-reported as "database unreachable" instead of failing fast. - No longer sleeps after the final failed attempt (~6-10s of dead wait that undermined the fail-fast contract); sleep now only happens between attempts. - Tests: assert sleep is called exactly 4 times on exhaustion, and add a wrapped-cause fail-fast case. |
||
|
+1
|
39418f9615 |
improvement(mothership): v0.2 (#4923)
* CONTRACTS * updates * Prompt caching * Fix regression * updates to prompt caching * Prompt caching trace * VFS updates * Changelog and plan * VFS update * Improvement/mothership (#4775) * ff * System role in cache * Fixes * Add dynamic user info * Improve tool search * improvement(platform): workspace UI/UX overhaul + integrations catalog Rework the workspace around the AI-workspace model: a Mothership home, a top-level Skills route, connected-credential and integration-detail pages, and a polished sidebar/settings surface. Replace the notifications store with a unified toast system (provider-level dismiss/pause, countdown ring). Integrations & catalog: - Add a BlockMeta layer (tags + catalog templates) scoped to catalog-visible integrations; every catalog integration carries >=7 grounded templates. - Rework the taxonomy: each block declares category tools|blocks|triggers. 3rd-party services are 'tools'; first-party primitives (postgres, mysql, knowledge, file, search, stt/tts, image/video generators, thinking, etc.) are 'blocks'. Versioned blocks follow the upgrade paradigm (old hidden, latest in toolbar/docs). - Generate integrations.json + tool docs canonically from block configs. Architecture & cleanup: - Consolidate block data extraction behind a single latest-version strategy (getCanonicalBlocksByCategory; version-consistent getBlockMeta). - Unify version-suffix handling in @sim/utils/string (stripVersionSuffix / isVersionedType, with tests); registry, generate-docs, tools/utils, and integrations all route through it. - Repair latent broken barrels, remove dead code, fix BlockMeta-related type errors and 5 broken docs links. Behavior-preserving for block execution and the toolbar's tool/block listing. * refactor(platform): remove forms, templates, and creators features Remove three standalone features and their supporting code: - Forms: form-deployment pages, API routes, execution path, and docs. - Templates: the template gallery (landing + workspace) and template APIs. - Creators: creator-profile routes and contracts. Add a super-user permissions module (lib/permissions/super-user) and an organizations API contract; update the audit/db/testing packages, billing, and the session/theme providers accordingly. * New doc setup * Fixes * Fixes * Logs tools * test(workflows): update archiveWorkflow update count after forms removal The forms feature was removed, dropping the form-table update from archiveWorkflow. Update the stale assertion from 8 to 7 tx.update calls. * Add finalization on error * Fixes * change dev CI to bun run db:push * Updates Please enter the commit message for your changes. Lines starting * Contrat update * improvement(nested): subagents * updates * upgrade * improvement(knowledge): polish tag filter dropdowns (#4816) * improvement(logs): object storage backed tracespans (#4787) * improvement(logs): obj storage backed tracespans * fix storage write context * fix tests * address comments * address comments * chore(db): remove migration 0219 to regenerate after staging merge Drops the 0219_robust_shard SQL, its snapshot, and the journal entry so the trace-spans/cost schema migration can be regenerated on top of the latest staging migration chain (avoids a number collision with staging's migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(billing): accurate per-member usage via shared ledger helper Per-member/per-user usage in the org-member routes now adds the usage_log ledger to the currentPeriodCost baseline (which is no longer incremented), via a shared getOrgMemberLedgerByUser helper to avoid repeating the subscription→period→ledger lookup across the admin and member-facing routes. Co-authored-by: Cursor <cursoragent@cursor.com> * regen migrations * update migration * address comments * more code cleanup * incorrect type cast --------- Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(providers): harden OpenAI-compatible providers + add tests (#4796) * improvement(providers): harden OpenAI-compatible providers + add tests * fix(vllm): let tool-loop errors propagate instead of returning silent partial success * fix(litellm): force tool_choice 'none' on final structured-output call The deferred final call used tool_choice 'auto', so the model could emit another tool_calls round instead of the structured answer, leaving content stale. Use 'none' (matching vLLM/Fireworks) on both the streaming and non-streaming final calls so the model must return the structured response. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers/ollama): drop tools from post-tool streaming call Ollama ignores tool_choice (not in its supported fields), so vLLM/Fireworks' tool_choice:'none' guard is a no-op here. Omit tools from the final streaming payload instead so the summarization turn can't emit dropped tool calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(litellm): spread payload into deferred final call so reasoning_effort carries over The non-streaming deferred finalPayload hand-picked fields and dropped reasoning_effort (and any future payload field), diverging from the streaming path which spreads ...payload. Spread payload here too for consistency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): restore enrichment TSDoc block Keeps parity with sibling Chat Completions providers (cerebras/mistral/xai). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(fireworks): restore TSDoc on utils helpers Restore the TSDoc blocks on supportsNativeStructuredOutputs, createReadableStreamFromOpenAIStream, and checkForForcedToolUsage — TSDoc is the codebase documentation standard and should not have been stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(litellm): remove inline rationale comments (codebase uses TSDoc) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers/ollama): drop orphaned enrichment TSDoc The block documented a function that now lives in trace-enrichment.ts, so it documents nothing in this file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(copilot): deprecate mcp server (#4797) * chore(copilot): deprecate mcp * update error codes * deprecate copilot api v1 route * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza (#4777) * feat(integrations): hosted API keys for Findymail, Prospeo, and Wiza Add hosted-key support across all credit-consuming Findymail, Prospeo, and Wiza operations so Sim provides the key when a workspace has not brought its own. Register the three BYOK providers, consolidate Wiza's two-step reveal into a single polling wiza_individual_reveal op, and hide the API key field on hosted Sim for hosted operations. * fix(integrations): harden Wiza reveal polling, soften enrichment getCost guards Address Greptile + Cursor Bugbot review on #4777: return explicit failures from the Wiza individual_reveal poller instead of throwing (thrown errors were swallowed into a false queued success), short-circuit when the initial reveal is already terminal, tolerate transient 5xx/429 during polling, and return 0 (not throw) from Findymail getCost when the contacts/employees array is absent. * chore(integrations): biome formatting after wiza merge resolution * fix(wiza): type isTerminalReveal param structurally for next build typecheck * feat(enrichments): add Findymail, Prospeo, Wiza to work-email waterfall * feat(enrichments): add Wiza + Prospeo phone reveal to phone-number waterfall * feat(enrichments): opportunistic identifiers + LinkedIn URL input across work-email & phone cascades * fix(tables): reduce column header chevron size and fix sidebar shadow bleed (#4800) * feat(slack): add install + privacy section to integration landing page (#4799) * feat(slack): add install + privacy section to integration landing page Adds a hand-authored, slug-keyed landing-content module (separate from the generated integrations.json so it survives regeneration) and renders an install walkthrough + privacy-policy link on integration pages when present. Also refreshes generated docs (data-enrichment entry, icon mappings, tool mdx). * fix(landing): render privacy section independently, align CTA analytics label * docs(landing): clarify the Slack install button is behind sign-in * refactor(landing): bake integration landing content into generated json via docs-gen Moves landing content (install walkthrough + privacy) out of a render-time augment and into the generation pipeline: generate-docs reads the pure-data content map and writes landingContent into integrations.json, so the page reads a single source (integration.landingContent). Canonical types live in integrations/data/types.ts. * improvement(enrichments): align enrichments sidebar with design system (#4801) * improvement(enrichments): align enrichments sidebar with design system * fix(enrichments): consistent close button pattern and fix url link hover * fix(misc): upgrade path change for new better-auth version, billing issue for workflow block agent usage (#4803) * fix(misc): upgrade path change for new better-auth version, double-billing for workflow block agent usage * fail loudly if stripe sub id missing * fix(copilot): seq migration (#4804) * chore(db): drop redundant idx_webhook_on_workflow_id_block_id index (#4809) Removed because (workflow_id, block_id) is a left-prefix of idx_webhook_on_workflow_id_block_id_updated_at_desc, which fully covers it. The dropped index was non-unique and enforced no constraint. * perf(copilot): read chat transcripts from copilot_messages (R+1 cutover) (#4808) * perf(copilot): read chat transcripts from copilot_messages, not JSONB Flip user-facing chat reads from the legacy copilot_chats.messages JSONB array (5.7GB, 99% TOAST) to the normalized copilot_messages table via a new loadCopilotChatMessages helper ordered by seq NULLS LAST, created_at, id — the verified canonical order. Both chat-detail getters (getAccessibleCopilotChat, getAccessibleCopilotChatWithMessages) now drop the messages column from their metadata select (no more whole-array detoast on every load) and assemble the transcript from the table after authorization. This cascades to the copilot + mothership GET endpoints and to resolveOrCreateChat's conversationHistory (the LLM payload). The normalize/effective-transcript pipeline is source-agnostic (copilot_messages.content == a JSONB array element), so transcripts are byte-identical. Dual-write and the JSONB column stay in place as the internal-logic source and fallback; removing JSONB writes is a later step. Prod integrity verified before cutover: 0 messages missing, 0 NULL-seq, 0 dup keys/seq, 0 orphans, order-parity vs JSONB = 0 mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(copilot): cover auth-deny on a found row skips the messages query Address PR review: exercise the `if (!authorized) return null` contract — when the chat row exists but authorization fails, the getter returns null and never issues the copilot_messages read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): right-align run/stop in embedded toolbar; workflow cells format like normal cells (#4806) * fix(tables): right-align run/stop in the embedded table toolbar Add a right-aligned `trailing` slot to ResourceOptionsBar and move the embedded mothership table's run/stop control into it, so Filter + Sort stay left-aligned and run/stop sits opposite on the right. No-op for the search-bearing consumers (logs, resource list), which don't pass `trailing`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): workflow-output cells format values like normal cells Workflow-output columns short-circuited in resolveCellRender and rendered their value as plain text, so a sim-resource URL / external URL / JSON / date produced by a workflow never got the chip, favicon link, or typed formatting a normal cell gets. Factor value formatting into a shared `resolveValueKind` helper used by both the workflow-value branch and the plain-cell branch; the workflow branch keeps the typewriter reveal for plain streaming text via a `typewriter` flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tables): detect resource/URL links on workflow output regardless of column type Workflow output columns default to `json` (columnTypeForLeaf), so routing their values through the type-based formatter (a) gated chip/URL promotion behind `column.type === 'string'` — a URL produced by a json-typed output never became a chip — and (b) JSON.stringify'd plain string values, adding quotes and losing the typewriter reveal. Detect links (sim-resource chip / favicon URL) on the value string directly for workflow outputs, falling back to the plain `value` kind; plain cells keep the type-based formatting. Addresses Greptile P2 on #4806. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(icons): repair broken integration icon rendering (#4810) * fix(icons): repair broken integration icon rendering Two distinct bugs left integration icons broken on the /integrations page (visible at 32-40px, hidden at the toolbar's 16px): 1. Corrupted SVG paths (Notion, Greptile, Granola, Calendly, Grafana, Bedrock): over-minified data dropped elliptical-arc flag digits (e.g. `A1 1 0 5.9 7` instead of `A1 1 0 0 0 5.9 7`); Granola's cubic stream was truncated. Browsers abort path parsing at the first invalid arc flag, so each rendered as a fragment or blank. Replaced with correct path data from canonical sources, preserving each icon's existing fill/gradient and bgColor. 2. Invisible glyph (Bright Data): its icon uses fill='currentColor' but bgColor was '#FFFFFF', and every surface forces text-white on the glyph - white-on-white. Changed bgColor to Bright Data's brand blue (#3d7ffc) so the white glyph reads, matching the white-glyph-on-brand-chip convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): restore Calendly dual-tone brand colors Addresses review feedback: the previous fix replaced the broken Calendly icon with a monochrome #006BFF path, dropping the cyan #0ae8f0 accent from the original dual-tone mark. Restored the two-tone logo (blue + cyan) using clean, valid path data, cropped to a tight square viewBox so it fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge icons, fix Zoom contrast and Quiver chip - Zoom: glyph was blue-on-blue (#0B5CFF on #2D8CFF chip); switched to currentColor so it renders as a white glyph on the blue chip. - Quiver: chip bgColor #000000 -> #FFFFFF to match the icon's near-white box, and enlarged the mark slightly (viewBox crop). - Enlarged (tightened viewBox, verified no clipping): RevenueCat, Prospeo, Granola, Firecrawl, Enrich.so, and the AWS icons (RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch, SES, Bedrock, S3). - ZoomInfo left unchanged: it is a full red rounded-square logo that already fills its frame, so a crop would clip it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data wordmark on white chip; repair Circleback - Bright Data: replaced the flame glyph with the official two-tone 'bright data' wordmark (provided asset), centered in a symmetric viewBox. Reverted the chip bgColor from #3d7ffc to #FFFFFF since the blue wordmark is invisible on a blue chip (the wordmark is designed for a light background). - Circleback: a minifier had rounded the pattern's image scale to scale(0), collapsing the embedded logo to zero size (invisible). Restored the correct scale (1/280 = 0.00357142857) so the C. mark renders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): sync Quiver block color card to white chip Reflects the Quiver bgColor change (#000000 -> #FFFFFF) in the docs block info card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge AWS/Cloudflare/Dagster icons, fully white Zoom - Enlarged (tighter viewBox, render-verified, no clipping): Cloudflare, Dagster, and the red AWS icons AWS IAM, Identity Center, Secrets Manager, SES, STS. Identity Center was anomalously small (filled ~32% of its frame); the group is now sized consistently (~80% fill). - Zoom: the camera lens triangle was still #0B5CFF (blue-on-blue); switched it to currentColor so the whole camera renders white on the blue chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(wiza): consolidate individual reveal into a single operation Merges the separate Start/Get Individual Reveal operations into one Individual Reveal operation in the Wiza docs and integrations data (operationCount 5 -> 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): size remaining AWS icons to match the set (~80% fill) Bring RDS, DynamoDB, SQS, CloudFormation, Athena, CloudWatch and S3 up to the same ~80% fill as the AWS IAM/Identity Center/Secrets Manager/SES/STS group, so all AWS icons are visually consistent. Bedrock left as-is (already ~92% fill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(icons): use Bright Data flame mark, enlarge ZoomInfo - Bright Data: the full 'bright data' wordmark was illegible at chip size. Replaced with just the flame-'i' brand mark (blue #4280f6 on the white chip), centered. - ZoomInfo: cropped the viewBox toward the white 'Zi' so it's larger; the red rounded-square background still fills the chip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * improvement(icons): enlarge CrowdStrike icon The falcon mark sat small in its chip because the icon used a wide 768x500 viewBox (letterboxed in the square chip). Switched to a square viewBox centered on the mark so it fills ~80%, consistent with the other icons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(tables): serialize schema mutations to prevent parallel column clobber (#4812) * Make workflow description nullable * fix(tables): serialize schema mutations to prevent parallel column clobber * fix(tables): load workflow outside schema lock; use DbOrTx for getTableById * fix(tables): scale idle timeout in updateColumnType to avoid aborting large type changes * fix(tables): skip stale remap types when workflowId changes concurrently * fix(tables): scale idle timeout in updateColumnConstraints for large tables * fix(wait): resume live/draft async waits and preserve cell context on chained waits (#4814) * Make workflow description nullable * fix(wait): resume live/draft async waits and preserve cell context on chained waits * improvement(knowledge): polish tag filter dropdowns * improvement(knowledge): soften filter section labels * improvement(knowledge): soften list filter labels * fix(security): harden SSO domain registration, webhook path isolation, and CSV export (#4813) * fix(security): harden KB file access, SSO domain registration, webhook path isolation, env secrets, and CSV export * fix(sso): scope domain conflict query with indexed lower(domain) filter Address PR review: avoid a full-table scan on every SSO provider registration by filtering candidate rows in SQL with lower(domain) = <normalized>, keeping the in-memory ownership check. Also tighten the normalizeSSODomain TSDoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: condense env route security comments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * icons update * chore(security): tighten inline comments in CSV export and KB file authorization Condense verbose comment blocks to concise TSDoc/single-line form; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): validate internal serve origin in KB file authorization Replace the bypassable isInternalFileUrl substring check in resolveInternalKbKey with an origin allow-list (base URL, internal API base URL, TRUSTED_ORIGINS). A crafted external host whose path is /api/files/serve/<victim-key> no longer resolves to the victim key. Relative same-origin URLs are unaffected. * style(sso): use idiomatic sql lower() comparison for domain conflict query Match the repo's prevailing `sql`lower(col) = value`` idiom for the case-insensitive SSO domain conflict lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): align workspace env admin gate with hasWorkspaceAdminAccess Use the same admin check the secrets UI uses (owner, admin permission, or org-admin) so owners and org-admins are not wrongly denied their own decrypted workspace secrets, while read-only members remain restricted to names only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sso): rely on lower(domain) match for conflict detection, drop dead in-memory recheck Address PR review: the SQL `lower(domain) = <normalized>` predicate already excludes rows that the in-memory `normalizeSSODomain(...) === domain` recheck claimed to catch, making that recheck dead/misleading code. Match on the canonical lower-cased domain and filter purely by ownership. Malformed legacy values (wildcards, schemes, ports) never match an email domain at sign-in, so excluding them is not a gap. Test DB mock now applies the lower() predicate so the casing-variant case is genuinely exercised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): scope webhook deploy path conflict to active webhooks findConflictingWebhookPathOwner omitted the isActive filter that the runtime dispatcher (findAllWebhooksForPath) applies, so an inactive but non-archived webhook from another workflow (e.g. after undeploy or failure auto-disable) would permanently block any new deployment on that path even though it never receives deliveries. Align the guard with the runtime isActive + archivedAt filter; the earliest-owner runtime check remains the authoritative cross-tenant protection. Also trims verbose TSDoc on the webhook path-isolation helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): exclude archived workflows from webhook deploy path conflict findConflictingWebhookPathOwner now joins workflow and filters isNull(workflow.archivedAt), matching the runtime dispatcher (findAllWebhooksForPath). A webhook on an archived workflow can never receive deliveries at runtime, so it must not block legitimate path reuse with a 409. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): anchor KB file ownership to earliest document in any state A KB file's owner is now the earliest document referencing its key regardless of state (active/archived/deleted/excluded); access is granted only when that owning document is still active. Closes the residual where an attacker could plant an active document to claim a file whose original document was archived or deleted. * updated greptile icon * revert(security): drop KB file authorization changes Reverts the knowledge-base file-access work (origin-pinning / owner-pinning / origin allow-list in verifyKBFileAccess) and its test. The other hardening fixes (SSO domain registration, webhook path isolation, workspace env secrets, CSV export) are unchanged. apps/sim/app/api/files/authorization.ts is restored to its origin/staging baseline. * fix(sso): treat caller's own user-scoped provider as owned during conflict check Self-hosters often register SSO user-scoped via the CLI script (no SSO_ORGANIZATION_ID). If they later enable organizations and reconfigure the same domain org-scoped through the UI, the conflict check previously treated their own user-scoped row as another tenant's and returned a misleading 409. Recognize the caller's own user-scoped provider as owned so that migration is allowed, while still blocking another user's or another org's domain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * revert(security): remove workspace-env admin gate Defer to a credential-based access model (separate change). Restores GET /api/workspaces/[id]/environment to main behavior and removes the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(security): consolidate webhook path-collision check into one helper Extract findConflictingWebhookPathOwner to lib/webhooks/utils.server.ts as the single source of truth for cross-tenant path-collision detection, used by both webhook creation paths (deploy sync and the manual /api/webhooks route). This also repairs two latent issues in the manual route's previous inline check, which queried with limit(1) and only webhook.archivedAt: - limit(1) inspected one arbitrary row, so a same-workflow row could mask a foreign collision (false negative). The shared helper scans all matching rows. - It omitted isActive/workflow.archivedAt, so inactive or archived-workflow webhooks (which never receive deliveries) permanently blocked path reuse. The helper mirrors the runtime dispatcher's filter. Same-workflow webhook reuse for upsert is now a separate, explicit lookup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF (#4818) * fix(security): block private/reserved IPs for hosted 1Password Connect SSRF * test(security): use real isPrivateOrReservedIP and cover IPv6 edge cases * improvement(integrations): validate and expand devin, cursor, and greptile (#4820) * improvement(integrations): validate and expand devin, cursor, and greptile - devin: fix missing org_id path segment on all session endpoints, add 7 session sub-resource tools (list messages/attachments, get/append/replace tags, archive, terminate), pagination, and is_archived output - cursor: add get_api_key_info, list_models, list_repositories tools - greptile: align block and docs - normalize array outputs to default [] and tighten types * refactor(cursor): simplify list_repositories v2 array normalization Collapse the redundant `?? []` + `Array.isArray` double-guard into a single Array.isArray check, per PR review feedback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): scope session-tag mapping to tag ops and normalize array tag inputs - Only map sessionTags into the tools tags param for append/replace operations, preventing stale sessionTags state from clobbering create_session tags - Fall back to a wired tags value when sessionTags is empty for tag operations - Normalize tag inputs (string or wired string[]) via normalizeTags so array values from other blocks no longer throw on .split Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cursor): restore base64 file data in legacy download_artifact metadata The legacy CursorBlock exposes only content + metadata (no v2 file output), so metadata.data was the only way legacy-block workflows could access downloaded artifact bytes. Restore the base64 data field and document it in the outputs/type instead of dropping it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(devin): coerce terminateArchive to archive flag for boolean-wired input * docs(integrations): regenerate tool docs for new devin and cursor operations --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(search-replace): don't auto-navigate when content edits invalidate the active match (#4819) * fix(search-replace): don't auto-navigate when content edits invalidate the active match * fix(search-replace): clear afterReplaceIndexRef on apply failure and zero matches * fix(search-replace): remove duplicate setActiveSearchTarget(null) on close * fix(search-replace): move afterReplaceIndexRef write inside handleApply past the guard * fix(search-replace): auto-navigate when hydration resolves with no prior active match * chore(search-replace): remove inline comments * fix(search-replace): revert !activeMatchId guard that caused immediate re-navigation after deselect * improvement(enrichments): limit company-info to fields both providers return (#4817) Hunter's company dataset returns null industry/foundedYear for many large companies (verified against the live API for Microsoft, Amazon, Google), so under the first-non-empty-wins cascade those columns appeared inconsistently across rows. Limit company-info outputs to employee count and description — the fields Hunter and PDL both reliably return — so every row is consistent. employeeCount is a string so Hunter's range bucket and PDL's exact count share the column. * fix(files): don't reject external URLs containing '..' in file parse validation (#4821) * fix(files): don't reject external URLs containing '..' in file parse validation The file block's file_fetch operation rejected any external URL whose path contained '..' (e.g. Slack files-pri slugs with a literal '...') with 'Access denied: path traversal detected'. Traversal checks only apply to local paths — external http(s) URLs are fetched with SSRF protection downstream and are never resolved against the filesystem, so they now short-circuit as valid. Internal /api/files/serve/ URLs keep full traversal protection. * test(files): fix external-URL assertion to handle undefined error * test(files): assert success explicitly in external-URL traversal test * fix(files): keep traversal protection for https URLs matching internal serve paths * feat(google-sheets): add row filtering to read with numeric operators (#4822) * feat(google-sheets): add row filtering to read with numeric operators Adds client-side row filtering to the Google Sheets read (v2) operation. Filter the returned rows by a header column using text operators (contains, not_contains, exact, not_equals, starts_with, ends_with) and numeric/ordering operators (gt, gte, lt, lte). Filtering lives in a pure, unit-tested helper (filterSheetRows) and runs over the fetched read range; an optional `filter` output reports whether the column was found and how many rows matched. Also hardens the surrounding tools: - trim spreadsheetId in write/update/append URL builders (matches read) - URL-encode the v1 read default range - expose valueInputOption for the update operation in the block Backwards compatible: with no filter requested, read output is byte- identical and the `filter` field is omitted. The filterMatchType union is widened additively (4 -> 10 values). * fix(google-sheets): correct filter metadata for missing column and header-only sheets - matchedRows is now 0 (not totalRows) when the filter column is not found, so it no longer contradicts applied=false / columnFound=false - columnFound now reflects an actual header lookup for empty/header-only sheets instead of being hardcoded true - add tests covering header-only and empty sheets with present/absent columns * fix(selectors): fetch all pages for paginated dropdown list routes (#4823) * fix(selectors): fetch all pages for paginated dropdown list routes Dropdown selectors fetched only the first page of paginated provider APIs, silently hiding results past page one. Add bounded server-side draining to the list routes across Microsoft Graph, Google, Notion, Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a shared client-side drain cap in the selector hook. Response shapes, stored values, and tool execution are unchanged; CloudWatch list tools still honor a caller-supplied limit. Also fixes the Word file picker that was searching for .xlsx files. * fix(selectors): harden JSM and Monday pagination draining - JSM service-desk/request-type drains advance `start` by the actual row count returned (not the fixed page size) and stop on an empty page, so a short non-final page can't skip items. - Monday boards drain now checks `response.ok` per page, surfacing a mid-drain HTTP failure instead of treating it as an empty final page and returning a partial 200. * docs(selectors): clarify JSM drain advances start by actual row count The offset-advancement fix (advance `start` by the rows returned, not the fixed page size) landed in 7b19788a8; update the TSDoc to match so it no longer reads as advancing by `limit`. * fix(selectors): drain fetchPage in direct fetchList callers Making `fetchList` optional left three direct callers (outside the useSelectorOptions hook) calling it unguarded, which broke the build's type check. Route them through a shared `loadAllSelectorOptions` helper that uses `fetchList` when present and otherwise drains `fetchPage`. This also prevents a regression: `confluence.spaces` / `knowledge.documents` now paginate via `fetchPage` only, and these callers (search/replace, value resolution) would otherwise have silently returned no options. * chore(selectors): rename MAX_PAGE_PAGES to MAX_NOTION_PAGES for readability * fix(sso): re-check domain conflict before write and reject IP-address domains (#4825) * improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write (#4826) Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes. * feat(tables): expand filter operators (not-contains, starts/ends-with, not-in, empty) (#4827) Add does-not-contain ($ncontains), starts-with ($startsWith), ends-with ($endsWith), not-in-array ($nin, previously executed server-side but unexposed in the UI), and is-empty/is-not-empty ($empty) filter operators end-to-end — SQL builder, condition types, query-builder converters/constants, the filter UI, the Table tools/block descriptions, and docs. Also fix correctness bugs in the filter builder surfaced by the wider operator set: - Same-column AND rules (e.g. age > 18 AND age < 65, or name startsWith 'A' AND name endsWith 'Z') silently overwrote each other because the AND group was keyed by column name. They now merge into one operator object, which also makes Filter -> rules -> Filter round-trip losslessly for multi-operator columns. - $nin values were not split into an array like $in, and textual-match values like "123" were numeric-coerced (breaking the ILIKE path). - A non-boolean $empty operand from the raw API silently inverted the check; it now coerces 'true'/'false' strings and otherwise returns a 400. * improvement(copilot): stop persisting tool-call result outputs in transcripts (#4829) Opening a Mothership task could take many seconds because a single persisted assistant message in copilot_messages.content can reach hundreds of MB, almost entirely inside contentBlocks[].toolCall.result.output (e.g. a get_workflow_logs or run_workflow result). The DB query is ~2ms; the cost is detoasting that payload, shipping it to the browser, and parsing it. These outputs are dead weight on the Sim side: they are never rendered (the thread shows only tool name/title/status) and never replayed to the model (the upstream copilot service owns conversation memory). So drop result.output before it is persisted, keeping result.success/error plus the tool metadata. - add stripToolResultOutput() in persisted-message.ts - apply it in messages-store toRow (covers every write path) and in loadCopilotChatMessages (existing rows render fast on read) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers (#4830) * feat(providers): add Together AI, Baseten, and Ollama Cloud model providers * fix(providers): guard Ollama streaming fast-path with hasActiveTools Match Together/Baseten/Fireworks: when tools are supplied but all are filtered out (usageControl 'none'), take the single streaming call instead of an extra non-streaming round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): filter non-chat model types from Together model list * refactor(providers): dedupe Ollama Cloud upstream schema ollamaCloudUpstreamResponseSchema was byte-for-byte identical to ollamaUpstreamResponseSchema (both /api/tags endpoints return the same { models: [{ name }] } shape). Drop the duplicate and reuse the shared schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(knowledge): calendar view sync, deduplicate popover animation classes, type-safe filter cast * cleanup(knowledge): remove TRIGGER_BORDER_CLASS duplication, inline displayLabel, drop enabledFilterParam alias --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> * feat(blocks): add BlockMeta to Quiver and Linq; fix invalid block config fields; update skills Block fixes: - Add QuiverBlockMeta (tags + 3 templates: icon generator, diagram creator, vectorizer) - Fix QuiverBlock: remove invalid tags field from BlockConfig, IntegrationType.Design → IntegrationType.AI (Design doesn't exist in the enum) - Fix GreptileBlock: remove invalid tags field from BlockConfig, IntegrationType.DeveloperTools → IntegrationType.DevOps - Fix LinqBlock: remove invalid tags field from BlockConfig (tags belong only in BlockMeta) Skills: - add-block: add dedicated BlockMeta section with structure, rules, and registration pattern; add BlockMeta checklist items - add-integration: add BlockMeta to block structure template, add rules clarifying that tags must NOT appear on BlockConfig and integrationType must be a valid enum value; update registry snippet to include blocksMeta; add checklist items * fix(integrations): fix category dropdown by defining missing LANDING_INTEGRATIONS_DATA_PATH and regenerating integrations.json The staging merge introduced landing-content.ts but forgot to define LANDING_INTEGRATIONS_DATA_PATH in generate-docs.ts, causing the script to crash before writing integrations.json. The stale JSON had integrationTypes (plural array) from an older script version, while the Integration type and workspace UI both read integrationType (singular string) — so ALL_CATEGORY_SECTIONS bucketed to undefined and the category filters never appeared in the dropdown. Fixed by adding the missing path constant and re-running the generator. integrations.json now has 192 entries with the correct integrationType field. * fix(sidebar): restore resize handle on all pages commit |
||
|
Vikhyath Mondreti
|
1933fc4f11 | fix(env): schema treatment of empty string (#4862) | ||
|
Waleed
|
8d7bbbc670 |
chore(utils): migrate to shared random/ID utilities and add enforcement linting (#4623)
* chore(utils): migrate to shared random/ID utilities and add enforcement linting - Replace all Math.random(), crypto.randomUUID(), crypto.randomBytes(), nanoid, and uuid usages with shared @sim/utils/random and @sim/utils/id helpers across 72 files - Add new @sim/utils exports: deepClone, omit, filterUndefined (object), truncate (string), backoffWithJitter, parseRetryAfter (retry), getErrorMessage (errors) - Sweep all getErrorMessage, sleep, deepClone callsites across 500+ files to use shared utilities - Add Biome noRestrictedImports rule to catch nanoid, uuid, and crypto named imports at lint time - Add scripts/check-utils-enforcement.ts to catch Math.random and crypto.* global property access - Add check:utils script to package.json * chore(utils): replace deepClone wrapper with structuredClone built-in deepClone() was a one-line wrapper around structuredClone(), which is universally available in Node 17+ and all modern browsers. Removing the abstraction reduces indirection and means contributors don't need to learn a project-specific name for a well-known built-in. - Remove deepClone from packages/utils/src/object.ts and index.ts - Replace all 17 call sites with structuredClone() directly - Update check:utils script suggestion text - Update CLAUDE.md and global.md docs * fix(utils): add missing biome noRestrictedImports rule and correct truncate docs - Add noRestrictedImports to biome.json under style — bans nanoid and uuid package imports at lint time (crypto.randomUUID/randomBytes are caught by the check:utils grep script which handles global property access) - Correct truncate() TSDoc and parameter name: sliceLength makes it clear that total output length is sliceLength + suffix.length, matching the behavior all callers were already written to expect * fix(utils): add missing getErrorMessage imports at 4 call sites The sweep agents added getErrorMessage calls without the corresponding import in 4 files, causing test failures. Added the missing imports. * fix(utils): fix build errors from getErrorMessage sweep and retry.ts Turbopack issue - Fix retry.ts cross-file import: Turbopack cannot resolve './random.js' for internal package imports; inline the jitter crypto call directly - Add missing getErrorMessage imports to 32 files where the sweep added calls without the corresponding import (caught by type-check and test runs) - Remove accidental getErrorMessage import from crowdstrike/query/route.ts which has its own domain-specific getErrorMessage for parsing CrowdStrike's JSON error format - Fix use-sub-block-value.ts type error from structuredClone narrowing: add 'as T' cast at emitValue callsite (safe — valueCopy is always a structural copy of newValue) * fix(tools): use toError in crowdstrike catch block instead of local getErrorMessage The catch block was calling the local getErrorMessage function which parses CrowdStrike API JSON responses, not JavaScript Error objects. Use toError(error).message to correctly extract the message from a caught value in this context. |
||
|
Waleed
|
b5dba82ac9 |
improvement(db): reduce connection saturation and egress hotspots (#4594)
* improvement(db): reduce connection saturation and egress hotspots * fix(vfs): preserve native content type in copilot SQL projection * fix(vfs): guard jsonb_array_elements against non-array contentBlocks |
||
|
Vikhyath Mondreti
|
43f53bb7d3 |
feat(execution): payload size bottlenecks with lazy execution value hydration, safer materialization, and batched parallel execution (#4560)
* improvement(resolver): lazy resolution for underlying fields greater than 10MB * progress * feat(parallel): batching * codegen to allow inline substitution * address comments * ui inconsistencies * cleanup redundant code * address more comments * address comments * replace helper * fix tests |
||
|
Vikhyath Mondreti
|
6cb779601a |
feat(search-replace): search & replace, cut, deploy modal ui flicker (#4507)
* feat(search): workflow search and replace * fix alignment * fix hidden fields bug * fix loops/parallel badge case * resource resolver * add cut * update docs * address comments * make source code for func blocks dispay resolved code instead * fix match issue * fix padding |
||
|
Theodore LiandClaude Opus 4.7
|
98f8e854eb |
improvement(tables): extract TablesDetail wrapper, ship trigger followups (#4476)
* ui improvements * Update status pils, make checkbox column sticky * add Run workflow to context menu * Refactor dispatching logic * fix checkbox width to be smaller if csv is small * Add drag behavior for workflows, stop workflow on multi select * fix z index of checkbox to left, add view workflow button * Switch to emcn buttons for Add inputs * Split up workflow sidebar from column sidebar, refactor cells * Lint and add auto run toggle * fix column reordering, add action bar * Create and use emcn square * Reconcile post-merge: drop positionMap, use rowId-based selection Staging refactored Tables UI to decouple from DB position (gutter from array index, checkedRows keyed by rowId, no PositionGapRows). Bring HEAD's action-bar / context-menu helpers in line: contextMenuRowIds, selectedRowIds, actionBarRowIds now key off row.id and walk `rows` directly. Drop the maxPosition / positionMap derived state. Collapse COLUMN_SIDEBAR_WIDTH_CSS to a numeric COLUMN_SIDEBAR_WIDTH used by both the sidebar shell and the table's reserved padding-right. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(table): backfill remapped workflow outputs from execution logs When a workflow column is re-pointed to a different (blockId, path), populate its existing rows with the new output's value pulled from saved execution logs instead of leaving them empty until the next run. Rows where the new mapping has no logged value clear (matching the previous behavior for those rows), but rows where the workflow already has the new output's value surface immediately. Refactor backfillAddedGroupOutputs into a generalized backfillGroupOutputsFromLogs helper with an `overwrite` flag — used in both the added-outputs path (preserves hand-edited values) and the new remapped path (overwrites since the new mapping is the source of truth). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): update column type when remapping workflow output A remap that changes the output's leaf type (string → number, json → boolean, etc.) was leaving the column's declared type stale. The clear- then-backfill flow then failed schema validation on every row, so the backfill silently aborted and the column stayed empty. Resolve the new leaf type via flattenWorkflowOutputs + columnTypeForLeaf for each mappingUpdate, and patch schema.columns[i].type before the schema write. The clear-tx then backfill ordering now works end-to-end across type changes. If the workflow or its target output can't be resolved (workflow deleted, block removed), fall back to leaving the column type alone — the backfill will skip rows whose picked value doesn't match, same as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): stringify objects instead of "[object Object]" in cells If a column's declared type lags its row data (e.g. a workflow column mid-remap, where the schema cache hasn't refetched yet but the row data already has the new mapping's value), formatValueForInput and the cell-render text variant fell through to String(value) and rendered "[object Object]". JSON-stringify objects in both spots so the transient skew shows the actual data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): drop extra left border on workflow group meta header The meta cell had border-r/b/l while regular headers have only border-r/b. With border-separate tables, that extra 1px left border shifted the meta cell's content one pixel right of the columns below it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): align workflow meta header without dropping its left border Restore border-l and pull the cell back -1px with -ml-px so the visible left border overlaps the previous cell's right border instead of adding 1px to the meta cell's box. Content lines up with the columns below. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): draw meta header left border via ::before pseudo Adding border-l to the meta cell shifted its content right by 1px because table-fixed + border-separate honors the border inside the colspan'd cell's width budget. -ml-px doesn't work on <th>. Render the visible left edge via a ::before at left: -1px instead — paints over the prior cell's right border without consuming any of the meta cell's content area. Content lines up with the columns below. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): add missing barrels, drop doubled-path imports Match the convention used by logs/components: every component folder exposes its public API via index.ts so consumers import from the folder name, not from its internal filenames. - New barrels: column-config-sidebar/, workflow-sidebar/, table-action-bar/, table/cells/, table/headers/. - Rename table-filter/index.tsx → index.ts (barrel is not a component). - Top-level components/index.ts re-exports every sibling folder so external consumers have one import path. - Replace `from '../foo/foo'` doubled paths in table.tsx with the shorter barrel-anchored form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): introduce TablesDetail wrapper as thin passthrough Phase 1 step 0 of the wrapper extraction (see plan okay-lets-make-a-shimmying-trinket.md). page.tsx now renders TablesDetail, which today is a passthrough to <Table>. Subsequent commits lift surface state out of <Table> into this wrapper one piece at a time. The mothership chat path (<Table embedded>) is untouched — <Table> stays exportable as a lower-level component for embedded contexts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift slideout panel state into TablesDetail wrapper The three right-edge slideout panels (column config, workflow config, execution details) move out of <Table> into the wrapper. The wrapper owns a single useReducer that encodes the at-most-one-open invariant as a discriminated union — opening any one panel automatically closes the others. <Table> emits open requests via three new callback props. Also extract <ExecutionDetailsSidebar> from inline-in-table.tsx to its own folder so the wrapper can compose it cleanly. Update the embedded mothership callsite (resource-content.tsx) to render <TablesDetail embedded> instead of <Table embedded>. Phase 1 step 1 of the wrapper extraction. <Table> shrinks from 3849 → 3787 lines; <TablesDetail> grows from 19 → 145 lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift delete-table modal + mutation into wrapper The delete-table confirmation modal and `useDeleteTable` mutation move out of <Table> into TablesDetail. <Table> exposes a new `onRequestDeleteTable` callback fired by the page-header Delete action. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift CSV import dialog into wrapper ImportCsvDialog moves out of <Table>. Grid exposes `onRequestImportCsv` fired by the page-header menu item; wrapper owns the open state and renders the dialog. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift RowModal (edit + delete) into wrapper Both RowModal instances move out of <Table> into the wrapper. Grid emits `onOpenRowModal(row)` (Space key) and `onRequestDeleteRows(snapshots)` (context menu). Post-delete cleanup (push undo, clear selection) needs grid-internal state, so the grid populates an `afterDeleteRowsSinkRef` callback that the wrapper's modal `onSuccess` invokes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift delete-columns modal into wrapper The destructive delete-columns confirmation modal moves into the wrapper. Grid emits `onRequestDeleteColumns(names)`; the cascade itself (per-column mutation, undo push, columnOrder + columnWidths cleanup) stays in the grid as a sink the wrapper invokes on confirm — too grid-internal to lift cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift run/stop mutations + TableActionBar to wrapper useRunGroup and useCancelTableRuns move out of <Table> into the wrapper, along with the <TableActionBar> render. Grid receives onRunGroup, onRunRows, onStopRow, onStopRows, onStopAll, and cancelRunsPending as props — used by the per-row gutter Play/Stop, the workflow-group meta-cell run menu, and the right-click context menu's Run/Stop on selection items. Action-bar selection state (actionBarRowIds, runningInActionBar, hasWorkflowColumns) is derived from grid-internal state, so the grid emits a `SelectionSnapshot` via `onSelectionChange` from a useEffect. Wrapper uses the snapshot to drive the floating <TableActionBar>. Phase 2 step 1 of the wrapper extraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift queryOptions to wrapper queryOptions (filter + sort) moves out of <Table> into the wrapper, making it a single source of truth that drives one useTable call. The wrapper passes the bundle down to the grid; sort/filter handlers in the grid call onQueryOptionsChange. Eliminates the previous double-useTable pattern (one for the grid's filtered/sorted view, one in the wrapper's hardcoded null/null query for sidebar metadata). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): lift page header (breadcrumbs/options/filter) to wrapper Phase 3 of the wrapper extraction. The full page-header surface moves out of <Table>: - ResourceHeader (breadcrumbs, table-rename UI, headerActions, createTrigger) - ResourceOptionsBar (sort + filter toggle) - TableFilter (filter panel — wrapper owns filterOpen state) - RunStatusControl (in the leading actions when runs are active) useRenameTable + useInlineRename for the breadcrumb name move to the wrapper. The grid populates pushTableRenameUndoSinkRef so the rename is still part of the grid's undo stack. Extract NewColumnDropdown and RunStatusControl from inline-in-table.tsx to their own folders so the wrapper composes them cleanly without reaching into the grid's internals. Hoist generateColumnName from grid-internal useCallback to a shared util so both the page-header and inline-header NewColumnDropdowns use the same logic. After this lift <Table> is the data grid only — no page surface, no modals, no slideouts, no breadcrumbs. The selection snapshot now includes totalRunning so the wrapper can render the page-header RunStatusControl from outside the grid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tables): cleanup pass on TablesDetail wrapper extraction Six-pass cleanup against the wrapper extraction diff: - Effects: add content-compare bailout to onSelectionChange emit so unchanged snapshots don't churn wrapper re-renders. - Memos: drop unnecessary activeSortState memo, fold into sortConfig. - Callbacks: remove ~10 useCallbacks with no observed reference (sidebars not memoized, modals not memoized, inline arrows on non-memoized children); keep the ones that feed into <DataRow>/<RunStatusControl>/ <ResourceHeader> (memoized) or grid-side useCallback deps. - Dead props: drop onQueryOptionsChange/onRequestDeleteTable/ onRequestImportCsv from <Table> — the page-header lift made them unused but the props weren't removed. - React Query: drop redundant tableWorkflowGroupsRef (created when onRunRows was useCallback-wrapped; after callback cleanup it can read the query data directly). - emcn: normalize Loader sizing to h-[14px] w-[14px] to match the codebase convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): re-seed columnOrder when columns change server-side The metadata-seed effect short-circuited after the first seed, so any later schema change (e.g. adding a workflow output column) couldn't push the new column into local columnOrder. The new column would then fall into the "remaining" bucket of `displayColumns` and render at the end of the table — until the user refreshed and the grid re-mounted with the now-current metadata. Drop the `metadataSeededRef.current` short-circuit from the early return so the effect can also reach the after-first-load re-seed branch, which already does the right thing (only re-seeds when the set of columns changes, leaves pure-reorder cases alone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(tables): rename wrapper to <Table>, grid to <TableGrid> Match the naming convention used elsewhere in the workspace (workflow.tsx → <Workflow>, base.tsx → <Base>, logs.tsx → <Logs>). - tables-detail.tsx → table.tsx (exports <Table>) - components/table/ → components/table-grid/ (exports <TableGrid>) - components/table-grid/table.tsx → table-grid.tsx - Drop <ExecutionDetailsSidebar> — was a 3-line passthrough (executionId → useLogByExecutionId → <LogDetails>); inline directly into table.tsx where it's used. - Flatten components/run-status-control/ folder to a single components/run-status-control.tsx file. 25-line single-use component with no internal subdirs — folder was overhead. Matches knowledge's max-badge.tsx precedent. Net: 1 wrapper rename + grid rename + 2 folder collapses, all imports updated. The mothership chat callsite updates from <TablesDetail embedded> to <Table embedded>. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): don't show "Waiting" for autoRun=false workflow groups A workflow group with autoRun=false never fires from the scheduler — the cell stays empty until the user clicks Run manually. Treating empty cells as "Waiting" misleads the user into thinking the group will auto-fire once deps are filled, which it won't. Skip autoRun=false groups when computing the per-row waiting labels so their cells render the empty-dash instead of the Waiting pill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(copilot): regenerate tool catalog from copilot dev (#247) Pulls in the workflow_group operations on user_table: add_workflow_group / update_workflow_group / delete_workflow_group / add_workflow_group_output / delete_workflow_group_output / run_workflow_group, plus the autoRun / blockId / dependencies / groupId parameters and a tightened mapping description for import_file. Also picks up biome import-order fixes from `bun run lint`. * improvement(table): action bar in mothership + per-execution mode Three related improvements to the table action bar: 1. Reposition from `position: fixed` to `position: absolute` inside the table's container. Fixed-positioning anchored to the viewport, which centered the bar across the whole window instead of the table panel — wrong in mothership embedded view, where the table sits in the right half. Absolute scopes the bar to the table's bounds. 2. Show the bar for single-execution highlights — when the user selects one workflow-output cell, or 1 row × N cols all within the same workflow group. The bar enters per-execution mode with Run / Stop / View execution buttons targeting that one cell or group. 3. Skip View execution for cancelled cells. A cancelled cell may have been cancelled before the worker ever picked the job up, so its executionId can't be relied on. Tighten the gate everywhere (context menu + action bar) to only `completed` / `error` / `running`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): backfill on add_workflow_group_output, don't re-run addWorkflowGroupOutput (the one-shot single-output add path used by the copilot user_table tool) was calling triggerWorkflowGroupRun({ mode: 'all' }) after appending the output — that re-fired the workflow on every row. Trace a307ed8fd5fe2d931aa84dedab5a60f0 shows ~75 workflow-group-cell jobs enqueued in the seconds after a single add_workflow_group_output call. Replace with backfillGroupOutputsFromLogs (overwrite: false), the same flow updateWorkflowGroup uses when receiving newOutputColumns. Reads each row's saved trace spans and writes the new output's value back — no compute beyond a JSONB write per row, no double-billing the user for runs they didn't ask for. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): drop sql.raw quote-escaping in column-name interpolation Six call sites in lib/table/service.ts built JSON-key string literals at runtime via `sql.raw(\`'\${name.replace(/'/g, "''")}'\`)` for use with PostgreSQL's `data->'key'` / `data->>'key'` operators. Practically safe (NAME_PATTERN gates column names to alphanumeric+underscore at insert time) but a smelly pattern that breaks the moment validation loosens. Both `data->` and `data->>` accept a parameterized text value as the key, so the `sql.raw` is unnecessary. Replace each with a normal `${name}::text` binding. No behavior change; eliminates the manual quote-escaping surface. Affected sites: renameColumn (the data-rewrite UPDATE), upsertRow's match filter, updateColumnType's IS-NOT-NULL gate, updateColumnConstraints' required-check + unique-duplicate-check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(copilot-tool): forward autoRun + mappingUpdates on update_workflow_group The sim-side service and contracts already accept both fields, but the copilot tool's update_workflow_group handler was dropping them on the floor. Now `args.autoRun` (toggle the persisted auto-fire flag) and `args.mappingUpdates` (per-output (blockId, path) swap) get forwarded through to updateWorkflowGroup. Pairs with the upcoming copilot-side change that exposes these in the tool catalog JSON / Go handler / prompting (see copilot branch redo-workflow-tools). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(table): keep gutter border visible when hovering Run-row button The per-row Run button sat flush against the row-gutter cell's right border. Its hover background (rounded-rect surface-2) painted over the border line for the 20px height of the button, making the gutter divider appear to disappear at the hovered row. Add mr-px to the button so the hover bg stops 1px short of the cell's right edge, leaving the divider intact. * fix(table): unify auto-fire and manual run paths in scheduler scheduleWorkflowGroupRuns now owns eligibility, autoRun semantics, dep evaluation, and enqueue for both paths. Auto-fire callers omit opts; manual callers (triggerWorkflowGroupRun) pass { groupId, isManualRun: true } to bypass the autoRun=false skip and (for autoRun=false groups) the dep check. Per-row /run-workflow-group route delegates to triggerWorkflowGroupRun with rowIds=[rowId]. Single server-side path for both manual entry points. Also: optimisticallyScheduleNewlyEligibleGroups skips autoRun=false groups so editing a row's data doesn't phantom-mark autoRun=false output cells as Queued. * fix(table): render empty cells as blank, not em-dash Empty cells (any column type) showed an em-dash placeholder. Drop it so empty cells render blank — matches what the user expects when nothing's there. * fix(table): per-row Run fires autoRun=false groups regardless of deps handleRunRow filtered out every group whose deps weren't satisfied, which silently dropped autoRun=false groups (since their deps usually aren't satisfied — that's the whole point of autoRun=false). Click Run row, the autoRun=false group's cells stayed empty. Mirror the scheduler's semantics: autoRun=false bypasses the dep check, autoRun=true still requires deps. * fix ui shape * improvement(table): collapse run ops into run_column, derive action-bar buttons from selection The action bar now reflects what's actually selected: - Selection-driven scope (cells the user highlighted, not their full rows) - Play visible when there's anything empty/failed; Refresh when there's anything completed; both for mixed - run_cell / run_row deleted; everything funnels through run_column - Per-row gutter Play, right-click "Run workflows on N rows", and column-header menu all share the canonical run path - Shared RunMode type from the contract; cleanup pass via /simplify (readExecution / isExecInFlight reuse, runScope helper, flat onViewExecution prop) * chore(copilot): regen tool catalog after dropping run_cell / run_row + dependencies.workflowGroups Mirror the copilot-side catalog change so the generated TS catalog matches the deployed copilot tool surface. * fix(table): atomic per-key writes for executions, plus run-op race fixes The executions blob on user_table_rows was read-modify-written wholesale on every update. Concurrent writers (a column edit and a manual-retry stamp, two pickup calls, a cancel and a cascade) each computed a merge from their own snapshot, and the last writer clobbered keys it never touched — producing stuck "queued" cells, vanished stamps, and stale completed exec records reappearing after retries. Fixes: - updateRow / batchUpdateRows now apply executionsPatch via a SQL jsonb merge expression. Each writer only mutates the keys it explicitly patches; other keys are preserved. Eliminates the cross-key clobber. - writeWorkflowGroupState bypasses the stale-worker guard for `queued` (new scheduler stamp) and `cancelled` (authoritative cancel) writes — those ARE the new authority for the cell. Previously the new run's stamp was being rejected by the same guard meant to block the OLD worker's writes. - skipScheduler flag on UpdateRowData / BatchUpdateByIdData lets the cancel path and runWorkflowGroupsInternal opt out of the implicit auto-fire pass (cancel was waking up siblings; manual-run was racing its own scheduler). - CELL_CONTENT pinned to h-[22px] so status badges don't grow rows. * chore(table): remove table-row sockets, both sides Tables don't use realtime sockets in prod — strip the dead path so we stop paying the per-row HTTP forward + socket emit on every cell write. Polling on running execs already covers reconciliation. Sim side: - service.ts: drop notifyTableRowUpdated/Deleted, notifyTableDeleted, the postRealtimeBridge helper, and all callsites. - hooks/queries/tables.ts: drop the socket subscription block in useTableRows; poll-on-running stays. Remove useEffect / useSocket imports. - app/.../tables/[tableId]/hooks/use-table.ts: drop the merge-on-event useEffect and unused imports. - app/workspace/providers/socket-provider.tsx: drop joinTable/leaveTable, onTableRowUpdated/Deleted/onTableDeleted, currentTableId state, related events + types. Realtime side: - handlers/tables.ts deleted; index.ts no longer wires it. - routes/http.ts: drop /api/table-row-updated, /api/table-row-deleted, /api/table-deleted endpoints. - rooms/{memory,redis}-manager.ts: drop emitToTable, handleTableRowUpdated/ Deleted, handleTableDeleted, related imports. - rooms/types.ts: drop method declarations, TableRowUpdatedPayload type, tableRoomName helper. - middleware/permissions.ts: drop unused verifyTableAccess. Bonus from parallel work: - cell-content typewriter trigger refinement. * fix(table): clearing a workflow output cell also clears its exec record When the user wipes a workflow output column value, the auto-fire reactor needs to be re-armed for that group. Previously, a stale cancelled / error exec record blocked the eligibility predicate (gate at line 79 hard-rejects those statuses on auto-fire) and the cell stayed stuck in its old terminal state — visible as "Cancelled" cells that wouldn't re-run no matter what. Both updateRow and batchUpdateRows now derive an `executionsPatch[gid] = null` for any output column the patch sets to empty. The data clear and the exec clear ride the same SQL transaction, so the row never lands in a stale- status-with-empty-data state. Symmetric to how `completed` already worked via `areOutputsFilled` in the predicate — clearing the cell wins over the prior exec status, regardless of what that status was. (Also revert typewriter-trigger experiment from a parallel session that was in-progress on this branch.) * fix(table): waiting state, optimistic UX, schema-mutation polling, exec cleanup A bundle of small UX + correctness fixes around workflow-cell run state. cell-render.tsx - In-flight (queued/running/pending) now wins over the existing value, so re-runs surface immediately instead of looking like nothing happened until the worker writes the new value. - "Waiting on X" wins over a stale `cancelled` / `error` exec when deps are unmet — clearing a dep now reads as actionable instead of stuck. useRunColumn (hooks/queries/tables.ts) - onSettled now cancels in-flight polls before invalidating. Stops a poll that landed mid-mutation from clobbering the optimistic state with stale data, which produced the queued → cancelled → queued flicker. addWorkflowGroup / updateWorkflowGroup (autoRun toggle on) - Awaits scheduleRunsForTable instead of fire-and-forget. The route returned before the queued exec stamps committed, so the post-mutation refetch saw no in-flight cells and polling never started — cells looked stuck even though the server eventually stamped them. deleteColumn / deleteColumns - Strip orphaned executions[gid] keys when deleting a column orphans its parent group. Without this, stale running/queued exec records lingered on every row forever and inflated the page-header "N running" counter even on tables with no actually-running cells. UI - Action-bar leading label: "Selected N workflow cell(s)". - Context menu: Run / Refresh items mirror the action bar's Play / Refresh split, gated on the same selection-status flags so both surfaces show the actions that match the current state. * refactor(table): consolidate exec-status helpers + fix N-running counter Cleanup pass on the recent table changes — pulls duplicated predicates and SQL snippets into shared helpers and fixes one drift bug along the way. - isExecInFlight: now single export from lib/table/deps.ts. Removed the duplicate in components/table-grid/utils.ts. Used by isGroupEligible (server eligibility) and runningByRowId (client counter). - isOptimisticInFlight: kept local to hooks/queries/tables.ts — renamed from isInFlight to disambiguate from the stricter isExecInFlight. The two predicates differ on `pending` without a jobId: optimistic patches and poll-trigger want the broader version, eligibility wants the strict one. - areOutputsFilled: single export from lib/table/deps.ts, dropped duplicate from workflow-columns.ts. - classifyExecStatusMix: shared row × group walker in table-grid/utils.ts. Replaces two copies of the same loop in table-grid.tsx (selectionStats + contextMenuStats). Both surfaces now have the same short-circuit semantics, including the seen-all-selected-rows early break that contextMenuStats was missing. - stripGroupExecutions: SQL helper in service.ts. Replaces three copies of the `UPDATE user_table_rows SET executions = executions - $gid::text` pattern across deleteColumn / deleteColumns / deleteWorkflowGroup. Drift bug: - runningByRowId / totalRunning counted only `running` and `queued`. Every other in-flight check in the codebase treats post-stamp `pending` as in-flight too, so the page-header "N running" badge briefly dropped to 0 between scheduler stamp and worker pickup. Now uses isExecInFlight. * fix(table): address pr review (drop dead workflowNameById prop, reset didDragRef on dragend, align sidebar width) * fix(table): scope post-clear schedule to targeted groups, forward mode Multi-group manual runs (Run row, gutter Play, action-bar Play across mixed completed + cancelled cells) re-fired completed-and-filled siblings. runWorkflowGroupsInternal cleared only the groups it filtered, then called scheduleRunsForRows with isManualRun: true and no group / mode filter — so the post-clear pass walked every group on the table with default mode 'all', and any autoRun=true completed sibling whose deps were satisfied got queued again. Scope the post-clear call to targetGroups and forward mode. * fix(table): meta-cell drag-leave flicker guard + plumb unique on create * fix(table): strip sibling deps when removing workflow output via updateWorkflowGroup deleteWorkflowGroup already stripped removed-column deps from sibling groups, but updateWorkflowGroup (the path the UI takes when deleting one output of a multi-output group) didn't — schema validation then rejected the update with 'Group X depends on missing column Y'. * improvement(table): debug logs at every cascade decision branch * improvement(table): parallelize queued-stamp writes within concurrency-cap chunks * Simplify stripping column names * fix lint, ci --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Theodore Li
|
31cfb74dc2 |
feat(table): add workflow execution column type (#4338)
* Add table triggers for columns and row added * Add async batching job for running column * Add ui improvements, stop mechanism * Use trigger dev for workflow runs * Use unified column sidebar for table * Add socket waits for tables, multi column workflow support * change back to cell based trigger jobs * Reuse code, add view log inline in table view * reorganize db to treat each column separately * adjust column naming strategy * Column ui improvements * fix live update on table * Change new column behavior * fixed errored workfows not showing as stopped * Column sidebar improvements * fix table column swapping behavior * fix bugs * add prompting, fix lint * fix ui stuff * flip feature flag * Add zod contracts fix initial auto-run of columns * ui improvements * Use db filter to query unran rows * Use live workflow run * Change wording for deleting workflow column * Update tools * Add tool to run selected rows * Add mothership tools * adjust col width * Restore ff * fix drizzle migration * fix test |
||
|
Vikhyath Mondreti
|
af859cd508 |
feat(workflows): lock/duplicate improvements for workflows (#4387)
* feat(workflows): lock/duplicate improvements * fix duplicate var remap bug * address comments * remove dead vars * fix tests * address comments * code cleanup * address comments * address comments * minor change * remove dead code |
||
|
Vikhyath Mondreti
|
b8959eb20d |
improvement(repo): zod based client-server boundary (#4355)
* improvement(repo): centralized zod contracts (#4336) * improvement(repo): zod schema contracts * type checks * fix(notion): correctly register tool (#4337) * fix func blokc * more improvements * fix tests * type check * remove v3 refs * minor type improvements * address comments * update jira contract * remove validateJsonBody * improvement(repo): consolidation of boundary helpers + better unknown usage (#4352) * improvement(repo): consolidation of boundary helpers + better unknown usage * address comments * improve file transfer error messaging * fix docs listing schema drift * fix inocrrect type casting * address council comments * remove prefix |
||
|
Waleed
|
ccb5f1e690 | fix(db): revert statement_timeout startup options breaking pooled connections (#4284) | ||
|
Theodore Li
|
c22ac38ab0 | Set statement timeout of 90 seconds (#4276) | ||
|
|
5f0f0edd63 |
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |