mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-30 19:32:46 +08:00
a20a5465ef586eaed2da4542df489dd6dfb80ad4
168
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
a20a5465ef |
perf(db): optimize recurring query paths (#7014)
* perf(db): optimize recurring query paths * perf(logs): batch keyset export reads * fix(workspaces): type nullable member count targets * fix(db): harden query performance changes * fix(logs): guard export stream cancellation |
||
|
Vikhyath MondretiandClaude Opus 5
|
43aba98527 |
fix(desktop): stop the OAuth connect callback from failing on a bare-path callback URL (#7005)
* fix(desktop): stop the OAuth connect callback from failing on a bare-path callback URL The desktop connect launcher passed better-auth a same-origin path as its callbackURL. Better Auth stores that value verbatim in the OAuth state, and the callback's credential-draft reader parsed it with a bare `new URL()`, which rejects a path. That throw happened inside the `account.create.before` database hook, which better-auth's OAuth callback does not guard, so the provider redirect landed on a 500 after authorization had already succeeded. Send an absolute URL from the connect page, matching the workspace-scoped branch and every other connect surface, and accept a path-absolute callback URL in the draft reader so the shape can never fail the callback again. Protocol-relative and malformed values still throw, keeping an unreadable binding loud. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(desktop): compose the connect completion URL through the URL API Concatenating `getBaseUrl()` with the completion path leaves the result dependent on how the deployment spelled `NEXT_PUBLIC_APP_URL`: the helper only adds a missing protocol, so a trailing slash produced `//desktop/connect/complete`, a pathname that matches no route. The completion page is what bounces the OAuth result to the desktop app's loopback, so that typo would have stranded the flow just past the callback it was meant to fix. Both callback URLs in the page — the launcher's and the workspace-scoped authorize redirect's — now go through one helper that resolves the path against the base with `new URL`, matching how the same function already builds the authorize URL, with coverage for a trailing-slash base. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(urls): give base URLs the no-trailing-slash form their call sites assume `getBaseUrl()` returned `NEXT_PUBLIC_APP_URL` as the operator spelled it, while almost every consumer builds `${base}/path`. A base configured with a trailing slash therefore produced a `//path` pathname that matches no route, and broke the `startsWith(`${base}/`)` prefix checks that decide whether a redirect target is our own — the OAuth authorize route rejected its own completion callback and fell back to the workspace page, so the desktop handoff never ran on those deployments. The previous commit fixed one such URL; this fixes the reason it was wrong, for the ~30 concatenation sites that share the assumption. `normalizeBaseUrl` now strips trailing slashes alongside the protocol it already added, which is the invariant SITE_URL has always documented. A path-prefixed base keeps its path. `getInternalApiBaseUrl` gets the same treatment, since its callers concatenate identically. `@sim/testing`'s urls mock is a hand-written mirror of this module, so it moves in step. `internal-api-base-url.test.ts` now unmocks the module it names — otherwise it asserts against that mirror and any drift between the two passes unnoticed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(urls): stop claiming path-prefixed base URLs are supported The previous commit's doc and test said a path-prefixed base keeps its path. That reads as support for a deployment shape the app does not have: there is no Next `basePath`, so routes are served at the origin root and such a value could not address them however the base were normalized. Every documented example is origin-only. Says only what is true — trailing slashes are the one spelling absorbed — and reframes the test as pinning the trim's shape rather than asserting a path-prefixed deployment works. No behavior change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Theodore Li
|
bbf408bf30 |
fix(security): harden public auth rate limits (#6997)
* fix(security): harden public auth rate limits * fix(security): fail closed without client IP * fix(security): backstop public OTP requests * fix(security): preserve independent rate-limit backstops |
||
|
Theodore Li
|
7173a3fb37 |
feat(status): surface major service incidents (#6987)
* feat(status): surface major service incidents * fix(status): isolate status endpoint failures |
||
|
Vikhyath MondretiandClaude Opus 5
|
71129cd112 |
feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces (#6950)
* feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces Joining a custom block's child run into its caller's trace shipped on by default, gated at read time by whether the person reading could already open the source workspace. That gate is doing the wrong job: a custom block's whole point is that consumers need no access to the source, so the check refuses exactly the readers the feature exists for, and it makes the answer depend on who is looking rather than on what the block's owner agreed to publish. The decision moves to the party whose data it is. `custom_block.trace_child_runs` is set by the publisher in Settings, applies org-wide, and is the entire policy — nothing downstream re-checks a caller. `getCustomBlockAuthority` already resolves per invocation and is the one lookup both the canvas handler and the Agent-tool runner pass through, so one column covers both surfaces and no consumer input can assert it. It defaults to FALSE. With the viewer check gone, an opted-in block publishes the source workflow's block names, inputs, outputs, and prompts to anyone who can read a consuming workflow's log. That is the same boundary curated outputs and redacted errors hold, so it opens by an affirmative act of the publisher or not at all — never as the residue of a column default on rows nobody revisited. Closed means the handle is withheld outright rather than persisted behind a flag: with no `childExecutionId` there is nothing for a reader, a migration, or a later refactor to join. What replaces it is a `_childTraceDisabled` marker, because a boundary span with no children renders exactly like a leaf block and an untraced run would otherwise read as one that did nothing. The consumer-facing failure `ref` is untouched either way — it is the only thing that makes an untraced failure reportable. Custom blocks invoked as Agent tools now join too. The child's handle already reached the agent's persisted `toolCalls[].result` (`postProcessToolOutput` strips only `__`-prefixed keys); nothing lifted it onto the tool span. Both span builders lift and strip it, and `hydrateChildTraces` needs no change — its boundary walk already recurses. The same handle is stripped from the model-facing copy of the tool result in `executeProviderTool`, the single point where the raw and model copies diverge: an opaque execution id in a tool result reads to a model like data the tool returned. The live SSE stream keeps one condition beyond the policy: an identified consumer. Not an authorization check — no workspace query — but chat deployments and the public API leave `liveTraceViewerUserId` unset because their consumer may be anonymous, and opting into org-wide tracing is not consent to stream a publisher's raw agent tokens to the internet. Copilot deliberately cannot set the field; exposing a team's internals org-wide is a human decision, not one an agent makes while publishing on their behalf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): read the publisher's trace policy at read time, not from the handle's presence Treating a persisted `childExecutionId` as proof of publisher consent is only true for handles this PR's writer produced. Every handle written before it meant something else — "a child ran; authorize the reader" — and the rows carrying them outlive the migration, so removing the reader check turned them into an open door: a consumer could open an old parent log and receive the source workflow's block names, inputs, outputs, and prompts from a block whose publisher never opted in. `hydrateChildTraces` now resolves the policy live, per boundary, from `custom_block.trace_child_runs`. The child log row's `workflowId` is the key — publish enforces one block per workflow — which also covers an Agent-tool boundary, whose span carries no block type to look up. A workflow with no block row (never published, or since deleted) has no publisher left to consent and stays shut, as does a failed policy read. This is not redundant with the write-time withholding. The handler still emits no handle for a block that was closed when the run executed, so such a run stays closed forever even if the block is opened later; this check decides whether the runs that DO carry a handle may still be shown. Turning the policy off therefore also closes what is already recorded, which is what a governance switch has to do to mean anything. Reported by Greptile on #6950. Also drops `any` from the trace-policy tests: outputs read through `Record<string, unknown>` (the handler's declared return does not name these internal keys) and failures narrow through `ChildWorkflowError.isChildWorkflowError`, which pins the failure type as well as its fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(logs): sum the child-trace drop counters from the struct, not a hand-listed set `totalDropped` re-listed four of the five counters, so a read whose only drops were policy refusals computed zero and skipped the log entirely. That is the commonest drop there is now — every handle written before the publisher policy existed refuses at that gate — so the one signal telling an operator the live check is closing joins went silent exactly when it started mattering. Summed from the struct instead. A hand-maintained list beside a struct is stale the moment a field is added, which is precisely how `policyClosed` was left out. Reported by Cursor Bugbot on #6950. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(db): renumber the custom-block trace migration around a 0299 collision Staging landed its own 0299 (`table_run_dispatches.heartbeat_at`) while this branch was open. The two migrations are independent — different tables, no shared statement — so only the number and drizzle's snapshot chain collided. Regenerated rather than hand-merged: a drizzle snapshot is a full-schema dump whose `prevId` links it to its parent, so editing one by hand to sit after a migration it was not generated against is how the chain silently stops matching the database. Staging's 0299 and its snapshot are taken verbatim; this is 0300, generated against them, and its SQL is byte-identical to what it replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
7b6c58113d |
fix(knowledge,tables): recover abandoned dispatches, bound the sweep and the workbook preview (#6945)
* fix(tables,knowledge): recover abandoned dispatches and bound the sweep
Three defects measured in production this afternoon.
A dispatcher killed by an OOM left `table_run_dispatches` at `dispatching`
forever. Every terminal transition on that table is user- or flow-initiated, so
nothing reclaimed the row: four dispatches were stranded in one afternoon,
pinning each table's "X running" overlay and blocking re-runs, with no way to
clear them from the product. The `table_run_dispatches_watchdog_idx` index has
existed for this sweep since the table was created, unused.
Liveness comes from a new `heartbeat_at`, stamped by the per-window writes that
already advance `cursor` and `processed_count`, so a slow-but-live dispatch is
spared however long it runs — the in-process path has no duration ceiling, so
ageing from `requested_at` would reclaim live self-hosted work. The sweep reads
`COALESCE(heartbeat_at, requested_at)` so rows written before the column stay
reclaimable rather than NULL-false forever, and runs as the last arm of the
existing stale-execution cron at the same 95-minute window its table-job sibling
uses. Rows are cancelled, not completed: the scope never finished.
The OOM itself is not a leak. Peak RSS is a flat plateau — 457 MB at 20-45s and
461 MB past 200s, so ten times the duration buys four megabytes — that has crept
about two percent per release for a month, from 446 MB in late July to 545 MB,
past the 512 MiB `small-1x` ceiling. CPU peaks at 0.19, so the larger preset is
bought for RAM alone. `maxAttempts` never covered the kill either: Trigger.dev
retries `TASK_PROCESS_OOM_KILLED` only when `retry.outOfMemory.machine` names a
preset, and all four runs recorded `attempt_count = 1` while the docstring
claimed they resumed from the persisted cursor.
The connector stuck-document sweep dispatched without a bound. Its chunk size
paced the loop but the candidate query had no limit, so one connector enqueued
2,959 documents in fifteen seconds onto the queue every workspace shares.
Nothing was double-billed — those documents were genuinely unindexed — but one
connector monopolized the queue, and each dispatch mints a fresh requestId, so
the idempotency key differs every pass and none of it deduplicates. Candidates
are now taken oldest-first and capped per sync; a deeper backlog is deferred to
the next sync rather than dropped.
* fix(file-parsers): read officeparser's entry point across module systems
`officeparser` is CommonJS — `main: officeParser.js`, no `type`, no `exports`
map — so what `await import('officeparser')` yields depends on who built the
code. Node and webpack synthesize named exports from `module.exports`, so
`.parseOfficeAsync` is there. esbuild, which builds the Trigger.dev worker
bundle, puts `module.exports` on `.default` and leaves the named export
undefined, and the package is in neither `build.external` nor
`additionalPackages`, so it is bundled.
Reading the named export directly therefore worked everywhere except the
worker, where calling it threw `TypeError: parseOfficeAsync is not a function`.
All four parsers treat that as "the library failed" and answer with a scrape of
the archive, which returns `degraded: true`, and the document pipeline rejects a
degraded parse outright. The visible result was every `.pptx` and legacy `.doc`
reporting "No text could be extracted from this file — it may be scanned,
image-only, or password-protected", naming a cause that had nothing to do with
the fault. 118 pptx and 14 doc failures landed in a single burst when one
connector's sync first succeeded after ten consecutive crashes.
Resolved in one shared loader rather than per bundler: externalizing the package
has to be repeated in every build config this code runs under and regresses
silently the day one is missed.
The shape handling is split into a pure `resolveParseOfficeAsync` because the
failing shape cannot be reproduced by mocking the specifier — Vitest's
module-namespace proxy throws on a missing export rather than yielding the
`undefined` a real bundle produces, so a test going through `import` can only
assert the shape that already worked. That is also why the existing parser
suites never caught this: each mocks `officeparser` with a fabricated named
export, which presupposes the interop being broken here.
* fix(knowledge): bound the workbook preview to the rows it emits
`sheet_to_json` allocates from a worksheet's DECLARED `!ref` range rather than
its populated cells, and Excel routinely writes an inflated range from stray
formatting. The 1,000-row preview cap was applied to the result, so it bounded
the emitted string while the allocation it was meant to bound had already
happened. An 880 KB workbook exhausted an 8 GB worker; the same content
exhausted 16 GB when this ran inside the connector sync. No machine size fixes
that, because the allocation scales with a number the file declares about
itself — fleet p99 for this task is 691 MB against 8 GB, so this is a cliff, not
pressure.
Passing the window into the conversion is what makes the cap real. `defval` goes
with it: defaulting every cell in the range made each row dense, so allocation
scaled with columns x declared rows rather than with populated cells, and
because no row was left empty it silently defeated the `blankrows: false` beside
it. Reported totals still come from the declared range, so bounding the
conversion does not change what the metadata says the workbook holds.
The eleven documents killed this way recorded `attempt_count = 1`: `maxAttempts`
does not cover `TASK_PROCESS_OOM_KILLED`, which Trigger.dev retries only when a
larger preset is named. Adding that escalation is a safety net rather than the
fix, and the same gap the dispatcher had.
Also corrects the machine comment, which claimed `large-1x` was 2 vCPU / 2 GB.
It is 4 vCPU / 8 GB, and believing the stale figure makes a resize look like the
answer when the parser is what is unbounded.
* fix(tables): keep a cancelled dispatch cancelled when a step claims it
`dispatcherStep` reads the dispatch, then awaits the table load before writing
`dispatching`. Keying that write on the id alone resurrected a dispatch
cancelled inside that window — a Stop-all, or now the stale-dispatch sweep —
and the fresh heartbeat it writes would then buy the resurrected row another
full window before the sweep could reclaim it again.
The race predates the sweep, but the sweep is a new writer of `cancelled` that
no user action drives, so it is newly reachable without anyone touching Stop.
Re-asserting the status the step already read is the whole fix.
* fix(tables,knowledge): spare a live window, and restore the truncation notice
A lease needs its heartbeat interval to sit well under its TTL. The dispatch
heartbeat is stamped between windows, not during them, and `batchTriggerAndWait`
checkpoints the loop for the whole window — so the interval is really "one
window", which nothing bounds: the window ends when its cells do, and the
in-process path has no ceiling at all. A window outliving the stale threshold
had its dispatch cancelled while it was plainly alive.
Its cells carry the signal the checkpointed parent cannot — `updatedAt` on every
in-flight row execution, written by the cell tasks themselves. Both signals must
be stale before a dispatch is reclaimed, so a slow window is spared for as long
as its cells keep reporting while a run with nothing beating and nothing
executing is still collected. The subquery rides the partial `(table_id,
status)` index that already covers exactly those three statuses.
Bounding the workbook conversion also made its truncation notice unreachable:
the converted length can no longer exceed the window it was compared against, so
every sheet larger than the preview cap silently stopped reporting that it had
been cut. Compared against the declared row count instead, which is what the
comparison meant before the conversion was bounded.
* fix(tables,knowledge): act on the claim outcome and scope liveness to the dispatch
Three defects, two of them created by the previous round's fixes.
Guarding the pending-to-dispatching claim without reading its outcome was the
worse half of a fix. When a Stop-all or the stale sweep won the race the row
correctly stayed `cancelled`, while the step went on to announce `dispatching`,
stamp cells and enqueue a window for it — and an empty window would then reach
the unguarded `markDispatchComplete` and overwrite `cancelled` with `complete`.
The step now ends when it did not claim the row.
The cell-liveness probe was table-scoped, and `table_row_executions` carries no
dispatch column, so a live dispatch's cells vouched for an abandoned dispatch
beside it and the abandoned row was never reclaimed — turning the stuck overlay
this sweep exists to clear into a permanent one. Narrowed to the dispatch's own
groups, which it already stores. Two active dispatches over the same groups can
still mask each other, but that is the state `markActiveDispatchesCancelled`
already prevents.
Truncation asks whether the window cut the sheet short — a question about the
declared range against the cap. Comparing the converted length to the cap made
it unreachable once the conversion was bounded; comparing the declared count to
the converted length then reported truncation for any sheet merely containing
blank rows, which are now skipped rather than defaulted into existence.
* fix(tables): scope dispatch liveness to its rows, not just its groups
The previous round narrowed the cell-liveness probe to the dispatch's groups on
the reasoning that two active dispatches over the same groups cannot coexist,
because starting a run cancels prior work on its scope. That reasoning was
wrong. `cancelPriorRuns` in `workflow-columns` requires `isManualRun`, so
auto-fired runs never cancel anything, and the per-row path is explicitly a
no-op for dispatch cancellation. Same-group coexistence is ordinary.
A dispatch that names rows now only accepts liveness from those rows, which
covers the auto-fired and row-scoped runs that reach this state. What remains is
two table-wide dispatches over the same groups, where nothing in the row
execution says whose work it is; closing that needs a `dispatch_id` column on
`table_row_executions` threaded through six write sites, including the shared
cell-write path every cell task uses. That residue is a delay rather than a
permanent mask — the live dispatch's cells stop updating when it finishes, and
the next sweep after a quiet window reclaims the abandoned row.
* refactor(tables): name the dispatch liveness predicate and bound its fan-out
Extracts the cell-activity check into `hasRecentCellActivity`, so the stale
predicate reads as its two conditions — nothing beating, nothing executing —
rather than a twenty-line SQL blob nested inside an `and()`. No behaviour
change; this is the code three review rounds found defects in, and being able
to read it is what makes those defects findable.
Bounds the terminal-event fan-out with `mapWithConcurrency`, matching how the
scheduler already fans out. The sibling cancel paths emit over one table's
dispatches; this sweep can carry a whole tick's worth across many tables, and
each event is its own write.
Also repairs the test that covers it. `collectChunks` walks into the
`tableRowExecutions` table object the fragment interpolates, so every column
name appears in the chunks whether the predicate references it or not — the
group, row, table and timestamp assertions all passed with their predicates
deleted. Matching the literal SQL instead makes them fail, which mutating each
clause now confirms.
* fix(tables): make the row bypass NULL-safe and guard the post-wait completion
`jsonb_typeof(scope -> 'rowIds') <> 'array'` was the table-wide bypass, but a
table-wide dispatch has no `rowIds`: the extraction is SQL NULL, `jsonb_typeof`
returns NULL, and `NULL <> 'array'` is UNKNOWN rather than TRUE. The bypass
never fired, so no live cell could satisfy the probe and the sweep reclaimed
exactly the long-running table-wide dispatches the row filter was added to
protect — inverting it. `IS DISTINCT FROM` is the NULL-safe form, and the same
pitfall is already handled with `coalesce` in `markActiveDispatchesCancelled`.
`completeDispatch` also wrote through the unguarded `markDispatchComplete`. Both
its callers run AFTER the window's wait, so a Stop-all or the sweep landing
during that wait leaves the row `cancelled` and the write overwrote it with
`complete`, publishing a completion event after the cancellation one. The claim
guard cannot cover this — the cancel arrives long after the claim. It now goes
through `completeDispatchIfActive`, which already exists for exactly this, and
emits nothing when the transition does not land.
* fix(knowledge): give connector sync logs a retention pass
Nothing pruned `knowledge_connector_sync_log`, so it grew by one row per sync
run forever — a connector on a fifteen-minute interval writes about 35,000 rows
a year by itself. That cost lands on `loadPreviousListingObservation`, which
reads the newest `completed` row per connector through an index covering
`connector_id` alone, so every retained row makes the sort behind the
deletion-safety corroboration slower.
Added as another arm of the cleanup cron, batched the same way as its two
sibling prunes. Two `exists` guards are load-bearing rather than defensive: the
newest row per connector always survives, and so does the newest `completed`
one, because that is the row `loadPreviousListingObservation` reconstructs the
previous listing from — and that reconstruction decides whether a suspect
listing is corroborated, i.e. whether reconciliation may delete documents.
Pruning it would silently change deletion behaviour. `started` rows are never
eligible; they are in flight or waiting on the scheduler's own sweep.
* fix(tables): funnel every post-claim completion through the guarded write
The empty-window exit still wrote through the unguarded `markDispatchComplete`,
and it runs after the claim like the other two — so a cancel landing during its
window query was overwritten with `complete`. Shorter window than the two
post-wait exits, same defect, and leaving one of three unguarded is how this
came back twice already.
All three now route through `completeDispatch`, so the guard lives in one place
and covering it once covers every exit. The redundant test for this path went
with it: it could not be made to fail against the mock, and a test that cannot
fail is worse than none — the guard is held by the test on the shared funnel.
* fix(tables): bound how long cell activity may spare a dispatch
The liveness probe cannot tell whose cells it is looking at when two table-wide
dispatches share a group, because `table_row_executions` carries no dispatch
column. On a quiet table that is only a delay — the neighbour finishes and the
next sweep reclaims — but a busy table with continuous auto-fired work can keep
an abandoned dispatch masked indefinitely, which is the stuck overlay this sweep
exists to clear.
A ceiling bounds it: past a day without a heartbeat, a dispatch is reclaimed
whatever its cells are doing. That is safe because a live dispatch stamps its
heartbeat between windows regardless of cell activity, so only a single window
outliving the ceiling could be reclaimed wrongly, and no window lasts a day on
any path — the Trigger.dev run ceiling is ninety minutes.
The real fix is a `dispatch_id` on the executions row. Threading it through the
patch layer and the upserts underneath it is a change to the hottest write path
in tables and belongs in its own review, not on the sixth round of this one.
* refactor(tables): give the stale predicate one definition of "last beat"
`COALESCE(heartbeat_at, requested_at)` was written twice — once for the stale
threshold and again for the absolute ceiling — so the two could drift into
disagreeing about what proof of life means. One `lastBeat` fragment, one
`notBeatingSince(cutoff)` helper, both cutoffs expressed through it.
Also corrects the ceiling's comment: it triggers a day past the stale
threshold, not a day past now.
* fix(tables): delete the unguarded completion rather than guard it a fourth time
The two pre-claim exits — table missing, no target groups — still wrote through
`markDispatchComplete`. Last round I argued they run before the claim, "where
forcing a terminal state is the intent". That was wrong twice over: the table
lookup is awaited, so a cancel lands in that window like any other, and a
dispatch cancelled mid-lookup has not completed its scope any more than one
cancelled mid-window has.
Routing them through `completeDispatchIfActive` left `markDispatchComplete` with
no callers, so it is gone. That is the part worth having: this is the fourth
place the same defect appeared, each time because an unguarded writer was
sitting there to be reached. With it deleted, `completeDispatchIfActive` is the
only way to complete a dispatch and the class cannot recur.
* fix(tables): re-read the dispatch before committing a window
Several round trips separate the claim from the enqueue — the window query, the
executions prefetch, the tombstone filter — and nothing rechecked the dispatch
across them. A Stop-all or the stale sweep landing in that gap had the step
stamp cells and run a whole window for a dispatch already recorded as cancelled;
the existing recheck sits after the window, which is too late to prevent it.
Mirrors that existing check on the other side of the enqueue. It narrows the gap
to a single statement rather than closing it — a cancel arriving after this read
still races the enqueue, and no check can fix that. The cell-level
`cancellationGuard` and the `isExecCancelledAfter` tombstone filter are what
catch the remainder.
|
||
|
Vikhyath Mondreti
|
81ebb37563 |
fix(delivery): stable provider idempotency tokens, and correctness fixes around them (#6943)
* fix(delivery): stable provider idempotency tokens, and correctness fixes around them Six independent fixes found while investigating a Slack transport failure. None of them are that failure; all of them are live. **Money writes could be delivered twice.** Square (8 tools), Brex (5) and Outlook Calendar minted their provider idempotency token with `generateId()` at request-build time. That is stable inside the transport retry loop — `prepareToolRequest` runs once above it — but a BLOCK-level retry re-enters the handler and mints a fresh one, defeating the provider's dedupe. A builder ticking "retry" on a Square block turned a committed write into a second card charge, and a Brex one into a second money transfer. Tokens now come from `deriveDeliveryKey`, a pure function of execution + block + tool + invocation, so every retry layer derives the same value. Stripe joins them: all ~50 tools previously sent no `Idempotency-Key` at all. The comment on `brex/create_transfer.ts` claimed a fresh key per transfer *prevents* duplicate money movement. That is inverted — fresh per *attempt* is what permits it — and is corrected here. **`invocationId` is required, not optional.** Deriving from `executionId` alone would be worse than the bug: five loop iterations paying five invoices would share one token, the provider would honour the first and silently drop four real payments, and it would look like five successes. Also: - `INTERNAL_API_BASE_URL` is now ignored on Trigger.dev workers. It names a route that resolves only inside the app container, and several modules run in both runtimes — `guardrails/mask-client.ts` says so in its own TSDoc — so setting it produced `PII redaction failed: Unable to connect` on every worker-side redaction. Mirrored into `packages/testing`'s urls mock, which reimplements the function and would otherwise have diverged. - `engines.bun` raised to >=1.3.14. Measured on 1.2.15, which the old floor permitted: a fully-delivered POST is silently replayed and the caller sees 200. - CloudWatch `put-metric-data` pinned to `maxAttempts: 1`. The AWS SDK default of 3 already replayed it, and `PutMetricData` aggregates rather than overwrites, so a duplicate silently corrupts the customer's metric series and alarm thresholds. - `webhookIdempotency` given a bounded in-progress lease. An untimed run held a SEVEN DAY lease while concurrent duplicates polled it once a second. Verified: `tsc --noEmit` clean, 30,100 tests pass. * fix(ci): restore staging files the branch split had reverted, and format Three problems, all from assembling this branch by checking paths out of a WIP branch built on an older `staging`. Anything `staging` changed since that base came back as a revert. - Root `package.json` had lost the `opentype.js` / `@types/opentype.js` dependencies `staging` added, which desynced `bun.lock` and failed `bun audit`. It also carried a `check:outbound-delivery` script belonging to other work. Every `package.json` is now taken from `staging` with only the `engines.bun` line re-applied. - `executor/utils/block-data.test.ts` — a 77-line file `staging` added — was deleted outright. Restored. - `executor/handlers/generic/generic-handler.test.ts` had lost a test `staging` added. Restored, with only the one `blockId` assertion re-applied. Also formats `keyed-invocation-identity.test.ts` and sorts imports in `internal-api-base-url.test.ts`, which is what `lint:check` failed on. * fix(providers): thread the model's tool-call id into keyed tool execution `prepareToolExecution` accepted an `invocationId` but no provider supplied one, so a keyed tool invoked through an agent always hit the incomplete-context fallback and minted a fresh token — leaving Stripe, Square, Brex and Outlook Calendar writes able to double-deliver under the hosted-key retry layer even though the block path was fixed. The id is now a positional parameter rather than another optional field on `request`, so a provider that cannot supply one fails to compile instead of silently falling through. 22 of the 27 call sites already had the OpenAI-shaped `toolCall` in scope; `tsc` identified the other five, of which Anthropic (`toolUse.id`) and Bedrock (`toolUse.toolUseId`) name it differently. Gemini is left deliberately unthreaded and documented: its function-call parts carry no model-supplied identifier — the streaming loop has to synthesize a local one — and a positional index would not survive the model re-emitting the call. A token that only looks stable is worse than the loud fallback, which names the missing fields. * fix(executor): keep execution order monotonic across a resume `executionOrder` is not carried in the pause snapshot, so a resumed run restarted the counter at 0 and a loop or parallel body executing on both sides of a pause could reuse a pre-pause value. That was cosmetic while the number only ordered logs. It stops being cosmetic once identity is derived from it: a `keyed` tool takes its provider idempotency token from this value, so two distinct writes would present the same token and the provider would silently drop the second. Suppressing a real payment is worse than the duplicate the token exists to prevent, because it looks like success. The counter is now seeded from the highest `executionOrder` among the restored block logs rather than from a new snapshot field, so snapshots written before this change are repaired on resume instead of needing a migration. * fix(providers): thread the tool-call id through the streaming loops too The previous commit only reached call sites written as a single-line three-argument call. The streaming loops are formatted across lines, so openai-compat (Groq, DeepSeek and everything else routing through it), Anthropic and Bedrock still omitted the id and kept falling back to a fresh token. Both Gemini paths now pass `part.functionCall?.id` — the RAW model id, not the `ensureToolCallId` value used for stream events. That helper allocates an execution-local id when Gemini supplies none, and it is freshly allocated per attempt: passing it would complete the keyed context, silencing the "could not derive" warning, while leaving the token just as unstable. Gemini frequently omits the id, in which case this is `undefined` and the loud fallback stands. All 27 call sites are now covered. * fix(providers): make the tool-call id argument required, not optional The TSDoc claimed a provider that cannot supply an id would fail to compile, but the parameter was declared `toolCallId?: string` — so a new call site could omit it entirely, typecheck, and silently take the unstable-token path the positional parameter exists to close. The comment promised a guarantee the type did not enforce. It is now `string | undefined`: required in position, nullable in value. A provider with no model-supplied id must pass `undefined` explicitly and take the loud fallback, rather than being able to forget the argument. All 27 existing call sites already pass it, so this is enforcement only. Verified by deleting the argument at one site: `tsc` rejects it. |
||
|
Waleed
|
63569a2459 | fix(connectors): count hard-kill failures and cap deletion blast radius (#6909) | ||
|
Vikhyath Mondreti
|
ab66ce9149 |
fix(admin): harden dashboard billing operations (#6914)
* feat(admin): make dashboard billing operations durable * fix(admin): close dashboard recovery gaps * fix(admin): preserve member operation lock order * fix(admin): harden durable operation boundaries * fix(admin): report member operation failures accurately * test(invitations): cover locked seat admission * feat(enterprise): gate owner activation on acceptance * fix(enterprise): keep owner activation recoverable |
||
|
Vikhyath Mondreti
|
aca152c7fb |
fix(env): put runtime config on <html> so client reads can't outrun it (#6923)
* fix(env): put runtime config on <html> so client reads can't outrun it The inline script that assigns `window.__ENV` is rendered from the component tree, so it lands ~13KB after the `<script async>` bootstrap tags React emits in the preamble. `appBootstrap` calls `hydrate()` synchronously whenever `self.__next_s` is empty — which it always is now that the script is a plain tag rather than a `beforeInteractive` one, that queue having been the only thing sequencing the assignment ahead of hydration. So module bodies and the first commit could both read env before the assignment landed: the socket URL fell back to the page origin for the life of the document, `getBaseUrl()` threw, and every module-scope flag in `env-flags` froze on nothing. Carry the same snapshot on `<html>`, the document's first tag, and read it in `getEnv` when `window.__ENV` is not yet assigned. Parsing is memoized against the raw attribute rather than against having run once, so the cache can never serve a value the document no longer carries. `window.__ENV` stays the public global and the preferred read, and both transports are built from one function so they cannot drift. Alongside: guard the read-only webhook-URL field so a base URL it cannot resolve is a blank field rather than a dead canvas; report what the workflow error boundary catches, which it previously swallowed entirely; and enable PostHog's native exception capture, since error boundaries only ever see their own subtree and chunk-load failures, rejected promises and throws from event or socket callbacks reached nothing. * fix(realtime): count each failed connect attempt once `manager.reconnect()` calls `open()`, whose error path emits `error` — which the socket re-emits as `connect_error` — and then emits `reconnect_error` itself. A failed reconnect therefore reached both handlers and advanced the counter twice, so the outage report tripped on the second real attempt while claiming three. Count in `connect_error` alone: it is the only handler that fires exactly once for both the initial failure and every retry. `reconnect_error` keeps its log line and states why it deliberately does not count. * fix(workflow): let an unresolvable webhook URL fail loudly Reverts the guard added earlier in this PR. A blank row titled "Webhook URL" is a worse outcome than a crash: it explains nothing, and the value is one a user copies into a third-party provider, so any substitute — a guessed page origin, an empty string — is a URL that provider accepts and then never delivers to. The read can no longer come back empty from the hydration race this PR fixes, so reaching it at all means the deployment has no application base URL, which breaks webhook registration and callbacks regardless. The error boundary now reports what it caught, so the throw names its own cause instead of surfacing as an unexplained fallback. |
||
|
Waleed
|
1ced0b6065 |
fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents (#6921)
* fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents The sweep gave 'processing' a staleness cutoff but gave 'pending' and 'failed' none, and the only cross-run guard was uploadedAt < syncStartedAt, which scopes within a sync but not across them. That was harmless while document processing awaited inline: documents were terminal by the time a sync ended. Since dispatch became asynchronous they sit 'pending' until a worker picks them up, so the next sync deleted their embeddings, reset them, and re-dispatched while the original runs were still executing — and each re-dispatch mints a fresh pass id, so it billed again. Queued documents now get a grace period derived from queue drain time rather than run duration: the processing queue's concurrency is 20 and global across workspaces, so a 2,600-document corpus takes roughly two hours to drain. The existing 45-minute threshold bounds a run, not a wait, and would still reclaim live documents on any corpus over about 900. No column records dispatch time, so the signal is processingStartedAt falling back to uploadedAt. The sweep now stamps that column when it re-dispatches, and the user-triggered retry stamps it instead of clearing it — without both, only a document's first dispatch was protected and the sweep churned once per sync forever on anything that kept waiting. The grace narrows the duplicate-billing window but cannot close it: a re-dispatch mints a new request id and the Trigger idempotency key is scoped per dispatch by design. Closing it durably needs a document-scoped key or a dispatch-generation column, recorded in TSDoc and deliberately not built here. * fix(knowledge): record dispatch time in its own column instead of overloading the start time Two review findings on this PR traced to the same root: the queue grace was reading processingStartedAt, a column that means something else. Externally, a pending row carrying a dispatch timestamp reported a processing start time for work that had not started. Internally, updateDocument sets a document pending and refreshes uploadedAt while leaving the previous run's processingStartedAt in place, and completion never clears it — so the sweep aged a re-dispatched document from a leftover stamp rather than its actual dispatch, and could reclaim it inside the grace window while the earlier queue entry was still live. That reopens the duplicate-billing race this PR closes. processing_queued_at is written on every re-dispatch and read by the sweep; processingStartedAt goes back to meaning what its name says, so its external contract is byte-identical to before this PR. Not stamped on first dispatch: the row is created and dispatched inside the same sync run, so uploadedAt is already accurate there and a guarded write per upload would buy no behavior. The internal document route returns a full table row through a passthrough schema, so the new column would have shipped as an undeclared raw Date. Declared and serialized explicitly instead. * fix(knowledge): stamp the queue time in the dispatch funnel, not at each call site Giving dispatch its own column was not enough. The connector content-update path updates a row in place with pending status and a fresh uploadedAt while leaving every prior-run processing column intact, so a document dispatched once already carried a stale queue stamp, the refreshed uploadedAt was never consulted, and the row aged from a dead run's timestamp — the same bug one level down. "Written on dispatch and only on dispatch" is now structural rather than a convention three call sites remember: the stamp lives in processDocumentsWithQueue, which every one of the nine dispatch paths already goes through, so none can forget it. The sweep's and the retry's own stamps are kept because they land inside the same transaction as their reset, so a document keeps its grace even if the dispatch that follows throws. The funnel also clears processingStartedAt, guarded on the row still being pending so it cannot disturb a worker's compare-and-set. That closes the API-facing half at its source: a pending row can no longer report a start time from any path, including the content-update path no serializer would have seen. |
||
|
|
42f6287911 |
feat(byok): add organization-wide key inheritance (#6834)
* feat(byok): add organization key management * feat(byok): inherit organization keys at runtime * feat(byok): add organization scope to BYOK settings * fix(byok): refresh org key state after mutations * fix(byok): hide stale inherited status badges * chore(db): drop colliding byok migration ahead of staging merge Staging independently claimed 0293. Remove ours so the merge is clean; it is regenerated at the next free index right after. * chore(db): regenerate byok migration at 0296 Staging claimed 0293-0295 during the merge; the regenerated SQL is byte-identical to the dropped 0293. * docs(byok): document organization scope, precedence, and the full provider list The BYOK section described workspace-scoped keys only. Add the organization scope, its Enterprise requirement, the per-provider precedence rule, what an entitlement lapse does, and the Pi sandbox exposure. Refresh the provider table from the settings page, which had drifted from 14 to 34 entries. * feat(byok): open organization keys to every organization plan Organization BYOK was gated on Enterprise, but an organization is the only thing that can hold the keys, so every plan that can own an organization should qualify — Pro for Teams, Max for Teams, and Enterprise. Add checkOrgPlan/resolveOrganizationPlan beside the Enterprise pair rather than widening checkEnterprisePlan, so the Enterprise-only gates (Access Control, whitelabeling) are untouched, and restore resolveOrganizationEnterprisePlan to module-private now that BYOK no longer needs it. * perf(byok): cache the organization entitlement, not the key material getBYOKKey runs once per agent block and once per hosted-capable tool call, so a loop over N items resolved N times — and each organization-inheriting resolution paid three sequential billing queries on top of the two key reads. Split the two reads by staleness tolerance. Key rows stay fresh, because revocation must be immediate. The entitlement is a billing gate that tolerates bounded staleness in the harmless direction (a lapsed organization keeps using its own key for <=60s), so cache it per organization with an in-flight share so concurrent blocks issue one query set. The management surfaces keep reading it fresh, so an organization that just upgraded is never told otherwise. Also run the block check and subscription read in parallel inside resolveOrganizationPlan, and carry the resolved scope on BYOKKeyResult so a log line can say whether a run used the workspace's key or an inherited one. * feat(byok): let workspaces store the Z.ai and Cohere keys the runtime reads Both ids were already in the BYOK contract enum and both are resolved at execution time — getApiKeyWithBYOK reaches 'zai' (GLM models are in the hosted catalog, so the BYOK branch runs), and 'cohere' backs both the Embeddings block and Knowledge Base reranking — but neither appeared in the settings list, so there was no way to store the key either path looks for. Cohere had no icon; add one from the official multi-color mark so it stays legible on a light and a dark page. Cohere's embed-v4.0 is kbEligible:false, so the description says 'Embeddings and Knowledge Base reranking' rather than claiming KB embeddings. * improvement(byok): shorten the workspace scope chip to 'Workspace' It sits beside 'Organization', so the scope reads from the pair; 'This' only added width. * fix(byok): do not cache a billing outage as an unentitled organization resolveOrganizationPlan maps a failed billing read to false, which is indistinguishable from a real plan lapse. The entitlement cache stored that, so one transient outage held the gate shut for the full TTL and every inheriting run silently fell back to a metered hosted key — and the cache's rejection path, which exists to prevent exactly this, was unreachable. Give the resolver the onError option its neighbours already have and let the cached read ask for 'throw', so a failure stays out of the cache and the next resolution retries. Behavior for the call that saw the error is unchanged: getBYOKKey still fails closed. Reported by Cursor Bugbot. * fix(byok): propagate the subscription read's failure too The previous commit threaded onError through resolveOrganizationPlan's own catch, but getOrganizationSubscriptionUsable soft-fails to null on its own, so a failed subscription read still arrived as an ordinary 'no usable subscription' and returned a successful false — which the entitlement cache then stored for the full TTL. Thread the option into that call as well. Test it at the billing layer rather than the cache layer: the entitlement test mocks resolveOrganizationPlan wholesale, so it could never have caught this. Verified the new test fails against the previous commit. Reported by Cursor Bugbot. * refactor(byok): cache the entitlement with LRUCache, like copilot entitlements The hand-rolled version reinvented three things the codebase already has a canonical answer for. lru-cache is a declared dependency of apps/sim and lib/copilot/entitlements.ts already caches an entitlement with it — by storing the in-flight Promise, which is what makes concurrent callers collapse onto one resolution with no in-flight bookkeeping at all. TTL and the size bound come from the library. That removes the second Map, the manual eviction (and its interaction with an in-flight entry), and the dead value-while-refreshing state: 23 executable lines. The one thing the library does not cover is dropping a rejected promise so a billing outage is not cached for the TTL, which is kept and pinned by a test that fails without it. TTL expiry is no longer re-tested — that is the library's behavior, not ours, and lru-cache reads its clock at module load so faking timers never moved it. * refactor(byok): coalesce the entitlement read with the shared singleflight lib/concurrency/singleflight.ts is the codebase's coalescing primitive and oauth/credential-service.ts already pairs it with a read-through cache. Adopting that shape fixes a case caching the promise directly did not: a *hung* billing read wedged every caller for the full 60s TTL, where coalesceLocally evicts and rejects at its settle deadline. It also removes the hand-rolled rejection eviction — the cache is written only on the success path, so an outage leaves no entry by construction. The cache now holds booleans, which introduces the one trap worth a test: a truthiness check would read a cached false as a miss and re-query billing on every resolution for lapsed organizations. Pinned. * fix(byok): keep an abandoned entitlement producer from writing the cache coalesceLocally does not cancel a producer it timed out — its docstring says so explicitly — so writing the cache from inside the producer let a late billing result overwrite a fresher answer a retry had already cached, and hold it for a full TTL. Move the write onto the value the caller actually received. A caller that timed out throws before reaching it, so an abandoned producer now resolves into nothing. The test reproduces the overwrite and fails against the previous shape. Reported by Cursor Bugbot. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
|
d9cfd7c68e |
improvement(mothership): v0.9 (#6815)
* checkpoint
* Checkpoint
* dot fixes
* Make async tool resume delivery recoverable
* Support split table tools and option recovery
* Harden VFS mutation handling
* feat(platform): platform subagent support — docs corpus VFS, search_docs, account context
Squash of the feat/platform-agent branch (sim side): mounts the Sim docs
corpus in the copilot VFS, wires search_docs and retires the legacy docs
search tools, and syncs the generated tool catalog and trace contracts for
the platform subagent.
* Align Copilot tools and resource handling
* Expand workflow log query support
* checkpoint
* Port desktop-improvements-0 desktop and browser-agent work
* fix(desktop): keep browser-agent input alive through live-SPA re-renders
* Harden workflow sanitization and Slack setup
* feat(desktop): coordinate clicks, caret insertion, and drag for the browser agent
* Revert subagent group eager auto-collapse
* fix(chat): keep sends FIFO across the streaming-to-idle drain gap
* Add the steering backend surface for mid-turn sends
* Sync generated contracts for async subagent orchestration
Pulls the mothership tool catalog (wait_agents / tail_agent / steer_agent /
interrupt_agent), trace spans (chat.async_subagent.*, chat.orchestrate.*), and
trace attributes (copilot.async_subagent.*) into the generated TS contracts.
* Add display titles for the async subagent orchestration tools
wait_agents / tail_agent / steer_agent / interrupt_agent get natural-language
running titles (naming the agent id being waited on, tailed, steered, or
stopped) and a Steering→Steered completed-verb rewrite.
* Show orchestrator-chosen subagent names on agent groups
A subagent_start whose payload data carries a name (the orchestrator's new
name trigger parameter) now labels the agent group with that mission name —
the agent-type icon stays. The name flows through the live stream path, the
turn model (AgentNode.displayName) and its serialize/rebuild round-trip, and
persisted transcripts (PersistedContentBlock.name), so reloads keep the label.
* Improve Copilot error handling and logging
* Backfill the subagent display name from the second start event
The dispatch-time subagent_start fires before the trigger args (and therefore
the name parameter) have streamed; the phase-3 start re-announces the lane with
the name. The block builder was dropping that duplicate wholesale, losing the
name on streaming providers — now it backfills subagentName onto the existing
block instead. (The home turn-model path already reconciled this case.)
* Support Slack bot connection flow
* Harden Copilot error and VFS handling
* Harden VFS resource operations
* Show 'Waiting for the first of N agents' for mode-any waits
The wait_agents title ignored the mode argument, so an any-mode wait over
three agents read 'Waiting for 3 agents' while the model narrated waiting for
the first — contradicting the transcript.
* Collapsed-by-default agent cards with live intent status lines
Subagents now narrate their work through <intent>3-5 words</intent> tags (a
fleet-wide prompt protocol on the mothership side). The turn model streams
each subagent's text through a split-safe tag parser: complete tags update the
agent's currentIntent and disappear from the prose, tags split across deltas
are carried until their close arrives, and a tag that never closes flushes
back as plain text.
The agent card renders as one line — display name (or agent label) plus the
latest intent, replaced inline as the agent shifts gears — and never
auto-expands; expanding to the full tool log is a deliberate click. Only an
outstanding permission prompt or a browser hand-back forces a group open.
Intents persist on the subagent block (and through the legacy persisted-
message paths) so reloads keep the last status, and a renamed reinvocation
now takes the latest name instead of pinning the first.
* Add the internal in-band tool execution route for live mothership turns
POST /api/copilot/tools/execute (INTERNAL_API_SECRET, Go→Sim) runs one
sim-server tool through the same server tool router the resume driver uses
and returns the result synchronously — no checkpoint. This is what lets
background (async) subagents write files/tables/knowledge, and lets the main
lane keep streaming (instead of checkpoint-pausing and killing every
background run) while async agents are live.
* Persist resource side effects for in-band tool execution
Files/tables created through the internal execute route now register on the
chat's resources exactly like the resume driver's executions — the route runs
the same handleResourceSideEffects pass (persistence only; an out-of-band
route has no live event sink, so mid-turn chip pushes are a follow-up).
* Extract intents from group text on every path, sync and async
The turn-model intent filter only fires for span-scoped subagent lanes, but
this surface also delivers subagent text through the legacy block path — so
<intent> tags flowed through unparsed and rendered as prose rows. Groups now
extract intents from their accumulated text at append time: the last complete
tag becomes the card's status line and every complete tag is stripped from
the rendered prose. Covers span-scoped, legacy, and persisted-reload paths
for both synchronous and background delegations.
* Fall back to the live tool title for the agent card status line
Persisted data proved tool-first subagents (grok search agents) emit zero
prose, so intent tags never stream no matter what the prompt says. The
collapsed card now always narrates: the agent's own <intent> tag when present,
else the latest tool's display title while the lane is live.
* Catch subagent <intent> tags in the server relay
The relay's subagent text handler now runs the split-safe intent extraction
as chunks stream: the latest complete tag is stamped onto the lane's persisted
subagent block (subagentIntent) and stripped from the stored prose, so live,
persisted, and replayed views all agree. Per-lane carry handles tags split
across chunks; a never-closing tag flushes back as plain text.
* Drop the tool-title fallback: the status line is the agent's intent
With the intent protocol now injected into every spawn's task message, agents
open with an <intent> tag; the card shows that narration or nothing.
* Replace intents with live tool-title status lines on agent cards
Intent parsing is fully removed (turn model, relay handler, persistence
fields, group extraction). The collapsed card's status is the latest tool
call in its RUNNING phrasing — never the completed rewrite, which stays in
the expanded log. Parallel tools show the most recently started still-running
title with a +N for concurrent siblings; between rounds the last title stays
frozen; a closed lane shows the bare name. Nested agent cards compute their
own status recursively from their own items.
* Keep the main Sim lane live-expanded; collapse only real subagent cards
The mothership group is the turn's own narration, not a delegation card —
collapsing it hid main-lane text and tools until manual expand, which read as
mis-ordered streaming while async subagents interleaved. It keeps the
original live-expand behavior and no status suffix.
* Persist subagent lane lifecycle blocks from the span handler
Lane-scoped span events route to the span handler, which only recorded trace
side effects — no subagent start block was ever persisted (verified: a
seven-agent run stored 104 blocks with zero starts). Grouping then fell back
to keying lane content by agent NAME, so a respawned agent of the same type
merged invisibly into the first one's card until it resolved. The handler now
persists the start block (spanId-keyed and deduped, carrying the display
name) and stamps endedAt on close, giving every invocation its own card.
* Name agents in orchestration titles; '+ n more' overflow format
wait/tail/steer/interrupt titles humanize the slugified agent ids back to
their display names ('Waiting for the first of Digest Workflow Build + 4
more'), and the agent card's parallel-tool suffix uses the same '+ n more'
format.
* Harden in-band tool execution and resources
* Route in-band execution through the comprehensive tool dispatcher
The internal execute route used the bare server-tool router, which rejects
VFS tools with 'Unknown server tool: read/glob/grep' — so nearly every
background agent's first discovery call failed (102 in-band calls in one run,
dozens rejected). It now uses the relay's executeTool dispatcher: registered
handlers (VFS, function execute) with permission checks and param
normalization, falling back to the app tool router — the same surface
foreground execution gets.
* Harden chat stream transition handling
* Harden VFS provenance and resource writes
* Standardize tool environment references
* Harden browser panel and chat cleanup
* Descriptive, user-language tool titles across the board
House rules applied everywhere: use every argument the call carries, never
name internal machinery, and never lead with Getting (the Got rewrite is
deleted so it cannot return).
- Deployments name the workflow: Deploying {workflow} as API/chat app/MCP tool
- Workflow reads name the part: Reading {workflow} meta/state/deployment/notes;
generic reads always name the file (Reading {leaf}), never bare Reading file
- Block runs name block and workflow: Running {block} in {workflow}, Running
from {block} in {workflow}, Running {workflow} until {block}, and
Enabling/Disabling {block} in {workflow}
- The six split-table tools get per-operation verbs (Adding column {name},
Updating rows, Wiring automation, Creating view {name}) instead of a wall
of Querying table
- The manage quartet drops X-action system-speak for gerunds
- get_* internal names become user language (Checking run settings, Tracing
block inputs, Reading the deployed version); web_fetch says Fetching
- Scheduled-task titles removed entirely (feature deleted from the Go catalog)
- New verb rewrites: Fetched, Traced, Wired, Configured, Looked, Rotated
* Deploying {workflow} as chat, not as chat app
* Loader gerunds; mv names both ends; mkdir names the folder
search_integration_tools -> Finding the right integration;
load_integration_tool -> Loading {integration} tools; load_skill ->
Loading skill {name}; run_enrichment -> Looking up {subject}. mv prefers the
model's phrasing, else reads 'Moving {files} to {destination}'; mkdir reads
'Creating folder {name}' from the path.
* Overflow counts read '+ n', dropping 'more'
* Unify workspace find and search
* Scale desktop title bar with page zoom
* Serialize account and organization truth into the copilot VFS
Workspace standing, membership, billing, org role, access-control
restrictions, published-block provenance, and fork topology were reachable
only through three parameterless tools (or not at all). They are ambient
read-only facts, so they belong in the VFS where they are greppable, cost no
tool round-trip, and every agent that can read gets them — the same move that
retired get_blocks_and_tools and list_user_workflows.
Adds account/{workspace,workspaces,members,billing}.json (always mounted) and
organization/{organization,access-control,custom-blocks,forks}.json (only when
the workspace is org-hosted). Every file projects an existing use case or util
after getOrMaterializeVFS's access assert — no new queries, no new
authorization. One relation per file, cross-referenced by id-and-name stub, so
overlapping facts cannot disagree. Volatile content (billing, access control,
forks) is lazy, so numbers are read-time fresh and unasked-for reads cost
nothing.
Projection follows the viewer: member emails are admin-only, fork detail
requires workspace admin on a forking-enabled org, and the whole organization/
namespace is absent for a personal workspace — which is itself the answer.
Retires get_account_billing, get_enterprise_context, and list_user_workspaces
along with their handlers; display titles stay for transcript replay.
* Fix insert_text refusing an editable field focused inside a frame
describeFocusedEditable descended shadow roots but not frames, while
activeElementReadback descends both. Focus inside a same-origin frame therefore
surfaced to the first as the FRAME element — not an input, not contentEditable,
not a canvas, no textbox role — so it fell through to 'not-editable' and
insert_text refused a field that press_key had just typed a character into.
Two functions answering 'what is focused' with different answers is the bug;
the descent loops now match exactly.
The refusal also names what actually held focus (tag, role, contenteditable).
A bare 'not-editable' gave the agent nothing to act on, so it guessed at the
cause — a real run spent twenty rounds on the wrong theory and had to be
stopped by the user.
* Keep retired browser takeover renderable in history
The tool is gone from the catalog, so its generated constant went with it and
every path that referenced it stopped compiling. Deleting those paths instead
would have silently downgraded every past transcript containing a takeover card
to a generic tool row, and dropped the no-timeout budget that an in-flight
takeover still needs while a rolling deploy finishes.
retired-tools.ts gives the literal a documented home that says what it is and
why it survives its tool.
* Follow the agent into a tab it opened to work in
browser_open_tab created the page with activate: false, so the agent worked in
a tab the user could not see while the panel sat on a page where nothing was
happening. The panel now follows a tab the agent deliberately opened.
Scoped to that tool only. A page spawning its own tab (popup, target=_blank) is
the site grabbing the view rather than the agent choosing a workspace, and
stays in the background as before — two existing tests pin that and caught the
first version of this change, which moved both.
A tab the user claimed still wins over both: the work starts in the background
instead of pulling the page out from under them mid-read.
* Make the browser tools agree with each other
An audit of the module found the frame-descent bug was one instance of a
pattern: six independent definitions of 'is this editable' and seven of 'what
is focused', disagreeing with each other. A tool refusing what its sibling
accepts on identical page state is invisible at runtime — the agent follows a
snapshot that says one thing into a tool that says another.
- browser_type now accepts role="textbox" like browser_insert_text does. The
snapshot advertises those elements as [textbox] with a ref, so refusing them
meant rejecting exactly what the outline told the model to type into. Both
the native and synthetic paths, and their descendant scans.
- pressKeyOnPage descends shadow roots and frames like every other focus
reader. It was dispatching synthetic keys at the shadow host or <iframe>
element, where they bubble but never reach the editor, while reporting
success — and contradicting the activeElement reported beside it.
- not-editable and ambiguous-editable name what was found: the element's tag
and role, and the candidate fields. Both had the data and discarded it, which
is what turns one blocked step into twenty rounds of guessing.
- obstructedAfterNavigation requires a dialog that ARRIVED with the
navigation. It compared against nothing, so every SPA route change under a
persistent role=dialog reported a successful click as obstructed. The test
that covered this asserted the false positive; it now pins both directions.
- browser_insert_text observes the top document when typing inside a frame,
like every other input tool. A submit that navigates the top page was
invisible to its frame-scoped observation.
* Let hover actually see what it mounted
Four independent defects made browser_hover blind to the most common thing a
hover produces — a row's action bar — so it reported no effect on a hover that
worked, and the agent fell back to clicking pixels off screenshots.
- The popup scan matched only role=tooltip/menu/listbox. Slack's message
shortcuts bar is a labelled toolbar/group, so it registered as nothing at
all. Added toolbar, menubar, labelled group, and [popover].
- The baseline was captured BEFORE prepareElementSurface scrolled the target
into view, so scrollChanged was always set by the tool's own probe. That
pinned every unproductive hover to 'background DOM churn' instead of the
honest 'nothing happened', and hid scrolling the hover really caused.
Re-baselined once the scroll settles and before the pointer moves.
- The MutationObserver attached only on the first observation, while the roots
list is rebuilt every call and grows as shadow roots mount. Components that
appeared later were never observed, so their DOM changes raised no revision.
Roots are now observed as they show up.
- observationTruncated was computed and never read, so a scan capped at 12k
nodes reported 'nothing appeared' with the same confidence as a complete
one — and portalled overlays live at the end of <body>, exactly what the cap
drops. Hover now says the page was too large to scan and to confirm visually.
* Stop the browser agent acting on the wrong element, and say why it refused
Four findings from the module audit, the first of which could silently do the
wrong thing rather than merely fail.
- A ref whose node is gone is re-adopted by structural resemblance, matching on
ORIGIN only so a pushState between snapshot and act does not kill every ref.
That leniency also let a ref to a row control in one view rebind to the
identical control in a view the app had since navigated to — acting on the
wrong message, signalled by nothing louder than recovered: true. Adoption now
requires the same path; a view swap reports the ref stale, and the caller
re-snapshots. Revalidating a still-connected node stays lenient, because that
is literally the node the model chose.
- A hit INSIDE the requested element is its own nested control, not an overlay.
Both produced 'covered by X — close or move the overlay', advice that cannot
be followed because there is nothing to close. Nested hits now say so and
point at retargeting.
- browser_click_at, browser_insert_text, and browser_drag listed targetChanged
in their effect formulas, but none passes an elementId, so no targetState is
ever captured and the term was always false — coverage that read as real.
Removed, with a test pinning the dependency.
- The seven effect formulas are deliberately NOT collapsed into one predicate:
drag must trust domChanged where others must not, hover must ignore field and
focus changes, click counts focus only for editables. Forcing one would make
each tool wrong differently. The differences are now documented in one place
next to the shared computation, so divergence is a declared policy rather
than an accident.
* Let edit_workflow configure block retries
* Updates
* Always focus the resource the agent is working on, and its browser tab
The resource panel had a carve-out: an already-open browser session declined
to replace another selection and only got an attention marker, so agent
browser work happened off-screen. The panel now follows the agent to whatever
it touches — browser included — and an event can still opt out explicitly.
The browser panel also follows the agent BETWEEN tabs: the store already
tracked automationTabId (and the strip marked it), but the visible tab never
changed. It now switches when the agent's target tab changes, so watching the
agent never means hunting for the tab it moved to. Keyed on the target
changing rather than on it being set, so a user who browses elsewhere
mid-run is only pulled along when the agent itself moves.
* Never paint a browser snapshot at stale geometry (the modal-open flash)
Opening a modal locks scroll, which removes the window scrollbar and reflows
the panel — so a capture taken before the lock describes a rect the panel no
longer occupies. The handshake painted that frame anyway and only then
retried, so the replacement landed visibly offset from the page it stands in
for: the flash. A capture is now checked against the host's live rect before
it is painted; a mismatched frame is skipped and re-captured at the settled
layout instead (modal retries go 2 -> 3 to absorb the extra settle).
* Name the workflow in deployment and workflow-scoped tool titles
'Checked deployment status' never said which workflow — nor did the deployed-
state read, run settings, block outputs/inputs, redeploy, promote, or the
global-variable write. These tools carry a workflowId (often defaulting to the
current workflow), so only the client can resolve a name: the enrichment layer
now resolves it for the whole workflow-scoped family and passes it as
workflowName, which every workflow title already reads.
Titles: Checking {workflow} deployment status, Reading deployed {workflow},
Checking {workflow} run settings, Reading {workflow} block outputs, Tracing
{workflow} block inputs, Redeploying {workflow}, Promoting {workflow} version
{n} to live, and 'Adding workflow variable {name} in {workflow}' — each
falling back to its unnamed form when no workflow resolves.
* Name the block that ran; never fall back to a raw block id
run_block and set_block_enabled carry only a blockId, so their titles would
have printed an opaque UUID ('Running 7f3a2b91-… in Invoice Sync'). The
enrichment layer now resolves blockId against the workflow store the same way
it already did for run_from_block's startBlockId, and the base titles no
longer accept an id as a name — an unresolved block reads 'Running block'
rather than a UUID.
* Add the missing Removing -> Removed rewrite
The table work introduced 'Removing automation'/'Removing enrichment' with no
past form, so those rows kept their present tense after completing.
* Name the target resource in the remaining tool titles
Table tools keep their operands nested under args and identify the table by
id, so their rows said 'Adding rows' with no hint where: enrichment now lifts
the nested args and resolves tableId against the cached workspace table list,
giving 'Adding rows to Runtimes', 'Adding column status in Runtimes',
'Reading views of Runtimes'.
Also: a block-schema read names the block instead of the file ('Loading
Slack', 'Loading Google Sheets tips'); browser type/insert show the text they
send, middle-ellipsized; downloads name the file; library-docs searches name
the library and query; knowledge-base searches include the query; generated
media names its output file; and diff_workflows, list_deployment_versions,
and publish_custom_block joined the workflow-name enrichment set.
* Bubble nested agents' tool calls into the parent's status line
The collapsed status only scanned a group's OWN tool items and skipped nested
agent groups, so a parent that had delegated froze on its last own tool while
its child did the actual work — the line described nothing that was running.
Status now walks the whole subtree: any tool at any depth counts, the most
recently started running one is shown, and the rest become the same '+ n'
overflow. With nothing running it falls back to the last tool at any depth,
so an idle parent still reflects where its subtree got to.
* Align nested tool call status rows
* Revert branch-local KB connector error-message edits
Restores apps/sim/connectors/ to staging state. Two copilot-focused commits
on this branch (
|
||
|
|
3b4d9e9149 |
fix(schedule): reconcile interrupted schedule executions (#6780)
* fix schedule execution recovery * simplify schedule recovery provider handling * fix(schedules): stop reconciliation rewriting healthy carriers and index its scan Follow-up hardening on the schedule recovery reconciliation. Reconciliation settled every carrier it examined, including ones that had already reached a terminal status and recorded their own outcome. Because the normal completion path never stamps the reconciled marker, each successful run's carrier was picked up on the next tick and rewritten: `completedAt` bumped, `error` nulled and `output` replaced with a recovery stub, all of which `GET /api/jobs/[jobId]` surfaces. A completed carrier whose execution log had aged out was additionally flipped to failed. Settle only carriers still in flight; a terminal one is owed schedule accounting and the marker, nothing more. The recovery scan matched no index. There was none on `async_jobs.updated_at`, and the unreconciled-terminal branch tested a jsonb extraction, so the whole OR fell back to a sequential scan and sort of `async_jobs` on every tick. Add the partial index, and spell the branch's status list and metadata key as SQL literals: Postgres cannot prove a parameterised predicate implies a literal index predicate, so a bound key would have left the new index unused. Irrecoverable carrier tombstones were exempted from retention with no secondary expiry, so the one class of row that can never reconcile grew without bound. Give them a longer bounded window instead. `WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN` had become a per-status budget when the stale-execution sweep gained its `redacting` pass, silently doubling the per-run cap. Share the budget across both passes, and add the matching partial indexes so the second pass keeps an index rather than seq-scanning. Also consolidates the carrier metadata keys, their predicates and the jsonb merge into one module -- the two routes were the writer and reader of the same keys with no shared symbol, so a rename type-checked clean while silently breaking retention. * fix(schedules): rotate deferred carriers and stop the redacting sweep failing live runs Addresses the review findings on the reconciliation hardening. A carrier whose accounting was deferred got no write at all, so it kept its `updatedAt` and stayed at the head of the `updatedAt`-ordered recovery batch on every tick, starving every other claimed carrier and never becoming eligible for retention. This was a regression from the previous commit: settling used to bump the timestamp for every examined row, and skipping the settle for already-terminal carriers removed the bump with it. A payload with no `scheduledFor` can never reconcile, so such a row pinned a batch slot permanently. Bump `updatedAt` unconditionally and keep only the reconciled marker conditional. The stale-execution sweep terminalized `redacting` logs on the execution deadline. That deadline bounds execution, while `redacting` covers payload masking after the run already finished -- so a run that used most of its budget entered redaction with the deadline due, and the sweep failed it five minutes later while the worker was still masking. Schedule recovery then read the log as a failed occurrence and counted a failure that never happened, even though the worker's terminal write later restored `completed`. Sweep `redacting` on the generic stale window only. `getScheduleNextRunAt` falls back to a daily cadence when a schedule has no cron expression. Deployment cannot persist such a schedule, so the branch is unreachable, but this change widened its use from failure recovery to every outcome -- log a warning when it fires rather than silently guessing a cadence. `executionDeadlineAt` was missing from the shared `workflowExecutionLogs` schema mock, so it read as `undefined` and assertions comparing against that column were trivially true. Add it. * refactor(schedules): drop vestigial recovery code and close a builder gap Follow-ups from a full re-read of the change. No behavior change except the removed dead code paths. The metadata merge stripped a `scheduleRecoveryBlocked` key on every write. That key has never been written by any shipped code -- it appears nowhere in staging and nowhere in history outside this branch -- so the strip guarded against a state that cannot exist, at the cost of an extra jsonb operation and a bind parameter on every reconciliation write. `processScheduleItem` set `carrierObservedOrLookupUncertain` immediately before returning on an ambiguous enqueue. The flag is only read from the surrounding catch block, which a normal return skips, so the assignment was dead and read as though it were load-bearing. Replaced with a comment stating why the occurrence is preserved. The stale-execution sweep carried two near-identical `jsonb_set` templates that differed only in their error expression, kept flat because the test mock renders nested SQL fragments as placeholders. The suite now has a recursive renderer, so the error expression is a named per-status value and there is one `jsonb_set`. Success was the only schedule outcome without a named update builder, which left `executeScheduleJob` using two idioms for the same guarded write and left the update shape untested. Add `buildScheduleSuccessUpdate` beside its cancellation and failure siblings, use it from both call sites, and cover it the way `buildScheduleCancellationUpdate` is covered -- a mutation of its `failedCount` reset previously passed every suite. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Waleed Latif <walif6@gmail.com> |
||
|
Vikhyath Mondreti
|
fcea50deb0 |
fix(provenance): stop size limits silently dropping secret provenance (#6867)
* fix(provenance): stop size limits silently dropping secret provenance A bundle-selection cap counted cells rather than rows, so a 25-column table insert lost secret provenance for every row past 400 — the whole batch was stamped unknown with nothing logged. The same number lived in the sender, the runtime type guard, and the route contract. Consolidate every provenance limit into one definition: an 8MB serialized envelope and 10,000 distinct secrets. The pair had been copied into seven modules under fourteen names, and several copies had drifted into bounding inputs — rows, cells, files, chunks — rather than the envelope. Remove every limit that could refuse a legal payload, add write-side cause logging and a workspace-visible audit entry when a read proceeds on unrecorded provenance, and repair the existing unknown rows. * fix(provenance): close a repair race and stop memory double-reporting The repair matched sidecars by the id its page captured, so a provenance-aware write committing between the snapshot and the delete had its fresh exact sidecar removed and its marker cleared behind it — a secret-bearing row left reading as legacy. The delete now re-checks status, which under READ COMMITTED re-evaluates against the writer's committed row so it no longer matches. Walk the candidate set by keyset over row_id. A page whose rows were all repaired concurrently clears nothing, and terminating on "cleared nothing" ended the walk with the rest of the backlog untouched. Memory reported unrecorded provenance twice, and counted records even when the surface was enforced — auditing a fail-open read that had actually failed closed. * fix(provenance): take the repair's locks in the writer's order The repair deleted the sidecar and only then updated its parent row, while mutateTableRowsWithSecretProvenance locks user_table_rows up front and upserts the sidecar inside the same transaction. Opposite orders, so an overlapping write deadlocked and Postgres resolved it by aborting either the deployment or somebody's table write. Lock the parent first, in id order, matching lockTableRows. Holding that lock is also what makes the status re-check decisive rather than racy: the writer commits its sidecar and its marker under the same lock, so once it is held the write is either wholly done or has not begun. |
||
|
Vikhyath MondretiandClaude Opus 5
|
9ee1b81b41 |
feat(custom-blocks): join cross-workspace runs into the caller's trace, and map blocks per environment (#6857)
* feat(custom-blocks): join cross-workspace runs into the caller's trace, and map blocks per environment Teams that orchestrate work across workspaces have two gaps that keep them on HTTP blocks instead of custom blocks: they cannot see what a custom block actually did, and a forked environment silently keeps calling the environment it was forked from. Debugging. A custom block is an invocation boundary — a published block is org-wide, so its internals must not reach every consumer by default. The child already writes its own log row in the source workspace, correlated to the invoking run; the trace existed, it just was not joined. The parent's span now carries only the child's opaque execution id, and `hydrateChildTraces` joins the child's spans at READ time, after authorizing the person reading against the child's workspace. Authorization follows the viewer rather than a flag set at publish time, re-evaluates on every read, and needs no second copy of the spans. Each hop of a nested chain is authorized against its own workspace. Boundaries left unexpanded — no access, no data, past a cap — say so, because a childless boundary span otherwise renders exactly like a leaf and a partial trace reads as a complete one. Live runs stream too, gated on `liveTraceViewerUserId`, which only surfaces with a single known authenticated viewer set. Chat deployments stream through the same callbacks and their consumer may be anonymous, so anything that does not opt in keeps the boundary shut. Child spans handed to such a viewer are projected through the CHILD's session: the invoking run's registry knows nothing about the publisher's secrets, so projecting there would leave a source-owner credential unmasked. They reach the live stream and stop — `createSpanFromLog` still refuses to persist them, which is what keeps read-time hydration the single authorization point. Environments. A fork inherits its parent's organization and `custom_block` is keyed `(organization_id, type)`, so a uat fork resolved to the same row and ran the prod workflow. Custom blocks become a fork-mappable resource, keyed by BLOCK TYPE — the rule every kind follows: key by whatever the workflow references, as `file` does with storage keys and `env-var` with names. A custom block is the only resource referenced by the canvas block's own type rather than a sub-block value, so the rewrite gets its own channel. Unmapped blocks keep the source type, because a type cannot be emptied without deleting the node; they surface as unmapped and block the promote, which is what stops uat from quietly invoking prod. Same-named environment copies now carry their source workspace, so an Access Control allowlist decision between three identical "Invoice Parser" rows is no longer a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(workspace-forking): let an explicit identity custom-block mapping resolve `remapForkBlockType` reported a mapping whose target equalled the source as unresolved, conflating "a mapping exists" with "the type string changed". Those are opposite states that produce an identical output `type`, and every caller uses the flag for the former — to decide whether the reference blocks a promote. The org-wide candidate list includes the source block, so binding an environment to the shared block is a normal pick. Under the old flag it raised `unmapped-custom-block` and refused the sync over a choice the user had explicitly made. The flag is now named `resolved` and reports mapping existence; whether the type moved is already visible from `type`. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): keep a streamed child's spans and markers off the parent's log Two leaks in the live-stream path, both the same mistake: treating a channel as viewer-scoped when it is actually persisted, so gating the stream on an authorized viewer bought nothing. `childTraceSpans` rode the block output to reach the stream. `filterOutputForLog` only dropped a hidden key when the block's own config declared it `hiddenFromDisplay` — true of the workflow block, never of a custom block, whose outputs are publisher-curated. The source run's spans therefore persisted into the parent's `span.output`, readable by anyone with parent-workspace access and never re-checked by `hydrateChildTraces`. A globally hidden key is now dropped at the top level, not only when nested, and `extractDisplayOutput` strips it again so no other producer can reintroduce it. The fan-out also called the invoking run's `onBlockStart`/`onBlockComplete`, which are persist-then-emit composites: they write block names and I/O into the parent's LoggingSession before reaching the stream. Those markers are keyed by the parent execution and outlive the per-viewer check entirely. Custom-block children now go through `liveStreamCallbacks`, the raw emit-only pair, and fail closed when a surface supplies none. Same-workspace workflow children keep the composites — they belong to the same run and their markers are legitimately the parent's. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): carry the emit-only stream sink into nested executions Routing custom-block events through `liveStreamCallbacks` forwarded the viewer id to the child but not the sink itself, so a nested hop cleared `canStreamCustomBlockToViewer` off the inherited id and then had nothing to stream through — `parentStreamSink` fell back to `{}` and live traces stopped at the first sub-executor. That hit a custom block nested inside a workflow block as readily as one inside another custom block. The sink now travels with the viewer id, and both are withheld together when streaming is not permitted. It is always the INHERITED chain, never `parentStreamSink`: for a same-workspace workflow block that is the persisting composite, so forwarding it would put a custom block nested inside one straight back onto the parent's progress markers — the leak the previous commit closed. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vikhyath MondretiandClaude Opus 5
|
521348b529 |
feat(secrets): record which secrets each run resolves, and surface it per secret (#6823)
* feat(secrets): record which secrets each run resolves and surface it per secret
Redaction stops a value at a boundary but cannot stop code that never emits it —
a Function block can print a key one character at a time and nothing ever matches
the secret. That is undecidable in general, so this adds the other half of the
posture: attribution.
Every run now records which configured secrets it actually resolved, under whose
identity, through which surface (workflow, Sim agent, MCP). The data already
existed in ResolvedSecretTraceRegistry.addActiveEntry and was persisted only for
paused runs; this persists it for every terminal path.
Execution logs cannot answer this. They store the whole available encrypted
environment rather than what a run referenced, they evidence a secret only where
value-matching redaction happened to fire, and they expire under
logRetentionHours — while "who has touched this key" outlives any single run.
- secret_usage: per-UTC-day rollup keyed by workspace, secret, scope, owner,
source, workflow, actor. A one-minute schedule touching three secrets would
otherwise write thousands of rows a day, which is also why this is not
audit_log. workflow_id/actor_user_id use '' sentinels rather than null so the
unique key works on Postgres 14 without NULLS NOT DISTINCT, and are not FKs:
they are historical facts, and an onDelete would rewrite a key column.
- secret_owner_user_id is part of the key. Two people can hold a personal secret
under one name and a shared personal secret resolves for a caller who does not
own it, so name and scope alone do not identify a secret. It is NOT the actor:
a scheduled run resolves the workflow owner's personal slice under the
workspace's execution actor.
- Direct environment reads are now detected in JS (TypeScript AST), Python
(tokenizer-checked) and shell (quote/heredoc-scanned), so a secret read as
environmentVariables['K'] or $K enters the run's provenance instead of going
unredacted. Each detector prescans for names that are actually configured
secrets before paying for a lex or quote-frame pass.
- Copilot integration tool calls are covered: resolveCopilotEnvReferences
substitutes {{SECRET}} into user-only params, which is a real use.
- See usage lives behind a credential-admin gate, using the same predicate that
reveals the value; members get a disabled chip explaining why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(audit): register the secret-usage route in the validation baseline
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): keep rollup metadata with its run, and stop shadowed bindings faking usage
Review round 1.
- record.ts: last_execution_id/last_trigger were assigned unconditionally while
last_used_at was chosen by greatest(), so two runs completing out of order split
one row between them — the newer run's timestamp beside the older run's execution
id, making "View log" open a run the row does not describe. Both are now guarded
on the timestamp actually advancing, so the row's metadata always belongs to the
run that owns its timestamp.
- javascript.ts: a local binding named environmentVariables (declaration, parameter,
destructured binding, or bare reassignment) made reads off the user's own object
look like mounted-secret reads. Any such binding now disables detection for the
file; the AST already had parent pointers, so this is a kind check during the
existing walk.
- python.ts: same class of bug with no parser available, so the rule is an allowlist
— every mention of the binding must be a literal subscript or .get(), otherwise
detection is off for the file. This also subsumes the cross-line attribute case
(other.\n environmentVariables['K']), which the previous space-and-tab look-behind
missed.
Under-reporting is the safe direction here: a trail that claims a use that never
happened is worse than one that misses a use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(db): format the generated migration snapshot
CI runs lint:check across every workspace; the drizzle-kit output in packages/db
had never been through biome, so the branch was green locally (where lint had
only been run inside apps/sim) and red on CI. Whitespace only — both files are
byte-for-byte identical once parsed, and drizzle-kit still reports no pending
schema diff against the reformatted snapshot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): detect every rebinding of the environment identifier, not just declarations
Review round 2. A bare `for (environmentVariables of rows)` has no declaration to
key off, so the previous check missed it and reads of the loop value were still
recorded as secret usage.
Rather than extend the hand-rolled node-kind list, this reuses the pair the same
file already applies to reject a placeholder in a write position:
isDeclarationIdentifier covers declarations, parameters, destructured bindings and
imports, and isWriteIdentifier covers every assignment operator, ++/--,
destructuring targets, and for-in / for-of initializers.
That also closes four forms neither the review nor the original check named:
logical (||=) and nullish (??=) assignment, and object and array destructuring
assignment. Six of the eight added cases fail against the previous check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): apply the rebinding rule to shell, and say when a run's log is gone
Review round 3, plus the docs that were left claiming the old behavior.
- shell.ts: a script that writes a configured name (API_KEY=local, export/local/
readonly, read, for, unset) expands its own value from that point on, not the
mounted secret, so recording it claimed a use that never happened. Every mention
of the name must now be a `$NAME` / `${NAME}` expansion, matching the allowlist
shape the Python detector already uses. Applied per name rather than per file:
JavaScript and Python shadow one object holding every secret, whereas rebinding
one shell variable says nothing about the rest.
- The usage trail deliberately outlives execution logs, so a row routinely names a
run whose log has been pruned. The read now left-joins workflow_execution_logs on
its unique execution_id and reports availability, and the panel renders the chip
disabled with the platform tooltip instead of linking into an empty Logs view.
Three states: no run to link, a run whose log is gone, and a live link.
- Docs said a direct environmentVariables/$KEY read does not activate masking,
which this branch changes. Corrected in credentials.mdx, function.mdx and the
logging FAQ, and the recognition limits are now written down: runtime-built
names, reassigned bindings, and reads that cannot be told apart from text.
Added a "See usage" section covering who can see it and why an empty trail
means "nothing recognized" rather than "never used".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): writing a name is not reading it, and a bare mention is not a rebinding
Review round 4.
- javascript.ts / python.ts: `environmentVariables.API_KEY = 'x'` and
`delete environmentVariables.API_KEY` touch the name without ever reading the
mounted value, but the detectors matched the member access and recorded a use
that never happened. JavaScript now asks the same isWriteIdentifier the
placeholder rewriter uses (its parameter is widened to ts.Node — the body
already walked generic nodes, so this is a type change, not a behaviour one)
plus a delete check; Python excludes a subscript followed by `=` and a `del`
target.
- shell.ts: requiring every mention of a name to be an expansion also fired on
text that binds nothing — a comment naming the key, or `echo "API_KEY=$API_KEY"`
where the literal is an argument rather than an assignment — and dropping those
cost masking on a genuine read. It now looks for actual writes: an assignment at
command-word position, a binding builtin, `printf -v`, or a `for` target.
The two directions are not symmetric, which is why this errs toward detecting
the read: missing a write records a use of a secret the script only had in its
environment, a misleading audit row and nothing more, since masking still
searches for the real value and will not find it. Over-detecting a write
suppresses masking on a value that does reach the log.
This also makes the code match what the docs already described — skipping after
a rebinding, not after any mention.
13 tests added; 11 fail against the previous code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): an update reads before it stores, and a del target may be parenthesized
Review round 5. The first of these is a regression from round 4.
- javascript.ts: reusing isWriteIdentifier to answer "is this a read" was wrong.
That predicate answers the rewriter's question — is this a target the
substitution must refuse — so it treats every assignment operator alike, which
is correct there and wrong here: `+=`, `||=`, `??=`, `++` and `--` all load the
current value before storing, so they are genuine reads and were silently
losing their masking. Only a plain `=` stores without reading. Replaced with a
purpose-named predicate, and isWriteIdentifier's parameter is narrowed back to
ts.Identifier now that nothing else needs it widened.
A test committed last round asserted the wrong behaviour for `+=`; it has been
corrected rather than left to pin the bug.
- python.ts: `del (environmentVariables['K'])` slipped past a check that looked
only at the characters immediately before the match. It now isolates the
enclosing logical line and tests whether that is a del statement, which also
covers `del((x))`, `del(x)`, `del a, x`, and a del after a semicolon.
12 tests added or corrected; 10 fail against the previous code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): stop excluding Python writes, which kept leaking in the unsafe direction
Review round 6. Greptile found that `del environmentVariables[environmentVariables['K']]`
had its inner access — which computes a key, so it is a genuine read — skipped
along with the delete, leaving that value unmasked.
The narrow fix was another textual rule. Instead this removes the write and delete
exclusions from the Python detector entirely, because they were optimizing the
wrong direction.
`resolvedSecretNames` feeds `outputSecretMatcher`, an exact-value matcher over the
output. Naming a secret the code never read costs nothing there: the matcher scans
for a value that does not appear. Failing to name one that was read leaves it
unmasked. The two error directions are therefore not comparable, and the
exclusions bought only audit-trail tidiness while every heuristic they needed has
so far leaked into the dangerous side — first a parenthesized target, now a nested
read. A `del` or an assignment is reported like any other access.
JavaScript keeps its exclusion: a real AST answers the question per node, with no
text to misread, and it has produced no such hole.
Net 30 lines removed from python.ts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): report recognized reads instead of proving they are not reads
Review round 7. Greptile flagged both directions at once — false usage from
reporting a write target, and unmasked secrets from the file-wide shadow flag —
so I traced what the signal actually drives before choosing.
The chain: the compiler's names feed outputSecretPlaintextsByName and the
exact-value matcher, NOT context.resolvedSecretNames, which starts empty. After
execution activateOutputSecretProvenance scans the output and adds only names
whose plaintext actually appeared; those become __resolvedSecretNames, which
tools/index.ts turns into recordResolved calls, which is what the usage trail
reads.
So a compile-time false positive produces no usage row on the ordinary path — it
only hands the matcher a value the code never emits. It does produce one on the
!projection.safe fallback, where the system already over-approximates by design.
A false negative, by contrast, keeps the value out of the matcher entirely, so a
genuinely read secret is never masked on any path.
That asymmetry decides it, so every "prove this is not a read" mechanism is gone:
- javascript.ts: the file-wide shadow flag. A helper declaring its own
environmentVariables discarded genuine reads of the mounted binding everywhere
else in the file — Greptile's security finding, and real.
- python.ts: the allowlist requiring every mention to be a subscript or .get().
Same hole: passing the dict to a function suppressed unrelated reads.
- shell.ts: the rebinding check. It had the same hole in a form nobody flagged —
`echo "$API_KEY"; API_KEY=local` dropped the first read, which is of the real
secret.
What stays is the question of whether the text is code at all — strings, comments,
single quotes, quoted heredocs — plus the receiver check that `other.environment
Variables['K']` is a different object, and JavaScript's node-precise write/delete
exclusion, which cannot suppress a read elsewhere.
Net 215 lines removed across the three detectors and their tests. Docs updated:
the rule is now stated as reporting rather than proving, and that See usage may
occasionally list a secret the code had available but did not read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(secrets): drop the last write-vs-read special case
`environmentVariables` is a plain object deserialized from the run payload
(route.ts:206), not a handle on the stored secret. Assigning to it changes
nothing outside the sandbox and is discarded when the run ends, so separating a
write from a read bought almost nothing while leaving JavaScript as the one
language still trying to prove a read is not a read.
Every language now follows the same rule: report a recognized read of a
configured secret name. The only exclusions left are facts rather than
inferences — the text is not executable (string, comment, single quote, quoted
heredoc), the receiver is a different object, or the name is not statically
knowable.
Docs note that assigning to the binding does not edit the secret.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(secrets): ship only the fields the trail actually shows
Five fields crossed the API and reached no reader: usageDate, firstUsedAt,
actorEmail, workflowId and actorUserId. The panel renders the timestamp, the
trigger, what used the secret, the actor's name, the run count and the run link;
everything else was projected, serialized and discarded.
first_used_at is dropped from the table as well. Nothing read it, and inside a
per-day bucket "first used that day" says nothing next to "last used that day" —
so it was a column written on every run for no question anyone asks. The upsert
loses its least() with it. Migration regenerated; the identifier columns behind
the joins stay, they simply are not returned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): report referenced code secrets, not only ones that surface in output
The Function route activated a secret's provenance — and therefore its usage row
and downstream masking — only when the exact value appeared in the result,
stdout, or error. That gate made the trail miss silent use entirely: a key that
authenticates an API call and is never echoed reported nothing, and so did the
founding scenario of this feature, a key exfiltrated character by character. The
innocent run that echoed a key got a row; the run worth catching did not.
Activation now follows the referenced set the compiler already computes: resolved
{{KEY}} bindings plus recognized direct reads, filtered to configured values —
the same set the unsafe-projection fallback already activated. An extra name only
hands the output matcher a value that never appears; configured-but-unreferenced
values are still never included. The output-scan activation path and its surface
helper are deleted rather than kept alongside.
One old test pinned the gate ("does not activate a referenced secret that does
not cross the Function result"); it now asserts the reverse, with the reasoning
attached. Two new tests pin the char-split exfiltration and the silent API-call
case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): shell escaping is backslash parity, not adjacency
Review round 8. `\\$API_KEY` is an escaped backslash followed by a LIVE expansion
— bash prints `\` plus the value — while `\$API_KEY` is an escaped dollar and
stays literal. Checking only the character adjacent to `$` read every even run as
escaped, dropping a real read from usage and masking alike; verified against
bash before fixing.
The scanner now counts the run of backslashes before the `$` and skips only odd
runs, the same parity rule logicalLineEndAfterContinuations in this file already
applies to line continuations. Six-case parity table added; the three even-run
cases fail against the previous check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): recognize destructured environment reads
Review round 9. `const { API_KEY } = environmentVariables` delivers the value by
name with no property- or element-access node in the AST, so the member-access
walk missed it entirely — and a missed read leaves an emitted value unmasked,
the dangerous direction.
The AST walk now also recognizes the declaration form (shorthand, renames,
defaults, string-literal keys), the assignment form ({ KEY } = env), and a
...rest element — which names no key but takes every value, so it reports every
configured name; the alternative left `const { ...all } = env; return all`
entirely unmasked. A computed key stays unrecognized, the same runtime-name
boundary as a computed subscript, and a receiver that is not the bare identifier
is not attributed.
Nine cases added; the six positive ones fail against the previous walk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): one receiver rule for destructured reads, parentheses included
Review round 10. Two accurate findings, folded into a generalization instead of
two more special cases:
- A parameter default (function f({ API_KEY } = environmentVariables)) and a
binding-element default are the same by-name delivery as a variable
declaration. The detector now keys on the ObjectBindingPattern itself and
checks its parent's initializer, so every declaration position follows one
rule instead of per-kind arms.
- Parentheses group without changing the receiver, so (environmentVariables) is
unwrapped before the identifier check — in the destructuring arm AND the
member-access arm, which had the same hole unreported.
Declined the for-of-over-array-literal finding: the receiver there is a
container, not the environment object, and following data flow through
containers has no fixed point — the same documented boundary as aliasing and
computed keys. A test pins the boundary so it reads as chosen, not missed.
Eight cases added; the seven receiver-rule cases fail against the previous code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): a dot in prose is not a qualifier, and a literal computed key is a subscript
Review round 11. Both findings were implementation-narrower-than-rule, fixed by
consulting authorities the detectors already had rather than adding new ones:
- python.ts: the receiver walk crosses whitespace so a parenthesized `other.` on
a previous line is seen — but it landed on a comment's final period
(`# Load the value.`) and discarded the genuine read on the next line. The
landing position is now checked against the same lexer ranges that filter the
candidates, which is also why the receiver check moves after lexing.
- javascript.ts: `const { ['API_KEY']: key } = environmentVariables` is the
element-access rule in pattern position, so a computed key holding a string
literal resolves like a literal subscript; any other computed key keeps the
runtime-name boundary a computed subscript already has.
Eight cases added; the comment-period case and all three literal-computed-key
cases fail against the previous code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Waleed
|
9328e66d69 |
feat(account): let users delete their own account (#6831)
* feat(account): let users delete their own account Adds a GDPR self-serve account deletion path: a preflight that reports what deletion would remove and every reason it would be refused, and a confirmed delete that erases the account and everything only it can reach. Deletion refuses while the account is still entangled rather than reassigning its content. Most tables reference user.id with ON DELETE CASCADE, and those cascades do not distinguish content in the account's own workspace from content it created inside somebody else's, so each blocker names the existing action that untangles it (leave the workspace, leave the organization, cancel the plan) — all of which already hand work over on their own tested paths. * fix(account): make deletion atomic and re-check privacy at delete time Reorders the teardown so nothing irreversible happens before the deletion is certain: anchors are handed over first (the fallible step, while everything is still recoverable), the workspace and user deletes now share one transaction, and the object-storage purge runs only after that commits. The workspace delete also re-checks inside the transaction that each workspace is still private, so a membership granted between the preview and the delete aborts the whole thing instead of destroying the new member's access. * fix(account): close deletion gaps found in review - Run the whole teardown in one transaction. The billing and ownership handovers now take the caller's transaction, so a refused deletion can no longer leave a workspace reassigned for a deletion that never happened. - Fail closed on a subscription read error. getHighestPriorityPersonalSubscription defaulted to returning null, which read as "no plan" and would have erased an account Stripe was still billing. - Erase the account's profile picture. It is personal data under our own storage prefix; an external provider avatar is left alone. - Enforce the storage purge cap while collecting keys rather than after, so an oversized account cannot exhaust memory before the cap applies. |
||
|
Waleed
|
51f1d45eff |
perf(table): collapse sequential round trips on the table read and write paths (#6808)
* test(table): characterize the single-row route before migrating it
The single-row surface (GET/PATCH/DELETE) had no route-level tests despite
being the hottest table write path. Pin the behavior it emits today so the
migration onto the shared internal route builder is verifiable rather than
hopeful.
Covers status codes, body shapes, ISO-8601 timestamp serialization, the
access level each method demands, collaborator invocation, and the
dual-caller wire keying — session callers pass column ids through untouched
while internal-JWT callers translate names to ids in both directions.
Verified to fail: mutating the wire translator, the deleted count, and the
workspace-ownership guard each turn the corresponding tests red.
* feat(table): model wire keying and actor attribution on row write use cases
The row write use cases assumed every caller speaks column names. That holds
for /api/v2, /api/v1 and the Copilot tools, but not for the first-party grid
or the internal /api/table routes, which address cells by stable storage id.
Feeding id-keyed data through the name remap drops every key it does not
recognise — a storage id names no column name — so the write would store
nothing and still report success.
Make the wire an explicit, required property of the input rather than an
assumption. `dataKeying: 'names' | 'ids'` sits alongside `strictWrite` and is
required for the same reason: a new write surface must state which contract it
publishes. Strictness now means the same thing on either wire — an unknown
column id is refused exactly as an unknown column name already was.
Single-row writes also gain optional actor attribution, so the acting tab can
skip refetching its own write. It is optional and absent by default, so every
existing caller keeps broadcasting to all subscribers as before. Only the
single-row create, update and delete paths accept it; a batch write is not
reconciled locally by the actor and must still refetch.
The attribution pin moves with the behaviour: what selects the actor-scoped
signal is no longer which file calls it but which surface supplies an actor,
so that is now what the test pins.
Verified to fail: ignoring the keying discriminator, and dropping actor
attribution, each turn the corresponding tests red.
* perf(table): remove two round trips from every single-row update
A single-cell PATCH spends far more time in sequential round trips to a remote
Postgres than in the UPDATE it issues. Prepared statements are disabled for
PgBouncer transaction mode, so every await is a full Parse/Bind/Execute.
Two of them were avoidable.
getRowById issued the row lookup and its executions sidecar in series, but the
sidecar is keyed on the row id the caller already supplied, so it never
depended on the lookup. Issuing both together makes it one round trip. A miss
now pays one redundant sidecar read, which is the rare path and costs no extra
wall time.
The uniqueness probe ran whenever the table had any unique column, passing the
fully merged row, so editing an unrelated cell re-probed every unique column —
its own transaction plus one query per column. It is now scoped to the columns
the patch actually writes. A merge cannot newly violate uniqueness on a column
it leaves alone: that value is the one already stored, and it satisfied the
constraint when it was written.
Sized before changing: few tables declare a unique column, but write traffic
concentrates in the ones that do, so this is the larger of the two savings in
practice.
Verified to fail: reverting the probe scoping turns the covering test red.
* perf(table): start the workspace load with the table load when a workspace is asserted
resolveActiveTableContext ran two sequential round trips: load the table,
then load the workspace it turned out to live in. When the caller asserts a
workspace the second input is already in hand, so both can start together.
What makes that safe is unchanged: requireTable still compares the table's
canonical workspaceId against the assertion and reports a mismatch as
not_found. The table outcome is inspected first and unconditionally, so any
path that returns has proven the assertion equal to the canonical id, and a
failing workspace load can never replace the concealing not_found. A final
identity check on the loaded context restates the invariant at the point of
return, so even with the first check removed the function cannot hand back a
foreign workspace. Promise.allSettled keeps the discarded branch from
surfacing as an unhandled rejection.
With no asserted workspace the path stays sequential — the table load is what
reveals which workspace to load, so there is nothing to start early.
One existing assertion changed: it required the workspace load not to have
been issued yet on a mismatched assertion, which is internal sequencing rather
than caller-observable behaviour and is definitionally untrue once the loads
overlap. The observable half is kept, and two timing tests now cover the
sequencing directly.
Verified to fail: removing either mismatch check, reading the workspace
outcome first, and swapping allSettled for bare awaits each turn the
corresponding tests red.
* perf(table): read a table and its latest job in one round trip
getTableById issued the table SELECT and then awaited latestJobForTable, so
every table request paid two sequential round trips. With prepared statements
disabled for PgBouncer transaction mode every await is a full round trip, and
this loader is on essentially every table route.
The job read cannot be skipped: a table's reported rowCount is the stored
count minus the job's pendingDeleteRemaining, so dropping it would overstate
the count during a pending delete and could wrongly reject inserts as over
capacity. An opt-out flag would have made that a caller's trap. Instead the
job is read in the same statement, as a correlated jsonb subquery in the
select list — the select-list form of a LEFT JOIN LATERAL, which is what
drizzle can type here. Output is unchanged for every input.
latestJobForTable is deleted rather than left dangling: getTableById was its
only caller, and keeping it would have carried a third copy of the
exports-excluded / newest-started_at / limit-one rule. mapJobRow is now
exported so the batch path and the lateral share one implementation of the
doomedCount and pendingDeleteRemaining logic. The batch DISTINCT ON path used
by the list endpoint is untouched.
Verified to fail: dropping the export filter, reversing or re-keying the sort,
dropping the limit, dropping the correlation, loosening either doomedCount
condition, and removing the rowCount subtraction each turn tests red. Dropping
the lateral from the projection initially survived, because the shared db mock
returns queued rows regardless of predicate; a projection assertion now
covers it.
* chore(table): drop the unused single-row GET contract
Authored as groundwork for migrating the internal row routes onto the shared
builder, which is not in this change. An exported contract nothing consumes is
dead code, so it lands with the migration that needs it instead.
* fix(table): resolve strict id-keyed columns through getColumnId
assertKnownColumnIds read column.id directly, but a column id is optional —
pre-backfill columns have none and are stored under their name, which is why
getColumnId exists and is what every other consumer of the schema uses. Such columns still
exist, so a strict id-keyed write naming one would have been refused as
unknown.
Latent today: no surface yet combines dataKeying 'ids' with strictWrite. Fixed
before one does.
Verified to fail: reading column.id turns the covering test red.
* refactor(table): apply review findings from the quality pass
Four parallel reviews (reuse, simplification, efficiency, altitude) converged
on the same set. Applied:
Removed actorClientId entirely. It had no supplier anywhere in the repo, so
every call reached signalTableRowsChangedByActor(id, undefined), which is
byte-identical to the broadcast it replaced — three optional fields, three
verbatim doc blocks and a pin test asserting the empty set, all inert. It
belongs with the route migration that supplies an actor. The attribution pin
is restored to its original form.
The uniqueness probe was only half-narrowed: the patched column list was
computed and then discarded, and the probe re-derived every unique column from
the full schema. It now receives only the columns the patch touched, so a
table with several unique columns runs one query instead of all of them.
assertKnownColumnIds hand-rolled an id index and duplicated the sibling
assert's message verbatim. It now reuses buildColumnNameById — which already
keys by getColumnId, so the legacy pre-backfill column case is handled by the
shared helper rather than by a special case here — and both asserts share one
message builder.
The job field list had become two copies, one drizzle-checked and one an
unchecked sql<T> cast that could silently return undefined for a renamed
field. The lateral now derives its jsonb pairs from JOB_PROJECTION, which
satisfies Record<keyof LatestJobRow, Column>. That compile-time guarantee
replaces the runtime drift test it makes redundant.
Also: hoisted the id index out of the batch loop to match the names path,
dropped a never-supplied parameter, narrowed an over-broad parameter type,
removed a dead timer cleanup and its now-unused import, replaced dynamic
re-imports with the static one already present, hoisted a repeated stub, and
documented why filters need no keying counterpart and how laxness differs
between the two wires.
Verified to fail: removing a field from JOB_PROJECTION breaks the build in two
places.
* test(table): share one table-definition fixture factory
buildTable was copy-pasted into 13 route test files under app/api/table, each
a near-identical TableDefinition literal. A required field added to that type
would have failed 13 files individually.
packages/testing already owned createTableColumn and createTableRow but no
definition factory, and — as it turns out — did not export any of them from
the barrel, so they were unreachable from @sim/testing. Adds
createTableDefinition beside them and exports all three.
The fixture type is a structural stand-in rather than TableDefinition itself,
because packages/* must not import from apps/* — the same approach the
existing factories in that file already take.
No assertion changed. Call sites that varied a field pass it as an override;
the four files with several call sites hoist a shared options const and spread
it so each call still gets a fresh object.
* perf(table): project only the job field the row count needs
Both job reads selected the whole payload jsonb, but mapJobRow reads exactly
one number out of it, and only for a running delete. The payload also carries
the delete job's filter and an unbounded excludeRowIds array, and the latest
non-export job is read on essentially every table request — a table that once
ran a large delete would ship that id list on every read, forever.
LatestJobRow.payload becomes doomedCount, extracted in SQL. Both readers share
JOB_PROJECTION so one edit reaches the batch DISTINCT ON and the correlated
subquery alike; the compile-time constraint widens to Column | SQL rather than
being dropped.
Behaviour is identical. `->` keeps the value jsonb, which postgres-js decodes
through its built-in JSON.parse handler, so it arrives as a number with no
boundary coercion. A null payload, a payload without the key, a non-object
payload and an explicit JSON null all collapse to the same `?? 0` the previous
optional chain produced.
Sized honestly before claiming a win: payloads are small in practice today, so
this is defensive rather than impactful — it removes an unbounded growth path,
not a measured cost.
Verified against a real Postgres, not just the mocked driver: the
generated correlated subquery returns doomedCount 12 for a delete job, null
for an import job, and a null row for a table with no job.
Also from the review pass: re-homes the strictWrite explanation onto
rowWriteOptions, where six {@link} references now point; records why
replaceProjectedWireRows carries no keying discriminator; notes the one case
the uniqueness-narrowing invariant does not cover; pins the lax id-wire
passthrough with a test; and renames a parameter that misled once only its
keys were read.
* chore(table): drop two comments the code already says
One restated the identifier below it (checkUniqueConstraintsDb), and one was a
section-divider banner in the factories barrel, which the repo's comment
convention rules out. Everything else that survives is a why the code cannot
express: round-trip rationale, sql.raw input safety, the narrowing invariant
and the one case it does not cover.
* refactor(table): move the internal row routes onto the application boundary (#6809)
* refactor(table): move the single-row route onto the application boundary
The hottest table write path authorized in its own handler and queried the
database from the adapter — the two things a surface adapter must never do. It
now declares itself with defineInternalJsonRoute against the readRow, updateRow
and deleteRow use cases: 127 lines instead of 274, with no db import, no
drizzle import and no checkAccess.
Doing that surfaced why the violation existed. Write-provenance resolution
needs the canonical schema to map a caller's column key to the storage column
it certifies, and the adapter could only do that because it was already loading
the table illegally. The envelope is now split along the real seam: the adapter
reads the header and payload field, which is transport, and the use case
resolves the selections against the canonical table, which is domain.
That split has to preserve a distinction the defaulting logic would erase. An
internal caller that sends no envelope stays deliberately untracked; defaulting
it to an exact-empty stamp would certify "this write introduced no secrets" on
a runtime write that may well have introduced some. Only an interactive caller
certifies exact-empty, over the storage columns its write actually persists.
Two further changes fell out of it:
present() now receives the same { principal, input } pair its sibling hooks
responseHeaders and finalizeResponse already got. This route serves a session
and a workflow execution on one path and owes them different column keyings, so
rendering per caller kind is presentation rather than domain. That was a gap in
the builder, not a special case for this route.
tableRowWireSchema describes what the single-row routes actually return. The
contract claimed a full TableRow, carrying the executions sidecar and Date
objects — true of the list and query routes, and never true here. The hand-
rolled handler was never checked against its own contract, so the drift was
invisible until the builder started validating it.
Wire changes, both deliberate and both narrower than before: a cross-tenant
table now conceals as 404 where the old blanket handler answered 403, while an
in-workspace role denial still answers 403. Nothing in hooks/queries/tables.ts
branches on either.
Verified to fail: forcing one keying, dropping the actor, pre-resolving the
envelope, certifying an untracked internal write, and skipping the bundle
completeness check each turn the covering tests red.
* refactor(table): move the upsert route onto the application boundary
Same shape as the single-row route: declares itself against upsertTableRow,
hands the provenance envelope over unresolved, and derives its column keying
from the principal rather than assuming one.
The keying and presentation helpers the two routes shared are now in row-wire
beside the translators they wrap, so a third route does not restate them.
Two response details changed on purpose. The row now carries `position`, which
the use case always had and this route alone omitted — every other single-row
response already returned it, and the contract now describes one shape instead
of three. The upsert result also carries read-back provenance, which the route
previously assembled for itself.
The surface had no route-level tests; it has six now, covering both caller
keyings, both operations, and the envelope handover.
* refactor(table): move the enrichment-detail route onto the application boundary
The last internal adapter that queried the database for itself. It now runs
through readTableRowEnrichmentDetail, a new use case that shares
tableOperations.readRow — reading a cell's cascade breakdown is a projection of
the same row under the same role, not a second semantic operation.
Its tests move to the same seam and gain one the old suite could not express:
a cross-tenant table now conceals rather than confirming it exists.
* fix(table): mirror the storage rule when keying write provenance
storageKeyByWireKey mapped an unrecognised id-keyed key to null, but
rowDataToStorage persists that key when the caller is not writing strictly. A
cell would have been written with no provenance recorded, under a stamp still
marked complete — the same failure the bundle completeness check exists to
prevent, arriving through the keying map instead of the selection set.
The two wires genuinely differ and the code now says so: the name path drops an
unrecognised key, so it maps to null; the id path stores what it is given, so
every key it sends is a storage key.
Unreachable today, since no delegated surface uses id keying and a session
bundle is refused earlier. Fixed because the function's stated invariant — that
it mirrors how the row data itself is normalized — was not true.
Also corrects the delegated principal fixture in these tests, which used a kind
that is not in the Principal union, so the subject-id branch was never actually
exercised. It is now, and the scope check is asserted to receive the acting
principal's own subject id.
Verified to fail: restoring the schema-based lookup turns the covering test red.
* fix(table): restore executor access to the migrated row routes
The migration swapped checkSessionOrInternalAuth for the delegation policy,
and that broke every Table block call to these endpoints in two ways at once.
The old policy accepted a legacy internal token. The new one requires a
delegation token, which the executor only mints when the tool asks for it —
and none of the four table row tools did, so get/update/delete/upsert row
would each have failed with a 401. The knowledge tools already declare it,
because their routes migrated first.
Even with a valid token the operations denied the caller: readRow, updateRow,
deleteRow and upsertRow ran under a policy whose delegatedServices is
['copilot'], so the executor got a 403. They now use the tool-facing policy
that already existed for the group operations.
Neither was visible to the route tests, which mock the auth policy wholesale —
so the gap is closed at the layer that actually decides: one test pinning that
each tool requests delegation and that its operation admits the executor,
mutation-verified against both failure modes.
Also fixes findings from the review pass: the read surfaces no longer load an
executions sidecar none of them put on the wire (two readers rather than a
flag, so a caller cannot silently read an empty one); the provenance name
index is built once per batch instead of once per row; the uniqueness comment
now names the concurrent-insert race as well as the retro-added constraint;
and the presenter context gets NoInfer plus a note that the v2 builder passes
something different.
* fix(table): keep the lock on a 423 and pin the remaining wire changes
The rows error policy was built on the concealment base rather than the
lock-aware one, so a TableLockedError fell through to the generic handler and
the response lost its `lock` field — the only thing that tells a client which
lock to clear. A row write is exactly as lockable as a group mutation, so it
now shares that base.
Also pins the two wire changes the review found undocumented: a mismatched
workspace assertion answers 404 rather than 400, which is a superset of the
cross-tenant concealment already intended, and an unclassified failure answers
the builder's shared "Internal server error" rather than the old per-route
text. Both are consistent with the ~80 routes already on this builder; they are
asserted so they read as decisions rather than drift.
Verified to fail: reverting the policy base turns the lock test red.
* refactor(table): apply the quality pass
Four parallel reviews (reuse, simplification, efficiency, altitude). The
highest-value finding cut against the branch's own purpose: the rows error
policy sat in the barrel-exported route-policies module, so its import of the
1,200-line row use-case graph was paid by every one of the ~28 table routes
that can never throw a row error — the barrel's import cost went from ~1.1s to
~1.7s. row-route-policies.ts already existed for exactly this and is
deliberately not re-exported; the policy now lives there with its v2 sibling.
The upsert path still loaded an executions sidecar no surface puts on the wire,
and did it inside the write transaction, holding it open for a discarded
result. The read path got that fix earlier; the write path next to it did not.
rowKeyingForPrincipal fell through to name keying for anything that was not a
session. The operation policy admits API-key principals, so the first one to
reach these routes would have had every id-keyed cell dropped and the write
reported as successful. It is now an exhaustive switch over the two kinds the
auth policy yields — which immediately failed three tests using a principal
kind that exists nowhere in the repo, so those fixtures are real now too.
Also: reuses toWireTimestamp and createUnknownTableRowSecretProvenance instead
of re-inlining them; shares one helper for the provenance choice the update and
upsert use cases both make; keeps one canonical actorClientId doc with two
cross-references; merges two maps keyed by the same string into one; stops
round-tripping the principal through the legacy AuthType enum; hoists the
presenter function type out of both conditional branches; drops a subsumed
test; and freezes the shared locks fixture so a mutating test cannot poison its
siblings.
|
||
|
Theodore Li
|
8fb571ac54 |
feat(credentials): add managed credential groups (#6697)
* feat(credentials): add managed credential groups * fix(audit): sync credential group mock * fix(credentials): serialize enrollment revocation * fix(credentials): isolate managed delegation * fix(credentials): serialize invitation lifecycle * fix(credentials): preserve enrollment lifecycle * refactor(credentials): migrate groups to application boundary * fix(credentials): serialize enrollment readiness * fix(credentials): preserve completed reconnect state * fix(credentials): revalidate policy before grant persistence * fix(credentials): prioritize expired invitations * fix(credentials): redirect unavailable oauth starts * fix(credentials): clarify managed oauth boundaries * fix(settings): complete feature flag test mocks * fix(credentials): clarify enrollment actions and entitlement errors * fix(credentials): preserve entitlement failure reasons * fix(credentials): refine managed oauth flow * fix(lint): use optional chain for pagination |
||
|
Vikhyath Mondreti
|
fa394e5e07 |
fix(execution): resolve secrets against the acting principal, not the workflow owner (#6690)
* fix(execution): resolve secrets against the acting principal, not the workflow owner * fix(execution): resolve anonymous public-API runs as the workspace billing account * fix(execution): propagate run identity across dispatch paths and scope public runs to workspace secrets |
||
|
Waleed
|
264d4f3469 |
feat(canvas): add a setting to turn off auto-focus when clicking blocks (#6685)
* feat(canvas): add a setting to turn off auto-focus when clicking blocks Clicking a block animates the camera to center it, which zooms in far enough that you lose sight of the rest of the workflow. Add an "Auto-focus on click" preference (on by default, so existing behavior is unchanged) that keeps the camera still on click. Also re-record the auto-connect and canvas-error-notification tooltip previews and re-encode all three at a smaller size. * fix(canvas): keep click-marked framing when auto-focus is off, recut tooltips Gating the whole click branch on the setting also skipped the userFocusedWorkflowIdRef write, which is what stops <ReactFlow onInit> from running fitView over the user's framing. That would have blown away the framing of exactly the users who turned auto-focus off to keep it. Mark the workflow as user-framed on any plain node click and gate only the camera move. Crop the auto-focus preview to the recording's viewport center so the blocks are legible at the 240px width Tooltip.Preview renders at, and trim the 2.45s of empty lead off the error-notification preview. * docs(canvas): correct useAutoFocusOnClick scope to clicks only The TSDoc claimed the preference also gated arrow-key navigation, which calls focusBlockInView without consulting it. State the click-only scope and why arrow-key navigation and block creation are excluded. |
||
|
Waleed
|
068422b3ff |
refactor(audit): derive updatedFields through one shared helper (#6604)
* refactor(audit): derive updatedFields through one shared helper Six copies of Object.keys(updateData).filter(k => k !== 'updatedAt') across four files decided, independently, which columns an audit row reports. The exclusion set is an audit convention, not a local detail, so it moves to @sim/audit as auditUpdatedFields and the exclusion becomes a single edit. The admin organizations route evaluated the expression twice in one handler and filed it under the metadata key `fields` while every other site uses `updatedFields`, so any consumer filtering on updatedFields silently missed org updates. It now computes once and uses the shared key; nothing reads metadata.fields. auditMock carries the real implementation rather than a stub, since callers under test derive their audit metadata through it. The two suites that hand-roll an @sim/audit factory source it from there. * test(audit): pin the testing mock's copy of auditUpdatedFields @sim/audit devDepends on @sim/testing, so the mock cannot import the real helper without closing a package cycle. Assert parity from the audit side instead, where the dependency already runs the safe direction, so a change to the exclusion convention cannot leave mocked callers validating behavior the deployed helper no longer has. |
||
|
Waleed
|
7c05e36049 |
fix(api): close five defects found auditing the v2 migration against main (#6575)
* fix(tables): stop a column retype from nulling empty-string cells A type conversion rewrote every cell holding '' to null. Main only nulled a blank the target type could not read; '' is a real stored value that both string and json columns accept, so string->json and json->string silently destroyed those cells. Worse on a required target: countEmptyCells matches only a missing key, SQL NULL, or '[]', so '' passes the required guard and the rewrite then wrote null behind a constraint that had just succeeded. The per-cell decision is now the pure retypeCellRewrite, restoring main's rule: null a blank only when the target cannot read it, otherwise coerce. * fix(execution): release the concurrency slot when a group cancel is refused The stop-the-work effects (durable Redis abort record, queue-job cancel, in-process abort) all fire before the workflow-group sidecar is consulted, and none can be undone. When the sidecar refuses the claim we throw a conflict, which skipped releaseExecutionSlot and stranded the plan concurrency reservation until it expired. Every conflict return is a terminal-or-absent state - a missing log row, a log already completed or errored, or a terminal cell - so a refusal never means the run is still executing. The slot is released before the throw, keeping the exact success && !isPausedCancellationPath predicate rather than a blanket finally that would free reservations for live runs. * fix(uploads): recover an ambiguous PUT instead of discarding the object Main recovered an upload whose bytes committed but whose response was lost, via a verify endpoint. The session client retries the PUT instead, but every provider now signs a create-only precondition, so the retry returns 409/412, is classified non-retryable, and the session aborts - deleting the object that had already landed. A transient blip on the final ack cost the whole upload. A conflict on a retry attempt is now treated as our own earlier PUT having committed, and completion proceeds. That is safe because completeUploadSession independently verifies the object through assertObjectIdentity, which rejects on uploadId mismatch before anything durable is registered. A first-attempt conflict still fails loudly. * fix(folders): enforce the workspace folder ceiling on the create path Readers bound the active path index at MAX_FOLDERS_PER_WORKSPACE and throw once a workspace exceeds it, but POST /api/folders reached createFolder, which has no maxFolderRows field and never counts. A workspace could therefore be driven past the ceiling, after which the 27 capped read sites failed on a state the product had allowed. createFolder now asserts room inside its transaction, right after the mutation lock, so the count cannot be raced. The refusal is a typed conflict rendering 409 with an actionable message rather than a 500. The check counts rows directly instead of loading the path index, so an already-over-cap workspace gets a clean refusal rather than a read error, and no reader gained a cap. folderMutationStatus also gained the payload_too_large mapping it was missing, which had been rendering a delete-cascade cap breach as an unexplained 500. * fix(skills): route internal skill writes through the shared use cases The internal route made the workspace authorization decision itself, never consulting the skills operation policy, never loading canonical workspace context, and recording an audit entry with no operation id or actor projection. v2 and Copilot already went through the use cases; only this surface did not. GET/POST/DELETE now authenticate, parse, call the shared use case, and present. Request and response shapes are unchanged. Two behavior changes fall out: a write against a deleted workspace is now refused with 404 rather than accepted, and permission-denial text matches the rest of the platform. Legacy internal-JWT auth is dropped because no principal kind expresses that caller and nothing calls it: the whole repo references /api/skills only in two comments, no tool declares an internalRoute to it, and the executor reads skills through a direct listSkills call rather than over HTTP. * fix(folders): enforce the workspace ceiling on the remaining create paths Folder duplication, admin workspace import, and workspace forking all inserted folders without consulting the ceiling that 27 read sites enforce, so any of them could leave a workspace whose reads then fail. Each now asserts room for the rows it is about to add rather than one at a time: duplication measures the whole subtree up front, forking counts its bulk insert, and import counts per segment because that is genuinely one row. assertFolderCollectionHasRoom gained an additionalRows notion for the bulk case, and short-circuits when nothing is being added so an over-cap workspace still reads and still syncs. Duplication deliberately does not take the folder mutation lock. Holding it across the copy would block folder creation workspace-wide for an unbounded time - there is no cap on workflows per subtree and duplicateWorkflow runs sequentially - and narrowing it is impossible because an advisory transaction lock cannot be released early; splitting the transaction would leave a half-copied tree on failure. A rare few-row overshoot near the ceiling is the better trade, and it matches what forking already does. A test asserts the lock is absent so re-adding it is a visible decision. Admin import gained the transaction and lock it never had. Its folder-full refusal escapes the per-workflow result list, because a full tree is a property of the workspace and would otherwise be buried as N failures behind a 200. The fork and promote routes had no catch at all, and withRouteHandler only classifies HttpError, so a refusal rendered as an opaque 500 - twice over, since drizzle wraps the throw. Both now project a classified conflict as 409 and rethrow anything unclassified. * fix(uploads): bound the signed PUT lifetime and advertise its real expiry A single-PUT transfer was signed for the whole 24h upload-session TTL, because expiresAt was reused as both the session lifetime and the signing lifetime. Multipart part URLs in the same file kept 1h, and the pre-migration presigned route signed every PUT for 1h, so the widening was unintended rather than a policy change. No provider clamps below 24h. The PUT presign is now clamped at the provider boundary by a shared UPLOAD_URL_TTL_MS, which the part-URL path also uses so the two cannot drift. An expired PUT URL is deliberately not recoverable: unlike multipart, which re-signs per part call because its progress is durable, a PUT is not resumable, so an expired URL and an interrupted PUT have identical recovery. Nothing leaks, since every provider signs a create-only precondition. Clamping alone would have made the contract lie: the URL would die an hour before the session's advertised expiresAt, with nothing telling an integrator why the 403 happened. The PUT transfer now carries its own expiresAt, mirroring the multipart part-URL field. It is provider-dependent on purpose - cloud transfers report the clamped signature expiry, while the local data plane has no signature and admits against the session, so reporting an hour there would have been a new inaccuracy in the other direction. * chore: delete the dead presigned-upload and skill-adapter paths The presigned upload routes and the internal skills adapters were both replaced during the v2 migration, leaving their implementations behind with no callers. Removed generatePresignedUploadUrl and verifyPresignedUploadReceipt with their three provider helpers, QUOTA_EXEMPT_STORAGE_CONTEXTS and the types it orphaned, and the performCreateSkill/performUpdateSkill/performDeleteSkill adapters with recordSkillEvent and statusForSkillOrchestrationError. Each was verified unreachable across apps, packages, scripts and ee, including barrel re-exports and string access, not just direct imports. recordSkillEvent needed the closest look, since deleting an audit writer can silently drop coverage. The use cases declare the same action, resource, and description, and the framework adds the operation and actor the old helper lacked; recordAudit back-fills actorName and actorEmail from the user table when both are omitted, so the one field the helper passed is not lost. The self-hosting architecture doc described a directUploadSupported flag on an endpoint that no longer exists, and now describes the upload-session flow that replaced it. * refactor(folders): keep the cheap resource facts out of the schema graph Reading a folder resource type's label or its lock support meant importing folderResourceConfig, which imports the db schema for every table it serves and from there reaches lib/table/service, the executor, and the tool registry. That mattered as soon as lib/folders/queries needed a label: queries is reached from workspace-file-manager, which is reached from the files and chat pages, so one import edge put roughly 4,700 modules into those page graphs and broke the tool-registry boundary audit. Labels and lock support now live in a leaf module that imports only a type, and config composes them so there is still one source of truth. The three folder routes that pulled the whole config in for a single boolean read the leaf instead. * fix(skills): apply an upsert batch in one transaction The internal skills route looped the batch, calling an independently committing use case per item. A rejection on a later item left the earlier ones written and audited while the request reported failure - the compound-mutation rule in CLAUDE.md exists for exactly this. No new transaction plumbing was needed: upsertSkills already wraps its whole item loop in one db.transaction, so the partial commit came from calling it N times rather than once. upsertSkillBatch now validates and per-skill authorizes every item before issuing a single write, and createSkill and updateSkill became thin wrappers over it so v2 and Copilot keep one authority for the rules. The compound operation declares the read floor that skills.update already used, and the use case additionally authorizes skills.create when any item lacks an id, still ahead of every write. A read-only member who is a skill editor keeps their edit, and creates are not authorized more loosely than before. Audit projects one entry per committed skill, and analytics moved after the commit so nothing is reported for a rolled-back item. Note metadata.operation for these writes is now skills.upsert rather than skills.create/update; the action field still carries the distinction. * fix(security): close two disclosure gaps and finish the slot-leak fix The payer-pool gate only covered the workspace branch. A personal API key that omits workspaceId takes the account branch, where getHighestPrioritySubscription resolves an organization subscription from any member row regardless of role - so a plain member read the organization-wide credit and storage pool by dropping one query parameter. The account branch is now gated by the same authority, and the storage pool is not queried when it may not be disclosed. Forcing that branch self-scoped instead would have downgraded plan, period, and status, which is what a member needs to see whether the org is blocked. GET /api/v1/logs/executions/[executionId] emitted the workflow snapshot raw, carrying password sub-block values and oauth-input credential ids. It now shares the sanitizer the v2 read already used, extracted so there is one implementation rather than two. Env-var references are still preserved. cancelWorkflowGroupExecution itself was unguarded, so an unexpected throw from its transaction escaped ahead of every release site - the same reservation leak this branch set out to close, still open on the adjacent path. It now releases through the shared predicate and rethrows, because a failed transition means the cell state is unknown and a success-shaped answer would be a lie. The comment claiming the abort record cannot be taken back was false and now states the real reason: a refusal is always a terminal-or-absent state. * fix(v2-api): cover every persisted run status and every capped body The workflow-runs endpoints carried the same omission the logs contract had: the execution logger persists redacting, the run schemas did not list it, and because validation is whole-response one such row returned 500 for an entire page. Both schemas now derive from PersistedWorkflowExecutionStatus behind the same AssertNever gate, so a future status is a type error rather than a production 500. The single-run read keeps its extra queued value, which only it can observe. That schema was also serving as the run-list status filter. Widening it would have accepted a filter value the application input cannot express, so the reported set and the accepted filter are now separate schemas. Routes declaring maxBodyBytes without payloadTooLargeResponse fell back to a bare string with no error code and no private cache header. Rather than patch the four, the default moved into the builders - all three had the hole - which covers 58 body-bearing handlers, and a route override still wins. The five per-route overrides that merely restated the default are gone. Also documents the 413 on the one knowledge route that has a real body cap, adds the rollout gate's 404 to the last v2 operation missing it, and rewords the nextCursor description, which read as though every list were a full-set list. * fix(api): make the shared traits and lifetimes single-sourced The folder resource-traits leaf composed labels into config but restated lock support independently, so the routes reading the trait and the orchestration reading the config could disagree about which resources lock. Config now composes both, and supportsLocking is required rather than optional so a new resource type cannot silently omit it. The upload commit claimed the PUT clamp and the multipart part URLs shared a constant and could not drift. That was true only of the advertised expiry - the three provider signers each hardcoded an hour, so changing the constant would have moved what we advertise while leaving what we sign, recreating exactly the mismatch the clamp removed. Each provider now receives the lifetime in its own unit from the one constant. Table restore hand-rolled its status map and returned the driver message verbatim at 500, leaking the failed statement and its bound parameters - the same defect this branch closed at nine other sites. It and import-csv now use the shared projection; import-csv's result type also had to carry the lock the classifier already set, so a 423 can name it. Deletes v2RowWriteError, which had no callers and would have rendered a locked table as 400 by discarding the 423 it was handed. |
||
|
Siddharth Ganesan
|
cb2809001e |
fix(mship, desktop): fix bugs (#6572)
* fix bugs * fix tests * fix(tests): stabilize archive and client boundary coverage |
||
|
|
263e3ca67e |
improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec (#5273)
* v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * improvement(external-endpoints): v2 versions with clean signatures + updated docs * feat(usage): accept X-API-Key on usage-logs list + export /api/users/me/usage-logs and /export now use checkHybridAuth — the same auth /api/users/me/usage-limits already accepts — so external monitors can read summary.bySourceCredits (the source breakdown of usage-limits' aggregate currentPeriodCost) instead of estimating Copilot spend by subtraction. Workspace-scoped keys are pinned to their own workspace's slice of the ledger: the filter defaults to the key's workspace and an explicit mismatch 403s. Both endpoints documented in openapi-core.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(billing): dedicated v2 usage endpoints; keep internal usage routes session-only Replaces the earlier X-API-Key enablement on /api/users/me/usage-logs with a dedicated public surface, so the internal Billing-settings endpoints can evolve with the UI while external monitors get a stable versioned contract: - GET /api/v2/billing/usage — current-billing-period summary with bySourceCredits (the source breakdown external monitors need to watch e.g. Copilot consumption without estimating by subtraction), plus limitCredits and plan - GET /api/v2/billing/usage/logs — cursor-paged credit ledger in the v2 envelope - workspace-scoped keys are pinned to their own workspace's slice; personal keys read the account ledger The public wire is credits-only: usage-logs rows now carry a hasCost boolean instead of dollarCost (the Billing UI only needed the >0 signal), and the rateLimit block is removed from the usage-limits response and docs (deploy-modal tab relabeled accordingly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(docs): validate OpenAPI specs against the Zod contracts in CI The specs in apps/docs are hand-authored because they carry what Zod never defines — error envelopes, status codes, prose, examples — so they can't be generated; check:openapi validates them instead: - spec integrity: $refs resolve, operationIds unique, 2xx documented, no orphaned component schemas - v2 conventions: every /api/v2 operation documents 401 + 429 and every 4xx/5xx resolves to the canonical { error: { code, message } } envelope - contract cross-check: contracts are auto-discovered from lib/api/contracts/v2 (each carries its method + path); doc<->contract coverage both ways, query/body/response field diffs via z.toJSONSchema - examples: documented request/response examples must parse with the matching contract's actual Zod schemas First run caught real drift, fixed here: 16 stale orphaned schemas in the core spec, the v2 billing ops referencing v1-shaped error components, deploy/rollback examples missing the required nullable lifecycle keys, CreateTableBody missing folderId, a legacy-grammar delete-rows example, and four knowledge document ops missing their required workspaceId query param. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(docs): recursive field diff in check:openapi + the deep drift it found A mutation test showed the doc<->contract field diff only compared top-level properties, so a typo inside the { data } envelope passed. The diff now descends through matching object properties and array items (both sides must expose a property set — passthrough contracts and prose-only docs end the descent instead of false-positive), with the Zod JSON-schema root doubling as the $defs context. Deep drift it immediately caught, fixed here: select-column config (options/multiple) missing from every tables column schema, AddColumnBody hand-rolling a third column shape (now composed from ColumnInput, with position/workflowGroupId as the per-op extensions the contracts actually admit), chunking strategyOptions undocumented, and the deployment lifecycle fields (activeDeployment/latestDeploymentAttempt) missing from DeploymentState. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(security): close the triggerType rate-limit bypass on workflow execute Caller-supplied triggerType flowed unchecked into preprocessExecution, whose checkRateLimit default turns OFF for 'manual'/'chat' — so any API-key caller, and any anonymous public-API caller billed to the workspace owner, could execute unthrottled by sending {"triggerType":"manual"} (async runs also skipped the worker-side check via admissionCompleted). External callers may now only send the redundant 'api' value; internal JWT callers ('workflow'/'mcp') are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * refactor(execution): extract enqueue/status/cancel into shared libs Prepares the v2 execution surface: handleAsyncExecution's queue logic moves to lib/workflows/executor/enqueue-execution.ts (slot/claim semantics encoded in a discriminated outcome, not HTTP statuses), the execution-status read to execution-status.ts, and the order-sensitive cancel machinery to lib/execution/cancel-workflow-execution.ts. The v1 routes re-render identically — their suites pass unmodified. Also: preprocessExecution gains rateLimitCounter ('sync'|'async') and its 429 now carries code RATE_LIMIT_EXCEEDED + retryAfterMs (previously indistinguishable from the concurrency 429 and Retry-After was discarded); and the duplicate cancel contract in contracts/logs.ts is unified on the full 5-value reason enum — its narrower copy made requestJson throw a client ZodError when cancelling a paused HITL run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): callable execution service + structured error classifier executeWorkflowService composes the same libs the v1 route holds inline (call-chain guard, execution-id claim, LoggingSession, preprocessing, deployed-state load + file-field processing, timeout-bound executeWorkflowCore, output hydration/compaction) for the deployed-state caller class — the seam the v2 execute route and in-process internal callers share, making the HTTP endpoint syntactic sugar. classifyExecutionError stops discarding the block context that buildBlockExecutionError already attaches at throw sites: failed runs now yield {message, code, blockId, blockName, blockType} with a stable append-only code enum (TIMEOUT/CANCELLED/USAGE_LIMIT_EXCEEDED/ INVALID_INPUT/BLOCK_EXECUTION_FAILED/CHILD_WORKFLOW_FAILED/ OUTPUT_TOO_LARGE/EXECUTION_FAILED), so callers route on error class instead of substring-matching messages — the single place raw errors are interpreted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): POST /api/v2/workflows/[id]/execute Thin route over executeWorkflowService: X-API-Key or anonymous public-API auth (sync/stream only for anonymous), strict body with body-flag async (no mode headers on v2), SSE passthrough for stream, and the execution resource response — executionId always present, in-band run failures are status:'failed' with the structured {message, code, blockId, blockName, blockType} error, sync timeout is status:'failed' + TIMEOUT instead of v1's 408, and a Response block's payload stays inside output (authors never control response status/headers on this origin). Async debits the async bucket and the 202 statusUrl points at the v2 executions resource. Adds CLIENT_CLOSED_REQUEST/SERVICE_UNAVAILABLE to the v2 error codes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): v2 executions status + cancel with queued backfill GET /api/v2/workflows/[id]/executions/[executionId] is the single status URL for sync and async runs: before the async worker writes the durable log row, status is backfilled from the job queue (deterministic job id) as 'queued'/'running' — closing v1's 202-to-pickup 404 window — and failed runs carry the structured error object. POST .../cancel renders the shared cancellation lib in the v2 envelope with the tightened 5-value reason enum. Both authenticate via the shared resolveV2WorkflowAccess (X-API-Key, authz masked as 404, allowPersonalApiKeys honored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): workflow tool + MCP bridge run in-process workflow_executor (workflow-as-agent-tool) short-circuits in executeTool through WorkflowBlockHandler — the same invocation boundary canvas child workflows use — mirroring the deployed_block_executor precedent. The MCP serve bridge calls executeWorkflowService directly instead of fetching its own execute endpoint; deployment-version pinning, MCP response-size rejection, and the actor override become typed options instead of header sniffing. Both callers drop the double admission slot and duplicate top-level log row the HTTP hop cost, and failed child runs now surface the structured error + child executionId so parents and MCP clients can route on error class and hand providers a reproducible handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(infra): CORS + CSP coverage for the v2 execute path /api/v2/workflows/:id/execute gets the same wildcard-origin, credential-free CORS policy as v1 (the default credentialed policy would block browser API-key calls and open a cookie CSRF surface) with X-Sim-Stream-Protocol allowed and no X-Execution-Mode (async is body-selected on v2), plus the COEP/COOP/CSP header block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(ui): deploy modal + copilot advertise the v2 execute surface All 20 API-tab snippets move to POST /api/v2/workflows/{id}/execute with the nested {"input": ...} body, async as the "async": true body flag (X-Execution-Mode gone), status polling against the v2 executions resource, the third tab renamed Usage and pointed at /api/v2/billing/usage, and {data} envelope unwraps in the printed responses. Fixes the latent baseUrl derivation (endpoint.split('/api/workflows/')) that would have silently built garbage URLs under a v2 endpoint, and deletes dead code (exampleCommand across 3 sites, getAsyncExampleTitle). Copilot deploy/manage/serializer endpoint builders and the api_trigger bestPractices example follow (the latter also drops its hardcoded staging host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * docs(api): document the v2 execution surface Adds execute, execution status, and cancel to openapi-v2-workflows.json with the structured ExecutionError schema (append-only code enum + block attribution) and the ExecutionResource contract, documenting the rules that differ from v1: modes are body-selected, a failed run is HTTP 200 with status 'failed', an executionId always means data (never the error envelope), queued status is visible immediately, and Response-block payloads stay inside output. Registers the three pages in the generated workflows meta.json and bumps the route-count baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): gate the whole /api/v2 surface behind one flag; UI stays on v1 Every v2 route now runs exactly one check immediately after auth — v2ApiGateError — and answers 404 when the `v2-api` flag is off, so the surface is invisible until it is deliberately rolled out. The gate is keyed on userId only: a workspace/org-keyed check would have to read membership for a caller-supplied id before authorization runs, and its 404-vs-403 split would leak cohort membership (the trap the per-domain table gate worked around by running late). The two executions routes inherit it from the shared access resolver; the tables-specific gate is removed so no route checks twice. `tables-v2-api` stays, now gating only the internal predicate-grammar route /api/table/[tableId]/query — note v2 tables routes move to the unified flag, so enabling them is a `v2-api` decision now. Reverts the deploy modal, copilot handlers, and api_trigger example to the v1 execute endpoint: v1 works unchanged, and the UI must not advertise a surface most users would get a 404 from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(executor): restore child-cost aggregation dropped by the staging merge Staging's custom-block rewrite deleted `aggregateChildCost` from workflow-handler.ts, and git merged that file cleanly — but this branch's workflow-tool-runner.ts, added for the v2 execute migration, still imports it. A silent semantic conflict: no marker, broken build. Taking staging's rewrite is correct, so the helper is defined locally in its one remaining consumer rather than resurrected in the file staging just rewrote. Same four lines over the still-exported `calculateCostSummary`, so a failed child workflow keeps billing the hosted-key spend it consumed instead of reporting $0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation (#6134) * refactor(orchestration): move the shared error contract out of lib/workflows OrchestrationErrorCode and statusForOrchestrationError are the contract every lib/[resource]/orchestration module returns against, but they lived inside the workflows module, so resource-neutral code (lib/folders) already had to import from a workflow path. Moved to lib/core/orchestration/types. Adds a 'locked' class mapping to 423. Both tables and workflows have a lock that forbids a mutation, and each caller was translating that to a status itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation Column update was implemented four times — the UI route, v1, v2, and the copilot table tool — each calling the same column services but owning its own guards, error mapping, and audit. The copies had drifted, and the drift was the bug: v2 was missing both guards, only the copilot copy minted stable option ids, and only v1/v2 audited. performUpdateTableColumn, performDeleteTable, and performDeleteTableRow now own that logic; all ten call sites reduce to auth, parse, call, render. The guards are asserted once in lib/table/orchestration rather than four times against four routes. Behavior this consolidates, previously true on only some paths: - The typeChanging guard. updateColumnType early-returns on an unchanged type and drops any options sent with it, so restating the current type alongside new options silently discarded them. v2 had no guard at all and, since its contract shares v1's body schema, accepted options and ignored them. - The select-unique guard. Each write is its own locked transaction, so a rename or type change paired with a constraint write that is going to fail commits first and then throws, half-applying the schema change. - Stable select-option ids. Cells reference the option id, so an edit that re-sends an option by name has to reuse it or every cell holding it is orphaned. Only the copilot path did this; normalizeSelectOptionsInput moves to lib/table/select-options and now covers every caller. It preserves a supplied id, so it is a no-op for the fully-formed options the HTTP contracts accept. - required forwarded into the type and options writes, so a conversion validates against the constraint the same request is setting. - An audit on every successful update. The UI route and the copilot tool emitted none. - Single-row delete through the row service. v2 did a raw db.delete, skipping assertRowDelete and deleteOrderedRow, so a delete-locked table returned 200 and the row-count bookkeeping never ran. - The delete actor handed to deleteTable, which audits only when a row was actually archived. v1 and v2 omitted it and audited themselves outside that check, emitting TABLE_DELETED for a no-op delete of an archived table. Failure classes come back as OrchestrationErrorCode; v2 renders them through a new v2ErrorForOrchestration, mirroring statusForOrchestrationError on the v1 and UI surfaces, so a given failure maps to the same status everywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(tables): bind the column-update tests to the orchestration function The base's route tests assert which column service each payload reaches — the behavior that now lives in performUpdateTableColumn. They mocked the `@/lib/table` barrel; the orchestration module imports the service directly, so they mock that too and keep asserting the same thing through the extracted implementation. The orchestration tests move onto the base's semantics: writes address the stable column id, a rename rides inside the write it accompanies rather than running first, and the currency guards replace the non-select options guard the service now owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(copilot): drop the column-type import the delegation made dead Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): move the audit log out of the table service `lib/table/service.ts` wrote its own audit rows, so whether an operation was audited depended on which function a caller reached for rather than on a user having performed it. That is what let v1 and v2 audit a no-op delete, and what made `deleteTable`'s optional `actingUserId` double as an audit opt-out flag. Worse, most sites fell back to `actingUserId ?? createdBy`, so an unattributed call was logged against the table's *creator*. The copilot `mv` path passed no actor at all: renaming someone else's table recorded them as the renamer. Audit now lives in the orchestration functions — performDeleteTable, performRenameTable, performMoveTableToFolder, performUpdateTableLocks — and the services just write. Internal callers (folder cascade, import rollback) keep calling the service and are silent by construction rather than by remembering to omit an argument. Two services now return what the audit needs: `deleteTable` reports whether it actually archived a row, so a repeat delete logs nothing; `updateTableLocks` returns the before/after locks, since only the locked write can observe the transition its description names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(tables): restore audit provenance and conflict status in orchestration Moving the audits into the orchestration functions dropped three things the routes had been carrying, and added one the orchestration now owns twice. - The v1 and v2 column-update routes passed `request` to `recordAudit`, so their audit rows recorded the caller's IP and user-agent. The orchestration function had no way to receive it. Every table orchestration function now takes an optional `OrchestrationRequestContext` and every HTTP route forwards it; the copilot and VFS callers, which have no request, omit it. - `classifyTableMutation` matched `TableConflictError` on "already exists" appearing in the message and reported it as `validation`, turning the UI route's 409 on a duplicate table rename into a 400. It now matches the type, the way `performRestoreTable` already did. - `captureServerEvent` ran on every delete while the audit was gated on a row actually being archived, so a repeat delete of an archived table still reported `table_deleted`. Both now hang off the same evidence. - The copilot delete path kept its own `captureServerEvent` from when the service did not emit one, double-counting every copilot table delete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * fix(tables): say which type a no-op column update restated A copilot `update_column` payload whose only content was the column's current type used to return success with the live schema, while the v1, v2, and UI routes rejected the same payload with "No updates specified". Delegating to `performUpdateTableColumn` unified them onto the routes' rejection — correct, but the message tells the caller its request was empty when it named a type. The orchestration function now reports the same thing `updateColumnType` reports when it loses this race concurrently: the column is already that type, re-issue without the type change. An empty payload still reads "No updates specified". Drops the copilot's `outcome.table ?? tableForUpdate` fallback with it — the comment described the no-op that can no longer reach that line, and a success always carries a table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * refactor(tables): classify failures by type instead of by message text The table module decided HTTP statuses by searching error messages for phrases. `VALIDATION_MESSAGE_FRAGMENTS` and `ROW_WRITE_ERROR_PATTERNS` held 32 substrings between them, and fifteen more lists were inlined in routes — 83 matchers over 17 files, each its own copy of the guesswork and already drifted apart. It made message wording load-bearing: `TableRowLimitError`'s own doc comment noted that its text had to contain "row limit" for a route to answer 400, and adding "already exists" to a rename message silently demoted a 409 to a 400 (the bug fixed one commit ago, by adding another special case). Services now throw `OrchestrationError`, which carries the transport-neutral `OrchestrationErrorCode` the layers above already speak. Classification is one `instanceof` in `orchestrationErrorResponse` (UI + v1) and `v2CaughtOrchestrationError` (v2). Every pattern list is gone. Wording is free to change; an unclassified error still becomes a generic 500, which is what an unexpected fault should be. `asOrchestrationError` walks the `cause` chain rather than testing the caught value directly: drizzle wraps a throw raised inside a transaction callback in a `DrizzleQueryError` whose own message is the failed SQL, so a bare `instanceof` would drop every failure raised inside `withLockedTable`. That is the same reason `rootErrorMessage` had to dig for a root cause before. Three throws stay bare `Error` deliberately — `Table ID mismatch`, `Workspace ID mismatch`, and `Failed to build upsert conflict predicate` are internal invariants no consumer classified, and they keep falling through to a 500. `Insufficient capacity` was in the pattern list with no producer anywhere in the codebase. Status changes, all deliberate: - `'forbidden'` joins the code union so the table-row-limit ceiling keeps its 403; without it this refactor would have flattened it to 400. - import-async's table-limit rejection: 400 -> 403, matching the two other create routes it had drifted from. - Renaming a table to an invalid name: 500 -> 400. `validateTableName` messages don't contain "Invalid", so no matcher ever caught them. - Restoring a table that isn't archived, or into an archived workspace: 500 -> 400. - A duplicate *column* name stays `validation`/400 rather than becoming a 409 like a duplicate table name. Both v1 and the orchestration have always answered 400 for it; changing a published status is not this refactor's job. The twelve tests that changed were asserting the substring mechanism itself, constructing plain `Error`s with magic strings. They now assert the real contract, plus new cases pinning that identical wording carrying no classification stays internal and keeps its message off the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials (#6150) * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials * fix(api): correct credential role, skill permission bar, MCP url identity, and custom-tool conflict mapping * fix(api): align credential mutation gating, provider-outage status, and unique-violation conflicts * fix(api): close unique-violation, revival, orphan-write, and env-rename gaps * fix(api): treat every provider-outage code as unavailable on create and update * fix(credentials): use the shared outage predicate on the session update path * fix(contracts): anchor the predicate double-cast annotation to the cast `check:api-validation:strict` counted 9 unannotated double-casts against a baseline of 8, failing CI. The predicate leaf schema was annotated, but the annotation sat above the declaration while the checker anchors on the line carrying the cast — five lines below, at the close of the object literal. The scanner walks back at most three lines and stops at the first non-comment one, so it hit `value: z.unknown().optional(),` and never saw the reason. Splitting the object schema from the cast puts them adjacent, so the existing reason binds. No behavior change — the cast, the schema, and the reasoning are unchanged. Also lowers the rawJsonReads ratchet 6 -> 5 to match the current count, which had drifted down; leaving it high lets a removed raw read silently come back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(skills): point the orchestration error contract at its moved module #6150 branched before #6134, so skill-lifecycle.ts imports @/lib/workflows/orchestration/types — the module #6134 moved to @/lib/core/orchestration/types. Git merged a file deletion on one side with a new file referencing it on the other: no textual conflict, broken build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(knowledge): make lib/knowledge/orchestration the single implementation (#6154) * refactor(knowledge): make lib/knowledge/orchestration the single implementation Knowledge base create was implemented four times — the internal route, v1, v2, and the copilot tool — and the orchestration around the shared write had drifted. Extract it the same way lib/table/orchestration was: services write, orchestration decides which writes run, guards them, audits them, and returns a transport-neutral failure. Behavior converged, not preserved: - One chunking default (DEFAULT_CHUNKING_CONFIG). The agent defaulted minSize to 1 against the API's 100, so identical input produced differently-chunked knowledge bases depending on who created it. The agent path now chunks at 100. - Every successful mutation is audited inside the orchestration function. The copilot tool called recordAudit zero times, so agent-created knowledge bases, document uploads, updates and deletes left no audit trail at all. - Failures classify by class, not by message text. The knowledge service errors are OrchestrationError subclasses and storage-quota rejections throw a shared StorageLimitExceededError, replacing four separate message greps for "already exists" / "does not have permission" / "storage limit". delete_connector reported the opposite of what happened. It reached the route through an internal HTTP self-call that sent no query string, so the route's keep-documents default always applied while the agent told the user the documents had been removed. The self-call is gone — all four connector operations run in-process — and the orchestration returns the real counts. Also: - OrchestrationErrorCode gains 'payload_too_large' (413 / PAYLOAD_TOO_LARGE). Without it, dropping the storage-limit message match would have regressed the documented 413 on knowledge base create and document upload to a 500. - messageForOrchestrationError renders a route's own wording for an unclassified fault, so a driver's message no longer reaches the client on a 500. - v1 and v2 knowledge base update now forward actorUserId, which the service requires for a workspace move; both omitted it. - The connector DELETE route reads deleteDocuments through parseRequest. Its contract declared z.boolean(), which would have rejected the string a query param actually is. - Drop the 409 from POST /api/v2/knowledge/{id}/documents in the OpenAPI spec. Nothing on the upload path throws a conflict; it was only ever reachable by the message match this change removes. Behavior change worth noting: a v1/v2 PUT carrying only the workspaceId scope field and no actual updates now returns 400 rather than 200 with the unchanged knowledge base. Deliberately deferred: document update remains internal-only. Extracting performUpdateKnowledgeDocument makes exposing it on v1/v2 a contract and a route away, but that is a new public surface rather than part of this consolidation. * fix(knowledge): make connector create atomic and stop flattening failures Review round 1 on #6154. - Resolve the billing payer before the connector is committed, not after. A malformed attribution header rejected post-commit left a live connector behind a 500, and a retry created a duplicate plus duplicate sync work. Manual sync resolves before writing its audit for the same reason. - Let the source-config validator carry its own failure class. Collapsing every rejection to `validation` flattened the connector PATCH route's 401 (stale stored credential) and 409 (missing workspace context) into a 400. - Add `unauthorized` to OrchestrationErrorCode. It is the class that 401 was already expressing on this route, and the v2 vocabulary already had UNAUTHORIZED; only the shared union was missing it. - Report a knowledge base that exists but failed to archive as failed, with the reason, rather than as not found. The copilot delete loop folded every non-not-found failure into `notFound`, telling the user it was never there. - Route copilot failures through the same message helper the HTTP surfaces use, so an unclassified fault's raw text (a driver's failed SQL) no longer reaches the agent verbatim while the UI and public APIs get the generic wording. * feat(api): expand the public v2 files surface (#6160) * feat(api): expand the public v2 files surface Adds folder support, rename/restore, move, bulk archive, share, and content replace to /api/v2/files, so managing files by API no longer stops at upload + download + archive-one. Routes are thin: auth -> parse -> perform* -> serialize. Share and content replace get their orchestration extracted first so the session routes and the public ones cannot diverge on the effective-authType resolution, the EE public-sharing gate, or the storage-quota classification. Presigned upload stays session-only: presign does an advisory quota check and the real debit happens in the separate register step, so a caller that never registers leaves unaccounted bytes with no reaper. The buffered multipart path debits inside uploadWorkspaceFile's own transaction. * fix(files): classify folder and content failures instead of 500ing them Bugbot round 1. The v2 routes map errorCode straight to a status, so every manager failure that arrived unclassified became a 500 for what is really a caller-fixable 400 or 404. - Folder manager throws OrchestrationError: missing target/folder -> not_found, reparent cycle / self-parent / restore-into-archived-workspace -> validation. - File manager does the same for the in-transaction 'File not found' paths that the earlier pass missed. - updateWorkspaceFileContent's outer catch re-wrapped everything in a bare Error, which stripped the class off StorageLimitExceededError and the new not_found alike. It now rethrows a classified failure untouched and attaches cause to the generic wrap, so asOrchestrationError can still walk the chain. - Every remaining perform* gained the asOrchestrationError branch. - renameWorkspaceFile returned the pre-update read, so the v2 PATCH reported a stale updatedAt; it now returns the timestamp it actually wrote. Docs: upload auto-suffixes a duplicate name rather than rejecting it, matching the in-app uploader. The description claimed 409 and was simply wrong. * fix(files): surface a failed upload read-back as the real error getWorkspaceFile swallows a query failure and returns null unless throwOnError is set, so a transient blip on the post-upload read reported as 'file could not be read back'. Distinguish the two: a real null after a just-committed write is an invariant break, a query failure is itself. * revert(api): drop the dedicated v2 file-folder routes File folders already live in the shared folder table as resourceType 'file' (#6045 cut them over, #6051 dropped workspace_file_folders), and the remaining file-specific folder machinery is being folded into the generic folder engine. Publishing /api/v2/files/folders/** would pin that transitional split into a public contract we'd then have to keep or break. Files stay folder-aware — folderId/folderPath on the projection, folderId on upload, and the move route — because a folder id is a folder.id and survives the unification untouched. Folder management belongs on /api/v2/folders once that surface serves resourceType 'file'; until then there is no v2 way to enumerate file folders, which is the deliberate gap. The orchestration classification fixes stay: the internal routes and the copilot file-folder tools still call those perform* functions. * fix(files): classify upload failures instead of matching their wording Bugbot round 2. uploadWorkspaceFile had the same outer-catch rewrap that updateWorkspaceFileContent did, so a blown storage quota reached the route as a bare Error and the v2 handler recovered the status by substring-matching the message. Any rewording silently demoted a 413 to a 500. - uploadWorkspaceFile rethrows a classified failure untouched and attaches cause to the generic wrap. - FileConflictError is now an OrchestrationError('conflict'), so a duplicate name classifies like every other conflict. Its 'FILE_EXISTS' discriminator had no readers and is gone; the instanceof checks elsewhere still hold. - The v2 upload handler uses v2CaughtOrchestrationError, dropping all three string matches. Also documents that bulk-archive is best-effort: unknown or already-archived ids are skipped rather than failing the call, and deletedItems is what actually happened. That asymmetry with the single-id DELETE was undocumented. * feat(api): add search, filtering, and sorting to the v2 list endpoints (#6189) * feat(api): add search, filtering, and sorting to the v2 list endpoints One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts: `search` (case-insensitive substring on the resource's natural name field), `sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship rather than inventing a third dialect alongside the Logs filters and the Tables predicate grammar. Every filter and sort is pushed into SQL. GET /api/v2/files previously read the whole scope and sorted/sliced it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Cursors are stamped with the sort they were minted under, so replaying one under a different sort is a 400 instead of silently duplicated or skipped rows. * fix(api): validate v2 cursor key values and compare timestamps at ms precision Two review findings, fixed at the root by making a keyset key own its cursor codec instead of hand-writing a decoder per sort. Cursor key values are caller-controlled, and matching the sort stamp and key count was not enough: an unparseable timestamp or a non-numeric size reached the query as an Invalid Date or NaN and surfaced as a 500. Each key now type- checks its own value and rejects a cursor it cannot hold, which both routes render as the documented 400. Timestamp keys now order and compare on date_trunc('milliseconds', col). Postgres keeps microseconds and defaultNow() populates them, but a cursor value round-trips through a millisecond-only JS Date — comparing the raw column against the truncated value re-admitted the page's own last row, duplicating it and stalling pagination outright at a page size of one. Reachable today via workspace_files.updated_at, which insertFileMetadata leaves to defaultNow(). * feat(api): complete the v2 workflows resource with versions and CRUD (#6184) * feat(api): complete the v2 workflows resource with versions and CRUD Adds version listing/detail plus create, update, and delete to the v2 workflows surface, which previously covered only execution and deployment. - GET /api/v2/workflows/[id]/versions — cursor-paginated, newest first - GET /api/v2/workflows/[id]/versions/[version] — version + pinned state - POST /api/v2/workflows, PATCH and DELETE /api/v2/workflows/[id] All six delegate to the existing orchestration and persistence helpers; no new domain logic. * fix(api): check folder containment before lock state; reject malformed version cursors assertFolderMutable walks a folder's ancestor chain without filtering on workspace, so inspecting it before containment let a caller tell a locked folder in someone else's workspace (423) from a nonexistent one (400). Create and update now assert containment first, matching the ordering import-workflow.ts already uses. A version cursor that decodes to JSON without a numeric version filtered every row out and returned an empty page with nextCursor null, which reads as a clean end-of-list. Malformed cursors are now a 400. * refactor(api): page workflow versions in the persistence helper listWorkflowVersions read every version row and the route filtered and sliced the result in memory, so the response was bounded but the query was not. It now takes optional limit/afterVersion, turning the cursor into a real keyset query; the route asks for limit + 1 and only trims the has-more probe. Both params are optional, so the internal, v1 admin, and copilot callers are unchanged. Also restores the untouched GET handler in [id]/route.ts to its original formatting — collapsing its signature had re-indented the whole body and buried the actual additions in whitespace churn. * feat(api): expand v2 tables with stateless multipart transfers (#6188) * feat(api): expand the public v2 tables surface Adds 16 operations so a v2 caller can do what the internal surface can: rename/move/lock a table, restore it, manage saved views, run enrichment columns, look up rows, and import/export with observable job control. Extracts lib/table/orchestration/import.ts (performTableCsvImport, performCreateTableFromCsv) and lib/table/export-stream.ts from the first-party routes, then repoints those routes at them, so v1 and v2 cannot drift on what an import or export actually does. events/stream, metadata and dispatches stay internal — they are editor state, not public API. * fix(api): make v2 table PATCH all-or-nothing and name the lock in every 423 Greptile P1: PATCH applied locks, rename and move as three sequential transactions, so a folder rejected mid-request left the earlier writes persisted while the response reported failure — and the schema-changed signal was skipped, leaving open clients on stale state. Every rejectable condition now runs before the first write, and the signal fires whenever anything did land. Cursor: v2TableLockError dropped the lock kind, so async import, column run, enrichment and table mutations returned a bare LOCKED. A table has four independent locks, so the caller could not tell which to clear. * fix(api): report the lock kind on classified 423s too, not just thrown ones The previous commit named the lock only where the rejection was thrown and caught at the route boundary. Where it instead arrives as a classified `errorCode: 'locked'` outcome — delete table, delete row, update column, and the table mutations — the kind was dropped, so those 423s stayed unactionable while their neighbours improved. The orchestration results now carry `lock`, and a shared `v2TableOrchestrationError` renders both arrival paths into the same `{ code, message, details: { lock } }` body. `details` is omitted rather than sent null when the kind is unknown, so a caller branching on it sees absence instead of a phantom value. * fix(api): make async table imports observable, not just startable `POST /import-async` pointed callers at `GET /api/v2/tables/jobs` to track progress, but that endpoint filters to `type = 'export'` — imports are derived onto the table itself, one write job at a time, and exports get a separate list precisely because they are excluded from that derivation. The public Table shape omitted those derived fields, so an async import could be started and cancelled but never observed to completion, failure, or progress. That is the gap the import/export/job-control set was meant to close. Table now carries `job` — id, type, status, rowsProcessed, error, or null when idle — and the import-async docs point at the table rather than the export list. * feat(api): make v2 table PATCH state which operations landed on failure Greptile held the PR at 4/5 on the residual non-atomicity and named two acceptable resolutions: make PATCH atomic, or have the contract adopt and expose partial-success explicitly. Atomicity would mean threading one transaction through renameTable, moveTableToFolder and updateTableLocks — three shared service functions with four non-test callers including the first-party route and two copilot tools — and deferring their per-operation audits to commit time. That is a refactor of shared write paths well outside this PR. So the contract states it instead. Every rejectable condition is already pre-validated, so a failure here is a genuine fault; when one follows a successful operation the error now carries `details.applied` listing what is live. Absent when nothing applied, so its presence always means "these changes took effect despite the error". Documented on the operation. `v2ErrorForOrchestration` gained the optional `details` this needs. * fix(api): make table lock flags read-only on the public v2 surface The new PATCH /api/v2/tables/[tableId] accepted a `locks` object, gated on workspace admin plus the table-locks feature. That still lets an API key clear the guard placed there to stop it: `write` is the floor for the endpoint, and admin keys are ordinary API keys, so a lock is no longer a boundary the key cannot cross. Locks stay readable on the table resource and enforcement is unchanged (a locked verb still returns 423). Changing one is now a first-party admin action only. The v2 body is declared here rather than reusing the first-party updateTableBodySchema, which keeps its `locks` field so the UI can still toggle them. It is .strict(), so a request carrying `locks` is rejected with a 400 naming the field instead of silently succeeding without applying it. * fix(api): keep reporting applied operations when the PATCH re-read fails The composite table PATCH promises that `error.details.applied` names the operations that are live despite an error, but `applied` was scoped inside the try. A rename or move that committed and was then followed by a throw in the final re-read — or a re-read finding the table archived — returned a bare 500/404 with no details, telling the caller nothing had landed. It would then retry into a duplicate-name conflict or repeat the move. `applied` is now function-scoped so every post-write exit carries it: the 404 on a missing re-read, a thrown lock error, a classified orchestration error, and the generic 500. `v2TableLockError` gains the same `extraDetails` parameter `v2TableOrchestrationError` already had. * feat(api): add workflow group writes to the v2 tables surface v2 exposed GET /groups but none of the writes, so the public API could run an enrichment or workflow column and read its binding, but never create one. A caller could add a plain data column and trigger the machine; wiring the two together still required the UI. Adds POST/PATCH/DELETE on /api/v2/tables/[tableId]/groups. The group is the unit that fills columns — one group feeds several — so creating one creates its output columns in the same call, matching the first-party shape rather than inverting it onto the column endpoint. Four departures from the first-party body, all public-surface concerns: - group.id is optional and server-generated. The UI mints an id to render optimistically; a public caller has no such need and a client-chosen id is a collision waiting to happen. - outputColumns[].workflowGroupId is dropped from the body and stamped from the resolved group, so it cannot disagree with it. - autoRun defaults to false. First-party defaults true so a UI add fills cells immediately; here it would make one POST fan out a metered run across every existing row. - A group naming neither a workflowId (type manual) nor an enrichmentId (type enrichment) is a 400 rather than a half-specified group the route has to guess about. Also rejects an outputColumns entry no group output feeds — the two arrays are joined by column name, and the first-party client builds both from one picker so it cannot desync, but a public caller can. Workspace containment on workflowId is asserted before it is persisted, on create and on any update that re-points the group; without it a table becomes a way to invoke workflows the key cannot otherwise reach. * improvement(api): make v2 table import and export async-only Drops the three synchronous entry points: POST /tables/[tableId]/import, POST /tables/import-csv, and GET /tables/[tableId]/export. Sync import tied a write to the lifetime of an HTTP request. The body *was* the data, so it carried a 10 MB cap that Next silently truncates past — a partial import reporting success. It also had no job, so a timeout mid-write left rows in place with nothing to poll and nothing to cancel. The async path reads the file from storage instead: upload via POST /api/v2/files for a key, start with POST /import-async, watch GET /tables/[tableId] -> job, stop with POST /job/cancel. Sync export carried no such hazard, but one shape per operation beats two: with both removed the surface has exactly one way to move a table in or out, and the CLI wraps the extra calls. This also removes the last multipart handling in v2 tables. Those were the only routes bypassing parseRequest — form fields were parsed by hand against separate form schemas, outside the contract system every other v2 write goes through. Create-a-table-from-CSV is now two calls: POST /tables, then /import-async with createColumns. csvImportModeSchema is append|replace, so there is no single-call create. Route baseline 1064 -> 1061. * docs(api): correct the import-async note about upload size limits The docstring claimed there is no synchronous upload endpoint and so no request-body size cliff. Both are wrong: POST /api/v2/files is a synchronous multipart upload with a 100 MB cap, and it is the only v2 upload path (presigned is deliberately absent). What async-only actually bought: the cap went 10 MB -> 100 MB, it fails on an explicit size check and a bounded body read rather than a proxy cap that silently truncates, authorization completes before any body is buffered, and the table write is a job that can be watched and cancelled. * feat(api): unify file and table transfers * improvement(api): make multipart transfers stateless * fix(api): make table import completion retries idempotent * feat(v2-tables): paginate the table list `GET /api/v2/tables` returned every table in the workspace in one response — it used the cursor envelope but hardcoded `nextCursor: null`, and had no `limit`. That was defensible when tables were only created through the UI; `POST /api/v2/tables` is public now, so a script can create them in bulk and the list has no way to ask for less. Adds `queryTables` alongside `listTables` rather than changing it, so the internal callers that genuinely want the whole scope are untouched — the same split `queryWorkspaceFiles` / `listWorkspaceFiles` already uses. Filter, order and slice all run in the query, so a `search` never costs a full-workspace read. A cursor whose values don't bind raises a validation error instead of being coerced to "no filter", which would have silently served page 1 under a resumed cursor. The keyset closes on `id` so a page boundary inside a run of equal names or timestamps stays stable. The shared `LimitQuery` doc component said "Maximum rows to return"; it now serves the table list too, so the wording is resource-neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(api): add multipart knowledge document uploads * fix(api): keep usage admission at knowledge upload session creation * feat(knowledge): wire knowledge base uploads to multipart sessions * fix(knowledge): refuse to abort an upload once a document is bound * fix(uploads): prevent multipart cleanup races * Unify file creation and signed upload sessions (#6264) * feat(uploads): unify signed upload sessions * fix(uploads): preserve attachment storage semantics * feat(files): add authored file creation * fix(uploads): omit hoisted S3 metadata headers * feat(api): add file metadata endpoint * improvement(api): scope folders to resource paths (#6284) * improvement(api): scope folders to resource paths * fix(files): serialize folder resolution with uploads * fix(files): release folder lock before upload setup * fix(api): normalize folder paths and unblock resource mutations * fix(api): make resource cleanup and metadata consistent * improvement(uploads): persist multipart sessions in postgres * fix(db): store table row trigger timestamps in UTC * improvement(api): default folder deletion to non-recursive * fix(billing): unify chat usage source * improvement(logs): expose trace spans on log detail * fix(logs): parse list trace spans * improvement(api): replace workflow jobs with execution resources (#6294) * improvement(api): replace workflow jobs with execution resources * fix(api): preserve legacy jobs while preferring v2 executions * fix(api): make execution polling resume-aware * fix(ui): hide async examples for public workflows * fix(api): bridge resume queue visibility lag * feat(api): add v2 workflow resume endpoint * fix(api): project pending resume attempts * fix(api): prefer terminal logs over stale resumes * improvement(api): unify v2 resource query layers (#6319) * improvement(api): unify v2 resource query layers * fix(api): address v2 review findings * fix(api): preserve cancelled queue status * fix(api): guard cancelled job transitions * fix(api): close v2 resume and log gaps * feat(api): rename v2 executions to runs * feat(api): split credentials and secrets * feat(api): add workspace metadata and email attribution * improvement(api): consolidate public v2 route handling * improvement(files): centralize operations across APIs and Copilot (#6392) * improvement(files): unify rename authorization * chore(skills): add file operation migration guide * improvement(files): consolidate file operation authorization * improvement(files): extract shared operation foundation * improvement(api): simplify internal route declarations * improvement(files): centralize application authorization * refactor(api): share workspace file name validation * refactor(files): centralize copilot application calls * docs(skills): generalize application operation migration * improvement(api): centralize remaining v2 resource operations (#6412) * improvement(api): centralize v2 resource operations * fix(api): preserve custom tool conflict errors * improvement(api): migrate policy-sensitive v2 reads (#6410) * improvement(workflows): centralize v2 application operations (#6411) * refactor(api): migrate v2 knowledge operations (#6413) * refactor(api): migrate v2 knowledge operations * fix(knowledge): fail upload completion on dispatch errors * fix(knowledge): preserve upload retry and VFS errors * improvement(tables): centralize v2 application operations (#6414) * improvement(tables): centralize v2 application operations * fix(tables): preserve run validation and signals * feat(auth): add scoped internal executor delegation (#6459) * feat(auth): add scoped internal executor delegation * fix(auth): derive delegation lifetime from one timestamp * Include share status in file metadata * feat(auth): centralize delegated identity policy (#6462) * improvement(copilot): consolidate application adapters (#6450) * improvement(api): harden application route boundaries (#6451) * improvement(api): harden application route boundaries * fix(folders): reject creates at workspace cap * fix(knowledge): enforce trusted workspace scope (#6452) * fix(knowledge): enforce trusted workspace scope * refactor(knowledge): declare v2 body lifecycle * finish knowledge application migration * refactor(knowledge): compose copilot batch commands * fix(knowledge): parse connector query flags * fix(knowledge): finalize partial batch effects * fix(knowledge): align merged application boundaries * fix(knowledge): close application boundary review gaps * style(knowledge): satisfy branch biome checks * fix(knowledge): page connector documents in editor * refactor: enforce Copilot table application boundary (#6453) * refactor: enforce copilot table application boundary * fix(tables): finish application boundary migration * fix(tables): restore scoped copilot imports * fix(tables): compose copilot commands atomically * fix(tables): preserve workflow group scheduling * fix(tables): complete fixed copilot composition * fix(tables): reject enrichment output mutation * fix(tables): complete authorized application boundary * fix(workflows): migrate Copilot application boundary (#6455) * fix(workflows): migrate Copilot application boundary * fix(workflows): finish delegated application migration * fix(workflows): encode VFS folder aliases * fix(workflows): close application composition gaps * fix(workflows): preserve VFS validation errors * fix(workflows): complete application boundary migration * test(workflows): format canonical binding coverage * fix(workflows): scope executor metadata reads * fix(workflows): bind executor metadata targets * improvement(skills): align application operation guidance (#6532) * feat(api): expose v2 resource owners * fix(api): distinguish visible resource authorization failures (#6537) * feat(api): generate v2 OpenAPI from contracts (#6509) * feat(api): generate v2 OpenAPI from contracts * fix(api): preserve string boolean wire defaults * fix(api): document file download headers * fix(docs): use TypeScript CLI with Next.js * fix(docs): avoid client-rendered theme script * fix(api): document departed audit default * feat(api): replace legacy core docs with v2 * feat(api): generate v2 OpenAPI from contracts * feat(api): refine generated v2 OpenAPI docs * fix(docs): align localized v2 execution examples * fix(ci): restore Helm diff and sync audit mock * fix CI regressions after staging merge --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
245ad1bd46 |
feat(workflows): new workflow block card, progress indicator, colors, dsl for natural language preview, retry configs (#6458)
* improvement(workflow): refine canvas interactions and rendering
* fix(workflow): keep outputs on the right, focus newly created blocks
Connection anchors: an output now always leaves a card from the right.
The cursor swell lets a drag start on any edge, but the left side is the
input, so anchoring an outgoing edge there drew a line out of the input
port and read as a second input. `normalizeCursorSourceHandleId` resolves
every drag to the right anchor, `normalizePositionedSourceHandleId`
collapses `source-left` alongside the legacy vertical anchors (so data
from the API, an older client, or a stale save self-heals on load), and
only the right-side source anchor is mounted.
Drops in `onConnectEnd` are always source -> target. The branch that
reversed the edge for a drag starting on an input could never run: the
`target` handle is `isConnectableStart={false}` and the positioned side
anchors are `isConnectable={false}`, so React Flow never reports an input
as a drag origin. Removed it and its now-unused imports.
A newly created block is centered once its node mounts and is measured,
so a card added from a drag-release, the block menu, or the toolbar is
never left off-screen or under the editor panel.
The editor panel's block icon uses the same type accent as the card's
badge instead of the block's legacy `bgColor`, which had left the panel
on the old per-integration brand colours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(workflow): floor header-only card height, adopt brand tag palette
The Start card intermittently collapsed after load, squashing the
action-menu tab so its icon row sat over the card.
`.workflow-drag-handle` is the host the border renderer measures, and both
it and the header row took their height from `blockHeight && blockHeight >
0`. `blockHeight` comes from the deterministic-dimensions pass and is
already floored at MIN_PAINTED_HEIGHT (48), but it is absent on the first
frames — and with no floor the host collapsed to its natural content
height (25.5px for a header-only trigger, exactly the title's line box).
The border builds its perimeter from `host.offsetHeight`, so that window
painted a sub-floor card: too little straight edge remained on the
vertical runs for the action-menu tab, which collapsed into the corner
arcs. Whether you saw it depended purely on whether the dimension publish
had landed, which is why it reproduced on one workflow and not another.
Floor all three: the host, the header row (so `items-center` centres the
title and type tag rather than pinning them to the top), and the border's
own `offsetHeight` read.
Also raise ACTION_MENU_CONTENT_READY_THRESHOLD to 0.9. At 0.8 the 24px
icon row was revealed while the swell had only reached 22.4px of its 28px
— shorter than the row it contains. Secondary to the above, but a real
overflow window on its own. The test now pins the ratio rather than the
constant.
Tag palette moves to fixed brand values (hex, not derived oklch) with two
inks — #F8F8F8 on dark fills, #1A1A1A on light. Tones are renamed to match
what they render. `green` (2.55:1) and `orange` (3.15:1) sit under WCAG AA
against their paired ink; both are deliberate brand decisions and are
documented in the component.
Deploy and Run take two new Button variants rather than className
overrides, so `tertiary` stays green everywhere else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* improvement(workflow): polish workflow canvas interactions
* fix(workflow): restyle loop drop target outline
* fix(workflow): shorten human block catalog label
* fix(workflow): canonicalize realtime edge handles
* code cleanup
* sizing fixes
* improvement(blocks): sentencify every block
* change tebse
* improvement(workflow): land notes UI and execution progress, consolidate duplicates
Ports the notes canvas editing and execution progress-indicator work, then
removes the parallel paths it arrived with so each concern has one owner.
Fixes found while consolidating:
- Note height was measured while the card was expanded to NOTE_EXPANDED_WIDTH,
where text re-wraps shorter, and published as the node's compact height. The
collapse then animated to a height never measured at the compact width.
- The edge pulse glow filter used the default objectBoundingBox units, so a
straight horizontal edge — what an auto-laid-out chain produces — resolved the
filter region to zero height and stopped the glow rendering entirely.
- A subflow's inner Start pill still read isNodeSelected while its border read
usesSelectedVisuals, so the two disagreed during execution.
- The Run/Stop button's disabled prop gated only Run while its handler cancelled
unconditionally, offering a Stop the cancel route answers with 403.
Consolidated:
- One note editor. The view's built-in textarea was unreachable in production
(the app always injects the markdown editor) and was kept alive only by tests
asserting against it; renderContentEditor is now required.
- onBlur/onCancel collapse to onEndEditing — content persists per keystroke, so
there was never a draft for a cancel path to discard.
- DEFAULT_NOTE_COLOR, the note height bounds, the note content reader and the
card width each had two or three definitions; each now has one.
- Removed with zero consumers: graphite/graphiteSubtle button variants,
data-subflow-selected, inputPlaceholderClassName, an effect that could never
fire, and getNoteColorOption's unreachable fallbacks.
Restores the role='status' announcement the progress rewrite dropped, and hardens
isNoteColor against inherited Object keys.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement(workflow): reuse the platform markdown editor in notes
Notes carried their own TipTap wiring — a second markdown editor that
reimplemented, more thinly, what `RichMarkdownField` already does for the skill
modal, skill fields and the deploy version description. It is now a ~20 line
skin: the Note supplies its type scale and per-colour selection tint, and the
field supplies the extension set, frontmatter held out-of-band, the round-trip
safety gate and its raw-source fallback, and markdown paste.
`RichMarkdownField` gains two additive props, both defaulting to today's
behaviour so the file editor is untouched: `surface` ('field' | 'bare') and
`proseClassName`. All three existing consumers pass an explicit `minHeight` and
no `surface`, so they take the original path unchanged.
Exiting the note editor moved to the card, because the editor's `/` and `@`
menus consume Escape to close themselves and ProseMirror checks `editorProps`
before plugin handlers — intercepting it inside the editor would have broken
both menus. The card now honours Escape only when nothing already consumed it,
which also let `onEndEditing` leave the injection contract.
The note editor is lazy now, matching every other consumer: it was pulling
TipTap and the full extension set into the canvas's initial chunk.
Also:
- One `areRunFromBlockDependenciesSatisfied`. The ActionBar, the canvas context
menu and the run-from-block handler each carried a byte-identical copy, and
the handler expressed the snapshot requirement differently, so the affordance
and the action could disagree. Each copy also re-scanned `edges` once per
incoming edge, on every ActionBar on the canvas.
- Reduced motion is one `usePrefersReducedMotion` in @sim/emcn rather than a
sixth ad-hoc `matchMedia`. The edge pulse now stops rendering instead of
hiding: `motion-reduce:hidden` is `display: none`, which left four SMIL
timelines running per edge.
- The pulse glow bleed covers the canvas minimum zoom. The strokes are
`non-scaling-stroke`, so the 6px tail spans 3/zoom user units — 30 at 0.1.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement(workflow): port canvas styling from workflow-updates
Ports the 14 styling commits your colleague added since the last sync, leaving
the ~68 staging PRs on that branch alone — those are platform/core work, not
this. Cherry-picked individually rather than merged so each conflict was small
enough to reason about.
What came in:
- Core block colors unify behind a two-level map: block type -> semantic role
-> accent, replacing the flat per-type table. Adds `purple` and `content`
tones to ChipTag, and a shared `WorkflowTypeIcon` that replaces the
hand-rolled ChipTag + accent lookup at each discovery surface.
- Native triggers take semantic colors; the deployments block moves to the
shared Rocket icon and drops its now-unused `iconColor`.
- Running-state polish: loader artwork and position, stop hover in dark mode,
the loader blended into the execution swell, and tooltips suppressed for
actions that are hidden mid-run.
- The toolbar drag preview clones the rendered icon container instead of
rebuilding a bgColor tile, so it matches what the canvas paints.
- The sidebar shows route-derived workspace identity instead of a skeleton
while the full record loads.
Conflict resolutions worth knowing:
- The running-loader artwork went through the shared `Loader` and back to the
custom SVG on their branch; the second commit is the intent, so that is what
landed — keeping our `role='status'` announcement layered on top.
- Two commits carried the lucide-react -> in-house icon migration along with
them. That migration is a staging change we have not taken, so our imports
stayed on lucide: adopting it in two files would leave the icon set split
across the app.
- `getMappedWorkflowTypeAccent` referenced a constant their refactor removed.
It had no consumers left once the search modal moved to `WorkflowTypeIcon`,
and their branch deletes it too, so it is gone here.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(workflow): make the subflow Start swell a real connection source
Dragging an edge into whitespace opens the add-block picker, but starting that
drag from a loop/parallel Start pill did nothing: the pill's border swell was
visual-only. Regular blocks and the container's own exit mint a draggable
cursor handle from their swell; the pill rendered only its invisible 14px
static strip, so grabbing the glowing affordance started no connection at all.
Everything downstream already worked and was nearly unreachable:
- the drop hit-test skips subflow containers, so a release inside the loop
opens the picker there (z 2000, above containers)
- handleToolbarDrop parents the new block into the container at the drop point
- it already carries the exact boundary rule for this source: a container
start handle only wires to a child of that container
The pill now runs the same cursor-handle machinery as the container view, with
one deliberate difference: its temporary handle carries the branch-cursor form
of the start id. The plain cursor id normalizes by block type — for a
container that is `loop-end-source`/`parallel-end-source`, the exit — so a
swell drag from Start would have persisted as an edge leaving the container.
The branch form passes `loop-start-source`/`parallel-start-source` through
normalization verbatim on both the picker and direct-connect paths; a test
pins that contract.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(notes): stop the field's prose classes recoloring bare-surface editing
Opening a note for editing shifted the text and turned it black: the
ProseMirror root unconditionally carried `rich-markdown-prose
rich-markdown-field-prose`, which pin the field's own ink and type ramp —
`--text-primary` at 15px/25px, then 14px/22px — overriding the card's
`text-current` at 14px/20px the moment the editor mounted.
`surface='bare'` means the host owns typography (the Note card mirrors its
rendered view via `proseClassName`), so on that surface the root now carries no
shared prose classes. The field surface is untouched. Edit mode inherits the
note colour's ink — including the caret — and sits on the same metrics as the
read view.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(workflow): give the in-flight connection line contrast inside containers
The drag line was drawn but camouflaged: its default stroke was the
resting-edge grey (#e0e0e0), which disappears against a loop body's opaque
`--surface-3` fill (~1.1:1) — so dragging an edge inside any container, nested
included, showed nothing. The z-order was never the problem; the connection
line layer already sits above every node.
The default token is now `--text-muted`, one value with contrast on every
canvas surface, still lighter than the `selected` variant so the variant
hierarchy holds. No per-surface special-casing.
Resting edges inside containers share the same camouflage (`--workflow-edge` on
`--surface-3`) — left alone deliberately: recoloring placed edges is a design
decision, not a bug fix.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(notes): match edit mode to the read view, and land the caret where clicked
Three defects, all from the read and edit views being built independently.
1. Blocks jumped up ~12px on entering edit mode. Streamdown wraps its output in
a container carrying `space-y-4` plus first/last margin resets, which outrank
the per-element margins in NOTE_COMPONENTS — so that wrapper, not those
margins, is what the read view actually paints. The editor had no equivalent.
The rhythm is now named (NOTE_MARKDOWN_FLOW), passed to Streamdown
explicitly so a dependency upgrade cannot move the read view out from under
the editor, and mirrored on the ProseMirror root. Tailwind's JIT only sees
literal strings so the mirror cannot be composed from the constant; a test
pins the two together instead, and fails if either side drifts.
2. The caret was barely visible: it inherited the note's 75%-opacity ink. The
palette owns per-colour chrome, so it now names the caret alongside the
selection tint.
3. The caret always landed at the document end. The read view sits under a
full-bleed overlay that must swallow the click to enter editing, so the
point never reached the editor and `autofocus: 'end'` was all that was left.
The view now forwards that point and the field resolves it through
`posAtCoords` on create — after the DOM is laid out, which `autofocus`
cannot wait for. Keyboard activation carries no point and still lands at the
end.
`autoFocusAt` is additive on the shared field and defaults to null, so the file
editor and the other three consumers are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(workflow): stop edges rendering behind top-level subflows
Containers are z-indexed by nesting depth, so a top-level subflow is 0. Edges
derived their z from their parent container — `+1`, or 0 with no parent — so a
root-level edge landed on exactly the same z as a root-level subflow. Equal
z-index falls back to DOM order, and React Flow paints the nodes layer after
the edges layer, so the container's opaque body won: any edge crossing a
top-level loop or parallel was drawn behind it, in-flight or persisted.
Edges now sit in their own band above the whole container scale and below
cards, keeping both the deeper-container-wins ordering and the rule that a line
always passes behind card chrome. This is why the edge became visible only once
a block was dropped: the new block is selected, and an edge inside the
container was already `containerZ + 1`, clear of the tie.
The in-flight connection line is declared in the same scale rather than
inheriting React Flow's stylesheet default of 1001, which is both below a
selected container child and outside the scale this file owns. Its stroke moves
to `--text-secondary`, the token the canvas already uses for an active edge —
the previous `--workflow-edge` grey is ~1.1:1 against a subflow body.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(notes): paste and drop images through the workspace-file pipeline
Traced the file editor's image path end to end and reused it verbatim rather
than minting a note-specific source: insertImages -> useUploadWorkspaceFile ->
POST /api/workspaces/{id}/files/presigned -> direct-to-S3 PUT -> workspace_files
row -> the editor persists the workspace-scoped
/api/workspaces/{id}/files/inline URL, which the serve route authorizes by
workspace membership and the embedded-image-ref machinery already recognizes
for share rewriting and referenced-by-doc tracking.
The shared field gains an optional `uploadImage(file) -> {url, alt} | null`
capability. With it, image paste/drop uploads sequentially and inserts each at
the evolving position, mirroring the file editor's flow, with a bail if the
editor unmounts mid-upload; without it, the existing swallow-guard on file
drops is unchanged, so the skill modal, skill fields and version-description
consumers behave exactly as before. The upload mutation owns its own toasts.
The note host wires the capability with folderId null, so note images land in
the workspace Files root — visible, manageable and deletable there like any
other upload. The note read view renders images through its Streamdown
components map with the card's own sizing.
Co-Authored-By: Claude <noreply@anthropic.com>
* improvement for notes, subflows
* fix(uploads): surface the server's message when a multipart upload is refused
A file over the 50MB direct-PUT threshold goes through multipart initiate, which
is where the storage quota is enforced — but the client threw away the response
body and reported `Failed to initiate multipart upload: Payload Too Large`. That
is the string the upload mutation puts in its toast, and it names neither which
limit was hit nor by how much, so the one place that answer surfaces didn't have
it.
It now prefers `errorBody.error` exactly as `getPresignedUploadInfo` already does
on the single-PUT path, and passes the body through as the error's details.
Control flow is unchanged: still throws, still `MULTIPART_ERROR`, and the
cloud-storage-absent branch above still claims its 400 first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(notes): restore GFM in the note read view
Streamdown's `remarkPlugins` prop REPLACES its default plugin list rather than
extending it, and remark-gfm is one of those defaults. The note passed
`[remarkBreaks]` — so the read view silently lost every GFM construct: task
lists, tables, strikethrough and autolinks.
The editor writes all of them (it has TaskList, TableKit and Strike), so a note
round-tripped through editing came back as raw source the moment editing closed:
`- [x] HELLO` rendered as a disc bullet followed by the literal text `[x] HELLO`.
`NOTE_COMPONENTS` has carried table/thead/tbody/tr/th/td entries this whole time
that could never fire.
Restoring the plugin is only half of it: remark-gfm marks a checklist
`contains-task-list` and emits a native checkbox, which under the note's generic
`ul` styling renders a checkbox sitting behind a disc bullet — the same defect
the editor had before the chrome/typography split. The read view now drops the
marker and indent for a task list, lays the row out as a flex line, and styles
the checkbox to match `.rich-markdown-nodes input[type="checkbox"]` declaration
for declaration, tick clip-path included, so the two views agree either side of a
click.
`remark-gfm` is now a declared dependency of the renderer package rather than one
borrowed transitively from streamdown.
Five tests cover the GFM surface — checked/unchecked boxes, no literal `[x]`, the
marker only dropped for checklists, tables, strikethrough — and four go red with
the plugin removed.
Checked the other three Streamdown call sites (Chat, the chat interface renderer,
the changelog): none override `remarkPlugins`, so none were affected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(canvas): drop dead markers and share the tile-brightness maths
Review pass over the branch against staging.
Dead code removed:
- `tileIconColorClass` in the renderer package — never called; only its
`isLightTileColor` sibling is.
- `data-connection-selector-search-frost`, `data-workflow-cursor-edge` and
`data-workflow-cursor-source-side` — written on three elements, read by no
stylesheet, selector or test.
- `CHIP_TARGET_SELECTOR_TYPES`, `MAX_CHIPS` and `chipPriority` were exported from
`canvas-rows.ts` but only used inside it.
Consolidated the one real divergence: the renderer package carried a hand-copied
mirror of the app's perceived-brightness maths, because it may not import app
code. The copy had already drifted — it dropped the `white`/`black` keyword
handling, so a block shipping `bgColor: 'white'` would render a white
`currentColor` icon on a white tile on the canvas while every other surface drew
it black. No block ships one today, which is exactly why nothing caught it. The
function now lives in `@sim/utils/color` and both sides import it; only the
0.75 threshold stays local to each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(canvas): share the z-scale, fix the preview's edge layering, and re-home strays
The preview canvas carried its own z numbers and had the collision the editor
canvas was fixed for: containers at nesting depth, top-level cards at an implicit
0, and edges at 0/5/10 by execution status — so a default edge tied with a
top-level subflow and painted behind it, while a success edge painted over
unselected cards.
The scale now lives once, in `@sim/workflow-renderer/canvas-layers`, and both
canvases read it. The preview keeps its status ordering, expressed inside the
shared edge band rather than as a second set of magic numbers.
Placement and duplication:
- `perceivedBrightness` moved to `@sim/utils/color`, with its unit test, and its
consumers import it directly. It had been re-exported through
`lib/colors/brightness.ts`, and the renderer package kept a hand-copy.
- `filterAcyclicEdges`/`wouldCreateCycle` were pass-through wrappers in the
workflow store's utils over the real implementations in `@sim/workflow-types`.
Deleted; the three consumers import the source.
- `lib/ui/glass-surface.ts` was a one-constant, one-consumer app-wide module, and
its consumer then aliased it a second time. Collapsed into the navbar shell.
- `nested-subflow-node` was set on nested container nodes in both canvases with no
stylesheet, selector or test behind it.
`packages/workflow-renderer` now has its own vitest config, so the four mount
tests for its components live with the components instead of in
`apps/sim/lib/workflows/**`. That immediately earned its keep: `apps/sim`
excludes test files from type-check, and once these were checked, tsc found three
`SubflowNodeView` renders being handed a `renderContentEditor` prop it does not
accept — a copy-paste from the note cases that had been silently ignored.
Verified: type-check 23/23, 21,143 app tests + 49 renderer + 147 utils, biome
clean, all 23 audits pass (`check:bare-icons` imported the moved helper and was
repointed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix field noun bug + notes
* fix(notes): let a note service the canvas actions the panel editor cannot
The panel editor clears any note put in front of it and renders nothing, but the
block menu still routed Rename and Open Editor through it.
Rename latched the editor's rename state onto the note — `handleStartRename`
reads the store directly, so it saw the id the menu had just set — and nothing
reset it when the clear ran. `handleSaveRename` writes to `renamingBlockIdRef`,
so the header went on showing a rename field over whatever was selected next and
saved that name to the note. Open Editor was a plain no-op that opened an empty
pane.
Rename now goes to the card, which expands and opens its own title — the same
menu-to-card routing Add Image already used, so both events now live in one
`lib/workflows/notes/canvas-requests.ts` and `add-image.ts` keeps only its
markdown concern. Open Editor is hidden for notes.
The panel editor also drops any rename whose block stops being the selected one.
That is belt-and-braces for notes now, but it closes the same hole for ordinary
blocks, where only the input's blur ended a rename and blur only fires if it held
focus. A rename interrupted that way is now discarded rather than left pending.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(canvas): author sentences for Snowflake and Mintlify, repoint Instagram's
Staging's two new integrations shipped without a `canvasPresentation`, so their
39 operations painted the field rows the rest of the canvas has stopped using.
The Instagram break is the more interesting one: staging renamed the insight
metrics subblock `metrics` -> `insightMetrics` while this branch was adding
sentences that named `metrics`. Both hunks merged cleanly — the union check
reports the file as an exact union — and the result was two clauses pointing at
a field that no longer exists, which resolves to nothing with no error and no
log. Only `check:canvas-sentences` sees it.
Two Snowflake sentences say something the block does not do, so they anchor
elsewhere: `taskName` filters `list_task_runs`/`get_task_run` rather than keying
them, and `table` filters `introspect_schema` — blank means "every one", not "not
filled in yet", and a core chip would have claimed otherwise.
Coverage is back to 4727/4727 operations across 321/321 blocks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(canvas): keep the block-type tag naming its type after a rename
The header tag dropped its label whenever the block's title already said the
same word, so the same block read two ways depending on nothing the user did
deliberately: a freshly dropped Wait showed a bare icon, and its second copy —
auto-named "Wait 2" — showed "Wait". The tag looked like a badge that appeared
on rename rather than a fixed part of the header.
It now always names the type, which is what loop and parallel containers already
do with their own tag, so every card on the canvas reads the same way.
`blockName` was only ever read for that comparison, so the prop is gone rather
than left behind for a future reader to wonder about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): compare edge handles by port, not by spelling
Two places answer "does this need redeploying?" and they load their sides
differently. The client diffs the live store against `/api/workflows/[id]/deployed`;
the server diffs the normalized tables against the version's raw jsonb. Only
some of those paths run handles through `loadWorkflowFromNormalizedTables`, so a
snapshot holding a side-anchored id (`source-right`) met a canonical one
(`source`) on the other side and the set comparison read it as every edge being
removed and re-added.
Each answer therefore differed, and they arrive on separate query timelines: the
button reads the client's, the modal badge reads the server's, so the state
flipped between Live and "Update deployment" with whichever query landed last
until both settled.
`normalizeEdge` now canonicalizes both handles, so the comparison cannot tell
two spellings of one port apart no matter how its inputs were loaded. The
existing normalization in `materializeDeploymentState` stays — that path also
feeds React Flow, which needs the handle it mounts to match.
The preview's error port had the mirror problem: it rendered for every
non-trigger block regardless of `errorEnabled`, so a card with no error row grew
a red knob anyway. It now gates the way the editor canvas does, keeping the port
mounted when an error edge already leaves it so React Flow cannot drop that edge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): stop counting the error flag twice in change detection
`errorEnabled` has two homes. It persists inside the block's `data` jsonb — the
realtime server `jsonb_set`s it there, and load mirrors it back onto the block as
a field — so it reached the diff twice, and only some paths populate the copy.
`setBlockErrorEnabled` writes the mirror alone, so right after toggling the port
the live block said `errorEnabled: true` with `data.errorEnabled: false`, while
the snapshot the deploy had just taken from the tables said true in both. The
diff read the stale `data` and reported the workflow as changed the instant it
finished deploying — then a state refetch rehydrated the block and it agreed
again. That is the flip between Live and "Update deployment": the button and the
modal read two different queries, so each landing swapped the answer. A block
created in-session had the same shape from the other side, its `data` carrying no
key at all against a persisted `false`.
Excluded from `normalizeBlockData` alongside the other fields that are duplicated
out of the block's own state. The block field is still compared on its own, with
`!!`, so absent and `false` agree and turning the flag on is still a change.
Fixing the store to write both homes was the other option and is not taken:
nothing reads the in-memory `data.errorEnabled` (save and load both let the block
field win), so it would add a second copy that only the diff could see — which is
the shape of this bug, not its fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(blocks): give the error-output flag a column instead of two homes
`errorEnabled` had no column, so it persisted inside the block's `data` jsonb
and was mirrored onto the block as a field on load. Every writer had to route
its `data` through `withPersistedErrorEnabled` or silently drop the toggle, the
realtime op `jsonb_set`, and change detection saw the same value twice — which
is what made the deploy badge flip between Live and "Update deployment" after
toggling the port.
Its siblings — `enabled`, `horizontal_handles`, `advanced_mode`, `trigger_mode`,
`locked` — are all boolean columns; `data` is for React Flow and subflow state.
The flag belongs with them, so it now has `error_enabled` and one home. The
shuttle helper, its `BlockData` mirror, the store's fallback read, and the
comparison exclusion the duplication forced are all gone.
Backwards compatibility, since released versions draw the error port with no
toggle in front of it: a block already wired to an error edge HAS the output on,
because there was no other way to draw that edge. That rule is now stated in
three places and none may be narrowed to read the flag alone —
- the migration backfills `error_enabled` from the edges, so live rows are true
before any new code reads them;
- `materializeDeploymentState` derives it for a version's frozen jsonb, which the
migration cannot reach — otherwise every workflow deployed before the toggle
would ask to be redeployed once;
- `workflow-block.tsx` keeps it at render time for states that reach the canvas
through neither (imports, copilot edits), where unmounting the port would make
React Flow drop the edge leaving it.
The migration also moves any `data.errorEnabled` a developer created on this
branch onto the column and strips the key; both statements match zero rows in
production, where it never shipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(canvas): realign the Snowflake and Dynatrace sentences with staging's blocks
Both breaks are the class the union check cannot see: separate hunks of the same
file merged cleanly, and the result names fields that no longer exist. A sentence
that does resolves to nothing, with no throw and no log.
Snowflake's rewrite (#6474) moved database, schema, table, warehouse and
procedure onto canonical selector pairs, so seven clauses anchored on ids that
are gone. Each now names both members of its pair, which is also what keeps the
card readable for someone working in advanced mode. Its nine new operations have
sentences.
Dynatrace (#6463) scoped the mute reason to the operations that mute, because
unmuting accepts exactly one — so the two unmute sentences were asking for a
field their card no longer shows. They drop the clause.
Coverage is 4736/4736 operations across 321/321 blocks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(api): drop the block-data error flag from the workflow contract
Left behind by the consolidation: the flag no longer lives in `data`, and a
schema that still declares it there invites the mirror back through the wire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(deploy): compare edge handles by port, so a falsy one cannot read as changed
`loadWorkflowFromNormalizedTables` now runs handles through the canonicalizer,
which falsy-coalesces — so an edge persisted with `sourceHandle: ''` loads as no
handle at all. The server diffs that against the deployment version's raw jsonb,
which still has `''`, and the set comparison reads one edge as removed and
another added. Every workflow holding such an edge would ask to be redeployed
the moment this ships, for nothing. Two write paths use `?? null` rather than
`|| null`, so `''` is reachable.
Canonicalized inside `normalizeEdge` rather than at either call site: the two
sides are loaded by different paths and only some of them normalize, so the
comparison has to be unable to tell two spellings of one port apart however its
inputs arrived.
This is the change reverted in
|
||
|
Waleed
|
9883543eb2 |
fix(knowledge): stop listing workspace knowledge bases on stale creator identity (#6454)
GET /api/knowledge without a workspaceId ORed on knowledge_base.user_id with no permission check, so a user removed from a workspace kept seeing metadata for every KB they created there. Scope the creator fallback to legacy KBs with no workspaceId, matching the workspace-filtered branch and the detail path. |
||
|
Vikhyath Mondreti
|
5cf1f9be84 |
improvement(provenance): cleanup secrets boundary (#6374)
* fix(secrets): preserve raw outputs with durable provenance * improvement(provenance): cleanup boundary * fix copy resources * fix fork copies to work with provenance * address comments * fix |
||
|
Waleed
|
1c0e82a4e2 |
perf(ci): parallelize repo audits, guard env-dependent tests, and fix the docs generator (#6358)
* perf(ci): parallelize the repo audits and guard env-dependent tests
The 21 independent audits ran as 21 sequential CI steps, each a single-threaded
read-only walk of the tree. scripts/run-audits.ts runs them concurrently:
28s serial -> 5.0s wall locally at 13-way. It buffers each audit's output and
replays only failures, so a green run stays quiet and a red one still names the
audit and shows why. Audits needing a git base ref (block registry, migration
safety) or that write files (drizzle generate) stay as their own steps.
Also fixes 5 tests that fail for every macOS dev and are invisible in CI. They
shell out to python3 using `match` statements and 3.12 f-string nesting, which
need >= 3.10; stock macOS ships 3.9.6, so `bun run test` produced raw Python
SyntaxErrors with no guard and nothing tying them to a missing tool. One also
needs ripgrep, which CI installs and a Mac usually does not.
@sim/testing/environment detects both and the tests skip with a reason via
vitest's ctx.skip(). Under CI it throws instead: these suites deliberately run
the real helper rather than a mock -- the cloud-review path/read-size bounds and
the placeholder compiler's generated Python are only observable that way -- so a
missing tool in CI means a security boundary silently stopped being covered,
which is worse than a red build.
Drops the Codecov upload. The workflow already documented it as a dead path:
nothing generates apps/sim/coverage, vitest runs without --coverage, and
fail_ci_if_error hides it, so it reported green having uploaded nothing.
* fix(ci): raise the python floor to 3.12 and stop the bridge audit serializing the batch
Two review findings, both real.
MIN_PYTHON was 3.10, chosen for the `match` statements the compiler suite
generates. But two of the three guarded tests also use PEP 701 f-strings --
reusing the outer quote, and embedding `#` -- which are 3.12. Verified on a real
3.11 interpreter: the match-guard test passes, the other two fail with
`f-string: unmatched '('` and `f-string expression part cannot include '#'`,
which is exactly the raw SyntaxError the guard exists to prevent. A 3.10 floor
let them through and failed anyway.
The audit parallelization did not speed CI up -- it slowed it down. Serially the
21 audits took ~31s; concurrently the batch took 39.2s wall, because
check:desktop-bridge went from 1s to 39.2s and became the entire wall clock while
the other 20 finished in 9s. It is the only audit that shells out through `bunx`,
which re-resolves the package against the shared install cache -- a network-backed
sticky-disk mount on CI. Cheap when it runs alone, serialized behind the others
when they run together. Spawning the resolved compiler entry point directly
removes that layer.
Verified the audit still fails on a breaking bridge change rather than passing
faster by doing less.
* fix(docs): unbreak the MDX build and read trigger config from the registry
The docs build has been failing on staging since the Smartlead merge:
./apps/docs/content/docs/en/integrations/smartlead.mdx
Expected a closing tag for `<original>` before the end of `paragraph`
Tool descriptions are emitted as prose, and that path escaped only braces --
every table-cell path already escaped angle brackets. MDX reads `<` as the start
of a JSX tag, so a description like 'The copy is named "<original> - copy"' fails
the build outright. escapeMdxProse handles the MDX-hostile characters and leaves
pipes, parens and brackets alone, which are legal in prose and whose escaping
would mangle markdown links.
Trigger configuration now comes from the evaluated registry instead of regex over
source. Static parsing silently dropped every field whose builder assembled its
array imperatively or took a description as a parameter -- all ten Jira triggers
lost `webhookSecret` and `jqlFilter` that way, and Monday lost its config too, so
regenerating the docs was destructive. Reading real objects also deletes 232 lines
of parsing. Note `required` may be a condition object rather than `true`; only an
unconditional `true` renders as Required, matching the previous behavior.
Tool headings now show the tool's name ("A2A Send Message") rather than its id
(`a2a_send_message`), unformatted, across 241 generated pages. Names come from
tools/generated/tool-metadata.ts, which CI keeps in sync. These headings feed each
page's table of contents. a2a.mdx is hand-written, so its headings were updated
directly.
Also consolidates five hand-inlined copies of the escape chain into the
escapeMdxCell that already existed, and drops 44 comments that restated the line
below them. Generator: 4306 -> 4069 lines.
Every refactor step was verified against a golden manifest of all 289 generated
files -- proven deterministic across runs and proven to catch a one-character
change -- so the only output differences are the intended ones.
KNOWN GAP: extractTriggerOutputs still parses source and has the same blind spot;
it already drops one Jira output section on main. Regenerating is now safe for
trigger config but still lossy for trigger outputs.
* refactor(ci): derive the audit list and stop shelling out through bunx
Review pass over the audit runner and the tool guards.
The audit list was hand-maintained alongside package.json with nothing linking
them, and it had already drifted: check:cron-parity exists, passes, and ran in no
CI step at all. The list is now derived from the check:* scripts with an explicit
exclusion map, so a new audit is opted out deliberately rather than forgotten.
That picks up cron-parity — 22 audits now, not 21.
check-realtime-prune-graph.ts still shelled out through `bunx turbo`, the same
pattern that took the bridge audit from 1s to 39s once the audits ran
concurrently. Both now go through scripts/local-bin.ts, which resolves
node_modules/.bin — the same path check:native-typecheck asserts is the native
TypeScript 7 compiler, so the one guarded path is the one that runs.
Audits are spawned as their script rather than `bun run <name>`, which started a
bun process only to read package.json and start a second one.
Tool detection is memoized per process; it was re-spawning python3 on each of the
5 call sites, in every vitest worker. The CI throw is deliberately NOT memoized —
memoizing it would turn every call after the first into a silent skip, which is
the failure mode the guard exists to prevent. Verified it still throws for all
three guarded tests, not just the first.
Also: dropped the environment module from the @sim/testing barrel so
node:child_process stays out of unrelated consumers' module graphs, restored the
per-audit reporting the 21 separate steps used to give (collapsible groups, error
annotations, and a timing table they never had), and trimmed comments that
restated their code or duplicated the runner's own docs.
* fix(devin): give the 11 Devin tools real display names
Every Devin tool had its id as its `name` (`list_session_messages`), so the
generated docs rendered `### list_session_messages` where every other integration
renders a human name. It was the only integration doing this -- 11 of 4427 tools.
Names take the service prefix, matching the majority convention (3200 of 4416
names start with their service).
Also points the ship skill at check:audits instead of hand-listing the audits.
That copy had drifted five behind package.json: cron-parity, import-specifiers,
sql-date-binding, trigger-block-cycle and native-typecheck were all missing, so
shipping never ran them. It was the third copy of that list; there is now one.
* fix(docs): read trigger outputs from the registry too
Closes the gap left by the config fix: extractTriggerOutputs still parsed source,
so triggers whose outputs come from a builder call lost their tables. jira_webhook
had no output section at all.
The registry was not a drop-in, which is why the naive swap deleted 10,298 lines
earlier. The two sides encode nesting differently. A TriggerOutput marks a group
by OMITTING type and holding children as sibling keys:
issue: { id: { type: 'number' }, title: { type: 'string' } }
while the renderer walks the JSON-Schema-ish shape the parser used to synthesize:
issue: { type: 'object', properties: { id: …, title: … } }
formatOutputStructure only descends into .properties, so handing it the raw
registry value collapsed every nested group to one untyped row and dropped its
children. normalizeTriggerOutputs converts between the two, preserving leaves
that already declare properties/items and merging the 13 hybrid nodes that carry
both a type and inline children.
Measured across all 368 triggers before changing anything: 155 identical, 213
divergent, and the divergence was purely the nesting encoding — no node has a
non-string type, and a group never carries its own string description, so
leaf-vs-group classification is unambiguous. That is what makes a nested property
literally named 'description' (42 of them) survive.
Deletes the static path: extractTriggerOutputs, resolveTriggerBuilderFunction,
resolveTriggerOutputsConstant, readTriggerSiblingModules,
getWebhookProviderConstants, plus resolveConstStringValue and matchQuotedProperty
which the config fix had already stranded.
20 output sections recovered (linear 79->93, tiktok 6->11, jira 44->45) and 1698
rows. Verified independently: zero sections lost across all 289 generated files,
no file lost rows, output deterministic across regeneration.
The 96 deletions are all corrections, not losses. 70 are confluence fields the
parser flattened out of `comment: { ...buildContentEntityFields(), parent: {…} }`
and rendered as top-level trigger outputs; they reappear nested under their
parent in the same hunk. 8 are greenhouse key ordering, 6 are intercom
descriptions the parser had dropped, 1 is a vercel row moving position.
Generator: 4069 -> 3903 lines.
* chore(test): silence vite 8 deprecation warnings in the sim vitest config
@vitejs/plugin-react v4 targets pre-rolldown Vite: it sets `esbuild.jsx`
and `optimizeDeps.rollupOptions`, both deprecated under Vite 8's oxc
pipeline, and self-reports that plugin-react-oxc should be used instead.
v6 is that plugin merged back under the original name — it requires Vite
^8, drops Babel entirely, and emits none of those options.
Vite 8 also resolves tsconfig paths natively, so vite-tsconfig-paths is
replaced by `resolve.tsconfigPaths`.
Full apps/sim suite unchanged: 1483 passed / 2 skipped files,
20415 passed / 30 skipped tests.
* refactor(docs): drop 33 more comments that restated their code
Second pass over the generator, e.g. `// Copy icons from sim app to docs app`
above `copyIconsFile()`. Kept the multi-line runs (those carry reasoning), the
ones with concrete examples, and the one marking a deliberate empty catch.
Verified byte-identical output across all 289 generated files.
Generator: 3903 -> 3870 lines, 4306 at the start of this branch.
* refactor(ci): read package.json once in the audit runner
auditScripts() re-read the manifest the module body had already loaded.
* fix(pdl): name the tools directory after the tool ids
People Data Labs declared `pdl_*` tool ids under `tools/peopledatalabs/`. Every
other integration names the directory after its id prefix -- 259 of 260 before
this, and PDL was the only exception.
The docs generator locates a tool's definition by deriving the directory from the
id prefix, so it looked in `tools/pdl/`, found nothing, and returned null for all
11 tools. peopledatalabs.mdx rendered eleven bare `###` headings with no
description, no Input table and no Output table.
Renaming the directory rather than the ids: tool ids are persisted in saved
workflows, so renaming those would break existing users. The directory is
internal -- 15 files' imports.
Fixed at the source rather than teaching the generator a fallback. A special case
would have left the invariant broken and the next integration free to break it
again; now 260 of 260 hold, and the generator needs no exception.
peopledatalabs.mdx: 11 empty headings -> 456 lines. Repo-wide: zero pages with an
empty action body.
|
||
|
Siddharth Ganesan
|
93b68f049a | feat(mship): async runs- #6329 | ||
|
Waleed
|
2ba455647b |
fix(db): bind every raw-sql Date through its column encoder (#6337)
* fix(db): bind every raw-sql Date through its column encoder `drizzle()` overwrites postgres-js's temporal serializers (OIDs 1082/1083/ 1114/1184/1182/1185/1115/1231) with an identity function because drizzle maps timestamps itself through the column's `mapToDriverValue`. A raw `sql` template carries no column context, so an interpolated `Date` skips that mapping, reaches the identity serializer unchanged, and the wire encoder throws `ERR_INVALID_ARG_TYPE`. The pools' `prepare` / `fetch_types` options are irrelevant: the serializer swap happens for all four combinations. Five live sites still interpolated a bare `Date`, the stale schedule-job filter among them — it has no try/catch, so a database async backend would surface a 500 from the schedule tick. Bind each cutoff with `sql.param(date, column)`. The testing `sql` mock's guard cannot see untested code or the tests that override the drizzle-orm mock, so add `check:sql-date-binding`: a Babel-AST audit over apps/** and packages/** that resolves Date-valued bindings per file and rejects any that reach a raw template unbound. Correct the mock's comment, which attributed the failure to postgres-js under `fetch_types: false`. * fix(scripts): require the documented sql-date-bound annotation form and a reason |
||
|
Waleed
|
82e654d73c |
fix(cron): bind stale async-job cutoffs through the column encoder (#6327)
The stale-processing predicate interpolated `Date` values straight into a raw `sql` template. A raw template carries no column context, so drizzle skips `PgTimestamp.mapToDriverValue` (which stringifies via `toISOString`) and postgres-js receives a `Date` it cannot serialize under the pools' `prepare: false` / `fetch_types: false` options. Every run of the job has failed its async-job sweep since the change shipped, leaving stuck jobs unreaped while the surrounding typed `lt(column, date)` sweeps succeeded. Bind both cutoffs with `sql.param(date, column)`, matching the workflow sweep in the same handler. The testing `sql` mock already rejected `sql.param(date)` for this reason but not the interpolated form that shipped, so extend it to cover both. That guard alone fails three existing tests when the fix is reverted, and the full suite shows no other route binding a bare Date this way. |
||
|
Waleed
|
ab257555d8 |
fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs (#6311)
* fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs
Better Auth 1.6.23 calls the account-linking handler with trustProviderByName:
false, which disables the trustedProviders allowlist for SSO entirely. Trust now
comes only from the provider's domainVerified flag, which Sim never set — so any
user who already had a Sim account was stranded on "account not linked". Entra
never sends email_verified, so this hit every Microsoft tenant.
Sim already proves domain ownership via sso_domain before a provider can be
registered, so the register route mirrors that decision onto domainVerified.
The column defaults to true so existing providers keep signing in across the
deploy, since enabling the option turns sign-in into a hard gate.
Also enforces the providerId uniqueness Better Auth already assumes: it rejects
any id that exists in any tenant and resolves providers by that column alone, so
a second customer picking "azure-ad" could not register at all and got an opaque
422. Sim now returns a 409 naming a free id, and a unique index makes the
duplicate-row state unreachable.
* fix(sso): revoke domain trust when verification is removed mid-update
The create path re-checks domain ownership after Better Auth persists the
provider and rolls the row back if the verified sso_domain row disappeared in
that window. The update path had no equivalent, so deleting the verified domain
while updateSSOProvider was in flight still set domainVerified, restoring
same-email account-linking trust for a domain the org no longer proves it owns.
The update path has no newly-created row to roll back, so it clears the flag
instead: that denies linking and blocks sign-in on the provider until the domain
is verified again.
* fix(sso): make domain-trust grants atomic and propagate revocation
Greptile flagged that the ownership check and the domainVerified write were
separate statements, so a domain deleted between them still ended with trust
granted. Two changes close it from both sides.
The grant now folds the ownership test into the UPDATE's WHERE clause, so
Postgres evaluates both in one statement and the write matches nothing once the
proof is gone.
Removing a verified domain now clears domainVerified for providers on that
domain, in the same transaction as the delete. This was a standing gap, not just
a race: deleting a domain previously left linking trust set indefinitely.
Together the provider cannot end up trusted without current ownership in either
commit order — if the grant lands first the delete clears it, and if the delete
lands first the grant no-ops.
* fix(sso): report a refused domain-trust grant instead of returning success
The conditional grant could match zero rows if the verified domain was deleted
between the pre-write check and the write. The route ignored that and returned
200, leaving a provider that cannot sign anyone in while telling the admin it
saved.
The grant now reports whether it matched, and that result is the single decision
point on both paths: the create path rolls the provider back, the update path
clears the flag, and both return SSO_DOMAIN_NOT_VERIFIED. This also drops the
separate post-write ownership read, since the UPDATE re-tests ownership itself.
* feat(sso): let admins map IdP claims, and trim setup comments
Identity providers disagree on which claim carries each value — Entra can send
the address as `upn` rather than `email` — and the mapping was hardcoded, so a
mismatch had no fix in the UI at all. Adds an Attribute mapping section for both
protocols, defaulting to each protocol's standard claim names shown as
placeholders, so the common case still needs no input.
Editing an existing provider now loads its stored mapping and only treats a
value as an override when it differs from the default, so a saved custom mapping
is never silently rewritten.
* feat(sso): expose the standard enterprise IdP options in the setup form
Rounds out the form with the options Better Auth already accepts but the UI hid,
so a non-standard IdP no longer dead-ends at a field that cannot be set.
SAML gains signature algorithm, digest algorithm and NameID format. Only SHA-256
and stronger are offered: Better Auth warns on SHA-1 as deprecated and rejects
anything outside its secure set, so weaker choices would only produce failed
saves.
SAML also surfaces the SP Entity ID beside the ACS URL. IdP admins are usually
handed a vendor metadata document; Sim does not publish one, and these are the
two values it would carry.
OIDC gains authorization, token and JWKS endpoint overrides for providers whose
discovery document is incomplete or unreachable. Discovery still fills them in
when they are left blank.
All of these load from the stored config when editing, so re-saving a provider
cannot quietly drop them.
* fix(sso): withhold domain trust from personal providers on the hosted deployment
A personal (org-less) provider has no verified domain behind it, but the trust
grant treated it as authoritative anyway. On the multi-tenant deployment that is
an account-takeover primitive: anyone able to register one could claim a domain
they do not own, point it at their own IdP, and have a sign-in auto-link to an
existing account on that domain.
Sim's UI always registers org-scoped, so this only reaches direct API callers.
Self-hosted deployments are single-tenant — the operator is the only tenant —
so the org-less path keeps working there.
Also clears the attribute mapping when the protocol changes: claim names are
protocol-specific, so an OIDC override carried into a SAML config would save a
mapping the IdP cannot resolve.
* docs(sso): correct the personal-provider trust note after the hosted gating
* fix(sso): drop the inert SAML algorithm selects, make NameID format clearable
The signature and digest algorithm selects were placebo controls. Tracing
@better-auth/sso 1.6.23, those two values are only read by validateConfigAlgorithms,
mergeSAMLConfig and sanitizeProvider — createSP and createIdP never pass them to
samlify, so nothing they select reaches the SAML exchange. Bugbot separately
noted they could not be cleared, since Better Auth merges with `??` and omitting
a key keeps the stored value. A control that neither applies nor clears should
not exist, so both are removed.
NameID format is genuinely wired (createSP passes it as nameIDFormat) and is
kept, but is now always sent rather than omitted when set to the provider
default. samlify falsy-guards the value, so an empty string reads as unset and
"Provider default" can actually clear a stored override.
The read-only provider view also now shows the SP Entity ID and the ACS label
for SAML — admins land there after saving and need the same two values the form
says their IdP requires.
* docs(sso): tighten the personal-provider note to the self-host path it describes
* fix(sso): forward an empty SAML NameID format so the provider default can be restored
The form sends an empty identifierFormat when the admin selects "Provider
default", but the route dropped it with a truthiness check. Better Auth merges
SAML config with `??`, so an omitted key retains the stored value — the selection
appeared to apply and silently did not.
Forwarding the empty string makes it reach the merge, and samlify falsy-guards
nameIDFormat, so it reads as unset. Selecting the provider default now actually
clears a stored override.
* fix(sso): revoke trust for providers whose domain is spelled with a wildcard
Migration 0268 grandfathered providers by normalizing their domain with
lower + btrim + a stripped leading `*.`, so sso_provider.domain can hold
`*.acme.com` while its verified sso_domain row holds `acme.com`. The revoke on
domain deletion compared the raw column, so such a provider matched nothing and
kept domainVerified after its ownership proof was gone.
The comparison now applies the same normalization 0268 used, so a grandfathered
row is matched the way it was written.
* fix(db): give the SSO index migration the concurrent-build convention it skipped
packages/db/scripts/migrate.ts documents the required shape for CONCURRENTLY
statements, and the previous migration follows it. 0284 did not, and the
omission is silently destructive.
migrate.ts sets a session lock_timeout of 5s, which survives the embedded
COMMIT. CREATE INDEX CONCURRENTLY waits on every concurrent write transaction in
the database — not only ones touching this table — so on a busy database the
build is cancelled with 55P03 and leaves an INVALID index. The retry then
replays the file, IF NOT EXISTS skips the invalid index, and DROP INDEX removes
the only working index on provider_id. The migration journals as applied and
exits 0 with provider_id unindexed and uniqueness unenforced, reopening the
cross-tenant provider resolution this migration exists to close.
Adds SET lock_timeout = 0 around the concurrent statements, a pre-drop of the
target index name so a replay rebuilds rather than skips, and restores the 5s
timeout afterwards. Verified by stranding an INVALID index and replaying: the
end state is a valid unique index with uniqueness enforced.
Also corrects the sso() comment that claimed domainVerified confines linking to
matching email domains. link-account.mjs blocks on
`!isTrustedProvider && !userInfo.emailVerified`, so an IdP asserting
email_verified links regardless of domain — the flag narrows nothing on its own.
* fix(sso): give the Enter shortcut the same guard as the Add domain button
The Enter handler called handleAdd unconditionally while the button was disabled
during an in-flight add, so repeated presses could issue overlapping requests.
Both now read one canAddDomain flag rather than duplicating the condition.
* fix(sso): stop the provider ID being editable after it is saved
Renaming it was never useful and always destructive. The value forms the
redirect URL registered with the identity provider, so changing it breaks
sign-in until the IdP is updated. Worse, the register route selects
create-vs-update by (providerId, organizationId), so a renamed id misses and
registers a SECOND provider; the settings page renders providers[0], so the
duplicate is invisible, there is no delete action to remove it, and existing
account rows still reference the old id.
Editing now shows it as a read-only copyable value, and the create form says up
front that it cannot be changed later.
Also hoists the suggestion list to module scope — it was rebuilding 44 objects
on every keystroke anywhere in the form.
* fix(sso): stop persisting generated IdP metadata so SAML cert rotation works
The route stored an IdP metadata document built from cert + entryPoint even when
the admin supplied none. The form loads that document back into its optional
metadata field and resends it, and on the next save it wins over the certificate
— so rotating a SAML signing certificate through the form appeared to succeed
and changed nothing.
Only metadata the admin actually pasted is persisted now. With none stored,
Better Auth's createIdP builds the IdP from issuer, entryPoint and cert, which
are the fields the form edits. No SAML providers exist in production, so this
changes no live tenant.
* fix(sso): always write SAML IdP metadata so clearing it takes effect on update
Not storing generated metadata fixed new providers but not existing ones: Better
Auth merges SAML config with `??`, so omitting the key let a previously stored
document survive and keep overriding the certificate.
The key is now always written, empty when the admin supplied none. createIdP
falsy-guards it and falls back to issuer/entryPoint/cert, so clearing the field
actually clears it.
* fix(sso): hold the domain proof under a row lock while granting trust
Two fixes from review.
The trust grant folded the ownership test into the UPDATE's WHERE clause, but
under READ COMMITTED the EXISTS subquery is evaluated against the statement's
original snapshot. A delete committing while the UPDATE waited on the provider
row could therefore still see the removed sso_domain row and grant trust after
ownership was gone. The grant now selects the proof FOR SHARE inside a
transaction before writing, so the delete blocks until it commits, and if the
delete committed first the select finds nothing and no trust is written.
Editing a SAML provider also broke on configs written by the previous commit:
hydration used `config.idpMetadata?.metadata || config.idpMetadata`, and
`{ metadata: '' }` is falsy at the property but truthy as an object, so an object
landed in a string field and failed validation on save. It now narrows on the
type and handles both the object and legacy bare-string shapes.
* refactor(sso): write the two merge-sensitive SAML fields the same way
idpMetadata and identifierFormat both exist to defeat Better Auth's `??` merge,
which silently keeps a stored value when a key is omitted, but they were written
differently — one always, one only when defined. Both are now always written,
empty when unset, under one comment explaining why and noting that each is
falsy-guarded downstream.
Also drops a redundant saveDisabled: false; the prop already defaults to false.
* fix(sso): report the row the trust grant actually matched
The grant returned true once it found the proof, without checking that the
provider UPDATE matched anything, so its boolean did not always mean what
callers read it to mean. It now reports the matched row.
* fix(sso): restore provider domain trust when a domain is re-verified
* fix(sso): correct the domain-removal warning now that it disables sign-in
* chore(sso): trim verbose comments
* fix(sso): revert a rejected SSO update instead of leaving it stored
|
||
|
Vikhyath Mondreti
|
c530d276b7 |
fix(env): flag combinations for sandboxes (#6308)
* fix(env): flag combinations for sandboxes * more changes |
||
|
Vikhyath MondretiandSiddharth Ganesan
|
117fe3137b |
feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift, workflow exec cancellations (#6247)
* feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift * fix(execution): harden compatibility and secret diagnostics * fix(execution): harden generated JavaScript literals * fix(execution): align timeout cleanup semantics * fix(tables): decouple stale job cleanup * fix(execution): drain stale workflow backlog * test(sandbox): make deadline assertions timing-safe * fix(execution): lock cleanup candidate batches * fix(execution): preserve cleanup failure metrics * cancel route fixes * separate out mship template and func template * fix * fix(execution): harden secret projection and block runs * fix(workflow): validate draft execution state * run from block ui disabling * feat(copilot): expose Sim sandboxes to mothership * feat(copilot): expose sandbox capability catalog in VFS * Updates * fix legacy logs showing up * fix(copilot): keep sandbox config visible * fix model provenance issues * fix lint' * more lint * more * test(files): align provenance copy query order * consolidate migrations, rollout compat * integration projections * update skills * fix * add provenance linters * fix: address review and compatibility regressions * fix: make tool boundary audit Bun 1.3 compatible --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com> |
||
|
Theodore Li
|
35fd4ef42f |
improvement(self-host): simplify capability setup configuration (#6230)
* feat(self-host): add capability-aware setup * fix(self-host): preserve capability compatibility * fix(copilot): honor preview availability server-side * improvement(self-host): centralize capability resolution * fix(self-host): preserve integration availability paths * fix(testing): align capability-aware config mocks * improvement(self-host): simplify capability setup configuration * fix(setup): preserve unowned storage overrides * fix(self-host): reconcile storage and allowlists * fix(integrations): preserve connect deep links |
||
|
Waleed
|
36aace765e |
fix(auth): correct callback URL resolution across SSR and hydration (#6217)
* fix(sso): derive the SSO callback URL during render
`callbackUrl` was seeded into `useState('/workspace')` and overwritten from
a `useEffect` that read `searchParams`, so the first painted frame always
carried the default. On any deep link with `?callbackUrl=`, the "Sign in
with email" and "Sign up" links briefly pointed at `/workspace` instead of
the requested destination, and a click landing in that window navigated to
the wrong place.
- derive `callbackUrl` from `searchParams` during render; the validation
gate is unchanged, so an off-origin or malformed value still falls back
to `/workspace`
- keep the warning for a rejected value in an effect, now keyed on the
param itself rather than the `searchParams` object, so it fires once per
actual change instead of once per identity change
- add first-frame tests via `renderToString`, which runs no effects and so
pins exactly the window the old code got wrong
* fix(auth): resolve callback URLs against the app origin server-side
`validateCallbackUrl` compared against a sentinel base
(`https://callback-url-validator.invalid`) when `window` was undefined, so
the server rejected every absolute URL — including the same-origin ones the
function documents as valid. A component deriving a callback URL during
render therefore produced one destination in the SSR markup and a different
one after hydration.
The exposure was not new to the SSO form: `login-form.tsx` and
`signup-form.tsx` already derive their callback URL during render on
`force-dynamic` pages, so both carried the same divergence.
- resolve against the deployment's own origin server-side, so the server
reaches the same verdict the browser will after hydration
- fall back to the sentinel when the app URL is unset or unparseable, which
keeps the server fail-closed: absolute URLs are rejected, as before
- cover the absolute same-origin case and the unset-app-URL fallback in the
existing suite; all 15 open-redirect rejection cases are unchanged
* fix(env): drop the getBaseUrl browser-origin fallback
The fallback was added in #6214 as a safety net while the real cause — the
hosted env script losing its `beforeInteractive` strategy — was fixed in the
same PR. With the injection ordering restored, `window.__ENV` is populated
before hydration, so the fallback is unreachable in any correctly configured
deployment.
Guessing the origin was also unsafe in the one case it could still fire. An
opaque origin — a sandboxed iframe, and `/chat/*` is deliberately embeddable
— serializes to the string `'null'`, which is truthy, so `getBaseUrl()` would
have returned `'null'` and every call site would have silently built
`null/api/...`. A throw surfaces the misconfiguration instead of encoding it
into request URLs.
- restore the unconditional throw when NEXT_PUBLIC_APP_URL is unset or blank
- mirror it back in the shared testing mock
- flip the two fallback tests to assert the throw, keeping whitespace-only
coverage
Server-side behavior is unchanged: there was never a `window` to fall back
to, so callers that already guard `getBaseUrl()` (`getBaseDomain`,
`validateCallbackUrl`) keep their existing fail-closed paths.
|
||
|
Waleed
|
dd7951563e |
fix(env): restore beforeInteractive on the hosted public env script (#6214)
The hosted `<PublicEnvScript>` rendered a plain `<script>`, which lands at the end of `<head>` — after the ~40 `<script async>` chunk tags Next emits at the top of the document. An async script runs as soon as its fetch resolves, so on a warm cache a Next chunk could execute (and hydration begin) before the parser reached the env tag, leaving `window.__ENV` undefined for the first render. That surfaced as "Something went wrong" on the workflow page, since `getBaseUrl()` throws during the deploy modal's render, and as the socket falling back to the page origin instead of NEXT_PUBLIC_SOCKET_URL. Regressed in #5522, which replaced next-runtime-env's PublicEnvScript (to avoid its unstable_noStore forcing dynamic rendering) with a static equivalent that dropped the beforeInteractive strategy. - render the library's own `<EnvScript>`, which defaults to beforeInteractive and does not call unstable_noStore — hosted and self-hosted now share one implementation and one loading strategy - drop the hand-rolled serialization and `<` escaping; Next's beforeInteractive path already runs the payload through htmlEscapeJsonString, which escapes `& > < U+2028 U+2029` - fall back to the browser origin in getBaseUrl() rather than throwing, so a missing injected env can never tear down a page through the error boundary; server-side callers still fail loudly - read NEXT_PUBLIC_EMAIL_PASSWORD_SIGNUP_ENABLED via getEnv() in the SSO form, matching login/signup/auth-modal — `env.X` returns the build-time placeholder, not the runtime value |
||
|
Vikhyath Mondreti
|
b8ec114382 |
improvement(chat): secrets mounting / exposure improvements and controls (#6191)
* fix(copilot): secrets injection into sandbox * improvement(chat): secrets mounting / exposure improvements and controls * fix(secrets): simplify copilot mounting flow * test(secrets): preserve standard tool permissions * fix(copilot): bind workflow tool completions * fix(secrets): preserve own environment keys * fix(copilot): release failed workflow claims * fix(copilot): trust compacted workflow completion |
||
|
|
13772565a3 |
fix(chat): show deployment passwords to admins (#6177)
* fix(chat): show deployment passwords to admins
* fix(chat): reject whitespace-only passwords
* fix(chat): preserve password visibility on regenerate
* fix(chat): harden password reveal and close deployment lockout paths
Follow-ups from a security review of the password reveal endpoint. The
permission model itself was correct — the reveal is gated on workspace
admin via the canonical resolver, so derived org-admin access is honored.
These address secret handling and validation around it.
- Cap set-path passwords at the same 1024 chars the chat login accepts.
Neither the input nor the schema bounded length, so a longer password
saved fine and then failed the login POST on length before auth ran,
locking every visitor out permanently.
- Discard the revealed password when the field is hidden. It previously
stayed in state and in the input's DOM value with Copy still armed, so
the field read as hidden while still handing out the plaintext.
- Evict the decrypted password from the mutation cache on unmount, and
correct the TSDoc claiming it was never retained — it sat in the
MutationCache for the default five minutes after the modal closed.
- Validate the password inside performChatDeploy, the writer both callers
must use. The copilot deploy_chat tool bypasses the route contract and
could still store a whitespace-only or over-long password, or create a
password-protected chat with no password at all.
- Stop echoing raw decryption errors from the reveal endpoint.
- Only persist a new password when the chat ends up password-protected;
PATCH { authType: 'email', password } used to re-arm the secret that the
auth-type branch had just cleared.
Also replaces the hand-rolled copy state with useCopyToClipboard, which
fixes an unawaited clipboard write that surfaced as an unhandled rejection
and a "Copied" confirmation shown even when the write failed.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(chat): correct password-change confirmation gate and stale reveal error
Addresses both open Bugbot findings.
- shouldConfirmPasswordChange keyed on "a chat exists" rather than "a password
exists", so switching a public chat to password protection for the first time
asked the admin to confirm changing a password that was never set. It now
takes the existing-password signal the component already computes.
- A failed reveal left "Failed to load the current password" on screen while the
admin typed or generated a replacement, because the mutation only drops its
error on the next attempt. Editing or regenerating now resets it.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Waleed
|
e5d2f7d80a |
fix(realtime): stop read-only members persisting block positions (#6174)
* fix(realtime): stop read-only members persisting block positions The socket operation ACL granted the `read` role block.updatePosition and blocks.batchUpdatePositions on the premise that they were ephemeral cursor sync. They are not: both are followed by persistWorkflowOperation, which UPDATEs workflow_blocks.positionX/positionY and bumps workflow.updatedAt. A member holding only `read` on a workspace could therefore permanently rewrite the coordinates of every block of every workflow in it — a write across the read/write boundary. Live cursors ride their own `cursor-update` event, and the smooth-drag broadcast is the UNCOMMITTED position path, which returns before persisting and never consults the role table — so the read role needs no grant at all. * test(realtime): assert the role ACL against production, not a fixture The shared ROLE_ALLOWED_OPERATIONS fixture still listed the two position operations for the read role, and three tests compared that fixture against itself — so they certified whatever it said, including the grants this PR removes. They now assert checkRolePermission over the protocol's complete operation list (the fixture's copy omits subblock/variable/admin-only ops), plus one test that pins the fixture to the production ACL so the two cannot drift apart again. |
||
|
Waleed
|
d89ab4a8ee |
fix(security): bound response bodies in secureFetchWithPinnedIP by default (#6169)
secureFetchWithPinnedIP only capped a response body when the caller passed maxResponseBytes; with the option absent the body streamed into an unbounded Buffer.concat. Several tool proxies whose target host is user-supplied (Jupyter, ClickHouse, Grafana, 1Password Connect) and the RSS poller called it without that option, so an attacker-controlled server answering with an endless chunked body could grow the shared process heap until it was OOM-killed. Make the cap fail-safe: default to 100MB (and treat a non-positive value as the default) so there is no unlimited mode, then pass tighter explicit caps at the user-supplied-host sites. Responses that carry no body (HEAD, 204, 304) are exempted from the content-length pre-check — they advertise the resource size as metadata, which would otherwise spuriously fail a HEAD probe of a large file or an RSS conditional-GET 304. |
||
|
|
18214158b9 |
fix(sanitization): secret exposure in function and agent trace spans (#6000)
* fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
452d82a636 |
perf(tools): read tool metadata instead of the registry on client paths (#6155)
* perf(tools): read tool metadata instead of the registry on client paths Cuts the last four edges that pulled `@/tools/registry` into the workspace shell. Every workspace route drops ~4,700 modules: route before after /w (canvas) 6,592 1,908 -71% /logs 6,227 1,543 -75% /tables 5,903 1,217 -79% /files 5,996 1,310 -78% workspace layout 5,751 1,063 -82% Dev cold compile of the canvas, n=3, cache cleared between runs: before 32.3s / 31.4s / 30.1s RSS 9.0-12.5 GB after 22.4s / 22.2s / 21.6s RSS 7.8-9.2 GB That lands where the `dev:minimal` escape hatch measured (20.0s / 6.7 GB) without its downside — `dev:minimal` swaps in curated registries that drop ~250 services, whereas this keeps every tool working. Rewired: - `block-outputs` -> `getToolOutputsMetadata` (needed `outputs`) - `serializer` -> `getToolParams` (needed `params`) - `validation` -> `hasToolId` (needed existence only) - `tools/params` -> `getToolMetadata` (needed `params`, `oauth`, `name`) `tools/params.ts` was the stubborn one: `mcp-dynamic-args.tsx` imports only `formatParameterLabel` from it, so the whole registry rode in behind a string helper — the same shape as the `mergeToolParameters` edge cut earlier. Adds a third generated artifact, `tool-ids.ts` (~110 KB). Resolution needs only the key set, so `@/tools/metadata` and `@/tools/metadata-outputs` both resolve through it and stay independent of each other, and an existence check costs ~110 KB instead of ~4 MB. Behaviour preservation was the risk here: `getTool` resolves an unversioned name onto its newest version, and a plain key lookup would have silently reported 246 versioned tools as missing. `resolveToolId` is reproduced against the id set and differentially tested — 4,404 probes (every id, every stripped base name, and an unknown) comparing old vs new resolution and existence: 0 mismatches. `ToolWithParameters.toolConfig` and `SubBlocksForToolInput.toolConfig` narrow from `ToolConfig` to `ToolMetadata`. The only external reader is `tool-input.tsx`, which uses `.name`. * docs(tools): point the boundary skill at the three metadata modules The skill still routed `hasToolMetadata` and `getToolIds` to `@/tools/metadata`, but this PR moved id resolution into `@/tools/tool-ids`. Left as-is it would send the next caller to the 4 MB module for an existence check that costs 110 KB — the exact mistake the skill exists to prevent. Also records the two properties a caller can silently get wrong: lookups guard with `Object.hasOwn` (a bare bracket lookup returns inherited prototype members), and they resolve unversioned names (246 tools are versioned, and a plain lookup reports them missing rather than crashing). * fix(tools): cut the settings-route registry edge and fix serializer test mocks Two findings from review, both real. The settings route still reached the registry: settings/[section]/page.tsx -> settings.tsx -> (dynamic import) ee/access-control/components/access-control.tsx -> group-detail.tsx -> tools/utils.ts -> tools/registry.ts It reads `getTool(id)?.name` — metadata — so it moves to `getToolMetadata`. The earlier audit missed it because it walked only from the canvas route, and the edge hides behind a dynamic `import()` that a static walk skips. Serializer tests mocked the wrong module. `Serializer` now reads params via `getToolParams` from `@/tools/metadata`, but the tests still only mocked `@/tools/utils`, so they controlled nothing and passed because the real generated artifacts happen to agree with the fixtures. Adds `toolsMetadataMock` to `@sim/testing/mocks`, backed by the same `mockToolConfigs` as `toolsUtilsMock` so a test mocking both sees one consistent tool universe, and mocks it in the three serializer suites. Verified the mock is now load-bearing: pointing it at a sentinel param makes the three user-only-required validation tests fail, and restoring it returns all 110 serializer tests to green. Before this they passed either way. * fix(tools): freeze the tool id array handed out by getToolIds `getToolIds()` returned the module's internal array by reference, so a caller doing `getToolIds().sort()` would reorder it in place and silently corrupt every later lookup — the in-place-mutation footgun `.claude/rules/sim-react-performance.md` calls out. Frozen rather than copied: the array is consumed in loops, so copying would allocate on every call. Freezing makes the mutation throw instead of corrupt, and `[...getToolIds()].sort()` still works. Return type is now `readonly string[]`, so the mistake is a compile error rather than a runtime surprise. No caller mutates it today; this is closing the hole, not fixing a live bug. * test(tools): enforce that the two tool-id resolvers never diverge `resolveToolId` now exists twice on purpose — `@/tools/utils` resolves against the live registry (so a tool added before regeneration still resolves at runtime), `@/tools/tool-ids` against the generated id list (so client code resolves without importing 4,300 tools). Nothing structurally kept them in step; a change to versioning logic in one would silently drift from the other. `tool-metadata:check` now asserts they agree across every id, every stripped base name, and an unknown — 4,404 probes — and only after the staleness check passes, so a missing regeneration reports as staleness rather than as drift. Verified it fails: breaking resolution for `gmail*` exits 1; restoring it passes. It cannot live in a vitest suite. `vitest.setup.ts` globally mocks `@/tools/registry` to an empty map, so `getTool` resolves nothing there — a parity test written as a spec passes or fails for the wrong reason. Both facts are recorded where the code is. Both resolvers stay exported. An earlier pass here un-exported the `@/tools/utils` one as dead; `tools/utils.server.ts` imports it through a multi-line import that a grep missed, and `tsc` caught it. Its doc now says which resolver a caller should reach for instead of leaving two identically-named functions unexplained. |
||
|
Waleedandmzxchandra
|
10bfb5d139 |
feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991)
* feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on |
||
|
Theodore LiandClaude Opus 5
|
c5cc6ce26c |
feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137)
* feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Waleed
|
ed23330f88 |
feat(knowledge): opt-in hybrid lexical + vector retrieval for KB search (#6124)
* feat(knowledge): hybrid lexical + vector retrieval for KB search KB search ranked purely on pgvector cosine distance, which retrieves exact tokens (error codes, ticket keys, identifiers, rare product names) poorly. Add a full-text leg over the already-present generated `embedding.content_tsv` column and its GIN index — no migration, no re-indexing — and fuse it with the vector leg by reciprocal rank. Both legs run concurrently and share the same visibility and tag-filter predicates; the lexical leg is best-effort and falls back to vector-only on failure. Hybrid is the default for every caller. `searchMode: 'vector'` on the internal and v1 contracts (and an advanced Retrieval Mode dropdown on the Knowledge block) restores the previous behavior. Both search routes now share one `executeKnowledgeSearch` dispatch instead of duplicating the three-branch retrieval logic. * change(knowledge): make vector the default search mode, hybrid opt-in Every existing caller — workflow block, v1 API, copilot, guardrail RAG — keeps its current ranking. Hybrid retrieval is now requested explicitly via `searchMode: 'hybrid'`. Also routes the copilot knowledge tool through the shared `executeKnowledgeSearch` dispatch so all four callers share one retrieval path, and documents `searchMode` on the public v1 search endpoint in the OpenAPI spec. * docs(knowledge): document the hybrid retrieval mode Regenerates the knowledge integration reference for the new searchMode tool param, and adds a Retrieval Mode section to the knowledge base workflow guide explaining when hybrid beats vector-only. * fix(knowledge): stop rank fusion from starving the lexical leg Rank n in one leg always ties rank n in the other, so ordering the fused list by score alone let whichever leg was scored first take every tied slot. At topK=1 that meant a hybrid search returned exactly the vector-only result and discarded the exact keyword match the mode exists to recover. Selection now orders by score and drains each tie group round-robin, taking from whichever leg has contributed fewest rows so far. The lexical leg is passed first so it wins a total tie, since a chunk the vector leg ranked below its distance threshold is the case hybrid was opted into for. * fix(knowledge): credit a shared hit to every leg that returned it Attributing a row found by both legs to a single leg left the round-robin owing the other leg a slot it had already been served. With a shared rank-1 hit and topK 2, that evicted the lexical-only row — the exact match hybrid was enabled to recover — in favor of the vector-only one. A shared row satisfied every leg that returned it, so every one of them is now charged for it. Tie-breaking prefers the candidate whose least-served leg has been served least, which also removes the arbitrary best-rank attribution. * fix(knowledge): reject a whitespace-only copilot query explicitly The shared dispatch treats a whitespace-only query as absent and throws when no tag filters accompany it, where the previous vector-only call would have embedded the blank string and searched. Tighten the existing guard so the tool returns its normal message instead. * fix(knowledge): fan the keyword leg out per knowledge base The vector leg caps candidates per base once getQueryStrategy sets useParallel, but the keyword leg always ran one global query with a single LIMIT. Searching several bases at once let whichever one ranks strongest lexically consume every slot, so an exact-token hit in a smaller base never reached fusion — the case hybrid exists to serve. The keyword leg now uses the same strategy: per-base queries under the same parallel limit, re-ranked globally on a selected ts_rank_cd. Both legs draw candidates the same way, so fusion combines rankings over the same pool. * perf(knowledge): stop the keyword leg detoasting every match's vector Selecting the cosine distance in the ranking query made Postgres detoast the 1536-dimension embedding and compute a distance for every full-text match before the LIMIT applied, so cost tracked how common the query term was rather than topK. On a 20k-chunk base with a term matching every row that was 61,055 buffer hits against 1,030 for the same query without the projection. Rank on ids and ts_rank_cd alone, then hydrate only the rows that survive the limit. Same results, and the worst case drops to ~27ms end to end. |
||
|
Vikhyath MondretiandClaude
|
7798e83489 |
feat(function): custom sandboxes (#6071)
* feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Waleed
|
0f1279979a |
fix(folders): stop archived rows skewing sortOrder, hide them from admin, and cover the two untested modules (#6062)
* fix(folders): stop archived rows skewing sortOrder, hide them from admin, and cover the two untested modules nextFolderSortOrder returned min - 1 over ALL rows including soft-deleted ones, so every delete ratcheted the floor further negative and never recovered — an archived folder at -400 forced the next new folder to -401 forever. Both minima (folders and child resources) now see only rows a user can still see, which is how the Files path has always worked. The admin workspace-folders endpoint counted and paginated soft-deleted folders, so an operator saw phantom folders and an inflated total, disagreeing with every user-facing list. Adds naming.test.ts and queries.test.ts. Both modules had zero tests and are mocked at every call site, so their bodies executed in no test anywhere. That left unasserted the two bug classes that caused real defects in the folder migration: the suffix sequence (must start at (1) and skip taken suffixes) and resourceType scoping on the id-keyed lookups, where a missing clause silently files a knowledge base under a table folder. Every assertion is mutation-checked. The first version of the sortOrder test was vacuous — for a root folder the parent condition is itself an isNull node, so a presence-only check passed with the soft-delete filter deleted; it now asserts the specific column. * refactor(testing): share the drizzle condition-tree helpers Asserting on WHERE clauses is the only way to pin a filter the row-queue mocks cannot enforce — a mock returns whatever was queued regardless of the predicate — so this pattern spreads to every test that guards a query's scoping. It had reached five local copies of the same flatten/has pair, four of them added by the tests in this branch. Moved to @sim/testing beside createMockSqlOperators, whose output shape they parse, so the helper and the node types it depends on live together. |