217 Commits

Include renames

Author	SHA1	Message	Date
Vikhyath MondretiandClaude Fable 5	957049e563	improvement(billing): paginate and parallelize the cycle-close sweep (#7103) ... * improvement(billing): paginate and parallelize the cycle-close sweep The sweep previously loaded every lagging candidate in one select and closed them serially — correct at any scale (per-subscription guarded marker claims make any interleaving safe) but O(fleet) in memory and wall clock. It now iterates candidates in keyset pages and closes each page with bounded concurrency; a total mapper keeps one failure from rejecting its page, and an interrupted sweep resumes from the durable markers on the next run. The member-lock select gains a deterministic ORDER BY: today member_user_id_unique keeps org rosters disjoint and every other locker is single-row, so no deadlock is currently constructible — the ordering is hygiene that also holds if membership exclusivity is ever relaxed. A partial index on exactly the sweep's candidate predicate (built CONCURRENTLY behind the runner's COMMIT breakpoint) keeps the scan O(lagging) rather than O(fleet); closes shrink it as they land. Validated end-to-end against Stripe test mode: one sweep over 304 staged lagging subscriptions (300 synthetic + 2 multi-member orgs closing concurrently + 2 real test-clock subscriptions with genuine overage) closed 304/304 with failed=0 in 667ms — the same fleet took ~302s serially — while both real subscriptions produced exactly one paid overage invoice each at the expected amounts, with zero invoices for the no-overage fleet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): restore the runner's lock_timeout and inline entitled statuses for the sweep index The concurrent-index migration now restores lock_timeout='5s' after its post-COMMIT section, matching the runner convention every prior CONCURRENTLY migration follows — without it, later pending files' DDL in the same session would wait indefinitely instead of hitting the runner's 55P03 retry path. The sweep's status filter becomes inlined literals derived from ENTITLED_SUBSCRIPTION_STATUSES: the planner can only prove a query implies a partial index's predicate from literals, and the parameterized generic plan demonstrably seq-scans where the literal form uses subscription_cycle_close_lagging_idx. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-25 23:17:11 -07:00
Waleed	0c38037bf1	fix(knowledge): harden ingestion pipeline (#7077) ... * fix(knowledge): harden ingestion pipeline * fix(knowledge): correct bounded ingestion edge cases * fix(knowledge): tighten redaction and chunk validation * fix(knowledge): close bounded ingestion gaps * fix(files): preserve parser complexity limits	2026-08-25 17:55:56 -07:00
Vikhyath MondretiandClaude Fable 5	a25d993266	improvement(billing): ledger-only usage + period-advance cycle close (#7078) ... * improvement(billing): make usage ledger-only and close cycles off period advance Usage is now the attributed usage_log ledger everywhere: the userStats baselines (currentPeriodCost / currentPeriodCopilotCost), the includeLegacyBaseline compatibility flag, the pro-snapshot join/leave machinery, and departedMemberUsage accrual are removed from all read and write paths. Cycle rollover (final sub-threshold overage collection, billedOverageThisPeriod reset, last-period bookkeeping) moves off the invoice.finalized payload parsing — dead for org subscriptions since May — onto a period-advance sweep with a durable per-subscription close marker (subscription.last_closed_period_start), transaction-enlisted Stripe outbox invoicing, and stamp-matched ledger sums. Enterprise closes are bookkeeping-only; reporting-anchor orgs advance the marker alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): include departed actors in close refresh and gate threshold billing on close currency Cycle close now unions current members with every actor holding org-stamped ledger rows in the closed period, so a departed member's daily-refresh consumption offsets the final overage exactly like their billed usage. Threshold billing defers with a pending-cycle-close no-op while a subscription's close marker lags its current period, so the shared billedOverageThisPeriod tracker can never mix an elapsed period's settlements with the new period's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): seal cycle-close races and align refresh actors with billed usage Threshold settlement revalidates the close marker and period under the tracker lock via the same isSubscriptionCycleCloseCurrent predicate the preflight uses, so a rollover between check and transaction aborts as a concurrent-state retry instead of settling against the wrong cycle. Terminal bookkeeping claims the close marker with its tracker reset, so a deletion racing an in-flight sweep close serializes through the one marker and the loser rolls back rather than re-billing settled overage; anchored enterprise deletions only claim the marker since their windows derive from the anchor, not Stripe bounds. A close with overage due but missing Stripe identifiers now defers loudly instead of claiming the marker and silently forgiving the money, the closed window's start derives from the ledger's own period stamps so anchor-day drift cannot misalign the refresh window, calculateSubscriptionOverage unions departed ledger actors into the org refresh deduction like the close does, and blocked accounts report their real ledger usage while staying blocked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): defer ownerless org closes and make the drift test load-bearing A close with overage due but no owner-role member now defers loudly like the missing-Stripe-identifier case instead of claiming the marker and silently forgiving the money. The stamp-drift test pins the marker before the stamped boundary so only the ledger-stamp lookup can produce the asserted window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(helm): bump chart to 1.6.0 for the billing-cycle-close cron job Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): revalidate the org roster under close locks and align invoice labels The close re-reads the member roster inside its transaction and defers on any change, mirroring threshold billing — an owner transfer moves the billed-overage tracker between rows, so a pre-lock roster could settle against the wrong tracker. Invoice labels now use the closed period's end month like every other overage path, and the sweep test's rows are shaped like rows the candidate query can actually return. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): hold cycle close for a settlement grace after rollover Billing attribution is frozen at run start, so a run straddling a rollover can insert elapsed-period-stamped rows after the period ends. Closing only once the rollover is older than any possible in-flight run guarantees the close's ledger sums are final; the sweep picks the period up on a later run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): resolve reporting windows through the canonical period resolver The close paths now ask resolveSubscriptionUsagePeriod whether a subscription derives its windows from a reporting anchor instead of re-checking metadata shape locally, so a malformed hand-edited anchor that the resolver rejects (falling back to Stripe bounds) books its Stripe-stamped ledger rows normally instead of skipping bookkeeping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): union departed ledger actors in org threshold settlement Organization threshold billing now reads the period ledger per user and unions the actors holding org-attributed rows with the current roster before computing refresh deductions — the same actor set calculateSubscriptionOverage and the cycle close use — so a departed member's usage cannot be settled without their daily-refresh offset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): claim the terminal period before deletion settlement and delete vestigial refresh bounds Subscription deletion now claims the close marker from the fresh subscription row before computing or charging final overage, serializing with the cycle-close sweep so both paths can never bill the same period — an in-flight close fails its guarded claim and rolls back, and the deletion settles against the row's real period instead of a possibly stale webhook payload. The per-user refresh bounds machinery is deleted outright: its only source was proPeriodCostSnapshotAt, which this PR stopped writing, and ledger entity stamps already scope refresh to org-attributed rows — a joiner's pre-join usage is user-stamped and can never enter the org refresh scan, while a departed member's org-stamped rows participate exactly like a current member's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(billing): scope daily refresh by entity stamps and close lagging periods before deletion settlement The refresh actor-list threading (userIds + departed-actor unions) violated the entity-stamp principle the ledger sums already follow: org-attributed rows from departed members counted in pooled usage but never consumed refresh on the monitor/resolved-usage paths. Daily refresh now scopes rows by the same write-time entity and period stamps as the ledger — no actor list anywhere — which deletes the unions, the rollup memberIds, and the org-specific refresh variant. Deletion settlement now closes any elapsed-but-unclosed period (grace bypassed — no later sweep revisits a canceled sub) before claiming the terminal period, so a deletion racing the sweep can no longer jump the marker past an unsettled period and silently forgive its final overage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): bucket refresh by clamped day so stamped stragglers stay in the deduction Refresh membership is now the entity/period stamps alone — identical to the ledger sums it offsets. A row written after the rollover but stamped to the elapsed period (attribution frozen at run start) is billed by the stamp-based close, so it must consume refresh too; created-at now only assigns the day bucket, clamped into the period, instead of excluding the row entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): pair the overage tracker with the marker's period and pin test clocks billedOverageThisPeriod only ever holds collections for the period that began at the close marker — the threshold gate blocks settlement whenever the marker lags. Both consumers now honor that pairing: a close that skipped forgiven periods counts nothing from the tracker against the period it bills, and the deletion settlement ignores the tracker when the marker was still lagging at claim time. Ignoring is provably safe in both cases because a lagging marker means no current-period collections exist. The cycle-close and daily-refresh suites pin the system clock: their grace and window checks compare fixed period fixtures against Date.now(), which made them dependent on the host date. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): reject lagging terminal claims and thread billingInterval into deletion bookkeeping claimTerminalPeriod no longer advances the marker over an unclosed elapsed period: a lagging marker is reported without a write so the deletion handler can run the elapsed close once more (healing a rollover that committed between close and claim) and only then seal the marker explicitly, with an error log, when the period is genuinely unclosable. Sealing preserves the in-flight-sweep abort guarantee. Deletion bookkeeping now passes the subscription's billingInterval through, so an enterprise reporting subscription whose interval lives on the row column (not metadata) still resolves as reporting-anchored and keeps its bookkeeping no-op, matching every other resolver call site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-25 16:34:44 -07:00
Vikhyath Mondreti	0a5b3801ea	feat(secrets): let workspace secrets opt out of redaction (#7045) ... * feat(secrets): let workspace secrets opt out of redaction * fix(secrets): certify no sandbox exemptions once the registry is incomplete * feat(secrets): carry visible secret values on the v2 list and document visibility * fix(secrets): read visible values by own property so prototype-named secrets cannot poison the list	2026-08-24 13:46:57 -07:00
Waleed	a20a5465ef	perf(db): optimize recurring query paths (#7014) ... * perf(db): optimize recurring query paths * perf(logs): batch keyset export reads * fix(workspaces): type nullable member count targets * fix(db): harden query performance changes * fix(logs): guard export stream cancellation	2026-08-23 17:15:10 -07:00
Vikhyath Mondreti	7ea8692e7d	fix(files): read a file nobody vouched for, as an untracked one already is (#6986) ... * fix(files): read a file nobody vouched for, as an untracked one already is A workspace file whose secret provenance could not be recorded was refused by every model and runtime boundary at once — attachments, mounts, and the tool routes that parse, transcribe or describe a file. A file that was never tracked at all returned exact-empty and worked fine. Both say exactly as much about their contents: nothing. So the refusal was a permanent penalty for having tried to record provenance and failed, with no way back, since nothing rewrites a file's provenance but another content write. Workspace files now do what the other durable surfaces do — proceed and record an audit entry naming the surface and the count, so the people who own the secrets are told. The state is distinguished from the four that really are unknown (missing row, version mismatch, stale binding, malformed), which stay refused; only a current sidecar that says so reads as unrecorded. Absence survives a merge rather than dropping through to exact, so a derived file cannot launder an unrecorded input into a positive claim. Script migration 0007 clears the files already stranded in the old state, taking the parent lock first in id order to match the live writer, with the unknown status re-checked under it so a concurrent write is never undone. Nothing changes for a legacy file: no sidecar row still means untracked, still reads exact-empty. The enforcement flag stays in place. * fix(files): narrow an initialization to a status the column accepts The union carries three states; the sidecar's CHECK constraint accepts two. Only the replace path discriminated — initialize forwarded the status verbatim, so an 'unrecorded' would have reached the database as a value it rejects, aborting the enclosing transaction rather than writing a bad row. Nothing passes one today, which is why it belongs here and not in a caller. Also names the storage mismatch on the type itself: a stored 'unknown' reads back as 'unrecorded', not as the union's 'unknown'. Same word, two layers, different meanings. * fix(provenance): repair only what the reader calls an absence The repair selected on status alone, which is wider than the branch it was meant to undo. The reader answers 'unrecorded' only for a sidecar that is version 1, bound to the file's current bytes, and holding a well-formed entries array; anything else is a fault it refuses and no policy relaxes. Clearing a fault was not a smaller claim but a larger one. Deleting the sidecar sets secret_provenance_version to NULL, and a NULL version reads as exact-empty — so a file refused because its provenance described bytes it no longer holds would have come back positively vouched for, silently, with no audit entry. Four such rows exist in production today. Both the candidate query and the delete now carry every condition, so a concurrent write cannot lose the distinction between an absence and a fault. Stale, unversioned and malformed sidecars stay refused exactly as the surface refuses them, and recover as they always have, on the next content write. * fix(provenance): store an absence apart from a refusal Relaxing a stored `unknown` assumed it always meant "nobody recorded this". It did not. The sidecar collapsed two opposite claims into one value: bytes nobody recorded, and bytes a writer refused on purpose — a child of an archive whose parent provably held secrets, a generated asset whose safety decision came back false, a transcode whose scanner knew secrets were present and could not locate them. Relaxing the second would have walked known-secret-bearing content through every model boundary at once. The decision type already told them apart: a registry that never ran is `safe: true`, while every refusal is `safe: false`. Only storage lost it. Writers now persist `unrecorded` and `unknown` separately, readers relax the first and refuse the second, and the constraint is widened to accept it. Only workspace files store this distinction, deliberately. Every other durable surface produces a non-exact sidecar from one condition — an incomplete incoming bundle or registry — which is always an absence. Files are the only surface that derives one stored object from another, so they are the only one that can refuse on purpose. The shared policy is unchanged and uniform: read an absence and audit it, refuse a taint. Also aligns the metadata batch classifier, which answered `unknown` where the single-file reader answers `unrecorded`, leaving two classifiers describing one policy differently. * test(provenance): assert the enforcement switch on a row it can act on The test that pins "closing the surface refuses an unrecorded file again" used a stored unknown, which the split had just turned into a taint. A taint is refused whatever the flag says, so the assertion held with enforcement off too and proved nothing about the switch this posture rests on. It now uses a recorded absence, and fails if the flag is ignored. * fix(files): copy a recorded absence as an absence The copy path required an exact source, so a stored unrecorded one fell in with the refusals and the target was written as a taint the source never carried. A workspace fork or a chat file copy therefore turned a readable file into a permanently refused one — the pathology this surface exists to undo, reached by copying — and nothing rewrites a file's provenance but another content write. Safe past the scope checks it now precedes: those exist to stop one workspace's secret entries landing in another's file, and a recorded absence has no entries to carry. The legacy Function export marker keeps writing unknown. Its record had resolved secret names in scope and an unreadable bundle, so whether that is an absence is not something this change can establish, and guessing in the readable direction is what caused this round.	2026-08-22 15:05:39 -07:00
Justin Blumencranz	3d40003482	improvement(tables): backfill default views for existing tables (#6899)	2026-08-21 19:25:58 -07:00
Vikhyath MondretiandClaude Opus 5	71129cd112	feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces (#6950) ... * feat(custom-blocks): let a publisher decide whether their block's runs reach consumer traces Joining a custom block's child run into its caller's trace shipped on by default, gated at read time by whether the person reading could already open the source workspace. That gate is doing the wrong job: a custom block's whole point is that consumers need no access to the source, so the check refuses exactly the readers the feature exists for, and it makes the answer depend on who is looking rather than on what the block's owner agreed to publish. The decision moves to the party whose data it is. `custom_block.trace_child_runs` is set by the publisher in Settings, applies org-wide, and is the entire policy — nothing downstream re-checks a caller. `getCustomBlockAuthority` already resolves per invocation and is the one lookup both the canvas handler and the Agent-tool runner pass through, so one column covers both surfaces and no consumer input can assert it. It defaults to FALSE. With the viewer check gone, an opted-in block publishes the source workflow's block names, inputs, outputs, and prompts to anyone who can read a consuming workflow's log. That is the same boundary curated outputs and redacted errors hold, so it opens by an affirmative act of the publisher or not at all — never as the residue of a column default on rows nobody revisited. Closed means the handle is withheld outright rather than persisted behind a flag: with no `childExecutionId` there is nothing for a reader, a migration, or a later refactor to join. What replaces it is a `_childTraceDisabled` marker, because a boundary span with no children renders exactly like a leaf block and an untraced run would otherwise read as one that did nothing. The consumer-facing failure `ref` is untouched either way — it is the only thing that makes an untraced failure reportable. Custom blocks invoked as Agent tools now join too. The child's handle already reached the agent's persisted `toolCalls[].result` (`postProcessToolOutput` strips only `__`-prefixed keys); nothing lifted it onto the tool span. Both span builders lift and strip it, and `hydrateChildTraces` needs no change — its boundary walk already recurses. The same handle is stripped from the model-facing copy of the tool result in `executeProviderTool`, the single point where the raw and model copies diverge: an opaque execution id in a tool result reads to a model like data the tool returned. The live SSE stream keeps one condition beyond the policy: an identified consumer. Not an authorization check — no workspace query — but chat deployments and the public API leave `liveTraceViewerUserId` unset because their consumer may be anonymous, and opting into org-wide tracing is not consent to stream a publisher's raw agent tokens to the internet. Copilot deliberately cannot set the field; exposing a team's internals org-wide is a human decision, not one an agent makes while publishing on their behalf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): read the publisher's trace policy at read time, not from the handle's presence Treating a persisted `childExecutionId` as proof of publisher consent is only true for handles this PR's writer produced. Every handle written before it meant something else — "a child ran; authorize the reader" — and the rows carrying them outlive the migration, so removing the reader check turned them into an open door: a consumer could open an old parent log and receive the source workflow's block names, inputs, outputs, and prompts from a block whose publisher never opted in. `hydrateChildTraces` now resolves the policy live, per boundary, from `custom_block.trace_child_runs`. The child log row's `workflowId` is the key — publish enforces one block per workflow — which also covers an Agent-tool boundary, whose span carries no block type to look up. A workflow with no block row (never published, or since deleted) has no publisher left to consent and stays shut, as does a failed policy read. This is not redundant with the write-time withholding. The handler still emits no handle for a block that was closed when the run executed, so such a run stays closed forever even if the block is opened later; this check decides whether the runs that DO carry a handle may still be shown. Turning the policy off therefore also closes what is already recorded, which is what a governance switch has to do to mean anything. Reported by Greptile on #6950. Also drops `any` from the trace-policy tests: outputs read through `Record<string, unknown>` (the handler's declared return does not name these internal keys) and failures narrow through `ChildWorkflowError.isChildWorkflowError`, which pins the failure type as well as its fields. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(logs): sum the child-trace drop counters from the struct, not a hand-listed set `totalDropped` re-listed four of the five counters, so a read whose only drops were policy refusals computed zero and skipped the log entirely. That is the commonest drop there is now — every handle written before the publisher policy existed refuses at that gate — so the one signal telling an operator the live check is closing joins went silent exactly when it started mattering. Summed from the struct instead. A hand-maintained list beside a struct is stale the moment a field is added, which is precisely how `policyClosed` was left out. Reported by Cursor Bugbot on #6950. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(db): renumber the custom-block trace migration around a 0299 collision Staging landed its own 0299 (`table_run_dispatches.heartbeat_at`) while this branch was open. The two migrations are independent — different tables, no shared statement — so only the number and drizzle's snapshot chain collided. Regenerated rather than hand-merged: a drizzle snapshot is a full-schema dump whose `prevId` links it to its parent, so editing one by hand to sit after a migration it was not generated against is how the chain silently stops matching the database. Staging's 0299 and its snapshot are taken verbatim; this is 0300, generated against them, and its SQL is byte-identical to what it replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-21 18:21:08 -07:00
Waleed	7b6c58113d	fix(knowledge,tables): recover abandoned dispatches, bound the sweep and the workbook preview (#6945) ... * fix(tables,knowledge): recover abandoned dispatches and bound the sweep Three defects measured in production this afternoon. A dispatcher killed by an OOM left `table_run_dispatches` at `dispatching` forever. Every terminal transition on that table is user- or flow-initiated, so nothing reclaimed the row: four dispatches were stranded in one afternoon, pinning each table's "X running" overlay and blocking re-runs, with no way to clear them from the product. The `table_run_dispatches_watchdog_idx` index has existed for this sweep since the table was created, unused. Liveness comes from a new `heartbeat_at`, stamped by the per-window writes that already advance `cursor` and `processed_count`, so a slow-but-live dispatch is spared however long it runs — the in-process path has no duration ceiling, so ageing from `requested_at` would reclaim live self-hosted work. The sweep reads `COALESCE(heartbeat_at, requested_at)` so rows written before the column stay reclaimable rather than NULL-false forever, and runs as the last arm of the existing stale-execution cron at the same 95-minute window its table-job sibling uses. Rows are cancelled, not completed: the scope never finished. The OOM itself is not a leak. Peak RSS is a flat plateau — 457 MB at 20-45s and 461 MB past 200s, so ten times the duration buys four megabytes — that has crept about two percent per release for a month, from 446 MB in late July to 545 MB, past the 512 MiB `small-1x` ceiling. CPU peaks at 0.19, so the larger preset is bought for RAM alone. `maxAttempts` never covered the kill either: Trigger.dev retries `TASK_PROCESS_OOM_KILLED` only when `retry.outOfMemory.machine` names a preset, and all four runs recorded `attempt_count = 1` while the docstring claimed they resumed from the persisted cursor. The connector stuck-document sweep dispatched without a bound. Its chunk size paced the loop but the candidate query had no limit, so one connector enqueued 2,959 documents in fifteen seconds onto the queue every workspace shares. Nothing was double-billed — those documents were genuinely unindexed — but one connector monopolized the queue, and each dispatch mints a fresh requestId, so the idempotency key differs every pass and none of it deduplicates. Candidates are now taken oldest-first and capped per sync; a deeper backlog is deferred to the next sync rather than dropped. * fix(file-parsers): read officeparser's entry point across module systems `officeparser` is CommonJS — `main: officeParser.js`, no `type`, no `exports` map — so what `await import('officeparser')` yields depends on who built the code. Node and webpack synthesize named exports from `module.exports`, so `.parseOfficeAsync` is there. esbuild, which builds the Trigger.dev worker bundle, puts `module.exports` on `.default` and leaves the named export undefined, and the package is in neither `build.external` nor `additionalPackages`, so it is bundled. Reading the named export directly therefore worked everywhere except the worker, where calling it threw `TypeError: parseOfficeAsync is not a function`. All four parsers treat that as "the library failed" and answer with a scrape of the archive, which returns `degraded: true`, and the document pipeline rejects a degraded parse outright. The visible result was every `.pptx` and legacy `.doc` reporting "No text could be extracted from this file — it may be scanned, image-only, or password-protected", naming a cause that had nothing to do with the fault. 118 pptx and 14 doc failures landed in a single burst when one connector's sync first succeeded after ten consecutive crashes. Resolved in one shared loader rather than per bundler: externalizing the package has to be repeated in every build config this code runs under and regresses silently the day one is missed. The shape handling is split into a pure `resolveParseOfficeAsync` because the failing shape cannot be reproduced by mocking the specifier — Vitest's module-namespace proxy throws on a missing export rather than yielding the `undefined` a real bundle produces, so a test going through `import` can only assert the shape that already worked. That is also why the existing parser suites never caught this: each mocks `officeparser` with a fabricated named export, which presupposes the interop being broken here. * fix(knowledge): bound the workbook preview to the rows it emits `sheet_to_json` allocates from a worksheet's DECLARED `!ref` range rather than its populated cells, and Excel routinely writes an inflated range from stray formatting. The 1,000-row preview cap was applied to the result, so it bounded the emitted string while the allocation it was meant to bound had already happened. An 880 KB workbook exhausted an 8 GB worker; the same content exhausted 16 GB when this ran inside the connector sync. No machine size fixes that, because the allocation scales with a number the file declares about itself — fleet p99 for this task is 691 MB against 8 GB, so this is a cliff, not pressure. Passing the window into the conversion is what makes the cap real. `defval` goes with it: defaulting every cell in the range made each row dense, so allocation scaled with columns x declared rows rather than with populated cells, and because no row was left empty it silently defeated the `blankrows: false` beside it. Reported totals still come from the declared range, so bounding the conversion does not change what the metadata says the workbook holds. The eleven documents killed this way recorded `attempt_count = 1`: `maxAttempts` does not cover `TASK_PROCESS_OOM_KILLED`, which Trigger.dev retries only when a larger preset is named. Adding that escalation is a safety net rather than the fix, and the same gap the dispatcher had. Also corrects the machine comment, which claimed `large-1x` was 2 vCPU / 2 GB. It is 4 vCPU / 8 GB, and believing the stale figure makes a resize look like the answer when the parser is what is unbounded. * fix(tables): keep a cancelled dispatch cancelled when a step claims it `dispatcherStep` reads the dispatch, then awaits the table load before writing `dispatching`. Keying that write on the id alone resurrected a dispatch cancelled inside that window — a Stop-all, or now the stale-dispatch sweep — and the fresh heartbeat it writes would then buy the resurrected row another full window before the sweep could reclaim it again. The race predates the sweep, but the sweep is a new writer of `cancelled` that no user action drives, so it is newly reachable without anyone touching Stop. Re-asserting the status the step already read is the whole fix. * fix(tables,knowledge): spare a live window, and restore the truncation notice A lease needs its heartbeat interval to sit well under its TTL. The dispatch heartbeat is stamped between windows, not during them, and `batchTriggerAndWait` checkpoints the loop for the whole window — so the interval is really "one window", which nothing bounds: the window ends when its cells do, and the in-process path has no ceiling at all. A window outliving the stale threshold had its dispatch cancelled while it was plainly alive. Its cells carry the signal the checkpointed parent cannot — `updatedAt` on every in-flight row execution, written by the cell tasks themselves. Both signals must be stale before a dispatch is reclaimed, so a slow window is spared for as long as its cells keep reporting while a run with nothing beating and nothing executing is still collected. The subquery rides the partial `(table_id, status)` index that already covers exactly those three statuses. Bounding the workbook conversion also made its truncation notice unreachable: the converted length can no longer exceed the window it was compared against, so every sheet larger than the preview cap silently stopped reporting that it had been cut. Compared against the declared row count instead, which is what the comparison meant before the conversion was bounded. * fix(tables,knowledge): act on the claim outcome and scope liveness to the dispatch Three defects, two of them created by the previous round's fixes. Guarding the pending-to-dispatching claim without reading its outcome was the worse half of a fix. When a Stop-all or the stale sweep won the race the row correctly stayed `cancelled`, while the step went on to announce `dispatching`, stamp cells and enqueue a window for it — and an empty window would then reach the unguarded `markDispatchComplete` and overwrite `cancelled` with `complete`. The step now ends when it did not claim the row. The cell-liveness probe was table-scoped, and `table_row_executions` carries no dispatch column, so a live dispatch's cells vouched for an abandoned dispatch beside it and the abandoned row was never reclaimed — turning the stuck overlay this sweep exists to clear into a permanent one. Narrowed to the dispatch's own groups, which it already stores. Two active dispatches over the same groups can still mask each other, but that is the state `markActiveDispatchesCancelled` already prevents. Truncation asks whether the window cut the sheet short — a question about the declared range against the cap. Comparing the converted length to the cap made it unreachable once the conversion was bounded; comparing the declared count to the converted length then reported truncation for any sheet merely containing blank rows, which are now skipped rather than defaulted into existence. * fix(tables): scope dispatch liveness to its rows, not just its groups The previous round narrowed the cell-liveness probe to the dispatch's groups on the reasoning that two active dispatches over the same groups cannot coexist, because starting a run cancels prior work on its scope. That reasoning was wrong. `cancelPriorRuns` in `workflow-columns` requires `isManualRun`, so auto-fired runs never cancel anything, and the per-row path is explicitly a no-op for dispatch cancellation. Same-group coexistence is ordinary. A dispatch that names rows now only accepts liveness from those rows, which covers the auto-fired and row-scoped runs that reach this state. What remains is two table-wide dispatches over the same groups, where nothing in the row execution says whose work it is; closing that needs a `dispatch_id` column on `table_row_executions` threaded through six write sites, including the shared cell-write path every cell task uses. That residue is a delay rather than a permanent mask — the live dispatch's cells stop updating when it finishes, and the next sweep after a quiet window reclaims the abandoned row. * refactor(tables): name the dispatch liveness predicate and bound its fan-out Extracts the cell-activity check into `hasRecentCellActivity`, so the stale predicate reads as its two conditions — nothing beating, nothing executing — rather than a twenty-line SQL blob nested inside an `and()`. No behaviour change; this is the code three review rounds found defects in, and being able to read it is what makes those defects findable. Bounds the terminal-event fan-out with `mapWithConcurrency`, matching how the scheduler already fans out. The sibling cancel paths emit over one table's dispatches; this sweep can carry a whole tick's worth across many tables, and each event is its own write. Also repairs the test that covers it. `collectChunks` walks into the `tableRowExecutions` table object the fragment interpolates, so every column name appears in the chunks whether the predicate references it or not — the group, row, table and timestamp assertions all passed with their predicates deleted. Matching the literal SQL instead makes them fail, which mutating each clause now confirms. * fix(tables): make the row bypass NULL-safe and guard the post-wait completion `jsonb_typeof(scope -> 'rowIds') <> 'array'` was the table-wide bypass, but a table-wide dispatch has no `rowIds`: the extraction is SQL NULL, `jsonb_typeof` returns NULL, and `NULL <> 'array'` is UNKNOWN rather than TRUE. The bypass never fired, so no live cell could satisfy the probe and the sweep reclaimed exactly the long-running table-wide dispatches the row filter was added to protect — inverting it. `IS DISTINCT FROM` is the NULL-safe form, and the same pitfall is already handled with `coalesce` in `markActiveDispatchesCancelled`. `completeDispatch` also wrote through the unguarded `markDispatchComplete`. Both its callers run AFTER the window's wait, so a Stop-all or the sweep landing during that wait leaves the row `cancelled` and the write overwrote it with `complete`, publishing a completion event after the cancellation one. The claim guard cannot cover this — the cancel arrives long after the claim. It now goes through `completeDispatchIfActive`, which already exists for exactly this, and emits nothing when the transition does not land. * fix(knowledge): give connector sync logs a retention pass Nothing pruned `knowledge_connector_sync_log`, so it grew by one row per sync run forever — a connector on a fifteen-minute interval writes about 35,000 rows a year by itself. That cost lands on `loadPreviousListingObservation`, which reads the newest `completed` row per connector through an index covering `connector_id` alone, so every retained row makes the sort behind the deletion-safety corroboration slower. Added as another arm of the cleanup cron, batched the same way as its two sibling prunes. Two `exists` guards are load-bearing rather than defensive: the newest row per connector always survives, and so does the newest `completed` one, because that is the row `loadPreviousListingObservation` reconstructs the previous listing from — and that reconstruction decides whether a suspect listing is corroborated, i.e. whether reconciliation may delete documents. Pruning it would silently change deletion behaviour. `started` rows are never eligible; they are in flight or waiting on the scheduler's own sweep. * fix(tables): funnel every post-claim completion through the guarded write The empty-window exit still wrote through the unguarded `markDispatchComplete`, and it runs after the claim like the other two — so a cancel landing during its window query was overwritten with `complete`. Shorter window than the two post-wait exits, same defect, and leaving one of three unguarded is how this came back twice already. All three now route through `completeDispatch`, so the guard lives in one place and covering it once covers every exit. The redundant test for this path went with it: it could not be made to fail against the mock, and a test that cannot fail is worse than none — the guard is held by the test on the shared funnel. * fix(tables): bound how long cell activity may spare a dispatch The liveness probe cannot tell whose cells it is looking at when two table-wide dispatches share a group, because `table_row_executions` carries no dispatch column. On a quiet table that is only a delay — the neighbour finishes and the next sweep reclaims — but a busy table with continuous auto-fired work can keep an abandoned dispatch masked indefinitely, which is the stuck overlay this sweep exists to clear. A ceiling bounds it: past a day without a heartbeat, a dispatch is reclaimed whatever its cells are doing. That is safe because a live dispatch stamps its heartbeat between windows regardless of cell activity, so only a single window outliving the ceiling could be reclaimed wrongly, and no window lasts a day on any path — the Trigger.dev run ceiling is ninety minutes. The real fix is a `dispatch_id` on the executions row. Threading it through the patch layer and the upserts underneath it is a change to the hottest write path in tables and belongs in its own review, not on the sixth round of this one. * refactor(tables): give the stale predicate one definition of "last beat" `COALESCE(heartbeat_at, requested_at)` was written twice — once for the stale threshold and again for the absolute ceiling — so the two could drift into disagreeing about what proof of life means. One `lastBeat` fragment, one `notBeatingSince(cutoff)` helper, both cutoffs expressed through it. Also corrects the ceiling's comment: it triggers a day past the stale threshold, not a day past now. * fix(tables): delete the unguarded completion rather than guard it a fourth time The two pre-claim exits — table missing, no target groups — still wrote through `markDispatchComplete`. Last round I argued they run before the claim, "where forcing a terminal state is the intent". That was wrong twice over: the table lookup is awaited, so a cancel lands in that window like any other, and a dispatch cancelled mid-lookup has not completed its scope any more than one cancelled mid-window has. Routing them through `completeDispatchIfActive` left `markDispatchComplete` with no callers, so it is gone. That is the part worth having: this is the fourth place the same defect appeared, each time because an unguarded writer was sitting there to be reached. With it deleted, `completeDispatchIfActive` is the only way to complete a dispatch and the class cannot recur. * fix(tables): re-read the dispatch before committing a window Several round trips separate the claim from the enqueue — the window query, the executions prefetch, the tombstone filter — and nothing rechecked the dispatch across them. A Stop-all or the stale sweep landing in that gap had the step stamp cells and run a whole window for a dispatch already recorded as cancelled; the existing recheck sits after the window, which is too late to prevent it. Mirrors that existing check on the other side of the enqueue. It narrows the gap to a single statement rather than closing it — a cancel arriving after this read still races the enqueue, and no check can fix that. The cell-level `cancellationGuard` and the `isExecCancelledAfter` tombstone filter are what catch the remainder.	2026-08-21 15:02:12 -07:00
Waleed	63569a2459	fix(connectors): count hard-kill failures and cap deletion blast radius (#6909)	2026-08-21 09:41:21 -07:00
Waleed	1ced0b6065	fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents (#6921) ... * fix(knowledge): stop the stuck-document sweep reclaiming still-queued documents The sweep gave 'processing' a staleness cutoff but gave 'pending' and 'failed' none, and the only cross-run guard was uploadedAt < syncStartedAt, which scopes within a sync but not across them. That was harmless while document processing awaited inline: documents were terminal by the time a sync ended. Since dispatch became asynchronous they sit 'pending' until a worker picks them up, so the next sync deleted their embeddings, reset them, and re-dispatched while the original runs were still executing — and each re-dispatch mints a fresh pass id, so it billed again. Queued documents now get a grace period derived from queue drain time rather than run duration: the processing queue's concurrency is 20 and global across workspaces, so a 2,600-document corpus takes roughly two hours to drain. The existing 45-minute threshold bounds a run, not a wait, and would still reclaim live documents on any corpus over about 900. No column records dispatch time, so the signal is processingStartedAt falling back to uploadedAt. The sweep now stamps that column when it re-dispatches, and the user-triggered retry stamps it instead of clearing it — without both, only a document's first dispatch was protected and the sweep churned once per sync forever on anything that kept waiting. The grace narrows the duplicate-billing window but cannot close it: a re-dispatch mints a new request id and the Trigger idempotency key is scoped per dispatch by design. Closing it durably needs a document-scoped key or a dispatch-generation column, recorded in TSDoc and deliberately not built here. * fix(knowledge): record dispatch time in its own column instead of overloading the start time Two review findings on this PR traced to the same root: the queue grace was reading processingStartedAt, a column that means something else. Externally, a pending row carrying a dispatch timestamp reported a processing start time for work that had not started. Internally, updateDocument sets a document pending and refreshes uploadedAt while leaving the previous run's processingStartedAt in place, and completion never clears it — so the sweep aged a re-dispatched document from a leftover stamp rather than its actual dispatch, and could reclaim it inside the grace window while the earlier queue entry was still live. That reopens the duplicate-billing race this PR closes. processing_queued_at is written on every re-dispatch and read by the sweep; processingStartedAt goes back to meaning what its name says, so its external contract is byte-identical to before this PR. Not stamped on first dispatch: the row is created and dispatched inside the same sync run, so uploadedAt is already accurate there and a guarded write per upload would buy no behavior. The internal document route returns a full table row through a passthrough schema, so the new column would have shipped as an undeclared raw Date. Declared and serialized explicitly instead. * fix(knowledge): stamp the queue time in the dispatch funnel, not at each call site Giving dispatch its own column was not enough. The connector content-update path updates a row in place with pending status and a fresh uploadedAt while leaving every prior-run processing column intact, so a document dispatched once already carried a stale queue stamp, the refreshed uploadedAt was never consulted, and the row aged from a dead run's timestamp — the same bug one level down. "Written on dispatch and only on dispatch" is now structural rather than a convention three call sites remember: the stamp lives in processDocumentsWithQueue, which every one of the nine dispatch paths already goes through, so none can forget it. The sweep's and the retry's own stamps are kept because they land inside the same transaction as their reset, so a document keeps its grace even if the dispatch that follows throws. The funnel also clears processingStartedAt, guarded on the row still being pending so it cannot disturb a worker's compare-and-set. That closes the API-facing half at its source: a pending row can no longer report a start time from any path, including the content-update path no serializer would have seen.	2026-08-20 18:59:31 -07:00
3 people	42f6287911	feat(byok): add organization-wide key inheritance (#6834) ... * feat(byok): add organization key management * feat(byok): inherit organization keys at runtime * feat(byok): add organization scope to BYOK settings * fix(byok): refresh org key state after mutations * fix(byok): hide stale inherited status badges * chore(db): drop colliding byok migration ahead of staging merge Staging independently claimed 0293. Remove ours so the merge is clean; it is regenerated at the next free index right after. * chore(db): regenerate byok migration at 0296 Staging claimed 0293-0295 during the merge; the regenerated SQL is byte-identical to the dropped 0293. * docs(byok): document organization scope, precedence, and the full provider list The BYOK section described workspace-scoped keys only. Add the organization scope, its Enterprise requirement, the per-provider precedence rule, what an entitlement lapse does, and the Pi sandbox exposure. Refresh the provider table from the settings page, which had drifted from 14 to 34 entries. * feat(byok): open organization keys to every organization plan Organization BYOK was gated on Enterprise, but an organization is the only thing that can hold the keys, so every plan that can own an organization should qualify — Pro for Teams, Max for Teams, and Enterprise. Add checkOrgPlan/resolveOrganizationPlan beside the Enterprise pair rather than widening checkEnterprisePlan, so the Enterprise-only gates (Access Control, whitelabeling) are untouched, and restore resolveOrganizationEnterprisePlan to module-private now that BYOK no longer needs it. * perf(byok): cache the organization entitlement, not the key material getBYOKKey runs once per agent block and once per hosted-capable tool call, so a loop over N items resolved N times — and each organization-inheriting resolution paid three sequential billing queries on top of the two key reads. Split the two reads by staleness tolerance. Key rows stay fresh, because revocation must be immediate. The entitlement is a billing gate that tolerates bounded staleness in the harmless direction (a lapsed organization keeps using its own key for <=60s), so cache it per organization with an in-flight share so concurrent blocks issue one query set. The management surfaces keep reading it fresh, so an organization that just upgraded is never told otherwise. Also run the block check and subscription read in parallel inside resolveOrganizationPlan, and carry the resolved scope on BYOKKeyResult so a log line can say whether a run used the workspace's key or an inherited one. * feat(byok): let workspaces store the Z.ai and Cohere keys the runtime reads Both ids were already in the BYOK contract enum and both are resolved at execution time — getApiKeyWithBYOK reaches 'zai' (GLM models are in the hosted catalog, so the BYOK branch runs), and 'cohere' backs both the Embeddings block and Knowledge Base reranking — but neither appeared in the settings list, so there was no way to store the key either path looks for. Cohere had no icon; add one from the official multi-color mark so it stays legible on a light and a dark page. Cohere's embed-v4.0 is kbEligible:false, so the description says 'Embeddings and Knowledge Base reranking' rather than claiming KB embeddings. * improvement(byok): shorten the workspace scope chip to 'Workspace' It sits beside 'Organization', so the scope reads from the pair; 'This' only added width. * fix(byok): do not cache a billing outage as an unentitled organization resolveOrganizationPlan maps a failed billing read to false, which is indistinguishable from a real plan lapse. The entitlement cache stored that, so one transient outage held the gate shut for the full TTL and every inheriting run silently fell back to a metered hosted key — and the cache's rejection path, which exists to prevent exactly this, was unreachable. Give the resolver the onError option its neighbours already have and let the cached read ask for 'throw', so a failure stays out of the cache and the next resolution retries. Behavior for the call that saw the error is unchanged: getBYOKKey still fails closed. Reported by Cursor Bugbot. * fix(byok): propagate the subscription read's failure too The previous commit threaded onError through resolveOrganizationPlan's own catch, but getOrganizationSubscriptionUsable soft-fails to null on its own, so a failed subscription read still arrived as an ordinary 'no usable subscription' and returned a successful false — which the entitlement cache then stored for the full TTL. Thread the option into that call as well. Test it at the billing layer rather than the cache layer: the entitlement test mocks resolveOrganizationPlan wholesale, so it could never have caught this. Verified the new test fails against the previous commit. Reported by Cursor Bugbot. * refactor(byok): cache the entitlement with LRUCache, like copilot entitlements The hand-rolled version reinvented three things the codebase already has a canonical answer for. lru-cache is a declared dependency of apps/sim and lib/copilot/entitlements.ts already caches an entitlement with it — by storing the in-flight Promise, which is what makes concurrent callers collapse onto one resolution with no in-flight bookkeeping at all. TTL and the size bound come from the library. That removes the second Map, the manual eviction (and its interaction with an in-flight entry), and the dead value-while-refreshing state: 23 executable lines. The one thing the library does not cover is dropping a rejected promise so a billing outage is not cached for the TTL, which is kept and pinned by a test that fails without it. TTL expiry is no longer re-tested — that is the library's behavior, not ours, and lru-cache reads its clock at module load so faking timers never moved it. * refactor(byok): coalesce the entitlement read with the shared singleflight lib/concurrency/singleflight.ts is the codebase's coalescing primitive and oauth/credential-service.ts already pairs it with a read-through cache. Adopting that shape fixes a case caching the promise directly did not: a *hung* billing read wedged every caller for the full 60s TTL, where coalesceLocally evicts and rejects at its settle deadline. It also removes the hand-rolled rejection eviction — the cache is written only on the success path, so an outage leaves no entry by construction. The cache now holds booleans, which introduces the one trap worth a test: a truthiness check would read a cached false as a miss and re-query billing on every resolution for lapsed organizations. Pinned. * fix(byok): keep an abandoned entitlement producer from writing the cache coalesceLocally does not cancel a producer it timed out — its docstring says so explicitly — so writing the cache from inside the producer let a late billing result overwrite a fresher answer a retry had already cached, and hold it for a full TTL. Move the write onto the value the caller actually received. A caller that timed out throws before reaching it, so an abandoned producer now resolves into nothing. The test reproduces the overwrite and fails against the previous shape. Reported by Cursor Bugbot. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-20 17:45:28 -07:00
3 people	3b4d9e9149	fix(schedule): reconcile interrupted schedule executions (#6780) ... * fix schedule execution recovery * simplify schedule recovery provider handling * fix(schedules): stop reconciliation rewriting healthy carriers and index its scan Follow-up hardening on the schedule recovery reconciliation. Reconciliation settled every carrier it examined, including ones that had already reached a terminal status and recorded their own outcome. Because the normal completion path never stamps the reconciled marker, each successful run's carrier was picked up on the next tick and rewritten: `completedAt` bumped, `error` nulled and `output` replaced with a recovery stub, all of which `GET /api/jobs/[jobId]` surfaces. A completed carrier whose execution log had aged out was additionally flipped to failed. Settle only carriers still in flight; a terminal one is owed schedule accounting and the marker, nothing more. The recovery scan matched no index. There was none on `async_jobs.updated_at`, and the unreconciled-terminal branch tested a jsonb extraction, so the whole OR fell back to a sequential scan and sort of `async_jobs` on every tick. Add the partial index, and spell the branch's status list and metadata key as SQL literals: Postgres cannot prove a parameterised predicate implies a literal index predicate, so a bound key would have left the new index unused. Irrecoverable carrier tombstones were exempted from retention with no secondary expiry, so the one class of row that can never reconcile grew without bound. Give them a longer bounded window instead. `WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN` had become a per-status budget when the stale-execution sweep gained its `redacting` pass, silently doubling the per-run cap. Share the budget across both passes, and add the matching partial indexes so the second pass keeps an index rather than seq-scanning. Also consolidates the carrier metadata keys, their predicates and the jsonb merge into one module -- the two routes were the writer and reader of the same keys with no shared symbol, so a rename type-checked clean while silently breaking retention. * fix(schedules): rotate deferred carriers and stop the redacting sweep failing live runs Addresses the review findings on the reconciliation hardening. A carrier whose accounting was deferred got no write at all, so it kept its `updatedAt` and stayed at the head of the `updatedAt`-ordered recovery batch on every tick, starving every other claimed carrier and never becoming eligible for retention. This was a regression from the previous commit: settling used to bump the timestamp for every examined row, and skipping the settle for already-terminal carriers removed the bump with it. A payload with no `scheduledFor` can never reconcile, so such a row pinned a batch slot permanently. Bump `updatedAt` unconditionally and keep only the reconciled marker conditional. The stale-execution sweep terminalized `redacting` logs on the execution deadline. That deadline bounds execution, while `redacting` covers payload masking after the run already finished -- so a run that used most of its budget entered redaction with the deadline due, and the sweep failed it five minutes later while the worker was still masking. Schedule recovery then read the log as a failed occurrence and counted a failure that never happened, even though the worker's terminal write later restored `completed`. Sweep `redacting` on the generic stale window only. `getScheduleNextRunAt` falls back to a daily cadence when a schedule has no cron expression. Deployment cannot persist such a schedule, so the branch is unreachable, but this change widened its use from failure recovery to every outcome -- log a warning when it fires rather than silently guessing a cadence. `executionDeadlineAt` was missing from the shared `workflowExecutionLogs` schema mock, so it read as `undefined` and assertions comparing against that column were trivially true. Add it. * refactor(schedules): drop vestigial recovery code and close a builder gap Follow-ups from a full re-read of the change. No behavior change except the removed dead code paths. The metadata merge stripped a `scheduleRecoveryBlocked` key on every write. That key has never been written by any shipped code -- it appears nowhere in staging and nowhere in history outside this branch -- so the strip guarded against a state that cannot exist, at the cost of an extra jsonb operation and a bind parameter on every reconciliation write. `processScheduleItem` set `carrierObservedOrLookupUncertain` immediately before returning on an ambiguous enqueue. The flag is only read from the surrounding catch block, which a normal return skips, so the assignment was dead and read as though it were load-bearing. Replaced with a comment stating why the occurrence is preserved. The stale-execution sweep carried two near-identical `jsonb_set` templates that differed only in their error expression, kept flat because the test mock renders nested SQL fragments as placeholders. The suite now has a recursive renderer, so the error expression is a named per-status value and there is one `jsonb_set`. Success was the only schedule outcome without a named update builder, which left `executeScheduleJob` using two idioms for the same guarded write and left the update shape untested. Add `buildScheduleSuccessUpdate` beside its cancellation and failure siblings, use it from both call sites, and cover it the way `buildScheduleCancellationUpdate` is covered -- a mutation of its `failedCount` reset previously passed every suite. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-20 11:55:45 -07:00
Vikhyath Mondreti	483ff12b6f	feat(billing): align enterprise reporting periods (#6851) ... * feat(billing): align enterprise reporting periods * fix(billing): harden enterprise reporting flow * fix(outbox): preserve handler compatibility * fix(billing): bound enterprise provisioning reads	2026-08-19 15:39:18 -07:00
Vikhyath MondretiandClaude Opus 5	9ee1b81b41	feat(custom-blocks): join cross-workspace runs into the caller's trace, and map blocks per environment (#6857) ... * feat(custom-blocks): join cross-workspace runs into the caller's trace, and map blocks per environment Teams that orchestrate work across workspaces have two gaps that keep them on HTTP blocks instead of custom blocks: they cannot see what a custom block actually did, and a forked environment silently keeps calling the environment it was forked from. Debugging. A custom block is an invocation boundary — a published block is org-wide, so its internals must not reach every consumer by default. The child already writes its own log row in the source workspace, correlated to the invoking run; the trace existed, it just was not joined. The parent's span now carries only the child's opaque execution id, and `hydrateChildTraces` joins the child's spans at READ time, after authorizing the person reading against the child's workspace. Authorization follows the viewer rather than a flag set at publish time, re-evaluates on every read, and needs no second copy of the spans. Each hop of a nested chain is authorized against its own workspace. Boundaries left unexpanded — no access, no data, past a cap — say so, because a childless boundary span otherwise renders exactly like a leaf and a partial trace reads as a complete one. Live runs stream too, gated on `liveTraceViewerUserId`, which only surfaces with a single known authenticated viewer set. Chat deployments stream through the same callbacks and their consumer may be anonymous, so anything that does not opt in keeps the boundary shut. Child spans handed to such a viewer are projected through the CHILD's session: the invoking run's registry knows nothing about the publisher's secrets, so projecting there would leave a source-owner credential unmasked. They reach the live stream and stop — `createSpanFromLog` still refuses to persist them, which is what keeps read-time hydration the single authorization point. Environments. A fork inherits its parent's organization and `custom_block` is keyed `(organization_id, type)`, so a uat fork resolved to the same row and ran the prod workflow. Custom blocks become a fork-mappable resource, keyed by BLOCK TYPE — the rule every kind follows: key by whatever the workflow references, as `file` does with storage keys and `env-var` with names. A custom block is the only resource referenced by the canvas block's own type rather than a sub-block value, so the rewrite gets its own channel. Unmapped blocks keep the source type, because a type cannot be emptied without deleting the node; they surface as unmapped and block the promote, which is what stops uat from quietly invoking prod. Same-named environment copies now carry their source workspace, so an Access Control allowlist decision between three identical "Invoice Parser" rows is no longer a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(workspace-forking): let an explicit identity custom-block mapping resolve `remapForkBlockType` reported a mapping whose target equalled the source as unresolved, conflating "a mapping exists" with "the type string changed". Those are opposite states that produce an identical output `type`, and every caller uses the flag for the former — to decide whether the reference blocks a promote. The org-wide candidate list includes the source block, so binding an environment to the shared block is a normal pick. Under the old flag it raised `unmapped-custom-block` and refused the sync over a choice the user had explicitly made. The flag is now named `resolved` and reports mapping existence; whether the type moved is already visible from `type`. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): keep a streamed child's spans and markers off the parent's log Two leaks in the live-stream path, both the same mistake: treating a channel as viewer-scoped when it is actually persisted, so gating the stream on an authorized viewer bought nothing. `childTraceSpans` rode the block output to reach the stream. `filterOutputForLog` only dropped a hidden key when the block's own config declared it `hiddenFromDisplay` — true of the workflow block, never of a custom block, whose outputs are publisher-curated. The source run's spans therefore persisted into the parent's `span.output`, readable by anyone with parent-workspace access and never re-checked by `hydrateChildTraces`. A globally hidden key is now dropped at the top level, not only when nested, and `extractDisplayOutput` strips it again so no other producer can reintroduce it. The fan-out also called the invoking run's `onBlockStart`/`onBlockComplete`, which are persist-then-emit composites: they write block names and I/O into the parent's LoggingSession before reaching the stream. Those markers are keyed by the parent execution and outlive the per-viewer check entirely. Custom-block children now go through `liveStreamCallbacks`, the raw emit-only pair, and fail closed when a surface supplies none. Same-workspace workflow children keep the composites — they belong to the same run and their markers are legitimately the parent's. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(custom-blocks): carry the emit-only stream sink into nested executions Routing custom-block events through `liveStreamCallbacks` forwarded the viewer id to the child but not the sink itself, so a nested hop cleared `canStreamCustomBlockToViewer` off the inherited id and then had nothing to stream through — `parentStreamSink` fell back to `{}` and live traces stopped at the first sub-executor. That hit a custom block nested inside a workflow block as readily as one inside another custom block. The sink now travels with the viewer id, and both are withheld together when streaming is not permitted. It is always the INHERITED chain, never `parentStreamSink`: for a same-workspace workflow block that is the persisting composite, so forwarding it would put a custom block nested inside one straight back onto the parent's progress markers — the leak the previous commit closed. Reported by Cursor Bugbot on #6857. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-19 15:25:28 -07:00
Vikhyath MondretiandClaude Opus 5	521348b529	feat(secrets): record which secrets each run resolves, and surface it per secret (#6823) ... * feat(secrets): record which secrets each run resolves and surface it per secret Redaction stops a value at a boundary but cannot stop code that never emits it — a Function block can print a key one character at a time and nothing ever matches the secret. That is undecidable in general, so this adds the other half of the posture: attribution. Every run now records which configured secrets it actually resolved, under whose identity, through which surface (workflow, Sim agent, MCP). The data already existed in ResolvedSecretTraceRegistry.addActiveEntry and was persisted only for paused runs; this persists it for every terminal path. Execution logs cannot answer this. They store the whole available encrypted environment rather than what a run referenced, they evidence a secret only where value-matching redaction happened to fire, and they expire under logRetentionHours — while "who has touched this key" outlives any single run. - secret_usage: per-UTC-day rollup keyed by workspace, secret, scope, owner, source, workflow, actor. A one-minute schedule touching three secrets would otherwise write thousands of rows a day, which is also why this is not audit_log. workflow_id/actor_user_id use '' sentinels rather than null so the unique key works on Postgres 14 without NULLS NOT DISTINCT, and are not FKs: they are historical facts, and an onDelete would rewrite a key column. - secret_owner_user_id is part of the key. Two people can hold a personal secret under one name and a shared personal secret resolves for a caller who does not own it, so name and scope alone do not identify a secret. It is NOT the actor: a scheduled run resolves the workflow owner's personal slice under the workspace's execution actor. - Direct environment reads are now detected in JS (TypeScript AST), Python (tokenizer-checked) and shell (quote/heredoc-scanned), so a secret read as environmentVariables['K'] or $K enters the run's provenance instead of going unredacted. Each detector prescans for names that are actually configured secrets before paying for a lex or quote-frame pass. - Copilot integration tool calls are covered: resolveCopilotEnvReferences substitutes {{SECRET}} into user-only params, which is a real use. - See usage lives behind a credential-admin gate, using the same predicate that reveals the value; members get a disabled chip explaining why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(audit): register the secret-usage route in the validation baseline Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): keep rollup metadata with its run, and stop shadowed bindings faking usage Review round 1. - record.ts: last_execution_id/last_trigger were assigned unconditionally while last_used_at was chosen by greatest(), so two runs completing out of order split one row between them — the newer run's timestamp beside the older run's execution id, making "View log" open a run the row does not describe. Both are now guarded on the timestamp actually advancing, so the row's metadata always belongs to the run that owns its timestamp. - javascript.ts: a local binding named environmentVariables (declaration, parameter, destructured binding, or bare reassignment) made reads off the user's own object look like mounted-secret reads. Any such binding now disables detection for the file; the AST already had parent pointers, so this is a kind check during the existing walk. - python.ts: same class of bug with no parser available, so the rule is an allowlist — every mention of the binding must be a literal subscript or .get(), otherwise detection is off for the file. This also subsumes the cross-line attribute case (other.\n environmentVariables['K']), which the previous space-and-tab look-behind missed. Under-reporting is the safe direction here: a trail that claims a use that never happened is worse than one that misses a use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(db): format the generated migration snapshot CI runs lint:check across every workspace; the drizzle-kit output in packages/db had never been through biome, so the branch was green locally (where lint had only been run inside apps/sim) and red on CI. Whitespace only — both files are byte-for-byte identical once parsed, and drizzle-kit still reports no pending schema diff against the reformatted snapshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): detect every rebinding of the environment identifier, not just declarations Review round 2. A bare `for (environmentVariables of rows)` has no declaration to key off, so the previous check missed it and reads of the loop value were still recorded as secret usage. Rather than extend the hand-rolled node-kind list, this reuses the pair the same file already applies to reject a placeholder in a write position: isDeclarationIdentifier covers declarations, parameters, destructured bindings and imports, and isWriteIdentifier covers every assignment operator, ++/--, destructuring targets, and for-in / for-of initializers. That also closes four forms neither the review nor the original check named: logical (||=) and nullish (??=) assignment, and object and array destructuring assignment. Six of the eight added cases fail against the previous check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): apply the rebinding rule to shell, and say when a run's log is gone Review round 3, plus the docs that were left claiming the old behavior. - shell.ts: a script that writes a configured name (API_KEY=local, export/local/ readonly, read, for, unset) expands its own value from that point on, not the mounted secret, so recording it claimed a use that never happened. Every mention of the name must now be a `$NAME` / `${NAME}` expansion, matching the allowlist shape the Python detector already uses. Applied per name rather than per file: JavaScript and Python shadow one object holding every secret, whereas rebinding one shell variable says nothing about the rest. - The usage trail deliberately outlives execution logs, so a row routinely names a run whose log has been pruned. The read now left-joins workflow_execution_logs on its unique execution_id and reports availability, and the panel renders the chip disabled with the platform tooltip instead of linking into an empty Logs view. Three states: no run to link, a run whose log is gone, and a live link. - Docs said a direct environmentVariables/$KEY read does not activate masking, which this branch changes. Corrected in credentials.mdx, function.mdx and the logging FAQ, and the recognition limits are now written down: runtime-built names, reassigned bindings, and reads that cannot be told apart from text. Added a "See usage" section covering who can see it and why an empty trail means "nothing recognized" rather than "never used". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): writing a name is not reading it, and a bare mention is not a rebinding Review round 4. - javascript.ts / python.ts: `environmentVariables.API_KEY = 'x'` and `delete environmentVariables.API_KEY` touch the name without ever reading the mounted value, but the detectors matched the member access and recorded a use that never happened. JavaScript now asks the same isWriteIdentifier the placeholder rewriter uses (its parameter is widened to ts.Node — the body already walked generic nodes, so this is a type change, not a behaviour one) plus a delete check; Python excludes a subscript followed by `=` and a `del` target. - shell.ts: requiring every mention of a name to be an expansion also fired on text that binds nothing — a comment naming the key, or `echo "API_KEY=$API_KEY"` where the literal is an argument rather than an assignment — and dropping those cost masking on a genuine read. It now looks for actual writes: an assignment at command-word position, a binding builtin, `printf -v`, or a `for` target. The two directions are not symmetric, which is why this errs toward detecting the read: missing a write records a use of a secret the script only had in its environment, a misleading audit row and nothing more, since masking still searches for the real value and will not find it. Over-detecting a write suppresses masking on a value that does reach the log. This also makes the code match what the docs already described — skipping after a rebinding, not after any mention. 13 tests added; 11 fail against the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): an update reads before it stores, and a del target may be parenthesized Review round 5. The first of these is a regression from round 4. - javascript.ts: reusing isWriteIdentifier to answer "is this a read" was wrong. That predicate answers the rewriter's question — is this a target the substitution must refuse — so it treats every assignment operator alike, which is correct there and wrong here: `+=`, `||=`, `??=`, `++` and `--` all load the current value before storing, so they are genuine reads and were silently losing their masking. Only a plain `=` stores without reading. Replaced with a purpose-named predicate, and isWriteIdentifier's parameter is narrowed back to ts.Identifier now that nothing else needs it widened. A test committed last round asserted the wrong behaviour for `+=`; it has been corrected rather than left to pin the bug. - python.ts: `del (environmentVariables['K'])` slipped past a check that looked only at the characters immediately before the match. It now isolates the enclosing logical line and tests whether that is a del statement, which also covers `del((x))`, `del(x)`, `del a, x`, and a del after a semicolon. 12 tests added or corrected; 10 fail against the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): stop excluding Python writes, which kept leaking in the unsafe direction Review round 6. Greptile found that `del environmentVariables[environmentVariables['K']]` had its inner access — which computes a key, so it is a genuine read — skipped along with the delete, leaving that value unmasked. The narrow fix was another textual rule. Instead this removes the write and delete exclusions from the Python detector entirely, because they were optimizing the wrong direction. `resolvedSecretNames` feeds `outputSecretMatcher`, an exact-value matcher over the output. Naming a secret the code never read costs nothing there: the matcher scans for a value that does not appear. Failing to name one that was read leaves it unmasked. The two error directions are therefore not comparable, and the exclusions bought only audit-trail tidiness while every heuristic they needed has so far leaked into the dangerous side — first a parenthesized target, now a nested read. A `del` or an assignment is reported like any other access. JavaScript keeps its exclusion: a real AST answers the question per node, with no text to misread, and it has produced no such hole. Net 30 lines removed from python.ts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): report recognized reads instead of proving they are not reads Review round 7. Greptile flagged both directions at once — false usage from reporting a write target, and unmasked secrets from the file-wide shadow flag — so I traced what the signal actually drives before choosing. The chain: the compiler's names feed outputSecretPlaintextsByName and the exact-value matcher, NOT context.resolvedSecretNames, which starts empty. After execution activateOutputSecretProvenance scans the output and adds only names whose plaintext actually appeared; those become __resolvedSecretNames, which tools/index.ts turns into recordResolved calls, which is what the usage trail reads. So a compile-time false positive produces no usage row on the ordinary path — it only hands the matcher a value the code never emits. It does produce one on the !projection.safe fallback, where the system already over-approximates by design. A false negative, by contrast, keeps the value out of the matcher entirely, so a genuinely read secret is never masked on any path. That asymmetry decides it, so every "prove this is not a read" mechanism is gone: - javascript.ts: the file-wide shadow flag. A helper declaring its own environmentVariables discarded genuine reads of the mounted binding everywhere else in the file — Greptile's security finding, and real. - python.ts: the allowlist requiring every mention to be a subscript or .get(). Same hole: passing the dict to a function suppressed unrelated reads. - shell.ts: the rebinding check. It had the same hole in a form nobody flagged — `echo "$API_KEY"; API_KEY=local` dropped the first read, which is of the real secret. What stays is the question of whether the text is code at all — strings, comments, single quotes, quoted heredocs — plus the receiver check that `other.environment Variables['K']` is a different object, and JavaScript's node-precise write/delete exclusion, which cannot suppress a read elsewhere. Net 215 lines removed across the three detectors and their tests. Docs updated: the rule is now stated as reporting rather than proving, and that See usage may occasionally list a secret the code had available but did not read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(secrets): drop the last write-vs-read special case `environmentVariables` is a plain object deserialized from the run payload (route.ts:206), not a handle on the stored secret. Assigning to it changes nothing outside the sandbox and is discarded when the run ends, so separating a write from a read bought almost nothing while leaving JavaScript as the one language still trying to prove a read is not a read. Every language now follows the same rule: report a recognized read of a configured secret name. The only exclusions left are facts rather than inferences — the text is not executable (string, comment, single quote, quoted heredoc), the receiver is a different object, or the name is not statically knowable. Docs note that assigning to the binding does not edit the secret. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(secrets): ship only the fields the trail actually shows Five fields crossed the API and reached no reader: usageDate, firstUsedAt, actorEmail, workflowId and actorUserId. The panel renders the timestamp, the trigger, what used the secret, the actor's name, the run count and the run link; everything else was projected, serialized and discarded. first_used_at is dropped from the table as well. Nothing read it, and inside a per-day bucket "first used that day" says nothing next to "last used that day" — so it was a column written on every run for no question anyone asks. The upsert loses its least() with it. Migration regenerated; the identifier columns behind the joins stay, they simply are not returned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): report referenced code secrets, not only ones that surface in output The Function route activated a secret's provenance — and therefore its usage row and downstream masking — only when the exact value appeared in the result, stdout, or error. That gate made the trail miss silent use entirely: a key that authenticates an API call and is never echoed reported nothing, and so did the founding scenario of this feature, a key exfiltrated character by character. The innocent run that echoed a key got a row; the run worth catching did not. Activation now follows the referenced set the compiler already computes: resolved {{KEY}} bindings plus recognized direct reads, filtered to configured values — the same set the unsafe-projection fallback already activated. An extra name only hands the output matcher a value that never appears; configured-but-unreferenced values are still never included. The output-scan activation path and its surface helper are deleted rather than kept alongside. One old test pinned the gate ("does not activate a referenced secret that does not cross the Function result"); it now asserts the reverse, with the reasoning attached. Two new tests pin the char-split exfiltration and the silent API-call case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): shell escaping is backslash parity, not adjacency Review round 8. `\\$API_KEY` is an escaped backslash followed by a LIVE expansion — bash prints `\` plus the value — while `\$API_KEY` is an escaped dollar and stays literal. Checking only the character adjacent to `$` read every even run as escaped, dropping a real read from usage and masking alike; verified against bash before fixing. The scanner now counts the run of backslashes before the `$` and skips only odd runs, the same parity rule logicalLineEndAfterContinuations in this file already applies to line continuations. Six-case parity table added; the three even-run cases fail against the previous check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): recognize destructured environment reads Review round 9. `const { API_KEY } = environmentVariables` delivers the value by name with no property- or element-access node in the AST, so the member-access walk missed it entirely — and a missed read leaves an emitted value unmasked, the dangerous direction. The AST walk now also recognizes the declaration form (shorthand, renames, defaults, string-literal keys), the assignment form ({ KEY } = env), and a ...rest element — which names no key but takes every value, so it reports every configured name; the alternative left `const { ...all } = env; return all` entirely unmasked. A computed key stays unrecognized, the same runtime-name boundary as a computed subscript, and a receiver that is not the bare identifier is not attributed. Nine cases added; the six positive ones fail against the previous walk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): one receiver rule for destructured reads, parentheses included Review round 10. Two accurate findings, folded into a generalization instead of two more special cases: - A parameter default (function f({ API_KEY } = environmentVariables)) and a binding-element default are the same by-name delivery as a variable declaration. The detector now keys on the ObjectBindingPattern itself and checks its parent's initializer, so every declaration position follows one rule instead of per-kind arms. - Parentheses group without changing the receiver, so (environmentVariables) is unwrapped before the identifier check — in the destructuring arm AND the member-access arm, which had the same hole unreported. Declined the for-of-over-array-literal finding: the receiver there is a container, not the environment object, and following data flow through containers has no fixed point — the same documented boundary as aliasing and computed keys. A test pins the boundary so it reads as chosen, not missed. Eight cases added; the seven receiver-rule cases fail against the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(secrets): a dot in prose is not a qualifier, and a literal computed key is a subscript Review round 11. Both findings were implementation-narrower-than-rule, fixed by consulting authorities the detectors already had rather than adding new ones: - python.ts: the receiver walk crosses whitespace so a parenthesized `other.` on a previous line is seen — but it landed on a comment's final period (`# Load the value.`) and discarded the genuine read on the next line. The landing position is now checked against the same lexer ranges that filter the candidates, which is also why the receiver check moves after lexing. - javascript.ts: `const { ['API_KEY']: key } = environmentVariables` is the element-access rule in pattern position, so a computed key holding a string literal resolves like a literal subscript; any other computed key keeps the runtime-name boundary a computed subscript already has. Eight cases added; the comment-period case and all three literal-computed-key cases fail against the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-18 18:32:20 -07:00
Theodore Li	8fb571ac54	feat(credentials): add managed credential groups (#6697) ... * feat(credentials): add managed credential groups * fix(audit): sync credential group mock * fix(credentials): serialize enrollment revocation * fix(credentials): isolate managed delegation * fix(credentials): serialize invitation lifecycle * fix(credentials): preserve enrollment lifecycle * refactor(credentials): migrate groups to application boundary * fix(credentials): serialize enrollment readiness * fix(credentials): preserve completed reconnect state * fix(credentials): revalidate policy before grant persistence * fix(credentials): prioritize expired invitations * fix(credentials): redirect unavailable oauth starts * fix(credentials): clarify managed oauth boundaries * fix(settings): complete feature flag test mocks * fix(credentials): clarify enrollment actions and entitlement errors * fix(credentials): preserve entitlement failure reasons * fix(credentials): refine managed oauth flow * fix(lint): use optional chain for pagination	2026-08-14 21:46:58 -04:00
Waleed	264d4f3469	feat(canvas): add a setting to turn off auto-focus when clicking blocks (#6685) ... * feat(canvas): add a setting to turn off auto-focus when clicking blocks Clicking a block animates the camera to center it, which zooms in far enough that you lose sight of the rest of the workflow. Add an "Auto-focus on click" preference (on by default, so existing behavior is unchanged) that keeps the camera still on click. Also re-record the auto-connect and canvas-error-notification tooltip previews and re-encode all three at a smaller size. * fix(canvas): keep click-marked framing when auto-focus is off, recut tooltips Gating the whole click branch on the setting also skipped the userFocusedWorkflowIdRef write, which is what stops <ReactFlow onInit> from running fitView over the user's framing. That would have blown away the framing of exactly the users who turned auto-focus off to keep it. Mark the workflow as user-framed on any plain node click and gate only the camera move. Crop the auto-focus preview to the recording's viewport center so the blocks are legible at the 240px width Tooltip.Preview renders at, and trim the 2.45s of empty lead off the error-notification preview. * docs(canvas): correct useAutoFocusOnClick scope to clicks only The TSDoc claimed the preference also gated arrow-key navigation, which calls focusBlockInView without consulting it. State the click-only scope and why arrow-key navigation and block creation are excluded.	2026-08-13 17:05:54 -07:00
Waleed	d96f3f95c8	perf(db): drop 0287's two zero-row scans from the ACCESS EXCLUSIVE hold (#6609) ... drizzle-orm 0.45.2 runs every pending migration inside ONE transaction (node_modules/drizzle-orm/pg-core/dialect.js:60-71), and migrate.ts:212 sets `statement_timeout = 0`. So the ACCESS EXCLUSIVE that 0287:2's ADD COLUMN takes on `workflow_blocks` is held, unbounded, through 0288 and 0289 to COMMIT — `lock_timeout` at migrate.ts:213 bounds acquisition only, exactly as that file's own TSDoc at :76-78 says. Every editor load, workflow save, executor block read, and realtime canvas op queues behind it platform-wide, and migrations run before image promotion (ci.yml:113-133), so the stall lands on 100% old-version traffic. Two of the three statements inside that hold did nothing. `data.errorEnabled` never existed in a released version — `git log origin/main -S errorEnabled` returns zero commits across main's entire history — so both statements filtered on it match zero rows, and the file's own comment said as much. There is no index on the `data` expression, so each was a full sequential scan of the whole table. Measured on PostgreSQL 17.9 against a 328 MB / 200k-row fixture built to the same bytes-per-row shape as the reported production table: 0287:2 ADD COLUMN 0.5 ms metadata-only, takes AccessExclusiveLock 0287:11 edge backfill 21 ms Nested Loop -> Index Scan on the PK 0287:20 (deleted) 47 ms Seq Scan, 200,000 rows removed, 0 matched 0287:22 (deleted) 47 ms Seq Scan, 200,000 rows removed, 0 matched A concurrent primary-key SELECT started 50 ms into the transaction was blocked 53-219 ms before and 22-25 ms after — the latter indistinguishable from the 21-33 ms control with no migration running at all. The two deleted statements accounted for 80,520 buffer accesses (~629 MB of in-lock I/O on that fixture) and 81% of the transaction's work. Editing an already-merged migration in place is safe here specifically because drizzle writes `hash` but never reads it back: the skip test at dialect.js:56-63 compares `created_at` against `folderMillis` only. The edit is therefore a no-op on every database that already applied 0287 (staging, dev, branch DBs) and takes effect only where it has not run. `meta/_journal.json` and the snapshot prevId chain are untouched. The only casualty is a stale `data.errorEnabled` key on branch databases a developer created it on. Nothing reads it: save.ts:50 and load.ts:89 both read the `error_enabled` COLUMN, and load.ts passes `data` through untouched. Alternatives rejected, each checked against the code rather than assumed: - An embedded `COMMIT;` to end the transaction early. 0289's four CREATE TYPE, its CREATE TABLE, and its three CREATE INDEX all lack IF NOT EXISTS, so a mid-batch failure in autocommit leaves them applied-but-unjournaled and the replay dies on 42710 — which migrate.ts:219 only retries for 55P03. That turns a transient stall into a wedged deploy. - Moving the statements to a new file after 0289. All pending files share one transaction, so it buys exactly zero lock reduction. - Rewriting the surviving backfill as `WHERE id IN (SELECT source_block_id FROM workflow_edges WHERE ...)`. Measured both: planner-equivalent. `source_handle` is unindexed, so both forms seq-scan `workflow_edges` (7.5 ms, identical) and then index-scan `workflow_blocks` on the primary key — it never scans that table. The IN form adds a HashAggregate and 1,170 more buffers, so it is marginally worse. Left as shipped. - Promoting the `data-backfill` lint from warn to annotate. `readAnnotation` only requires a non-empty reason and 0287 already supplies one per statement, so the rule would fire zero findings. 0288's nullable `retry` column is correct as-is and unchanged. Both delete-and-reinsert save paths on the deployed version (save.ts:30-63 and the realtime REPLACE_STATE handler) reset `error_enabled` to false and `retry` to NULL for a workflow saved by an old replica during the rollout; the ordinary realtime block upsert does not, because its `set` clause omits both columns. That residue is tolerable: everything the surviving backfill writes is re-derived from the edge set at workflow-block.tsx:754 and lib/workflows/persistence/utils.ts:196-201, and `retry` ships in this same release so it has no installed base. `bun run check:migrations origin/main` drops from three data-backfill warnings to one. The real migrator applied all 289 journal entries to a fresh PostgreSQL 17.9 database with the edited file, producing `error_enabled boolean not null default false` and `retry jsonb`.	2026-08-12 01:32:00 -07:00
6 people	263e3ca67e	improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec (#5273) ... * v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * improvement(external-endpoints): v2 versions with clean signatures + updated docs * feat(usage): accept X-API-Key on usage-logs list + export /api/users/me/usage-logs and /export now use checkHybridAuth — the same auth /api/users/me/usage-limits already accepts — so external monitors can read summary.bySourceCredits (the source breakdown of usage-limits' aggregate currentPeriodCost) instead of estimating Copilot spend by subtraction. Workspace-scoped keys are pinned to their own workspace's slice of the ledger: the filter defaults to the key's workspace and an explicit mismatch 403s. Both endpoints documented in openapi-core.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(billing): dedicated v2 usage endpoints; keep internal usage routes session-only Replaces the earlier X-API-Key enablement on /api/users/me/usage-logs with a dedicated public surface, so the internal Billing-settings endpoints can evolve with the UI while external monitors get a stable versioned contract: - GET /api/v2/billing/usage — current-billing-period summary with bySourceCredits (the source breakdown external monitors need to watch e.g. Copilot consumption without estimating by subtraction), plus limitCredits and plan - GET /api/v2/billing/usage/logs — cursor-paged credit ledger in the v2 envelope - workspace-scoped keys are pinned to their own workspace's slice; personal keys read the account ledger The public wire is credits-only: usage-logs rows now carry a hasCost boolean instead of dollarCost (the Billing UI only needed the >0 signal), and the rateLimit block is removed from the usage-limits response and docs (deploy-modal tab relabeled accordingly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(docs): validate OpenAPI specs against the Zod contracts in CI The specs in apps/docs are hand-authored because they carry what Zod never defines — error envelopes, status codes, prose, examples — so they can't be generated; check:openapi validates them instead: - spec integrity: $refs resolve, operationIds unique, 2xx documented, no orphaned component schemas - v2 conventions: every /api/v2 operation documents 401 + 429 and every 4xx/5xx resolves to the canonical { error: { code, message } } envelope - contract cross-check: contracts are auto-discovered from lib/api/contracts/v2 (each carries its method + path); doc<->contract coverage both ways, query/body/response field diffs via z.toJSONSchema - examples: documented request/response examples must parse with the matching contract's actual Zod schemas First run caught real drift, fixed here: 16 stale orphaned schemas in the core spec, the v2 billing ops referencing v1-shaped error components, deploy/rollback examples missing the required nullable lifecycle keys, CreateTableBody missing folderId, a legacy-grammar delete-rows example, and four knowledge document ops missing their required workspaceId query param. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(docs): recursive field diff in check:openapi + the deep drift it found A mutation test showed the doc<->contract field diff only compared top-level properties, so a typo inside the { data } envelope passed. The diff now descends through matching object properties and array items (both sides must expose a property set — passthrough contracts and prose-only docs end the descent instead of false-positive), with the Zod JSON-schema root doubling as the $defs context. Deep drift it immediately caught, fixed here: select-column config (options/multiple) missing from every tables column schema, AddColumnBody hand-rolling a third column shape (now composed from ColumnInput, with position/workflowGroupId as the per-op extensions the contracts actually admit), chunking strategyOptions undocumented, and the deployment lifecycle fields (activeDeployment/latestDeploymentAttempt) missing from DeploymentState. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(security): close the triggerType rate-limit bypass on workflow execute Caller-supplied triggerType flowed unchecked into preprocessExecution, whose checkRateLimit default turns OFF for 'manual'/'chat' — so any API-key caller, and any anonymous public-API caller billed to the workspace owner, could execute unthrottled by sending {"triggerType":"manual"} (async runs also skipped the worker-side check via admissionCompleted). External callers may now only send the redundant 'api' value; internal JWT callers ('workflow'/'mcp') are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * refactor(execution): extract enqueue/status/cancel into shared libs Prepares the v2 execution surface: handleAsyncExecution's queue logic moves to lib/workflows/executor/enqueue-execution.ts (slot/claim semantics encoded in a discriminated outcome, not HTTP statuses), the execution-status read to execution-status.ts, and the order-sensitive cancel machinery to lib/execution/cancel-workflow-execution.ts. The v1 routes re-render identically — their suites pass unmodified. Also: preprocessExecution gains rateLimitCounter ('sync'|'async') and its 429 now carries code RATE_LIMIT_EXCEEDED + retryAfterMs (previously indistinguishable from the concurrency 429 and Retry-After was discarded); and the duplicate cancel contract in contracts/logs.ts is unified on the full 5-value reason enum — its narrower copy made requestJson throw a client ZodError when cancelling a paused HITL run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): callable execution service + structured error classifier executeWorkflowService composes the same libs the v1 route holds inline (call-chain guard, execution-id claim, LoggingSession, preprocessing, deployed-state load + file-field processing, timeout-bound executeWorkflowCore, output hydration/compaction) for the deployed-state caller class — the seam the v2 execute route and in-process internal callers share, making the HTTP endpoint syntactic sugar. classifyExecutionError stops discarding the block context that buildBlockExecutionError already attaches at throw sites: failed runs now yield {message, code, blockId, blockName, blockType} with a stable append-only code enum (TIMEOUT/CANCELLED/USAGE_LIMIT_EXCEEDED/ INVALID_INPUT/BLOCK_EXECUTION_FAILED/CHILD_WORKFLOW_FAILED/ OUTPUT_TOO_LARGE/EXECUTION_FAILED), so callers route on error class instead of substring-matching messages — the single place raw errors are interpreted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): POST /api/v2/workflows/[id]/execute Thin route over executeWorkflowService: X-API-Key or anonymous public-API auth (sync/stream only for anonymous), strict body with body-flag async (no mode headers on v2), SSE passthrough for stream, and the execution resource response — executionId always present, in-band run failures are status:'failed' with the structured {message, code, blockId, blockName, blockType} error, sync timeout is status:'failed' + TIMEOUT instead of v1's 408, and a Response block's payload stays inside output (authors never control response status/headers on this origin). Async debits the async bucket and the 202 statusUrl points at the v2 executions resource. Adds CLIENT_CLOSED_REQUEST/SERVICE_UNAVAILABLE to the v2 error codes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): v2 executions status + cancel with queued backfill GET /api/v2/workflows/[id]/executions/[executionId] is the single status URL for sync and async runs: before the async worker writes the durable log row, status is backfilled from the job queue (deterministic job id) as 'queued'/'running' — closing v1's 202-to-pickup 404 window — and failed runs carry the structured error object. POST .../cancel renders the shared cancellation lib in the v2 envelope with the tightened 5-value reason enum. Both authenticate via the shared resolveV2WorkflowAccess (X-API-Key, authz masked as 404, allowPersonalApiKeys honored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): workflow tool + MCP bridge run in-process workflow_executor (workflow-as-agent-tool) short-circuits in executeTool through WorkflowBlockHandler — the same invocation boundary canvas child workflows use — mirroring the deployed_block_executor precedent. The MCP serve bridge calls executeWorkflowService directly instead of fetching its own execute endpoint; deployment-version pinning, MCP response-size rejection, and the actor override become typed options instead of header sniffing. Both callers drop the double admission slot and duplicate top-level log row the HTTP hop cost, and failed child runs now surface the structured error + child executionId so parents and MCP clients can route on error class and hand providers a reproducible handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(infra): CORS + CSP coverage for the v2 execute path /api/v2/workflows/:id/execute gets the same wildcard-origin, credential-free CORS policy as v1 (the default credentialed policy would block browser API-key calls and open a cookie CSRF surface) with X-Sim-Stream-Protocol allowed and no X-Execution-Mode (async is body-selected on v2), plus the COEP/COOP/CSP header block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(ui): deploy modal + copilot advertise the v2 execute surface All 20 API-tab snippets move to POST /api/v2/workflows/{id}/execute with the nested {"input": ...} body, async as the "async": true body flag (X-Execution-Mode gone), status polling against the v2 executions resource, the third tab renamed Usage and pointed at /api/v2/billing/usage, and {data} envelope unwraps in the printed responses. Fixes the latent baseUrl derivation (endpoint.split('/api/workflows/')) that would have silently built garbage URLs under a v2 endpoint, and deletes dead code (exampleCommand across 3 sites, getAsyncExampleTitle). Copilot deploy/manage/serializer endpoint builders and the api_trigger bestPractices example follow (the latter also drops its hardcoded staging host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * docs(api): document the v2 execution surface Adds execute, execution status, and cancel to openapi-v2-workflows.json with the structured ExecutionError schema (append-only code enum + block attribution) and the ExecutionResource contract, documenting the rules that differ from v1: modes are body-selected, a failed run is HTTP 200 with status 'failed', an executionId always means data (never the error envelope), queued status is visible immediately, and Response-block payloads stay inside output. Registers the three pages in the generated workflows meta.json and bumps the route-count baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): gate the whole /api/v2 surface behind one flag; UI stays on v1 Every v2 route now runs exactly one check immediately after auth — v2ApiGateError — and answers 404 when the `v2-api` flag is off, so the surface is invisible until it is deliberately rolled out. The gate is keyed on userId only: a workspace/org-keyed check would have to read membership for a caller-supplied id before authorization runs, and its 404-vs-403 split would leak cohort membership (the trap the per-domain table gate worked around by running late). The two executions routes inherit it from the shared access resolver; the tables-specific gate is removed so no route checks twice. `tables-v2-api` stays, now gating only the internal predicate-grammar route /api/table/[tableId]/query — note v2 tables routes move to the unified flag, so enabling them is a `v2-api` decision now. Reverts the deploy modal, copilot handlers, and api_trigger example to the v1 execute endpoint: v1 works unchanged, and the UI must not advertise a surface most users would get a 404 from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(executor): restore child-cost aggregation dropped by the staging merge Staging's custom-block rewrite deleted `aggregateChildCost` from workflow-handler.ts, and git merged that file cleanly — but this branch's workflow-tool-runner.ts, added for the v2 execute migration, still imports it. A silent semantic conflict: no marker, broken build. Taking staging's rewrite is correct, so the helper is defined locally in its one remaining consumer rather than resurrected in the file staging just rewrote. Same four lines over the still-exported `calculateCostSummary`, so a failed child workflow keeps billing the hosted-key spend it consumed instead of reporting $0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation (#6134) * refactor(orchestration): move the shared error contract out of lib/workflows OrchestrationErrorCode and statusForOrchestrationError are the contract every lib/[resource]/orchestration module returns against, but they lived inside the workflows module, so resource-neutral code (lib/folders) already had to import from a workflow path. Moved to lib/core/orchestration/types. Adds a 'locked' class mapping to 423. Both tables and workflows have a lock that forbids a mutation, and each caller was translating that to a status itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation Column update was implemented four times — the UI route, v1, v2, and the copilot table tool — each calling the same column services but owning its own guards, error mapping, and audit. The copies had drifted, and the drift was the bug: v2 was missing both guards, only the copilot copy minted stable option ids, and only v1/v2 audited. performUpdateTableColumn, performDeleteTable, and performDeleteTableRow now own that logic; all ten call sites reduce to auth, parse, call, render. The guards are asserted once in lib/table/orchestration rather than four times against four routes. Behavior this consolidates, previously true on only some paths: - The typeChanging guard. updateColumnType early-returns on an unchanged type and drops any options sent with it, so restating the current type alongside new options silently discarded them. v2 had no guard at all and, since its contract shares v1's body schema, accepted options and ignored them. - The select-unique guard. Each write is its own locked transaction, so a rename or type change paired with a constraint write that is going to fail commits first and then throws, half-applying the schema change. - Stable select-option ids. Cells reference the option id, so an edit that re-sends an option by name has to reuse it or every cell holding it is orphaned. Only the copilot path did this; normalizeSelectOptionsInput moves to lib/table/select-options and now covers every caller. It preserves a supplied id, so it is a no-op for the fully-formed options the HTTP contracts accept. - required forwarded into the type and options writes, so a conversion validates against the constraint the same request is setting. - An audit on every successful update. The UI route and the copilot tool emitted none. - Single-row delete through the row service. v2 did a raw db.delete, skipping assertRowDelete and deleteOrderedRow, so a delete-locked table returned 200 and the row-count bookkeeping never ran. - The delete actor handed to deleteTable, which audits only when a row was actually archived. v1 and v2 omitted it and audited themselves outside that check, emitting TABLE_DELETED for a no-op delete of an archived table. Failure classes come back as OrchestrationErrorCode; v2 renders them through a new v2ErrorForOrchestration, mirroring statusForOrchestrationError on the v1 and UI surfaces, so a given failure maps to the same status everywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(tables): bind the column-update tests to the orchestration function The base's route tests assert which column service each payload reaches — the behavior that now lives in performUpdateTableColumn. They mocked the `@/lib/table` barrel; the orchestration module imports the service directly, so they mock that too and keep asserting the same thing through the extracted implementation. The orchestration tests move onto the base's semantics: writes address the stable column id, a rename rides inside the write it accompanies rather than running first, and the currency guards replace the non-select options guard the service now owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(copilot): drop the column-type import the delegation made dead Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): move the audit log out of the table service `lib/table/service.ts` wrote its own audit rows, so whether an operation was audited depended on which function a caller reached for rather than on a user having performed it. That is what let v1 and v2 audit a no-op delete, and what made `deleteTable`'s optional `actingUserId` double as an audit opt-out flag. Worse, most sites fell back to `actingUserId ?? createdBy`, so an unattributed call was logged against the table's *creator*. The copilot `mv` path passed no actor at all: renaming someone else's table recorded them as the renamer. Audit now lives in the orchestration functions — performDeleteTable, performRenameTable, performMoveTableToFolder, performUpdateTableLocks — and the services just write. Internal callers (folder cascade, import rollback) keep calling the service and are silent by construction rather than by remembering to omit an argument. Two services now return what the audit needs: `deleteTable` reports whether it actually archived a row, so a repeat delete logs nothing; `updateTableLocks` returns the before/after locks, since only the locked write can observe the transition its description names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(tables): restore audit provenance and conflict status in orchestration Moving the audits into the orchestration functions dropped three things the routes had been carrying, and added one the orchestration now owns twice. - The v1 and v2 column-update routes passed `request` to `recordAudit`, so their audit rows recorded the caller's IP and user-agent. The orchestration function had no way to receive it. Every table orchestration function now takes an optional `OrchestrationRequestContext` and every HTTP route forwards it; the copilot and VFS callers, which have no request, omit it. - `classifyTableMutation` matched `TableConflictError` on "already exists" appearing in the message and reported it as `validation`, turning the UI route's 409 on a duplicate table rename into a 400. It now matches the type, the way `performRestoreTable` already did. - `captureServerEvent` ran on every delete while the audit was gated on a row actually being archived, so a repeat delete of an archived table still reported `table_deleted`. Both now hang off the same evidence. - The copilot delete path kept its own `captureServerEvent` from when the service did not emit one, double-counting every copilot table delete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * fix(tables): say which type a no-op column update restated A copilot `update_column` payload whose only content was the column's current type used to return success with the live schema, while the v1, v2, and UI routes rejected the same payload with "No updates specified". Delegating to `performUpdateTableColumn` unified them onto the routes' rejection — correct, but the message tells the caller its request was empty when it named a type. The orchestration function now reports the same thing `updateColumnType` reports when it loses this race concurrently: the column is already that type, re-issue without the type change. An empty payload still reads "No updates specified". Drops the copilot's `outcome.table ?? tableForUpdate` fallback with it — the comment described the no-op that can no longer reach that line, and a success always carries a table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * refactor(tables): classify failures by type instead of by message text The table module decided HTTP statuses by searching error messages for phrases. `VALIDATION_MESSAGE_FRAGMENTS` and `ROW_WRITE_ERROR_PATTERNS` held 32 substrings between them, and fifteen more lists were inlined in routes — 83 matchers over 17 files, each its own copy of the guesswork and already drifted apart. It made message wording load-bearing: `TableRowLimitError`'s own doc comment noted that its text had to contain "row limit" for a route to answer 400, and adding "already exists" to a rename message silently demoted a 409 to a 400 (the bug fixed one commit ago, by adding another special case). Services now throw `OrchestrationError`, which carries the transport-neutral `OrchestrationErrorCode` the layers above already speak. Classification is one `instanceof` in `orchestrationErrorResponse` (UI + v1) and `v2CaughtOrchestrationError` (v2). Every pattern list is gone. Wording is free to change; an unclassified error still becomes a generic 500, which is what an unexpected fault should be. `asOrchestrationError` walks the `cause` chain rather than testing the caught value directly: drizzle wraps a throw raised inside a transaction callback in a `DrizzleQueryError` whose own message is the failed SQL, so a bare `instanceof` would drop every failure raised inside `withLockedTable`. That is the same reason `rootErrorMessage` had to dig for a root cause before. Three throws stay bare `Error` deliberately — `Table ID mismatch`, `Workspace ID mismatch`, and `Failed to build upsert conflict predicate` are internal invariants no consumer classified, and they keep falling through to a 500. `Insufficient capacity` was in the pattern list with no producer anywhere in the codebase. Status changes, all deliberate: - `'forbidden'` joins the code union so the table-row-limit ceiling keeps its 403; without it this refactor would have flattened it to 400. - import-async's table-limit rejection: 400 -> 403, matching the two other create routes it had drifted from. - Renaming a table to an invalid name: 500 -> 400. `validateTableName` messages don't contain "Invalid", so no matcher ever caught them. - Restoring a table that isn't archived, or into an archived workspace: 500 -> 400. - A duplicate *column* name stays `validation`/400 rather than becoming a 409 like a duplicate table name. Both v1 and the orchestration have always answered 400 for it; changing a published status is not this refactor's job. The twelve tests that changed were asserting the substring mechanism itself, constructing plain `Error`s with magic strings. They now assert the real contract, plus new cases pinning that identical wording carrying no classification stays internal and keeps its message off the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials (#6150) * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials * fix(api): correct credential role, skill permission bar, MCP url identity, and custom-tool conflict mapping * fix(api): align credential mutation gating, provider-outage status, and unique-violation conflicts * fix(api): close unique-violation, revival, orphan-write, and env-rename gaps * fix(api): treat every provider-outage code as unavailable on create and update * fix(credentials): use the shared outage predicate on the session update path * fix(contracts): anchor the predicate double-cast annotation to the cast `check:api-validation:strict` counted 9 unannotated double-casts against a baseline of 8, failing CI. The predicate leaf schema was annotated, but the annotation sat above the declaration while the checker anchors on the line carrying the cast — five lines below, at the close of the object literal. The scanner walks back at most three lines and stops at the first non-comment one, so it hit `value: z.unknown().optional(),` and never saw the reason. Splitting the object schema from the cast puts them adjacent, so the existing reason binds. No behavior change — the cast, the schema, and the reasoning are unchanged. Also lowers the rawJsonReads ratchet 6 -> 5 to match the current count, which had drifted down; leaving it high lets a removed raw read silently come back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(skills): point the orchestration error contract at its moved module #6150 branched before #6134, so skill-lifecycle.ts imports @/lib/workflows/orchestration/types — the module #6134 moved to @/lib/core/orchestration/types. Git merged a file deletion on one side with a new file referencing it on the other: no textual conflict, broken build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(knowledge): make lib/knowledge/orchestration the single implementation (#6154) * refactor(knowledge): make lib/knowledge/orchestration the single implementation Knowledge base create was implemented four times — the internal route, v1, v2, and the copilot tool — and the orchestration around the shared write had drifted. Extract it the same way lib/table/orchestration was: services write, orchestration decides which writes run, guards them, audits them, and returns a transport-neutral failure. Behavior converged, not preserved: - One chunking default (DEFAULT_CHUNKING_CONFIG). The agent defaulted minSize to 1 against the API's 100, so identical input produced differently-chunked knowledge bases depending on who created it. The agent path now chunks at 100. - Every successful mutation is audited inside the orchestration function. The copilot tool called recordAudit zero times, so agent-created knowledge bases, document uploads, updates and deletes left no audit trail at all. - Failures classify by class, not by message text. The knowledge service errors are OrchestrationError subclasses and storage-quota rejections throw a shared StorageLimitExceededError, replacing four separate message greps for "already exists" / "does not have permission" / "storage limit". delete_connector reported the opposite of what happened. It reached the route through an internal HTTP self-call that sent no query string, so the route's keep-documents default always applied while the agent told the user the documents had been removed. The self-call is gone — all four connector operations run in-process — and the orchestration returns the real counts. Also: - OrchestrationErrorCode gains 'payload_too_large' (413 / PAYLOAD_TOO_LARGE). Without it, dropping the storage-limit message match would have regressed the documented 413 on knowledge base create and document upload to a 500. - messageForOrchestrationError renders a route's own wording for an unclassified fault, so a driver's message no longer reaches the client on a 500. - v1 and v2 knowledge base update now forward actorUserId, which the service requires for a workspace move; both omitted it. - The connector DELETE route reads deleteDocuments through parseRequest. Its contract declared z.boolean(), which would have rejected the string a query param actually is. - Drop the 409 from POST /api/v2/knowledge/{id}/documents in the OpenAPI spec. Nothing on the upload path throws a conflict; it was only ever reachable by the message match this change removes. Behavior change worth noting: a v1/v2 PUT carrying only the workspaceId scope field and no actual updates now returns 400 rather than 200 with the unchanged knowledge base. Deliberately deferred: document update remains internal-only. Extracting performUpdateKnowledgeDocument makes exposing it on v1/v2 a contract and a route away, but that is a new public surface rather than part of this consolidation. * fix(knowledge): make connector create atomic and stop flattening failures Review round 1 on #6154. - Resolve the billing payer before the connector is committed, not after. A malformed attribution header rejected post-commit left a live connector behind a 500, and a retry created a duplicate plus duplicate sync work. Manual sync resolves before writing its audit for the same reason. - Let the source-config validator carry its own failure class. Collapsing every rejection to `validation` flattened the connector PATCH route's 401 (stale stored credential) and 409 (missing workspace context) into a 400. - Add `unauthorized` to OrchestrationErrorCode. It is the class that 401 was already expressing on this route, and the v2 vocabulary already had UNAUTHORIZED; only the shared union was missing it. - Report a knowledge base that exists but failed to archive as failed, with the reason, rather than as not found. The copilot delete loop folded every non-not-found failure into `notFound`, telling the user it was never there. - Route copilot failures through the same message helper the HTTP surfaces use, so an unclassified fault's raw text (a driver's failed SQL) no longer reaches the agent verbatim while the UI and public APIs get the generic wording. * feat(api): expand the public v2 files surface (#6160) * feat(api): expand the public v2 files surface Adds folder support, rename/restore, move, bulk archive, share, and content replace to /api/v2/files, so managing files by API no longer stops at upload + download + archive-one. Routes are thin: auth -> parse -> perform* -> serialize. Share and content replace get their orchestration extracted first so the session routes and the public ones cannot diverge on the effective-authType resolution, the EE public-sharing gate, or the storage-quota classification. Presigned upload stays session-only: presign does an advisory quota check and the real debit happens in the separate register step, so a caller that never registers leaves unaccounted bytes with no reaper. The buffered multipart path debits inside uploadWorkspaceFile's own transaction. * fix(files): classify folder and content failures instead of 500ing them Bugbot round 1. The v2 routes map errorCode straight to a status, so every manager failure that arrived unclassified became a 500 for what is really a caller-fixable 400 or 404. - Folder manager throws OrchestrationError: missing target/folder -> not_found, reparent cycle / self-parent / restore-into-archived-workspace -> validation. - File manager does the same for the in-transaction 'File not found' paths that the earlier pass missed. - updateWorkspaceFileContent's outer catch re-wrapped everything in a bare Error, which stripped the class off StorageLimitExceededError and the new not_found alike. It now rethrows a classified failure untouched and attaches cause to the generic wrap, so asOrchestrationError can still walk the chain. - Every remaining perform* gained the asOrchestrationError branch. - renameWorkspaceFile returned the pre-update read, so the v2 PATCH reported a stale updatedAt; it now returns the timestamp it actually wrote. Docs: upload auto-suffixes a duplicate name rather than rejecting it, matching the in-app uploader. The description claimed 409 and was simply wrong. * fix(files): surface a failed upload read-back as the real error getWorkspaceFile swallows a query failure and returns null unless throwOnError is set, so a transient blip on the post-upload read reported as 'file could not be read back'. Distinguish the two: a real null after a just-committed write is an invariant break, a query failure is itself. * revert(api): drop the dedicated v2 file-folder routes File folders already live in the shared folder table as resourceType 'file' (#6045 cut them over, #6051 dropped workspace_file_folders), and the remaining file-specific folder machinery is being folded into the generic folder engine. Publishing /api/v2/files/folders/** would pin that transitional split into a public contract we'd then have to keep or break. Files stay folder-aware — folderId/folderPath on the projection, folderId on upload, and the move route — because a folder id is a folder.id and survives the unification untouched. Folder management belongs on /api/v2/folders once that surface serves resourceType 'file'; until then there is no v2 way to enumerate file folders, which is the deliberate gap. The orchestration classification fixes stay: the internal routes and the copilot file-folder tools still call those perform* functions. * fix(files): classify upload failures instead of matching their wording Bugbot round 2. uploadWorkspaceFile had the same outer-catch rewrap that updateWorkspaceFileContent did, so a blown storage quota reached the route as a bare Error and the v2 handler recovered the status by substring-matching the message. Any rewording silently demoted a 413 to a 500. - uploadWorkspaceFile rethrows a classified failure untouched and attaches cause to the generic wrap. - FileConflictError is now an OrchestrationError('conflict'), so a duplicate name classifies like every other conflict. Its 'FILE_EXISTS' discriminator had no readers and is gone; the instanceof checks elsewhere still hold. - The v2 upload handler uses v2CaughtOrchestrationError, dropping all three string matches. Also documents that bulk-archive is best-effort: unknown or already-archived ids are skipped rather than failing the call, and deletedItems is what actually happened. That asymmetry with the single-id DELETE was undocumented. * feat(api): add search, filtering, and sorting to the v2 list endpoints (#6189) * feat(api): add search, filtering, and sorting to the v2 list endpoints One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts: `search` (case-insensitive substring on the resource's natural name field), `sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship rather than inventing a third dialect alongside the Logs filters and the Tables predicate grammar. Every filter and sort is pushed into SQL. GET /api/v2/files previously read the whole scope and sorted/sliced it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Cursors are stamped with the sort they were minted under, so replaying one under a different sort is a 400 instead of silently duplicated or skipped rows. * fix(api): validate v2 cursor key values and compare timestamps at ms precision Two review findings, fixed at the root by making a keyset key own its cursor codec instead of hand-writing a decoder per sort. Cursor key values are caller-controlled, and matching the sort stamp and key count was not enough: an unparseable timestamp or a non-numeric size reached the query as an Invalid Date or NaN and surfaced as a 500. Each key now type- checks its own value and rejects a cursor it cannot hold, which both routes render as the documented 400. Timestamp keys now order and compare on date_trunc('milliseconds', col). Postgres keeps microseconds and defaultNow() populates them, but a cursor value round-trips through a millisecond-only JS Date — comparing the raw column against the truncated value re-admitted the page's own last row, duplicating it and stalling pagination outright at a page size of one. Reachable today via workspace_files.updated_at, which insertFileMetadata leaves to defaultNow(). * feat(api): complete the v2 workflows resource with versions and CRUD (#6184) * feat(api): complete the v2 workflows resource with versions and CRUD Adds version listing/detail plus create, update, and delete to the v2 workflows surface, which previously covered only execution and deployment. - GET /api/v2/workflows/[id]/versions — cursor-paginated, newest first - GET /api/v2/workflows/[id]/versions/[version] — version + pinned state - POST /api/v2/workflows, PATCH and DELETE /api/v2/workflows/[id] All six delegate to the existing orchestration and persistence helpers; no new domain logic. * fix(api): check folder containment before lock state; reject malformed version cursors assertFolderMutable walks a folder's ancestor chain without filtering on workspace, so inspecting it before containment let a caller tell a locked folder in someone else's workspace (423) from a nonexistent one (400). Create and update now assert containment first, matching the ordering import-workflow.ts already uses. A version cursor that decodes to JSON without a numeric version filtered every row out and returned an empty page with nextCursor null, which reads as a clean end-of-list. Malformed cursors are now a 400. * refactor(api): page workflow versions in the persistence helper listWorkflowVersions read every version row and the route filtered and sliced the result in memory, so the response was bounded but the query was not. It now takes optional limit/afterVersion, turning the cursor into a real keyset query; the route asks for limit + 1 and only trims the has-more probe. Both params are optional, so the internal, v1 admin, and copilot callers are unchanged. Also restores the untouched GET handler in [id]/route.ts to its original formatting — collapsing its signature had re-indented the whole body and buried the actual additions in whitespace churn. * feat(api): expand v2 tables with stateless multipart transfers (#6188) * feat(api): expand the public v2 tables surface Adds 16 operations so a v2 caller can do what the internal surface can: rename/move/lock a table, restore it, manage saved views, run enrichment columns, look up rows, and import/export with observable job control. Extracts lib/table/orchestration/import.ts (performTableCsvImport, performCreateTableFromCsv) and lib/table/export-stream.ts from the first-party routes, then repoints those routes at them, so v1 and v2 cannot drift on what an import or export actually does. events/stream, metadata and dispatches stay internal — they are editor state, not public API. * fix(api): make v2 table PATCH all-or-nothing and name the lock in every 423 Greptile P1: PATCH applied locks, rename and move as three sequential transactions, so a folder rejected mid-request left the earlier writes persisted while the response reported failure — and the schema-changed signal was skipped, leaving open clients on stale state. Every rejectable condition now runs before the first write, and the signal fires whenever anything did land. Cursor: v2TableLockError dropped the lock kind, so async import, column run, enrichment and table mutations returned a bare LOCKED. A table has four independent locks, so the caller could not tell which to clear. * fix(api): report the lock kind on classified 423s too, not just thrown ones The previous commit named the lock only where the rejection was thrown and caught at the route boundary. Where it instead arrives as a classified `errorCode: 'locked'` outcome — delete table, delete row, update column, and the table mutations — the kind was dropped, so those 423s stayed unactionable while their neighbours improved. The orchestration results now carry `lock`, and a shared `v2TableOrchestrationError` renders both arrival paths into the same `{ code, message, details: { lock } }` body. `details` is omitted rather than sent null when the kind is unknown, so a caller branching on it sees absence instead of a phantom value. * fix(api): make async table imports observable, not just startable `POST /import-async` pointed callers at `GET /api/v2/tables/jobs` to track progress, but that endpoint filters to `type = 'export'` — imports are derived onto the table itself, one write job at a time, and exports get a separate list precisely because they are excluded from that derivation. The public Table shape omitted those derived fields, so an async import could be started and cancelled but never observed to completion, failure, or progress. That is the gap the import/export/job-control set was meant to close. Table now carries `job` — id, type, status, rowsProcessed, error, or null when idle — and the import-async docs point at the table rather than the export list. * feat(api): make v2 table PATCH state which operations landed on failure Greptile held the PR at 4/5 on the residual non-atomicity and named two acceptable resolutions: make PATCH atomic, or have the contract adopt and expose partial-success explicitly. Atomicity would mean threading one transaction through renameTable, moveTableToFolder and updateTableLocks — three shared service functions with four non-test callers including the first-party route and two copilot tools — and deferring their per-operation audits to commit time. That is a refactor of shared write paths well outside this PR. So the contract states it instead. Every rejectable condition is already pre-validated, so a failure here is a genuine fault; when one follows a successful operation the error now carries `details.applied` listing what is live. Absent when nothing applied, so its presence always means "these changes took effect despite the error". Documented on the operation. `v2ErrorForOrchestration` gained the optional `details` this needs. * fix(api): make table lock flags read-only on the public v2 surface The new PATCH /api/v2/tables/[tableId] accepted a `locks` object, gated on workspace admin plus the table-locks feature. That still lets an API key clear the guard placed there to stop it: `write` is the floor for the endpoint, and admin keys are ordinary API keys, so a lock is no longer a boundary the key cannot cross. Locks stay readable on the table resource and enforcement is unchanged (a locked verb still returns 423). Changing one is now a first-party admin action only. The v2 body is declared here rather than reusing the first-party updateTableBodySchema, which keeps its `locks` field so the UI can still toggle them. It is .strict(), so a request carrying `locks` is rejected with a 400 naming the field instead of silently succeeding without applying it. * fix(api): keep reporting applied operations when the PATCH re-read fails The composite table PATCH promises that `error.details.applied` names the operations that are live despite an error, but `applied` was scoped inside the try. A rename or move that committed and was then followed by a throw in the final re-read — or a re-read finding the table archived — returned a bare 500/404 with no details, telling the caller nothing had landed. It would then retry into a duplicate-name conflict or repeat the move. `applied` is now function-scoped so every post-write exit carries it: the 404 on a missing re-read, a thrown lock error, a classified orchestration error, and the generic 500. `v2TableLockError` gains the same `extraDetails` parameter `v2TableOrchestrationError` already had. * feat(api): add workflow group writes to the v2 tables surface v2 exposed GET /groups but none of the writes, so the public API could run an enrichment or workflow column and read its binding, but never create one. A caller could add a plain data column and trigger the machine; wiring the two together still required the UI. Adds POST/PATCH/DELETE on /api/v2/tables/[tableId]/groups. The group is the unit that fills columns — one group feeds several — so creating one creates its output columns in the same call, matching the first-party shape rather than inverting it onto the column endpoint. Four departures from the first-party body, all public-surface concerns: - group.id is optional and server-generated. The UI mints an id to render optimistically; a public caller has no such need and a client-chosen id is a collision waiting to happen. - outputColumns[].workflowGroupId is dropped from the body and stamped from the resolved group, so it cannot disagree with it. - autoRun defaults to false. First-party defaults true so a UI add fills cells immediately; here it would make one POST fan out a metered run across every existing row. - A group naming neither a workflowId (type manual) nor an enrichmentId (type enrichment) is a 400 rather than a half-specified group the route has to guess about. Also rejects an outputColumns entry no group output feeds — the two arrays are joined by column name, and the first-party client builds both from one picker so it cannot desync, but a public caller can. Workspace containment on workflowId is asserted before it is persisted, on create and on any update that re-points the group; without it a table becomes a way to invoke workflows the key cannot otherwise reach. * improvement(api): make v2 table import and export async-only Drops the three synchronous entry points: POST /tables/[tableId]/import, POST /tables/import-csv, and GET /tables/[tableId]/export. Sync import tied a write to the lifetime of an HTTP request. The body *was* the data, so it carried a 10 MB cap that Next silently truncates past — a partial import reporting success. It also had no job, so a timeout mid-write left rows in place with nothing to poll and nothing to cancel. The async path reads the file from storage instead: upload via POST /api/v2/files for a key, start with POST /import-async, watch GET /tables/[tableId] -> job, stop with POST /job/cancel. Sync export carried no such hazard, but one shape per operation beats two: with both removed the surface has exactly one way to move a table in or out, and the CLI wraps the extra calls. This also removes the last multipart handling in v2 tables. Those were the only routes bypassing parseRequest — form fields were parsed by hand against separate form schemas, outside the contract system every other v2 write goes through. Create-a-table-from-CSV is now two calls: POST /tables, then /import-async with createColumns. csvImportModeSchema is append|replace, so there is no single-call create. Route baseline 1064 -> 1061. * docs(api): correct the import-async note about upload size limits The docstring claimed there is no synchronous upload endpoint and so no request-body size cliff. Both are wrong: POST /api/v2/files is a synchronous multipart upload with a 100 MB cap, and it is the only v2 upload path (presigned is deliberately absent). What async-only actually bought: the cap went 10 MB -> 100 MB, it fails on an explicit size check and a bounded body read rather than a proxy cap that silently truncates, authorization completes before any body is buffered, and the table write is a job that can be watched and cancelled. * feat(api): unify file and table transfers * improvement(api): make multipart transfers stateless * fix(api): make table import completion retries idempotent * feat(v2-tables): paginate the table list `GET /api/v2/tables` returned every table in the workspace in one response — it used the cursor envelope but hardcoded `nextCursor: null`, and had no `limit`. That was defensible when tables were only created through the UI; `POST /api/v2/tables` is public now, so a script can create them in bulk and the list has no way to ask for less. Adds `queryTables` alongside `listTables` rather than changing it, so the internal callers that genuinely want the whole scope are untouched — the same split `queryWorkspaceFiles` / `listWorkspaceFiles` already uses. Filter, order and slice all run in the query, so a `search` never costs a full-workspace read. A cursor whose values don't bind raises a validation error instead of being coerced to "no filter", which would have silently served page 1 under a resumed cursor. The keyset closes on `id` so a page boundary inside a run of equal names or timestamps stays stable. The shared `LimitQuery` doc component said "Maximum rows to return"; it now serves the table list too, so the wording is resource-neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(api): add multipart knowledge document uploads * fix(api): keep usage admission at knowledge upload session creation * feat(knowledge): wire knowledge base uploads to multipart sessions * fix(knowledge): refuse to abort an upload once a document is bound * fix(uploads): prevent multipart cleanup races * Unify file creation and signed upload sessions (#6264) * feat(uploads): unify signed upload sessions * fix(uploads): preserve attachment storage semantics * feat(files): add authored file creation * fix(uploads): omit hoisted S3 metadata headers * feat(api): add file metadata endpoint * improvement(api): scope folders to resource paths (#6284) * improvement(api): scope folders to resource paths * fix(files): serialize folder resolution with uploads * fix(files): release folder lock before upload setup * fix(api): normalize folder paths and unblock resource mutations * fix(api): make resource cleanup and metadata consistent * improvement(uploads): persist multipart sessions in postgres * fix(db): store table row trigger timestamps in UTC * improvement(api): default folder deletion to non-recursive * fix(billing): unify chat usage source * improvement(logs): expose trace spans on log detail * fix(logs): parse list trace spans * improvement(api): replace workflow jobs with execution resources (#6294) * improvement(api): replace workflow jobs with execution resources * fix(api): preserve legacy jobs while preferring v2 executions * fix(api): make execution polling resume-aware * fix(ui): hide async examples for public workflows * fix(api): bridge resume queue visibility lag * feat(api): add v2 workflow resume endpoint * fix(api): project pending resume attempts * fix(api): prefer terminal logs over stale resumes * improvement(api): unify v2 resource query layers (#6319) * improvement(api): unify v2 resource query layers * fix(api): address v2 review findings * fix(api): preserve cancelled queue status * fix(api): guard cancelled job transitions * fix(api): close v2 resume and log gaps * feat(api): rename v2 executions to runs * feat(api): split credentials and secrets * feat(api): add workspace metadata and email attribution * improvement(api): consolidate public v2 route handling * improvement(files): centralize operations across APIs and Copilot (#6392) * improvement(files): unify rename authorization * chore(skills): add file operation migration guide * improvement(files): consolidate file operation authorization * improvement(files): extract shared operation foundation * improvement(api): simplify internal route declarations * improvement(files): centralize application authorization * refactor(api): share workspace file name validation * refactor(files): centralize copilot application calls * docs(skills): generalize application operation migration * improvement(api): centralize remaining v2 resource operations (#6412) * improvement(api): centralize v2 resource operations * fix(api): preserve custom tool conflict errors * improvement(api): migrate policy-sensitive v2 reads (#6410) * improvement(workflows): centralize v2 application operations (#6411) * refactor(api): migrate v2 knowledge operations (#6413) * refactor(api): migrate v2 knowledge operations * fix(knowledge): fail upload completion on dispatch errors * fix(knowledge): preserve upload retry and VFS errors * improvement(tables): centralize v2 application operations (#6414) * improvement(tables): centralize v2 application operations * fix(tables): preserve run validation and signals * feat(auth): add scoped internal executor delegation (#6459) * feat(auth): add scoped internal executor delegation * fix(auth): derive delegation lifetime from one timestamp * Include share status in file metadata * feat(auth): centralize delegated identity policy (#6462) * improvement(copilot): consolidate application adapters (#6450) * improvement(api): harden application route boundaries (#6451) * improvement(api): harden application route boundaries * fix(folders): reject creates at workspace cap * fix(knowledge): enforce trusted workspace scope (#6452) * fix(knowledge): enforce trusted workspace scope * refactor(knowledge): declare v2 body lifecycle * finish knowledge application migration * refactor(knowledge): compose copilot batch commands * fix(knowledge): parse connector query flags * fix(knowledge): finalize partial batch effects * fix(knowledge): align merged application boundaries * fix(knowledge): close application boundary review gaps * style(knowledge): satisfy branch biome checks * fix(knowledge): page connector documents in editor * refactor: enforce Copilot table application boundary (#6453) * refactor: enforce copilot table application boundary * fix(tables): finish application boundary migration * fix(tables): restore scoped copilot imports * fix(tables): compose copilot commands atomically * fix(tables): preserve workflow group scheduling * fix(tables): complete fixed copilot composition * fix(tables): reject enrichment output mutation * fix(tables): complete authorized application boundary * fix(workflows): migrate Copilot application boundary (#6455) * fix(workflows): migrate Copilot application boundary * fix(workflows): finish delegated application migration * fix(workflows): encode VFS folder aliases * fix(workflows): close application composition gaps * fix(workflows): preserve VFS validation errors * fix(workflows): complete application boundary migration * test(workflows): format canonical binding coverage * fix(workflows): scope executor metadata reads * fix(workflows): bind executor metadata targets * improvement(skills): align application operation guidance (#6532) * feat(api): expose v2 resource owners * fix(api): distinguish visible resource authorization failures (#6537) * feat(api): generate v2 OpenAPI from contracts (#6509) * feat(api): generate v2 OpenAPI from contracts * fix(api): preserve string boolean wire defaults * fix(api): document file download headers * fix(docs): use TypeScript CLI with Next.js * fix(docs): avoid client-rendered theme script * fix(api): document departed audit default * feat(api): replace legacy core docs with v2 * feat(api): generate v2 OpenAPI from contracts * feat(api): refine generated v2 OpenAPI docs * fix(docs): align localized v2 execution examples * fix(ci): restore Helm diff and sync audit mock * fix CI regressions after staging merge --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-11 05:45:25 -04:00
4 people	245ad1bd46	feat(workflows): new workflow block card, progress indicator, colors, dsl for natural language preview, retry configs (#6458) ... * improvement(workflow): refine canvas interactions and rendering * fix(workflow): keep outputs on the right, focus newly created blocks Connection anchors: an output now always leaves a card from the right. The cursor swell lets a drag start on any edge, but the left side is the input, so anchoring an outgoing edge there drew a line out of the input port and read as a second input. `normalizeCursorSourceHandleId` resolves every drag to the right anchor, `normalizePositionedSourceHandleId` collapses `source-left` alongside the legacy vertical anchors (so data from the API, an older client, or a stale save self-heals on load), and only the right-side source anchor is mounted. Drops in `onConnectEnd` are always source -> target. The branch that reversed the edge for a drag starting on an input could never run: the `target` handle is `isConnectableStart={false}` and the positioned side anchors are `isConnectable={false}`, so React Flow never reports an input as a drag origin. Removed it and its now-unused imports. A newly created block is centered once its node mounts and is measured, so a card added from a drag-release, the block menu, or the toolbar is never left off-screen or under the editor panel. The editor panel's block icon uses the same type accent as the card's badge instead of the block's legacy `bgColor`, which had left the panel on the old per-integration brand colours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(workflow): floor header-only card height, adopt brand tag palette The Start card intermittently collapsed after load, squashing the action-menu tab so its icon row sat over the card. `.workflow-drag-handle` is the host the border renderer measures, and both it and the header row took their height from `blockHeight && blockHeight > 0`. `blockHeight` comes from the deterministic-dimensions pass and is already floored at MIN_PAINTED_HEIGHT (48), but it is absent on the first frames — and with no floor the host collapsed to its natural content height (25.5px for a header-only trigger, exactly the title's line box). The border builds its perimeter from `host.offsetHeight`, so that window painted a sub-floor card: too little straight edge remained on the vertical runs for the action-menu tab, which collapsed into the corner arcs. Whether you saw it depended purely on whether the dimension publish had landed, which is why it reproduced on one workflow and not another. Floor all three: the host, the header row (so `items-center` centres the title and type tag rather than pinning them to the top), and the border's own `offsetHeight` read. Also raise ACTION_MENU_CONTENT_READY_THRESHOLD to 0.9. At 0.8 the 24px icon row was revealed while the swell had only reached 22.4px of its 28px — shorter than the row it contains. Secondary to the above, but a real overflow window on its own. The test now pins the ratio rather than the constant. Tag palette moves to fixed brand values (hex, not derived oklch) with two inks — #F8F8F8 on dark fills, #1A1A1A on light. Tones are renamed to match what they render. `green` (2.55:1) and `orange` (3.15:1) sit under WCAG AA against their paired ink; both are deliberate brand decisions and are documented in the component. Deploy and Run take two new Button variants rather than className overrides, so `tertiary` stays green everywhere else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * improvement(workflow): polish workflow canvas interactions * fix(workflow): restyle loop drop target outline * fix(workflow): shorten human block catalog label * fix(workflow): canonicalize realtime edge handles * code cleanup * sizing fixes * improvement(blocks): sentencify every block * change tebse * improvement(workflow): land notes UI and execution progress, consolidate duplicates Ports the notes canvas editing and execution progress-indicator work, then removes the parallel paths it arrived with so each concern has one owner. Fixes found while consolidating: - Note height was measured while the card was expanded to NOTE_EXPANDED_WIDTH, where text re-wraps shorter, and published as the node's compact height. The collapse then animated to a height never measured at the compact width. - The edge pulse glow filter used the default objectBoundingBox units, so a straight horizontal edge — what an auto-laid-out chain produces — resolved the filter region to zero height and stopped the glow rendering entirely. - A subflow's inner Start pill still read isNodeSelected while its border read usesSelectedVisuals, so the two disagreed during execution. - The Run/Stop button's disabled prop gated only Run while its handler cancelled unconditionally, offering a Stop the cancel route answers with 403. Consolidated: - One note editor. The view's built-in textarea was unreachable in production (the app always injects the markdown editor) and was kept alive only by tests asserting against it; renderContentEditor is now required. - onBlur/onCancel collapse to onEndEditing — content persists per keystroke, so there was never a draft for a cancel path to discard. - DEFAULT_NOTE_COLOR, the note height bounds, the note content reader and the card width each had two or three definitions; each now has one. - Removed with zero consumers: graphite/graphiteSubtle button variants, data-subflow-selected, inputPlaceholderClassName, an effect that could never fire, and getNoteColorOption's unreachable fallbacks. Restores the role='status' announcement the progress rewrite dropped, and hardens isNoteColor against inherited Object keys. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(workflow): reuse the platform markdown editor in notes Notes carried their own TipTap wiring — a second markdown editor that reimplemented, more thinly, what `RichMarkdownField` already does for the skill modal, skill fields and the deploy version description. It is now a ~20 line skin: the Note supplies its type scale and per-colour selection tint, and the field supplies the extension set, frontmatter held out-of-band, the round-trip safety gate and its raw-source fallback, and markdown paste. `RichMarkdownField` gains two additive props, both defaulting to today's behaviour so the file editor is untouched: `surface` ('field' | 'bare') and `proseClassName`. All three existing consumers pass an explicit `minHeight` and no `surface`, so they take the original path unchanged. Exiting the note editor moved to the card, because the editor's `/` and `@` menus consume Escape to close themselves and ProseMirror checks `editorProps` before plugin handlers — intercepting it inside the editor would have broken both menus. The card now honours Escape only when nothing already consumed it, which also let `onEndEditing` leave the injection contract. The note editor is lazy now, matching every other consumer: it was pulling TipTap and the full extension set into the canvas's initial chunk. Also: - One `areRunFromBlockDependenciesSatisfied`. The ActionBar, the canvas context menu and the run-from-block handler each carried a byte-identical copy, and the handler expressed the snapshot requirement differently, so the affordance and the action could disagree. Each copy also re-scanned `edges` once per incoming edge, on every ActionBar on the canvas. - Reduced motion is one `usePrefersReducedMotion` in @sim/emcn rather than a sixth ad-hoc `matchMedia`. The edge pulse now stops rendering instead of hiding: `motion-reduce:hidden` is `display: none`, which left four SMIL timelines running per edge. - The pulse glow bleed covers the canvas minimum zoom. The strokes are `non-scaling-stroke`, so the 6px tail spans 3/zoom user units — 30 at 0.1. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(workflow): port canvas styling from workflow-updates Ports the 14 styling commits your colleague added since the last sync, leaving the ~68 staging PRs on that branch alone — those are platform/core work, not this. Cherry-picked individually rather than merged so each conflict was small enough to reason about. What came in: - Core block colors unify behind a two-level map: block type -> semantic role -> accent, replacing the flat per-type table. Adds `purple` and `content` tones to ChipTag, and a shared `WorkflowTypeIcon` that replaces the hand-rolled ChipTag + accent lookup at each discovery surface. - Native triggers take semantic colors; the deployments block moves to the shared Rocket icon and drops its now-unused `iconColor`. - Running-state polish: loader artwork and position, stop hover in dark mode, the loader blended into the execution swell, and tooltips suppressed for actions that are hidden mid-run. - The toolbar drag preview clones the rendered icon container instead of rebuilding a bgColor tile, so it matches what the canvas paints. - The sidebar shows route-derived workspace identity instead of a skeleton while the full record loads. Conflict resolutions worth knowing: - The running-loader artwork went through the shared `Loader` and back to the custom SVG on their branch; the second commit is the intent, so that is what landed — keeping our `role='status'` announcement layered on top. - Two commits carried the lucide-react -> in-house icon migration along with them. That migration is a staging change we have not taken, so our imports stayed on lucide: adopting it in two files would leave the icon set split across the app. - `getMappedWorkflowTypeAccent` referenced a constant their refactor removed. It had no consumers left once the search modal moved to `WorkflowTypeIcon`, and their branch deletes it too, so it is gone here. Co-Authored-By: Claude <noreply@anthropic.com> * feat(workflow): make the subflow Start swell a real connection source Dragging an edge into whitespace opens the add-block picker, but starting that drag from a loop/parallel Start pill did nothing: the pill's border swell was visual-only. Regular blocks and the container's own exit mint a draggable cursor handle from their swell; the pill rendered only its invisible 14px static strip, so grabbing the glowing affordance started no connection at all. Everything downstream already worked and was nearly unreachable: - the drop hit-test skips subflow containers, so a release inside the loop opens the picker there (z 2000, above containers) - handleToolbarDrop parents the new block into the container at the drop point - it already carries the exact boundary rule for this source: a container start handle only wires to a child of that container The pill now runs the same cursor-handle machinery as the container view, with one deliberate difference: its temporary handle carries the branch-cursor form of the start id. The plain cursor id normalizes by block type — for a container that is `loop-end-source`/`parallel-end-source`, the exit — so a swell drag from Start would have persisted as an edge leaving the container. The branch form passes `loop-start-source`/`parallel-start-source` through normalization verbatim on both the picker and direct-connect paths; a test pins that contract. Co-Authored-By: Claude <noreply@anthropic.com> * fix(notes): stop the field's prose classes recoloring bare-surface editing Opening a note for editing shifted the text and turned it black: the ProseMirror root unconditionally carried `rich-markdown-prose rich-markdown-field-prose`, which pin the field's own ink and type ramp — `--text-primary` at 15px/25px, then 14px/22px — overriding the card's `text-current` at 14px/20px the moment the editor mounted. `surface='bare'` means the host owns typography (the Note card mirrors its rendered view via `proseClassName`), so on that surface the root now carries no shared prose classes. The field surface is untouched. Edit mode inherits the note colour's ink — including the caret — and sits on the same metrics as the read view. Co-Authored-By: Claude <noreply@anthropic.com> * fix(workflow): give the in-flight connection line contrast inside containers The drag line was drawn but camouflaged: its default stroke was the resting-edge grey (#e0e0e0), which disappears against a loop body's opaque `--surface-3` fill (~1.1:1) — so dragging an edge inside any container, nested included, showed nothing. The z-order was never the problem; the connection line layer already sits above every node. The default token is now `--text-muted`, one value with contrast on every canvas surface, still lighter than the `selected` variant so the variant hierarchy holds. No per-surface special-casing. Resting edges inside containers share the same camouflage (`--workflow-edge` on `--surface-3`) — left alone deliberately: recoloring placed edges is a design decision, not a bug fix. Co-Authored-By: Claude <noreply@anthropic.com> * fix(notes): match edit mode to the read view, and land the caret where clicked Three defects, all from the read and edit views being built independently. 1. Blocks jumped up ~12px on entering edit mode. Streamdown wraps its output in a container carrying `space-y-4` plus first/last margin resets, which outrank the per-element margins in NOTE_COMPONENTS — so that wrapper, not those margins, is what the read view actually paints. The editor had no equivalent. The rhythm is now named (NOTE_MARKDOWN_FLOW), passed to Streamdown explicitly so a dependency upgrade cannot move the read view out from under the editor, and mirrored on the ProseMirror root. Tailwind's JIT only sees literal strings so the mirror cannot be composed from the constant; a test pins the two together instead, and fails if either side drifts. 2. The caret was barely visible: it inherited the note's 75%-opacity ink. The palette owns per-colour chrome, so it now names the caret alongside the selection tint. 3. The caret always landed at the document end. The read view sits under a full-bleed overlay that must swallow the click to enter editing, so the point never reached the editor and `autofocus: 'end'` was all that was left. The view now forwards that point and the field resolves it through `posAtCoords` on create — after the DOM is laid out, which `autofocus` cannot wait for. Keyboard activation carries no point and still lands at the end. `autoFocusAt` is additive on the shared field and defaults to null, so the file editor and the other three consumers are unchanged. Co-Authored-By: Claude <noreply@anthropic.com> * fix(workflow): stop edges rendering behind top-level subflows Containers are z-indexed by nesting depth, so a top-level subflow is 0. Edges derived their z from their parent container — `+1`, or 0 with no parent — so a root-level edge landed on exactly the same z as a root-level subflow. Equal z-index falls back to DOM order, and React Flow paints the nodes layer after the edges layer, so the container's opaque body won: any edge crossing a top-level loop or parallel was drawn behind it, in-flight or persisted. Edges now sit in their own band above the whole container scale and below cards, keeping both the deeper-container-wins ordering and the rule that a line always passes behind card chrome. This is why the edge became visible only once a block was dropped: the new block is selected, and an edge inside the container was already `containerZ + 1`, clear of the tie. The in-flight connection line is declared in the same scale rather than inheriting React Flow's stylesheet default of 1001, which is both below a selected container child and outside the scale this file owns. Its stroke moves to `--text-secondary`, the token the canvas already uses for an active edge — the previous `--workflow-edge` grey is ~1.1:1 against a subflow body. Co-Authored-By: Claude <noreply@anthropic.com> * feat(notes): paste and drop images through the workspace-file pipeline Traced the file editor's image path end to end and reused it verbatim rather than minting a note-specific source: insertImages -> useUploadWorkspaceFile -> POST /api/workspaces/{id}/files/presigned -> direct-to-S3 PUT -> workspace_files row -> the editor persists the workspace-scoped /api/workspaces/{id}/files/inline URL, which the serve route authorizes by workspace membership and the embedded-image-ref machinery already recognizes for share rewriting and referenced-by-doc tracking. The shared field gains an optional `uploadImage(file) -> {url, alt} | null` capability. With it, image paste/drop uploads sequentially and inserts each at the evolving position, mirroring the file editor's flow, with a bail if the editor unmounts mid-upload; without it, the existing swallow-guard on file drops is unchanged, so the skill modal, skill fields and version-description consumers behave exactly as before. The upload mutation owns its own toasts. The note host wires the capability with folderId null, so note images land in the workspace Files root — visible, manageable and deletable there like any other upload. The note read view renders images through its Streamdown components map with the card's own sizing. Co-Authored-By: Claude <noreply@anthropic.com> * improvement for notes, subflows * fix(uploads): surface the server's message when a multipart upload is refused A file over the 50MB direct-PUT threshold goes through multipart initiate, which is where the storage quota is enforced — but the client threw away the response body and reported `Failed to initiate multipart upload: Payload Too Large`. That is the string the upload mutation puts in its toast, and it names neither which limit was hit nor by how much, so the one place that answer surfaces didn't have it. It now prefers `errorBody.error` exactly as `getPresignedUploadInfo` already does on the single-PUT path, and passes the body through as the error's details. Control flow is unchanged: still throws, still `MULTIPART_ERROR`, and the cloud-storage-absent branch above still claims its 400 first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(notes): restore GFM in the note read view Streamdown's `remarkPlugins` prop REPLACES its default plugin list rather than extending it, and remark-gfm is one of those defaults. The note passed `[remarkBreaks]` — so the read view silently lost every GFM construct: task lists, tables, strikethrough and autolinks. The editor writes all of them (it has TaskList, TableKit and Strike), so a note round-tripped through editing came back as raw source the moment editing closed: `- [x] HELLO` rendered as a disc bullet followed by the literal text `[x] HELLO`. `NOTE_COMPONENTS` has carried table/thead/tbody/tr/th/td entries this whole time that could never fire. Restoring the plugin is only half of it: remark-gfm marks a checklist `contains-task-list` and emits a native checkbox, which under the note's generic `ul` styling renders a checkbox sitting behind a disc bullet — the same defect the editor had before the chrome/typography split. The read view now drops the marker and indent for a task list, lays the row out as a flex line, and styles the checkbox to match `.rich-markdown-nodes input[type="checkbox"]` declaration for declaration, tick clip-path included, so the two views agree either side of a click. `remark-gfm` is now a declared dependency of the renderer package rather than one borrowed transitively from streamdown. Five tests cover the GFM surface — checked/unchecked boxes, no literal `[x]`, the marker only dropped for checklists, tables, strikethrough — and four go red with the plugin removed. Checked the other three Streamdown call sites (Chat, the chat interface renderer, the changelog): none override `remarkPlugins`, so none were affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(canvas): drop dead markers and share the tile-brightness maths Review pass over the branch against staging. Dead code removed: - `tileIconColorClass` in the renderer package — never called; only its `isLightTileColor` sibling is. - `data-connection-selector-search-frost`, `data-workflow-cursor-edge` and `data-workflow-cursor-source-side` — written on three elements, read by no stylesheet, selector or test. - `CHIP_TARGET_SELECTOR_TYPES`, `MAX_CHIPS` and `chipPriority` were exported from `canvas-rows.ts` but only used inside it. Consolidated the one real divergence: the renderer package carried a hand-copied mirror of the app's perceived-brightness maths, because it may not import app code. The copy had already drifted — it dropped the `white`/`black` keyword handling, so a block shipping `bgColor: 'white'` would render a white `currentColor` icon on a white tile on the canvas while every other surface drew it black. No block ships one today, which is exactly why nothing caught it. The function now lives in `@sim/utils/color` and both sides import it; only the 0.75 threshold stays local to each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(canvas): share the z-scale, fix the preview's edge layering, and re-home strays The preview canvas carried its own z numbers and had the collision the editor canvas was fixed for: containers at nesting depth, top-level cards at an implicit 0, and edges at 0/5/10 by execution status — so a default edge tied with a top-level subflow and painted behind it, while a success edge painted over unselected cards. The scale now lives once, in `@sim/workflow-renderer/canvas-layers`, and both canvases read it. The preview keeps its status ordering, expressed inside the shared edge band rather than as a second set of magic numbers. Placement and duplication: - `perceivedBrightness` moved to `@sim/utils/color`, with its unit test, and its consumers import it directly. It had been re-exported through `lib/colors/brightness.ts`, and the renderer package kept a hand-copy. - `filterAcyclicEdges`/`wouldCreateCycle` were pass-through wrappers in the workflow store's utils over the real implementations in `@sim/workflow-types`. Deleted; the three consumers import the source. - `lib/ui/glass-surface.ts` was a one-constant, one-consumer app-wide module, and its consumer then aliased it a second time. Collapsed into the navbar shell. - `nested-subflow-node` was set on nested container nodes in both canvases with no stylesheet, selector or test behind it. `packages/workflow-renderer` now has its own vitest config, so the four mount tests for its components live with the components instead of in `apps/sim/lib/workflows/**`. That immediately earned its keep: `apps/sim` excludes test files from type-check, and once these were checked, tsc found three `SubflowNodeView` renders being handed a `renderContentEditor` prop it does not accept — a copy-paste from the note cases that had been silently ignored. Verified: type-check 23/23, 21,143 app tests + 49 renderer + 147 utils, biome clean, all 23 audits pass (`check:bare-icons` imported the moved helper and was repointed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix field noun bug + notes * fix(notes): let a note service the canvas actions the panel editor cannot The panel editor clears any note put in front of it and renders nothing, but the block menu still routed Rename and Open Editor through it. Rename latched the editor's rename state onto the note — `handleStartRename` reads the store directly, so it saw the id the menu had just set — and nothing reset it when the clear ran. `handleSaveRename` writes to `renamingBlockIdRef`, so the header went on showing a rename field over whatever was selected next and saved that name to the note. Open Editor was a plain no-op that opened an empty pane. Rename now goes to the card, which expands and opens its own title — the same menu-to-card routing Add Image already used, so both events now live in one `lib/workflows/notes/canvas-requests.ts` and `add-image.ts` keeps only its markdown concern. Open Editor is hidden for notes. The panel editor also drops any rename whose block stops being the selected one. That is belt-and-braces for notes now, but it closes the same hole for ordinary blocks, where only the input's blur ended a rename and blur only fires if it held focus. A rename interrupted that way is now discarded rather than left pending. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(canvas): author sentences for Snowflake and Mintlify, repoint Instagram's Staging's two new integrations shipped without a `canvasPresentation`, so their 39 operations painted the field rows the rest of the canvas has stopped using. The Instagram break is the more interesting one: staging renamed the insight metrics subblock `metrics` -> `insightMetrics` while this branch was adding sentences that named `metrics`. Both hunks merged cleanly — the union check reports the file as an exact union — and the result was two clauses pointing at a field that no longer exists, which resolves to nothing with no error and no log. Only `check:canvas-sentences` sees it. Two Snowflake sentences say something the block does not do, so they anchor elsewhere: `taskName` filters `list_task_runs`/`get_task_run` rather than keying them, and `table` filters `introspect_schema` — blank means "every one", not "not filled in yet", and a core chip would have claimed otherwise. Coverage is back to 4727/4727 operations across 321/321 blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(canvas): keep the block-type tag naming its type after a rename The header tag dropped its label whenever the block's title already said the same word, so the same block read two ways depending on nothing the user did deliberately: a freshly dropped Wait showed a bare icon, and its second copy — auto-named "Wait 2" — showed "Wait". The tag looked like a badge that appeared on rename rather than a fixed part of the header. It now always names the type, which is what loop and parallel containers already do with their own tag, so every card on the canvas reads the same way. `blockName` was only ever read for that comparison, so the prop is gone rather than left behind for a future reader to wonder about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(deploy): compare edge handles by port, not by spelling Two places answer "does this need redeploying?" and they load their sides differently. The client diffs the live store against `/api/workflows/[id]/deployed`; the server diffs the normalized tables against the version's raw jsonb. Only some of those paths run handles through `loadWorkflowFromNormalizedTables`, so a snapshot holding a side-anchored id (`source-right`) met a canonical one (`source`) on the other side and the set comparison read it as every edge being removed and re-added. Each answer therefore differed, and they arrive on separate query timelines: the button reads the client's, the modal badge reads the server's, so the state flipped between Live and "Update deployment" with whichever query landed last until both settled. `normalizeEdge` now canonicalizes both handles, so the comparison cannot tell two spellings of one port apart no matter how its inputs were loaded. The existing normalization in `materializeDeploymentState` stays — that path also feeds React Flow, which needs the handle it mounts to match. The preview's error port had the mirror problem: it rendered for every non-trigger block regardless of `errorEnabled`, so a card with no error row grew a red knob anyway. It now gates the way the editor canvas does, keeping the port mounted when an error edge already leaves it so React Flow cannot drop that edge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(deploy): stop counting the error flag twice in change detection `errorEnabled` has two homes. It persists inside the block's `data` jsonb — the realtime server `jsonb_set`s it there, and load mirrors it back onto the block as a field — so it reached the diff twice, and only some paths populate the copy. `setBlockErrorEnabled` writes the mirror alone, so right after toggling the port the live block said `errorEnabled: true` with `data.errorEnabled: false`, while the snapshot the deploy had just taken from the tables said true in both. The diff read the stale `data` and reported the workflow as changed the instant it finished deploying — then a state refetch rehydrated the block and it agreed again. That is the flip between Live and "Update deployment": the button and the modal read two different queries, so each landing swapped the answer. A block created in-session had the same shape from the other side, its `data` carrying no key at all against a persisted `false`. Excluded from `normalizeBlockData` alongside the other fields that are duplicated out of the block's own state. The block field is still compared on its own, with `!!`, so absent and `false` agree and turning the flag on is still a change. Fixing the store to write both homes was the other option and is not taken: nothing reads the in-memory `data.errorEnabled` (save and load both let the block field win), so it would add a second copy that only the diff could see — which is the shape of this bug, not its fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(blocks): give the error-output flag a column instead of two homes `errorEnabled` had no column, so it persisted inside the block's `data` jsonb and was mirrored onto the block as a field on load. Every writer had to route its `data` through `withPersistedErrorEnabled` or silently drop the toggle, the realtime op `jsonb_set`, and change detection saw the same value twice — which is what made the deploy badge flip between Live and "Update deployment" after toggling the port. Its siblings — `enabled`, `horizontal_handles`, `advanced_mode`, `trigger_mode`, `locked` — are all boolean columns; `data` is for React Flow and subflow state. The flag belongs with them, so it now has `error_enabled` and one home. The shuttle helper, its `BlockData` mirror, the store's fallback read, and the comparison exclusion the duplication forced are all gone. Backwards compatibility, since released versions draw the error port with no toggle in front of it: a block already wired to an error edge HAS the output on, because there was no other way to draw that edge. That rule is now stated in three places and none may be narrowed to read the flag alone — - the migration backfills `error_enabled` from the edges, so live rows are true before any new code reads them; - `materializeDeploymentState` derives it for a version's frozen jsonb, which the migration cannot reach — otherwise every workflow deployed before the toggle would ask to be redeployed once; - `workflow-block.tsx` keeps it at render time for states that reach the canvas through neither (imports, copilot edits), where unmounting the port would make React Flow drop the edge leaving it. The migration also moves any `data.errorEnabled` a developer created on this branch onto the column and strips the key; both statements match zero rows in production, where it never shipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(canvas): realign the Snowflake and Dynatrace sentences with staging's blocks Both breaks are the class the union check cannot see: separate hunks of the same file merged cleanly, and the result names fields that no longer exist. A sentence that does resolves to nothing, with no throw and no log. Snowflake's rewrite (#6474) moved database, schema, table, warehouse and procedure onto canonical selector pairs, so seven clauses anchored on ids that are gone. Each now names both members of its pair, which is also what keeps the card readable for someone working in advanced mode. Its nine new operations have sentences. Dynatrace (#6463) scoped the mute reason to the operations that mute, because unmuting accepts exactly one — so the two unmute sentences were asking for a field their card no longer shows. They drop the clause. Coverage is 4736/4736 operations across 321/321 blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(api): drop the block-data error flag from the workflow contract Left behind by the consolidation: the flag no longer lives in `data`, and a schema that still declares it there invites the mirror back through the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(deploy): compare edge handles by port, so a falsy one cannot read as changed `loadWorkflowFromNormalizedTables` now runs handles through the canonicalizer, which falsy-coalesces — so an edge persisted with `sourceHandle: ''` loads as no handle at all. The server diffs that against the deployment version's raw jsonb, which still has `''`, and the set comparison reads one edge as removed and another added. Every workflow holding such an edge would ask to be redeployed the moment this ships, for nothing. Two write paths use `?? null` rather than `|| null`, so `''` is reachable. Canonicalized inside `normalizeEdge` rather than at either call site: the two sides are loaded by different paths and only some of them normalize, so the comparison has to be unable to tell two spellings of one port apart however its inputs arrived. This is the change reverted in 066e18ac28. That revert reasoned only about side-anchored ids, which are genuinely unreachable — it missed that the same coalesce collapses the empty string, which is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(deploy): keep ignoring the error flag's old home in block data Deploying never converged: the badge asked to redeploy again the moment it finished, forever. The flag lived in `data` before it had a column. `0287` moves it, but a migration only reaches the live tables — every deployment version already written is frozen jsonb that keeps the old key. The wire schema no longer declares it either, so Zod strips it from the live state on its way to the client. So the two sides of the check genuinely differ: live `data: {}` against a snapshot's `data: {errorEnabled: false}`, reported as `data.errorEnabled` changed. Deploying cannot fix that — the next snapshot is taken from rows that still carry the key. Confirmed against a real stuck workflow: 18 versions, both blocks reporting `data.errorEnabled`, and the same two states comparing equal with this restored. Removing the exclusion in 7934df7f88 assumed the migration could reach every copy of the value. It cannot reach a frozen snapshot, so the comparison has to keep tolerating the old key regardless of where it survives. The block field is still compared on its own, with `!!`, so the flag itself is not ignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert(deploy): stop tolerating a block-data shape that never shipped The flag's stint inside `data` began and ended on this branch: `main` and `staging` have zero mentions of `errorEnabled`, and `ci.yml` gates every deploy job on a push to main/staging/dev, so opening a PR deploys nothing. No released version ever wrote the key, which means no production row and no production deployment snapshot can hold one. That makes the exclusion permanent code apologizing for a shape that cannot reach the database it defends. Migration 0287 already strips the key from the live tables, which is where a one-time data fix belongs; in production it matches zero rows, and on a developer database it makes the next deploy write a clean snapshot. Reverts 5ece9f9e7e. That fix was correct about the mechanism and wrong about the scope: it read a local database as evidence about production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): retract bystander cards during a run, slant the sweep mark Two things about a running canvas. A run pinned every card's action bar open and suspended every card's hover on top of it, so the canvas became a wall of open swells that could neither retract nor respond to a pointer — and the `isWorkflowRunning && !isRunning` hover treatment already written for those cards was unreachable. Only the card that is actually running is pinned now; the rest behave as they do at rest, which is what makes hovering one bring its bar up again. The sweep's filled slot painted a full 24px square. It now paints a slanted band across the slot, as a hard-stop gradient rather than a `clip-path` — the two end slots already carry one for the swell silhouette and a second would have to win a specificity race with it. The stops hold `--surface-2` exactly, so only the shape changes. Each variant is spelled out because Tailwind's JIT reads literal class strings and a composed `hover-hover:${FILL}` compiles to nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): fill the running sweep one way, and tighten its mark The sweep drained back to empty after each pass, which reads as undoing the progress the block is making. It fills left to right and starts over. The direction flag goes with it — the state is just the count now. The slanted mark also sat too far off its neighbours. Slant and tightness trade against each other here: the transparent wedge has to be at least as wide as the edge's horizontal travel, or the cut clips a corner instead of crossing the slot. Leaning 7° off vertical instead of 17° travels 2.9px across the 24px slot rather than 7.3px, which brings the wedge in from 26% to 12% — 3.2px a side against 7.8px, so the gap between marks drops from ~17.6px to ~8.4px with the slant still crossing cleanly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): fill the running bar once, edge to edge The sweep restarted from empty every time it filled, so the bar kept re-running ground it had already covered. It fills left to right once and holds. The mark also sat inset in its slot, which put a gap on both sides of every join and made the row read as separate chunks instead of one bar. It now spans its slot edge to edge, leaving only the row's own `gap-[2px]` between marks, and takes its weight off vertically instead: `bg-clip-content` with symmetric padding paints a 10px band inside the 24px slot without changing the slot's size, so the swell measured around it does not move. `--surface-2` is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): paint the running bar as right-leaning uprights The fill read as a row of horizontal slabs. Each filled slot now paints one narrow upright bar leaning right, so a run fills as `/ / / /`. Geometry, since the two constraints fight: leaning the edge 15° off vertical carries it 4.3px across the bar's 16px height, so the transparent margin has to stay above 15% or the cut clips a corner instead of crossing top to bottom. 38%/62% leaves a 7px bar with room to spare. Height comes from `bg-clip-content` plus symmetric padding, which does not change the slot's own size, so the swell measured around it stays put. `--surface-2` is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): pitch the running hatch to the row, not to one bar a slot One bar per 24px button left the rest of the button empty, so the marks inherited the button grid's rhythm and sat ~19px apart — a row of isolated ticks rather than a loader. The fill repeats now, at a pitch that divides the row's own rhythm: a slot plus its `gap-[2px]` is 26px, so a 13px horizontal pitch puts exactly two bars in every slot and stays in phase across the gaps, including the 40px end slots. Bars land every 13px with a uniform 6px between them, whatever the run's length. Stops are measured along the 105° axis rather than horizontally, so they carry the `sin(105°)` factor: a 7px bar on a 13px pitch is 6.76px on a 12.56px period. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(canvas): paint the running hatch once across the row, not per slot The bars came out bunched in some places and spread in others. Per-slot backgrounds cannot avoid that: each button starts its own gradient at its own origin, so the phase resets at every slot — and the end slots are 40px against the others' 24px, so the resets are not even uniform. Three passes of tuning the stops were all chasing a constraint the approach could not satisfy. The hatch is now one element spanning the row, so there is one gradient and one phase. It sits behind the buttons and grows by width: the run/stop button keeps an opaque fill while running and masks the part growing underneath it, and every other slot is transparent mid-sweep so the hatch reads through. The slots no longer paint anything themselves, and the per-slot filled flag goes with them. `--surface-2` is unchanged; only where it is painted moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): close the running hatch's gaps The hatch ran a 50/50 duty cycle — 6px of fill to 6px of air — which read as sparse. It now runs 8px to 3px. Both stops are measured along the 105° axis rather than horizontally, so each carries a `sin(105°)` factor; the note records that, and that closing the gap further is a matter of moving the first stop toward the second. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(emcn): drop the brand highlight from popover menus (#6506) Context menus opted into a palette of their own — `variant='secondary'` for a brand-blue row highlight and `colorScheme='inverted'` for a dark card — so the canvas, block, toolbar, terminal, sidebar, and preview menus looked nothing like the menus everywhere else in the product. Removes both overrides so they inherit the same surface, border, and `--surface-active` highlight the terminal's overflow menu already uses, and drops the brand state from the Popover itself along with the `variant` prop that only ever selected it. One fewer way to style a menu. * feat(executor): opt-in per-block retry (#6505) * feat(executor): opt-in per-block retry Adds a per-block retry policy, off by default, surfaced in the editor's additional-fields disclosure alongside the block's other advanced settings. A block that opts in replays its handler while tries remain, then rethrows the final error so the error port behaves exactly as it does for a block that never retried — retrying only delays the existing outcome, never changes it. Retry is deliberately indiscriminate about the failure, since there is no reliable way to tell a transient error from a permanent one and classifying would silently do nothing for the generic errors people turn it on for. Only throws that are not failures are excluded: a deliberate stop, a child workflow whose own blocks already ran their policies, and the block types whose throw is control flow (human-in-the-loop, sentinels, subflow containers, notes, triggers). Eligibility lives in one predicate read by both the editor and the executor, so a block can never keep retrying after an edit that hides its control. `retry` is a nullable jsonb column; NULL means "runs once", which is how every existing block already behaves, so the change is inert until someone opts in. Bounds are clamped on read rather than rejected, so a value written before a bound moved still resolves to something runnable. Also decouples the additional-fields disclosure from `block.advancedMode`. That flag decides which member of a canonical pair serializes, so opening the disclosure used to be able to drop a block's configured credential. Expansion is now view state; the stored flag is no longer written by the editor. Retried blocks report their try count on the trace span, shown in log details. * fix(realtime): allow the write role to persist a block retry policy `update-retry` was added to the protocol but not to the write-role allowlist, so the editor applied the change optimistically while the server dropped it and the policy never reached the database. Adds a test asserting the write role holds every per-block operation the protocol declares, so the next block setting cannot repeat this silently. * fix(editor): keep a retry number field's value when it is blurred untouched Committing on blur normalized the draft unconditionally, and an untouched field's draft is null — which normalizes to the default. Focusing and leaving Max tries silently reset a configured 5 back to 3. * improvement(canvas): close the running fill to solid, slant its leading edge Gaps gone entirely: the bar is one solid fill now. The slant moves onto the growing edge, because a repeat with its gaps closed has no edges left to show. 4px of run across the 16px height is the same 15° lean the bars carried, so the fill still leans right — it just leans at its front instead of throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * improvement(canvas): return the running fill to the squares' rhythm, sheared Marks are 24px with the row's 2px gap after them again — the geometry the slots carried before any of this — so they land where the squares did. The shear is the only thing that is new. It stays on the single spanning element rather than going back to per-slot backgrounds: one gradient means one phase, which is what lets the 24/2 rhythm hold across the 40px end slots instead of resetting at every boundary. Stops are measured along the 105° axis rather than horizontally, so both carry a `sin(105°)` factor: 24px of mark is 23.18px of stop, and the 26px pitch is 25.11px of period. Writing 24/26 directly renders ~3.5% wide and drifts out of the squares' rhythm across the row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(canvas): trim the running fill to the swell's tapered end The fill ran off the block. The row is a rectangle but the swell is not — its last slot cuts a diagonal so the shape narrows toward the top, and a rectangular overlay therefore painted past the gray edge up there while still sitting inside it at the bottom. The per-slot version never showed this because each button's own clip contained its fill; moving the paint onto one spanning element took that containment away with it. The overlay now carries the same taper, read off that slot's own path: 16.67px in from the row's right at the overlay's top, 3.33px at its bottom, a slope of 20/24. Only applied to the swell variant, which is the shape that tapers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(canvas): stop the handoff highlight pinning a bystander's toolbar open Dropping `isWorkflowRunning` from `forceOpen` was not enough: it also read `usesSelectedVisuals`, which is `isNodeSelected || isExecutionHighlighted`, and the handoff highlight covers the block feeding the running one. So the upstream card kept its bar down for the whole run — the wall of open swells this was supposed to end, one card smaller. Those are two different questions. `usesSelectedVisuals` still drives the TREATMENT — the graphite silhouette and `data-node-selected`, so the eye can follow the baton — while whether the toolbar is pinned open now keys off selection alone. The container keeps `isRunning` by itself. Selection was never a pin there, and its own tests hold it to opening on hover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(realtime): refresh the error flag on a block upsert `BATCH_ADD_BLOCKS` wrote `errorEnabled` on insert but left it out of the conflict clause, so re-adding an existing block id kept whatever the row already held while every sibling flag — enabled, advancedMode, triggerMode, retry, locked — was refreshed from `excluded`. The client's value was silently discarded and the old error-output state came back on the next load. Mine: the insert side gained the field when the column landed and the conflict set did not. The other two block writers delete before inserting, so no stale row survives them; this upsert was the only path that merged into one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * update loader animation * change the loader --------- Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com>	2026-08-10 19:21:32 -07:00
Bill LeoutsakosandBill Leoutsakos	fb6e18f5db	refactor(tiktok): align webhook routing with shared dispatcher (#6261) ... * refactor(tiktok): route webhooks by account key * refactor(tiktok): remove custom webhook ingress jobs * fix(tiktok): constrain webhook routing migration * test(tiktok): remove redundant assertions * fix(tiktok): preserve routing backfill state * fix(tiktok): preserve legacy webhook routing during rollout * fix(tiktok): preserve webhook delivery guarantees * refactor(tiktok): remove rollout compatibility * refactor(tiktok): remove unused webhook backfill --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>	2026-08-08 15:33:33 -07:00
Waleed	ea5df41a57	fix(webhooks): resolve the AgentMail tenant before verifying the signature (#6431) ... * fix(webhooks): resolve the AgentMail tenant before verifying the signature The receiver identified the delivering workspace by trying every stored webhook secret in turn, so each request cost one HMAC over the full body per tenant. Resolve the workspace from the envelope's inbox id first and check a single secret, and make that id uniquely indexed so the one-check property holds by construction. Also gates the route through the shared admission gate, like the other public receivers, and gives it a body cap sized for an email envelope. * fix(db): declare the inbox provider index partial in the schema too The migration built a partial unique index but the Drizzle definition and snapshot omitted the predicate, so a regenerate would have diverged. Also records at the registration site why only message-bearing event types can be subscribed. * fix(db): drop the duplicate 0285 journal entry The regenerate that added the partial-index predicate appended a second entry for the same migration, so drizzle would have replayed it — dropping and rebuilding the index, leaving a window with no unique constraint.	2026-08-08 12:59:45 -07:00
Waleed	ab257555d8	fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs (#6311) ... * fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs Better Auth 1.6.23 calls the account-linking handler with trustProviderByName: false, which disables the trustedProviders allowlist for SSO entirely. Trust now comes only from the provider's domainVerified flag, which Sim never set — so any user who already had a Sim account was stranded on "account not linked". Entra never sends email_verified, so this hit every Microsoft tenant. Sim already proves domain ownership via sso_domain before a provider can be registered, so the register route mirrors that decision onto domainVerified. The column defaults to true so existing providers keep signing in across the deploy, since enabling the option turns sign-in into a hard gate. Also enforces the providerId uniqueness Better Auth already assumes: it rejects any id that exists in any tenant and resolves providers by that column alone, so a second customer picking "azure-ad" could not register at all and got an opaque 422. Sim now returns a 409 naming a free id, and a unique index makes the duplicate-row state unreachable. * fix(sso): revoke domain trust when verification is removed mid-update The create path re-checks domain ownership after Better Auth persists the provider and rolls the row back if the verified sso_domain row disappeared in that window. The update path had no equivalent, so deleting the verified domain while updateSSOProvider was in flight still set domainVerified, restoring same-email account-linking trust for a domain the org no longer proves it owns. The update path has no newly-created row to roll back, so it clears the flag instead: that denies linking and blocks sign-in on the provider until the domain is verified again. * fix(sso): make domain-trust grants atomic and propagate revocation Greptile flagged that the ownership check and the domainVerified write were separate statements, so a domain deleted between them still ended with trust granted. Two changes close it from both sides. The grant now folds the ownership test into the UPDATE's WHERE clause, so Postgres evaluates both in one statement and the write matches nothing once the proof is gone. Removing a verified domain now clears domainVerified for providers on that domain, in the same transaction as the delete. This was a standing gap, not just a race: deleting a domain previously left linking trust set indefinitely. Together the provider cannot end up trusted without current ownership in either commit order — if the grant lands first the delete clears it, and if the delete lands first the grant no-ops. * fix(sso): report a refused domain-trust grant instead of returning success The conditional grant could match zero rows if the verified domain was deleted between the pre-write check and the write. The route ignored that and returned 200, leaving a provider that cannot sign anyone in while telling the admin it saved. The grant now reports whether it matched, and that result is the single decision point on both paths: the create path rolls the provider back, the update path clears the flag, and both return SSO_DOMAIN_NOT_VERIFIED. This also drops the separate post-write ownership read, since the UPDATE re-tests ownership itself. * feat(sso): let admins map IdP claims, and trim setup comments Identity providers disagree on which claim carries each value — Entra can send the address as `upn` rather than `email` — and the mapping was hardcoded, so a mismatch had no fix in the UI at all. Adds an Attribute mapping section for both protocols, defaulting to each protocol's standard claim names shown as placeholders, so the common case still needs no input. Editing an existing provider now loads its stored mapping and only treats a value as an override when it differs from the default, so a saved custom mapping is never silently rewritten. * feat(sso): expose the standard enterprise IdP options in the setup form Rounds out the form with the options Better Auth already accepts but the UI hid, so a non-standard IdP no longer dead-ends at a field that cannot be set. SAML gains signature algorithm, digest algorithm and NameID format. Only SHA-256 and stronger are offered: Better Auth warns on SHA-1 as deprecated and rejects anything outside its secure set, so weaker choices would only produce failed saves. SAML also surfaces the SP Entity ID beside the ACS URL. IdP admins are usually handed a vendor metadata document; Sim does not publish one, and these are the two values it would carry. OIDC gains authorization, token and JWKS endpoint overrides for providers whose discovery document is incomplete or unreachable. Discovery still fills them in when they are left blank. All of these load from the stored config when editing, so re-saving a provider cannot quietly drop them. * fix(sso): withhold domain trust from personal providers on the hosted deployment A personal (org-less) provider has no verified domain behind it, but the trust grant treated it as authoritative anyway. On the multi-tenant deployment that is an account-takeover primitive: anyone able to register one could claim a domain they do not own, point it at their own IdP, and have a sign-in auto-link to an existing account on that domain. Sim's UI always registers org-scoped, so this only reaches direct API callers. Self-hosted deployments are single-tenant — the operator is the only tenant — so the org-less path keeps working there. Also clears the attribute mapping when the protocol changes: claim names are protocol-specific, so an OIDC override carried into a SAML config would save a mapping the IdP cannot resolve. * docs(sso): correct the personal-provider trust note after the hosted gating * fix(sso): drop the inert SAML algorithm selects, make NameID format clearable The signature and digest algorithm selects were placebo controls. Tracing @better-auth/sso 1.6.23, those two values are only read by validateConfigAlgorithms, mergeSAMLConfig and sanitizeProvider — createSP and createIdP never pass them to samlify, so nothing they select reaches the SAML exchange. Bugbot separately noted they could not be cleared, since Better Auth merges with `??` and omitting a key keeps the stored value. A control that neither applies nor clears should not exist, so both are removed. NameID format is genuinely wired (createSP passes it as nameIDFormat) and is kept, but is now always sent rather than omitted when set to the provider default. samlify falsy-guards the value, so an empty string reads as unset and "Provider default" can actually clear a stored override. The read-only provider view also now shows the SP Entity ID and the ACS label for SAML — admins land there after saving and need the same two values the form says their IdP requires. * docs(sso): tighten the personal-provider note to the self-host path it describes * fix(sso): forward an empty SAML NameID format so the provider default can be restored The form sends an empty identifierFormat when the admin selects "Provider default", but the route dropped it with a truthiness check. Better Auth merges SAML config with `??`, so an omitted key retains the stored value — the selection appeared to apply and silently did not. Forwarding the empty string makes it reach the merge, and samlify falsy-guards nameIDFormat, so it reads as unset. Selecting the provider default now actually clears a stored override. * fix(sso): revoke trust for providers whose domain is spelled with a wildcard Migration 0268 grandfathered providers by normalizing their domain with lower + btrim + a stripped leading `*.`, so sso_provider.domain can hold `*.acme.com` while its verified sso_domain row holds `acme.com`. The revoke on domain deletion compared the raw column, so such a provider matched nothing and kept domainVerified after its ownership proof was gone. The comparison now applies the same normalization 0268 used, so a grandfathered row is matched the way it was written. * fix(db): give the SSO index migration the concurrent-build convention it skipped packages/db/scripts/migrate.ts documents the required shape for CONCURRENTLY statements, and the previous migration follows it. 0284 did not, and the omission is silently destructive. migrate.ts sets a session lock_timeout of 5s, which survives the embedded COMMIT. CREATE INDEX CONCURRENTLY waits on every concurrent write transaction in the database — not only ones touching this table — so on a busy database the build is cancelled with 55P03 and leaves an INVALID index. The retry then replays the file, IF NOT EXISTS skips the invalid index, and DROP INDEX removes the only working index on provider_id. The migration journals as applied and exits 0 with provider_id unindexed and uniqueness unenforced, reopening the cross-tenant provider resolution this migration exists to close. Adds SET lock_timeout = 0 around the concurrent statements, a pre-drop of the target index name so a replay rebuilds rather than skips, and restores the 5s timeout afterwards. Verified by stranding an INVALID index and replaying: the end state is a valid unique index with uniqueness enforced. Also corrects the sso() comment that claimed domainVerified confines linking to matching email domains. link-account.mjs blocks on `!isTrustedProvider && !userInfo.emailVerified`, so an IdP asserting email_verified links regardless of domain — the flag narrows nothing on its own. * fix(sso): give the Enter shortcut the same guard as the Add domain button The Enter handler called handleAdd unconditionally while the button was disabled during an in-flight add, so repeated presses could issue overlapping requests. Both now read one canAddDomain flag rather than duplicating the condition. * fix(sso): stop the provider ID being editable after it is saved Renaming it was never useful and always destructive. The value forms the redirect URL registered with the identity provider, so changing it breaks sign-in until the IdP is updated. Worse, the register route selects create-vs-update by (providerId, organizationId), so a renamed id misses and registers a SECOND provider; the settings page renders providers[0], so the duplicate is invisible, there is no delete action to remove it, and existing account rows still reference the old id. Editing now shows it as a read-only copyable value, and the create form says up front that it cannot be changed later. Also hoists the suggestion list to module scope — it was rebuilding 44 objects on every keystroke anywhere in the form. * fix(sso): stop persisting generated IdP metadata so SAML cert rotation works The route stored an IdP metadata document built from cert + entryPoint even when the admin supplied none. The form loads that document back into its optional metadata field and resends it, and on the next save it wins over the certificate — so rotating a SAML signing certificate through the form appeared to succeed and changed nothing. Only metadata the admin actually pasted is persisted now. With none stored, Better Auth's createIdP builds the IdP from issuer, entryPoint and cert, which are the fields the form edits. No SAML providers exist in production, so this changes no live tenant. * fix(sso): always write SAML IdP metadata so clearing it takes effect on update Not storing generated metadata fixed new providers but not existing ones: Better Auth merges SAML config with `??`, so omitting the key let a previously stored document survive and keep overriding the certificate. The key is now always written, empty when the admin supplied none. createIdP falsy-guards it and falls back to issuer/entryPoint/cert, so clearing the field actually clears it. * fix(sso): hold the domain proof under a row lock while granting trust Two fixes from review. The trust grant folded the ownership test into the UPDATE's WHERE clause, but under READ COMMITTED the EXISTS subquery is evaluated against the statement's original snapshot. A delete committing while the UPDATE waited on the provider row could therefore still see the removed sso_domain row and grant trust after ownership was gone. The grant now selects the proof FOR SHARE inside a transaction before writing, so the delete blocks until it commits, and if the delete committed first the select finds nothing and no trust is written. Editing a SAML provider also broke on configs written by the previous commit: hydration used `config.idpMetadata?.metadata || config.idpMetadata`, and `{ metadata: '' }` is falsy at the property but truthy as an object, so an object landed in a string field and failed validation on save. It now narrows on the type and handles both the object and legacy bare-string shapes. * refactor(sso): write the two merge-sensitive SAML fields the same way idpMetadata and identifierFormat both exist to defeat Better Auth's `??` merge, which silently keeps a stored value when a key is omitted, but they were written differently — one always, one only when defined. Both are now always written, empty when unset, under one comment explaining why and noting that each is falsy-guarded downstream. Also drops a redundant saveDisabled: false; the prop already defaults to false. * fix(sso): report the row the trust grant actually matched The grant returned true once it found the proof, without checking that the provider UPDATE matched anything, so its boolean did not always mean what callers read it to mean. It now reports the matched row. * fix(sso): restore provider domain trust when a domain is re-verified * fix(sso): correct the domain-removal warning now that it disables sign-in * chore(sso): trim verbose comments * fix(sso): revert a rejected SSO update instead of leaving it stored	2026-08-06 00:52:39 -07:00
Vikhyath MondretiandSiddharth Ganesan	117fe3137b	feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift, workflow exec cancellations (#6247) ... * feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift * fix(execution): harden compatibility and secret diagnostics * fix(execution): harden generated JavaScript literals * fix(execution): align timeout cleanup semantics * fix(tables): decouple stale job cleanup * fix(execution): drain stale workflow backlog * test(sandbox): make deadline assertions timing-safe * fix(execution): lock cleanup candidate batches * fix(execution): preserve cleanup failure metrics * cancel route fixes * separate out mship template and func template * fix * fix(execution): harden secret projection and block runs * fix(workflow): validate draft execution state * run from block ui disabling * feat(copilot): expose Sim sandboxes to mothership * feat(copilot): expose sandbox capability catalog in VFS * Updates * fix legacy logs showing up * fix(copilot): keep sandbox config visible * fix model provenance issues * fix lint' * more lint * more * test(files): align provenance copy query order * consolidate migrations, rollout compat * integration projections * update skills * fix * add provenance linters * fix: address review and compatibility regressions * fix: make tool boundary audit Bun 1.3 compatible --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com>	2026-08-05 19:22:04 -07:00
Waleed	5dbe95e0c1	fix(files): reserve image layout space so images stop reflowing on load (#6299) ... * fix(files): reserve image layout space so images stop reflowing on load A markdown image with no stored dimensions reserved zero vertical space until it downloaded, then snapped to its natural height and pushed content below it down (cumulative layout shift). Reserve the box up front from the image's intrinsic aspect ratio instead. Store intrinsic width/height as workspace_file metadata (not in the markdown — it stays clean ``), read it synchronously from the already-loaded file list to reserve a responsive aspect-ratio box on first render, and lazily backfill it once per image on first view via a write-gated, idempotent PATCH. The node view falls back to on-load measurement for the first-ever view and for external images. Images stay fluid (max-width:100%, height:auto). * fix(files): address review — reserve on stale memo, clear dims on content swap - onLoad guards on the memoized storedDimensions the render uses (not a fresh cache read), so a sibling's non-reactive backfill can't leave a view unreserved. - updateWorkspaceFileContent clears width/height when it swaps bytes, so stale dimensions can't be reserved for new content (and the null re-enables backfill). - Keep optimistically-cached dimensions when the PATCH fails (correct measurement; a 403/transient error shouldn't wipe sibling reservations). - Test imports the sibling via the absolute @/ path. * fix(files): re-derive image dimensions on content swap instead of clearing Clearing width/height to NULL on a content swap reopened the width IS NULL backfill path, so a late fire-and-forget PATCH for the previous image could write its stale size onto the new content. Instead, measure the new bytes' intrinsic dimensions server-side (image-size, headers only) and store those (or null for a non-image), so the row always matches the current content and a stale backfill can't apply. * fix(files): self-heal image dimensions from the browser instead of server-measuring Round-3 review: server-side image-size returns raw (non-EXIF) dimensions, and clearing dims on content swap reopened the stale-PATCH race for non-image or unmeasurable content. Move authority to the browser's own naturalWidth/Height (EXIF-correct): the node view reserves from it and reports on any mismatch, and updateWorkspaceFileDimensions overwrites (no width IS NULL gate) so stale values self-correct on the next view. Reverts the server-side measurement and the content-swap dimension touch entirely. * fix(files): clear image dimensions on content swap (completes self-heal) The self-heal rework left the old image's dimensions in the row after a content replacement, so the next view of the new bytes reserved a wrong-sized box before correcting. Clear width/height on the content-swap write so the row never describes stale content: the next view falls back to the baseline first-load reflow and the browser's measurement backfills the correct size. No server-side decode (EXIF-safe), and the client's overwrite-on-mismatch handles a late PATCH. * fix(files): guard dimension writes by content key so a stale PATCH can't persist Ties the dimensions write to the storage key the client measured. The key is regenerated on every content replacement, so an in-flight PATCH measured against superseded bytes is rejected at the DB (WHERE key = measured key) instead of persisting the old aspect ratio for new content. Closes the last stale-ordering window Greptile flagged — the write is now content-version-conditioned, not just corrected on the next render. * chore(files): fix stale route TSDoc and hoist a regex literal (cleanup pass) Post-review /cleanup: the dimensions route TSDoc still described backfill-once behavior (now overwrite-on-mismatch via the content-key CAS); the bare-pixel width regex is hoisted to module scope. No behavior change. * fix(files): reflect the content-version guard outcome in the dimensions response The route returned success:true even when updateWorkspaceFileDimensions matched 0 rows (the CAS rejected a write whose measured key no longer matches the row). Return success:<whether a row was written> and widen the contract response to { success: boolean }. Not an error path — the client's next measurement persists once its file list has the new key; this just stops the API claiming a persist that did not happen. * fix(files): reconcile the cache when a dimension write is content-version-rejected Previously the client discarded a success:false (CAS-rejected) response, leaving its optimistic patch — which is for superseded bytes — lingering in the file-list cache. On rejection, invalidate the list so the cache reconciles with the new content (whose real size persists on its next load). Deliberately NOT a retry: re-sending the old measurement under the new key would write the wrong size. A transport error / read-only 403 still keeps the optimistic value (it's the real displayed size). * docs(files): align stale dimension docs with the overwrite/self-heal behavior Cleanup audit: the ImageDimensionsSource/reportImageDimensions interface docs and one route log string still said backfill-once/no-op; the mechanism overwrites on mismatch to self-correct. Wording only, no behavior change.	2026-08-05 17:53:41 -07:00
Waleed	0bc4fb4656	fix(security): meter and throttle the deployed-chat TTS relay (#6212) ... * fix(security): meter and throttle the deployed-chat TTS relay POST /api/proxy/tts/stream treated "a live public chat exists" as authorization to spend the platform ElevenLabs key. A public chat id is handed to every visitor, so any anonymous caller could synthesize speech with no length cap, no rate limit and no usage accounting. Bring the relay in line with its STT sibling (/api/speech/token): - Resolve the chat's workspace and bill synthesized characters to that payer via a new `voice-output` usage source, so spend is attributable and counts against the plan's usage limit (402 once exceeded). - Throttle per IP before any database work, and per chat afterwards, to bound both one caller hammering many chats and many callers hammering one chat. - Cap `text` at 2000 characters and allowlist `voiceId`/`modelId`, so the caller can no longer choose an unbounded charge, a premium or cloned voice, or the billing model. - Drop `Access-Control-Allow-Origin: *`, which let any third-party page read the audio; deployed chat and the Office embed are same-origin. * fix(security): correct TTS metering, pricing and body cap Follow-up review of the previous commit found five defects in it: - Usage rows collided. `usage_log.event_key` is unique and inserts are conflict-do-nothing, and the key is derived from the entry's stable fields. With no explicit sourceReference, two synthesis calls of equal character count in the same workspace produced the same key, so every repeat length went unbilled — defeating the metering this change is for. Each call now carries a unique sourceReference. - Priced at $0.10 per 1k characters, twice the published ElevenLabs Flash/Turbo rate of $0.05, which would have overcharged customers 2x. - No body cap, so an anonymous caller could make the route buffer up to the shared 50 MB default before validation. Now 16 KB, as the STT sibling does. - Threshold settlement ran per sentence: several queries and a possible Stripe call on a realtime path. The workflow execution that produced the text already settles the payer. - The per-IP bucket was described as preventing database amplification. getClientIp trusts the leftmost X-Forwarded-For, so an attacker rotates past it; the comment now says the per-chat bucket is load-bearing. * refactor(chat): share the deployed-chat auth gate across voice routes Review of the previous commits surfaced duplication and one more gap: - The TTS and STT routes had grown near-identical copies of the chat auth + payer lookup. Extracted to resolveDeployedChatCaller, so the gate and the payer resolve together and cannot drift per route — that duplication is how the unmetered TTS path shipped in the first place. - Neither copy filtered chat.archivedAt, so an archived chat could still authorize spend against its former owner's workspace. The shared lookup now filters it, fixing both routes at once. Note: not covered by a test — the db chain mock does not evaluate WHERE clauses, so an assertion here could not fail. - Replaced the route's hand-rolled 429 builder with the existing enforceIpRateLimit helper, and added enforceChatRateLimit alongside the per-user/IP/workspace helpers. Gains the standard Retry-After and X-RateLimit-Reset headers plus throttle logging. - Dropped a test that asserted a module the route no longer imports was never called: it could not fail. - Narrowed the contract: unexported the single-use allowlists and dropped .passthrough() now that the body is a closed shape. * fix(security): fail closed when voice-output usage cannot be recorded Review round 1 findings: - A ledger write failure previously logged and streamed the audio anyway, leaving the spend unrecorded and the payer's usage understated. The caller is anonymous, so serving audio we could not charge for is the unmetered spend this route exists to prevent — it now returns 500. - Use generateId() from @sim/utils/id rather than crypto.randomUUID, per the AGENTS.md ID rule. generateId returns a full UUID v4, so the per-call uniqueness the usage_log event_key depends on is unchanged. * fix(chat): split long TTS text so the relay cap cannot drop audio The client sentence-splits on Western `.!?` only, so text that never matches — CJK punctuation, or a list with no terminal punctuation — accumulates and is flushed as one block at the end of the stream. Against the new 2000-character relay cap that block is rejected and the whole message plays no audio, a regression introduced by adding the cap. Split to cap-sized pieces at the single point that enqueues synthesis, so both the per-sentence path and the end-of-stream flush are covered. Prefers a whitespace or CJK punctuation boundary, falling back to a hard cut when a block has none. The server cap stays as the enforcement point. * fix(security): release the vendor stream when metering rejects the request The fail-closed branch returned 500 with the ElevenLabs response body still open, so synthesis and download kept consuming vendor and runtime resources for a caller that was already rejected. Cancel it before returning, and assert the cancellation in the test.	2026-08-03 11:38:46 -07:00
Vikhyath Mondreti	b8ec114382	improvement(chat): secrets mounting / exposure improvements and controls (#6191) ... * fix(copilot): secrets injection into sandbox * improvement(chat): secrets mounting / exposure improvements and controls * fix(secrets): simplify copilot mounting flow * test(secrets): preserve standard tool permissions * fix(copilot): bind workflow tool completions * fix(secrets): preserve own environment keys * fix(copilot): release failed workflow claims * fix(copilot): trust compacted workflow completion	2026-08-02 10:21:27 -07:00
Waleedandmzxchandra	10bfb5d139	feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991) ... * feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on b8f28b04b: - Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to avoid clobbering a local in-progress resize), so refetching the definition on a peer never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not a no-op refetch. Structural changes still propagate via 'schema'. - Overlay now also observes the content layer with the ResizeObserver, so a column resize (which grows the content, not the scroll container) re-measures remote outlines. - Presence-merge comment now states both sides of the trade-off. * fix(tables): re-broadcast local selection on (re)join Cursor Medium: a selection made before the room join completes (or held across a reconnect) was dropped server-side and never re-sent, so peers didn't see it until the local user moved it again. Track the current selection in a ref (set on every emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is joined. * fix(tables): re-broadcast selection when a peer's row change shifts it End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable (rowId, columnId) only on selection/editing change, not when a live edit/schema refetch inserted/deleted/reordered rows. The index-based local selection then sat on a different logical row than the rowId peers held, so your outline showed on the old row until you moved. Re-run the emit on rows/displayColumns change and dedup an unchanged result (also drops the redundant null-on-open emit) so the broadcast stays consistent with the local highlight. * fix(tables): schema invalidates run-state/enrichment + guard stale join Cursor round on cdc8796b8 (2 Medium): - schema handler used detail exact:true, so it skipped the activeDispatches + enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a prefix match. After a peer deletes/restructures a workflow group, peers could keep a stale running badge or enrichment panel. Now invalidates both siblings too (rows stay on the debounce). - Guard against a stale join stealing the room: a fast table A->B switch could let A's async authorize finish after B, leave B, and strand the socket in A. Added a per-socket monotonic join generation checked after authorize (mirrors the file-doc relay's guard) + a test. * feat(tables): live column width/pin/order sync Collaborators now see each other's column resizes, pins, and reorders live — the last piece of Google-Sheets-style layout parity. - New lightweight `metadata` durable event kind (distinct from `schema`): only the table definition carries UI metadata, so peers refetch the definition alone — no rows/run-state refetch. The metadata PUT route now signals it. - The grid reconciles server metadata against its in-progress gesture: the column being actively resized keeps its live local width, and an in-flight column drag blocks a reorder apply — so a peer's change never reverts the local action. Each field is reference-guarded (React Query structural sharing keeps unchanged sub-objects stable), so an unrelated peer change doesn't re-apply the others. * fix(tables): escalate to schema signal when a reorder scrubs group deps Independent audit of the metadata-sync commit found a stale-run-state hole: a columnOrder PUT that moves a column left of a workflow group's leftmost column makes updateTableMetadata scrub that group's dependencies and write a new schema — a real structural change. But the route only fired the lightweight 'metadata' signal (detail-only refetch), so peers' and the actor's activeDispatches / enrichmentDetails queries stayed stale (a lingering running badge / enrichment panel) — exactly what the 'schema' handler exists to prevent. updateTableMetadata now reports whether it scrubbed the schema; the route emits signalTableSchemaChanged in that case and the light signalTableMetadataChanged otherwise. Width/pin/plain-reorder stay on the cheap detail-only path. * feat(realtime): accurate in-file presence + collaborative-caret polish (#5965) Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests. * feat(files): smarter bullet delete/indent and untitled-file title sync (#5971) * improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption Backspace at the start of a list item now outdents a nested item or clears a top-level item to a paragraph in place instead of deleting the row and jumping the caret to the previous block; Enter on an empty nested item outdents. Empty non-trailing top-level items still collapse cleanly since they cannot round-trip as a lifted paragraph. Also strips nested empty list-item marker lines on serialize: a nested empty bullet re-parsed as a Setext heading underline, silently turning its parent line into an H2 and dropping the bullet. Top-level empty items are preserved. * feat(files): sync an untitled file's name with its leading heading While a file is still named untitled(.md), typing a leading heading auto-renames the file after it (debounced), and renaming the file first seeds a leading H1 from the new name. One-shot: coupling stops once the file has a real name, and the heading seed always prepends so existing content is never clobbered. * fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace Addresses review findings on the list Backspace logic: - Emptiness now uses the caret block's content.size (counts inline images/mentions), not textContent, so a bullet holding only a non-text atom is no longer treated as empty and deleted. - An empty first block whose item has sibling blocks removes only that block instead of lifting the whole item out of the list. * fix(files): preserve the untitled to named heading seed across a rename during editor load The parent captures the file name at mount (before content/session finish loading) and passes it as the transition baseline, so a rename that lands in the loading window is still seen as an untitled to named transition and the leading heading seed is not skipped. * fix(files): drop the name-to-heading seed, keep title sync one-way Removes the effect that inserted a leading H1 when an untitled file was renamed. On the collaborative Files page every open client observed the untitled-to-named transition and inserted into the shared doc, producing duplicate headings; it could also re-insert a heading a user had just deleted while a rename was in flight. Seeding document content from an async rename transition is the wrong model on a shared editor. The primary direction — typing a leading heading renames a still-untitled file — is unaffected (it never mutates the doc). * fix(files): keep empty lines between paragraphs on reload The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds from runs of blank lines — a saved visual blank line silently vanished on the next load (the settle/reopen re-seed goes through the chunker). The whole-document parser preserves them, but whether a gap yields an empty paragraph is a global, block-type- dependent decision (kept between two paragraphs, dropped after a heading), so it can't be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes; ordinary single-blank-line separation still takes the fast chunked path. Adds a suite asserting chunked output matches the whole-document parser for leading/trailing/between gaps and around lists/headings. * fix(files): only auto-name an untitled file when the user can edit The debounced untitled→filename hook ran on every onUpdate — including the mount-time seed and for view-only viewers — without checking edit permission, so a read-only user could schedule a rename they have no permission to make (a spurious, server-rejected write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready, the same signal the autosave path uses), at both schedule and fire time. * fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry - markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic \r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked, dropping empties. Normalize line endings once up front so the routing guards, the chunker, and the parser all see the same \n. +CRLF/CR test cases. - keymap: Enter on an empty first block of a multi-block item now removes only that block (removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks case — the trailing check no longer swallows multi-block items. +test. * fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip) A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness) surfaced these: - HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines → serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are unaffected. +regression tests. - Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits only; false for non-collab surfaces). - Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty item DIRECTLY under a shallower parent line — matching the function's own docstring intent. Probe-verified. +test. - Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling removed during review. * fix(files): a remote edit must not cancel the local rename debounce The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so a peer's edit arriving within the 600ms window cancelled the local user's pending rename. Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it. * docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip The stacked trailing-empty-paragraph strip made the older comment overstate a correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks, and advertise dead CRLF handling. Reword to match what the code actually does. --------- Co-authored-by: Waleed Latif <walif6@gmail.com> * fix(realtime): access-revalidation multi-room safety + cleanup pass Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the access-revalidation sweep (staging's workflow-only #5917) treated every entry in socket.rooms as a workflow id, but the generalized multi-room model puts namespaced files/tables/file-doc rooms on the same io. It would resolve those as bogus workflows, get null, and evict files/tables collaborators every ~30s. collectScanTargets now decodes each room name with parseRoomName and sweeps only workflow rooms; added a regression test and fixed the now-false TSDoc. Other audit fixes (all behavior-preserving): - workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated from avatar.ts) - PresenceAvatars: mr-1 was baked into the shared component, silently adding a margin to the workflow sidebar stack; moved to an optional layout className, re-applied on the tables/file-doc header surfaces only - table DELETE routes only signal collaborators when rows were actually removed (matches PUT) - events.ts definition kind: drop the never-emitted reason:'schema', fix its doc - event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis success path too, not just the memory fallback) - remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals in handleWorkflowDeletion; gate the table columnIndexById map on remote selections; move file-doc module TSDoc off the FileDocOwner interface; fix a stale @returns * fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps Validated each issue with subagents before implementing the cleanest fix. - tables LEAVE in-flight-join race (B8): the table handler tracked no current-table intent, so an unscoped/same-table leave during an in-flight authorize left the socket stranded in the room (present in the roster, broadcasting a ghost until disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that advances joinGeneration to cancel the racing join. + 3 regression tests. - v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update. Add the signals at those call sites (add-only, matching the existing route seam) — never in the service, so execution writes can't double-emit. Sync-only for copilot bulk ops, guarded on affected/deleted count; async job branches stay covered by their kind:'job' events; create/delete/get untouched. - table join read consolidation (B5): sweepStalePresence returns its roster so the same-tab dedup reuses it instead of a second getRoomUsers. - shared authorize slice (B6): extract only the guard-safe authorize->allowed branch into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays inline — file-doc's generation capture sits mid-ladder and must not move). - resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps local widths leading until the width PUT settles. - embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the embedded surface stops broadcasting cell selections the server drops. * fix(tables): close two copilot live-collab signal gaps + harden presence sweep Follow-ups from a comprehensive review of the branch: - copilot batch_update_rows and import_file's inline append branch wrote rows but emitted no live-collab signal, so collaborators didn't see those edits live (the append's sibling replace branch already signalled). Add the guarded signal to both, matching the internal route. - sweepStalePresence now reads the roster before the fetchSockets liveness probe and returns it on a probe failure, so same-tab dedup still runs during a transient fetchSockets outage instead of being skipped. - reword an internal comment off the retired "mothership" term. * fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup Review round on #5991: - Table join re-checked the generation only once after authorize, then awaited leave/sweep/avatar before joining + registering presence. A table switch or leave in that window stranded the socket in the wrong room, and the failure catch could tear down a newer successful join. Resolve the avatar up-front, re-check generation immediately before the membership commit (matching the file-doc join), and skip the rollback/error for a superseded join. + a post-authorize-window regression test. - access-revalidation cleanup treated removeUserFromRoom's no-op false as a transport failure and re-enqueued a still-connected socket forever. Only retry when the socket is still mapped to the room (a healthy null mapping means the entry is already gone). Repurposed the expired-mapping test to lock it. * fix(realtime): guard table join leave-prior against superseding join Round 2 on #5991: a superseded join's leave-prior could still run — during its getRoomForSocket await a newer join commits to its room, so currentRoom is that newer room and the superseded join would leave/remove/broadcast it before the final guard aborts. Re-check the generation immediately after the lookup await, before the leave mutation. Extended the post-authorize-window test to assert the superseded join never tears down the newer join's room. * fix(realtime): roll back a table join superseded during addUserToRoom Round 3 on #5991: after the final generation guard, A could join + register presence while a newer join B commits to its room during addUserToRoom's await — B's leave-prior can't observe A's half-written entry, so A's late write wins and strands the socket. Re-check after addUserToRoom and roll back A's own Socket.IO join + presence (scoped to A's room, never touching B). + a regression test hanging addUserToRoom mid-commit. * refactor(realtime): DRY table-join supersession guards; fix stale comment Cleanliness pass after the review rounds (no behavior change): - Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected` checks into a named `superseded()` helper (the catch keeps its intentionally narrower check). - Remove a stale guard comment that was left stranded above the avatar resolve. - Document the best-effort rollback catch. * fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost Two Cursor findings on the file-doc client-id ownership rebind: - On a document switch, the prior room was left BEFORE the ownership check, so a CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining the new one (contradicting its own comment). Run the ownership check first, and leave the previous doc only once the rebind is guaranteed to succeed. - Reclaiming a client id removed the stale prior socket from owners + awareness only; its socketToRoomName + Socket.IO membership remained, and handleMessage's SYNC path gates on socketToRoomName (not owners), so it stayed able to write document frames until disconnect. Fully evict the reclaimed socket. + 2 tests. * refactor(realtime): serialize table join/leave to fix map-corruption at the root Round 4 on #5991 surfaced a race the generation guards structurally cannot fix: two concurrent joins for one socket race on the single-valued socket→room map — a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry to A, and the rollback then wipes it, stranding the socket (map empty while it holds table B). Guards protect JS suspension points; they can't stop an in-flight Redis write from landing late. Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket promise chain so their multi-step async Redis commits can never interleave — restoring the atomic-commit property the synchronous sibling handlers get for free. This DELETES the leave-prior guard and the post-commit rollback (the code that caused the bug); four generation guards collapse to two identical superseded() checks (skip a superseded queued op + one pre-commit check). Reworked the interleaving-specific tests into a fast-switch-skips-superseded test; the leave- cancels-join tests are unchanged. Local to the tables handler — no shared-infra change. * fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim Two review findings: - Table join: the catch skipped rollback when superseded, but a socket.join that landed before addUserToRoom threw leaves the socket in the Socket.IO room with no matching socket->room map entry — unreclaimable by any later op (cleanup keys off the map). Under serialization the skip is unnecessary (the newer op hasn't committed), so always roll back. Simpler + fixes the strand. - File-doc reclaim: fully evicting the prior socket didn't release the seeder role if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set) never re-elected and an unseeded doc stayed empty until the deadline. Clear the role on eviction so the join's election picks a new seeder. + 2 regression tests. * fix(realtime): close revoke-race ghost presence in workflow join An access-revalidation revoke landing between socket.join and addUserToRoom socketsLeaves the socket while its presence mapping does not yet exist, so cleanupEvictedSocket finds nothing to remove and the join then writes presence for a socket already out of the room — a ghost collaborator until the stale sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth check so the whole re-auth -> socket.join -> addUserToRoom section is await-free, matching the invariant the handler already relies on for the pre-join re-auth. + ordering regression test. * refactor(realtime): serialize workflow join/leave; drop dead room-authz limb Comprehensive independent audit follow-ups: - workflow.ts join/leave now use the same opChain + joinGeneration serialization as the sibling handlers (tables, file-doc, workspace-files). It was the only async presence path left unserialized, so a rapid workflow switch A->B (or a leave racing an in-flight join) could strand presence in room A — a ghost collaborator still receiving A's operation broadcasts until disconnect. The join now aborts a superseded op at start and again right before the membership commit, and the catch always rolls back a partial join. - leave-workflow drops the '&& session' gate: an idle user whose 1h session key expired (while the 24h room mapping is still live) can now leave cleanly instead of being stranded until disconnect. The room ref alone suffices. - authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its getActiveWorkflowContext import. Workflow authorizes through its own path and never flows through authorizeRoom; the map now honestly covers only the workspace-scoped types (files, file-doc, table). - Remove unused isSameRoom (zero callers) and a needless useMemo in PresenceAvatars (plain derivation, single copy). - Tests: 4 workflow serialization/leave regressions. All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol tests, biome, api-validation, boundaries, prune 14/25. * fix(realtime): align session TTL, harden committed joins from post-success rollback Per-module comprehensive audit follow-ups: - redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room set outlived the session, and since getRoomForSocket reads the room set while the workflow handlers gate edits/presence on `room && session`, an active-but-idle collaborator got wedged into 'session expired' after 1h — sticky until reload (activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it). Both keys refresh together, so they now expire together (restores the pre-refactor consistency, where both shared one TTL). - workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live collaborator after success was already acked). - workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow re-join no longer leave→re-adds and flickers peers' presence), and the join ack is liveness-filtered via filterVisiblePresence — both for parity with the tables handler. - platform-authz: honest docstring + 400 message for the workspace-scoped-only authorizeRoom map (workflow authorizes via its own path). - caret-presence: corrected an over-stated batching comment. - +1 workflow regression test (post-success failure keeps the user joined). Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast Two Cursor findings on the prior audit-fix commit: - The 'committed' guard in join-workflow/join-table was too broad: a failure BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit 'if (committed) return' and emitted neither success nor error, hanging the client while it sat in the room. Replaced with the narrower shape: only the purely-decorative post-success steps (peer broadcast + log-metric) are wrapped best-effort; anything before the success ack still rolls back and surfaces a retryable error, so the client retries instead of hanging — while the original goal (a benign broadcast/metric blip never kicking a live, acked user) holds. - broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis transport error to []. On a disconnect broadcast that emitted an empty roster and cleared every remaining collaborator's presence until the next healthy update. Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the broadcast on a read failure (getRoomUsers keeps its swallow contract). - Tests: pre-success failure rolls back + retryable error (no hang); post-success failure keeps the user joined. Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune. * fix(realtime): harden seeder recovery, join-generation, and misc robustness Final line-by-line audit follow-ups (all LOW/MED, no P0/P1): - file-doc: a sole client whose seed FETCH fails was added to triedSeeders, re-election found nobody, and the document stayed permanently empty until reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before giving up. Also bound clientId to a non-negative integer (it is an ownership key). - tables + workspace-files: validate the room id BEFORE advancing joinGeneration, so a malformed/rejected join can't cancel a legitimate in-flight join. - workflow + tables + workspace-files: suppress the client-facing join error when the op was already superseded (a retryable error naming the abandoned room could make a client re-join and cancel its newer join). The rollback still runs. - redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it on failure) so isReady() can't report ready while the Lua SHAs are null. - connection: apply the presence-bearing filter to the manager-removed set too (symmetry with the fallback path). - http.ts: validate workflowId on the four workflow endpoints (matching the files one). - client: clear a pending join-retry timer before rescheduling (reconnect churn no longer orphans a stray extra join); clear caret fade timers on plugin destroy; seed-effect cleanup reports NOT-ready (safe direction). - Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip, presenceEventName. Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): gate isReady() on loaded script SHAs, not just connection Follow-up to the prior isConnected change, which was incomplete: the redis client's 'ready' event flips isConnected=true on connect — before initialize() loads the Lua scripts — so a bare isConnected check reports ready while removeUserFromRoom / updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs too, so the POST endpoints return a retryable 503 during that startup window instead of proceeding against unloaded scripts. Standard readiness-probe discipline. * fix(files): offline read-only fallback when the realtime doc never syncs When the realtime server is unreachable (offline, server down, socket never connects), the collaborative editor would sit blank and read-only forever — content only arrives via provider sync events that never fire. Add a bounded connect-deadline to the Yjs provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a synthetic non-retryable join-error — the exact path a real fatal rejection already uses, which seeds the file's stored content read-only. Latching fatal also stops a late reconnect from syncing server state in and merge-duplicating the locally-seeded content (the documented Yjs 'non-empty doc ignores initial value' gotcha). Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap can't run the markdown->Yjs conversion server-side (Collaboration extension errors under jsdom), and a durable binary snapshot would create a dual source of truth with the markdown file (edited by copilot / PUT / download). The client-seeder + bounded re-election is the correct architecture for a markdown-is-truth model; this closes its one real user-facing gap without persistence, a migration, or dual-truth. Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests. Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification. * fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target Two Cursor findings: - join-workflow bumped joinGeneration before validating workflowId (unlike tables / workspace-files, which I'd already fixed). A malformed/empty join could advance the counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test. - The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in client-id reclaim), that binding still points at the socket's PRIOR, valid document — so the rollback tore down a document the socket was validly in. Only run that cleanup when the binding already points at THIS join's target; otherwise the socket never registered as an owner here and the only leftover is a freshly-created empty room, dropped by destroyRoomIfIdle. Gates: realtime tsc 0, 209 tests, biome, boundaries, prune. * fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset Final safety-audit findings: - CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect deadline latched fatal and the editor fell back to a read-only local seed, a late SyncStep2 (slow server / flaky network / deploy) was still applied — merging server state into the seeded doc (content duplication) and flipping synced=true, which un-gated autosave and would persist the duplicate to the real file. fatal guarded (re)join but not inbound sync. Now handleMessage returns early when fatal. +test (late SyncStep2 after the deadline is ignored, doc stays empty + gated). - file-doc join catch now suppresses the client-facing error when superseded (matches workflow/tables/workspace-files) so a retryable error for an abandoned file can't make a client re-join and cancel the newer one. - table/workspace-files room hooks reset the retry budget on (re)connect so a prior full exhaustion doesn't block retries after a reconnect. - presence-visibility: corrected a stale TTL comment. Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune. * feat(collab-doc): server-authoritative Yjs seeding (#6008) Server-authoritative Yjs seeding for collaborative file documents: the realtime relay fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and applies it once per room, replacing the client-seeder election/handshake entirely. - DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc) reusing the client markdown engine for parity by construction - Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the client readiness deadline, guard-release for join-driven retry, read-only fallback on persistent failure - Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone build (serverExternalPackages + outputFileTracingIncludes) Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C. * feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation) buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them. Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the current state and applies the returned diff. Tested incl. concurrent-edit no-clobber. * feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff, applies it (relaying to every editor, reconciled with concurrent user edits), and reports 'no-live-room' so the caller falls back to a direct file write when nothing is open. - Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a shared POST helper + fetchFileDocSeed/fetchFileDocMerge - POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied } - Tests for the seeded-room merge relay and the no-live-room fallback * feat(collab-doc): stream copilot edits into open editors (Stage C) edit_content now, after its durable file write, best-effort merges the same markdown into the file's live collaborative document (markdown files only). If a collaborator has it open, the edit streams into their editor as a CRDT merge — reconciled with their concurrent typing — instead of the file silently changing under them; the editor's existing autosave mirrors the merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit. * fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown - buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as the seed does. Copilot passes full-file content, so without this the frontmatter merged into the doc as editor content and autosave wrote it back over the file (corruption). - Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the collaborative editor never renders. * fix(collab-doc): make frontmatter collaborative so a merge can't revert it The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge (which triggers an autosave with no user action) could write stale YAML back over a copilot frontmatter change — silently dropping it. Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed stores it, the merge updates it (only when it actually changed, preserving the no-op diff), and the editor re-attaches THAT value on save — falling back to the locked copy before the seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it. * fix(collab-doc): re-sync draft on a frontmatter-only merge A server edit that changes only the frontmatter updates the config map but not the body fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time frontmatter, and an explicit save could revert the live change. Observe the config map and, on a frontmatter-only change, re-attach the new frontmatter to the current body and push a fresh draft (guarded on a synced body so it never races the seed's own onUpdate). * fix(collab-doc): order the apply-edit/merge timeouts (outer > inner) The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge timeout (8s), so the outer call could abort while the relay was still merging — the relay then applied the merge after edit_content had returned, racing a follow-on edit. Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s so it always outlives the inner merge. Cross-referenced in comments to prevent drift. * fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races) A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core patterns are idiomatic) surfaced these real issues: - The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it. - The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness) living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in @sim/realtime-protocol with a test asserting the ordering; both apps import it. - The copilot live-merge gate used an extension-only check, but the editor treats any text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware isMarkdownFile mirroring the client, so those files stream too. - applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each diff the same stale snapshot and apply out of order. Serialize per file via a promise chain. - Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift). - Add .max() bounds to the merge contract body; document the best-effort-merge failure window honestly (open editor + merge failure can drop a copilot edit until reload — closed by the deferred durable-doc work). * feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence Make collaborative file-doc editing correct across multiple ECS tasks (the per-process Y.Doc previously assumed one replica per file). - Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each file's stream is the ordered, replayable log of updates; a multiplexed XREAD tailer converges every task's in-memory doc. Coordinated single-seeder election (SET NX + empty-stream recheck) fixes split-brain seeding. - Doc-sync fans out to local clients + the stream; awareness stays on the Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries. - Server-side persistence: project the live doc back to markdown via a new /api/internal/file-doc/persist endpoint, debounced during editing and flushed on last-disconnect, from the authoritative stream state. Collaborative editors no longer client-autosave, closing the copilot clobber-window. - Copilot merges apply through the stream (reach the live doc on any task) and serialize cross-task via a Redis merge lock. - Degrades to the original single-replica behavior when REDIS_URL is unset. * fix(collab-doc): harden distributed locks and durability from review Address Greptile + Cursor review of the multi-replica backend: - Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never merges against a shared base while a peer holds the lock) and AWAIT the stream write before releasing, so the next task never diffs a stale base. - Distributed locks (seed/merge/compact) now use ownership tokens + a compare-and-delete release (Lua), so a lock that expired and was re-acquired by another task is never stolen; acquisition fails CLOSED on Redis error. - Seed: publish the seed to the stream AWAITED under the lock before releasing, so a later seeder's empty-stream fence always sees it (closes the fence's publish-after-release gap); TTL kept at the readiness deadline. - Persist: persist the AUTHORITATIVE stream state even when this task's local doc was never seeded, and capture the local fallback synchronously so a last-disconnect flush never encodes an already-destroyed doc. - Publish: retry a transient xAdd failure so a Redis blip can't silently drop an edit from the shared log. * fix(collab-doc): only persist a doc a user actually edited Cursor review: a copilot durable write landing while a doc is being seeded could have the stale seed projected back over it on last-disconnect, clobbering the copilot edit even with no user changes. Gate server-side persistence on a genuine user edit (socket-origin update): a seed-only or merge-only doc is never projected back to the file (copilot writes the file durably itself), so it can't clobber a concurrent external write. * fix(collab-doc): close review-round race/durability gaps Address Greptile P1 + Cursor findings: - Seed publishes to the shared stream AWAITED *before* seeding the local doc, so a publish failure leaves the doc unseeded and the stream empty for a clean retry rather than serving an unpublished local seed a peer would re-seed over (split-brain). - flushPersist falls back to the synchronously-captured local snapshot when getStreamState throws (not only when it returns null), so a transient Redis read no longer drops the final durable write as the room is torn down. - streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no longer let the seed fence pass and double-seed. - Client collabReady initializes from the collaborative prop, so a collaborative editor never has a mount-window where client autosave could clobber the server write. * fix(collab-doc): unstick seed retry and persist tailed edits Cursor review: - ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the room unseeded with no retry. - Persistence dirty-tracking now marks a doc edited on any post-seed update, including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a seededObserved flag. The last task to leave persists real edits even if it only tailed them; the seed transition itself is still never counted, so a seeded-but-unedited doc is never projected back over the file. * fix(collab-doc): match client markdown post-processing on server persist Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only, but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter. Server persist could therefore write markdown differing from a client save (empty list markers, callout un-escaping) — spurious blob churn / round-trip drift despite the byte-identical claim. Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is byte-identical to a client save and the client's dirty-check baseline. Add a regression test guarding the composition. * fix(collab-doc): close durability gaps at deploy boundaries + audit polish From a comprehensive from-scratch audit (correctness, SOTA, cleanliness, feature-completeness): - Persist max-wait: a continuous edit burst kept resetting the 5s debounce and never persisted; cap it so a burst flushes at least every 20s, bounding unpersisted edits. - Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling deploy / scale-in secures open edited rooms to durable markdown before exit, instead of relying on the stream + a surviving task. - Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a snapshot folds seed+edits into one frame, so a task catching up purely from it no longer treats real edits as an unedited seed and skips persisting. - Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer object-identity guard; fix stale comments (seed route, edit-content autosave). * improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle - collab editor no longer latches a spurious 'Unsaved changes' prompt: report dirty only when the client owns durability (canAutosave), since in a collaborative session the relay persists the doc server-side - guard shutdown against a double SIGINT/SIGTERM running teardown twice - disconnect local sockets before httpServer.close so shutdown exits gracefully instead of hitting the forced-exit timer (local-only, deploy-safe) - return 400 (not silent 200) on an invalid workspaceId in the files-changed fanout - clear a pending join-retry timer on reconnect so it can't fire a duplicate join * fix(realtime): make collab-doc seeding atomic to close split-brain window An adversarial concurrency audit found a split-brain double-seed vector: the seed used an advisory SET NX PX lock + a SEPARATE xLen fence + an unconditional xAdd (a non-atomic check-then-append). If the seed lock's TTL expired mid-seed (a >4s stall after the 8s seed fetch), a second task could acquire the freed lock over a still-empty stream, both fences read empty, and both append seeds with distinct Yjs client ids -> duplicated document content. - add an atomic SEED_IF_EMPTY_SCRIPT (append-iff-empty in one Redis step) + store.seedIfEmpty(); the emptiness check and append are now inseparable, so two tasks racing (even both past an expired lock) can never both seed - ensureServerSeed uses seedIfEmpty instead of streamHasContent-fence + publishAndWait; the seed lock is now purely an efficiency optimization (avoid a duplicate fetch), not a correctness dependency - fix the misleading comment that claimed a copilot merge is not counted as an edit in the multi-replica path (it round-trips as REDIS_ORIGIN and does count; a safe idempotent over-persist, never a lost edit) - add interleaving tests: seedIfEmpty atomicity/fence, the split-brain regression under an expired lock, a peer edit during attach catch-up, and concurrent two-task compaction * docs(realtime): align seed comments with the atomic-append correctness model Greptile flagged lingering doc drift: the module + shouldSeed comments still credited the seed lock + empty-stream check as the split-brain fix. Reframe them so the atomic seedIfEmpty is the exactly-once guarantee and shouldSeed is an efficiency gate only. * feat(realtime): live workspace tables list, sharing one invalidation-room impl (#6053) * feat(realtime): live workspace tables list, sharing one invalidation-room impl Bring the tables list to parity with the files list: a create/rename/move/delete/ restore now propagates to every viewer live instead of waiting out the 30s staleTime. Following the files pattern, but factoring the two into one shared implementation rather than copy-pasting. - add ROOM_TYPES.WORKSPACE_TABLES + its authz resolver (workspace-id-addressed, reuses the workspace resolver like workspace-files) - extract setupWorkspaceInvalidationRoom (server) and useWorkspaceInvalidationRoom (client) — the presence-free, workspace-scoped live-list room; files and tables now both bind to it, so they can never drift. Event/room names derive from the room type. Replaces the standalone workspace-files handler + hook - notifyWorkspaceTablesChanged fanout fired from the table service (createTable, renameTable, moveTableToFolder, deleteTable, restoreTable) so it covers both the HTTP routes AND copilot, which call the service directly - relay /api/workspace-tables-changed endpoint; wire the hook into the tables page - consolidate the handler test into one suite run against both room types * feat(realtime): live tables list also covers table-folder mutations Fold in the follow-up: a table folder create/rename/move/delete/restore now propagates to the tables list live too, so the browser is fully consistent. - generic notifyFolderResourceChanged(resourceType, workspaceId) dispatches the workspace live-list signal by resource type (a map, not a special-case if), so file/knowledge_base/workflow are no-ops today and gain liveness by adding a map entry when they adopt an invalidation room - fired from the shared folder lifecycle (createFolder/updateFolder/deleteFolder/ restoreFolder), covering routes AND copilot - the tables room hook now invalidates the table folders query too, not just the tables list, since the page renders both * fix(realtime): skip per-table live-list notify during a folder cascade A folder delete/restore already fires one folder-level notifyFolderResourceChanged for the whole subtree, but the cascade also calls deleteTable/restoreTable per table — each awaiting its own notifyWorkspaceTablesChanged. A folder with many tables would run N+1 sequential relay calls (each bounded by NOTIFY_TIMEOUT_MS), blocking the mutation. Add a skipNotify option the cascade passes so only the one folder-level notify fires. * feat(collab-doc): Hocuspocus binary persistence + Next 16 seed/persist fixes (#6059) * fix(collab-doc): make server-side seed conversion work under Next 16 / Turbopack Opening a file left both collaborators read-only and stalled ~12s: the server-side seed (markdown -> Yjs, run through the headless editor engine) was failing, so the doc never seeded and the editor never left its readiness gate. Two root causes, both latent until a real build/runtime (typecheck + unit tests don't exercise either), surfaced by the Next 16 upgrade: 1. Build boundary: the server seed route imported the shared editor schema (`createMarkdownContentExtensions`), which pulled in the React node-view components (`useEffect`) -> 'client component in a Server Component'. Split each node's React-free schema into its own `*-schema.ts` (code-block, image, raw-markdown-snippet); the client editor still injects the React node views via the existing `nodeViews` param, unchanged. 2. Runtime DOM: the converter installs a jsdom `window` on `globalThis`, but Turbopack's server bundle gives bundled `@tiptap/core` a `window` that does NOT read `globalThis`, so `elementFromString` threw 'no window object available'. Externalize the `@tiptap/*` packages the converter uses (native Node require, so their `window` reads the real global) and fix the converter's DOM guard to gate on `window` (what TipTap checks) with no sticky flag. Verified: seed route returns 200 with the Yjs update; 514 collab-doc + editor tests pass; schema byte-identical after the split. * feat(collab-doc): persist the Yjs binary and load it on cold-start (Hocuspocus pattern) Adopt the industry-standard Hocuspocus store/load-document pattern so a cold room open loads the file's last-persisted Yjs binary directly instead of re-converting markdown -> Yjs on every open. Rebuilding the CRDT from markdown on each connect is the exact anti-pattern Tiptap/Yjs warn against (fresh client ids -> duplicated content); it also forced the fragile server-side headless-editor conversion on every open. Now conversion runs only on a genuine first open or an external markdown edit. - new table workspace_file_collab_state(file_id PK->workspace_files cascade, doc_state bytea, source_hash, updated_at): the Yjs binary + a hash of the markdown it was derived from (bounded <=~1MB by the 256KB round-trip gate). Mirrors Hocuspocus's extension-database (binary in a DB column). Migration 0275. - persist upserts the binary (tagged with the exact markdown just written) - cold-start seed returns the cached binary when its source_hash matches the file's current markdown; otherwise converts (and the next persist refreshes the cache) - also externalize yjs / y-protocols / lib0 alongside @tiptap: bundling loaded a second yjs copy, so @tiptap/y-tiptap's 'instanceof Y.XmlElement' failed on app-created nodes ('Unexpected case') during Yjs -> markdown Verified end-to-end: seed -> persist -> seed returns the exact persisted binary (a cache hit, no re-conversion). 18 collab-doc + 51 realtime file-doc tests pass. * fix(collab-doc): best-effort cache read + drop dead barrel - seed: a cache-read failure (transient DB error, not-yet-migrated cache table) no longer aborts a cold room open — the durable markdown is already in hand, so fall through to conversion. Symmetric with persist's best-effort cache write. Addresses the Cursor Bugbot finding on the read/write asymmetry. - remove the collab-doc index.ts barrel: nothing imported it (every consumer uses direct ./seed / ./merge / ./converter imports), so it was dead re-export surface. De-export COLLAB_DOC_FIELD accordingly — it is used only inside converter.ts. * fix(collab-doc): stream every external file write into open editors, not just edit_content (#6070) * fix(collab-doc): stream every external file write into open editors, not just edit_content A copilot/mothership edit to an open markdown file did not appear live in another user's editor: the live-doc merge bridge (mergeEditIntoLiveFileDoc) was wired into the edit_content tool ONLY. Every other server-side write — the file tool (/api/tools/file/manage), function_execute (/api/function/execute via writeWorkspaceFileByPath), create_file overwrite, and the PUT /content route — went straight to updateWorkspaceFileContent and skipped the merge, so the durable file changed but the open editor never updated. Confirmed from live logs (the mothership 'Prepend sentence' ran read + file + function_execute — zero apply-edit calls) and Redis (the prepended text was absent from the doc stream). Centralize the merge at the one chokepoint every external writer shares: - updateWorkspaceFileContent gains an opt-out "syncLiveDoc" (default on) and, after the durable write, merges markdown writes into any open collaborative doc (best-effort; no-op when nobody has it open). Any current OR future writer is covered automatically. - persist.ts opts out (syncLiveDoc:false) — it IS the doc→markdown projection, so merging it back would be a persist→merge→persist self-loop. - create_file opts its empty shell out (real content arrives via a later write) so an open editor never flickers to empty on overwrite; threaded through writeWorkspaceFileByPath. - edit_content drops its now-redundant explicit merge call (the chokepoint handles it). - binary writers (image/video/audio/ffmpeg/download) are naturally excluded — the merge is gated to markdown, the only format the collaborative editor renders. Also bump the api-validation route baseline 994→996 to match the true route count already on this branch (pre-existing ratchet drift from an earlier merge; NOT added by this PR). * fix(collab-doc): defer setEditable out of the render phase (flushSync warning) The collab editability-reapply effect called editor.setEditable synchronously. In collab mode isEditable flips from readiness (synced + seeded), which is driven by a Yjs config.observe firing synchronously inside Y.applyUpdate — so the effect can run while React is mid-render. TipTap's React binding commits setEditable's transaction with flushSync, which throws "flushSync was called from inside a lifecycle method. React cannot flush when React is already rendering." Defer the setEditable to a microtask (runs right after the current commit, before paint), guarding against a destroyed editor or a stale value before it fires. Only the collab path (this effect) hit the warning; the streaming/settle effect's setEditable calls run on the non-collab path where isEditable isn't driven by a mid-render Yjs observer. * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) (#6073) * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) The non-collaborative streaming/settle effect called editor.setContent / setEditable / setTextSelection / focus directly in the effect body. setContent mounts the custom node views synchronously through the @tiptap/react flushSync path (tiptap#3764), so when this effect runs while React is mid-render it throws "flushSync was called from inside a lifecycle method." This is the second flushSync source (the collab editability effect was the first, fixed separately); it fires on the agent-streaming-into-a-non-collab-editor surface. Defer the effect-body view mutations to a microtask via a small runOffRender helper (runs right after the current commit, before paint; no-ops if the editor was torn down). The settle block is deferred as ONE microtask so setContent -> collapse selection -> setEditable -> focus keep their order. The streaming rAF tick is left untouched — it already runs off-render, so it keeps writing content directly. queueMicrotask is TipTap's own documented remedy for this warning. 497 rich-markdown-editor tests (incl. stream-settle-selection) pass; tsc + lint + api-validation + boundary + prune green. Needs a live check: stream an agent into a non-collab markdown file and confirm it still renders smoothly. * chore(rich-markdown-editor): trim verbose flushSync-defer comments * fix(rich-markdown-editor): drop superseded settle/stream microtasks via a run token runOffRender previously only guarded editor.isDestroyed, so if React ran the next reconcile pass (a newer stream or settle) before a queued microtask flushed, the stale microtask could still apply setContent/setEditable/setTextSelection over the newer state. Tag each effect run with an incrementing token; a deferred mutation applies only when its run is still the latest (and the editor is alive). A run token fits this effect's several early-return exits better than a per-exit cleanup flag. Addresses Greptile/Cursor review. * fix(rich-markdown-editor): never drop the settle selection-collapse under a superseded run The run token drops a superseded settle's microtask, but the settle had already flipped its state flags synchronously — so a pre-empting steady-sync run took the non-settle path and never collapsed the selection, leaving a post-stream select-all painting the leaf-in-selection decoration. Track the collapse as a debt (pendingCollapseRef): whichever deferred run ultimately applies — settle or the steady-sync path — clears it, so the collapse runs exactly once on the latest content. Addresses the Cursor review finding. * feat(tables): show live cell-selection carets in the embedded chat panel (#6081) The table cell-selection presence room was joined only on the dedicated /tables/[id] page (useTableRoom was passed an empty id in embedded mode). Join it in embedded too, so the mothership chat resource panel shows collaborators' live cell selections and broadcasts the local one. tableId is already resolved from props in embedded (the data event stream already uses it un-gated), and authz runs on join, so this is safe. Avatars are unaffected — they render only in the !embedded Resource.Header, so the panel gets carets without avatars. * feat(collab-doc): If-Match optimistic concurrency so persist never clobbers an out-of-band edit (#6085) * feat(collab-doc): optimistic-concurrency guard so persist never clobbers an out-of-band edit The relay projected the live Yjs doc back to durable markdown unconditionally (last-write-wins), so a persist already in flight when an external write landed could overwrite it. Add RFC 7232 If-Match optimistic concurrency end to end, reconciling through the CRDT (never rejecting user work): - updateWorkspaceFileContent gains an expectedUpdatedAt guard: the write commits only if the file is still at that version (checked against the SELECT ... FOR UPDATE-locked row, so it is atomic with the write), else it throws the new ContentVersionConflictError without clobbering. - persistFileDoc takes expectedVersion and returns a discriminated result (persisted | missing | conflict). On conflict it returns the current durable content + version instead of writing. - The relay tracks the durable version its live doc is synced to — set on seed, advanced when a durable write is merged in (apply-edit carries the version), and on each successful persist. It is held cluster-wide in Redis (filedoc:syncver:{name}) so whichever task persists reads the same version, with the per-room value as the single-pod fallback. - flushPersist sends that version as If-Match. On a conflict it merges the current durable content into the live doc (so the out-of-band edit AND the live edits converge) and retries (bounded), so even a last-leave flush racing an external write persists the reconciled result rather than losing the session's edits. Threads the version through the seed + persist contracts and the apply-edit payload. No schema change (reuses workspace_files.updatedAt as the version token). Tests: app-side CAS (match writes, mismatch throws + cleans up the orphan upload), relay conflict handled gracefully without clobber/loop; 236 realtime + 76 sim collab/uploads tests, tsc x2, lint, api-validation, boundaries, prune all green. * chore(collab-doc): heartbeat-refresh the synced-version key TTL alongside its stream Keep filedoc:syncver:{name} alive as long as the room's stream (it was only re-set on seed/merge/persist), so an open-but-idle doc's persist If-Match token can't expire and force a needless reconcile. * fix(collab-doc): stop persist-conflict retries when there is no live doc to reconcile On an If-Match conflict with no live doc to reconcile into (last collaborator gone, no shared stream), applyMarkdownToLiveFileDoc returns no-live-room; re-projecting the same pre-teardown snapshot would only re-conflict, so break the retry loop immediately and leave the out-of-band (durable) content authoritative — the intended conflict policy. Addresses Greptile review. * fix(collab-doc): close three optimistic-concurrency edge cases from review - Single-pod persist retry projected the pre-reconcile snapshot (captureState always returned the initial localState), while the synced version had been advanced by the reconcile — so the If-Match could pass and clobber the reconciled edit. captureState now re-reads the live doc on each attempt (falling back to the pre-teardown snapshot only once the room is gone). - The synced version was recorded from this task's own seed FETCH before knowing whether this task's seed actually won; a peer winning with a different version could leave a newer token than the stream content. Record it only inside the didSeed branch (the task whose seed won); peer-seeded tasks read the winner's cluster value. - Persist wrote UNCONDITIONALLY when no version was available (relay version momentarily missing), which could clobber non-empty durable content. It now returns conflict for a non-empty file with no version (reconcile/retry once the version is re-established); an empty file's first write stays unconditional. * fix(collab-doc): defer (not reconcile) on missing version, and use the freshest version token - Missing-version persist now returns 'deferred' instead of 'conflict'. A missing version token (a Redis blip on a peer-seeded task) is NOT a genuine out-of-band change, so triggering a reconcile would wipe live edits (incoming-wins) even though nothing changed durably. Deferred means: don't write, don't reconcile — leave the edits in the stream and let a later persist write them once the version is re-established. - currentVersion now takes the MAX of the cluster (Redis) and local room versions rather than always preferring Redis, so a lagged/failed fire-and-forget Redis set can't shadow a newer local value and cause spurious If-Match conflicts. Versions are monotonic epoch-ms, so the larger is the later sync. * fix(collab-doc): make persist If-Match teardown-race-immune and recover missing version on final flush Close two last-leave concurrency holes Cursor flagged: - Thread the reconciled version LOCALLY through the persist retry loop. After a conflict+reconcile the correct next If-Match is exactly result.version, so carry it in a local var instead of re-deriving from room.syncedVersion/Redis. On a last-leave flush destroyRoomIfIdle removes the room from the map before the async flush finishes, so mergeMarkdownIntoRoom's recordVersion can no longer update room.syncedVersion — threading makes each retry's precondition correct by construction, immune to that dropped mutation and to a best-effort Redis re-read. - Cache the resolved version back into room.syncedVersion in currentVersion() so a peer-seeded/tail-only task (which never sets it locally) or a later transient Redis read failure still resolves it from the last value seen (monotonic max, never regresses). - On a FINAL flush, briefly retry resolving the If-Match when the version read momentarily fails, rather than deferring and stranding the session's edits in the TTL'd stream — the version is cluster-wide and heartbeat-refreshed. * fix(collab-doc): stamp cluster sync version the moment the seed wins, before the liveness guard The winning seeder set the If-Match token (room + Redis filedoc:syncver) only after the liveness/seeded guard that follows seedIfEmpty. But the tailer can integrate the just-appended seed DURING the seedIfEmpty await, so isDocSeeded(room.doc) is already true when the guard runs and it returns early — leaving the stream holding seed content with no cluster version. Later persists then send no If-Match, the app returns `deferred`, and session edits stay only in the TTL'd stream (the exact stranding this PR prevents elsewhere). Move the version stamp to immediately after seedIfEmpty wins, before the guard. Recording it only once our seed won (not from the fetch) is preserved, so it still can't shadow a peer's winning seed. * fix(collab-doc): make the synced-version token monotonic at every write site The If-Match token is written fire-and-forget from the seed stamp, merges, and persists, both locally and to Redis. An out-of-order write (e.g. a seed's lagged setSyncedVersion landing after a later merge's) could regress it below the version the live doc already incorporates, causing spurious If-Match conflicts — and on a last-leave flush with no live room to reconcile into, a spurious conflict leaves durable authoritative and drops the session's edits. - setSyncedVersion now writes via SET_VERSION_IF_NEWER_SCRIPT (Redis-side compare-and-set): it overwrites only when the new value is greater, refreshing the TTL either way. - recordVersion / the persisted branch / the seed stamp all take Math.max instead of assigning room.syncedVersion directly. Versions are monotonic epoch-ms, so "newer" is a plain numeric compare, exact within a Lua double. * fix(collab-doc): close three last-leave persist edge cases from review - Stale snapshot after reconcile (High): the multi-task captureState fell back to the pre-await localState snapshot even after a reconcile advanced ifMatch, so a failed stream re-read could persist the pre-reconcile state against the new version and clobber the out-of-band edit the reconcile just incorporated. NULL localState after a reconcile so a failed read aborts instead. - Lock miss aborts reconcile (Medium): a merge-lock acquisition failure returned 'no-live-room', indistinguishable from an absent stream, so flushPersist treated transient contention as terminal. Return a distinct 'merge-unavailable' and handle it as retry-later (edits stay in the stream), never as "nothing to reconcile into". - Peer syncver never recovers (Medium): the winner's setSyncedVersion was fire-and-forget with swallowed errors — the only way a peer-seeded task learns the durable version — so a dropped write left that peer deferring forever. Make it retry (bounded) like appendUpdate/seedIfEmpty; the monotonic script keeps a racing retry a no-op. * fix(collab-doc): scope the persist If-Match to a content version so metadata bumps can't clobber edits The optimistic-concurrency validator was `updatedAt`, which rename/move/delete/restore also bump with no content change. A racing live-doc persist then saw a stale token, got `conflict`, reconciled the pre-edit durable body via updateYFragment (incoming-wins on overlap), and wiped the user's in-flight edits. Scope the validator to content (RFC 7232 semantics — validate the representation, not the row): - New `workspace_files.content_updated_at` (NOT NULL, `now()` fast-default — no table rewrite). Advances ONLY on content writes (upload / overwrite / create); metadata writes never touch it. - The FOR UPDATE CAS, the merge-notify version, and the seed version all use `content_updated_at`. A rename now leaves it unchanged, so the persist If-Match still matches -> no spurious conflict, no reconcile, no lost edits. Genuine out-of-band content writes still conflict and reconcile. - Consolidated the collab schema into one migration (the collab-state table + the new column) per request, rather than a separate follow-up migration. Relay/store/contracts unchanged (still a numeric monotonic version). * chore(collab-doc): condense the densest persist comments (no behavior change) Cleanup pass: tighten the three longest comment blocks added while hardening the persist path (currentVersion cache, ifMatch threading, final-flush version retry) without dropping any invariant. No dead code found (biome lint clean; all new symbols referenced). * fix(collab-doc): persist must return the content version, not updatedAt Follow-up to the content-scoped If-Match: persistFileDoc still returned `updatedAt` as the version in both the persisted and conflict results, while the CAS/seed/merge all guard on `content_updated_at`. A content write sets both to the same instant, so it was coincidentally correct — until they diverge: if a metadata write bumps `updatedAt` past `content_updated_at`, the conflict path returned the larger `updatedAt`, so the relay's re-persist sent an If-Match the CAS (which checks `content_updated_at`) could never match → perpetual conflict → dropped reconciled edits. Return `contentUpdatedAt` in both paths so the relay's token always matches what it's checked against. * fix(collab-doc): defer persist whenever the version is missing; guard the content-version test - Empty-file CAS race (Medium): the unconditional-write carve-out for size===0 read `record.size` outside the write transaction, so a concurrent first content write could land after the check and be clobbered. With content_updated_at NOT NULL every existing file always has a real version, so a missing expectedVersion is always transient — always defer, never write unconditionally. Removes the TOCTOU hole. - Content-version test (Low): the merge-chokepoint test kept updatedAt == contentUpdatedAt, so it passed even if wired to the wrong field. Mock distinct values and assert contentUpdatedAt, so a regression to updatedAt now fails the test. * fix(collab-doc): don't reconcile a conflict the live doc already reflects (would wipe newer edits) flushPersist reconciled the durable body into the live doc on every conflict. But when the conflict comes from a racing self-persist (or an apply-edit the chokepoint already merged), the durable body is a STALE SUBSET of the live stream, and the incoming-wins updateYFragment merge moves the doc backward — wiping newer in-flight edits, which the retry then persists. Before reconciling, re-check the freshest synced version. If it already covers the conflict version, the live doc has already incorporated that content (or is ahead), so skip the reconcile and just retry with the freshest version as If-Match — the re-projection captures the current live stream, preserving every edit. Only a genuine out-of-band change the live doc hasn't incorporated (freshest < conflict version) is reconciled in. freshest never exceeds the durable version, so this can't loop. * fix(collab-doc): make content_updated_at monotonic per file; skip-reconcile can't loop The If-Match token was stamped with app-local new Date() on each content write, so cross-instance clock skew could stamp a later write with an EARLIER content_updated_at — breaking the version ordering the whole optimistic-concurrency scheme (and the skip-reconcile branch's freshest>=version assumption) depends on. Under skew the relay's monotonic syncedVersion could exceed the durable version, sticking the If-Match: persist conflicts forever, exhausts retries, drops the session's edits. - Stamp content_updated_at strictly after the current committed value (we hold the row's FOR UPDATE lock): new Date(max(now, currentFile.contentUpdatedAt + 1ms)). Monotonic per file regardless of clocks; also removes same-millisecond collisions. updatedAt stays plain wall-clock (display/sort). - Skip-reconcile branch retries with result.version (the durable value the CAS will match), never freshest (which could exceed it and loop). Belt-and-suspenders now that the version is monotonic. * refactor(collab-doc): drop the destructive in-persist reconcile; adopt-version-and-retry on conflict The in-persist reconcile projected the durable body back over the live doc via updateYFragment ("make the doc match"). That is destructive: when the live stream is already ahead — the common case, because the write chokepoint (mergeEditIntoLiveFileDoc) already merged the out-of-band change into the stream — it moved the doc backward and wiped newer in-flight edits. This produced a run of races (stale snapshot, wipe-newer-edits, version-lag skip miss) that a full-document reconcile fundamentally can't avoid, since deciding when it's safe relies on a laggy cross-task version token. Remove it. On conflict, adopt the durable version as the new If-Match and retry: captureState re-reads the current stream (which holds the out-of-band change AND the live edits), so the re-projection persists the converged result. The durable change reaches the live doc via the chokepoint, never here. Trade-off: the only unmerged out-of-band write is one whose chokepoint merge itself failed (rare, logged), which we accept over the frequent reconcile-wipes-edits race. - flushPersist: conflict -> ifMatch = result.version, retry (bounded). No applyMarkdownToLiveFileDoc. - conflict response drops `markdown` (contract + relay type + persist) — no body needed, saves a blob fetch. applyMarkdownToLiveFileDoc stays (still used by the apply-edit route / the chokepoint). * fix(collab-doc): don't let a last-leave conflict retry clobber via the stale local snapshot Regression from dropping the reconcile: on conflict the retry adopts result.version and re-reads captureState. But after single-pod last-leave teardown the room is already destroyed, so captureState falls back to the pre-teardown localState (which lacks the out-of-band change); the retry then CAS-passes and overwrites the committed external write — undoing the external-wins last-leave policy. Null localState on the first conflict, so the retry can only use freshly-read authoritative state (stream / live doc). When none is available (single-pod room gone, or a transient stream-read failure) captureState returns null and the retry stops, leaving durable content authoritative. Covers both the single-pod and multi-task-stream-unavailable variants of the stale-snapshot clobber. * fix(collab-doc): stop (don't re-persist) on a persist conflict — closes the commit-window clobber The conflict retry adopted the durable version and immediately re-persisted the current stream, assuming the stream already held the out-of-band change. But an external write commits durable BEFORE its chokepoint merge (mergeEditIntoLiveFileDoc) reaches the stream, so a persist landing in that window CAS-passed with a stream that still lacked the external content and clobbered the committed write — not just the rare merge-failed path, but a race on every external write, worst at last-leave flushes. Make persist a single attempt: on conflict, STOP and leave durable authoritative. The chokepoint merges the change into the stream and — only once it is actually there — advances the synced version via its own recordVersion; a later flush (debounced or final) then projects the converged stream with a matching token. The session's edits stay in the stream meanwhile. The conflict handler deliberately does NOT advance the synced version, or the next flush would clobber with a still-behind stream. Removes the retry loop and PERSIST_CONFLICT_RETRIES. * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run (#6094) * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run These three table operations mutate row data but emitted no `rows` change signal, so open editors' grids stayed stale until a manual refresh (enrichment *results* already stream live via `cell` events; these are the bulk paths that don't emit per-cell events): - Async row delete (`runTableDelete`): signal as rows drop out (throttled with the existing progress event) and once more on completion — the `job` progress event only drives the delete meter, not the rows query. Covers the delete-async route and the copilot bulk-delete, since both share the runner. - Cancel runs (`cancel-runs` route): cancelling clears each affected row's exec state; the `dispatch: cancelled` events drop the run overlay but the client then renders authoritative DB state, so refetch. Only when something was actually cancelled. - Run column (`columns/run` route): starting a run bulk-clears the target group's cells to pending; refetch so the cleared cells show. Only when a dispatch was actually created. Guarded so no signal fires on a no-op/failure. Adds a delete-runner test asserting the completion signal. * fix(tables): guarantee the live-rows signal on every mutating path (review) - Delete runner (Greptile P1): a batch could commit and the job then cancel/supersede before the next throttled progress signal or `markJobReady`, bypassing both signals and leaving deleted rows on screen. Track `deletedAny` and fire the grid refetch in a `finally`, so it runs on EVERY exit — completion, cancel/supersede, mid-batch lock, or a rethrown error after a partial delete. - cancel-runs / columns/run routes (Cursor): the `cancelled > 0` / `if (dispatchId)` guards don't always reflect DB row changes — cancel tombstones exec state even when 0 dispatches were active, and a run bulk-clears cells then can return a null dispatchId. Signal unconditionally; a stale-but-harmless refetch beats a missed one. - Tests: assert the delete signal fires on the mid-run-cancel-after-delete path and NOT when nothing was deleted. * fix(tables): mark deletedAny before the page delete so a mid-page lock still refreshes the grid `deletePageByIds` commits in internal batches, so a delete lock landing mid-page can persist earlier batches and THEN throw TableLockedError — the catch returns without a count, so setting `deletedAny` from the return value missed it and the finally skipped the grid refetch. Set `deletedAny = true` before the call (any attempt may commit rows); an attempt that commits nothing only over-refetches (harmless). Adds a test asserting the signal fires when a page throws a mid-page lock. * fix(files): make embedded resource file view collaborative (#6095) * fix(files): make embedded resource file view collaborative The /chat resource panel rendered saved files through FileViewer without the collaborative opt-in, so a file open on the Files page and the same file open in the embedded panel never joined the same file-doc room — no live carets and no live content sync between the two surfaces. Pass collaborative on the EmbeddedFile FileViewer. Collaboration still self-gates on canEdit + non-streaming + workspace doc, so the agent token-stream preview (the dedicated streaming-file path, canEdit=false) is untouched. * fix(files): refcount file-doc room membership per shared socket Two collaborative surfaces in one tab (the Files editor and the embedded chat resource panel) share one Socket.IO connection, so both providers for the same file JOIN the same room over that socket. The server's LEAVE does socket.leave(name) with no membership refcount, so the first provider's destroy() would strand the second still-mounted one — no more live content or presence. Count live providers per file per socket (keyed by the stable Socket object, so it survives reconnects) and emit LEAVE only when the last provider for a file tears down. The single-provider path is unchanged (0->1->0). * feat(tables): propagate shared saved-view changes to collaborators live (#6100) Table views (named filter/sort/layout presets) are table-wide shared state — every reader sees every view — but view create/update/delete had no realtime signal, so a collaborator only saw another user's view changes on their own staleTime/focus refetch. Add a 'views' table event kind + signalTableViewsChanged, emitted from the views service (createTableView/updateTableView/deleteTableView, on real success only), and a client handler that invalidates the views query alone (no rows/definition refetch — a view is presentation state on the loaded table). Mirrors how row/schema/metadata changes already propagate. * test(tables): cover the views realtime signal (emit + on-success-only) (#6101) - events.test.ts: signalTableViewsChanged appends a single 'views' event carrying the tableId (through the real memory buffer). - views/service.test.ts: create/update/delete emit signalTableViewsChanged on real success, and DON'T on a no-op (a PATCH/DELETE targeting a missing view changes nothing, so it must not signal). Mirrors delete-runner's signal-path coverage; drives the DB via the shared dbChainMock. - Add tableViews to the comprehensive @sim/db/schema test mock so the service tests can queue the in-transaction existence row. * chore(ci): reconcile api-validation baselines after the staging merge The staging merge unioned realtime-rooms's own `as unknown as` cast (lib/collab-doc/converter.ts) with staging's zod-recursive-type cast (lib/api/contracts/tables.ts), so the non-test double-cast count is 9 — both casts pre-existed and were individually accepted on their branches. Also tighten rawJsonReads 6->5 to the true current count. Fixes the strict API contract boundary audit on realtime-rooms. * feat(copilot): stream file edits into the live collaborative Y.Doc (keep embedded view collaborative) (#6108) * feat(copilot): stream file edits into the live collaborative Y.Doc Copilot's file edits previously only reached the live doc once, at the final edit_content write, so a collaborative editor watching the file saw nothing until completion (streaming looked broken) and the client-side preview path was suppressed in collab mode. Make copilot a CRDT peer: as it streams append/update/patch content, merge the growing markdown into the file's live Y.Doc via the existing apply-edit path (a minimal updateYFragment diff, concurrent-edit-safe), throttled to ~250ms. version is omitted for these intermediate merges — they advance the live doc for viewers but are not durable checkpoints; the final edit_content write carries the real contentUpdatedAt and reconciles the durable file. Per the relay's persist gating, server-internal merges never schedule a persist, so a copilot-only stream produces zero intermediate file writes. - notify.ts: mergeEditIntoLiveFileDoc version is now optional (streaming omits it). - file-preview-adapter.ts: throttled live-doc merge at the edit_content stream hook. * fix(copilot): order + gate streaming live-doc merges; fast collab first render Harden the streaming merge (adversarial review): - Order + bound: dispatch through a per-file in-flight guard (drop-while-in-flight) so a stale out-of-order snapshot can never land after a newer one and regress the doc, and relay load is capped at one request per file regardless of rate. - No wipe: gate append/patch on the base file content having loaded — a base-less snapshot would diff to a delete-everything wipe of the seeded doc; update streams a full rewrite from scratch and needs no base. - Markdown-only gate: non-markdown files have no collaborative room, so skip the wasted relay round-trip. Fast collab first render (Issue 2): render the already-fetched markdown read-only via generateHTML while the collaborative doc seeds, with the editor mounted-but- hidden in the same layout box for a seamless swap on collabReady. Pure HTML — it never touches the Y.Doc (client seeding duplicates the doc), and generateHTML escapes text (raw-HTML snippets render escaped), so no XSS. * test(copilot): cover streaming file edits into the live collaborative Y.Doc Drives edit_content args_delta stream events through processFilePreviewStreamEvent and asserts the live-doc merge: fires with the growing FULL previewText and no version arg; is throttled (~250ms per file); is skipped for non-markdown files and for a base-less append (the delete-everything wipe guard); and runs at most one-in-flight per file. Verified to fail if any gate/guard is removed. * fix(collab-doc): coordinate live-doc merge ordering in one place; close durable-clobber race The second review found a residual: the durable edit_content write went through a different path than the adapter's in-flight guard, so a late straggler streaming merge could land after it and, via a persist, clobber the durable file's tail. Move the per-file coordination into mergeEditIntoLiveFileDoc (the one place both the streaming and durable paths call): a streaming (versionless) merge is dropped while one is in flight for the file; a durable (versioned) write instead WAITS for the in-flight streaming merge, so the final content is always the last merge applied and can't be regressed by a straggler. Simplifies the adapter (drops its Set + helper). Relocate the one-in-flight test to notify.test.ts (streaming-drops-while-busy + durable-waits-then-applies-last); the adapter test keeps throttle/gates/previewText. * fix(copilot): address review — order merges, exclude update, gate throttle, unhide stream Review round on #6108: - Greptile P1 (durable merges lose ordering): serialize ALL merges per file on one chain in mergeEditIntoLiveFileDoc (each chains after the current tail), so concurrent durable writes can't resume-and-fire out of order. notify now exposes isLiveDocMergeInFlight. - Cursor High (update stream blanks the doc): only append/patch stream — they build on the loaded base; update is a from-scratch rewrite whose partial snapshot would diff the full doc toward a fragment, so it applies atomically at the durable write. - Cursor Medium (throttle advances on a dropped merge): the adapter gates on !isLiveDocMergeInFlight, so the send throttle advances only on an actual dispatch — no lag, no backlog behind a slow relay. - Cursor Medium (placeholder hides a live stream): show the fast-render placeholder only when not streaming, so a stream that starts before the doc seeds shows through the editor. - Soften merge.ts/notify.ts comments per the lifecycle audit: only UNTOUCHED regions are preserved; a region the merge rewrites reconciles toward copilot's content. Tests updated: notify covers chain ordering + isLiveDocMergeInFlight; adapter covers append streaming, throttle, non-markdown/base-less/update skips, and the in-flight skip. * fix(collab-doc): reject stale durable merges at the relay (cross-process ordering) The in-process merge chain only orders merges within one apps/sim process. Two durable writes for the same file on DIFFERENT processes could reach the relay out of dispatch order; the relay recorded the version monotonically but still APPLIED the older markdown, regressing the live doc while the token stayed high (a later persist could then write the stale content back over the durable file). Enforce ordering at the relay — the single cross-process coordination point — using the existing Redis primitives: under the per-file Redis merge lock, read the cluster-wide synced version and SKIP a versioned merge that is not newer (a newer durable write already landed). Make recordVersion await setSyncedVersion so it is durable before the lock releases, so the next holder's staleness check reads a consistent value. Streaming (versionless) merges are unaffected — they carry no durable version and are ordered per-process by the caller. Adds a relay test asserting a stale/idempotent versioned merge returns 'stale' and never computes or publishes a diff. * fix(copilot): match durable path — detect markdown by MIME type + name at the stream gate The streaming gate checked isMarkdownFile with only the filename, while the durable merge uses type + name — so a text/markdown file without a .md extension was skipped mid-stream (it self-corrected at the durable write). Pass editIntent.contentType so streaming detects the same set of markdown files as the durable path. * test(copilot): assert throttle follow-through after an in-flight merge clears * fix(collab-doc): order streaming merges by streamedAt so a late snapshot can't regress a newer durable write * refactor(collab-doc): tidy merge-order docs + relay order object; cover multi-replica streaming stale-check * fix(collab-doc): order streaming merges by causal base version, not wall-clock A streaming snapshot now carries baseVersion (the durable contentUpdatedAt it was built from) instead of a wall-clock streamedAt. The relay drops the snapshot when a newer durable write landed since that base, so a concurrent human save can no longer be clobbered in the live doc and then persisted over the durable file. Skew-immune: both keys are DB-monotonic contentUpdatedAt values. * fix(collab-doc): derive streaming baseVersion as contentUpdatedAt ?? updatedAt Match the version line the seed/persist use so a legacy file with no content version still ships an ordered streaming snapshot instead of an unordered one. * fix(collab-doc): fail-closed on a streaming snapshot with no baseVersion The live-merge gate now requires a numeric baseVersion, not just loaded base content. A rare base with no file record (hence no version) would otherwise ship an unordered snapshot the relay can't stale-check, risking a clobber of a concurrent durable write. Skip the live merge instead; the durable write reconciles. * docs(collab-doc): document the accepted concurrent-independent-streams limitation * chore(ci): reconcile api-validation baseline after the staging merge The merge commit auto-merged the baseline at 1000; bump totalRoutes/zodRoutes to 1003 for staging's three new contract-bound routes (nonZodRoutes still 0). * test(files): update storage-accounting assertion to the mergeEditIntoLiveFileDoc options object * fix(collab-doc): trace the full yjs/tiptap external stack into the file-doc route bundles The seed/merge/persist internal routes run the collab-doc converter (markdown <-> Yjs via headless TipTap) server-side. Those deps are serverExternalPackages, and the standalone tracer only force-included jsdom — it does NOT follow yjs's ESM subpath imports of lib0 (lib0/logging, ...), so Docker/standalone builds shipped node_modules without them and the seed route 500'd (Cannot find module 'lib0/logging'). That left every collaborative document unseeded and permanently read-only on deployed envs. Force yjs, lib0, y-protocols, and @tiptap into the trace for all three routes. * fix(collab-doc): copy the full yjs/lib0 stack into the app image The seed/merge/persist routes run the converter (markdown <-> Yjs) server-side. yjs is a serverExternalPackage and the Next standalone tracer copies lib0 only partially — it drops the ESM subpath file lib0/logging.js that yjs.mjs imports via lib0's exports map, so the seed 500s ('Cannot find module lib0/logging') and every collaborative doc is stuck read-only. Verified in the running dev container: /app/node_modules/lib0 had 37/38 files, logging.js missing. outputFileTracingIncludes can't fix it — its globs resolve against apps/sim, but these deps hoist to the monorepo-root node_modules, so the glob matches nothing (my prior next.config attempt was a no-op; reverted). Instead COPY the complete lib0/yjs/y-protocols from the deps stage in the runner, overwriting the partial trace — the same pattern already used for isolated-vm. * feat(files): stream copilot edits into the collaborative doc smoothly (#6122) * feat(files): stream copilot edits into the collaborative doc smoothly - apply the agent stream client-side into the live Yjs binding as minimal updateYFragment diffs (like main's setContent, but incremental) so it renders smoothly AND broadcasts to every peer via CRDT — a collaborator on /files sees the stream for free - gate the apply on collabReady so diffs never land on an unseeded doc; keep the read-only placeholder visible until the seed swaps in - run streamed ops under a dedicated tx origin so they stay out of the user's undo stack - delete the throttled server-side streaming merge and the baseVersion ordering machinery it needed (relay + notify + session contract); the durable final write still reconciles open editors and seeds late joiners * fix(files): apply agent stream as a true CRDT peer + guard base-less snapshots Review round 1 (Greptile P1s): - apply the stream against a private shadow replica (seeded from the live doc at stream start) and relay only the agent's own delta into the shared doc, so a concurrent peer edit to a region the agent snapshot didn't include is no longer reverted (previously the whole-body reconcile deleted it) - gate append snapshots on "must extend the base": a base-less append fragment (emitted before the base loads) can no longer reconcile the seeded doc to a wipe; patch still legitimately replaces a mid-region - gate the apply on collabReady so diffs never land on an unseeded doc; keep the placeholder visible until the seed swaps in - plumb streamOperation through the preview surfaces to drive the append gate - add a peer-edit-preservation test (fails under whole-body reconcile) and refresh the undo-isolation + broadcast tests for the session API * fix(files): destroy the agent shadow deterministically on settle Cursor round 1 (Low): endAgentStream ran inside runOffRender, whose microtask is dropped when a rapid follow-up stream bumps the run token — leaking the shadow Y.Doc. Split it out into an unguarded microtask queued after the (droppable) final apply, so the shadow is always destroyed. * fix(files): agent stream frames skip the relay's durable persist Cursor round 1 (High): client-applied stream frames broadcast over the sync channel, so the relay stamped a socket origin and ran schedulePersist — durably writing partial agent content mid-stream, attributed to the watching user (the old server-merge applied with no origin and never did). Restore that behavior: - new FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST wire tag; the provider tags AGENT_STREAM_ORIGIN updates with it (normal user edits stay SYNC) - the relay applies it under an AgentSyncOrigin (carries the socket id for broadcast exclusion, but is not a plain string) so originSocketId() is null → no edited/schedulePersist/lastEditorUserId; excludeSocketId() still excludes the sender, and the update still publishes to the stream so peers converge - the copilot's final edit_content write remains the authoritative durable persist - tests: relay applies+fans-out but never persists a SYNC_NO_PERSIST frame (verified it fails if applied as a socket edit); provider tags agent edits * fix(files): open the stream shadow at start + private extend baseline Cursor round 2: - High (settle skips apply without session): the stream shadow is now opened on the first ready frame, BEFORE the extend gate — so an `update` rewrite (whose every frame is gated out until settle) and a stream that finishes before seed still get a session, and settle applies the final body via the reused-or-on-demand shadow instead of leaving the doc stale until the durable reconcile. - Medium (peer edits stall the stream): the extend gate now reads a private `lastStreamedBodyRef` (the agent's own last frame), snapshotted at stream start, not `lastSyncedBodyRef` which `onUpdate` clobbers on peer edits — so a collaborator typing can't make the growing snapshot stop prefixing the shown body and freeze it. - Medium (multi-replica over-persist): pre-existing, documented "safe over-persist" (a peer task tails the frame as REDIS_ORIGIN and marks edited) — refreshed the stale comment to describe the SYNC_NO_PERSIST source; copilot's edit_content write remains the authoritative durable persist. * fix(files): fail-close base-less previews + operation-based stream hold Cursor/Greptile round 3 (High + Medium) — remove the fragile string-prefix "extend gate", which was the root of both findings: - Server: `buildFilePreviewText` now fails closed for an `append` whose base content hasn't loaded (returns undefined, like patch/update), so a base-less fragment never reaches the client. This eliminates the base-less wipe at settle (Greptile P1) at the source; an empty file (existingContent === '') still previews normally. - Client: the collab streaming tick no longer string-prefixes the raw preview against the editor's canonical markdown (the '*' vs '-' / emphasis mismatch that froze every append frame — Cursor). The mid-stream hold is now purely operation-based: `update` waits for settle; append/patch/create apply each frame via the (peer-safe) shadow reconcile. lastStreamedBodyRef is now a plain dedup guard, not a prefix baseline. Keeps the shadow, durable write, and SYNC_NO_PERSIST unchanged. * fix(files): elect a single agent-stream writer across tabs Cursor round 4 (High): with the stream applied client-side, two tabs/windows on the same chat could each derive streamingContent (the reconnect/resume path re-consumes preview events) and each independently insert the stream under a different Yjs clientID, duplicating content until the durable reconcile. Fix — single-writer election via the file-doc awareness (new agent-stream-leader): - a client applying an agent stream announces `agentApplying` on its own awareness - only the leader (min clientID among announcers) applies mid-stream AND at settle; a non-leader renders the leader's ops via Yjs and does not apply (a non-leader applying the final body would re-insert the whole doc as a duplicate) - re-checked each frame, so it converges to one writer the moment awareness propagates; the sub-frame startup race is reconciled by the durable write - single-client (the common case) is unaffected: it is the only announcer, so it always leads * fix(files): gate the settle apply locally, not on a settle-time re-election Cursor round 5 (High): the settle recomputed leadership from live awareness and the leader cleared its announcement immediately, so a straggler peer that settled afterward became the sole announcer, self-elected, and applied finalBody through its base-seeded shadow — re-inserting the whole doc as a duplicate. Fix: gate the settle apply on a LOCAL didApplyStreamRef (set only when this client actually applied a mid-stream frame — i.e. it was the mid-stream leader whose shadow is up to date), not on a settle-time re-election. A client that never applied (non-leader, a held `update`, or a pre-seed stream) skips the final apply and converges via Yjs + the durable write. The mid-stream leader election (isAgentStreamLeader) is unchanged, so exactly one client's didApplyStreamRef is ever true. * fix(files): open the agent-stream shadow lazily on lead (no stale handoff) Greptile round 6 (P1): the leader race — (a) a mid-stream leadership handoff could apply from a stale pre-stream shadow, and (b) two tabs starting the same stream before awareness converges could both lead briefly. - (a) fixed: the shadow is now opened LAZILY in the tick, only when this client actually leads, seeded from the CURRENT doc — so a handoff successor diffs against the prior leader's ops (never a stale base) and a non-leader builds no shadow at all. Announce candidacy via a dedicated ref (decoupled from the shadow); settle still gates the final apply on didApplyStreamRef (leader-only). - (b) the pure startup race is inherent to eventually-consistent election. It is now the only residual: bounded to two tabs starting the SAME stream within the awareness-propagation window, transient (converges in a frame or two), and never persisted (SYNC_NO_PERSIST + the durable edit_content reconcile). Resumes are sequential, so the common multi-tab case elects cleanly. Documented inline; a server-granted lease would close it fully but at a round-trip cost on the common single-tab path, which isn't worth it. * fix(files): idempotent settle apply (update lands client-side; no straggler dup) Cursor round 6 (Medium): a lone client's `update` never applied client-side — held mid-stream, then skipped by the didApplyStreamRef settle gate — so the rewrite depended entirely on the durable merge (stale if delayed/failed). Root cause was over-correcting round 5. Now that the shadow is opened lazily in the tick (current-seeded), the round-5 base-shadow duplication is already gone, so didApplyStreamRef is unnecessary. Replaced it: settle applies the final body via `agentStreamSessionRef.current ?? beginAgentStream(editor)` — the leader reuses its up-to-date shadow (last throttled frame), while a client that never applied (non-leader, held `update`, pre-seed) opens a FRESH current-seeded shadow. Reconciling current->final is idempotent: a straggler that settles after another wrote the final reconciles to a noop. So a lone `update` applies at settle (no wait on the merge), and there's still no settle-time election or base-shadow dup. * fix(files): broadcast agent frames to the whole room (same-socket siblings) Cursor round 7 (Medium): SYNC_NO_PERSIST frames applied under an origin carrying the sender socket id, and excludeSocketId dropped that whole socket from the relay fan-out. A second FileDocProvider on the same socket (chat preview + Files editor) then missed all mid-stream ops and stayed stale until the durable reconcile — a regression from the old no-origin server merge, which reached both. Fix: the agent origin is now a plain AGENT_SYNC_ORIGIN symbol, and agent frames broadcast to the WHOLE room (no socket excluded), matching the old behavior — so a same-socket sibling provider stays live; the emitting provider no-ops on its own echo (the ops are already applied locally). originSocketId still returns null for the symbol, so it keeps skipping edited/schedulePersist. Removed excludeSocketId and the socket-carrying origin object. Updated the relay test to assert the whole-room broadcast (verified it fails if the sender is excluded). * fix(files): tag agent stream frames no-persist across replicas A peer task tailing an agent-streamed preview frame previously applied it as REDIS_ORIGIN, marking the seeded room edited and making a transient startup-race duplicate eligible for that task's last-disconnect flush. Mark agent frames with a stream field so peers apply them as REDIS_AGENT_ORIGIN, excluded from the edited/persist gate. The copilot's durable edit_content write stays the sole authority over file bytes. * fix(files): reseed agent shadow on lead regain + agent-only compaction Two multi-writer edge cases surfaced in review: - rich-markdown-editor: a client that led, lost leadership, then regained it reused its stale shadow (which never saw the interim leader's ops), re-emitting ops for content already present. Tear the shadow down when a client observes it is not the leader, so a regain rebuilds fresh from the current doc. - file-doc-store: compaction always stamped its snapshot REDIS_SNAPSHOT_ORIGIN (marks peers edited). A long agent-only stream crossing the threshold could fold preview content into a persist-eligible snapshot. Track whether a room integrated any real edit and stamp an agent-only snapshot REDIS_AGENT_ORIGIN so it stays no-persist. Both covered by falsification-verified tests. * fix(files): close realEdited data-loss race + elect a settle writer Independent audit surfaced two real gaps: - file-doc-store: realEdited was latched AFTER appendUpdate's awaits, but the edit already sits in room.doc synchronously. A concurrent agent-frame compaction could read realEdited=false, snapshot that real content, and stamp it a no-persist agent frame — a lost edit. Latch it synchronously (same tick as the doc mutation) before any await. Deterministic falsifiable test added. - rich-markdown-editor: at settle every tab applied the final body, and a non-leader's local microtask runs before the leader's final propagates, so both insert the tail (Yjs keeps both) -> duplicated tail. Elect a single settle writer (reliable — awareness is long converged by settle), reading leadership before clearing the announcement. Corrects the overclaiming idempotency comment and the handoff pick-up comment. Adds a y-tiptap internals upgrade-guardrail test. * fix(files): own presence per client id, not one-per-socket The shared workspace socket hosts one collaborative provider per mounted view, so the chat file preview and the standalone Files editor for the same file each bind their own Yjs client id over ONE socket. The relay owned a single client id per socket, so the later JOIN overwrote the earlier and dropped its awareness — which silently broke the single-writer agent-stream election (a peer stopped seeing the streaming provider's announcement and could self-elect, duplicating streamed text for the whole stream). Track ownership per (socket, client id): a socket owns a set of client ids; the awareness gate accepts a frame only if every id it carries is owned; cleanup drops all of a socket's ids; the roster stays one-entry-per-session. Reclaim and the same-user reconnect path evict just the reclaimed id, dropping the old socket only if it empties. Falsification-verified test added. * fix(files): make streamed file-preview accumulation replay-safe Guard deriveFilePreviewSession against re-delivered/replayed content events: apply a delta/snapshot only when previewVersion strictly advances, so a client re-render or stream replay can't double-append the tail (the duplicated-content bug) or regress on an older snapshot. * fix(files): fix new-file collab streaming latch and agent-edit duplication - Latch collab readiness so a new file's post-seed `synced` flap can no longer re-gate agent streaming (the stream previously showed only the seed and the rest appeared only on reload) - Relay defers the durable edit_content merge to an actively-streaming client: the client shadow stream and the server merge were both writing the same content into the live doc, duplicating it when the server ran ahead - Render the collaborator caret bar out of flow so a peer caret never nudges the surrounding text by ~1px - Remove dead code: unused FileDocMessageType alias, unnecessary LiveFileDocMergeOrder export Covered by tests: readiness latch (flap/offline/latch cases), relay merge deferral (single- and multi-replica), plus verified-failing guards. * test(copilot): fix loadWorkspaceFileTextForPreview mock to return { text } not a bare string The adapter reads previewBase.text to seed an append/patch base; the mock returned a bare '' so previewBase.text was undefined, making a base-less append fail closed (no file_preview_content). My PR's fail-close change exposed the wrong-shaped mock. --------- Co-authored-by: mzxchandra <129460234+mzxchandra@users.noreply.github.com>	2026-07-31 18:48:12 -07:00
Theodore Li	4ef7bbad53	fix(db): index large-value workflow_id FKs so workflow deletes don't seq-scan (#6136)	2026-07-31 19:48:52 -04:00
Vikhyath MondretiandClaude	7798e83489	feat(function): custom sandboxes (#6071) ... * feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 16:33:35 -07:00
Waleed	4793607ee9	chore(db): drop the legacy folder tables and adopt the deferred folder_id FKs (#6051) ... * chore(db): drop the legacy folder tables and adopt the deferred folder_id FKs Contract migration closing out the generic-folders cutover: drops workflow_folder and workspace_file_folders, and adopts the two folder_id foreign keys 0272 deliberately deferred. Runs a final INSERT-ONLY reconcile first so no stranded legacy folder is lost by the drop. Insert-only because 0272 deliberately renamed 47 workflow folders to dedupe them, and an upsert would revert all 47. In production this is a no-op (verified 0 stranded); it exists for deployments that never ran the post-drain reconcile as an operational step, self-hosted upgrades above all. Hand-written rather than drizzle-generated: the generated form drops the tables BEFORE adding the FKs with no step to re-root unresolvable folder_ids, so one dangling id fails ADD CONSTRAINT after the legacy data is already gone; it also validates the FK inline, holding ACCESS EXCLUSIVE across a full scan of the 1.7M-row workspace_files, and uses DROP TABLE CASCADE. The generated snapshot is kept so drizzle-kit generate reports no schema changes. Every re-root path dedupes, because all three destinations are partial unique indexes keyed on a coalesced nullable column: folder, workflow, and workspace_files. Parents must also be reachable — an active row re-roots off a soft-deleted parent, while a soft-deleted row may keep one. Also hardens the retention sweep, which the new ON DELETE SET NULL exposes: a surviving active child re-rooted by the FK can collide at the workspace root, and chunkedBatchDelete turns that 23505 into a permanent per-chunk stall. The folder cleanup target now renames children first, covering workflows, files and subfolders, re-asserts eligibility so a restore mid-batch is not stripped, and treats the deduplicated name as a hint since both allocators can return a colliding one. * fix(db): stop the re-root dedupe renaming personal workflows workflow.workspace_id is nullable, and NULL is treated as EQUAL by PARTITION BY but UNKNOWN by the = in base_taken. So for personal workflows rn incremented across the group while no collision was ever detected: the dedupe could only fire spuriously, renaming a user-visible workflow that needed no rename, since the unique index treats NULL workspace_id rows as distinct anyway. The file block already carried the equivalent guard. Also gives step 1's inserts ON CONFLICT (id) DO NOTHING. It restates the stranded guard, closing the window between that read's snapshot and the index check — an operational re-run of 0274, or a live pod, committing into folder mid-statement would otherwise raise 23505, and migrate.ts retries only 55P03. 0272 and 0274 were already written this way; step 1 was the exception. Corrects three comments that claimed more than the code delivers: the header's 'no legacy folder is lost' (an id already present under another resource_type cannot be rescued), the reachability note (a cycle among stranded rows survives a per-row parent check), and a note made stale by the ON CONFLICT above.	2026-07-29 12:25:39 -07:00
Theodore Li	985f9e6172	feat(tables): saved views with filter, sort, and column presets (#5961) ... * feat(tables): saved views with filter, sort, and column presets * fix(tables): views own column layout, preserve deep-linked sort, seed update cache * fix(tables): merge view layout writes server-side, keep layout on save-from-All * fix(tables): send view saves as a merge patch so concurrent writes can't clobber * fix(tables): persist explicit All selection, prune view state for deleted columns * fix(tables): key-order-stable dirty check, reset layout when switching to All * fix(tables): return 404 when patching a missing view * improvement(tables): order the bar filter, sort, columns and drop the hidden count * fix(tables): record newly created columns in the active view's order * fix(tables): don't write layout as a reader, clear dead sort, reset dead view id * feat(tables): add New view to the views menu, starting from All * chore(api): bump route-count baseline to 981 after staging merge * fix(tables): guard default demotion, use the applied filter for dirty/save * fix(tables): clear state when the active view is deleted externally * revert(tables): drop the ineffective rows-query gate for view resolution * feat(tables): enable saved views in the embedded mothership table * fix(tables): resolve the views flag on the chat route too * improvement(tables): right-align the embedded run/stop control * fix(tables): live layout on new views, prune stored config, order cache writes * fix(tables): live layout on save-as-view, reset it per view, ignore inherited param when embedded * fix(tables): route undo/redo column-layout writes to the active view * fix(tables): scope layout undo to the view that recorded it Layout is view-owned, but UndoEntry carried no owner, so the persistence sink — rebound every render to the active view — decided the target at undo time rather than at record time. Reordering in view A then undoing from view B wrote A's column order into B and left A unchanged. Entries are now stamped with the active view id, and the three layout-bearing action types (create-column, delete-column, reorder-columns) are pruned from both stacks when the active view changes. Row and schema actions are table-scoped and survive the switch untouched. Inert when views are disabled: the id is always null, so nothing is ever pruned. * fix(tables): send only the layout keys an action changes Pin, reorder, and insert-column each shipped a full columnWidths snapshot they never modified. Both sinks merge at top-level key granularity, so an earlier-issued patch landing after a concurrent resize replaced the newer width map with its stale copy. Resize, auto-resize, and delete-column still send columnWidths — those genuinely change it. * fix(tables): don't strand layout writes while views load, keep schema undos across views Two more instances of layout being written without a known owner. The sink was left unbound until a view resolved, so a resize/reorder/pin (or the column-append effect) during the views fetch fell through to the table's shared metadata — corrupting All for a table about to adopt a view, and losing the edit to the re-seed. The sink is now bound while the query is in flight and suppresses the write. An error counts as settled, so a failed views fetch falls back to All instead of suppressing layout writes for the session. Pruning was also too broad: create-column and delete-column are table-scoped schema ops that merely have a layout side-effect, so dropping them on a view switch made a deleted column unrecoverable. Only reorder-columns is purely layout and still prunes; the other two survive and have just their layout half suppressed at replay when the recorded view isn't active. * fix(tables): flush layout buffered during load when the owner settles to All Suppressing the write while the views query was in flight stopped All from being corrupted, but nothing resolved the buffer afterwards, so a resize during load looked applied and vanished on refresh. Settling on All re-seeds nothing (viewLayoutKey never changed), so the gesture is still on screen and is now persisted to shared metadata. Adopting a view re-seeds the grid from that view and has already replaced the gesture on screen, so the buffer is dropped to match. * fix(tables): resolve undo layout ownership at write time, not dispatch time The layout writes for column create/delete happen in mutation success callbacks, and persistLayoutRef is rebound every render. Resolving entryOwnsLayout up front meant the guard could still hold from before a view switch while the sink it guarded already pointed at the destination, writing the recorded view's layout into whichever view was now active. Now a function, so the guard and the sink are read at the same moment: switch away mid-mutation and the schema half still lands while the layout half is dropped, matching the rule that undo only ever affects the view on screen. * fix(tables): read layout from the grid instead of mirroring it The wrapper kept two shadow copies of the grid's column layout — liveLayoutRef and pendingLayoutRef — and all three of this round's findings were that mirror going stale: - liveLayoutRef was only cleared on activeView.id change, so widths buffered before the views query settled survived into All, where layout writes bypass the mirror entirely. Both create paths spread it last, so a saved view stored those snapped-back widths. - currentViewConfig memoized a spread of that ref, and mutating a ref doesn't re-run a memo, so Save as view sent the pre-gesture layout. - The flush effect keyed on activeView, but adoption writes the view id through the URL, so for one render the query had settled while activeView was still null — flushing to All in exactly the case that had to drop. The grid owns this state, so it now publishes a reader through a sink ref and the wrapper asks at the moment it needs a value. Nothing to keep in sync, so nothing to go stale. Only whether an unowned change happened is tracked, and the resolve effect — which is what actually picks the owner — decides to persist or drop. * fix(tables): flush unowned layout when the views fetch fails The resolve effect is the only caller of resolvePendingLayout and gated on isSuccess, which never becomes true on a query error — so layout touched during the load window was never persisted on the error path, even though the table had already settled to All and later writes worked. The error branch now flushes to shared metadata, matching the rest of the error path's fall-back-to-All behavior. * fix(tables): gate the on-screen layout restore by ownership, not just the persist The undo success callbacks applied the recorded view's order/widths/pinning to the grid unconditionally and only gated the PATCH, so switching views before a column create/delete mutation resolved left the destination displaying the origin view's layout until switched away and back. Each callback now checks ownership where its layout work begins: three are purely layout and return at the top; delete-column undo restores cell data first (row data, runs everywhere) and gates only the layout block below it. In a non-owning view the restored column still appears via the grid's append effect — at the end, leaving that view's layout untouched. * fix(tables): route all layout writes through one owner-aware sink, reconcile order at seed Two holes closed: The sink binding toggled on activeView, leaving a render-frame gap after the views query settled but before the resolve effect adopted a default — writes in that gap fell through to shared metadata. The sink is now always bound while views are enabled and handlePersistLayout is the single router, reading the owner at call time: unresolved buffers, a view patches the view, All writes metadata. The error branch stamps the owner so post-error writes stop buffering. The append effect only fires when the schema changes, so a column that arrived while another source owned the layout rendered via the displayColumns fallback but was never written into the adopted owner's stored order until a drag happened to heal it. Seeding now reconciles the incoming order against the schema and persists the appended tail through the current sink. * fix(tables): capture the layout owner when a schema action is dispatched Insert-column and the delete chain persist layout from mutation callbacks through updateMetadataRef, which always targets the current sink — so a view switch mid-flight wrote the origin view's order/widths/pins into the destination. Undo got this guard already; the live paths never did. Both now capture viewLayoutKey at dispatch and compare at the callback. On a mismatch the layout work is skipped: the destination re-seeded its own layout, the new column lands there via the append effect when the refetch arrives, and the deleted column's dangling keys are pruned on read. pushUndo also takes the captured owner as an override — stamped at callback time it would record the destination, letting a later undo apply the origin's layout to it. * fix(tables): resolve views on list availability, not query success A failed background refetch flips isError while the cached list stays usable, and every view mutation invalidates the views query — so one blip made the resolve effect treat views as terminally failed and stop applying switches until the next successful refetch. The axis is now whether a list exists: error with no list ever fetched settles to All; error with a cached list resolves normally against the cache; owner is unknown only while the initial fetch is in flight. * ci: raise Build App timeout to 25 minutes The build outgrew the 15-minute cap over the last two days of merges: 10m02 (c6acc62a07), then 14m44 after the folders/desktop/library batch (adc557a2f2, 16s under the limit), then two consecutive timeouts on 7b1af4121b after the outlook merge. Staging's own latest runs show the same signature (one cancelled). GitHub labels a job timeout "cancelled", which is why these read as cancellations.	2026-07-29 14:57:58 -04:00
Waleed	a3413cf5bd	feat(folders): cut file folders over, and give knowledge bases and tables folders (#6045) ... * feat(folders): cut file folders over, and give knowledge bases and tables folders Builds on the generic resourceType-driven folder engine to finish the migration and extend it to two more resource trees. File-folder cutover - Repoints all 30 `workspace_file_folders` query sites onto `folder` scoped to `resourceType = 'file'`, id-keyed lookups included — a caller can hand a workflow folder's id to a file-folder endpoint, and only the predicate stops it - Adds the missing restore-conflict handling: a file folder whose name was taken while it was archived now deduplicates against the RESOLVED parent (restore re-roots when the original parent is still archived) instead of being permanently unrestorable - The forking mapper's `resourceType` predicate lives in the `leftJoin` condition, not the WHERE, where it would silently make the outer join inner - `servedFolderResourceTypeSchema` gains `file`; the soft-delete sweep enumerates all four types rather than dropping its predicate Knowledge Base and Tables folders - `knowledge_base.folder_id` and `user_table_definitions.folder_id` are now written and served; both create/move paths admit a folder only through `findActiveFolder`, scoped to workspace AND resourceType - Restoring a knowledge base whose folder is still archived re-roots it rather than filing it somewhere the page never renders; a workspace move re-roots for the same reason - Both pages get folder rows, breadcrumbs, inline rename on rows and on the breadcrumb, move-to submenus, pin/unpin, and drag-a-row-onto-a-folder Shared folder UI - One `components/folders/` module now backs Knowledge, Tables, and Files: `folderBreadcrumbItems` (data for `Resource.Header`, which owns the crumb chrome), `folderRow`, `folderRowId`/`parseFolderedRowId`, `buildMoveOptions` / `buildDescendantIndex` / `renderMoveOptions`, `nextUntitledFolderName`, `FolderContextMenu`, `useFolderNavigation`, `useFolderRowDragDrop` - Deletes the per-page copies: `files/move-options.tsx` and the Tables-local folder context menu Recently Deleted - Restores folders of every type, not just workflow folders, driven by one declarative table rather than a branch per resource Folder pins resolve against `folder` unscoped: file, knowledge-base, and table folders are all pinnable and all live there, and a folder id addresses exactly one row. * fix(folders): close the folder-engine audit findings Data loss - Migration 0274 catches up file folders created AFTER 0272. That backfill is guarded by `WHERE NOT EXISTS (… resource_type = 'file')`, so it fires once; every file folder created between it and this cutover exists only in `workspace_file_folders`. Without the catch-up they vanish from Files on deploy and their contents become unreachable Authorization - Archived folders were mutable, which bypassed the folder lock: the folder routes and `updateFolder` filtered id + resourceType but never `deletedAt`, while `getFolderLockStatus` does — so an archived-but-locked folder reported unlocked. Lock a subfolder, delete its parent, and any write-level member could rename and reparent it. PUT and the engine now require an active row; DELETE deliberately still does not, because it reuses an archived folder's own `deletedAt` to make a partial cascade retryable - `v1/admin/workflows/import` wrote `folderId` with no validation at all. Migration 0272 dropped the FK, so it accepted a knowledge-base folder, another workspace's folder, or garbage — and the workflow then rendered under no sidebar node while still executing, billing, and escaping the folder delete cascade. It now runs the same `assertFolderInWorkspace` + `assertFolderMutable` pair as `v1/workflows/import` Restore no longer loses state - A folder restore ran a bare `archivedAt = null` over its workflows, while the delete had routed through `archiveWorkflow` — which also disables schedules, webhooks, chats, and MCP tools. Restoring a folder reported success while every schedule stayed disabled forever and every chat 404'd. The workflow tree now restores through `restoreWorkflow`, like the knowledge-base and table trees already did. Deployment state is still deliberately not revived, matching the single-workflow restore - Those canonical restores re-root a resource whose folder is archived — correct on its own, catastrophic inside a cascade, which restores children BEFORE the folder rows and would therefore dump every child at the workspace root. `resolveRestoredFolderId` takes the subtree being restored and exempts it. `restoreTable` gained the re-root check it lacked Correctness - `getFolderPath` in `lib/folders/tree.ts` walked `parentId` with no `visited` set, so a cycle in the client cache — reachable through `useReorderFolders`' unchecked optimistic write — hung the tab. Its twin already had one. Dead `buildFolderMap` deleted - The sidebar's "New folder" inside a folder hardcoded the name, so the second one 409'd, was swallowed with only a log line, and the user got no folder and no message. It now names through the existing `generateSubfolderName` and surfaces failures - `chunkedBatchDelete` deleted by id with no re-check, so a resource restored between the SELECT and the DELETE was hard-deleted anyway. Callers now re-assert their soft-delete predicate on the DELETE - Recently Deleted and the `restore_resource` tool cover every folder tree, not just workflow folders, so deleted knowledge-base and table folders are recoverable Cleanup - The folder duplicate route stops steering control flow through `throw new Error('literal')` matched by string, reuses the engine's `nextFolderSortOrder` instead of recomputing it, and names its workflow scope once - `servedFolderResourceTypeSchema` documents that its default applies only to an omitted value; a present-but-unrecognized one is rejected, never coerced to `workflow` - The `workspace_file_folders` comment described the table as unread and unwritten and invited a DROP. It was live until this cutover, and it is now the rollback copy — the comment says so, and names what must be true before anyone drops it - Pinned rows carry a small non-interactive pin glyph again; they sort to the top and had nothing explaining why since the inline pin button was removed Tests - New `lib/folders/lifecycle.test.ts` covers create/update/delete/restore, the re-root, and the 23505 mappings; `cleanup-soft-deletes.test.ts` now pins the folder target's resourceType predicate, which is what keeps the sweep off a not-yet-cut-over type * improvement(folders): show the pin glyph on Files rows and pin the deploy contract Files rows were left without the pin indicator the other two lists gained, so a pinned file or folder still sorted to the top of the Files page with nothing explaining why. Adds `lib/api/contracts/folders.test.ts`, which pins the three cases that together ARE the rolling-deploy contract: an omitted `resourceType` defaults to `workflow` (so a client predating the field keeps working), a present-but-unrecognized one is rejected rather than coerced, and every served tree is accepted. * fix(folders): close the findings from the full-diff audit Migration 0274 reconciles instead of catching up - The deployed manager writes `workspace_file_folders` EXCLUSIVELY, so `folder`'s file rows are frozen at the 0272 snapshot: every rename, move, delete, and restore since then lives only in the legacy table. An insert-only catch-up left all of that diverged — renames reverted, deleted folders came back as active phantoms, and a stale-active row could hold the name a newer folder legitimately took - That last case was silent: the bare `ON CONFLICT DO NOTHING` swallows a NAME violation as readily as an id one, so the newer folder was discarded and every file inside it stranded. The conflict target is now `(id)`, so a name collision — which would mean the source violated its own unique index — fails loudly instead - Reconciles by parking mirrored names first, then upserting, so no transient state can collide with the partial unique index. Both statements are idempotent, and the header says what nothing else would: this has to be run again after the deploy drains, because old pods keep writing the legacy table until they are gone `/api/folders` no longer serves file folders - Adding `'file'` to `servedFolderResourceTypeSchema` opened a second writer over the same rows. It bypassed the `workspace_file_folders:${workspaceId}` advisory lock that makes the manager's check-then-write pairs atomic, and it accepted names containing `/`, `\`, `.` and `..`, which the file surface forbids because a folder name becomes a path segment — a folder named `a/b` is then unreachable by path forever, and breadcrumbs resolve it as two segments. The storage cutover never needed a second API, so there isn't one Conflicts that returned 500 - `v1/admin/workspaces/[id]/import` created its folder with an unserialized SELECT-then- INSERT, so two imports sharing a folder path failed the whole import. The name is server-chosen, so it now adopts whichever folder won, like `ensureWorkspaceFileFolderPath` - Folder duplicate never caught 23505. `newId` is client-supplied, so replaying a duplicate whose response was lost hit the primary key and returned 500 instead of a 409 the caller can act on Performance - `knowledgeKeys` moved to `hooks/queries/utils/knowledge-keys.ts`. Reading it from `hooks/queries/kb/knowledge` pulled a ~1000-line hook module — and the `@sim/emcn` barrel behind it — into `hooks/queries/folders`, which three server prefetches import. That is the same import edge the navigation-speed audit measured at 11.8MB per workspace route; `tableKeys` already lived in a keys-only module for exactly this reason Merge drop - Knowledge never got the chrome Tables did: its prefetch now fetches the folder tree alongside the bases, so the list does not paint ungrouped and a `?folderId=` deep link does not render an empty breadcrumb, and its loading fallback declares the "New folder" action so the header does not shift on hydration Comments corrected to match reality - The shared folder row and context menu claimed Files as a consumer; Files still builds its own row and routes folders through `FileRowContextMenu`, which the TSDoc now says - `schema.ts` prescribed a row COUNT comparison before dropping the legacy table. The divergence 0274 repairs is in row CONTENTS, which a count check passes straight through * fix(folders): address the review round and the full-PR audit Reverts the workflow restore hook — it was the wrong fix - Greptile flagged that restoring a folder leaves schedules disabled and webhooks/chats inactive. The hook added last round routed workflows through `restoreWorkflow` to address it, but `restoreDependents` ALREADY clears exactly those columns, in bulk, inside the restore transaction and matched on the archive timestamp. The hook bought nothing and cost a per-workflow read/transaction/read OUTSIDE that transaction — roughly 1600 round trips for a folder of 200 workflows — plus a window where the workflows were active and the folder was not, and it made `restoreDependents` dead code - What no restore path can undo is the state archive OVERWRITES: `status: 'disabled'` with `nextRunAt` cleared, `isActive: false`. Archive does not record what those were, so restoring them to a constant would re-enable a schedule the user had disabled and re-run a completed one. Left explicit — redeploy re-activates a schedule — matching deployment state, which restore also deliberately leaves off. Documented on the config rather than guessed - Kept the genuine bug the review surfaced underneath it: `restoreWorkflow` un-archived every dependent by `workflowId` alone, resurrecting a webhook or chat the user had archived days earlier. Now matched on the workflow's own `archivedAt`, which is the semantics this feature documents Bugbot: orphaned knowledge bases vanished from every view - Knowledge filtered on a raw `folderId === currentFolderId` and never re-rooted a base whose folder is missing or archived, so a base restored on its own — or left behind by a partial cascade — was invisible everywhere. Tables already fell those back to the root A dead `?folderId=` is now healed instead of being a dead end - A bookmark to a deleted folder rendered the root title over an empty grid, hid everything actually at the root, and — worse — left every create/upload action targeting the dead id, filing new resources somewhere nothing could reach. `useFolderNavigation` clears it once the folder list resolves, which fixes Files, Knowledge, and Tables at once Parity gaps found by auditing against Files - Tables sorted folders newest-first on a clean URL while Files and Knowledge sorted A→Z: its sort params are defaulted, so the raw column was never null. Reads `activeSort` now - On Tables you could not delete the folder you were standing in — its own row is not in the list, and the crumb menu offered only Rename, which also made the step-out branch unreachable dead code - A failed knowledge-base move was logged and never surfaced, so a rejected move — from the submenu or from a drag — looked like nothing happened - A drag begun in another mount of the page could never be dropped: `onDragOver` bailed before `preventDefault`, so no drop event ever fired and the `dataTransfer` fallback in `onDrop` was unreachable. External/foreign drags are still ignored, so an OS file dropped on the list cannot navigate the tab away Files: a folder cycle crashed the page - The folder-size roll-up recursed with no `visited` guard, so a parent/child cycle in the cached tree — reachable through the optimistic folder-move write — recursed until the stack blew and the page went blank. Also indexes children once instead of re-scanning per node Migration 0274 is now atomic - 0272 ends with an embedded COMMIT for its CONCURRENTLY index builds, so this file is not guaranteed to run inside drizzle's batch transaction. Wrapped in a DO block so the parking pass cannot commit without the reconcile, which would leave every mirrored folder holding a placeholder name - Validated with a read-only dry run: no active name duplicates, no orphaned or cross-workspace parents, and no id collisions with `workflow_folder`, so the insert cannot trip the unique index, the FKs, or the resource-type trigger Smaller corrections - Knowledge indexed its members for the owner comparator instead of two linear scans per comparison, and logs a load failure from an effect rather than the render body - Files uses the shared root sentinel instead of a bare `'__root__'` in two places - `restore_resource` documents that its two new folder types are unreachable until the enum in the copilot service's tool catalog — a different repository — widens - Corrected a forking comment that still described file folders as a separate table * docs(folders): record the 0272 + 0274 dry-run results on the migration A read-only dry run materialises the complete post-0272 `folder` table from both source tables and asserts every constraint it declares: no NULL names, no primary-key collisions between the two source tables, no unique-index violations, no resource-type or workspace trigger violations, and no parent/user/workspace FK violations. 0272's dedup renames only workflow folders, never file folders — the latter is what lets pass 2 write raw names. * fix(folders): close the findings from the line-by-line audit of the PR Data integrity - Workspace fork copied `folderId` verbatim onto the child's tables and knowledge bases. The column carries no workspace, so a forked row pointed at a folder owned by the SOURCE workspace — invisible in the fork, and mutated from under it when the source later deleted that folder (`ON DELETE SET NULL`). Latent until now because nothing wrote a non-null `folderId` for those two resources; this PR is what makes it reachable. Forked resources land at the root, like forked files already did Interaction bugs - Clearing a dead `?folderId=` wrote through the params' default `history: 'push'`, so Back returned to the dead URL, which healed and pushed again — the Back button was dead on that page. It replaces now: opening a folder is navigation, correcting a URL that never pointed anywhere is not - That same heal keyed off `isLoading`, which is false for a DISABLED query, false for an ERRORED one, and false while `keepPreviousData` is showing the previous workspace's folders. In all three the list is empty or stale, so a transient fetch failure — or a workspace switch — threw away a perfectly good open folder. Gated on `isSuccess && !isPlaceholderData` - "New folder" while a search was active created the folder but never showed it: the search filters folders too, so the new row did not match, the rename field never appeared, and the create read as a no-op. Both pages clear the search so the thing just created is on screen - The drag ghost was removed only by `dragend`, which fires on the source row. The table is virtualized, so scrolling the source out of view mid-drag unmounted it, the event never arrived, and the ghost stayed on the page with every row stuck at drag opacity. Cleaned up on unmount as the backstop - A drag begun in another mount highlighted every folder as a valid target — including the dragged folder itself — then silently did nothing on drop. It only highlights when the source is known and was actually checked Correctness - The folder-duplicate route caught 23505 across the WHOLE handler, so a unique violation raised while copying the workflows inside the folder was relabelled "a folder with this name already exists". Scoped to the folder INSERT, which is the only place the two conflicts the caller can act on (client-supplied `newId` replay, concurrent name take) actually arise - The optimistic folder create derived its sort order from the workspace's WORKFLOWS regardless of resource type. Only the workflow tree interleaves folders and resources in one ordering space, so a knowledge-base or table folder was placed against an unrelated one - A table archived between `checkAccess` and the move surfaced as a 500 with a "not found" body; it is a 404 - `FOLDER_RESOURCE_TYPE_BY_RESTORABLE` was a `Partial` record, so adding a folder tree without a mapping would compile and silently restore into the workflow tree. Total record now - Recently Deleted paired query results to folder trees by ARRAY POSITION; reordering the const would have filed knowledge folders under Tables with no type error. Keyed by type - The knowledge move's no-op guard compared against the snapshot taken when the menu opened rather than the live row Performance — the module split now actually does what it claimed - `knowledge-keys.ts` justified itself as keeping server prefetches off the ~1000-line `kb/knowledge` module, but `knowledge/prefetch.ts` still imported its stale-time constant from exactly that module, whose first line pulls the `@sim/emcn` barrel. Worse, this PR had ADDED the same class of edge to four prefetches via `FOLDER_LIST_STALE_TIME`/`mapFolder` - `FOLDER_LIST_STALE_TIME`, `mapFolder`, and `KNOWLEDGE_BASE_LIST_STALE_TIME` now live beside their key factories, so all four server prefetches import only keys-and-constants modules. `mapFolder` keeps its explicit field list rather than a spread, so the cached shape cannot drift from a client fetch Honesty in comments - The KB retention `deleteFilter` narrows the restore race but does not close it — documents hard-deleted by `onBatch` do not come back, so a restore landing mid-batch returns an emptied base. Said so - A failed folder restore leaves children active while their folders are still archived. They are reachable (both lists fall an unresolvable `folderId` back to the root) but they sit at the root until the restore is retried. Said so Product policy made visible - Restore deliberately does not re-enable schedules, webhooks, or chats, because archive overwrites their state without recording it. A deliberately-disabled schedule or webhook is a common state rather than an edge case, so reactivating to a constant would be wrong more often than right. Recently Deleted now says so at the moment of restore instead of leaving it to be discovered Smaller - Pin glyph: dropped the doubled gap, added `role='img'` so the state is announced - "Move here" carries the folder glyph its sibling leaves do - `useMoveTable` had stolen `useDeleteTable`'s doc block - Dead `.returning` column and a `?? null` on a non-nullable param in `moveTableToFolder` - Tables reads one sort source for both blocks, matching Knowledge - New test pins that `updateFolder` refuses an archived folder — the guard that stops a delete from unlocking its locked subfolders * fix(tables): re-read live placement before skipping a move as a no-op `handleMoveTable` and `handleMoveFolder` decided "already there, nothing to do" from `activeTable`/`activeFolder`, which are snapshots taken when the context menu opened. A refetch or a concurrent move since then makes that comparison stale, so the write the user just asked for is silently skipped and the row stays where it was. Both now compare against the live list, matching the knowledge-base move. * chore(folders): drop production figures from migration and code comments This repository is public. The 0274 header and the Recently Deleted policy comment carried concrete counts and ratios taken from the production database, which is operational detail that does not belong in open source. Both now state the property that matters — what was asserted, and why disabled-on-restore is the safe default — without the underlying numbers. * fix(folders): close the findings from auditing the whole release landing unit Audited as the unit that actually ships — #6014 (pinning), #6025 (workflow folders onto the generic table), #6037 (the engine) and this branch together against `main`, rather than this branch against `staging`. Authorization - `PUT /api/folders/reorder` still operated on archived folders. `getFolderLockStatus` skips archived rows, so `assertFolderMutable` there was a guaranteed no-op — meaning a locked folder became freely reparentable the moment its parent was deleted. This branch closed exactly that hole on `PUT /api/folders/[id]` and left its sibling open, then widened reorder to three resource trees. Reordering an archived folder is also wrong on its own terms: `collectArchivedSubtreeIds` walks the cascade by parent, so moving a branch out of an archived subtree silently drops it from that folder's restore Data loss in the purge - `batchDeleteByWorkspaceAndTimestamp` forwarded its eligibility predicate only into the SELECT, never the DELETE. The `folder` target runs through it, so a restore committing between the two statements had its folder hard-deleted anyway — taking the placement of the children the restore had just brought back. The workflow purge re-checks for precisely this reason and the knowledge-base sweep gained the same guard earlier in this branch; `folder` had neither. The wrapper now re-asserts eligibility on the DELETE for every caller Wire boundary - `toPinnedItemApi` had no return type, so `pinned_item.resource_type` — plain `text` by design, against a closed contract enum — was never checked. Annotating it surfaced the real hole immediately: during a rolling deploy an older pod can read a pin a newer one wrote, and returning it would fail response validation and take the WHOLE list down rather than the one row. Unknown kinds are now narrowed out explicitly at the boundary instead of surviving by accident on a filter whose stated job is something else Interaction - The knowledge-base FOLDER move still compared against the snapshot taken when the menu opened. The resource move on that page and both Tables handlers were fixed earlier in this branch; this was the last one Operational - 0274's "re-run after drain" instruction needed a precondition. Cleanup hard-deletes from `folder` but never from `workspace_file_folders`, so a re-run after a purge would reinstate every purged file folder as a soft-deleted phantom whose files are already gone. Retention is far longer than any drain, so running it promptly is sufficient — but the instruction said nothing about ordering Accuracy - Four comments named symbols that do not exist or no longer apply: `useFolderCreateWithDedup` (never existed — it is `nextUntitledFolderName`), `collectActiveSubtreeIds`, `restoreFolderCascade` as the live restore path, and a claim that a server `createSearchParamsCache` reads the shared folder param. The shared param doc also now admits Files declares its own `?folderId=` rather than claiming to be the single declaration - `FOLDER_RESOURCES.file` is unreachable at runtime — `servedFolderResourceTypeSchema` does not serve `'file'` — and a reader would reasonably assume otherwise. Says so, and says what routing file folders through the generic engine would bypass - `pinnedItem.resourceType`'s inline comment was missing `'folder'` - `folders.test.ts` fixtures still carried `color` and `isExpanded`, both deliberately dropped from the generic table — pre-consolidation rows masquerading as folders - `getFolders` in the folder cache had one caller, in its own file * fix(folders): gate the orphan fallback on a resolved folder index, not a loading flag The URL heal was fixed to use `isSuccess && !isPlaceholderData`, but the list filters that decide "this resource's folderId does not resolve, so show it at the root" were left on `isLoading` — the same footgun, one layer down. `isLoading` is false for a disabled query (no workspaceId), false for an errored one, and — because `useFolders` sets `keepPreviousData` — false while the previous workspace's folders are still on screen during a switch. In all three the index is empty or belongs to another workspace, so every foldered knowledge base and table was treated as an orphan and dragged to the root, or filtered out of a deep-linked folder entirely. A transient fetch failure was enough to make a folder look empty. `useFolderNavigation` now exposes `foldersResolved` instead of `isLoading`, so the heal and both list filters share one signal and the footgun is no longer reachable from the hook's surface.	2026-07-28 23:11:22 -07:00
5 people	1d64b92b41	feat(desktop): desktop app (#5998) ... * top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai>	2026-07-28 19:25:59 -07:00
Waleed	cb3611bad2	feat(folders): add resource pinning and generalize the folders contract (#6014) ... - Adds per-user pinning for workflows, files, knowledge bases, and tables: new `pinned_item` table, `/api/pinned-items` routes, React Query hooks, and a shared `PinButton` wired into Tables, Knowledge, and Files with pinned-first ordering - Adds the generic `folder` table with an idempotent, replay-safe, collision-aware backfill from `workflow_folder` and `workspace_file_folders`; no cutover yet, folder reads still go through a documented legacy adapter - Moves `folderSchema` to the generic vocabulary (`resourceType`, `deletedAt`) and drops the unused `color`/`isExpanded`	2026-07-28 14:42:11 -07:00
Waleed	02311cafb9	perf(db): stop scanning every workspace file on workflow archive, cover the billing period aggregate (#6015) ... * perf(files): stop loading every workspace file to archive a workflow's backing files `cleanupWorkflowAliasBacking` runs on every workflow delete. It loaded every `context='workspace'` row for the workspace — active and soft-deleted, all columns — then discarded >99% of them in JS to find the handful of `.changelogs/<workflowId>.md` and `.plans/<workflowId>/**` rows it owns. In production this was the single worst query at 37.83% of total database runtime: p50 6s, p95 9s, max 13.1s, 641 calls/day, 34.1M rows read. It is index-served, so the cost is not a missing index — it is 59,061 rows and ~630MB of cold buffers read to archive a few files. A file's `folderPath` is derived solely from its `folderId`, so matching on folder membership is equivalent to the path comparison it replaces. The load and JS filter become one targeted UPDATE keyed on a handful of folder ids. Folders that are soft-deleted are still included when resolving which files a workflow owns: path resolution ignores `deletedAt`, so a live file parented to an archived folder previously matched and must continue to. Also: - Add `getWorkspaceShares`, replacing an id-list `IN` clause that grew with the file count (59,061 elements on the worst workspace) with one indexed lookup on `workspace_id`. Callers read the map by id, so a superset is equivalent. - Drop `all` from the client-reachable file scope enum. It drops the `deleted_at` predicate and so cannot use the partial index serving the other two. No client requests it; server callers reach that scope directly. - Set `fetch_types: false` on the app and realtime pools. postgres.js otherwise runs a blocking `pg_catalog.pg_type` roundtrip before each new connection's first query — 95,722 of them per day. It builds array parsers only; Drizzle already parses this schema's two `text[]` columns itself. * perf(nav): add route loading boundaries and cover the billing period aggregate Dynamic routes prefetch only down to the nearest loading boundary, and the server stops prefetching at the first one. With no loading.tsx anywhere in the workspace tree and a dynamic layout, Link prefetch was yielding almost nothing and every navigation waited on a full server round trip with no feedback. Add loading.tsx only where the fallback is provably what renders today, so perceived speed improves without changing what users see: - home, chat/[chatId]: reuse HomeFallback, already each page's own Suspense fallback. - integrations, skills: reuse the tab-header chrome, byte-identical to each page's own Suspense fallback. Deliberately not added to workspace root, settings, or w, whose pages are redirect-only or already self-fallbacking — a boundary there would paint a skeleton that does not match the destination. Billing: - Add usage_log_billing_period_cost_idx, trailing the remaining predicate columns and `cost` so the period aggregates resolve index-only. Confirmed against production: the aggregate currently runs as an Index Scan touching 478,559 buffers because `cost` is absent from the existing index. That index is superseded but left in place; dropping it is a separate migration so a planner regression costs nothing to revert. - Collapse the two aggregates behind /api/billing into one scan using SUM(...) FILTER. Besides halving the work on a nav-path query, it removes a latent inconsistency: as separate statements the two sums could observe different snapshots, making the copilot subset exceed the total. Caching was considered and rejected for the usage aggregate. Its callers include usage enforcement, threshold billing, and overage calculation, where a stale-low read permits overspend and a stale-high read double-charges. * test(files): cover cleanupWorkflowAliasBacking and correct a stale share mock cleanupWorkflowAliasBacking had no test coverage, and the rewrite that replaced its load-everything-then-filter body rests on a subtle equivalence: a file's folderPath is derived solely from its folderId, and path resolution ignores deletedAt. The second half is the easy part to get wrong — a live file parented to an archived folder still resolves to a backing path and must still be archived, so folders are filtered by deletedAt only when choosing which folders to archive, never when deciding which files the workflow owns. These tests pin that distinction: they fail if the archived-folder case is dropped from file ownership, and separately assert that archived folders stay out of the folder update and that unrelated workflows are never touched. Also point the workspace files route test at getWorkspaceShares. Its mock still named getSharesForResources, which the route no longer imports. The suite passed regardless because all three cases exercise the upload path, so the GET listing has no coverage — but the stale name would have handed the first GET test an undefined function. * fix(perf): drop the loading boundaries and correct the usage_log index order Adversarial review found real regressions in both. Route loading boundaries — all four removed: The premise in their TSDoc was wrong. Each page's existing `<Suspense>` exists so nuqs can prerender; `useSearchParams` only suspends during SSR, so on a client navigation those fallbacks never painted. Hoisting them to loading.tsx did not "show the same frame" — it made a previously invisible blank frame visible. For chat, Next keys the Suspense boundary by cache key, so a chatId change mounts a fresh suspended boundary and always commits its fallback. Switching chats would have gone from "previous chat stays on screen" to a blank surface for the whole RSC round trip, on the highest-frequency navigation in the product. Partial prefetch only warms the fallback, so no latency was saved to offset it. The integrations and skills fallbacks were correct for their own pages but also became the fallback for four detail routes that render different chrome, inserting a wrong intermediate frame. Fixing that needs per-child boundaries and new skeleton UI that cannot be verified without a browser, for pages whose only server work is `await params`. Not worth it. Every remaining loading.tsx in this app paints real chrome. A blank one was against the grain, and the measurable win was zero. usage_log index: Column order was wrong. The daily-refresh rollup filters entity type, id and period_start but NOT period_end, so putting period_end fourth ended the usable prefix at column three and left user_id and created_at as in-index filters rather than scan boundaries — turning a ~2.2k-entry bitmap scan into a ~266k-entry scan. Verified in production that period_start functionally determines period_end (13,612 groups, zero with more than one end), so the slot bought no selectivity. user_id and created_at now follow the shared prefix; period_end rides as payload. Also corrected the claim that this supersedes usage_log_billing_entity_period_idx. That index deduplicates to 17 MB across 1.42M entries because it has no high-cardinality key column, which is what keeps prefix-only bitmap scans cheap. It is retained deliberately, not pending a drop. Harden cleanupWorkflowAliasBacking: gate the UPDATE on the ownership filter list itself. `and()` and `or()` both drop undefined, so a clause that resolved to nothing would have left a WHERE of workspace + context + not-deleted and archived every file in the workspace. Tests now assert no UPDATE is issued when the workflow owns nothing, plus the filename and context predicates. Note fetch_types also disables array serializers, not just parsers; documented. * docs(db): correct the fetch_types constraint note after differential testing Ran both settings against a real Postgres with the schema's actual text[] columns. Drizzle-typed selects and .returning() are byte-identical either way; only a raw db.execute projecting an array column differs, yielding the wire form. The previous note also claimed a raw JS-array bind fails because the serializers come from the same catalog fetch. It does fail — but under both settings, because Drizzle expands an array into a row constructor before postgres.js ever sees it. That is unrelated to this flag, so the claim is removed rather than left implying a constraint this change introduces. * chore(db): generate the drizzle snapshot for the usage_log index migration 0271 was hand-written, so drizzle-kit's state never learned about the new index — the next `generate` would have re-emitted it as a fresh migration against an already-migrated database. Ran `drizzle-kit generate` to produce the snapshot, then restored the CONCURRENTLY form: drizzle emits a plain CREATE INDEX, which takes an ACCESS EXCLUSIVE lock and would block writes on a 4M-row table for the duration of the build. The generated column order matched the hand-written SQL exactly, which also confirms schema.ts and the migration agree. `generate` is now a no-op, and check:migrations still passes.	2026-07-28 12:39:10 -07:00
Theodore Li	0dcbc56ef6	feat(tables): per-table mutation locks (schema/insert/update/delete) (#5960) ... * feat(tables): per-table mutation locks (schema/insert/update/delete) Adds four independent, admin-only locks to a table so a workspace can make it append-only, read-only, or schema-frozen. Enforced at the lib/table service layer rather than the routes, so the API, workflow blocks, and Mothership (which calls the services directly) are all covered by one assert. Violations return 423; changing a lock requires workspace admin and returns 403. * fix(tables): address review — CI snapshot, lock-aware jobs, UI gating - Add the drizzle meta snapshot the hand-written migration was missing, which made CI regenerate the lock columns as a phantom 0271 - Stop a queued or retried delete/update job that starts after its lock is enabled; a run that has already committed pages still finishes, since pages are never rolled back and aborting would leave an uncancelled half-done state - Map TableLockedError to 423 on the internal single-row DELETE (was a 500) - Gate the lock settings UI behind NEXT_PUBLIC_TABLE_LOCKS so it can't open onto a Save that 403s against the server-side flag - Respect the delete lock on column drops in the grid, matching assertColumnDestructive, instead of failing only on click * fix(tables): close lock gaps in async import and undo/redo - Assert insert (and delete for replace) at the start of the import worker. Replace deleted every row before the first insert assert, so an insert-locked table was wiped and then failed, leaving it empty. - Run the delete/update worker start checks through assertRowDelete / assertRowUpdate instead of reading the lock flags directly, so the worker and its enqueue site apply identical rules — including the workflow-column exemption a bulk update was being cancelled despite. - Make undo/redo verb mapping direction-aware for column ops: dropping or retyping a column needs the delete lock clear too, matching assertColumnDestructive, so those steps are skipped instead of stranded. * fix(tables): stop locks over-blocking workflow runs, duplicate, and imports - Run / Re-run / Stop no longer inherit the cell-edit gate: they write only workflow-output columns (which the update lock exempts) and Stop is a cancel - Duplicate needs only the insert lock — it inserts a full copied row, so it stays available on an append-only table - updateWorkflowGroup asserts the destructive rule only when a patch actually drops or remaps output columns; rename / autoRun / mapping edits need just the schema lock - Gate the import-async enqueue on insert (and delete for replace) so a locked table 423s instead of claiming the write-job slot and failing in the worker - Disable Import CSV under an insert lock, and wire the expanded cell editor and the previously-unused stable add-row handler to the lock-aware flags * fix(tables): keep column menu usable and let locks always be cleared - Pass the blocked-delete handler instead of undefined so ColumnOptionsMenu still mounts on a delete-locked table; withholding it hid the entire menu, including Insert column, which the delete lock must not affect - Restore ColumnHeaderMenu readOnly to permission-only: it swaps the header for a static label, which was also disabling pin, column-select and open-config - Assert the schema lock at the import-async enqueue when createColumns is set - Allow a locks PATCH that only clears locks even when the feature flag is off, and keep the settings entry reachable on a locked table, so flipping the kill switch can't strand a table with locks nobody can remove * fix(tables): scope the update-lock exemption and re-check locks per page - Make the workflow-output carve-out opt-in via `computedWrite`, set only by the cell-write path. It was caller-agnostic, so an ordinary API caller could PATCH a workflow-output column on an update-locked table - Re-read the lock before every page in the delete and update runners. The single worker-start check went stale immediately, so enabling a lock could not stop a job already deleting or overwriting rows. Committed pages still stay committed, exactly as with an explicit cancel. Import keeps its one-shot check: a half-imported table needs the deletes the lock now forbids - Route Insert column to the locked-action modal under a schema lock instead of leaving it live (regular header) or hiding the whole menu (group header) * fix(tables): revalidate locks in-transaction and unbreak backfill + partial unlock - Re-assert the lock inside each page-write transaction, under the same advisory lock updateTableLocks takes, so check-then-write is atomic against a lock change. The per-page check alone still left the page-selection await between the assert and the write - Pass computedWrite from the backfill runner: batchUpdateRows is the workflow-output backfill path, so scoping the exemption had made rebuilding outputs 423 on an update-locked table while live cell writes still worked - Gate the feature flag on a lock actually going off->on rather than on an all-false payload. The settings UI always submits all four flags, so with the flag off an admin could not clear one lock while another stayed on - Hoist the lock-kind -> flag map to lib/table/types as TABLE_LOCK_FLAGS * fix(tables): guard import batches in-transaction and split paste by verb - Re-assert the insert lock inside each import batch's insert transaction, under the same advisory lock updateTableLocks takes. Reversing the earlier "let the file finish" call: an admin can lift the lock to clean up a partial import, so honouring it beats letting the rest of the file land - Gate paste per verb. Overwriting existing rows is an update and extending past the last row is a full-row insert, so a paste-append still works on an append-only table while an overwrite explains itself. Refuses the whole paste rather than applying half of it - Space opens the same row editor as double-click, so it now follows the update lock instead of filling in a form that 423s on save * fix(tables): guard the remaining import writes and explain blocked Shift+Enter - Route the replace-mode wipe, the inferred-schema write and createColumns through the same guardBatch transaction as the batch inserts, so a delete or schema lock committed while the file is downloading or being sampled is seen instead of the job-start snapshot - Shift+Enter takes the manual-add path, so it now opens the lock modal like the Add row button instead of returning silently * improvement(tables): surface blocked lock actions as a toast, not a modal - Replace TableLockedModal with a warning toast carrying a "Lock settings" action button for admins. Being told you can't edit shouldn't cost a dismiss click, and the button still routes admins straight to the panel - Dedupe by id so repeated attempts on a locked cell replace one notice instead of stacking a column of them - Move the copy into lock-copy.ts alongside the rest of the lock vocabulary and tighten it for toast length * fix(tables): don't show the update-lock notice to users without write access Double-click checked the update lock before any permission check, so a read-only member on a locked table got "Editing rows is locked" — misleading (the lock isn't why they can't edit) and it swallowed the expanded-cell viewer, which is a legitimate read-only affordance. * improvement(tables): trim the update-lock toast copy * fix(tables): map append-import locks to 423 and stop conflating locks with permissions - The sync append branch returns instead of rethrowing, so the outer catch's mapper never saw a TableLockedError and every lock violation became a 500. Map it in that catch, with a regression test (replace mode already rethrows) - Stop mounting the workflow-group column menu for users without edit access: passing the blocked handlers unconditionally made it appear for read-only members and report a lock even on an unlocked table - Enter/F2 now raises the same lock notice as double-click and Space instead of silently doing nothing - guardBatch returns the freshly-read definition so addTableColumnsWithTx asserts live state; a schema lock cleared mid-import no longer fails the createColumns step. The snapshot pre-asserts now only run as the fallback for callers that pass no revalidator - Append-only no longer labels a table whose schema is also locked, which claimed columns were mutable when they weren't - Import dialog withholds Replace on a delete-locked table and create-column on a schema-locked one, instead of offering a configuration that only 423s * fix(tables): revalidate sync imports under the advisory lock, explain every blocked key - The sync append/replace paths asserted only the request-start snapshot, so a lock committed while the CSV was parsed still let the write through. Both now re-read under the schema advisory lock at the top of their own transaction, taken before acquireRowOrderLock so the order stays advisory -> rows_pos -> definitions - Delete/Backspace, Cmd+D, typeahead and cut raised no notice on an update-locked table; they now explain the lock, and stay silent for users without write access - The multipart import fetch threw a plain Error, dropping the 423 status, so the lock self-heal never ran and the stale detail cache survived. It throws ApiClientError now and both import-into-table hooks call handleTableLockRejection - Force append at submit when the delete lock landed while the dialog was open with Replace already selected * fix(tables): restore the workspace ownership check on copilot row deletes Switching deleteRow/deleteRowsByIds to take a TableDefinition dropped the workspaceId argument that previously scoped the query, and the two branches I added loaded the table without the ownership comparison every other operation in the tool performs — letting a caller delete rows from a table in another workspace. Both now reject a foreign table as not found.	2026-07-25 14:21:05 -04:00
Vikhyath Mondreti	17d77795b4	feat(providers): prompt caching capability + usage-based cache pricing (#5922) ... * improvement(providers): validation pass, and stream tool loop improvements * remove deploy options correctly * fix * feat(providers): prompt caching capability and usage-based cache pricing Replace the arbitrary cached-rate heuristic with a single cache-aware pricing function, and add prompt caching as an opt-in capability for Anthropic. Pricing: priceModelUsage in cost-policy.ts is now the only place cache arithmetic happens. Provider adapters normalize their wire shape into ModelUsage (input always excludes cache buckets); the pricing function never branches on provider. This removes five divergent behaviors, including the !!request.context heuristic that gave Router and Evaluator an unearned 10x input discount, and the overwrite that silently billed Anthropic cache reads and writes at zero. Also parses OpenAI cache_write_tokens, previously ignored. Caching: Anthropic gets a capability-gated advanced switch that places cache_control on the last tool and last system block; system is now always a TextBlockParam array. OpenAI gets a stable per-block prompt_cache_key with no UI, since its caching is automatic. * fix(providers): route OpenAI and Gemini block cost through cache-aware pricing Cache-aware pricing only reached trace segments. The billable block cost still called calculateCost on the cache-inclusive prompt total, so OpenAI cache hits and Gemini implicit-cache hits were charged at the full input rate and GPT-5.6+ cache writes went unbilled. Both providers now accumulate cache buckets and price through priceModelUsage, matching the Anthropic token convention where input excludes cache reads and writes. Cached counts are clamped to the prompt total so an over-reporting payload cannot bill more input than the request contained. * fix(streaming): redact tool payloads on selected outputs in public chat Redaction only ran on the empty-selection branch, but a deployment almost always selects outputs, so it was dead in the case it exists for. Selecting toolCalls streamed the raw arguments and results to a public chat client in a chunk frame, and providerTiming carried thinking content the same way. Both paths now extract from the sanitized block output rather than the raw log: the streamed selected output, which is the reachable vector, and the final envelope. Sanitizing the source rather than per selected path means a newly selectable field cannot reopen the hole. * refactor(providers): drop unreachable billing fallbacks Every provider pricing helper took a policy parameter no caller passed. Worse than dead: passing one would have double-applied the margin the central layer already applies. Removed, so providers can only price at list. Also removed guards that cannot fire. The central fallback normalized cache buckets no provider can reach it with (all three that report cache usage price themselves) and did so at a 1x write multiplier no vendor charges. priceModelUsage re-validated token counts the adapter had already clamped, and applyModelCostPolicy defaulted a required total field. Validation now happens once, in the adapter that parses the vendor payload and is the only layer that knows cache buckets are a subset of the prompt total.	2026-07-24 15:46:11 -07:00
Waleed	513292f17b	feat(sso): DNS domain verification gating org SSO registration (#5909) ... * feat(sso): DNS domain verification gating org SSO registration Add org-scoped domain ownership verification (DNS TXT challenge) as the security precondition for configuring SSO. Closes the first-come domain-claim vuln where any org could wire another company's domain to its own IdP. - New sso_domain table + migration 0266; existing org SSO domains are grandfathered as verified so live tenants are unaffected - Verified-domains settings UI (enterprise-gated) with add/verify/remove - Register route now requires a verified domain for org-scoped registration; personal SSO and already-grandfathered domains are unaffected - Self-host register script writes the verified sso_domain row directly, so script-driven registration stays backwards compatible * fix(sso): harden domain verification against concurrency + fix CI lint Addresses review findings on state invariants under concurrent/failed writes: - Add unique index on (organization_id, domain) so concurrent claims can't create duplicate pending rows; POST re-reads and stays idempotent on conflict - Verify flips the row only if it's still the exact pending challenge checked (guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique index violation to 409 instead of an unhandled 500 - Wrap the self-host script's provider write + verified-domain upsert in a transaction so a failed ownership write can't leave a provider committed - Format 0266 snapshot/journal with biome (fixes @sim/db lint:check) * fix(sso): re-check domain verification before provider write (TOCTOU) The register gate checked the verified sso_domain row only at handler entry, then ran OIDC discovery before writing the provider. A verified row removed during that window could still complete registration. Extract the check into a closure and call it both as an entry fast-fail and authoritatively right before registerSSOProvider, alongside the existing domain-conflict re-check. * fix(sso): stop rotating verification token on idempotent re-add Re-adding a pending domain rotated its verification token, which invalidated a TXT record the admin may have already published and — under two concurrent re-adds — could return a token the racing write had already superseded, so the admin's DNS record would never verify. Return the existing row unchanged instead; the pending token is always shown in the UI, so it is never lost. * fix(sso): close register TOCTOU with compensating delete + harden edges Audit-driven hardening: - Close the residual register TOCTOU: registerSSOProvider is create-only (throws if the providerId exists), so a compensating delete after the write is provably safe — it can only remove the just-created row. If verification was revoked during the write, roll the provider back and 403. - Verify is now idempotent under concurrency: a same-org row already flipped to verified by a racing request returns 200, not a confusing 409. - Grandfather backfill + self-host script now match normalizeSSODomain's dominant transforms (lower + trim + strip leading wildcard) so a non-canonical legacy domain can't miss the runtime gate's lookup. Prod backfill result is unchanged. - Cleanup: drop dead default export, align card radius to sibling convention. * fix(sso): redact domain tokens from non-admins + fix script stale-update Round-5 review findings: - GET /domains redacted the pending TXT verification token (a management secret) to any org member. Now only owner/admins read it; members see the list and status without it. Non-Enterprise orgs get an empty list (entitlement flag only), never the domains/tokens. - Self-host script decided update-vs-insert from a read taken OUTSIDE the transaction; a provider deleted mid-flight made the UPDATE match zero rows silently while the verified-domain upsert still committed (orphaned domain). The decision now happens inside the transaction from the UPDATE's row count. * docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains Verified domains currently only gate SSO configuration. Remove the forward-looking references to enforcing SSO and auto-joining members (deferred to a later release) from the docs, settings copy, nav description, and schema comment so we don't promise unshipped features. * fix(sso): guard rollback to new providers only + Enterprise-gate domain removal Round-6 review findings: - The compensating provider rollback now only fires when the provider did not exist before this request (providerExistedBefore). registerSSOProvider is create-only today so reaching the rollback already implies a fresh create, but this makes the safety local and future-proof: if Better Auth ever allowed updating an existing provider, a revoked-verification rollback must not delete that pre-existing row. - DELETE /domains now requires an Enterprise plan like add/list/verify, so all domain mutations share one entitlement (the UI already hides removal from non-Enterprise orgs). Adds a delete-route test. * fix(sso): roll back the SSO provider by row id, not logical keys The compensating rollback deleted by (providerId, orgId). providerId is unique, so if this request's row were deleted and recreated by a concurrent registration in the narrow window before the rollback, the logical-key delete would remove that other request's provider. Delete by the primary-key id registerSSOProvider returns instead, so only the exact row this request created is ever removed. * chore(sso): final-review polish — trim script read, unify copy, doc migration edge Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the new logic): - Self-host script: narrow the pre-transaction existence read to select({ id }) instead of SELECT * (it only feeds a log line now). - Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere 409 wording ("is already verified by another organization") across routes. - p-3 shorthand on the domain row card. - Document the migration's rare two-orgs-share-a-domain grandfather behavior (login unaffected; validated no such duplicates in prod). * fix(sso): apply attribute mapping + make SSO edit work; drop dead guard Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW, script-registered with the default mapping, so neither change affects it): - Attribute mapping was passed at the top level of the register payload, which Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it so custom mappings actually apply. (Default mapping is unchanged, so existing logins are unaffected.) - Editing an SSO provider was broken: registerSSOProvider is create-only and threw on the existing providerId → generic 500. Route now detects a provider the caller already owns and updates it via Better Auth's updateSSOProvider, and surfaces Better Auth's own error status/message instead of a blanket 500. Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes by the created row's primary-key id and register is create-only) and the earlier final-review polish (script read, unified copy, migration edge note). Smoke-test SSO login + edit on staging before merge (auth-path change). * fix(sso): require null org on personal-mode provider lookups (gate bypass) The personal branch of both provider-ownership lookups keyed on (providerId, userId) without requiring organizationId IS NULL. Because org providers store userId = their creator and providerId is globally unique, an org admin could send a personal-mode request (no orgId) — which skips the membership check and the domain-verification gate — yet still match, and then via the new update path move, their org's provider to an unverified domain. Add isNull(organizationId) to the personal branch of both clauses so it can only match a genuinely personal provider, matching the route's own isOwnedByCaller. Found by an adversarial review of the update path added in 394bda9f7. * fix(sso): script updates the observed provider by id, not providerId Inside the registration transaction the script updated WHERE providerId — the logical key. If the observed provider was deregistered and a replacement created with the same providerId before the transaction ran, that update would clobber the replacement's config and ownership. Update the specific observed row by its primary-key id instead; if it's gone we insert, which fails cleanly on the providerId unique constraint rather than overwriting the replacement. * fix(sso): script upserts provider via delete-then-insert (no unique constraint) sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate duplicates, so the previous "update by id, else insert" could create a duplicate provider when the observed row was deregistered and replaced before the transaction — the fallback insert would succeed. Delete every row for the providerId then insert exactly one, inside the transaction, so the providerId ends up as exactly this config atomically. Linked accounts key on the providerId string (not the row id), so existing logins are unaffected. * fix(sso): guard compensating-delete row id so rollback can't silently no-op * chore(sso): regenerate migration as 0268 after merging staging Staging landed migrations 0266/0267, colliding with our 0266. Removed our migration, merged staging, and regenerated cleanly with drizzle-kit as 0268_sso_domain_verification (identical sso_domain table + indexes), then re-appended the grandfather backfill. api-validation baseline reconciled to 973 (staging 970 + our 3 domain routes). Also make the register-route test's registerSSOProvider mock return an id so the guarded compensating delete runs. * refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate The self-host script canonicalized SSO domains with a minimal inline transform (lower+trim+wildcard) that diverged from the app's full normalizeSSODomain (protocol, port, path, trailing dot, email local part) — equivalent spellings could store a different ownership key than the runtime gate looks up. Move normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register route, the domain-claim route, and the script all use the identical canonicalizer. The script now skips the verified-domain record when SSO_DOMAIN isn't a valid registrable domain instead of storing a malformed key.	2026-07-24 00:34:16 -07:00
3 people	6dcc65be89	feat(skills): add skill editors (#5705) ... * feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-23 19:52:00 -07:00
4 people	d24bc7eccb	feat(agent-stream): thinking and tool streaming (#5671) ... * feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-23 19:39:03 -07:00
Waleed	2b5a92a3c8	feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862) ... * feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name)	2026-07-22 16:42:59 -07:00
Waleed	93fbf584a2	feat(chat): soft-delete sidebar chats with restore from Recently Deleted (#5830) ... * feat(chat): soft-delete sidebar chats with restore from Recently Deleted * fix(chat): review round 1 — restore workspace authz, purge recheck, archived-list invalidation * test(chat): update SSE handler assertions for workspaceLists invalidation * fix(chat): bump updatedAt on restore, recheck retention cutoff in task cleanup * fix(chat): drop explicit feedback delete in task cleanup — chat FK cascade covers it * test(chat): cover restore route; guard legacy copilot delete from hard-deleting mothership chats * chore: revert unintended bun.lock drift from worktree install * fix(cleanup): recheck workflow archive cutoff on delete; export deletedAt in chat drain	2026-07-21 20:10:13 -07:00
Vikhyath Mondreti	86e6e1d26c	feat(forking): excluded workflows (#5727) ... * feat(forking): excluded workflows * improve sync preview	2026-07-16 20:23:26 -07:00
5 people	f03b4337fa	feat(mothership): mixture of models, search agent, persistent subagents, fork chat, inline questions (mothership v0.8) (#5410) ... * feat(scout): add scout agent * fix(contracts): update contracts to include scout agent * feat(copilot): search agent (research+scout merge) + read-only table/KB tool handlers Mirrors mothership dev f90f9b05: - regenerated tool-catalog/tool-schemas mirrors (search trigger replaces research + scout; QueryUserTable / SearchKnowledgeBase entries) - queryUserTableServerTool / searchKnowledgeBaseServerTool: read-only wrappers delegating to the full user_table / knowledge_base handlers with hard operation allowlists (and outputPath export rejection on query_user_table) - display maps: 'search' agent label/title/icon added; research + scout entries retained so historical transcripts keep rendering - Search.id replaces Research.id in LONG_RUNNING_TOOL_IDS (it inherits research's long crawls) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(copilot): run_code compute-only handler; docs lint fix Mirrors mothership dev db60da94: run_code is the compute-only variant of function_execute for the search agent — same sandbox and inputs, no outputs.files / outputTable, so it cannot create or overwrite workspace resources. Wrapper handler hard-rejects the write vectors and delegates to executeFunctionExecute; run_code is deliberately absent from OUTPUT_PATH_TOOLS and the table output post-processor, so the name gating blocks writes even for leaked args. Added to LONG_RUNNING_TOOL_IDS, display title/icon maps, and the regenerated catalog/schema mirrors. Also removes two ineffective biome suppression comments in the docs workflow-preview (the rule doesn't fire in the docs app config). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(copilot): failed tool calls must surface their error in terminal data A failed handler result that carried a defined-but-empty output (the app-tool executor's 'Tool not found' ships output: {}) won the priority race in getToolCallTerminalData, so the resume payload's data — the only thing the model reads — was a bare {} with the error text dropped. The search agent retried run_code 20+ times blind against a stale server because every failure rendered as empty instead of 'Tool not found'. Failed calls now always carry error in their terminal data: merged into object outputs, wrapped alongside non-object outputs, preserved when the output already has an error field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(chat): render inline question tags from the agent in chat * fix(chat): let inert multi-step questions browse all prompts * improvement(chat): guard question answer formatting against sparse arrays * chore(copilot): drop user_memory from generated contracts and tool display Companion to mothership 8ae32e97 (user_memory tool removed — the feature no longer exists). Regenerates the mothership contract mirrors via generate-mship-contracts.ts, which also picks up the pending telemetry contract additions (gen_ai.agent.name labels, llm.client.context_tokens, llm.client.compactions, llm.request.compaction_trigger, llm.compaction.pause, gen_ai.usage.context_tokens), and removes the user_memory display title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): answered question card becomes the user turn; two select types only UI ordering: answering a question card no longer echoes a duplicate user bubble. The combined answer still goes on the wire as a user message, but the chat pairs it back to its card (strict 'Prompt — Answer' match, now uniform for single questions too) and renders the card as the answered recap — the card IS the user turn, and the next assistant message streams below it. The pairing is derived from the transcript, so live and reloaded renders are identical; a dismissed card followed by an unrelated typed message does not match and renders normally. Messages ending with a question card also drop the copy/thumbs actions row — the card is an input surface, not a reactable assistant turn. Question types are now single_select and multi_select only: text is removed (the free-text 'Something else' row covers it) and confirm collapses into single_select with Yes/No options. multi_select rows toggle with a check and the free-text row's arrow submits the step; answers are comma-joined labels plus any typed entry. Agent-supplied catch-all options ('Other', 'Something else', 'None of the above') are stripped at parse — the card always provides its own free-text row; a question left with no real options is invalid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): question cards are single_select only Removes multi_select (and its toggle/check UI). The card is one shape: pick one option or type into the always-present 'Something else' row. Catch-all stripping and the transcript pairing/recap behavior are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): bring back multi_select question cards Re-adds multi_select with a reworked interaction: option rows carry real checkboxes (emcn Checkbox chrome) instead of numbers and arrows, an option-styled Submit row confirms the step, and the "Something else" row reads as a plain option until clicked — then it becomes the focused text box, auto-checks, and can be unchecked without losing the typed text (blur with nothing typed reverts it). single_select behavior, catch-all stripping, and the transcript pairing/recap format are unchanged; multi_select answers are the checked labels comma-joined. * chore(copilot): regenerate mothership contract mirror (chat blob span attrs) * chore(copilot): regenerate mothership contract mirror (chat blob metrics) * feat(secrets): make output of generate api key a secret * feat(cli): add mkdir, mv, cp to mship tool set * feat(fork-chat): add fork chat to mothership * fix(fork-chat): fix messageid handling in fork chat * feat(credentials): agent-initiated oauth credential reconnect (#5488) * feat(credentials): agent-initiated oauth credential reconnect * fix(credentials): address reconnect review findings * improvement(credentials): log when connect draft name lookups degrade * fix(conflicts): remove migration * fix(conflicts): fix conflicts * fix(fork-chat): add migrations back * fix(ci): fix lint * fix(ci): fix bad import * fix(vfs): fix 500 char limit in vfs for skills and custom tools * feat(copilot): gate user skills to explicit slash-attach (#5536) Stop the mothership from adopting a workspace user-skill on its own: - Remove the load_user_skill tool and its three payload callers (chat payload, mothership execute route, inbox executor); delete lib/mothership/skills.ts + its test. Skills no longer autoload as the agent's own instructions. - Rename the workspace "## Skills" inventory to "## Agent Block Skills — NOT FOR YOU" with a one-line guardrail so a skill's description (e.g. "respond like a pirate") is not treated as an instruction. Skills reach the model as behavior only via explicit /-attach. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(lots-of-things): lots of things * feat(subagents): add persistent subagents * fix(copilot): let edit_workflow set knowledge-base tag filters, and stop it clearing them (#5546) * fix(copilot): persist KB tag subblocks as JSON strings from edit_workflow The edit_workflow tool normalizes array-with-id subblocks (via normalizeArrayWithIds) but only re-stringifies the keys listed in JSON_STRING_SUBBLOCK_KEYS. `tagFilters` (knowledge-tag-filters) and `documentTags` (document-tag-entry) were missing, so agent-authored tag filters were stored as raw JSON arrays while those UI components read their value with JSON.parse (expecting a string). The result: an agent edit to a Knowledge block's tag filter persisted correctly but rendered as an empty filter in the editor (JSON.parse on an array throws -> []). - Add `tagFilters` and `documentTags` to JSON_STRING_SUBBLOCK_KEYS so edit_workflow stores them in the same shape the UI writes. - Make both components' parsers tolerate an already-parsed array on read, self-healing values already persisted in the broken (array) shape. Search execution was unaffected (parseTagFilters accepts arrays), so the value was never lost — only the editor render and round-trip were broken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): expose KB tag definitions in VFS meta.json Surface each knowledge base's defined tags (displayName -> tagSlot) inline in its meta.json via serializeKBMeta, loaded in one batched query (loadKbTagDefinitions), so the agent can bind a knowledge-tag filter to a real tag slot instead of guessing a tag name it cannot otherwise see. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stringify KB tag subblocks on the nested-node edit path The nested-node merge path normalized array-with-id subblocks but never re-serialized the JSON_STRING_SUBBLOCK_KEYS, so editing a block nested in a loop/parallel container still persisted tagFilters/documentTags (and conditions/routes) as raw arrays -- the exact shape the subblock components cannot JSON.parse. Route all four write paths through a single normalizeSubblockValue helper so the normalize and re-stringify steps cannot drift apart again, and extract the duplicated string-or-array read logic into parseJsonArrayValue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): tighten subblock serialization helpers Derive KbTagDefinitionSummary from the canonical TagDefinition instead of restating its fields, make parseJsonArrayValue generic so callers drop their `as T[]` casts, and unexport the three builders helpers that no longer have consumers outside the module now that normalizeSubblockValue fronts them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stop stripping tagFilters/documentTags from the agent's workflow view sanitizeForCopilot dropped `tagFilters` and `documentTags` from the workflow state the agent reads (workflows/{name}/state.json), while edit_workflow is allowed to write both. The field was therefore write-only: on a follow-up edit the agent read back an absent field, concluded no filter was set, and cleared the user's tag filter. The redaction was introduced for workflow *export* (#1628) and is already enforced there by sanitizeWorkflowForSharing's key list. The duplicate in the copilot-only sanitizeSubBlocks was redundant for export and destructive for the agent. Removes it and pins the contract with a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): reject malformed KB tag values instead of clearing the filter `knowledge-tag-filters` and `document-tag-entry` had no arm in the `edit_workflow` input validator, so they fell through to the pass-through default. Any non-array value the agent supplied -- a double-encoded JSON string, an object, an unparseable string -- reached `normalizeSubblockValue`, where `normalizeArrayWithIds` coerces unparseable input to `[]`. The write path then persisted `"[]"` over the tag filter the user had configured. `condition-input` and `router-input` already guard against exactly this and return an actionable error to the model. Extend that arm to cover the two KB subblock types. It keys on subblock type, so the unrelated `tagFilters` short-input on the Algolia block is unaffected. `null`/`undefined` and empty arrays still clear the field, so intentional clears keep working. Also wrap `loadKbTagDefinitions` in try/catch. Tag definitions are an optional meta.json enrichment, but the query ran inside the top-level `Promise.all`, so a transient failure would reject the entire workspace VFS materialize and leave the agent unable to read any file. Now it degrades to a meta.json without tag definitions, matching the sibling materializers. Adds regression tests for both, plus the first tests for `parseJsonArrayValue`, the helper that keeps pre-fix raw-array rows readable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): collapse duplicate JSON-array parsing in edit-workflow builders `normalizeArrayWithIds` and `normalizeConditionRouterIds` each hand-rolled the same "accept a raw array or the JSON string these subblocks persist" parse. Extract `parseJsonArray`, which returns null when the value is neither, so each caller keeps its own distinct fallback: `[]` for the former, the untouched original value for the latter. Behavior-preserving. An empty array is truthy, so `[]` and `"[]"` still parse through rather than hitting either fallback. `validation.ts` has a third copy, but `builders.ts` already imports from it, so sharing the helper across the two would introduce an import cycle. Left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): specify tag name and legal operators in KB meta.json `tagDefinitions` exposed `displayName`, but a `tagFilters` entry must carry the key `tagName`. An entry written with `displayName` passes validation and persists, then filters nothing -- a silent failure. Rename the field at the serializer boundary; the DB column is untouched. Also emit the operators legal for each tag's `fieldType`, reusing `getOperatorsForFieldType`. `between` is valid for number and date but not for text or boolean, and the agent has no way to infer that. An unrecognized fieldType yields an empty list rather than throwing. Still unspecified, and deliberately out of scope: a filter entry's value key is `tagValue` (but `value` on documentTags), and `between` needs `valueTo`. Those describe the subblock entry shape, not the knowledge base, so meta.json is the wrong place for them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): pass a nullish subblock clear through instead of serializing "[]" `validateValueForSubBlockType` accepts null as an explicit clear, but `normalizeSubblockValue` then ran it through `normalizeArrayWithIds`, which coerces any non-array to `[]`, and persisted the string "[]". No data is lost either way -- "[]" and an absent field both mean "no filters". But it left the field present when the caller asked for it to be unset, so `sanitizeForCopilot` showed the agent an empty filter rather than an absent one, contradicting the absent-means-unset invariant the sanitizer documents. It also made Algolia's `if (params.tagFilters)` see a set value, since "[]" is truthy. An explicitly empty array still serializes to "[]" -- clearing with a value is distinct from clearing by omission. Reported by Cursor Bugbot on #5546. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(changes): huge changes * fix(subagents): lanes * fix(mship): transcript stuff * fix(subagents): thinking lanes * fix(superagent): fix superagent tools and checkpoints * fix(scope): scope subagent tools * fix(lint): fix lint * chore(db): regenerate workspace_files.message_id migration as 0260 on staging base * fix(superagent): fix superagent integration tools * improvement(questions): make something else a placeholder * chore(copilot): regenerate mothership contract mirror after staging rebase * feat(mship): add external mcps to mship * fix(ci): fix dev build * fix(stream): show thinking text * fix(ci): force redeploy * fix(mothership): keep chat forks outside workspace storage billing Preserve the product invariant that Mothership chat files are not charged as workspace file storage after the billing storage merge. * fix(uploads): restore listWorkspaceFiles throwOnError option dropped in rebase * fix(subagent-streaming): remove italics * fix(mothership): treat subagent lanes closed by subagent_end as settled so the between-steps thinking indicator isn't suppressed * fix(ui): thinking loader and rool names * fix(ui): add file * fix(thinking): show thinking during subagents * fix(chat): drop dead thinking-channel ternary after lanes skip thinking blocks * fix(thinking): remove thinking text * improvement(function execute): add timeout to function execute and stop showing text in subagents * fix(subagents): hide thinking text * fix(ff): move ff to go * improvement(superagent): nuke superagent * feat(main-agent): superagent into main agent * chore(db): regenerate workspace_files.message_id migration as 0262 on staging base * fix(credentials): restore reconnect params on shared createConnectDraft * fix(migrations): rebase with staging --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu> Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-16 11:24:46 -07:00
Vikhyath Mondreti	54b35a4f0e	improvement(deployments): bugfixes for run-block, airtable + external sub management (#5680) ... * improvement(webhooks): external subscription management * ui/ux * remove test file * fix tests * address comments * address comments * update to grain v2 api * improvement(grain): hide auto-registered webhook URL on v2 triggers * Revert "improvement(grain): hide auto-registered webhook URL on v2 triggers" This reverts commit c89660cc3e. * address comments * address comments * address rollback * fix grain v2 * fix more comments	2026-07-15 18:01:53 -07:00
Vikhyath Mondreti	ef7c8e24b2	feat(platform): settings permissions, admin, billing attribution (#5545) ... * fix(invites): preserve active organization for external access Keep organization activation server-owned so failed membership checks cannot clear a valid session context. * feat(admin, billing, settings): cleanup settings visibility, billing actor resolution, new admin routes * address comments * chore(db): reset pending migrations before staging merge Remove locally generated migrations so they can be regenerated against the latest staging schema without preserving stale snapshots or numbering. * regen migrations * address comments * chore(db): reset generated migrations before staging merge Remove this branch's generated migrations so they can be regenerated against the latest staging schema with fresh numbering. * upgrade global work * fix lint * address comments * legacy callbacks correctness * address comments * update * guardrail attribution	2026-07-13 00:53:49 -07:00
Theodore LiandClaude Opus 4.8	f4d47ed826	feat(slack): reusable custom bot credentials, slack_v2 block (preview), redesigned trigger (#5323) ... * feat(slack): enable assistant-agent tools via assistant:write scope Add assistant:write, app_mentions:read, and im:history to the Slack bot OAuth scopes so the Set Assistant Status / Title / Suggested Prompts tools (assistant.threads.*) work with users' existing Slack credentials — no new app or credentials required. Restore the action_assistant trigger capability (scope assistant:write) in the manifest generator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpeT8J5yVCrrNQB9Hzm9uS * Add slack trigger * fix channel picker in slack trigger * improvement(slack-trigger): reorder app type, gate account to sim mode, add channel-id input * fix(slack-trigger): drop unmapped events from filter, resolve oauth token for reaction text + file downloads * fix(slack-trigger): empty operation selection fires nothing; resolve token via credential owner not execution actor * fix(slack-trigger): ignore message edit/delete/system subtypes; prefer channel picker over stale manual ids * feat(slack-trigger): single-event model with contextual filters and full event catalog * fix(slack-trigger): apply event/channel/bot filters on custom-app path too * fix(slack-trigger): don't drop edit/delete events when channel_type is absent * feat(slack): reusable custom bot credentials, slack_v2 block, interactivity triggers - Custom bot as a workspace service-account credential (set up once, shared ingest URL /api/webhooks/slack/custom/{credentialId}, reused across triggers and actions) - slack_v2 action block: credential-based Custom Bot auth alongside Sim OAuth; v1 hidden from toolbar - Interaction triggers (block_actions / view_submission) with optional action/callback id filter; settings.interactivity in generated manifests - Setup wizard: name + description, full permissions by default with ChipDropdown customization; reconnect mode rotates secrets in place - Centralized service-account token resolution (unknown provider fails loudly) - Shared Slack webhook fan-out dispatcher for native + custom ingest routes * chore(api-validation): bump route baseline to 924 after staging merge * feat(slack): preview-gate slack_v2 and the custom-bot credential surfaces slack_v2 (block + hosted slack_oauth trigger) ships preview: true — hidden from all discovery until revealed via block-visibility AppConfig or PREVIEW_BLOCKS. v1 stays toolbar-visible with the legacy slack_webhook trigger until v2 GAs. The integrations-page custom-bot setup surface rides the same flag via isHiddenUnder(slack_v2); placed instances, existing credentials, and ingest/execution paths are never gated. * fix(slack): v1 keeps slack_webhook trigger subblocks; handle object-form event channels - v1 spread had been swapped to slack_oauth's trigger subblocks (shared with v2), leaving its slack_webhook deploy path without signing-secret config (Bugbot high). v1 now carries the legacy trigger set again; v2 swaps them for slack_oauth's. - resolveSlackEventChannel reads channel.id for channel_created/channel_rename payloads, so channel filters no longer drop every rename event. * fix(slack): default absent appType to custom at deploy; deactivate custom-bot webhooks on credential delete - appType is hidden and seeded 'custom' by value(), which only covers editor-created blocks; defaultValue now persists it via buildProviderConfig and the deploy fallback flips to custom (the only exposed mode this ship) - deleting a slack-custom-bot credential now also deactivates provider='slack' webhooks routed by that credential id, not just native slack_app rows * fix(slack): resolve credential owner for deploy-time team_id lookup A teammate deploying a trigger wired to a shared Slack credential isn't the credential owner; refreshAccessTokenIfNeeded only loads tokens for the owning user. Resolve the account owner first, mirroring the runtime formatInput path. * chore(slack): reconcile staging merge - nullable webhook.path coalesced at correlation/payload/tiktok boundaries - slack dispatch delegates to staging's dispatchResolvedWebhookTarget (shared preprocess/deployment/filter/enqueue lifecycle), keeping the skip-reason diagnostics; route tests reworked around that seam - api-validation route baseline 924 -> 926 * fix(slack): workspace-scope bot credentials at deploy; recreate webhooks on routing transitions - a bot credential id is semi-public (embedded in Slack Request URLs), so the custom deploy branch now rejects credentials outside the workflow's workspace - needsRecreation also compares path/routingKey, so a row from an older routing model can't survive redeploy as a stale delivery surface * test(slack): pin fail-closed behavior for empty/missing event selection * fix(slack): 409 on custom-bot name collision instead of silently returning the existing credential The service-account dedupe matches on displayName, which defaults to the Slack team name — shared by every bot in that workspace. A second unnamed bot create returned the first credential as success, orphaning the new id already pasted into the Slack Request URL. Same-id replays stay idempotent; different-id collisions now fail loudly so the wizard prompts for a distinct name. * fix(slack): reconnect surfaces Atlassian error codes and persists name/description edits - PUT credential route now returns the Atlassian provider code (providerErrorCode -> code) so reconnect failures map to specific token/domain messages, matching create - Google/Atlassian reconnect send + seed displayName/description (parity with Slack); edits are no longer silently discarded, and empty fields don't clobber existing values * fix(slack): require bot name; propagate rotated bot_user_id to webhooks on reconnect - the setup wizard now requires a bot name (canAdvance), so the credential name, manifest app name, and uniqueness key all use the user's choice instead of the shared Slack team-name fallback that collided for a second bot in one workspace - reconnect that changes the bot user id (recreated Slack app) now updates the bot_user_id cached in each bound webhook's providerConfig, so reaction self-drop keeps working instead of letting the bot's own reactions re-enter --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 19:43:52 -04:00
4 people	5d3809a0e3	feat(tiktok): add tiktok trigger, block (#5504) ... * feat(tiktok): add TikTok integration Adds TikTok as a full OAuth-based integration: provider registration (with TikTok's comma-separated scope and client_key requirements), 9 tools covering profile info, video listing/querying, creator info, direct video/photo posting (URL or file upload), inbox drafts, and post status polling, plus the TikTok block, icon, and generated docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): add avatarFile output to Get User Info Adds a file-typed avatarFile output (sourced from the largest available avatar URL) alongside the existing string avatar fields, so the profile picture can be materialized as a UserFile and chained into file-consuming blocks (e.g. attached to an email), per PR review feedback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): lower upload memory cap, drop redundant avatar string outputs Cap the file-upload video buffer at 250MB instead of TikTok's 4GB ceiling — relaying that much through this server's memory per request isn't safe under concurrent load, and larger files can still go through the PULL_FROM_URL path, which never buffers on our server. Also drop the now-redundant avatarUrl/avatarUrl100/avatarLargeUrl string outputs from Get User Info in favor of the file-typed avatarFile output alone, since the feature is unreleased and the raw URL is still reachable via avatarFile.url. Cover image URLs on List/Query Videos are confirmed to be signed, expiring TikTok CDN links; left as strings (no file-output conversion path exists for fields nested inside array items) but documented the expiry behavior more clearly. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ci): bump API validation route-count baseline for TikTok publish-video route The TikTok integration adds one new Zod-backed internal API route (app/api/tools/tiktok/publish-video), which trips the route-count ratchet in check-api-validation-contracts.ts. Bumping totalRoutes and zodRoutes from 917 to 918 (nonZodRoutes stays 0) to acknowledge the new route is properly validated. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): drop unused avatar_url_100 from default user fields After removing the avatar string outputs, avatar_url_100 was still requested from TikTok's user info endpoint but never surfaced anywhere. Removed it from the default field list and the field descriptions, and noted that avatar_url/avatar_large_url feed the avatarFile output. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop returning raw 'credential' subBlock id from tools.config.params The block's params function built a local `credential` variable from params.oauthCredential and returned it under the key `credential` in every switch case. That literal token is the raw subBlock id, which is deleted after canonical transformation into `oauthCredential` — the blocks.test.ts canonical-param-validation suite flags any params function that still references it. It was also redundant: oauthCredential is already part of the base resolved inputs, which the executor merges into the tool call before config.params overrides are applied, so the OAuth token resolution (which reads contextParams.oauthCredential) worked regardless. Removed the explicit credential plumbing, matching the convention already used by other OAuth blocks like dropbox.ts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): send empty JSON body on Query Creator Info POST query_creator_info had no request.body function, and formatRequestParams() only attaches a body when tool.request.body is defined at all — so despite sending Content-Type: application/json, the request went out with no body whatsoever. Added body: () => ({}), matching the convention already used by other parameterless-POST tools in this codebase (Google Vault, Supabase, Square, Gmail, etc.). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop dropping valid zero values in optional numeric fields cursor, photoCoverIndex, and videoCoverTimestampMs all used a truthy check (params.x && {...}) to decide whether to include an optional numeric override, which drops a legitimate 0 (first page has no cursor issue aside, photoCoverIndex 0 is TikTok's own default cover photo, and timestamp 0 is a valid first-frame cover). Switched to explicit undefined/empty-string checks, matching the !== undefined convention the underlying tools already use. In today's resolution pipeline these fields always arrive as strings (even chained block references get stringified by the template resolver), and a non-empty string like "0" is truthy, so this wasn't actively broken end-to-end - but it was relying on that subtlety rather than being correct by construction, and was inconsistent with the tools' own undefined checks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): accept newline-separated video IDs in Query Videos videoIds is a long-input (multiline textarea), the same widget used for the newline-separated photoImages field on this block, but its parser only split on commas. Entering one ID per line - the natural pattern for a multiline field, and the one already used elsewhere on this block - produced a single concatenated garbage string instead of an array, so TikTok's query would fail or return nothing. Now splits on commas or newlines, and updated the placeholder/description to reflect both formats. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(tiktok): add app-level webhook ingress and triggers * fix(tiktok): only count actually queued webhook executions Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): bump API validation baseline for staging merge Co-authored-by: Cursor <cursoragent@cursor.com> * cleanup code * fix type issues * misc code cleanup * remove photos and add upload for videos * move shared video output properties to types.ts so docs generation resolves them Co-authored-by: Cursor <cursoragent@cursor.com> * hide TikTok from toolbar and docs until the integration is ready to ship Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): ratchet API validation baseline to 924 after staging merge Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-10 15:08:24 -07:00