107 Commits

Include renames

Author	SHA1	Message	Date
Waleed	9b9f4ee596	fix(v2-api): close three secret disclosures, make the surface consistent, and align docs with signatures (#6560) ... * fix(v2-api): close two secret disclosures and align docs with signatures Two P0 disclosures, five correctness bugs, and the standardization and guard work that came out of auditing them. **Secret disclosure — workflow version state.** `GET /api/v2/workflows/{id}/ versions/{version}` served the deployed graph unsanitized, so a read-role workspace API key received plaintext block-password values and OAuth credential ids. The sibling export route has always sanitized. Every other v2 response is protected structurally because the builder re-parses it, but this field is `z.custom<WorkflowState>()` — a predicate that validates nothing — which is why it survived earlier audits. Sanitization now lives in the use case, secure by default, with a named `includeCredentialValues` opt-in that only the session-authed deploy-preview route sets. **Secret disclosure — MCP headers.** The internal list and update routes returned custom `Authorization` headers verbatim to any read-role member; headers are stored unencrypted. Values are now gated on write permission and projected through one shared helper. The settings UI genuinely prefills from them, so blanking outright would wipe headers on unrelated edits — write-only headers plus encryption at rest are the follow-up. Correctness: - v2 execute ignored `X-Sim-Via`, resetting the call chain on every hop and defeating the recursion guard. Wired on both the keyed and anonymous paths. - v2 knowledge search accepted `searchMode` and dropped it, silently serving vector-only results for a hybrid request, and allowed 50MB bodies where internal caps at 2MiB. - v2 run cancel never released the plan concurrency slot and half-cancelled group runs; a group conflict now returns 409 instead of reporting success. - v2 table row writes stamped no secret provenance, so the next internal read reported the whole page incomplete. `secretProvenance` is now required on the primitives, making the next omission a compile error. - Folder conflicts and malformed paths returned 500; they are 409/404/400 now. `FolderPathError` splits from `FolderHierarchyError` so a corrupt stored tree stays a 500 and stays in 5xx alerting. Standardization and documentation: - `PUT /files/{id}/share` -> PATCH. The resource is not round-trippable (`hasPassword`, never the password), so merge-on-omission is the only implementable semantics. - ~40 spec truthfulness fixes: a 410 the API cannot emit, eight 423s with no lock guard, ~30 reachable-but-undocumented 404/400/413s, and six inverted field claims. Eleven operations that always reject a workspace key now say so — four of them answer 404, so a workspace key was told the resource did not exist. - `NAME_PATTERN` lost its `/i` through `z.toJSONSchema`, publishing 15 patterns that reject names the runtime accepts. Every generated client rejected any capitalized table or column name, and two of the spec's own examples failed the spec's own schema. Guards, so these classes cannot recur: - `check:route-verbs` (new) cross-checks all 212 builder routes' exported verb and path against their contract. The builders only compare at runtime, so a half-done rename previously passed CI and 500'd in production. - Example validation now runs against the published JSON Schema with formats on, covering 225 nodes instead of 100 — this is what caught the regex bug. - The list-pagination sweep is union-aware and fails loudly on a schema it cannot introspect, rather than counting it compliant. * refactor(v2-api)!: flatten the single-resource response envelope BREAKING: 31 endpoints that returned `{ data: { <resource>: T } }` now return `{ data: T }`. This corrects drift, not a design decision. PR #5273 added skills, custom tools, MCP servers, secrets, and knowledge nested while adding workflows, files, and logs flat — and in the same commit wrote the `v2/shared.ts` docblock declaring `single resource: { data: T }` is the standard. The nested half appears to have been modelled on the v2 tables surface (#6067), which landed twelve days earlier. Lists were already `{ data: T[], nextCursor }`, so flat single-resource is what actually matches them; nesting made every client destructure a layer that carries nothing. Doing it now because the cost only grows: `v2-api` is still dark-launched, so today this breaks no one. After GA it needs a deprecation window. Payloads that carry real information were deliberately left alone — this was a classification exercise, not a mechanical sweep. Unchanged: delete acknowledgements (`{ id, deleted }`, `{ path, deleted, deletedItems }`), the knowledge search envelope (which echoes query, knowledgeBaseIds, topK and totalResults alongside hits), upload payloads carrying signed tokens and transfer instructions, bulk-operation counts, `{ row, operation }` upserts, named acknowledgement scalars (`{ dispatchId }`, `{ cancelled }`), and `{ columns: [...] }` — a collection, where a bare `{ data: T[] }` would be indistinguishable from the list envelope but without `nextCursor`. Also flattened the two file-share responses, which were not in the original survey: leaving them would have put one resource in two shapes on one path. `GET /files/{id}/share` now returns `{ "data": null }` when a file has never been shared. No consumer is affected. Both SDKs touch exactly two v2 endpoints — execute and run status — and both were already flat. No docs MDX, client hook, or internal caller reads a changed response; Copilot table tools call the application use cases directly rather than the HTTP surface. The shared `v2FolderSchema` is untouched: every folder flatten was achievable at the response site, which is itself evidence flat was the intended shape. * fix(v2-api): close a third secret disclosure and make concealment coherent **Secret disclosure — run snapshot.** `GET /api/v2/logs/{runId}` returned `workflowState` straight from `workflowExecutionSnapshots.stateData`, which is the workflow graph: `blocks[].subBlocks[].value` holds `password: true` field values and `oauth-input` credential ids. Nothing on that path sanitized it, and the field was typed `z.unknown()`, so the builder's response parse stripped nothing. A read-role workspace API key could read plaintext credentials. This is the third instance of one pattern, and the pattern is the finding: the builder protects every response by re-parsing it, so the only fields that can leak are the ones typed `z.unknown()` or `z.custom()`. Both prior disclosures sat behind exactly such a field. The snapshot is now sanitized in the use case and the field is typed object-or-null. An inventory of every remaining `z.unknown()` in the v2 contracts is in the PR description; two carry data with no projection behind them and are named there as follow-ups. **Concealment was bypassable.** `createV2ResourceConcealmentPolicy` rewrites resource-authorization failures to 404 so a caller cannot probe for existence. Workflows and files applied it on every verb; tables and knowledge applied it only on reads. A caller could therefore probe with PATCH, read the 403, and learn the resource exists — the read-side concealment bought nothing. Nine mutation sites now conceal, plus the three table-column verbs, which were inconsistent with their own sibling sub-resources. `lib/logs/api/route-policies.ts` was a second, divergent implementation that sniffed `response.status === 403` and so also swallowed workspace-policy denials the canonical helper deliberately preserves. It now uses the helper. A third such sniff survives in the upload-control helper and is noted as a follow-up. Also: - `DELETE /tables/{tableId}/rows/{rowId}` returned the bulk `{deletedCount, deletedRowIds}` shape while nine sibling single-resource deletes return `{id, deleted}`. It now matches them. - Nine operations can 404 on an unknown folder path and did not document it; `createWorkflow` could 413 on an oversized folder tree and did not; getting a run can 409 when trace data was truncated and did not. - `queryTableRows` documented a 413 it cannot emit and `resumeWorkflowRun` a 423 with no lock guard anywhere in its path — the same un-producible-status class already cleared for 410 elsewhere. - Execute's 409 description covered only the run-id case after the recursion-guard fix added a second cause, and named a code the route does not emit: the wire carries `error.code: CONFLICT` with the specific cause in `error.details.code`. `x-sim-via` is now a declared request header. - Deploy and rollback published examples that were impossible: `isDeployed: true` beside `activeDeployment: null`, where the route computes the former from the latter. - `afterRowId`/`beforeRowId` were published on row insert and silently dropped by the route, so a positional insert became a tail append. - A generated document whose script fails permanently answered "still being generated, try again" forever; the underlying cause is now preserved. * docs(v2-api): correct eleven false or misleading spec claims Structural parity between contracts and specs is CI-enforced; semantic truth is not. These are claims the spec made that the code does not honour. Outright false: - `DELETE /files/{fileId}` said it deletes "the stored bytes". It archives: the row is retained with a deletion timestamp and the bytes are never removed. Restore exists, but only on the internal API, so the description now says so rather than implying v2 offers it. - Execute documented `409 EXECUTION_ID_CONFLICT` in three places. The wire carries `error.code: CONFLICT` with `error.details.code: RUN_ID_CONFLICT`; only v1 ever emitted the documented string. - The files spec claimed every endpoint uses the canonical envelopes while `GET /files/{fileId}` returns octet-stream. - The shared timestamp rule justified itself with a rendering claim that is false — 29 bare-form sites publish `format: date-time` identically. The real difference is runtime validation, so the rule now says that. It was softened rather than enforced: responses are re-parsed, so adding `.datetime()` to a field whose producer can emit a non-ISO string turns a working read into a 500, and that could not be proven for all 29 without a much larger audit. Misleading: - The billing ledger silently defaults to a 30-day window, so a client paginating to `nextCursor: null` believes it has the whole ledger. - Deleting a connector-backed knowledge document does not delete its chunks — the row survives as excluded and the embeddings remain. - `listTables` said "all tables"; it is keyset-paged with a default limit. - `GET /files/{id}/share` omitted the `data: null` never-shared case its own schema and example already declare. - The share PATCH matrix omitted two hard 400s, so following it literally against a never-shared file fails. - Five knowledge operations render a canonical folder path back and can 413 on an oversized tree without carrying the sentence that says so. Also: the upload-control helper was a third implementation of concealment by sniffing `response.status === 403`, which masks workspace-policy denials the canonical helper deliberately preserves. It now uses the shared policy, so those denials keep their 403. And the shared docblock's search-field enumeration was presented as exhaustive while omitting two lists, and its error-envelope claim omitted the two upload data-plane routes that emit a bare `{error: string}` — both now carry the carve-out the CI allowlist already had. * test(v2-api): align upload concealment test with cross-tenant-only semantics #6557 narrowed `createV2ResourceConcealmentPolicy` to conceal only the three cross-tenant authorization classes, deliberately letting a same-workspace policy denial keep its 403 so the caller learns why. My test predated that and asserted a workspace-key denial was concealed as 404. Split into two cases that pin the distinction rather than paper over it: a cross-tenant reach conceals, a workspace-key policy denial does not. * fix(v2-api): accept the redacting log status and envelope the knowledge-search 413 The v2 log presenters parsed status against a five-value enum, but the execution logger persists a sixth, redacting, while a finished run's output is scrubbed. Any such row failed the response parse; on the list route one row 500'd the whole page. The enum is now derived from PersistedWorkflowExecutionStatus with a compile-time exhaustiveness assertion, so a future status is a type error rather than a production 500. POST /api/v2/knowledge/search declared maxBodyBytes without payloadTooLargeResponse, so its 413 returned a bare string instead of the v2 error envelope. It now matches the sibling deploy/rollback routes. * fix(uploads): restore archive extraction folder parity Archive extraction into workspace files/ was rewritten onto the authorized application-operation boundary, and three behavioral regressions came with that move. Together they broke every archive containing a subdirectory, and 100% of copilot extract() calls (materialize-file always passes rootFolderSegments: [baseName], and its catch only handles ArchiveError). 1. Non-canonical folder path. The extractor joined the folder segments with "/" and passed the result as `path` to createWorkspaceFileFolderOperation. That path reaches requireNonRootFolderPath -> parseFolderPath, which requires a leading "/" and byte-for-byte canonical per-segment encoding, so "bundle/data" threw FolderPathError before anything was written — and a folder name containing a space or a reserved character would still have thrown after merely prefixing a slash. 2. exactName: true. createWorkspaceFileFromBuffer was told to demand the exact leaf name, which sets maxAttempts = 1 and raises FileConflictError when the name already exists. The extractor's rollback then deleted every file written so far, so one colliding name destroyed the whole extraction. Reachable today for flat archives through the unzip action of POST /api/tools/file/manage. Restored to auto-suffixing via allocateUniqueWorkspaceFileName. 3. Wrong folder primitive. createWorkspaceFileFolderAtPath creates exactly one leaf, conflicts on an existing path, and requires the parent to exist already. The extractor never creates intermediates and caches by full path, so the first nested entry asked for a folder whose parent was never created. The correct semantics are ensureWorkspaceFileFolderPath: walk every segment, reuse what exists, create only what is missing. Rather than bypass the operation boundary by calling the manager primitive directly, this adds ensureWorkspaceFileFolderPathOperation — an authorized application use case under files.folders.create that expresses "ensure this whole chain exists" — and routes the extractor through it with raw decoded segments, so no path string is built and no encoding can be malformed. archive.test.ts previously mocked the folder operation and asserted the broken shape (path: 'bundle'), which is why this shipped. The suite now fakes the workspace-file store in memory while enforcing the real rules: folder paths run through the production parseFolderPath family, the create-one-leaf operation conflicts and requires a parent, and exactName governs conflict vs auto-suffix. Nested, reuse, encoded-name, and collision cases are covered and each fails against the pre-fix code. * chore(files): tidy archive extraction cleanup * fix(uploads): roll back folders archive extraction created Extraction now materializes folders before uploading files, but the failure path only deleted the extracted files — every folder the call created was left behind. That is not cosmetic: `materialize_file` guards re-extraction by looking up the root folder path and refusing when it has any child, so a half-extracted nested archive turned every retry into "already extracted — delete that folder first" until a human cleaned up the tree by hand. The rollback must delete only folders this call actually inserted, never one it reused: extracting into an existing path is normal (a sibling entry, an earlier successful extraction), and deleting a pre-existing folder would destroy unrelated user data. `ensureWorkspaceFileFolderPath` already distinguishes the two while walking the segment chain, so it (and its application operation) now reports `createdFolderIds` alongside the leaf id. The extractor accumulates those ids in creation order and, on failure, deletes them in reverse — parents are recorded before their children, so reverse order is deepest-first and a parent is never removed out from under a child. Folder cleanup is best-effort like the existing file cleanup, so a cleanup failure never masks the original error. * fix(billing): withhold the payer credit pool from v2 status readers `GET /api/v2/billing/status` resolved the workspace's payer and projected that payer's pooled allowances — credits used, credit limit, credits remaining, and the payer entity's storage usage and quota — to any caller holding only `read` on the workspace, including a personal API key. The payer pool is shared across every workspace that payer funds, and the platform already treats it as privileged: the workspace credit-availability surface computes `canViewPayerPool` from `canManageWorkspaceBilling` and substitutes member-scoped or null figures for everyone else. The new versioned endpoint had no equivalent gate. `credits` and `storage` are now projected only to a caller who may manage the resolved payer's billing: the billed account holder of a personally hosted workspace, an admin of the hosting organization, or a workspace API key, which only a workspace admin can provision. The endpoint stays at `read` so a plain member keeps the plan, period, and standing the workspace UI already shows them, and an exceeded pooled limit still reports as `limit_exceeded` without disclosing the numbers behind it. Both fields are nullable on the wire and in the regenerated OpenAPI spec. The decision lives in the application use case, resolved from canonical workspace state, not in the route: billing authority is payer identity and organization role, which the workspace permission ladder cannot express — a plain workspace `admin` is deliberately not enough. * chore(api): remove the unused public API route builder and dead endpoint labels `withPublicApiRouteHandler` and 27 `ApiEndpoint` union members landed together in #5273, but the v2 surface shipped on `defineV2JsonRoute` + `v2RateLimits` instead. The builder had no production caller — only its own test — and the v2 rate limiter never reads an `ApiEndpoint` label, so those members were never emitted to telemetry by symbol or by string literal. Remaining members are exactly the labels a v1 route passes to `checkRateLimit` or `authenticateRequest`. Drops the now-unreachable `hasZodUsage` branch from the API validation audit; no ratchet metric moves (route total stays 1093). * fix(billing): deny the payer pool to actor-less workspace API keys The first pass gated `credits` and `storage` on billing authority for personal API keys but let a `workspace_api_key` principal through unconditionally, which left the excluded role a way back in. Any workspace `admin` may mint a workspace API key, and a workspace `admin` is deliberately not a billing manager, so an admin who reads `null` as themselves could mint a key and read the full pool with it. On an organization-hosted workspace that pool is the organization's, spanning workspaces the admin has no standing in. Billing authority is payer identity or an organization admin role — a property of a person. A workspace API key is deliberately actor-less, so it can never satisfy it and now reads both fields as `null`. Attributing the key to its creator was rejected: it would launder the same workspace-admin role, it breaks when the creator's authority is revoked while the key lives on, and substituting a key's owner for the acting principal is what the application operation boundary forbids. The reasoning sits in TSDoc at the decision point. The key keeps the plan, period, and standing it needs to monitor a workspace, including `limit_exceeded` and `billing_blocked`. No in-repo caller reads `credits` or `storage` from this endpoint. The payer storage pool is now read only once disclosure is authorized, so a caller who may not see it no longer triggers the query at all. * fix(folders): bound the workflow folderId-branch path index reads `createWorkflow` and `updateWorkflow` each resolve a folder two ways inside one function. The folderPath branch goes through `resolveWorkflowFolderPath`, which loads the path index with `maxRows: MAX_FOLDERS_PER_WORKSPACE`; the folderId branch loaded it with no bound at all, issuing a `SELECT` over every active folder row in the workspace. In `updateWorkflow` the unbounded read and the bounded fallback sit thirty lines apart in the same function. Passes the cap at both sites, matching the read sites that already opt in. Exceeding it throws `FolderCollectionLimitExceededError` rather than truncating, because a partial path index resolves real folder paths to `undefined` and re-roots resources at the workspace root. `maxRows` deliberately stays opt-in rather than becoming the default. Folder creation does not refuse at the same ceiling on every path — `POST /api/folders` goes through the `createFolder` name/parentId variant, which passes no `maxFolderRows`, so the count guard in `executeCreateFolderAtPath` never runs and a workspace can already hold more than `MAX_FOLDERS_PER_WORKSPACE` folders. Defaulting the bound would make every path-index consumer throw for a state the product allows to exist. Reconciling reader and writer is a separate change with a user-facing limit, not a chore. * chore(billing): tidy payer-pool concealment cleanup * fix(api): reject an undecodable offset cursor on v2 table rows GET /api/v2/tables/{tableId}/rows coerced an undecodable pagination cursor to offset 0 and re-served page one. A client paging forward reads that as a fresh first page and can loop over it forever. Every sibling v2 cursor list — logs, files, workflows, workflow runs, workflow versions, workspace members, tables, knowledge documents — already rejects with a validation error instead. Extracts the offset-cursor decode both offset-paginated v2 routes had inlined into `decodeOffsetCursor`, next to the existing `decodeSortedCursor`, so the reject-don't-restart rule has one home. * fix(api): restore v1 table error-response parity and stop internal message leak The v1 table routes were rewritten to consume `lib/table/orchestration` results, and two response behaviors drifted from what the live API returned. Information disclosure: an unclassified failure's `outcome.error` carries whatever text the fault happened to have. Drizzle wraps a throw raised inside a transaction in an error whose own message is the failed statement and its bound parameters, so `DELETE /api/v1/tables/{tableId}` and `DELETE /api/v1/tables/{tableId}/rows/{rowId}` returned that verbatim in the 500 body to any API-key holder. Previously these returned a fixed generic string. Lost `lock` field: the 423 body used to be `{ error, lock }`. The delete, row-delete, and column-update routes (v1 and internal) dropped the lock kind the orchestration result already computes, leaving clients unable to tell which lock to clear. Both are fixed at one altitude: `orchestrationOutcomeErrorResponse` in `app/api/table/utils.ts` is now the only way a table route projects an orchestration failure onto the wire. It renders the route's fallback for an unclassified failure and the real message for a classified one (validation, not-found, conflict, locked keep their specific text), and carries `lock` on a 423. A future route cannot reintroduce either bug by hand-spelling the body. Duplicate table names on `POST /api/v1/tables` keep answering 409 rather than reverting to the previous 400. 409 is the correct semantic, and every other v1 duplicate-name surface (knowledge, files, workflow import) already answers 409; the tables 400 was the outlier. v1 tables appears in no published OpenAPI document and no in-repo client branches on the status, so the compatibility cost is limited to a caller matching 400 specifically for a name collision. * fix(skills): only reject a built-in name collision on an actual rename The built-in-name guard ran on every update that carried a `name`, without comparing it to the skill's current persisted name. Skills created before the guard existed can legitimately carry a built-in's name (they simply shadowed the built-in at read time), and the skill modal always submits the full object including the unchanged name — so every save of such a skill returned 400 with "The skill name ... is reserved by a built-in skill", with no way to fix it short of renaming. Move the guard in `updateSkill` to after the canonical row is loaded and run it only when the submitted name differs from the current one. Creating a skill with a built-in name, and renaming an existing skill into one, are still rejected. The check stays in the shared orchestration primitive because that is the only layer both the internal `/api/skills` adapter (via `performUpdateSkill`) and `updateSkillUseCase` (v2 + Copilot) pass through, and it is where the current name is in hand. * chore(tables): tidy v1 error projection cleanup * chore(skills): tidy collision guard cleanup	2026-08-11 16:41:27 -07:00
Waleed	be5db68644	fix(agiloft): repoint the block at the alrest surface and fix EWLogin (#6562) ... * fix(agiloft): make the block work, and align it with the REST documentation The native Agiloft block could not authenticate against any instance. A customer reported it; production traces for their workspace confirm every failure mode verbatim. Fixing that exposed a second, larger problem, and a per-endpoint audit against the full published documentation found the rest. Authentication - EWLogin sent only $KB/$login/$password as query parameters. A live instance answers `400 EWWrongDataException ... One has to specify $table, $KB, $lang parameters`. $table is required even though only $KB/$login/$password/$lang are documented. Parameters now travel in a form-encoded body, which the docs permit and which keeps the password out of URLs and access logs. - The authentication scheme is read from the login response and trimmed; Agiloft returns it as "Bearer " with a trailing space. - EWLogout was missing $lang. Surfaces - Record create, read, update, search and saved-search now use the endpoints that accept the token EWLogin issues; the legacy operations authenticate from inline credentials, which is what that surface expects. Nothing sends both forms at once — the documented 400 for doing so is what the original report had run into. - EWSelect passes credentials in a POST body, one of the five operations documented to support it. - Attachment retrieval uses the documented EWRetrieve endpoint, with filePosition rather than position, and no longer needs a login/logout pair. Defects found in the audit - remove_attachment reported zero on every call: its body is the EWREST assignment form but the route ran JSON.parse then Number(), yielding NaN. - The EWREST parser could not read EWActionButton's documented response, which puts both assignments on one line. - EWLock treated any 200 as success, including the documented {error, error_description} envelope, and invented an 'UNKNOWN' status. - EWTable discarded the linked-field details, required flag and text field type it had asked for, making includeLinkedInfo inert. - select_records had no result ceiling at all; both it and search now cap and report a truncated flag rather than reporting a capped length as a total. - Optional string inputs rejected null, so a blank Page field failed validation before any request was made. - Upsert treated the documented 202 async acknowledgement as a missing-ID failure, and returned no callback ID for the caller to poll. - Every response contract required an output that the 401 and 500 paths never return. Coverage added - Table and field discovery (EWTable), upsert (EWUpsert), async status (EWAsyncStatus), natural language search (EWNLPSearch), action buttons (EWActionButton), the REPLACE_WITH_ANOTHER delete rule with its substitute records, $async on upsert, and <fieldName>$overwrite on attach. - Reads with a named field list go through the search projection; an unfiltered contract record runs to roughly 184KB and swamps downstream agent context. - Errors are readable: Agiloft wraps failures in HTML around a typed exception and an internal task id, and the JSON endpoints now request real status codes rather than a 200 the caller has to interpret. Not implemented: $searchSQL and $operationHints=NOLOCK are EWRead/EWUpdate parameters and those operations do not run on that surface here; EWQuestion, EWHotlinks, EWOData, EWBroadcast and webhook registration have no documentation beyond their names. Verified against the published documentation, not against a live instance. * fix(agiloft): give natural language search a sentence that paints check:canvas-sentences failed: the nlp_search card resolved to nothing on an untouched canvas, so it painted empty. Its only basic-mode field was the long-input query, and the field list is advanced, so every segment dropped. The sentence now leads with the knowledge base, matching the shape List Tables already uses — both operations are knowledge-base scoped rather than table-scoped, so it also reads more accurately. * fix(agiloft): stop retrying refusals, and expose the outputs the new operations return Five findings from review that had gone unanswered. An Agiloft refusal was surfacing as HTTP 500. readAlrestJson throws when the envelope reports success:false, the route catch mapped that to 500, and the tool runner retries 500s — so a create the server had already rejected could be retried and duplicate the record. Refusals now return a settled failure with the message intact; genuine faults still 500. list_tables could not run in its primary mode. EWTable is knowledge-base scoped, but some instances reject EWLogin without a $table, so whole-knowledge-base discovery failed at login with nothing to fall back to. It now says what the caller can do about it rather than surfacing the raw login error. Upsert corrupted structured values. Every field went through String(), so a multi-value field collapsed into one joined string instead of the documented repeated key/value pairs, and an object silently wrote "[object Object]" into the record. Arrays now encode as repeated pairs and objects are refused, since Agiloft documents no encoding for them. Two outputs were invisible in the editor. `records` was conditioned on search_records alone, so natural language search results could not be chained, and `callbackId` on run_action_button alone, so a queued upsert's callback could not be wired into Async Status even though both values exist at runtime.	2026-08-11 16:31:32 -07:00
6 people	263e3ca67e	improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec (#5273) ... * v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * improvement(external-endpoints): v2 versions with clean signatures + updated docs * feat(usage): accept X-API-Key on usage-logs list + export /api/users/me/usage-logs and /export now use checkHybridAuth — the same auth /api/users/me/usage-limits already accepts — so external monitors can read summary.bySourceCredits (the source breakdown of usage-limits' aggregate currentPeriodCost) instead of estimating Copilot spend by subtraction. Workspace-scoped keys are pinned to their own workspace's slice of the ledger: the filter defaults to the key's workspace and an explicit mismatch 403s. Both endpoints documented in openapi-core.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(billing): dedicated v2 usage endpoints; keep internal usage routes session-only Replaces the earlier X-API-Key enablement on /api/users/me/usage-logs with a dedicated public surface, so the internal Billing-settings endpoints can evolve with the UI while external monitors get a stable versioned contract: - GET /api/v2/billing/usage — current-billing-period summary with bySourceCredits (the source breakdown external monitors need to watch e.g. Copilot consumption without estimating by subtraction), plus limitCredits and plan - GET /api/v2/billing/usage/logs — cursor-paged credit ledger in the v2 envelope - workspace-scoped keys are pinned to their own workspace's slice; personal keys read the account ledger The public wire is credits-only: usage-logs rows now carry a hasCost boolean instead of dollarCost (the Billing UI only needed the >0 signal), and the rateLimit block is removed from the usage-limits response and docs (deploy-modal tab relabeled accordingly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(docs): validate OpenAPI specs against the Zod contracts in CI The specs in apps/docs are hand-authored because they carry what Zod never defines — error envelopes, status codes, prose, examples — so they can't be generated; check:openapi validates them instead: - spec integrity: $refs resolve, operationIds unique, 2xx documented, no orphaned component schemas - v2 conventions: every /api/v2 operation documents 401 + 429 and every 4xx/5xx resolves to the canonical { error: { code, message } } envelope - contract cross-check: contracts are auto-discovered from lib/api/contracts/v2 (each carries its method + path); doc<->contract coverage both ways, query/body/response field diffs via z.toJSONSchema - examples: documented request/response examples must parse with the matching contract's actual Zod schemas First run caught real drift, fixed here: 16 stale orphaned schemas in the core spec, the v2 billing ops referencing v1-shaped error components, deploy/rollback examples missing the required nullable lifecycle keys, CreateTableBody missing folderId, a legacy-grammar delete-rows example, and four knowledge document ops missing their required workspaceId query param. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(docs): recursive field diff in check:openapi + the deep drift it found A mutation test showed the doc<->contract field diff only compared top-level properties, so a typo inside the { data } envelope passed. The diff now descends through matching object properties and array items (both sides must expose a property set — passthrough contracts and prose-only docs end the descent instead of false-positive), with the Zod JSON-schema root doubling as the $defs context. Deep drift it immediately caught, fixed here: select-column config (options/multiple) missing from every tables column schema, AddColumnBody hand-rolling a third column shape (now composed from ColumnInput, with position/workflowGroupId as the per-op extensions the contracts actually admit), chunking strategyOptions undocumented, and the deployment lifecycle fields (activeDeployment/latestDeploymentAttempt) missing from DeploymentState. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(security): close the triggerType rate-limit bypass on workflow execute Caller-supplied triggerType flowed unchecked into preprocessExecution, whose checkRateLimit default turns OFF for 'manual'/'chat' — so any API-key caller, and any anonymous public-API caller billed to the workspace owner, could execute unthrottled by sending {"triggerType":"manual"} (async runs also skipped the worker-side check via admissionCompleted). External callers may now only send the redundant 'api' value; internal JWT callers ('workflow'/'mcp') are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * refactor(execution): extract enqueue/status/cancel into shared libs Prepares the v2 execution surface: handleAsyncExecution's queue logic moves to lib/workflows/executor/enqueue-execution.ts (slot/claim semantics encoded in a discriminated outcome, not HTTP statuses), the execution-status read to execution-status.ts, and the order-sensitive cancel machinery to lib/execution/cancel-workflow-execution.ts. The v1 routes re-render identically — their suites pass unmodified. Also: preprocessExecution gains rateLimitCounter ('sync'|'async') and its 429 now carries code RATE_LIMIT_EXCEEDED + retryAfterMs (previously indistinguishable from the concurrency 429 and Retry-After was discarded); and the duplicate cancel contract in contracts/logs.ts is unified on the full 5-value reason enum — its narrower copy made requestJson throw a client ZodError when cancelling a paused HITL run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): callable execution service + structured error classifier executeWorkflowService composes the same libs the v1 route holds inline (call-chain guard, execution-id claim, LoggingSession, preprocessing, deployed-state load + file-field processing, timeout-bound executeWorkflowCore, output hydration/compaction) for the deployed-state caller class — the seam the v2 execute route and in-process internal callers share, making the HTTP endpoint syntactic sugar. classifyExecutionError stops discarding the block context that buildBlockExecutionError already attaches at throw sites: failed runs now yield {message, code, blockId, blockName, blockType} with a stable append-only code enum (TIMEOUT/CANCELLED/USAGE_LIMIT_EXCEEDED/ INVALID_INPUT/BLOCK_EXECUTION_FAILED/CHILD_WORKFLOW_FAILED/ OUTPUT_TOO_LARGE/EXECUTION_FAILED), so callers route on error class instead of substring-matching messages — the single place raw errors are interpreted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): POST /api/v2/workflows/[id]/execute Thin route over executeWorkflowService: X-API-Key or anonymous public-API auth (sync/stream only for anonymous), strict body with body-flag async (no mode headers on v2), SSE passthrough for stream, and the execution resource response — executionId always present, in-band run failures are status:'failed' with the structured {message, code, blockId, blockName, blockType} error, sync timeout is status:'failed' + TIMEOUT instead of v1's 408, and a Response block's payload stays inside output (authors never control response status/headers on this origin). Async debits the async bucket and the 202 statusUrl points at the v2 executions resource. Adds CLIENT_CLOSED_REQUEST/SERVICE_UNAVAILABLE to the v2 error codes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): v2 executions status + cancel with queued backfill GET /api/v2/workflows/[id]/executions/[executionId] is the single status URL for sync and async runs: before the async worker writes the durable log row, status is backfilled from the job queue (deterministic job id) as 'queued'/'running' — closing v1's 202-to-pickup 404 window — and failed runs carry the structured error object. POST .../cancel renders the shared cancellation lib in the v2 envelope with the tightened 5-value reason enum. Both authenticate via the shared resolveV2WorkflowAccess (X-API-Key, authz masked as 404, allowPersonalApiKeys honored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): workflow tool + MCP bridge run in-process workflow_executor (workflow-as-agent-tool) short-circuits in executeTool through WorkflowBlockHandler — the same invocation boundary canvas child workflows use — mirroring the deployed_block_executor precedent. The MCP serve bridge calls executeWorkflowService directly instead of fetching its own execute endpoint; deployment-version pinning, MCP response-size rejection, and the actor override become typed options instead of header sniffing. Both callers drop the double admission slot and duplicate top-level log row the HTTP hop cost, and failed child runs now surface the structured error + child executionId so parents and MCP clients can route on error class and hand providers a reproducible handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(infra): CORS + CSP coverage for the v2 execute path /api/v2/workflows/:id/execute gets the same wildcard-origin, credential-free CORS policy as v1 (the default credentialed policy would block browser API-key calls and open a cookie CSRF surface) with X-Sim-Stream-Protocol allowed and no X-Execution-Mode (async is body-selected on v2), plus the COEP/COOP/CSP header block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(ui): deploy modal + copilot advertise the v2 execute surface All 20 API-tab snippets move to POST /api/v2/workflows/{id}/execute with the nested {"input": ...} body, async as the "async": true body flag (X-Execution-Mode gone), status polling against the v2 executions resource, the third tab renamed Usage and pointed at /api/v2/billing/usage, and {data} envelope unwraps in the printed responses. Fixes the latent baseUrl derivation (endpoint.split('/api/workflows/')) that would have silently built garbage URLs under a v2 endpoint, and deletes dead code (exampleCommand across 3 sites, getAsyncExampleTitle). Copilot deploy/manage/serializer endpoint builders and the api_trigger bestPractices example follow (the latter also drops its hardcoded staging host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * docs(api): document the v2 execution surface Adds execute, execution status, and cancel to openapi-v2-workflows.json with the structured ExecutionError schema (append-only code enum + block attribution) and the ExecutionResource contract, documenting the rules that differ from v1: modes are body-selected, a failed run is HTTP 200 with status 'failed', an executionId always means data (never the error envelope), queued status is visible immediately, and Response-block payloads stay inside output. Registers the three pages in the generated workflows meta.json and bumps the route-count baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): gate the whole /api/v2 surface behind one flag; UI stays on v1 Every v2 route now runs exactly one check immediately after auth — v2ApiGateError — and answers 404 when the `v2-api` flag is off, so the surface is invisible until it is deliberately rolled out. The gate is keyed on userId only: a workspace/org-keyed check would have to read membership for a caller-supplied id before authorization runs, and its 404-vs-403 split would leak cohort membership (the trap the per-domain table gate worked around by running late). The two executions routes inherit it from the shared access resolver; the tables-specific gate is removed so no route checks twice. `tables-v2-api` stays, now gating only the internal predicate-grammar route /api/table/[tableId]/query — note v2 tables routes move to the unified flag, so enabling them is a `v2-api` decision now. Reverts the deploy modal, copilot handlers, and api_trigger example to the v1 execute endpoint: v1 works unchanged, and the UI must not advertise a surface most users would get a 404 from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(executor): restore child-cost aggregation dropped by the staging merge Staging's custom-block rewrite deleted `aggregateChildCost` from workflow-handler.ts, and git merged that file cleanly — but this branch's workflow-tool-runner.ts, added for the v2 execute migration, still imports it. A silent semantic conflict: no marker, broken build. Taking staging's rewrite is correct, so the helper is defined locally in its one remaining consumer rather than resurrected in the file staging just rewrote. Same four lines over the still-exported `calculateCostSummary`, so a failed child workflow keeps billing the hosted-key spend it consumed instead of reporting $0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation (#6134) * refactor(orchestration): move the shared error contract out of lib/workflows OrchestrationErrorCode and statusForOrchestrationError are the contract every lib/[resource]/orchestration module returns against, but they lived inside the workflows module, so resource-neutral code (lib/folders) already had to import from a workflow path. Moved to lib/core/orchestration/types. Adds a 'locked' class mapping to 423. Both tables and workflows have a lock that forbids a mutation, and each caller was translating that to a status itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation Column update was implemented four times — the UI route, v1, v2, and the copilot table tool — each calling the same column services but owning its own guards, error mapping, and audit. The copies had drifted, and the drift was the bug: v2 was missing both guards, only the copilot copy minted stable option ids, and only v1/v2 audited. performUpdateTableColumn, performDeleteTable, and performDeleteTableRow now own that logic; all ten call sites reduce to auth, parse, call, render. The guards are asserted once in lib/table/orchestration rather than four times against four routes. Behavior this consolidates, previously true on only some paths: - The typeChanging guard. updateColumnType early-returns on an unchanged type and drops any options sent with it, so restating the current type alongside new options silently discarded them. v2 had no guard at all and, since its contract shares v1's body schema, accepted options and ignored them. - The select-unique guard. Each write is its own locked transaction, so a rename or type change paired with a constraint write that is going to fail commits first and then throws, half-applying the schema change. - Stable select-option ids. Cells reference the option id, so an edit that re-sends an option by name has to reuse it or every cell holding it is orphaned. Only the copilot path did this; normalizeSelectOptionsInput moves to lib/table/select-options and now covers every caller. It preserves a supplied id, so it is a no-op for the fully-formed options the HTTP contracts accept. - required forwarded into the type and options writes, so a conversion validates against the constraint the same request is setting. - An audit on every successful update. The UI route and the copilot tool emitted none. - Single-row delete through the row service. v2 did a raw db.delete, skipping assertRowDelete and deleteOrderedRow, so a delete-locked table returned 200 and the row-count bookkeeping never ran. - The delete actor handed to deleteTable, which audits only when a row was actually archived. v1 and v2 omitted it and audited themselves outside that check, emitting TABLE_DELETED for a no-op delete of an archived table. Failure classes come back as OrchestrationErrorCode; v2 renders them through a new v2ErrorForOrchestration, mirroring statusForOrchestrationError on the v1 and UI surfaces, so a given failure maps to the same status everywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(tables): bind the column-update tests to the orchestration function The base's route tests assert which column service each payload reaches — the behavior that now lives in performUpdateTableColumn. They mocked the `@/lib/table` barrel; the orchestration module imports the service directly, so they mock that too and keep asserting the same thing through the extracted implementation. The orchestration tests move onto the base's semantics: writes address the stable column id, a rename rides inside the write it accompanies rather than running first, and the currency guards replace the non-select options guard the service now owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(copilot): drop the column-type import the delegation made dead Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): move the audit log out of the table service `lib/table/service.ts` wrote its own audit rows, so whether an operation was audited depended on which function a caller reached for rather than on a user having performed it. That is what let v1 and v2 audit a no-op delete, and what made `deleteTable`'s optional `actingUserId` double as an audit opt-out flag. Worse, most sites fell back to `actingUserId ?? createdBy`, so an unattributed call was logged against the table's *creator*. The copilot `mv` path passed no actor at all: renaming someone else's table recorded them as the renamer. Audit now lives in the orchestration functions — performDeleteTable, performRenameTable, performMoveTableToFolder, performUpdateTableLocks — and the services just write. Internal callers (folder cascade, import rollback) keep calling the service and are silent by construction rather than by remembering to omit an argument. Two services now return what the audit needs: `deleteTable` reports whether it actually archived a row, so a repeat delete logs nothing; `updateTableLocks` returns the before/after locks, since only the locked write can observe the transition its description names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(tables): restore audit provenance and conflict status in orchestration Moving the audits into the orchestration functions dropped three things the routes had been carrying, and added one the orchestration now owns twice. - The v1 and v2 column-update routes passed `request` to `recordAudit`, so their audit rows recorded the caller's IP and user-agent. The orchestration function had no way to receive it. Every table orchestration function now takes an optional `OrchestrationRequestContext` and every HTTP route forwards it; the copilot and VFS callers, which have no request, omit it. - `classifyTableMutation` matched `TableConflictError` on "already exists" appearing in the message and reported it as `validation`, turning the UI route's 409 on a duplicate table rename into a 400. It now matches the type, the way `performRestoreTable` already did. - `captureServerEvent` ran on every delete while the audit was gated on a row actually being archived, so a repeat delete of an archived table still reported `table_deleted`. Both now hang off the same evidence. - The copilot delete path kept its own `captureServerEvent` from when the service did not emit one, double-counting every copilot table delete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * fix(tables): say which type a no-op column update restated A copilot `update_column` payload whose only content was the column's current type used to return success with the live schema, while the v1, v2, and UI routes rejected the same payload with "No updates specified". Delegating to `performUpdateTableColumn` unified them onto the routes' rejection — correct, but the message tells the caller its request was empty when it named a type. The orchestration function now reports the same thing `updateColumnType` reports when it loses this race concurrently: the column is already that type, re-issue without the type change. An empty payload still reads "No updates specified". Drops the copilot's `outcome.table ?? tableForUpdate` fallback with it — the comment described the no-op that can no longer reach that line, and a success always carries a table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * refactor(tables): classify failures by type instead of by message text The table module decided HTTP statuses by searching error messages for phrases. `VALIDATION_MESSAGE_FRAGMENTS` and `ROW_WRITE_ERROR_PATTERNS` held 32 substrings between them, and fifteen more lists were inlined in routes — 83 matchers over 17 files, each its own copy of the guesswork and already drifted apart. It made message wording load-bearing: `TableRowLimitError`'s own doc comment noted that its text had to contain "row limit" for a route to answer 400, and adding "already exists" to a rename message silently demoted a 409 to a 400 (the bug fixed one commit ago, by adding another special case). Services now throw `OrchestrationError`, which carries the transport-neutral `OrchestrationErrorCode` the layers above already speak. Classification is one `instanceof` in `orchestrationErrorResponse` (UI + v1) and `v2CaughtOrchestrationError` (v2). Every pattern list is gone. Wording is free to change; an unclassified error still becomes a generic 500, which is what an unexpected fault should be. `asOrchestrationError` walks the `cause` chain rather than testing the caught value directly: drizzle wraps a throw raised inside a transaction callback in a `DrizzleQueryError` whose own message is the failed SQL, so a bare `instanceof` would drop every failure raised inside `withLockedTable`. That is the same reason `rootErrorMessage` had to dig for a root cause before. Three throws stay bare `Error` deliberately — `Table ID mismatch`, `Workspace ID mismatch`, and `Failed to build upsert conflict predicate` are internal invariants no consumer classified, and they keep falling through to a 500. `Insufficient capacity` was in the pattern list with no producer anywhere in the codebase. Status changes, all deliberate: - `'forbidden'` joins the code union so the table-row-limit ceiling keeps its 403; without it this refactor would have flattened it to 400. - import-async's table-limit rejection: 400 -> 403, matching the two other create routes it had drifted from. - Renaming a table to an invalid name: 500 -> 400. `validateTableName` messages don't contain "Invalid", so no matcher ever caught them. - Restoring a table that isn't archived, or into an archived workspace: 500 -> 400. - A duplicate *column* name stays `validation`/400 rather than becoming a 409 like a duplicate table name. Both v1 and the orchestration have always answered 400 for it; changing a published status is not this refactor's job. The twelve tests that changed were asserting the substring mechanism itself, constructing plain `Error`s with magic strings. They now assert the real contract, plus new cases pinning that identical wording carrying no classification stays internal and keeps its message off the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials (#6150) * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials * fix(api): correct credential role, skill permission bar, MCP url identity, and custom-tool conflict mapping * fix(api): align credential mutation gating, provider-outage status, and unique-violation conflicts * fix(api): close unique-violation, revival, orphan-write, and env-rename gaps * fix(api): treat every provider-outage code as unavailable on create and update * fix(credentials): use the shared outage predicate on the session update path * fix(contracts): anchor the predicate double-cast annotation to the cast `check:api-validation:strict` counted 9 unannotated double-casts against a baseline of 8, failing CI. The predicate leaf schema was annotated, but the annotation sat above the declaration while the checker anchors on the line carrying the cast — five lines below, at the close of the object literal. The scanner walks back at most three lines and stops at the first non-comment one, so it hit `value: z.unknown().optional(),` and never saw the reason. Splitting the object schema from the cast puts them adjacent, so the existing reason binds. No behavior change — the cast, the schema, and the reasoning are unchanged. Also lowers the rawJsonReads ratchet 6 -> 5 to match the current count, which had drifted down; leaving it high lets a removed raw read silently come back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(skills): point the orchestration error contract at its moved module #6150 branched before #6134, so skill-lifecycle.ts imports @/lib/workflows/orchestration/types — the module #6134 moved to @/lib/core/orchestration/types. Git merged a file deletion on one side with a new file referencing it on the other: no textual conflict, broken build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(knowledge): make lib/knowledge/orchestration the single implementation (#6154) * refactor(knowledge): make lib/knowledge/orchestration the single implementation Knowledge base create was implemented four times — the internal route, v1, v2, and the copilot tool — and the orchestration around the shared write had drifted. Extract it the same way lib/table/orchestration was: services write, orchestration decides which writes run, guards them, audits them, and returns a transport-neutral failure. Behavior converged, not preserved: - One chunking default (DEFAULT_CHUNKING_CONFIG). The agent defaulted minSize to 1 against the API's 100, so identical input produced differently-chunked knowledge bases depending on who created it. The agent path now chunks at 100. - Every successful mutation is audited inside the orchestration function. The copilot tool called recordAudit zero times, so agent-created knowledge bases, document uploads, updates and deletes left no audit trail at all. - Failures classify by class, not by message text. The knowledge service errors are OrchestrationError subclasses and storage-quota rejections throw a shared StorageLimitExceededError, replacing four separate message greps for "already exists" / "does not have permission" / "storage limit". delete_connector reported the opposite of what happened. It reached the route through an internal HTTP self-call that sent no query string, so the route's keep-documents default always applied while the agent told the user the documents had been removed. The self-call is gone — all four connector operations run in-process — and the orchestration returns the real counts. Also: - OrchestrationErrorCode gains 'payload_too_large' (413 / PAYLOAD_TOO_LARGE). Without it, dropping the storage-limit message match would have regressed the documented 413 on knowledge base create and document upload to a 500. - messageForOrchestrationError renders a route's own wording for an unclassified fault, so a driver's message no longer reaches the client on a 500. - v1 and v2 knowledge base update now forward actorUserId, which the service requires for a workspace move; both omitted it. - The connector DELETE route reads deleteDocuments through parseRequest. Its contract declared z.boolean(), which would have rejected the string a query param actually is. - Drop the 409 from POST /api/v2/knowledge/{id}/documents in the OpenAPI spec. Nothing on the upload path throws a conflict; it was only ever reachable by the message match this change removes. Behavior change worth noting: a v1/v2 PUT carrying only the workspaceId scope field and no actual updates now returns 400 rather than 200 with the unchanged knowledge base. Deliberately deferred: document update remains internal-only. Extracting performUpdateKnowledgeDocument makes exposing it on v1/v2 a contract and a route away, but that is a new public surface rather than part of this consolidation. * fix(knowledge): make connector create atomic and stop flattening failures Review round 1 on #6154. - Resolve the billing payer before the connector is committed, not after. A malformed attribution header rejected post-commit left a live connector behind a 500, and a retry created a duplicate plus duplicate sync work. Manual sync resolves before writing its audit for the same reason. - Let the source-config validator carry its own failure class. Collapsing every rejection to `validation` flattened the connector PATCH route's 401 (stale stored credential) and 409 (missing workspace context) into a 400. - Add `unauthorized` to OrchestrationErrorCode. It is the class that 401 was already expressing on this route, and the v2 vocabulary already had UNAUTHORIZED; only the shared union was missing it. - Report a knowledge base that exists but failed to archive as failed, with the reason, rather than as not found. The copilot delete loop folded every non-not-found failure into `notFound`, telling the user it was never there. - Route copilot failures through the same message helper the HTTP surfaces use, so an unclassified fault's raw text (a driver's failed SQL) no longer reaches the agent verbatim while the UI and public APIs get the generic wording. * feat(api): expand the public v2 files surface (#6160) * feat(api): expand the public v2 files surface Adds folder support, rename/restore, move, bulk archive, share, and content replace to /api/v2/files, so managing files by API no longer stops at upload + download + archive-one. Routes are thin: auth -> parse -> perform* -> serialize. Share and content replace get their orchestration extracted first so the session routes and the public ones cannot diverge on the effective-authType resolution, the EE public-sharing gate, or the storage-quota classification. Presigned upload stays session-only: presign does an advisory quota check and the real debit happens in the separate register step, so a caller that never registers leaves unaccounted bytes with no reaper. The buffered multipart path debits inside uploadWorkspaceFile's own transaction. * fix(files): classify folder and content failures instead of 500ing them Bugbot round 1. The v2 routes map errorCode straight to a status, so every manager failure that arrived unclassified became a 500 for what is really a caller-fixable 400 or 404. - Folder manager throws OrchestrationError: missing target/folder -> not_found, reparent cycle / self-parent / restore-into-archived-workspace -> validation. - File manager does the same for the in-transaction 'File not found' paths that the earlier pass missed. - updateWorkspaceFileContent's outer catch re-wrapped everything in a bare Error, which stripped the class off StorageLimitExceededError and the new not_found alike. It now rethrows a classified failure untouched and attaches cause to the generic wrap, so asOrchestrationError can still walk the chain. - Every remaining perform* gained the asOrchestrationError branch. - renameWorkspaceFile returned the pre-update read, so the v2 PATCH reported a stale updatedAt; it now returns the timestamp it actually wrote. Docs: upload auto-suffixes a duplicate name rather than rejecting it, matching the in-app uploader. The description claimed 409 and was simply wrong. * fix(files): surface a failed upload read-back as the real error getWorkspaceFile swallows a query failure and returns null unless throwOnError is set, so a transient blip on the post-upload read reported as 'file could not be read back'. Distinguish the two: a real null after a just-committed write is an invariant break, a query failure is itself. * revert(api): drop the dedicated v2 file-folder routes File folders already live in the shared folder table as resourceType 'file' (#6045 cut them over, #6051 dropped workspace_file_folders), and the remaining file-specific folder machinery is being folded into the generic folder engine. Publishing /api/v2/files/folders/** would pin that transitional split into a public contract we'd then have to keep or break. Files stay folder-aware — folderId/folderPath on the projection, folderId on upload, and the move route — because a folder id is a folder.id and survives the unification untouched. Folder management belongs on /api/v2/folders once that surface serves resourceType 'file'; until then there is no v2 way to enumerate file folders, which is the deliberate gap. The orchestration classification fixes stay: the internal routes and the copilot file-folder tools still call those perform* functions. * fix(files): classify upload failures instead of matching their wording Bugbot round 2. uploadWorkspaceFile had the same outer-catch rewrap that updateWorkspaceFileContent did, so a blown storage quota reached the route as a bare Error and the v2 handler recovered the status by substring-matching the message. Any rewording silently demoted a 413 to a 500. - uploadWorkspaceFile rethrows a classified failure untouched and attaches cause to the generic wrap. - FileConflictError is now an OrchestrationError('conflict'), so a duplicate name classifies like every other conflict. Its 'FILE_EXISTS' discriminator had no readers and is gone; the instanceof checks elsewhere still hold. - The v2 upload handler uses v2CaughtOrchestrationError, dropping all three string matches. Also documents that bulk-archive is best-effort: unknown or already-archived ids are skipped rather than failing the call, and deletedItems is what actually happened. That asymmetry with the single-id DELETE was undocumented. * feat(api): add search, filtering, and sorting to the v2 list endpoints (#6189) * feat(api): add search, filtering, and sorting to the v2 list endpoints One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts: `search` (case-insensitive substring on the resource's natural name field), `sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship rather than inventing a third dialect alongside the Logs filters and the Tables predicate grammar. Every filter and sort is pushed into SQL. GET /api/v2/files previously read the whole scope and sorted/sliced it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Cursors are stamped with the sort they were minted under, so replaying one under a different sort is a 400 instead of silently duplicated or skipped rows. * fix(api): validate v2 cursor key values and compare timestamps at ms precision Two review findings, fixed at the root by making a keyset key own its cursor codec instead of hand-writing a decoder per sort. Cursor key values are caller-controlled, and matching the sort stamp and key count was not enough: an unparseable timestamp or a non-numeric size reached the query as an Invalid Date or NaN and surfaced as a 500. Each key now type- checks its own value and rejects a cursor it cannot hold, which both routes render as the documented 400. Timestamp keys now order and compare on date_trunc('milliseconds', col). Postgres keeps microseconds and defaultNow() populates them, but a cursor value round-trips through a millisecond-only JS Date — comparing the raw column against the truncated value re-admitted the page's own last row, duplicating it and stalling pagination outright at a page size of one. Reachable today via workspace_files.updated_at, which insertFileMetadata leaves to defaultNow(). * feat(api): complete the v2 workflows resource with versions and CRUD (#6184) * feat(api): complete the v2 workflows resource with versions and CRUD Adds version listing/detail plus create, update, and delete to the v2 workflows surface, which previously covered only execution and deployment. - GET /api/v2/workflows/[id]/versions — cursor-paginated, newest first - GET /api/v2/workflows/[id]/versions/[version] — version + pinned state - POST /api/v2/workflows, PATCH and DELETE /api/v2/workflows/[id] All six delegate to the existing orchestration and persistence helpers; no new domain logic. * fix(api): check folder containment before lock state; reject malformed version cursors assertFolderMutable walks a folder's ancestor chain without filtering on workspace, so inspecting it before containment let a caller tell a locked folder in someone else's workspace (423) from a nonexistent one (400). Create and update now assert containment first, matching the ordering import-workflow.ts already uses. A version cursor that decodes to JSON without a numeric version filtered every row out and returned an empty page with nextCursor null, which reads as a clean end-of-list. Malformed cursors are now a 400. * refactor(api): page workflow versions in the persistence helper listWorkflowVersions read every version row and the route filtered and sliced the result in memory, so the response was bounded but the query was not. It now takes optional limit/afterVersion, turning the cursor into a real keyset query; the route asks for limit + 1 and only trims the has-more probe. Both params are optional, so the internal, v1 admin, and copilot callers are unchanged. Also restores the untouched GET handler in [id]/route.ts to its original formatting — collapsing its signature had re-indented the whole body and buried the actual additions in whitespace churn. * feat(api): expand v2 tables with stateless multipart transfers (#6188) * feat(api): expand the public v2 tables surface Adds 16 operations so a v2 caller can do what the internal surface can: rename/move/lock a table, restore it, manage saved views, run enrichment columns, look up rows, and import/export with observable job control. Extracts lib/table/orchestration/import.ts (performTableCsvImport, performCreateTableFromCsv) and lib/table/export-stream.ts from the first-party routes, then repoints those routes at them, so v1 and v2 cannot drift on what an import or export actually does. events/stream, metadata and dispatches stay internal — they are editor state, not public API. * fix(api): make v2 table PATCH all-or-nothing and name the lock in every 423 Greptile P1: PATCH applied locks, rename and move as three sequential transactions, so a folder rejected mid-request left the earlier writes persisted while the response reported failure — and the schema-changed signal was skipped, leaving open clients on stale state. Every rejectable condition now runs before the first write, and the signal fires whenever anything did land. Cursor: v2TableLockError dropped the lock kind, so async import, column run, enrichment and table mutations returned a bare LOCKED. A table has four independent locks, so the caller could not tell which to clear. * fix(api): report the lock kind on classified 423s too, not just thrown ones The previous commit named the lock only where the rejection was thrown and caught at the route boundary. Where it instead arrives as a classified `errorCode: 'locked'` outcome — delete table, delete row, update column, and the table mutations — the kind was dropped, so those 423s stayed unactionable while their neighbours improved. The orchestration results now carry `lock`, and a shared `v2TableOrchestrationError` renders both arrival paths into the same `{ code, message, details: { lock } }` body. `details` is omitted rather than sent null when the kind is unknown, so a caller branching on it sees absence instead of a phantom value. * fix(api): make async table imports observable, not just startable `POST /import-async` pointed callers at `GET /api/v2/tables/jobs` to track progress, but that endpoint filters to `type = 'export'` — imports are derived onto the table itself, one write job at a time, and exports get a separate list precisely because they are excluded from that derivation. The public Table shape omitted those derived fields, so an async import could be started and cancelled but never observed to completion, failure, or progress. That is the gap the import/export/job-control set was meant to close. Table now carries `job` — id, type, status, rowsProcessed, error, or null when idle — and the import-async docs point at the table rather than the export list. * feat(api): make v2 table PATCH state which operations landed on failure Greptile held the PR at 4/5 on the residual non-atomicity and named two acceptable resolutions: make PATCH atomic, or have the contract adopt and expose partial-success explicitly. Atomicity would mean threading one transaction through renameTable, moveTableToFolder and updateTableLocks — three shared service functions with four non-test callers including the first-party route and two copilot tools — and deferring their per-operation audits to commit time. That is a refactor of shared write paths well outside this PR. So the contract states it instead. Every rejectable condition is already pre-validated, so a failure here is a genuine fault; when one follows a successful operation the error now carries `details.applied` listing what is live. Absent when nothing applied, so its presence always means "these changes took effect despite the error". Documented on the operation. `v2ErrorForOrchestration` gained the optional `details` this needs. * fix(api): make table lock flags read-only on the public v2 surface The new PATCH /api/v2/tables/[tableId] accepted a `locks` object, gated on workspace admin plus the table-locks feature. That still lets an API key clear the guard placed there to stop it: `write` is the floor for the endpoint, and admin keys are ordinary API keys, so a lock is no longer a boundary the key cannot cross. Locks stay readable on the table resource and enforcement is unchanged (a locked verb still returns 423). Changing one is now a first-party admin action only. The v2 body is declared here rather than reusing the first-party updateTableBodySchema, which keeps its `locks` field so the UI can still toggle them. It is .strict(), so a request carrying `locks` is rejected with a 400 naming the field instead of silently succeeding without applying it. * fix(api): keep reporting applied operations when the PATCH re-read fails The composite table PATCH promises that `error.details.applied` names the operations that are live despite an error, but `applied` was scoped inside the try. A rename or move that committed and was then followed by a throw in the final re-read — or a re-read finding the table archived — returned a bare 500/404 with no details, telling the caller nothing had landed. It would then retry into a duplicate-name conflict or repeat the move. `applied` is now function-scoped so every post-write exit carries it: the 404 on a missing re-read, a thrown lock error, a classified orchestration error, and the generic 500. `v2TableLockError` gains the same `extraDetails` parameter `v2TableOrchestrationError` already had. * feat(api): add workflow group writes to the v2 tables surface v2 exposed GET /groups but none of the writes, so the public API could run an enrichment or workflow column and read its binding, but never create one. A caller could add a plain data column and trigger the machine; wiring the two together still required the UI. Adds POST/PATCH/DELETE on /api/v2/tables/[tableId]/groups. The group is the unit that fills columns — one group feeds several — so creating one creates its output columns in the same call, matching the first-party shape rather than inverting it onto the column endpoint. Four departures from the first-party body, all public-surface concerns: - group.id is optional and server-generated. The UI mints an id to render optimistically; a public caller has no such need and a client-chosen id is a collision waiting to happen. - outputColumns[].workflowGroupId is dropped from the body and stamped from the resolved group, so it cannot disagree with it. - autoRun defaults to false. First-party defaults true so a UI add fills cells immediately; here it would make one POST fan out a metered run across every existing row. - A group naming neither a workflowId (type manual) nor an enrichmentId (type enrichment) is a 400 rather than a half-specified group the route has to guess about. Also rejects an outputColumns entry no group output feeds — the two arrays are joined by column name, and the first-party client builds both from one picker so it cannot desync, but a public caller can. Workspace containment on workflowId is asserted before it is persisted, on create and on any update that re-points the group; without it a table becomes a way to invoke workflows the key cannot otherwise reach. * improvement(api): make v2 table import and export async-only Drops the three synchronous entry points: POST /tables/[tableId]/import, POST /tables/import-csv, and GET /tables/[tableId]/export. Sync import tied a write to the lifetime of an HTTP request. The body *was* the data, so it carried a 10 MB cap that Next silently truncates past — a partial import reporting success. It also had no job, so a timeout mid-write left rows in place with nothing to poll and nothing to cancel. The async path reads the file from storage instead: upload via POST /api/v2/files for a key, start with POST /import-async, watch GET /tables/[tableId] -> job, stop with POST /job/cancel. Sync export carried no such hazard, but one shape per operation beats two: with both removed the surface has exactly one way to move a table in or out, and the CLI wraps the extra calls. This also removes the last multipart handling in v2 tables. Those were the only routes bypassing parseRequest — form fields were parsed by hand against separate form schemas, outside the contract system every other v2 write goes through. Create-a-table-from-CSV is now two calls: POST /tables, then /import-async with createColumns. csvImportModeSchema is append|replace, so there is no single-call create. Route baseline 1064 -> 1061. * docs(api): correct the import-async note about upload size limits The docstring claimed there is no synchronous upload endpoint and so no request-body size cliff. Both are wrong: POST /api/v2/files is a synchronous multipart upload with a 100 MB cap, and it is the only v2 upload path (presigned is deliberately absent). What async-only actually bought: the cap went 10 MB -> 100 MB, it fails on an explicit size check and a bounded body read rather than a proxy cap that silently truncates, authorization completes before any body is buffered, and the table write is a job that can be watched and cancelled. * feat(api): unify file and table transfers * improvement(api): make multipart transfers stateless * fix(api): make table import completion retries idempotent * feat(v2-tables): paginate the table list `GET /api/v2/tables` returned every table in the workspace in one response — it used the cursor envelope but hardcoded `nextCursor: null`, and had no `limit`. That was defensible when tables were only created through the UI; `POST /api/v2/tables` is public now, so a script can create them in bulk and the list has no way to ask for less. Adds `queryTables` alongside `listTables` rather than changing it, so the internal callers that genuinely want the whole scope are untouched — the same split `queryWorkspaceFiles` / `listWorkspaceFiles` already uses. Filter, order and slice all run in the query, so a `search` never costs a full-workspace read. A cursor whose values don't bind raises a validation error instead of being coerced to "no filter", which would have silently served page 1 under a resumed cursor. The keyset closes on `id` so a page boundary inside a run of equal names or timestamps stays stable. The shared `LimitQuery` doc component said "Maximum rows to return"; it now serves the table list too, so the wording is resource-neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(api): add multipart knowledge document uploads * fix(api): keep usage admission at knowledge upload session creation * feat(knowledge): wire knowledge base uploads to multipart sessions * fix(knowledge): refuse to abort an upload once a document is bound * fix(uploads): prevent multipart cleanup races * Unify file creation and signed upload sessions (#6264) * feat(uploads): unify signed upload sessions * fix(uploads): preserve attachment storage semantics * feat(files): add authored file creation * fix(uploads): omit hoisted S3 metadata headers * feat(api): add file metadata endpoint * improvement(api): scope folders to resource paths (#6284) * improvement(api): scope folders to resource paths * fix(files): serialize folder resolution with uploads * fix(files): release folder lock before upload setup * fix(api): normalize folder paths and unblock resource mutations * fix(api): make resource cleanup and metadata consistent * improvement(uploads): persist multipart sessions in postgres * fix(db): store table row trigger timestamps in UTC * improvement(api): default folder deletion to non-recursive * fix(billing): unify chat usage source * improvement(logs): expose trace spans on log detail * fix(logs): parse list trace spans * improvement(api): replace workflow jobs with execution resources (#6294) * improvement(api): replace workflow jobs with execution resources * fix(api): preserve legacy jobs while preferring v2 executions * fix(api): make execution polling resume-aware * fix(ui): hide async examples for public workflows * fix(api): bridge resume queue visibility lag * feat(api): add v2 workflow resume endpoint * fix(api): project pending resume attempts * fix(api): prefer terminal logs over stale resumes * improvement(api): unify v2 resource query layers (#6319) * improvement(api): unify v2 resource query layers * fix(api): address v2 review findings * fix(api): preserve cancelled queue status * fix(api): guard cancelled job transitions * fix(api): close v2 resume and log gaps * feat(api): rename v2 executions to runs * feat(api): split credentials and secrets * feat(api): add workspace metadata and email attribution * improvement(api): consolidate public v2 route handling * improvement(files): centralize operations across APIs and Copilot (#6392) * improvement(files): unify rename authorization * chore(skills): add file operation migration guide * improvement(files): consolidate file operation authorization * improvement(files): extract shared operation foundation * improvement(api): simplify internal route declarations * improvement(files): centralize application authorization * refactor(api): share workspace file name validation * refactor(files): centralize copilot application calls * docs(skills): generalize application operation migration * improvement(api): centralize remaining v2 resource operations (#6412) * improvement(api): centralize v2 resource operations * fix(api): preserve custom tool conflict errors * improvement(api): migrate policy-sensitive v2 reads (#6410) * improvement(workflows): centralize v2 application operations (#6411) * refactor(api): migrate v2 knowledge operations (#6413) * refactor(api): migrate v2 knowledge operations * fix(knowledge): fail upload completion on dispatch errors * fix(knowledge): preserve upload retry and VFS errors * improvement(tables): centralize v2 application operations (#6414) * improvement(tables): centralize v2 application operations * fix(tables): preserve run validation and signals * feat(auth): add scoped internal executor delegation (#6459) * feat(auth): add scoped internal executor delegation * fix(auth): derive delegation lifetime from one timestamp * Include share status in file metadata * feat(auth): centralize delegated identity policy (#6462) * improvement(copilot): consolidate application adapters (#6450) * improvement(api): harden application route boundaries (#6451) * improvement(api): harden application route boundaries * fix(folders): reject creates at workspace cap * fix(knowledge): enforce trusted workspace scope (#6452) * fix(knowledge): enforce trusted workspace scope * refactor(knowledge): declare v2 body lifecycle * finish knowledge application migration * refactor(knowledge): compose copilot batch commands * fix(knowledge): parse connector query flags * fix(knowledge): finalize partial batch effects * fix(knowledge): align merged application boundaries * fix(knowledge): close application boundary review gaps * style(knowledge): satisfy branch biome checks * fix(knowledge): page connector documents in editor * refactor: enforce Copilot table application boundary (#6453) * refactor: enforce copilot table application boundary * fix(tables): finish application boundary migration * fix(tables): restore scoped copilot imports * fix(tables): compose copilot commands atomically * fix(tables): preserve workflow group scheduling * fix(tables): complete fixed copilot composition * fix(tables): reject enrichment output mutation * fix(tables): complete authorized application boundary * fix(workflows): migrate Copilot application boundary (#6455) * fix(workflows): migrate Copilot application boundary * fix(workflows): finish delegated application migration * fix(workflows): encode VFS folder aliases * fix(workflows): close application composition gaps * fix(workflows): preserve VFS validation errors * fix(workflows): complete application boundary migration * test(workflows): format canonical binding coverage * fix(workflows): scope executor metadata reads * fix(workflows): bind executor metadata targets * improvement(skills): align application operation guidance (#6532) * feat(api): expose v2 resource owners * fix(api): distinguish visible resource authorization failures (#6537) * feat(api): generate v2 OpenAPI from contracts (#6509) * feat(api): generate v2 OpenAPI from contracts * fix(api): preserve string boolean wire defaults * fix(api): document file download headers * fix(docs): use TypeScript CLI with Next.js * fix(docs): avoid client-rendered theme script * fix(api): document departed audit default * feat(api): replace legacy core docs with v2 * feat(api): generate v2 OpenAPI from contracts * feat(api): refine generated v2 OpenAPI docs * fix(docs): align localized v2 execution examples * fix(ci): restore Helm diff and sync audit mock * fix CI regressions after staging merge --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-11 05:45:25 -04:00
Theodore Li	56910c002e	feat(embeddings): add OpenRouter support (#6396) ... * feat(knowledge): add OpenRouter embedding fallback * fix(knowledge): preserve successful embedding batches * feat(embeddings): add OpenRouter provider * fix(knowledge): bill only platform embedding tokens * test(embeddings): include OpenRouter provider * feat(embeddings): load OpenRouter model catalog * fix(embeddings): preserve legacy provider default * fix(embeddings): batch OpenRouter requests * fix(embeddings): reset stale OpenRouter model	2026-08-10 15:15:18 -04:00
Waleed	303986f45f	feat(snowflake): credential-based auth, object pickers, and 9 new operations (#6474) ... * feat(snowflake): credential-based auth, object pickers, and 9 new operations Replace the per-block host + PAT fields with a Snowflake service-account credential, move the credential picker to the top of the block, back the object fields with metadata-only pickers, and add nine operations. - credential: snowflake-service-account token service account (account host + programmatic access token), verified against the SQL API with the same headers the tools use - selectors: database, schema, table, warehouse, execution role, file format and procedure pickers behind one /api/tools/snowflake/objects route - new operations: unload_data, list_databases, list_schemas, list_tables, alter_warehouse, resume_task, suspend_task, list_query_history, list_copy_history * fix(snowflake): migrate renamed subblock IDs and authenticate before parsing - add SUBBLOCK_ID_MIGRATIONS entries so the renamed object fields map onto their pickers and the removed host/apiKey values are parked - authenticate the caller before contract validation in the selector route, per the API route convention * fix(snowflake): close unload-query breakouts, drop parked secrets, correct docs - assertBalancedQuery now skips // line comments, $$ dollar quoting and rejects ambiguous nested block comments; each hid a paren that let an injected OVERWRITE = TRUE escape the derived table - always emit OVERWRITE so an injected duplicate is rejected by Snowflake rather than silently replacing staged files - _removed_ migration targets now drop the stored value instead of parking it under a dead key, where export scrubbing (which walks the block config) would never clear it - 403 falls back to the shared invalid-credentials message, which names the network policy and SQL API causes Snowflake does not distinguish in the body - correct the network-policy-by-user-type claim: only SERVICE_AGENT is exempt - correct MAX_FILE_SIZE and errorOnly tool descriptions to match the fixed code * fix(snowflake): stop untouched switches emitting clauses; retarget migration - an untouched switch serializes as null, and advanced mode emits every advanced subblock, so alter_warehouse silently sent AUTO_RESUME = FALSE and permanently disabled auto-resume on the warehouse; normalize optional booleans to undefined in tools.config.params - point the subblock migration at the advanced text members: a migrated block has no credential, so a picker cannot hydrate a stored name, and legacy fileFormat values were qualified while the picker lists bare names - add the missing json-object wand type and scope the SQL wand prompt, which promised bindings that unload_data does not accept * fix(migrations): sweep already-parked subblock values; align picker 403 - an earlier version of this migration renamed retired fields into _removed_* keys instead of deleting them, so deployed workflows still hold those values; they match no oldId, so a dedicated sweep clears them for every block type - the picker now treats a Snowflake 403 like a 401: it means a network policy or a disabled SQL API, which the credential validator already reports as a credential problem rather than a bad request * fix(wand): add json-array generation type for array-contract fields The json-object reinforcement tells the model the response must start with { and end with }, which fights any field whose contract is an array. Snowflake's rows, matchColumns and procedureArguments all ask for arrays, so they were being steered toward an object that the JSON parse would then reject. Adds a sibling json-array type that strips fences the same way but reinforces brackets, and points the three array fields at it. bindings and filters are genuine objects and stay on json-object. * fix(snowflake): unload a table, not an inline query The COPY INTO grammar places the source immediately before its copy options, so an inlined query sits one parenthesis from being able to rewrite them. Guarding that means matching Snowflake's tokenizer exactly, and three successive versions of the guard were each defeated: // line comments, $$ dollar quoting, and a bare carriage return, which the scanner did not treat as a line terminator but Snowflake does. Each fix was a guess at a lexer the public docs do not specify. Removes the inline-query source instead of guessing a fourth time. A table name goes through qualifiedIdentifier, which is provably safe. Exporting a query result now means materializing it first — a view, or CREATE TABLE AS SELECT via Execute SQL — which the tool description, the block skill and the docs all say. Also from the final audit: - optionalBoolean accepts the string forms a direct tool call delivers, matching the other boolean readers on this block, and its TSDoc no longer states the serializer rule backwards - the five JSON editors declare language: 'json', so invalid JSON is caught inline instead of at execution - bound the RESULT_SCAN read in SQL, not only by rows_per_resultset - pin every migration target to a live subblock id, for all blocks	2026-08-09 00:20:09 -07:00
mzxchandraandWaleed Latif	87aeca6f0c	feat(zoho-desk): add Zoho Desk integration (#6157) ... * feat(zoho-desk): add Zoho Desk integration Add a full Zoho Desk integration: tools, block, icon, and a webhook trigger. Tools (tools/zoho_desk): list/get/update tickets, list/add comments, list/get threads, get contact, list organizations, and download attachments as UserFiles via an internal route. Registered in tools/registry.ts. Block (blocks/blocks/zoho-desk.ts): operation dropdown, OAuth credential, an organization selector backed by GET /organizations, per-operation fields, and BlockMeta templates. Wires the Zoho Desk trigger. OAuth (zoho-desk provider): authorize/token at accounts.zoho.com with access_type=offline + prompt=consent; the Desk REST base is derived from the token response api_domain and persisted so calls honor data residency instead of assuming desk.zoho.com. Every call sends Authorization: Zoho-oauthtoken and the orgId header. Trigger + webhook handler (triggers/zoho_desk, lib/webhooks/providers/zoho-desk.ts): Sim creates and tears down the Zoho Desk webhook subscription. Inbound events are verified with JWT RS256 (X-ZDesk-JWT) against the data-center JWKS, ACKed via the durable queue to meet Zoho's 5s deadline, and fail loudly on Free/Standard editions that cannot create webhooks. * fix(zoho-desk): OAuth PKCE, DC scope-marker parsing, SSRF, and e2e fixes OAuth: forward code_verifier in the custom getToken (PKCE is enabled, so the exchange must echo the verifier or Zoho rejects the request with invalid_request). Surface Zoho's error/error_description, which it returns in the JSON body with HTTP 200, instead of collapsing every failure into "no access token". Data-center base parsing: better-auth persists Zoho's scopes comma-joined with no spaces, so the greedy \S+ marker regex swallowed the whole scope list into the host. Stop the capture at a comma or whitespace in both read sites (token route and webhook handler), so apiDomain resolves to the real Desk host. Attachment SSRF: replace the permissive host regex (which accepted attacker domains like zoho.attacker.com) with a strict Zoho-apex suffix allowlist. Block: guard Number() pagination so a non-numeric typo can't send NaN; add the ignoreSourceId -> sourceId loop-guard header to update_ticket (matching add_comment). Organizations route: surface fetch/Zoho failures with a real status instead of a 200 with an empty list, so the org selector no longer fails silently. * fix(zoho-desk): webhook creation, attachment naming, and HTML content handling Webhook trigger (verified end-to-end against a live Enterprise org): - Omit ignoreSourceId; Zoho rejects a non-Zoho UUID with INVALID_DATA. Drop the generateId() fallback and its providerConfig persistence. - Answer Zoho's create-time notification-URL probe via the existing pending webhook verification mechanism (GET/HEAD matchers) so subscription creation no longer 405s. - mapZohoWebhookError now surfaces Zoho's real errorCode / message / field errors instead of a catch-all edition message, and attaches an HTTP status so 4xx flow through NonRetryableDeploymentError while 429/5xx stay retryable. - Propagate the real status through deploy.ts so failed creates don't retry-loop. get_attachment polish: - Return the downloaded file's name under `name` (ToolFileData key) instead of `filename`, and derive it (explicit -> Content-Disposition -> URL segment -> fallback) so attachments are no longer stored as "untitled". - Gate the add_comment-only `contentType` param so it isn't sent to get_attachment. HTML content handling (Zoho content fields emit raw HTML): - Add a Zoho-local html-to-text converter mirroring the Outlook dual-field pattern: when contentType is 'html', derive a plain-text `contentText` alongside the untouched raw `content` + `contentType`; plainText mirrors. - Apply to comments (list/add), threads (list/get), the ticket description (descriptionText), and the webhook trigger payload. Trigger org selector: Organization is now a credential-scoped combobox that lists the connected account's Zoho Desk organizations. * fix(zoho-desk): review round - DC-base derivation, org-loader resilience, batched-event visibility - deriveZohoDeskBaseFromApiDomain: preserve an already-regional desk.zoho.<tld> api_domain instead of falling back to the US (.com) data center, and map the DC TLD from any zoho(apis).<tld> host - keeps Desk calls in the right data center for residency. - fetchZohoDeskOrganizationOptions: wrap the token/org fetch in try/catch and degrade to an empty list (the org field is a free-text combobox, so manual entry still works) instead of hard-failing the selector on token/DC/network errors. - formatInput: warn (not silently drop) if Zoho ever delivers more than one event in a single payload. * fix(zoho-desk): harden attachment download against redirect-based SSRF/token leak Replace the raw fetch in the attachment route with secureFetchWithValidation (the same guarded fetch the copilot file-download tool uses). The download URL is user/LLM-influenced and Zoho may redirect, so auto-following redirects could send the OAuth token / orgId to an untrusted or internal host. The guarded fetch pins the resolved IP, blocks private/reserved targets on every hop, drops the Authorization header if a redirect leaves the origin (stripAuthOnRedirect), and enforces the 50MB cap while streaming. The strict Zoho apex allowlist still gates the initial origin as defense in depth. * fix(zoho-desk): only add the edition hint when Zoho's error indicates it mapZohoWebhookError appended the "requires Professional edition or higher" guidance to every 403, but a 403 can also mean a wrong org, a missing scope, or a bad token. Gate the hint on Zoho's own errorCode / message matching the permission/edition pattern instead of the bare status, so unrelated 403s surface Zoho's real reason without the misleading suffix. Adds a test for the non-edition 403 path. * fix(zoho-desk): stop duplicating /api/v1 when resolving a relative attachment href A relative attachment href that already starts with `api/v1` (as Zoho's hrefs often do) was concatenated onto getZohoDeskApiBase (which ends in /api/v1), producing `/api/v1/api/v1/...` and a failing download. Extract a tested resolveZohoAttachmentUrl helper that uses absolute hrefs as-is and strips a leading slash + `api/v1/` prefix from relative ones before joining, so the path is correct for absolute, root-relative, and api/v1-prefixed hrefs alike. * fix(zoho-desk): reject an empty update_ticket PATCH with a clear error update_ticket built its PATCH body from optional fields via filterUndefined, so a call with no fields set sent `{}` and surfaced an opaque Zoho failure. Guard the body builder to throw an actionable "provide at least one field" error before the request. Adds a test for the empty and populated body paths. * fix(zoho-desk): fall back to the credential Desk domain in webhook JWT verify verifyAuth chose the JWKS host from providerConfig.apiDomain and otherwise defaulted to the US host (desk.zoho.com), so a non-US webhook row missing apiDomain would verify against the wrong JWKS and reject legitimate events. When apiDomain is absent, resolve it from the OAuth credential's __zoho_domain__ scope marker (mirroring deleteSubscription). The persisted-apiDomain fast path stays DB-free to respect the 5s delivery deadline. Adds tests for both paths. * fix(zoho-desk): apply the Zoho host allowlist to the organizations route The organizations route built its URL from the client-supplied apiDomain and attached the OAuth token without the https-Zoho-host allowlist the attachment route already enforced, so a session-access caller could point the server at an arbitrary origin and leak the token. Extract the shared isZohoHost allowlist and an assertZohoUrl guard into tools/zoho_desk/utils (two consumers now), guard the organizations URL before fetching, and refactor the attachment route to reuse the shared helper. Adds tests for the allowlist and guard. * fix(zoho-desk): propagate provider 4xx in the stable webhook prepare path The v2 stable deploy preparation flattened every registration failure (except path conflicts) to HTTP 500, so a provider-attached permanent 4xx - e.g. Zoho's edition/validation failures from createSubscription - retried instead of failing the deploy terminally. Propagate the attached status (`?? 500`), matching the legacy save path's status-aware mapping so both deploy paths route 4xx through NonRetryableDeploymentError. * fix(zoho-desk): make createSubscription config failures non-retryable createSubscription threw plain Errors (no status) for missing orgId, event type, or credentials, and for a Zoho success with no webhook id - so the deploy outbox mapped them to 500 and retried permanent configuration failures. Attach a 4xx via statusError (400 for missing config/credentials; 422 for the no-id anomaly, where a retry risks duplicate webhooks) so they fail the deploy terminally like the mapped Zoho API 4xx responses. Tests assert the 400 status on the guard paths. * fix(zoho-desk): enrich prevState with contentText symmetrically with payload formatInput derived plain-text contentText only on payload, so an update event for a comment/thread left prevState as raw HTML while payload carried contentText - inconsistent shapes for before/after comparisons. Apply withDerivedContentText to prevState too. Test asserts both are enriched. * docs(zoho-desk): regenerate integration docs Regenerate zoho_desk.mdx from the current tool definitions: removes the stale add_comment `ignoreSourceId` input row (the field was dropped because Zoho rejects arbitrary values) and adds the derived `contentText` / `descriptionText` plain-text fields on comments, threads, and tickets. * fix(zoho-desk): validate the persisted Desk base against the strict host allowlist deriveZohoDeskBaseFromApiDomain trusted any host matching `desk.zoho.[a-z.]+`, so a crafted api_domain like `desk.zoho.com.attacker.com` passed and was persisted as the credential's `__zoho_domain__` REST base - later receiving the OAuth token on every Desk tool/webhook call. Gate the derivation on the strict isZohoHost apex allowlist (which rejects that lookalike), extracted with assertZohoUrl into a dependency-free host-allowlist module so the auth token-exchange path validates hosts without pulling in the tool utilities. The attachment and organizations routes now import the shared guard from there. Also: formatInput now emits the normalized null trigger shape for an empty/ malformed event array instead of leaking a raw `[]` to downstream steps. Tests cover the empty-array shape and the lookalike-host rejection. * fix(zoho-desk): correct API field names, scopes, and host validation Validation pass against Zoho's published Desk API surfaced six defects that typecheck, lint, and the existing suite all passed over, because each one fails silently against the live API rather than erroring. Wire-name mismatches (Zoho ignores unknown keys, so all three were silent): - update_ticket sent `customFields`; the ticket PATCH body names it `cf`. `customFields` exists only as a deprecated alias on other Desk resources and on the separate validate-field-updates endpoint, so updates reported success and applied nothing. - ZOHO_DESK_TICKET_PROPERTIES and ZOHO_DESK_CONTACT_PROPERTIES advertised a `customFields` output; both resources return `cf`. The declared field always resolved undefined and the real one was undeclared. - list_tickets sent `departmentId`; the query param is `departmentIds`, so the department filter was dropped and every department's tickets came back. Content handling: - deriveZohoContentText matched `contentType === 'html'`, but Zoho spells the discriminator per resource: comments use `html`, threads use the MIME form `text/html`. Every thread's `contentText` was therefore raw markup - the exact opposite of the field's purpose. Now normalized across both spellings, parameterized values, and casing, with regression tests. Scopes (least privilege): - Desk.tickets.ALL -> Desk.tickets.READ + Desk.tickets.UPDATE. No tool creates or deletes a ticket; ALL additionally granted ticket DELETE. - Dropped Desk.search.READ (no search tool exists) and Desk.webhooks.READ / .UPDATE (the provider only creates and deletes), plus their orphaned SCOPE_DESCRIPTIONS entries. Host validation - the webhook provider was the only token-carrying path not anchored to the Zoho apex allowlist, including the JWKS fetch, where an unrecognized host would have stood in as the JWT issuer: - createSubscription, deleteSubscription, and verifyAuth now route their base through a shared allowlist check. - getZohoDeskApiBase validates rather than trusting injection precedence. - The organizations route uses secureFetchWithValidation with stripAuthOnRedirect, matching the attachment route it had diverged from. Block and trigger: - The trigger's department field is renamed `triggerDepartmentIds`; sharing the `departmentIds` id let a value typed as a list_tickets filter become the webhook subscription's filter when switching modes. - `isPublic` no longer serializes onto all ten operations, matching the existing gating for `contentType`. - from/limit reject negatives and fractions instead of forwarding them. - update_ticket gains description, resolution, and classification (all already declared as outputs), and a departmentId input so a ticket can be moved. Accuracy corrections to user-facing text, all against the published parameter tables: `from` is 0-based (0-4999, default 0), not 1-based; per-endpoint limits are tickets 1-100/10, comments 1-100/50, threads 1-200/100; sortBy lists Zoho's actual allowed values; the two `include` sets genuinely differ per endpoint; status and priority accept comma-separated lists. Also: path IDs are trimmed via requireZohoDeskId so a pasted trailing space fails with a clear message instead of a %20 404; comment `commenter` and thread `status`/`isDescriptionThread`/`visibility`/`canReply` are now declared; ZOHO_CLIENT_ID/SECRET added to the oauth test env; docs page gains a MANUAL-CONTENT intro covering capabilities, the Professional-edition webhook requirement, and the US-data-center limitation. Not verified from documentation, needs a live account before merge: - the OAuth scope for the attachment content sub-path (Zoho publishes none, and there is an unanswered SCOPE_MISMATCH report against it) - 12 of the 17 offered webhook event ids (5 are confirmed); Ticket_Delete is documented but not offered - the ticket `descriptionContentType` key, and the POST /api/v1/webhooks body shape, neither of which appears in any reachable Zoho reference * chore(zoho-desk): regenerate tool metadata The param and description corrections in the previous commit changed the generated tool surface, so tool-metadata:check failed in CI. Regenerated; the diff is two Zoho-only lines. * fix(zoho-desk): stop posting null for untouched update_ticket fields `filterUndefined` strips only `undefined`, but an untouched subBlock never arrives as `undefined`: the workflow serializer initializes every subBlock value to `null` (stores/workflows/utils.ts) and extractBlockParams writes those nulls straight into tool params, with nothing between the serializer and request.body filtering them. Reproduced against the real serializer and block with only `status` set: basic {"subject":null,"status":"Closed"} advanced {"subject":null,"status":"Closed","priority":null,...,"cf":null} `subject` leaks even in basic mode because it declares no `mode`, so shouldSerializeSubBlock never drops it. Zoho documents subject as a writable field, so every status-only edit either failed the PATCH or blanked the ticket's subject; in advanced mode the whole update surface nulled out, including `cf`. Two things hid this. The empty-PATCH guard was unreachable from the block (the body always carried at least `subject`), and the existing test called buildBody with fields *absent* rather than null - the shape the block never produces - so it could not fail on the real path. Replaces filterUndefined with a local omitUnset that drops undefined, null, and '' (a cleared input means "leave unchanged", not "set to empty"). Adds three tests using the real serializer shape, all verified to fail before the fix. Also fixes the same null-blindness in the block's param mapping, where Number(null) === 0 injected from=0 on every operation, and corrects the shared limit placeholder, which claimed max 100 while list_threads allows 200. * feat(zoho-desk): add Self Client service-account credential Adds a second way to connect Zoho Desk, alongside the interactive OAuth flow: a Zoho Self Client, pasted as client id + client secret + organization id. Built on the existing client-credential-accounts framework rather than a new credential path, so it behaves like the Zoom Server-to-Server and Box CCG accounts already in the repo - a short-lived token minted on demand, no refresh token. Two Zoho behaviors the generic framework does not cover: - `scope` must be COMMA-separated on Zoho's token endpoint; a space-separated list is rejected as an invalid scope. The list comes from getCanonicalScopesForProvider('zoho-desk'), so the Self Client and the OAuth flow can never drift apart on scopes. - Zoho reports OAuth failures in the JSON body, frequently with HTTP 200 (e.g. {"error":"invalid_client"}), so the success body is inspected for an `error` field before the token is read - a status-only check would accept a failed mint. deriveZohoDeskBaseFromApiDomain moves out of auth.ts into the dependency-free host-allowlist module so the minter and the OAuth path share one derivation instead of duplicating it, and the mint response's api_domain now flows through to tools as `apiDomain` (the SA branch of the token route previously returned none, so SA calls would have assumed desk.zoho.com). Docs: hand-authored zoho-desk-service-account.mdx following the existing *-service-account.mdx pages, registered in meta.json and in the generator's keep-list so stale-page cleanup does not delete it. Known limitation, documented in the descriptor helpText and the docs page: webhook triggers still require an OAuth connection. Webhook provisioning resolves credentials through getCredentialOwner/refreshAccessTokenIfNeeded, which is OAuth-account-only for every provider in the repo - not a Zoho-specific gap. Unverified from documentation, needs a live Zoho org before merge: - the `ZohoDesk.` soid prefix. Zoho documents only the syntax {servicename}.{zsoid} with a single CRM example; no first-party doc states the Desk prefix. normalizeZohoDeskSoid passes through any value already containing a '.', so an operator can paste a corrected full soid without a code change. - whether zsoid is the same identifier as the Desk orgId header value. - whether the client-credentials endpoint accepts Desk.webhooks.CREATE/DELETE for a Self Client. - whether the mint response populates api_domain for Desk (documented for CRM); if absent the derivation falls back to the US Desk host. * fix(zoho-desk): derive descriptionText for ticket-shaped payloads Cursor Bugbot: webhook ticket events reached workflows as raw HTML with no plain-text sibling. `withDerivedContentText` only looked at `content` / `contentType`, but ticket resources carry their body on `description` / `descriptionContentType`, so trigger output disagreed with get_ticket. The helper now derives both, which also removed two inconsistencies on the tool side: get_ticket had its own inline copy of the derivation (now one shared implementation that cannot drift), and update_ticket returned its PATCH response raw despite the shared output map declaring descriptionText. `descriptionContentType` remains the one field name unconfirmed in any Zoho reference. It degrades safely - an absent key makes deriveZohoContentText return the value unchanged, so descriptionText mirrors description rather than breaking, exactly as get_ticket already behaved - and it is now one helper to correct if Zoho names it differently. * feat(zoho-desk): let the service account pick its data center Zoho's accounts server is per region, and the integration pinned every call to the US host. For the interactive OAuth flow that is currently unavoidable - better-auth's authorize/token URLs are static per provider - but the service account mints its own token, so the region can simply be chosen. This makes the Self Client the only way a non-US Zoho org can connect. Adds an optional `dataCenter` field to the client-credential framework. Optional matters: ClientCredentialAccountFieldId and ClientCredentialAccountFields are shared with Zoom, Box and Salesforce, whose descriptors and minters are unchanged. Blank keeps the previous behavior (US), so existing credentials are unaffected. Only us/eu/in/au are offered - the four regions where both the accounts server and the Desk REST host are confirmed. CA is deliberately absent: Zoho's accounts docs say accounts.zohocloud.ca while Zoho's own Desk SDK says accounts.zoho.ca, and the two cannot both be right. JP/SA/CN/UK lack a confirmed Desk host. The Desk base is now derived from the selected region rather than inferred from the mint response, which also removes a dependency on `api_domain` being populated for Desk (Zoho documents it for CRM only). When `api_domain` IS present and disagrees with the region, it wins - it is authoritative about where the token actually works - and the mismatch is logged so a mis-selected region is diagnosable. deriveZohoDeskBaseFromApiDomain gains a `try` variant returning undefined so an untrusted api_domain can no longer masquerade as an authoritative US answer and silently override a correct region. A wrong region fails loudly rather than silently: the minter runs as verification on both create and reconnect, so the credential is never persisted in a broken state. Because Zoho reports it as `invalid_client` - a Self Client only exists on its own region's accounts server - the operator hint for that code now names the data center as a candidate cause. Copy is scoped per path rather than blanket "US only": the OAuth service description, trigger setup instructions, and the docs intro now say which path each limitation applies to, and the service-account page documents the four regions with a sign-in-domain to region-code table. * fix(zoho-desk): strip ticket description HTML, classify body-reported refresh failures Final validation pass findings. descriptionText never stripped anything. It was gated on a `descriptionContentType` discriminator that Zoho does not send: the Ticket_Add webhook sample ships `"description": "<div>Description</div>"` with no such key, and the ticket GET/PATCH response field lists have no content-type sibling either. So get_ticket, update_ticket, and every webhook ticket payload emitted descriptionText as a byte-identical copy of the raw HTML, while the declared output promised stripped text. The tests did not catch it because they fabricated the shape - both fixtures constructed `descriptionContentType: 'html'`, a key Zoho never emits, proving the branch works without proving it is ever taken. Ticket descriptions are HTML by convention, so the strip is now unconditional (html-to-text is a near-identity on genuinely plain text), an explicit descriptionContentType is still honored if Zoho ever adds one, and the fixtures now use Zoho's real shape with no content-type key anywhere. A body-reported refresh failure was unclassified. Zoho answers a revoked refresh token with HTTP 200 and `{"error":"invalid_client"}`; refreshOAuthToken only checked `data.ok === false` (a Slack-ism), so the request fell through to the "no access token" guard and returned no errorCode. isTerminalRefreshError could therefore never recognize invalid_client as terminal, the credential was never marked dead, and every later execution retried a refresh that cannot succeed - with the user shown "No access token in refresh response" instead of a reconnect prompt. The body is now classified before the status is trusted, matching what the token exchange and the service-account mint already did. That guard also stopped logging the whole response body, which carries live tokens on a partial success. Also: an unrecognized dataCenter now fails with a named error instead of quietly resolving to US and surfacing as an opaque invalid_client (blank still means US); the webhook JWKS cache is bounded, since its key derives from a providerConfig field that SYSTEM_MANAGED_FIELDS protects from diffing but not from being written; and the attachment `size` output no longer asserts bytes, a unit Zoho documents as KB. * feat(zoho-desk): canonical selectors and BlockMeta skills The block picked its organization with an ad-hoc `combobox` + `fetchOptions`. Only five blocks in the repo did that, and the other four are core blocks (agent/credential/function/logs) - no other OAuth integration used it. Every other resource a user has to identify was a bare short-input taking an opaque numeric id. Zoho Desk now uses the same machinery as the other 25 selector providers: hooks/selectors/providers/zoho-desk/selectors.ts registered in the selector registry, consumed from the block as basic selector + advanced manual input sharing one canonicalParamId, for organization, update-ticket department, and the list-tickets department filter. The trigger's org field moves to the same selector. zoho-desk-org-options.ts is deleted rather than left beside the new path, so blocks/ has zero fetchOptions usages outside the core blocks. Wire params are unchanged (orgId, departmentId, departmentIds, assigneeId, ticketId, contactId) - this is a UI change, not an API change. The organizations route now resolves the credential server-side. It previously had the browser fetch an access token and POST it back, which an earlier audit flagged as the one place a Zoho token left the server; the new selector-credential resolver keeps it server-side for both the OAuth and service-account credential types and re-anchors every outbound host to the Zoho apex allowlist. No agents selector: the endpoint is documented but its OAuth scope is not, and the nearest evidence points at Desk.agents.READ, which we do not request. Adding it would force every existing Zoho Desk user to reconnect for a convenience field, so assigneeId stays a manual input until the scope can be confirmed against a live org. Adds the skills array BlockMeta was missing - 227 of 300 blocks declare one and this did not. Seven skills, each grounded in a use case Zoho or the ecosystem actually advertises (auto-triage, SLA escalation, digest, AI draft reply, customer context, engineering handoff, knowledge-gap report) and each exercising only tools in tools.access. CSAT surveys, ticket creation, dedup and keyword search were deliberately left out: the integration has no tool for them, and a skill implying an unsupported action is worse than a shorter list. * feat(zoho-desk): agents selector and free-text trigger organization Three improvements that were previously deferred only to avoid forcing existing users to reconnect or orphaning saved workflows. This integration is unmerged and has no users, so the constraint does not apply and the better option wins. assigneeId was the last field still asking for an opaque numeric id. It is now a canonical selector pair backed by a new zoho_desk.agents selector, which required adding the Desk.agents.READ scope - the reason it was skipped before. Route follows the departments one exactly: auth before parseRequest, host anchored to the Zoho apex allowlist, secureFetchWithValidation with stripAuthOnRedirect, and a page drain capped at 20 pages with 204 treated as end-of-list. Scope caveat: Zoho publishes no explicit scope line for the list-all GET /api/v1/agents. Every other endpoint in the Agents module documents Desk.agents.READ (get by id, get by email, roles/{id}/agents), and it is the only agents-module scope Zoho defines, so that is the basis. Inference across a module rather than a direct quote - worth one live call before merge, same as the existing attachment-scope note. The trigger regained free-text organization entry, lost when the org field became a selector. The earlier concern - that a manual value would land under its raw subBlock id and never reach the provider - turned out not to hold: buildProviderConfig already collapses canonical pairs and writes the active member under the canonical key. The real gap is narrower and does exist: when canonicalModes pins the group to basic while only the manual field has a value, the collapse deletes the canonical key even though the required-field check passes, so the deploy succeeds and then fails at subscription time. resolveConfigOrgId closes that, with a test. The block/trigger `orgId` id overlap stays shared, now with a comment. Two earlier audits disagreed; renaming turns out to be the wrong call. buildCanonicalIndex has an explicit guard for trigger-mode reuse and blocks.test.ts codifies it as a valid pattern, orgId means the same portal in both modes (unlike departmentIds, which is correctly distinct), and a separate triggerManualOrgId would put two advanced members in one canonical group - getCanonicalValues takes the first non-empty, so a stale tool-mode value could silently supply the trigger's organization. * fix(zoho-desk): make the attachment cap reachable, unbreak selector paging Final audit round. The 50 MB attachment ceiling could never be hit. This route returns the file as base64 inside its JSON body, and the executor reads internal tool responses through readToolResponseBody, capped at 10 MB. Base64 inflates 4/3, so ~7.5 MB of raw bytes is the real ceiling - and the old limit meant a larger attachment was downloaded, encoded and serialized in full (peaking near 250 MB of live allocation, with nothing bounding concurrent downloads) purely to be rejected afterwards. The cap is now the reachable size, so the limit enforces itself while the bytes are still streaming, and an overflow returns 413 with the actual ceiling instead of a generic 500. Raising it properly means uploading in the route and returning a file reference, as the WhatsApp media route does - not a bigger constant. Selector paging assumed a 0-based `from`. Zoho's docs contradict themselves: the pagination section says "range 0-4999, default 0" while the listing examples read as 1-based ("from=5 and limit=50 retrieves records 5 to 54"). Under the 1-based reading, stepping by exactly the page size re-fetches the boundary record and the dropdown shows a duplicate per page. Rather than pick a base that cannot be confirmed without a live tenant, the department and agent drains dedupe by id, which is correct under either reading. The organization list was unpaginated, and Zoho's listing APIs default to ten per page. An account with more accessible portals silently got a truncated dropdown, and since every other selector and every tool call is gated on orgId, a missing portal was unreachable except through the advanced manual field. Both the selector route and list_organizations now request the documented maximum. Docs: regenerated so the trigger table includes manualOrgId, and two service-account claims are hedged to match what the code already says it cannot verify - that zsoid equals the Desk orgId header value, and that every tool works under the requested scopes (Zoho publishes no scope for the attachment content sub-path). Also: status and priority move out of advanced mode - they are the fields most often changed on a ticket update; the custom-fields wand prompt now ends with the required "Return ONLY" clause; and the shared-orgId rationale comment cites the mechanism that actually applies (buildCanonicalIndex dedupe plus the first-non- empty rule in getCanonicalValues) rather than a blocks.test.ts branch that never evaluates this pair. * fix(zoho-desk): five-audit round - serializer trigger-advanced leak, scopes, paging Five independent audits (OAuth/scopes, tools-vs-docs, block/selectors, blast-radius, /validate-trigger). Findings, most severe first. A trigger-mode field was a live tool-mode required param. `shouldSerializeSubBlock` excluded `mode: 'trigger'` but not `'trigger-advanced'`, so the trigger's required `manualOrgId` validated on every tool operation. Reproduced against the real serializer: with the Organization field pinned to advanced, running List Organizations failed with "Missing required fields: Organization ID" - a field that operation does not even render, and which the user could not clear without switching operations. Fixed in the serializer rather than locally, because the Google Sheets/Drive/Calendar pollers have the identical shape. `limit=200` on /organizations was an undocumented parameter I added by extrapolating from /departments and /agents. Zoho documents NO parameters for that endpoint and its sample is a bare GET; the other siblings cap at 100 and Zoho answers out-of-range with 422. Since orgId gates every tool and both other selectors, a 422 there would have made the whole integration unreachable. Reverted to Zoho's documented shape. `descriptionText` was HTML-stripping plain text. The previous round made the strip unconditional after finding Zoho sends no `descriptionContentType`, but Zoho's REST samples show plain descriptions while only the webhook payload is HTML - and the webhook path runs this over contact/account/department bodies too. html-to-text is not identity on plain text: it decodes entities and deletes tag-shaped content ("a < b > c", XML snippets). Now sniffs for markup first. `omitUnset` made every documented field-clear impossible. Zoho's own PATCH sample uses `"classification": ""` and `"productId": ""` to clear. Dropping `''` meant no scalar field could be cleared. Now drops only undefined/null - the serializer-null case it was written for - and forwards `''`. status/priority leaked between operations. One shared subBlock served both the list_tickets filter and the update_ticket value, and subBlock values survive an operation switch, so a filter of "Open,On Hold" could be PATCHed onto a ticket and an update value could silently filter a later list. Split per operation. Auth: `invalid_code` added to TERMINAL_ERRORS - it is Zoho's code for a revoked refresh token, so without it the previous round's refresh fix never actually dead-flagged the credential it was written for. The shared refresh body-error branch now also requires `!data.access_token`, so no provider can have a successful refresh misclassified. The token route now uses the validating `extractZohoDeskBaseFromScope` instead of a private regex with no https/allowlist check - that value is injected into every tool call. Scope list falls back to the requested scopes when Zoho omits `scope`, which would otherwise flag every credential as needing reconnect. The Self Client mint no longer sends `aaaserver.profile.READ`, a scope that grant never uses. Trigger: `includePrevState` now set for every *_Update event, not just tickets - it defaults to false, so prevState was permanently null for contact/agent/task/ article updates while the trigger advertised it. `departmentIds` is only sent for events Zoho documents as accepting it, and the field is conditioned accordingly. Empty filters serialize as `null`, matching Zoho's examples, rather than `{}`. JWKS fetch bounded to 1.5s - jose's default is 5000ms, exactly Zoho's whole delivery deadline, and Zoho publishes no retry. The create-time validation POST fallback is now matched by the pending-verification probe. Ticket_Delete added. All 17 webhook event ids, the POST /api/v1/webhooks body contract, and the JWT claim/JWKS specifics are now confirmed verbatim against Zoho's webhook documentation - previously 12 of 17 events and the entire subscription contract were unverified. * revert(zoho-desk): back out both shared lib/oauth changes Reverting two changes to shared OAuth code because their premise is inferred rather than proven, and neither meets the bar for touching a path every provider runs. `refreshOAuthToken` body-error branch. The premise was that Zoho reports refresh failures with HTTP 200 and an `error` body. That is documented and empirically confirmed for the authorization-code EXCHANGE (see the comment on getToken in auth.ts), but I never confirmed it for the REFRESH grant specifically - and if Zoho returns a proper 4xx there, the existing `!response.ok` path already classifies it via extractErrorCode, making the branch dead code that every one of the ~34 providers still executes on each refresh. A shared branch whose only justification is an unverified inference about one provider is not worth its blast radius. `invalid_code` in TERMINAL_ERRORS. Same problem, worse downside: the code is sourced from a Zoho community post rather than official docs, TERMINAL_ERRORS is consulted for every provider, and a false positive marks a credential dead for an hour. Not adding it simply preserves today's behavior (retry rather than dead-flag), so reverting costs nothing that was previously working. Both are cheap to reinstate, correctly scoped, once a live Zoho account shows what a revoked refresh token actually returns. Kept: the token-redaction on the "no access token" warn, which is an unambiguous improvement independent of Zoho. Also kept, deliberately, is the serializer `trigger-advanced` exclusion - that one rests on a reproduced bug rather than an inference, and it aligns the serializer with the convention the rest of the codebase already follows (blocks.test.ts treats `trigger` and `trigger-advanced` identically in six places, as does the copilot block-metadata tool, and blocks/types.ts documents trigger-advanced as "the advanced side of a trigger field"). * fix(zoho-desk): carry the stored data center through a credential reconnect A reconnect rebuilds the service-account secret blob from the submitted fields only, and the connect modal never prefills - correctly, since for every other field in this family the stored value is a secret the admin must retype. The data center is the first non-secret member of that set, so it was being silently dropped: rotating a client secret on an EU/IN/AU credential moved it back to the US accounts server, where the next mint fails with an opaque invalid_client. performUpdateCredential now reads the stored dataCenter out of the existing blob when the caller does not supply one. The read is failure-tolerant - an undecryptable or unparseable blob yields undefined rather than throwing, so it can never block a reconnect, and the provider default applies as before. Raised independently by three reviewers; I twice argued it was acceptable because the mint fails loudly rather than corrupting silently. That was true and beside the point - the operator still had to guess why. * fix(zoho-desk): delta-audit findings - prevState scope, status leak, HTML sniffer An audit of the commits the earlier five audits never saw. All four findings are in code written as fixes for those audits, which is where this branch has repeatedly introduced new problems. `includePrevState` was sent for Ticket_Comment_Update. The previous commit gated it on an `_Update` suffix and claimed Zoho supports it on every update event. Zoho's webhook doc lists the attribute on Ticket/Contact/Agent/Task/Article update events but NOT on Ticket_Comment_Update, which documents only `departmentIds`. That made it an undocumented filter key on a live subscription create - the same class of risk the same commit reverted `limit=200` for, so it failed that commit's own stated bar. Now an explicit set rather than a suffix rule. The status/priority split did not stop the leak it was written for. The mapping used `operation === 'list_tickets' ? filterValue : updateValue`, whose bare else covers all eight other operations - so a stale Update Ticket status was forwarded into get_ticket, list_comments and the rest. Harmless on the wire (those tools ignore it) but exactly the stale-value pattern the neighbouring gates exist to prevent. Both fields are now scoped to the two operations that declare them. The HTML sniffer destroyed plain text. `/<[a-z!\/][^>]*>/` fires on any `<` followed by a letter with a later `>`, so realistic ticket bodies lost content: "if x<y then z>0" became "if x0", and "replace <username> with the real name" lost the placeholder. It now requires a real element - a paired tag, a self-closing tag, a comment/doctype - or an entity, and the entity arm covers hex references it previously missed. Regression tests verified by reverting to the loose pattern and watching them go red. The reconnect data-center carry-forward is scoped to client-credential providers. As written it added a DB read plus a decrypt to every service-account reconnect for every provider - Slack, Atlassian, all token-paste providers - to carry a field only Zoho has. Also: the JWKS cache-bound TSDoc had been orphaned onto the wrong constant by an earlier insertion, and `cooldownDuration` was dropped since it restated jose's default while only `timeoutDuration` needed justifying. * test(zoho-desk): cover the webhook subscription filter rules The subscription filter logic had no test coverage at all, and it is where the last two rounds both found bugs - includePrevState on an event Zoho does not document it for, and departmentIds sent to events that accept no filters. Adds six cases against the real createSubscription: includePrevState is set for each of the five documented update events and NOT for Ticket_Comment_Update, departmentIds is kept for a filterable event and dropped for one that is not, and an event with no filters serializes as null rather than an empty object. Verified the guard bites: reverting PREV_STATE_EVENTS to the `endsWith('_Update')` rule turns the Ticket_Comment_Update case red. The Ticket_Comment_Update assertion checks the with-departments case as well as the bare one - asserting only `not.toHaveProperty` on the bare filter would pass vacuously, since that filter is legitimately null. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-01 21:55:00 -07:00
3 people	13772565a3	fix(chat): show deployment passwords to admins (#6177) ... * fix(chat): show deployment passwords to admins * fix(chat): reject whitespace-only passwords * fix(chat): preserve password visibility on regenerate * fix(chat): harden password reveal and close deployment lockout paths Follow-ups from a security review of the password reveal endpoint. The permission model itself was correct — the reveal is gated on workspace admin via the canonical resolver, so derived org-admin access is honored. These address secret handling and validation around it. - Cap set-path passwords at the same 1024 chars the chat login accepts. Neither the input nor the schema bounded length, so a longer password saved fine and then failed the login POST on length before auth ran, locking every visitor out permanently. - Discard the revealed password when the field is hidden. It previously stayed in state and in the input's DOM value with Copy still armed, so the field read as hidden while still handing out the plaintext. - Evict the decrypted password from the mutation cache on unmount, and correct the TSDoc claiming it was never retained — it sat in the MutationCache for the default five minutes after the modal closed. - Validate the password inside performChatDeploy, the writer both callers must use. The copilot deploy_chat tool bypasses the route contract and could still store a whitespace-only or over-long password, or create a password-protected chat with no password at all. - Stop echoing raw decryption errors from the reveal endpoint. - Only persist a new password when the chat ends up password-protected; PATCH { authType: 'email', password } used to re-arm the secret that the auth-type branch had just cleared. Also replaces the hand-rolled copy state with useCopyToClipboard, which fixes an unawaited clipboard write that surfaced as an unhandled rejection and a "Copied" confirmation shown even when the write failed. Co-Authored-By: Claude <noreply@anthropic.com> * fix(chat): correct password-change confirmation gate and stale reveal error Addresses both open Bugbot findings. - shouldConfirmPasswordChange keyed on "a chat exists" rather than "a password exists", so switching a public chat to password protection for the first time asked the admin to confirm changing a password that was never set. It now takes the existing-password signal the component already computes. - A failed reveal left "Failed to load the current password" on screen while the admin typed or generated a replacement, because the mutation only drops its error on the next attempt. Editing or regenerating now resets it. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude <noreply@anthropic.com>	2026-08-01 20:06:54 -07:00
Waleedandmzxchandra	10bfb5d139	feat(realtime): shared room spine + live Files/Tables collaboration + Yjs document editing (#5991) ... * feat(realtime): add shared room identity + authorization spine (#5929) Introduces the foundation for a unified realtime "room" model spanning the Socket.IO presence server (apps/realtime), the durable SSE event log, and the ephemeral pub/sub fanout — all of which today reinvent their own room identity, naming, and authorization. - @sim/realtime-protocol/rooms: RoomRef { type, id }, ROOM_TYPES, and a roomName/parseRoomName codec. WORKFLOW deliberately maps to the bare id so the ~40 existing io.to(workflowId) callsites and presence state keys are unchanged; every other room type is namespaced so id spaces cannot collide. - @sim/platform-authz/rooms: authorizeRoom(userId, room, action) generalizing the exemplary authorizeWorkflowByWorkspacePermission — one resource->workspace resolver per room type, then the shared resolveEffectiveWorkspacePermission + permissionSatisfies gate. Pure foundation, no behavior change: nothing consumes these yet. Prune graph stays at 14/25 (platform-authz already depended transitively on realtime-protocol via apps/realtime). * refactor(realtime): generalize presence server to multi-room [2/N] (#5930) * refactor(realtime): generalize presence server to multi-room (RoomRef) Generalizes the Socket.IO presence layer from single-workflow-room-per-socket to a domain-neutral, multi-room-per-socket model keyed by RoomRef, so a second domain (workspace files, next PR) can reuse the same membership + presence engine. Behavior-preserving for workflow collaboration. IRoomManager is now domain-neutral (addUserToRoom/removeUserFromRoom/ getRoomForSocket/getRoomUsers/updateUserActivity/... all take a RoomRef). The workflow lifecycle broadcasts (deletion/revert/update/deploy) move out of the manager into WorkflowRoomService, composed over the generic manager. Backward-compat by design (no workflow migration, no regression): - Workflow Socket.IO room name stays the bare workflowId (roomName() maps workflow -> bare id), so the ~40 io.to(workflowId) callsites are untouched. - Workflow Redis presence keys stay workflow:{id}:users/:meta (the type prefix IS "workflow"). Multi-room correctness (from adversarial audit): - socket:{id}:workflow single-value key -> socket:{id}:rooms HASH (type->id). - The SHARED socket:{id}:session key is deleted only when the socket leaves its LAST room (refcount via HLEN) — a leave from one room no longer breaks the other room's handlers. - disconnect enumerates the socket's stored rooms and rebroadcasts presence per room, instead of picking an arbitrary socket.rooms entry. - presence broadcasts use a per-room-type event name (workflow keeps the bare presence-update; others are namespaced). Workflow handlers wrap manager calls with a shared workflowRoom(id) helper; UserPresence.workflowId -> room (the client never reads that field). Tests: existing 112 realtime tests pass unchanged (behavior gate) + 7 new multi-room tests (refcounted session, presence isolation, multi-room disconnect, per-type event names). tsc clean, boundaries + prune (14/25) green. * fix(realtime): harden multi-room disconnect + id-guard room removal Two fixes from an adversarial regression audit of the multi-room refactor: - Disconnect now handles `disconnecting` (where `socket.rooms` is still populated and authoritative) and falls back to the live Socket.IO room set for any room the manager's stored state no longer tracked. This restores reliable presence cleanup + departure broadcast even if the Redis `socket:{id}:rooms` key was evicted or TTL-expired — the one behavioral gap vs the pre-refactor disconnect. - REMOVE_ROOM_SCRIPT now only drops the socket's room mapping (and runs the last-room session cleanup) when the stored id matches the room being removed, matching the memory manager's existing id guard. Prevents a mismatched-room call from wiping a different room's mapping or the shared session. +1 test (id-guarded no-op removal). 120 realtime tests pass, tsc clean. * fix(realtime): only rebroadcast disconnect-fallback rooms whose removal succeeded Greptile 4/5 follow-up: the disconnecting-time fallback ignored removeUserFromRoom's boolean and rebroadcast presence even when the removal reported false. Now it only treats a room as removed (and rebroadcasts) when the manager confirms it — symmetric with removeSocketFromAllRooms, which already only returns rooms it actually removed. * fix(realtime): exclude the disconnecting socket from its farewell broadcast Greptile follow-up (transient-Redis-failure edge): if removeUserFromRoom fails on disconnect, the socket's presence entry can outlive it (room hashes have no TTL) and reappear as a ghost. Disconnect now broadcasts a correction to EVERY room the socket was in (union of the manager's removed rooms and the live Socket.IO membership) and passes the disconnecting socket id as excludeSocketId, so it is never shown as a collaborator regardless of whether the Redis delete succeeded. Any orphaned entry is still reclaimed by the next join's stale-presence sweep. broadcastPresenceUpdate gains an optional excludeSocketId; normal broadcasts are unchanged. +1 test. * fix(realtime): make presence broadcasts liveness-aware (root-cause ghost fix) Presence broadcasts now reconcile the stored list against the live Socket.IO membership (io.in(room).fetchSockets()) before emitting, via a shared filterVisiblePresence helper. This closes the residual behind the earlier disconnect fixes: an entry orphaned by a failed removal (room hashes have no TTL) could reappear in a LATER join's presence snapshot until the 75-min stale sweep. Now such an entry is never emitted, because a non-live socket is filtered out of every broadcast. Combined with excludeSocketId (which handles the disconnecting socket, still momentarily live). Fail-safe: on a fetchSockets throw or an empty result while entries remain, emit the unfiltered list rather than hide live collaborators. Also drops a dead guard in the disconnect union loop (rooms already removed are skipped by the wasInRooms check) and the now-unused isSameRoom import. +1 ghost-guard test. 122 realtime tests pass. * feat(files): live presence avatars + live file tree via realtime rooms (#5932) * refactor(tables): adopt shared durable event-log core (#5934) * fix(realtime): address post-merge review-comment findings (#5937) * fix(realtime): address post-merge review-comment findings A re-audit of every inline review comment on the merged stack surfaced real issues that the thread-resolutions and prior audits missed. Fixes: Presence server (#5930 comments): - connection.ts: snapshot `socket.rooms` SYNCHRONOUSLY before the first await. Socket.IO clears the room set once the synchronous part of a `disconnecting` handler returns, so reading it after `await removeSocketFromAllRooms` saw an empty set — the eviction fallback was dead. (Cursor: "Disconnect fallback misses live rooms".) - workflow-room-service: restore the original managers' final unconditional room-state wipe via a new `deleteRoom(room)` manager method, so a deleted workflow leaves no lingering presence/meta even if a per-socket removal failed or a socket joined mid-teardown. (Cursor: "Deletion skips final room wipe".) Files (#5932 comments): - workspace-file-manager.uploadWorkspaceFile now fans out the live-tree signal (all direct-upload paths: multipart fallback, copilot create, /api/files/upload, v1 files — the presigned path already notified). (Cursor: "Creates miss live tree fan-out".) - use-workspace-files-room: clear the pending retry timer on join success; and a module-scoped intended-room guard defers the unmount `leave` so a rapid remount re-claims the room and skips a stale leave — fixing presence flap + a leave-after-join race. (Cursor: "Retry timer survives join success" + "Remount churns files presence".) - workspace-files handler: roll back a partial join (leave room + remove presence) in the catch, mirroring the workflow join. (Cursor: "Join failure skips membership rollback".) +2 tests (deleteRoom). 127 realtime tests pass, both apps tsc clean, api-validation + boundaries green. * fix(files): scope workspace-files leave to a workspace (deferred-leave safety) Self-review of the deferred-leave guard found a real bug: leave-workspace-files was not workspace-scoped, so after a workspace switch (A->B) the deferred leave from A would evict the socket from its new room B. The leave now carries the workspaceId and the server no-ops if the socket's current files room differs. Also excludes the leaving socket from the leave broadcast (consistent with disconnect). * fix(realtime): close files-room presence leak + validate join payload Architecture-audit findings: - S1 (real Redis leak): the files room inherited the shared manager but not the workflow join's liveness sweep, so an UNGRACEFUL disconnect (pod crash — no `disconnecting` event) left its presence entry in the no-TTL room hash forever. Added a shared `sweepStalePresence(manager, room)` (fetchSockets liveness + remove not-live-AND-stale entries, matching the workflow 75min threshold) and run it on files join; also filter the join ack through `filterVisiblePresence` so a joiner never briefly sees an un-swept ghost. - S2: validate the client-supplied `workspaceId` on files join before it reaches the DB query (matches the /api/workspace-files-changed guard; fails closed). - N2: corrected the notify doc — it is awaited (guaranteed dispatch before a Node route returns) and hard-bounded to NOTIFY_TIMEOUT_MS, not "never block". +1 test (sweepStalePresence keeps live/fresh, reclaims not-live-stale). 128 realtime tests pass, both apps tsc clean, biome clean. * fix(realtime): workflow-deletion always notifies + cleans by socket.io membership Review-round findings on #5937: - Always emit `workflow-deleted` (was guarded by users.length>0), so a socket still in the Socket.IO room after a Redis presence eviction is told the workflow is gone before socketsLeave kicks it — the editor no longer keeps showing a deleted workflow. (Cursor: "Silent kick skips deletion event".) - Clean per-socket state for the UNION of live Socket.IO members and presence-tracked sockets, so an evicted/late-joined socket's room mapping + session are dropped too — not just presence-snapshot sockets. (Greptile: "Room deletion leaves reverse state".) - deleteRoom now logs AND rethrows on Redis failure (like addUserToRoom) so a failed wipe isn't reported as a clean deletion; the request surfaces it. (Greptile: "Room deletion failures are suppressed".) The two "deferred leave drops new membership" P1s were already fixed by the workspace-scoped leave in a prior commit (leave carries { workspaceId }; server no-ops on mismatch). 128 tests pass, tsc + biome clean. * refactor(files): drop module-scoped deferred-leave; rely on workspace-scoped leave Removes the one non-idiomatic construct (a module-level mutable `intendedFilesWorkspaceId` + queueMicrotask). It only guarded a same-workspace CONCURRENT remount, which doesn't occur in production (folder nav is shallow/no remount; list<->detail is sequential) — a dev-StrictMode-only case. The real cross-workspace race is already handled by the workspace-scoped leave: if B's join runs first (auto-leaving A), A's leave no-ops because the socket's current files room is B. Simpler, idiomatic, prod-correct. * feat(realtime): Yjs relay server for collaborative document editing [4/N] (#5941) Server-side Yjs relay for collaborative document editing (live carets + text selection) in the Files rich-markdown editor. Faithful y-websocket-style relay over the existing authenticated Socket.IO connection + shared room abstraction; in-memory Y.Doc + Awareness per file; awareness ownership binding, userId-keyed client-id uniqueness, seeder election with deadline re-election, concurrent-JOIN generation guard. 25 relay tests. Reviewed to Greptile 5/5 + Cursor pass across multiple rounds, plus an independent 4-lens audit (correctness/security/conventions/simplicity) and /simplify + /cleanup passes. * feat(files): collaborative document editing — client provider + editor (#5946) Client Yjs provider (FileDocProvider over the authenticated socket) + TipTap Collaboration/CollaborationCaret wiring for live carets + text-selection in the Files rich-markdown editor. Collaboration is a Files-page-only surface (explicit `collaborative` opt-in), disjoint from agent-streaming. Read-only + autosave-gated until synced+seeded. Merges into the realtime-rooms integration branch. * feat(tables): live collaboration — cell-selection presence + live mutation propagation (#5957) * feat(tables): live cell-selection presence — protocol + server + client hook The realtime spine for Google-Sheets-style table presence (mode A, socket): - @sim/realtime-protocol/table-presence: centralized wire protocol (events + TableCellSelection {anchor, focus, editing} + payloads) so server emits and client subscriptions can't drift. - ROOM_TYPES.TABLE + resolveTableWorkspace registered in ROOM_WORKSPACE_RESOLVERS (tableId -> workspace via userTableDefinitions, honoring archivedAt); roomName / presenceEventName / disconnect cleanup / authorizeRoom all derive automatically. - apps/realtime/src/handlers/tables.ts: join/leave (mirrors workspace-files) + a table-cell-selection relay (mirrors the workflow selection channel), broadcasting via roomName(room) since table rooms are namespaced. UserPresence gains a cell field threaded through the memory + Redis managers (Lua ARGV[7], null clears). - Extracted the duplicated resolveAvatarUrl into handlers/avatar.ts. - use-table-room.ts client hook: joins over the shared socket, tracks the roster (avatars) + patches per-socket cell deltas, exposes a throttled emitCellSelection. Grid UI (avatars + selection overlay) lands next; concurrent cell-value edits (last-write-wins via the durable log) are the follow-up PR. * feat(tables): render live cell-selection presence in the grid Wires the table presence room into the grid UI: - Page (table.tsx): useTableRoom (gated off in embedded/mothership mode) — renders <PresenceAvatars> in the header and passes remoteSelections + emitCellSelection down to the grid. - Grid emits its local selection: an effect resolves the index-based anchor/focus to stable (rowId, columnId) via refs and broadcasts it (with an editing flag for the active cell) through the throttled emitter. - RemoteSelectionOverlay: draws each remote viewer's selection in their color (getUserColor), a darker fill while editing, and name-on-hover — measured from live cell rects in the content wrapper's space (scrolls with the grid), hidden when rows are virtualized off-window, pointer-events-none so it never blocks cell clicks (hover via pointer hit-test). * test(tables): cover the table presence handler Mirrors workspace-files.test.ts: join auth/unavailable/denied/success, plus the cell-selection relay (asserts it persists via updateUserActivity and broadcasts on the namespaced roomName, not the bare id) and leave. * feat(tables): propagate manual cell edits live (last-write-wins) A manual row edit now appends a lightweight 'edit' event to the durable table stream; collaborators refetch the row (via the existing debounced rows-invalidate the job events use) so the winning value shows live. The event carries no value — peers refetch in their own wire format, so there's no auth-specific value translation on the wire, and last-write-wins falls out of the DB's committed order (the Google-Sheets model). Edits that also trigger a dispatch already emit dispatch/cell events; the debounce coalesces the two. * refactor(tables): apply /simplify findings - Drop the dead 'add unknown peer' upsert branch in use-table-room (Socket.IO ordering guarantees a peer is in the roster before their selection delta). - TableCellSelectionBroadcast = TablePresenceUser & { cell } (was a copy-paste). - Make TableGrid's presence props required + drop the unused empty-default/guard (only table.tsx mounts it, always passing both). - Drop the unused rowId from the 'edit' event (the handler invalidates all rows). - Overlay: subscribe scroll/resize/pointer listeners once per scroll element and cache the wrapper origin, so incoming deltas re-measure without re-subscribing and the pointer hit-test never forces a per-move layout read. - Server: cache the immutable socket session so a selection delta no longer reads it from Redis every time. * refactor(tables): apply /cleanup findings - Fix the remote-selection name label contrast: text-white is unreadable on the light-pastel user colors (same bug the Files caret fixed) → fixed dark #1a1a1a. - Re-measure via useLayoutEffect so a moving peer selection updates before paint (no one-frame position lag). - Drop 'mothership' from a comment (constitution copy rule). Six cleanup passes ran (effect, memo/callback, state, react-query, emcn, comment); the rest confirmed clean — all state/memos/callbacks/effects are load-bearing, presence correctly lives in useState (socket-pushed), and the edit→rows-invalidate granularity is right. * feat(tables): propagate every table mutation live (edit + schema signals) Comprehensive live collaboration for all user table mutations, via two value-less durable signals + named helpers (signalTableRowsChanged / signalTableSchemaChanged): - edit (rows refetch): single + batch row create, cell/row update, batch update, delete by id/filter, and upsert. - schema (definition + rows refetch): column add/update/delete, workflow-group add/update/delete, table rename, and CSV import (which can add columns). - Client handles 'schema' by invalidating the table detail (exact) + rows. Execution paths (column run, cancel-runs) and async jobs (delete/import-async, job-cancel) already propagate via cell/dispatch/job events — verified applyJob refetches on terminal. No reorder routes exist. Table archive (route DELETE) is a deliberate follow-up: it needs a table-deleted redirect event, not a refetch signal (which would 404). * refactor(tables): apply comprehensive /cleanup audit findings Holistic + react-query + comment audits over the whole PR: - Security/crash fix: a remote peer's rowId flowed unescaped into the overlay's querySelector — a hostile id ('x"]') threw SyntaxError inside a useLayoutEffect, crashing every other viewer's page. CSS.escape it, and validate + whitelist the untrusted cell payload server-side (shape + 200-char id bound) before it is stored/rebroadcast. - Simplify the CELL_SELECTION relay: the delta attached userId/userName/avatarUrl that the client discarded (identity comes from the roster). Drop them + the getUserSession lookup/cache entirely — the delta is now { socketId, cell }. - React Query: schema handler also invalidates lists() (parity with the local column-mutation set); document that the mutating client self-refetches by design. - Comment tightenings; biome fixed a stale import order in workspace-files.ts. * fix(tables): broadcast single-cell selections (focus falls back to anchor) Cursor High: a normal cell click leaves selectionFocus null (the grid treats it as a one-cell selection via focus ?? anchor), but the presence emit required BOTH anchor and focus to resolve — so the most common selection never broadcast and clicking even cleared a prior remote outline. Mirror the grid's focus ?? anchor semantics. * fix(tables): reviewer + regression + per-LOC audit findings Cursor review round (5 findings) + regression audit + per-LOC audit: - Presence roster snapshot now KEEPS the cell we already hold for a known socket, so a join/leave broadcast can't revert a fresher CELL_SELECTION delta. - Reset the selection throttle on table switch (was unmount-only), so a pending selection for table A can't flush into table B's room after a switch. - Metadata writes (column widths, display) use a new lightweight 'metadata' signal that refetches only the definition — a resize no longer forces peers to refetch rows. - Overlay re-measures on row add/remove/reorder via a tbody childList MutationObserver (a live refetch moves cells without a scroll/resize). - Document the actor self-refetch create caveat (scrolled multi-page insert) accurately. - isCellRef narrows to a partial instead of casting to the full type then re-checking; drop a redundant mount measure() (the layout effect covers it); text-[11px]→text-xs. * fix(tables): drop ineffective metadata propagation + re-measure overlay on column resize Cursor round on b8f28b04b: - Remove the 'metadata' signal entirely. The grid seeds columnWidths/pinnedColumns from metadata ONCE (metadataSeededRef) and deliberately never re-applies them (to avoid clobbering a local in-progress resize), so refetching the definition on a peer never surfaced their width/pin change — an ineffective path. Width/pin live-sync needs reconciliation that doesn't clobber a local resize; that's a deliberate follow-up, not a no-op refetch. Structural changes still propagate via 'schema'. - Overlay now also observes the content layer with the ResizeObserver, so a column resize (which grows the content, not the scroll container) re-measures remote outlines. - Presence-merge comment now states both sides of the trade-off. * fix(tables): re-broadcast local selection on (re)join Cursor Medium: a selection made before the room join completes (or held across a reconnect) was dropped server-side and never re-sent, so peers didn't see it until the local user moved it again. Track the current selection in a ref (set on every emit, cleared on table switch) and re-emit it from handleJoinSuccess once the room is joined. * fix(tables): re-broadcast selection when a peer's row change shifts it End-to-end lifecycle audit (Low-Med): the selection emit resolved the stable (rowId, columnId) only on selection/editing change, not when a live edit/schema refetch inserted/deleted/reordered rows. The index-based local selection then sat on a different logical row than the rowId peers held, so your outline showed on the old row until you moved. Re-run the emit on rows/displayColumns change and dedup an unchanged result (also drops the redundant null-on-open emit) so the broadcast stays consistent with the local highlight. * fix(tables): schema invalidates run-state/enrichment + guard stale join Cursor round on cdc8796b8 (2 Medium): - schema handler used detail exact:true, so it skipped the activeDispatches + enrichmentDetails sibling queries the local invalidateTableSchema refreshes via a prefix match. After a peer deletes/restructures a workflow group, peers could keep a stale running badge or enrichment panel. Now invalidates both siblings too (rows stay on the debounce). - Guard against a stale join stealing the room: a fast table A->B switch could let A's async authorize finish after B, leave B, and strand the socket in A. Added a per-socket monotonic join generation checked after authorize (mirrors the file-doc relay's guard) + a test. * feat(tables): live column width/pin/order sync Collaborators now see each other's column resizes, pins, and reorders live — the last piece of Google-Sheets-style layout parity. - New lightweight `metadata` durable event kind (distinct from `schema`): only the table definition carries UI metadata, so peers refetch the definition alone — no rows/run-state refetch. The metadata PUT route now signals it. - The grid reconciles server metadata against its in-progress gesture: the column being actively resized keeps its live local width, and an in-flight column drag blocks a reorder apply — so a peer's change never reverts the local action. Each field is reference-guarded (React Query structural sharing keeps unchanged sub-objects stable), so an unrelated peer change doesn't re-apply the others. * fix(tables): escalate to schema signal when a reorder scrubs group deps Independent audit of the metadata-sync commit found a stale-run-state hole: a columnOrder PUT that moves a column left of a workflow group's leftmost column makes updateTableMetadata scrub that group's dependencies and write a new schema — a real structural change. But the route only fired the lightweight 'metadata' signal (detail-only refetch), so peers' and the actor's activeDispatches / enrichmentDetails queries stayed stale (a lingering running badge / enrichment panel) — exactly what the 'schema' handler exists to prevent. updateTableMetadata now reports whether it scrubbed the schema; the route emits signalTableSchemaChanged in that case and the light signalTableMetadataChanged otherwise. Width/pin/plain-reorder stay on the cheap detail-only path. * feat(realtime): accurate in-file presence + collaborative-caret polish (#5965) Per-session file-doc presence (avatars count other sessions like the canvas), StrictMode-safe stable Y.Doc (fixes blank-doc on join), flush caret cap + restored hover hit-slop, and three join-lifecycle race fixes unifying file-doc + workspace-files on one intent-tracked monotonic generation model. All findings root-caused with regression tests. * feat(files): smarter bullet delete/indent and untitled-file title sync (#5971) * improvement(files): smarter bullet delete/indent, fix empty-nested-bullet heading corruption Backspace at the start of a list item now outdents a nested item or clears a top-level item to a paragraph in place instead of deleting the row and jumping the caret to the previous block; Enter on an empty nested item outdents. Empty non-trailing top-level items still collapse cleanly since they cannot round-trip as a lifted paragraph. Also strips nested empty list-item marker lines on serialize: a nested empty bullet re-parsed as a Setext heading underline, silently turning its parent line into an H2 and dropping the bullet. Top-level empty items are preserved. * feat(files): sync an untitled file's name with its leading heading While a file is still named untitled(.md), typing a leading heading auto-renames the file after it (debounced), and renaming the file first seeds a leading H1 from the new name. One-shot: coupling stops once the file has a real name, and the heading seed always prepends so existing content is never clobbered. * fix(files): count inline atoms in list-item emptiness, keep multi-block items on Backspace Addresses review findings on the list Backspace logic: - Emptiness now uses the caret block's content.size (counts inline images/mentions), not textContent, so a bullet holding only a non-text atom is no longer treated as empty and deleted. - An empty first block whose item has sibling blocks removes only that block instead of lifting the whole item out of the list. * fix(files): preserve the untitled to named heading seed across a rename during editor load The parent captures the file name at mount (before content/session finish loading) and passes it as the transition baseline, so a rename that lands in the loading window is still seen as an untitled to named transition and the leading heading seed is not skipped. * fix(files): drop the name-to-heading seed, keep title sync one-way Removes the effect that inserted a leading H1 when an untitled file was renamed. On the collaborative Files page every open client observed the untitled-to-named transition and inserted into the shared doc, producing duplicate headings; it could also re-insert a heading a user had just deleted while a rename was in flight. Seeding document content from an async rename transition is the wrong model on a shared editor. The primary direction — typing a leading heading renames a still-untitled file — is unaffected (it never mutates the doc). * fix(files): keep empty lines between paragraphs on reload The chunked markdown parser (parseMarkdownToDoc) parses each block stripped of the blank lines between them, so it dropped the empty paragraphs @tiptap/markdown builds from runs of blank lines — a saved visual blank line silently vanished on the next load (the settle/reopen re-seed goes through the chunker). The whole-document parser preserves them, but whether a gap yields an empty paragraph is a global, block-type- dependent decision (kept between two paragraphs, dropped after a heading), so it can't be reconstructed block-locally. Route documents with empty-paragraph blank-line spacing to the whole-document parser for exact fidelity — the same tradeoff NON_CHUNKABLE makes; ordinary single-blank-line separation still takes the fast chunked path. Adds a suite asserting chunked output matches the whole-document parser for leading/trailing/between gaps and around lists/headings. * fix(files): only auto-name an untitled file when the user can edit The debounced untitled→filename hook ran on every onUpdate — including the mount-time seed and for view-only viewers — without checking edit permission, so a read-only user could schedule a rename they have no permission to make (a spurious, server-rejected write). Gate the derive-title on editor.isEditable (canEdit + settled + collab-ready, the same signal the autosave path uses), at both schedule and fire time. * fix(files): normalize line endings before the empty-paragraph guard; Enter/Backspace symmetry - markdown-parse: EMPTY_PARAGRAPH_SPACING/NON_CHUNKABLE tested the raw body, but a classic \r-only file (blank lines are \r) would miss the \n-anchored guard and still be chunked, dropping empties. Normalize line endings once up front so the routing guards, the chunker, and the parser all see the same \n. +CRLF/CR test cases. - keymap: Enter on an empty first block of a multi-block item now removes only that block (removeEmptyWrappedBlock) instead of exiting the list, mirroring the Backspace hasSiblingBlocks case — the trailing check no longer swallows multi-block items. +test. * fix(files): editor audit follow-ups (trailing-blank read-only, collab rename, over-strip) A 4-agent independent audit (UX vs inkeep + SOTA, cleanliness, adversarial correctness) surfaced these: - HIGH regression: files ending in a blank line opened READ-ONLY. The empty-paragraph routing preserved a TRAILING empty paragraph, but postProcess collapses trailing newlines → serialize/parse non-idempotent → isRoundTripSafe flipped the file read-only. A trailing empty paragraph can't be serialized stably, so parseMarkdownToDoc now strips trailing empty paragraphs and the guard no longer routes on trailing blanks. Interior/leading empties are unaffected. +regression tests. - Medium: the debounced untitled→filename rename fired on remote Yjs edits too, so every peer renamed and could rename from a not-yet-synced heading. Gate on isChangeOrigin (local edits only; false for non-collab surfaces). - Medium: stripEmptyListItemLines over-stripped a nested empty item that follows a same-indent sibling (a real placeholder the parser keeps). Narrowed to the actual Setext hazard — an empty item DIRECTLY under a shallower parent line — matching the function's own docstring intent. Probe-verified. +test. - Low: corrected untitled-title.ts docstring that described a reverse name→heading coupling removed during review. * fix(files): a remote edit must not cancel the local rename debounce The isChangeOrigin gate cleared the debounce timer BEFORE bailing on a remote update, so a peer's edit arriving within the 600ms window cancelled the local user's pending rename. Bail on isChangeOrigin first, before touching the timer; only local edits clear/reschedule it. * docs(files): correct EMPTY_PARAGRAPH_SPACING rationale after trailing-strip The stacked trailing-empty-paragraph strip made the older comment overstate a correctness necessity it no longer owns, mislabel trailing runs of 2+ blanks, and advertise dead CRLF handling. Reword to match what the code actually does. --------- Co-authored-by: Waleed Latif <walif6@gmail.com> * fix(realtime): access-revalidation multi-room safety + cleanup pass Fix a blocker surfaced by a full cleanup/simplify audit of the branch: the access-revalidation sweep (staging's workflow-only #5917) treated every entry in socket.rooms as a workflow id, but the generalized multi-room model puts namespaced files/tables/file-doc rooms on the same io. It would resolve those as bogus workflows, get null, and evict files/tables collaborators every ~30s. collectScanTargets now decodes each room name with parseRoomName and sweeps only workflow rooms; added a regression test and fixed the now-false TSDoc. Other audit fixes (all behavior-preserving): - workflow.ts reuses resolveAvatarUrl (drops db/user/eq imports duplicated from avatar.ts) - PresenceAvatars: mr-1 was baked into the shared component, silently adding a margin to the workflow sidebar stack; moved to an optional layout className, re-applied on the tables/file-doc header surfaces only - table DELETE routes only signal collaborators when rows were actually removed (matches PUT) - events.ts definition kind: drop the never-emitted reason:'schema', fix its doc - event-log: rename buildMemory -> buildEntry (it builds the entry on the Redis success path too, not just the memory fallback) - remove dead resolveWorkspaceIdForRoom export; parallelize per-socket removals in handleWorkflowDeletion; gate the table columnIndexById map on remote selections; move file-doc module TSDoc off the FileDocOwner interface; fix a stale @returns * fix(realtime,tables): close table-presence race + v1/copilot live-collab gaps Validated each issue with subagents before implementing the cleanest fix. - tables LEAVE in-flight-join race (B8): the table handler tracked no current-table intent, so an unscoped/same-table leave during an in-flight authorize left the socket stranded in the room (present in the roster, broadcasting a ghost until disconnect). Mirror workspace-files: a closure-local currentTableId + a leave that advances joinGeneration to cancel the racing join. + 3 regression tests. - v1 + copilot live-collab signal gap (D1): tables edited via the v1 public API or Sim/copilot emitted no edit/schema signal, so open collaborators didn't live-update. Add the signals at those call sites (add-only, matching the existing route seam) — never in the service, so execution writes can't double-emit. Sync-only for copilot bulk ops, guarded on affected/deleted count; async job branches stay covered by their kind:'job' events; create/delete/get untouched. - table join read consolidation (B5): sweepStalePresence returns its roster so the same-tab dedup reuses it instead of a second getRoomUsers. - shared authorize slice (B6): extract only the guard-safe authorize->allowed branch into resolveRoomJoinAuth, shared by the three room handlers (the full preamble stays inline — file-doc's generation capture sits mid-ladder and must not move). - resize-revert flicker (E3): a peer's value-less metadata event forces a refetch that could momentarily revert a just-finished local resize; a pendingWidthWriteRef keeps local widths leading until the width PUT settles. - embedded-mode stray emit (E5): gate emitCellSelection on a bound table id so the embedded surface stops broadcasting cell selections the server drops. * fix(tables): close two copilot live-collab signal gaps + harden presence sweep Follow-ups from a comprehensive review of the branch: - copilot batch_update_rows and import_file's inline append branch wrote rows but emitted no live-collab signal, so collaborators didn't see those edits live (the append's sibling replace branch already signalled). Add the guarded signal to both, matching the internal route. - sweepStalePresence now reads the roster before the fetchSockets liveness probe and returns it on a probe failure, so same-tab dedup still runs during a transient fetchSockets outage instead of being skipped. - reword an internal comment off the retired "mothership" term. * fix(realtime): guard table join commit + rollback against supersession; drop no-op eviction cleanup Review round on #5991: - Table join re-checked the generation only once after authorize, then awaited leave/sweep/avatar before joining + registering presence. A table switch or leave in that window stranded the socket in the wrong room, and the failure catch could tear down a newer successful join. Resolve the avatar up-front, re-check generation immediately before the membership commit (matching the file-doc join), and skip the rollback/error for a superseded join. + a post-authorize-window regression test. - access-revalidation cleanup treated removeUserFromRoom's no-op false as a transport failure and re-enqueued a still-connected socket forever. Only retry when the socket is still mapped to the room (a healthy null mapping means the entry is already gone). Repurposed the expired-mapping test to lock it. * fix(realtime): guard table join leave-prior against superseding join Round 2 on #5991: a superseded join's leave-prior could still run — during its getRoomForSocket await a newer join commits to its room, so currentRoom is that newer room and the superseded join would leave/remove/broadcast it before the final guard aborts. Re-check the generation immediately after the lookup await, before the leave mutation. Extended the post-authorize-window test to assert the superseded join never tears down the newer join's room. * fix(realtime): roll back a table join superseded during addUserToRoom Round 3 on #5991: after the final generation guard, A could join + register presence while a newer join B commits to its room during addUserToRoom's await — B's leave-prior can't observe A's half-written entry, so A's late write wins and strands the socket. Re-check after addUserToRoom and roll back A's own Socket.IO join + presence (scoped to A's room, never touching B). + a regression test hanging addUserToRoom mid-commit. * refactor(realtime): DRY table-join supersession guards; fix stale comment Cleanliness pass after the review rounds (no behavior change): - Extract the four identical `joinGeneration !== joinAttempt || socket.disconnected` checks into a named `superseded()` helper (the catch keeps its intentionally narrower check). - Remove a stale guard comment that was left stranded above the avatar resolve. - Document the best-effort rollback catch. * fix(realtime): file-doc rebind must not drop the current doc or leave a writable ghost Two Cursor findings on the file-doc client-id ownership rebind: - On a document switch, the prior room was left BEFORE the ownership check, so a CLIENT_ID_IN_USE rejection dropped the socket from the old doc without joining the new one (contradicting its own comment). Run the ownership check first, and leave the previous doc only once the rebind is guaranteed to succeed. - Reclaiming a client id removed the stale prior socket from owners + awareness only; its socketToRoomName + Socket.IO membership remained, and handleMessage's SYNC path gates on socketToRoomName (not owners), so it stayed able to write document frames until disconnect. Fully evict the reclaimed socket. + 2 tests. * refactor(realtime): serialize table join/leave to fix map-corruption at the root Round 4 on #5991 surfaced a race the generation guards structurally cannot fix: two concurrent joins for one socket race on the single-valued socket→room map — a stalled addUserToRoom for table A lands late, clobbers a newer join's map entry to A, and the rollback then wipes it, stranding the socket (map empty while it holds table B). Guards protect JS suspension points; they can't stop an in-flight Redis write from landing late. Fix per architecture review: serialize this socket's JOIN + LEAVE on a per-socket promise chain so their multi-step async Redis commits can never interleave — restoring the atomic-commit property the synchronous sibling handlers get for free. This DELETES the leave-prior guard and the post-commit rollback (the code that caused the bug); four generation guards collapse to two identical superseded() checks (skip a superseded queued op + one pre-commit check). Reworked the interleaving-specific tests into a fast-switch-skips-superseded test; the leave- cancels-join tests are unchanged. Local to the tables handler — no shared-infra change. * fix(realtime): always roll back a failed table join; re-elect file-doc seeder on reclaim Two review findings: - Table join: the catch skipped rollback when superseded, but a socket.join that landed before addUserToRoom threw leaves the socket in the Socket.IO room with no matching socket->room map entry — unreclaimable by any later op (cleanup keys off the map). Under serialization the skip is unnecessary (the newer op hasn't committed), so always roll back. Simpler + fixes the strand. - File-doc reclaim: fully evicting the prior socket didn't release the seeder role if it held it, so electSeederIfNeeded (which no-ops while seederSocketId is set) never re-elected and an unseeded doc stayed empty until the deadline. Clear the role on eviction so the join's election picks a new seeder. + 2 regression tests. * fix(realtime): close revoke-race ghost presence in workflow join An access-revalidation revoke landing between socket.join and addUserToRoom socketsLeaves the socket while its presence mapping does not yet exist, so cleanupEvictedSocket finds nothing to remove and the join then writes presence for a socket already out of the room — a ghost collaborator until the stale sweep. Hoist resolveAvatarUrl (the only await in that gap) above the re-auth check so the whole re-auth -> socket.join -> addUserToRoom section is await-free, matching the invariant the handler already relies on for the pre-join re-auth. + ordering regression test. * refactor(realtime): serialize workflow join/leave; drop dead room-authz limb Comprehensive independent audit follow-ups: - workflow.ts join/leave now use the same opChain + joinGeneration serialization as the sibling handlers (tables, file-doc, workspace-files). It was the only async presence path left unserialized, so a rapid workflow switch A->B (or a leave racing an in-flight join) could strand presence in room A — a ghost collaborator still receiving A's operation broadcasts until disconnect. The join now aborts a superseded op at start and again right before the membership commit, and the catch always rolls back a partial join. - leave-workflow drops the '&& session' gate: an idle user whose 1h session key expired (while the 24h room mapping is still live) can now leave cleanly instead of being stranded until disconnect. The room ref alone suffices. - authorizeRoom: remove the dead ROOM_TYPES.WORKFLOW resolver + its getActiveWorkflowContext import. Workflow authorizes through its own path and never flows through authorizeRoom; the map now honestly covers only the workspace-scoped types (files, file-doc, table). - Remove unused isSameRoom (zero callers) and a needless useMemo in PresenceAvatars (plain derivation, single copy). - Tests: 4 workflow serialization/leave regressions. All gates green: tsc (sim/realtime/packages) 0, 204 realtime + 11 protocol tests, biome, api-validation, boundaries, prune 14/25. * fix(realtime): align session TTL, harden committed joins from post-success rollback Per-module comprehensive audit follow-ups: - redis-manager: SESSION_TTL now tracks SOCKET_ROOMS_TTL (was 1h vs 24h). The room set outlived the session, and since getRoomForSocket reads the room set while the workflow handlers gate edits/presence on `room && session`, an active-but-idle collaborator got wedged into 'session expired' after 1h — sticky until reload (activity only EXPIREs the already-gone session; only addUserToRoom re-HSETs it). Both keys refresh together, so they now expire together (restores the pre-refactor consistency, where both shared one TTL). - workflow.ts + tables.ts: a 'committed' flag stops the join catch from rolling back a genuinely-joined user when a trailing ack/broadcast/metric step fails on a Redis blip (a pure getUniqueUserCount log-metric failure could otherwise kick a live collaborator after success was already acked). - workflow.ts: leave-prior now guards `currentRoom.id !== workflowId` (a same-workflow re-join no longer leave→re-adds and flickers peers' presence), and the join ack is liveness-filtered via filterVisiblePresence — both for parity with the tables handler. - platform-authz: honest docstring + 400 message for the workspace-scoped-only authorizeRoom map (workflow authorizes via its own path). - caret-presence: corrected an over-stated batching comment. - +1 workflow regression test (post-success failure keeps the user joined). Gates: tsc (sim/realtime/packages) 0, 205 realtime + 11 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): narrow join commit-guard to post-success; skip empty presence broadcast Two Cursor findings on the prior audit-fix commit: - The 'committed' guard in join-workflow/join-table was too broad: a failure BETWEEN the membership commit and the success ack (e.g. getWorkflowState) hit 'if (committed) return' and emitted neither success nor error, hanging the client while it sat in the room. Replaced with the narrower shape: only the purely-decorative post-success steps (peer broadcast + log-metric) are wrapped best-effort; anything before the success ack still rolls back and surfaces a retryable error, so the client retries instead of hanging — while the original goal (a benign broadcast/metric blip never kicking a live, acked user) holds. - broadcastPresenceUpdate read the roster via getRoomUsers, which swallows a Redis transport error to []. On a disconnect broadcast that emitted an empty roster and cleared every remaining collaborator's presence until the next healthy update. Split out a throwing readRoomUsers; broadcastPresenceUpdate now skips the broadcast on a read failure (getRoomUsers keeps its swallow contract). - Tests: pre-success failure rolls back + retryable error (no hang); post-success failure keeps the user joined. Gates: realtime tsc 0, 206 realtime tests, biome, boundaries, prune. * fix(realtime): harden seeder recovery, join-generation, and misc robustness Final line-by-line audit follow-ups (all LOW/MED, no P0/P1): - file-doc: a sole client whose seed FETCH fails was added to triedSeeders, re-election found nobody, and the document stayed permanently empty until reload. Re-offer seeding a bounded number of rounds (MAX_SEED_ROUNDS) before giving up. Also bound clientId to a non-negative integer (it is an ownership key). - tables + workspace-files: validate the room id BEFORE advancing joinGeneration, so a malformed/rejected join can't cancel a legitimate in-flight join. - workflow + tables + workspace-files: suppress the client-facing join error when the op was already superseded (a retryable error naming the abandoned room could make a client re-join and cancel its newer join). The rollback still runs. - redis-manager: set isConnected=true only after scriptLoad succeeds (and reset it on failure) so isReady() can't report ready while the Lua SHAs are null. - connection: apply the presence-bearing filter to the manager-removed set too (symmetry with the fallback path). - http.ts: validate workflowId on the four workflow endpoints (matching the files one). - client: clear a pending join-retry timer before rescheduling (reconnect churn no longer orphans a stray extra join); clear caret fade timers on plugin destroy; seed-effect cleanup reports NOT-ready (safe direction). - Tests: bounded seeder recovery, cell-selection strip-junk, TABLE round-trip, presenceEventName. Gates: tsc (sim/realtime/packages) 0, 208 realtime + 12 protocol tests, biome, api-validation, boundaries, prune. * fix(realtime): gate isReady() on loaded script SHAs, not just connection Follow-up to the prior isConnected change, which was incomplete: the redis client's 'ready' event flips isConnected=true on connect — before initialize() loads the Lua scripts — so a bare isConnected check reports ready while removeUserFromRoom / updateUserActivity would silently no-op on a null SHA. Gate isReady() on the SHAs too, so the POST endpoints return a retryable 503 during that startup window instead of proceeding against unloaded scripts. Standard readiness-probe discipline. * fix(files): offline read-only fallback when the realtime doc never syncs When the realtime server is unreachable (offline, server down, socket never connects), the collaborative editor would sit blank and read-only forever — content only arrives via provider sync events that never fire. Add a bounded connect-deadline to the Yjs provider: if no first sync lands within CONNECT_DEADLINE_MS, latch fatal and emit a synthetic non-retryable join-error — the exact path a real fatal rejection already uses, which seeds the file's stored content read-only. Latching fatal also stops a late reconnect from syncing server state in and merge-duplicating the locally-seeded content (the documented Yjs 'non-empty doc ignores initial value' gotcha). Deliberately NOT adding durable Yjs snapshot persistence / server-side seeding: TipTap can't run the markdown->Yjs conversion server-side (Collaboration extension errors under jsdom), and a durable binary snapshot would create a dual source of truth with the markdown file (edited by copilot / PUT / download). The client-seeder + bounded re-election is the correct architecture for a markdown-is-truth model; this closes its one real user-facing gap without persistence, a migration, or dual-truth. Timer cleared on first sync, on a real fatal rejection, and on destroy. +2 tests. Gates: sim tsc 0, 496 editor tests, biome. Needs live offline->reconnect verification. * fix(realtime): validate workflow join id before generation bump; scope file-doc join rollback to its target Two Cursor findings: - join-workflow bumped joinGeneration before validating workflowId (unlike tables / workspace-files, which I'd already fixed). A malformed/empty join could advance the counter and cancel a legitimate in-flight workflow switch. Validate the id first. +test. - The file-doc join catch called cleanupFileDocForSocket unconditionally, which keys off socketToRoomName. During a document SWITCH that fails before rebinding (e.g. a throw in client-id reclaim), that binding still points at the socket's PRIOR, valid document — so the rollback tore down a document the socket was validly in. Only run that cleanup when the binding already points at THIS join's target; otherwise the socket never registered as an owner here and the only leftover is a freshly-created empty room, dropped by destroyRoomIfIdle. Gates: realtime tsc 0, 209 tests, biome, boundaries, prune. * fix(files): drop late sync frames once fatal; file-doc join error suppression + retry-budget reset Final safety-audit findings: - CRITICAL: FileDocProvider.handleMessage had no fatal guard. After the connect deadline latched fatal and the editor fell back to a read-only local seed, a late SyncStep2 (slow server / flaky network / deploy) was still applied — merging server state into the seeded doc (content duplication) and flipping synced=true, which un-gated autosave and would persist the duplicate to the real file. fatal guarded (re)join but not inbound sync. Now handleMessage returns early when fatal. +test (late SyncStep2 after the deadline is ignored, doc stays empty + gated). - file-doc join catch now suppresses the client-facing error when superseded (matches workflow/tables/workspace-files) so a retryable error for an abandoned file can't make a client re-join and cancel the newer one. - table/workspace-files room hooks reset the retry budget on (re)connect so a prior full exhaustion doesn't block retries after a reconnect. - presence-visibility: corrected a stale TTL comment. Gates: tsc (sim/realtime) 0, 209 realtime + collab/hooks suites, biome, boundaries, prune. * feat(collab-doc): server-authoritative Yjs seeding (#6008) Server-authoritative Yjs seeding for collaborative file documents: the realtime relay fetches a Yjs seed built from the file's markdown (via the shared TipTap engine) and applies it once per room, replacing the client-seeder election/handshake entirely. - DOM-free markdown<->Yjs conversion core (markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc) reusing the client markdown engine for parity by construction - Internal x-api-key seed endpoint + realtime fetch; single attempt bounded under the client readiness deadline, guard-release for join-driven retry, read-only fallback on persistent failure - Client readiness gate = synced && server seed flag; jsdom wired for the Next standalone build (serverExternalPackages + outputFileTracingIncludes) Foundation only — copilot-into-doc + markdown projection + durable persistence are Stage C. * feat(collab-doc): Sim merge endpoint for copilot-into-doc (Stage C foundation) buildFileDocMergeUpdate(docState, markdown) computes the minimal Yjs diff that turns a live document into target markdown, via applyMarkdownToYDoc (a real updateYFragment diff, not a replace) — so a copilot rewrite merges with concurrent user edits instead of clobbering them. Exposed over the internal x-api-key /api/internal/file-doc/merge endpoint the realtime relay will call: the relay owns the doc, the app owns the conversion engine, so the relay ships the current state and applies the returned diff. Tested incl. concurrent-edit no-clobber. * feat(collab-doc): realtime apply-edit — merge copilot markdown into a live doc The relay can now stream a copilot edit into open editors: applyMarkdownToLiveFileDoc finds the seeded live room, ships its state to the app's /merge endpoint for a minimal CRDT diff, applies it (relaying to every editor, reconciled with concurrent user edits), and reports 'no-live-room' so the caller falls back to a direct file write when nothing is open. - Generalize the realtime->app request module (file-doc-seed.ts -> file-doc-app.ts) with a shared POST helper + fetchFileDocSeed/fetchFileDocMerge - POST /api/file-doc/apply-edit on the internal x-api-key HTTP surface, returning { applied } - Tests for the seeded-room merge relay and the no-live-room fallback * feat(collab-doc): stream copilot edits into open editors (Stage C) edit_content now, after its durable file write, best-effort merges the same markdown into the file's live collaborative document (markdown files only). If a collaborator has it open, the edit streams into their editor as a CRDT merge — reconciled with their concurrent typing — instead of the file silently changing under them; the editor's existing autosave mirrors the merged doc back to the file. No-op when nothing is open. Never blocks or fails the edit. * fix(collab-doc): strip frontmatter on merge; gate live-merge to markdown - buildFileDocMergeUpdate now strips YAML frontmatter (splitFrontmatter().body) exactly as the seed does. Copilot passes full-file content, so without this the frontmatter merged into the doc as editor content and autosave wrote it back over the file (corruption). - Gate the live-doc merge on isMarkdownFileName (new server-safe helper) instead of the over-broad !isDoc, so code/text edits don't pay the realtime round-trip for a format the collaborative editor never renders. * fix(collab-doc): make frontmatter collaborative so a merge can't revert it The editor re-attaches its open-time frontmatter on every autosave, so the Stage C merge (which triggers an autosave with no user action) could write stale YAML back over a copilot frontmatter change — silently dropping it. Carry the file's frontmatter in the doc's config map instead of locking it at open: the seed stores it, the merge updates it (only when it actually changed, preserving the no-op diff), and the editor re-attaches THAT value on save — falling back to the locked copy before the seed lands and for non-collaborative docs. A server-side frontmatter change is now reflected rather than reverted. New FILE_DOC_SEED.frontmatterKey; seed/merge tests cover it. * fix(collab-doc): re-sync draft on a frontmatter-only merge A server edit that changes only the frontmatter updates the config map but not the body fragment, so TipTap's onUpdate never fires — the autosave draft kept the stale open-time frontmatter, and an explicit save could revert the live change. Observe the config map and, on a frontmatter-only change, re-attach the new frontmatter to the current body and push a fresh draft (guarded on a synced body so it never races the seed's own onUpdate). * fix(collab-doc): order the apply-edit/merge timeouts (outer > inner) The Sim->realtime apply-edit timeout (4s) was shorter than the nested realtime->Sim merge timeout (8s), so the outer call could abort while the relay was still merging — the relay then applied the merge after edit_content had returned, racing a follow-on edit. Split the shared realtime->app timeout: the seed keeps 8s (it reads a cold blob), the merge gets a tight 3s (it is a pure conversion, no I/O), and the outer apply-edit is raised to 6s so it always outlives the inner merge. Cross-referenced in comments to prevent drift. * fix(collab-doc): address deep-audit findings (jsdom trace, timeouts, gate, races) A 4-agent LOC audit + precedent research (TipTap/Hocuspocus/Yjs docs confirm the core patterns are idiomatic) surfaced these real issues: - The /merge route also lazy-requires jsdom but was missing its outputFileTracingIncludes entry — a Docker/standalone build would 500 with MODULE_NOT_FOUND. Add it. - The four conversion timeouts encoded an ordering invariant (merge<applyEdit, seed<readiness) living only in prose across two apps. Hoist to a shared FILE_DOC_TIMEOUTS in @sim/realtime-protocol with a test asserting the ordering; both apps import it. - The copilot live-merge gate used an extension-only check, but the editor treats any text/markdown-MIME file as markdown. Replace isMarkdownFileName with a MIME-aware isMarkdownFile mirroring the client, so those files stream too. - applyMarkdownToLiveFileDoc had no per-file serialization; overlapping merges could each diff the same stale snapshot and apply out of order. Serialize per file via a promise chain. - Editor hardcoded 'config'/'initialContentLoaded' instead of FILE_DOC_SEED constants (drift). - Add .max() bounds to the merge contract body; document the best-effort-merge failure window honestly (open editor + merge failure can drop a copilot edit until reload — closed by the deferred durable-doc work). * feat(collab-doc): multi-replica shared Yjs backend + server-side markdown persistence Make collaborative file-doc editing correct across multiple ECS tasks (the per-process Y.Doc previously assumed one replica per file). - Shared Yjs backend over Redis Streams (apps/realtime file-doc-store): each file's stream is the ordered, replayable log of updates; a multiplexed XREAD tailer converges every task's in-memory doc. Coordinated single-seeder election (SET NX + empty-stream recheck) fixes split-brain seeding. - Doc-sync fans out to local clients + the stream; awareness stays on the Socket.IO adapter. Snapshot+XTRIM compaction trims only integrated entries. - Server-side persistence: project the live doc back to markdown via a new /api/internal/file-doc/persist endpoint, debounced during editing and flushed on last-disconnect, from the authoritative stream state. Collaborative editors no longer client-autosave, closing the copilot clobber-window. - Copilot merges apply through the stream (reach the live doc on any task) and serialize cross-task via a Redis merge lock. - Degrades to the original single-replica behavior when REDIS_URL is unset. * fix(collab-doc): harden distributed locks and durability from review Address Greptile + Cursor review of the multi-replica backend: - Merge lock: retry LONGER than the lock TTL (guaranteed acquisition, never merges against a shared base while a peer holds the lock) and AWAIT the stream write before releasing, so the next task never diffs a stale base. - Distributed locks (seed/merge/compact) now use ownership tokens + a compare-and-delete release (Lua), so a lock that expired and was re-acquired by another task is never stolen; acquisition fails CLOSED on Redis error. - Seed: publish the seed to the stream AWAITED under the lock before releasing, so a later seeder's empty-stream fence always sees it (closes the fence's publish-after-release gap); TTL kept at the readiness deadline. - Persist: persist the AUTHORITATIVE stream state even when this task's local doc was never seeded, and capture the local fallback synchronously so a last-disconnect flush never encodes an already-destroyed doc. - Publish: retry a transient xAdd failure so a Redis blip can't silently drop an edit from the shared log. * fix(collab-doc): only persist a doc a user actually edited Cursor review: a copilot durable write landing while a doc is being seeded could have the stale seed projected back over it on last-disconnect, clobbering the copilot edit even with no user changes. Gate server-side persistence on a genuine user edit (socket-origin update): a seed-only or merge-only doc is never projected back to the file (copilot writes the file durably itself), so it can't clobber a concurrent external write. * fix(collab-doc): close review-round race/durability gaps Address Greptile P1 + Cursor findings: - Seed publishes to the shared stream AWAITED *before* seeding the local doc, so a publish failure leaves the doc unseeded and the stream empty for a clean retry rather than serving an unpublished local seed a peer would re-seed over (split-brain). - flushPersist falls back to the synchronously-captured local snapshot when getStreamState throws (not only when it returns null), so a transient Redis read no longer drops the final durable write as the room is torn down. - streamHasContent fails CLOSED (returns true on xLen error): a Redis blip can no longer let the seed fence pass and double-seed. - Client collabReady initializes from the collaborative prop, so a collaborative editor never has a mount-window where client autosave could clobber the server write. * fix(collab-doc): unstick seed retry and persist tailed edits Cursor review: - ensureServerSeed clears serverSeedStarted when it aborts at the streamHasContent fence, so a fail-closed Redis xLen (or a genuine peer-seed) no longer strands the room unseeded with no retry. - Persistence dirty-tracking now marks a doc edited on any post-seed update, including a peer's edit relayed via the tailer (REDIS_ORIGIN), tracked via a seededObserved flag. The last task to leave persists real edits even if it only tailed them; the seed transition itself is still never counted, so a seeded-but-unedited doc is never projected back over the file. * fix(collab-doc): match client markdown post-processing on server persist Cursor review: yDocToFileMarkdown serialized the body with yDocToMarkdown only, but the editor save path runs postProcessSerializedMarkdown before applyFrontmatter. Server persist could therefore write markdown differing from a client save (empty list markers, callout un-escaping) — spurious blob churn / round-trip drift despite the byte-identical claim. Apply postProcessSerializedMarkdown in yDocToFileMarkdown so a server persist is byte-identical to a client save and the client's dirty-check baseline. Add a regression test guarding the composition. * fix(collab-doc): close durability gaps at deploy boundaries + audit polish From a comprehensive from-scratch audit (correctness, SOTA, cleanliness, feature-completeness): - Persist max-wait: a continuous edit burst kept resetting the 5s debounce and never persisted; cap it so a burst flushes at least every 20s, bounding unpersisted edits. - Graceful-shutdown flush: flushAllFileDocRooms awaited in shutdown so a rolling deploy / scale-in secures open edited rooms to durable markdown before exit, instead of relying on the stream + a surviving task. - Compacted-snapshot catch-up now marks the doc edited (REDIS_SNAPSHOT_ORIGIN): a snapshot folds seed+edits into one frame, so a task catching up purely from it no longer treats real edits as an unedited seed and skips persisting. - Polish: delete dead __setFileDocStoreForTest; bounded retry loop; rename acquirePersistSlot -> tryClaimPersistWindow with accurate docs; tailer object-identity guard; fix stale comments (seed route, edit-content autosave). * improvement(realtime): post-review fixes for collab dirty-state + relay lifecycle - collab editor no longer latches a spurious 'Unsaved changes' prompt: report dirty only when the client owns durability (canAutosave), since in a collaborative session the relay persists the doc server-side - guard shutdown against a double SIGINT/SIGTERM running teardown twice - disconnect local sockets before httpServer.close so shutdown exits gracefully instead of hitting the forced-exit timer (local-only, deploy-safe) - return 400 (not silent 200) on an invalid workspaceId in the files-changed fanout - clear a pending join-retry timer on reconnect so it can't fire a duplicate join * fix(realtime): make collab-doc seeding atomic to close split-brain window An adversarial concurrency audit found a split-brain double-seed vector: the seed used an advisory SET NX PX lock + a SEPARATE xLen fence + an unconditional xAdd (a non-atomic check-then-append). If the seed lock's TTL expired mid-seed (a >4s stall after the 8s seed fetch), a second task could acquire the freed lock over a still-empty stream, both fences read empty, and both append seeds with distinct Yjs client ids -> duplicated document content. - add an atomic SEED_IF_EMPTY_SCRIPT (append-iff-empty in one Redis step) + store.seedIfEmpty(); the emptiness check and append are now inseparable, so two tasks racing (even both past an expired lock) can never both seed - ensureServerSeed uses seedIfEmpty instead of streamHasContent-fence + publishAndWait; the seed lock is now purely an efficiency optimization (avoid a duplicate fetch), not a correctness dependency - fix the misleading comment that claimed a copilot merge is not counted as an edit in the multi-replica path (it round-trips as REDIS_ORIGIN and does count; a safe idempotent over-persist, never a lost edit) - add interleaving tests: seedIfEmpty atomicity/fence, the split-brain regression under an expired lock, a peer edit during attach catch-up, and concurrent two-task compaction * docs(realtime): align seed comments with the atomic-append correctness model Greptile flagged lingering doc drift: the module + shouldSeed comments still credited the seed lock + empty-stream check as the split-brain fix. Reframe them so the atomic seedIfEmpty is the exactly-once guarantee and shouldSeed is an efficiency gate only. * feat(realtime): live workspace tables list, sharing one invalidation-room impl (#6053) * feat(realtime): live workspace tables list, sharing one invalidation-room impl Bring the tables list to parity with the files list: a create/rename/move/delete/ restore now propagates to every viewer live instead of waiting out the 30s staleTime. Following the files pattern, but factoring the two into one shared implementation rather than copy-pasting. - add ROOM_TYPES.WORKSPACE_TABLES + its authz resolver (workspace-id-addressed, reuses the workspace resolver like workspace-files) - extract setupWorkspaceInvalidationRoom (server) and useWorkspaceInvalidationRoom (client) — the presence-free, workspace-scoped live-list room; files and tables now both bind to it, so they can never drift. Event/room names derive from the room type. Replaces the standalone workspace-files handler + hook - notifyWorkspaceTablesChanged fanout fired from the table service (createTable, renameTable, moveTableToFolder, deleteTable, restoreTable) so it covers both the HTTP routes AND copilot, which call the service directly - relay /api/workspace-tables-changed endpoint; wire the hook into the tables page - consolidate the handler test into one suite run against both room types * feat(realtime): live tables list also covers table-folder mutations Fold in the follow-up: a table folder create/rename/move/delete/restore now propagates to the tables list live too, so the browser is fully consistent. - generic notifyFolderResourceChanged(resourceType, workspaceId) dispatches the workspace live-list signal by resource type (a map, not a special-case if), so file/knowledge_base/workflow are no-ops today and gain liveness by adding a map entry when they adopt an invalidation room - fired from the shared folder lifecycle (createFolder/updateFolder/deleteFolder/ restoreFolder), covering routes AND copilot - the tables room hook now invalidates the table folders query too, not just the tables list, since the page renders both * fix(realtime): skip per-table live-list notify during a folder cascade A folder delete/restore already fires one folder-level notifyFolderResourceChanged for the whole subtree, but the cascade also calls deleteTable/restoreTable per table — each awaiting its own notifyWorkspaceTablesChanged. A folder with many tables would run N+1 sequential relay calls (each bounded by NOTIFY_TIMEOUT_MS), blocking the mutation. Add a skipNotify option the cascade passes so only the one folder-level notify fires. * feat(collab-doc): Hocuspocus binary persistence + Next 16 seed/persist fixes (#6059) * fix(collab-doc): make server-side seed conversion work under Next 16 / Turbopack Opening a file left both collaborators read-only and stalled ~12s: the server-side seed (markdown -> Yjs, run through the headless editor engine) was failing, so the doc never seeded and the editor never left its readiness gate. Two root causes, both latent until a real build/runtime (typecheck + unit tests don't exercise either), surfaced by the Next 16 upgrade: 1. Build boundary: the server seed route imported the shared editor schema (`createMarkdownContentExtensions`), which pulled in the React node-view components (`useEffect`) -> 'client component in a Server Component'. Split each node's React-free schema into its own `*-schema.ts` (code-block, image, raw-markdown-snippet); the client editor still injects the React node views via the existing `nodeViews` param, unchanged. 2. Runtime DOM: the converter installs a jsdom `window` on `globalThis`, but Turbopack's server bundle gives bundled `@tiptap/core` a `window` that does NOT read `globalThis`, so `elementFromString` threw 'no window object available'. Externalize the `@tiptap/*` packages the converter uses (native Node require, so their `window` reads the real global) and fix the converter's DOM guard to gate on `window` (what TipTap checks) with no sticky flag. Verified: seed route returns 200 with the Yjs update; 514 collab-doc + editor tests pass; schema byte-identical after the split. * feat(collab-doc): persist the Yjs binary and load it on cold-start (Hocuspocus pattern) Adopt the industry-standard Hocuspocus store/load-document pattern so a cold room open loads the file's last-persisted Yjs binary directly instead of re-converting markdown -> Yjs on every open. Rebuilding the CRDT from markdown on each connect is the exact anti-pattern Tiptap/Yjs warn against (fresh client ids -> duplicated content); it also forced the fragile server-side headless-editor conversion on every open. Now conversion runs only on a genuine first open or an external markdown edit. - new table workspace_file_collab_state(file_id PK->workspace_files cascade, doc_state bytea, source_hash, updated_at): the Yjs binary + a hash of the markdown it was derived from (bounded <=~1MB by the 256KB round-trip gate). Mirrors Hocuspocus's extension-database (binary in a DB column). Migration 0275. - persist upserts the binary (tagged with the exact markdown just written) - cold-start seed returns the cached binary when its source_hash matches the file's current markdown; otherwise converts (and the next persist refreshes the cache) - also externalize yjs / y-protocols / lib0 alongside @tiptap: bundling loaded a second yjs copy, so @tiptap/y-tiptap's 'instanceof Y.XmlElement' failed on app-created nodes ('Unexpected case') during Yjs -> markdown Verified end-to-end: seed -> persist -> seed returns the exact persisted binary (a cache hit, no re-conversion). 18 collab-doc + 51 realtime file-doc tests pass. * fix(collab-doc): best-effort cache read + drop dead barrel - seed: a cache-read failure (transient DB error, not-yet-migrated cache table) no longer aborts a cold room open — the durable markdown is already in hand, so fall through to conversion. Symmetric with persist's best-effort cache write. Addresses the Cursor Bugbot finding on the read/write asymmetry. - remove the collab-doc index.ts barrel: nothing imported it (every consumer uses direct ./seed / ./merge / ./converter imports), so it was dead re-export surface. De-export COLLAB_DOC_FIELD accordingly — it is used only inside converter.ts. * fix(collab-doc): stream every external file write into open editors, not just edit_content (#6070) * fix(collab-doc): stream every external file write into open editors, not just edit_content A copilot/mothership edit to an open markdown file did not appear live in another user's editor: the live-doc merge bridge (mergeEditIntoLiveFileDoc) was wired into the edit_content tool ONLY. Every other server-side write — the file tool (/api/tools/file/manage), function_execute (/api/function/execute via writeWorkspaceFileByPath), create_file overwrite, and the PUT /content route — went straight to updateWorkspaceFileContent and skipped the merge, so the durable file changed but the open editor never updated. Confirmed from live logs (the mothership 'Prepend sentence' ran read + file + function_execute — zero apply-edit calls) and Redis (the prepended text was absent from the doc stream). Centralize the merge at the one chokepoint every external writer shares: - updateWorkspaceFileContent gains an opt-out "syncLiveDoc" (default on) and, after the durable write, merges markdown writes into any open collaborative doc (best-effort; no-op when nobody has it open). Any current OR future writer is covered automatically. - persist.ts opts out (syncLiveDoc:false) — it IS the doc→markdown projection, so merging it back would be a persist→merge→persist self-loop. - create_file opts its empty shell out (real content arrives via a later write) so an open editor never flickers to empty on overwrite; threaded through writeWorkspaceFileByPath. - edit_content drops its now-redundant explicit merge call (the chokepoint handles it). - binary writers (image/video/audio/ffmpeg/download) are naturally excluded — the merge is gated to markdown, the only format the collaborative editor renders. Also bump the api-validation route baseline 994→996 to match the true route count already on this branch (pre-existing ratchet drift from an earlier merge; NOT added by this PR). * fix(collab-doc): defer setEditable out of the render phase (flushSync warning) The collab editability-reapply effect called editor.setEditable synchronously. In collab mode isEditable flips from readiness (synced + seeded), which is driven by a Yjs config.observe firing synchronously inside Y.applyUpdate — so the effect can run while React is mid-render. TipTap's React binding commits setEditable's transaction with flushSync, which throws "flushSync was called from inside a lifecycle method. React cannot flush when React is already rendering." Defer the setEditable to a microtask (runs right after the current commit, before paint), guarding against a destroyed editor or a stale value before it fires. Only the collab path (this effect) hit the warning; the streaming/settle effect's setEditable calls run on the non-collab path where isEditable isn't driven by a mid-render Yjs observer. * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) (#6073) * fix(rich-markdown-editor): defer non-collab settle/stream mutations off the render phase (flushSync) The non-collaborative streaming/settle effect called editor.setContent / setEditable / setTextSelection / focus directly in the effect body. setContent mounts the custom node views synchronously through the @tiptap/react flushSync path (tiptap#3764), so when this effect runs while React is mid-render it throws "flushSync was called from inside a lifecycle method." This is the second flushSync source (the collab editability effect was the first, fixed separately); it fires on the agent-streaming-into-a-non-collab-editor surface. Defer the effect-body view mutations to a microtask via a small runOffRender helper (runs right after the current commit, before paint; no-ops if the editor was torn down). The settle block is deferred as ONE microtask so setContent -> collapse selection -> setEditable -> focus keep their order. The streaming rAF tick is left untouched — it already runs off-render, so it keeps writing content directly. queueMicrotask is TipTap's own documented remedy for this warning. 497 rich-markdown-editor tests (incl. stream-settle-selection) pass; tsc + lint + api-validation + boundary + prune green. Needs a live check: stream an agent into a non-collab markdown file and confirm it still renders smoothly. * chore(rich-markdown-editor): trim verbose flushSync-defer comments * fix(rich-markdown-editor): drop superseded settle/stream microtasks via a run token runOffRender previously only guarded editor.isDestroyed, so if React ran the next reconcile pass (a newer stream or settle) before a queued microtask flushed, the stale microtask could still apply setContent/setEditable/setTextSelection over the newer state. Tag each effect run with an incrementing token; a deferred mutation applies only when its run is still the latest (and the editor is alive). A run token fits this effect's several early-return exits better than a per-exit cleanup flag. Addresses Greptile/Cursor review. * fix(rich-markdown-editor): never drop the settle selection-collapse under a superseded run The run token drops a superseded settle's microtask, but the settle had already flipped its state flags synchronously — so a pre-empting steady-sync run took the non-settle path and never collapsed the selection, leaving a post-stream select-all painting the leaf-in-selection decoration. Track the collapse as a debt (pendingCollapseRef): whichever deferred run ultimately applies — settle or the steady-sync path — clears it, so the collapse runs exactly once on the latest content. Addresses the Cursor review finding. * feat(tables): show live cell-selection carets in the embedded chat panel (#6081) The table cell-selection presence room was joined only on the dedicated /tables/[id] page (useTableRoom was passed an empty id in embedded mode). Join it in embedded too, so the mothership chat resource panel shows collaborators' live cell selections and broadcasts the local one. tableId is already resolved from props in embedded (the data event stream already uses it un-gated), and authz runs on join, so this is safe. Avatars are unaffected — they render only in the !embedded Resource.Header, so the panel gets carets without avatars. * feat(collab-doc): If-Match optimistic concurrency so persist never clobbers an out-of-band edit (#6085) * feat(collab-doc): optimistic-concurrency guard so persist never clobbers an out-of-band edit The relay projected the live Yjs doc back to durable markdown unconditionally (last-write-wins), so a persist already in flight when an external write landed could overwrite it. Add RFC 7232 If-Match optimistic concurrency end to end, reconciling through the CRDT (never rejecting user work): - updateWorkspaceFileContent gains an expectedUpdatedAt guard: the write commits only if the file is still at that version (checked against the SELECT ... FOR UPDATE-locked row, so it is atomic with the write), else it throws the new ContentVersionConflictError without clobbering. - persistFileDoc takes expectedVersion and returns a discriminated result (persisted | missing | conflict). On conflict it returns the current durable content + version instead of writing. - The relay tracks the durable version its live doc is synced to — set on seed, advanced when a durable write is merged in (apply-edit carries the version), and on each successful persist. It is held cluster-wide in Redis (filedoc:syncver:{name}) so whichever task persists reads the same version, with the per-room value as the single-pod fallback. - flushPersist sends that version as If-Match. On a conflict it merges the current durable content into the live doc (so the out-of-band edit AND the live edits converge) and retries (bounded), so even a last-leave flush racing an external write persists the reconciled result rather than losing the session's edits. Threads the version through the seed + persist contracts and the apply-edit payload. No schema change (reuses workspace_files.updatedAt as the version token). Tests: app-side CAS (match writes, mismatch throws + cleans up the orphan upload), relay conflict handled gracefully without clobber/loop; 236 realtime + 76 sim collab/uploads tests, tsc x2, lint, api-validation, boundaries, prune all green. * chore(collab-doc): heartbeat-refresh the synced-version key TTL alongside its stream Keep filedoc:syncver:{name} alive as long as the room's stream (it was only re-set on seed/merge/persist), so an open-but-idle doc's persist If-Match token can't expire and force a needless reconcile. * fix(collab-doc): stop persist-conflict retries when there is no live doc to reconcile On an If-Match conflict with no live doc to reconcile into (last collaborator gone, no shared stream), applyMarkdownToLiveFileDoc returns no-live-room; re-projecting the same pre-teardown snapshot would only re-conflict, so break the retry loop immediately and leave the out-of-band (durable) content authoritative — the intended conflict policy. Addresses Greptile review. * fix(collab-doc): close three optimistic-concurrency edge cases from review - Single-pod persist retry projected the pre-reconcile snapshot (captureState always returned the initial localState), while the synced version had been advanced by the reconcile — so the If-Match could pass and clobber the reconciled edit. captureState now re-reads the live doc on each attempt (falling back to the pre-teardown snapshot only once the room is gone). - The synced version was recorded from this task's own seed FETCH before knowing whether this task's seed actually won; a peer winning with a different version could leave a newer token than the stream content. Record it only inside the didSeed branch (the task whose seed won); peer-seeded tasks read the winner's cluster value. - Persist wrote UNCONDITIONALLY when no version was available (relay version momentarily missing), which could clobber non-empty durable content. It now returns conflict for a non-empty file with no version (reconcile/retry once the version is re-established); an empty file's first write stays unconditional. * fix(collab-doc): defer (not reconcile) on missing version, and use the freshest version token - Missing-version persist now returns 'deferred' instead of 'conflict'. A missing version token (a Redis blip on a peer-seeded task) is NOT a genuine out-of-band change, so triggering a reconcile would wipe live edits (incoming-wins) even though nothing changed durably. Deferred means: don't write, don't reconcile — leave the edits in the stream and let a later persist write them once the version is re-established. - currentVersion now takes the MAX of the cluster (Redis) and local room versions rather than always preferring Redis, so a lagged/failed fire-and-forget Redis set can't shadow a newer local value and cause spurious If-Match conflicts. Versions are monotonic epoch-ms, so the larger is the later sync. * fix(collab-doc): make persist If-Match teardown-race-immune and recover missing version on final flush Close two last-leave concurrency holes Cursor flagged: - Thread the reconciled version LOCALLY through the persist retry loop. After a conflict+reconcile the correct next If-Match is exactly result.version, so carry it in a local var instead of re-deriving from room.syncedVersion/Redis. On a last-leave flush destroyRoomIfIdle removes the room from the map before the async flush finishes, so mergeMarkdownIntoRoom's recordVersion can no longer update room.syncedVersion — threading makes each retry's precondition correct by construction, immune to that dropped mutation and to a best-effort Redis re-read. - Cache the resolved version back into room.syncedVersion in currentVersion() so a peer-seeded/tail-only task (which never sets it locally) or a later transient Redis read failure still resolves it from the last value seen (monotonic max, never regresses). - On a FINAL flush, briefly retry resolving the If-Match when the version read momentarily fails, rather than deferring and stranding the session's edits in the TTL'd stream — the version is cluster-wide and heartbeat-refreshed. * fix(collab-doc): stamp cluster sync version the moment the seed wins, before the liveness guard The winning seeder set the If-Match token (room + Redis filedoc:syncver) only after the liveness/seeded guard that follows seedIfEmpty. But the tailer can integrate the just-appended seed DURING the seedIfEmpty await, so isDocSeeded(room.doc) is already true when the guard runs and it returns early — leaving the stream holding seed content with no cluster version. Later persists then send no If-Match, the app returns `deferred`, and session edits stay only in the TTL'd stream (the exact stranding this PR prevents elsewhere). Move the version stamp to immediately after seedIfEmpty wins, before the guard. Recording it only once our seed won (not from the fetch) is preserved, so it still can't shadow a peer's winning seed. * fix(collab-doc): make the synced-version token monotonic at every write site The If-Match token is written fire-and-forget from the seed stamp, merges, and persists, both locally and to Redis. An out-of-order write (e.g. a seed's lagged setSyncedVersion landing after a later merge's) could regress it below the version the live doc already incorporates, causing spurious If-Match conflicts — and on a last-leave flush with no live room to reconcile into, a spurious conflict leaves durable authoritative and drops the session's edits. - setSyncedVersion now writes via SET_VERSION_IF_NEWER_SCRIPT (Redis-side compare-and-set): it overwrites only when the new value is greater, refreshing the TTL either way. - recordVersion / the persisted branch / the seed stamp all take Math.max instead of assigning room.syncedVersion directly. Versions are monotonic epoch-ms, so "newer" is a plain numeric compare, exact within a Lua double. * fix(collab-doc): close three last-leave persist edge cases from review - Stale snapshot after reconcile (High): the multi-task captureState fell back to the pre-await localState snapshot even after a reconcile advanced ifMatch, so a failed stream re-read could persist the pre-reconcile state against the new version and clobber the out-of-band edit the reconcile just incorporated. NULL localState after a reconcile so a failed read aborts instead. - Lock miss aborts reconcile (Medium): a merge-lock acquisition failure returned 'no-live-room', indistinguishable from an absent stream, so flushPersist treated transient contention as terminal. Return a distinct 'merge-unavailable' and handle it as retry-later (edits stay in the stream), never as "nothing to reconcile into". - Peer syncver never recovers (Medium): the winner's setSyncedVersion was fire-and-forget with swallowed errors — the only way a peer-seeded task learns the durable version — so a dropped write left that peer deferring forever. Make it retry (bounded) like appendUpdate/seedIfEmpty; the monotonic script keeps a racing retry a no-op. * fix(collab-doc): scope the persist If-Match to a content version so metadata bumps can't clobber edits The optimistic-concurrency validator was `updatedAt`, which rename/move/delete/restore also bump with no content change. A racing live-doc persist then saw a stale token, got `conflict`, reconciled the pre-edit durable body via updateYFragment (incoming-wins on overlap), and wiped the user's in-flight edits. Scope the validator to content (RFC 7232 semantics — validate the representation, not the row): - New `workspace_files.content_updated_at` (NOT NULL, `now()` fast-default — no table rewrite). Advances ONLY on content writes (upload / overwrite / create); metadata writes never touch it. - The FOR UPDATE CAS, the merge-notify version, and the seed version all use `content_updated_at`. A rename now leaves it unchanged, so the persist If-Match still matches -> no spurious conflict, no reconcile, no lost edits. Genuine out-of-band content writes still conflict and reconcile. - Consolidated the collab schema into one migration (the collab-state table + the new column) per request, rather than a separate follow-up migration. Relay/store/contracts unchanged (still a numeric monotonic version). * chore(collab-doc): condense the densest persist comments (no behavior change) Cleanup pass: tighten the three longest comment blocks added while hardening the persist path (currentVersion cache, ifMatch threading, final-flush version retry) without dropping any invariant. No dead code found (biome lint clean; all new symbols referenced). * fix(collab-doc): persist must return the content version, not updatedAt Follow-up to the content-scoped If-Match: persistFileDoc still returned `updatedAt` as the version in both the persisted and conflict results, while the CAS/seed/merge all guard on `content_updated_at`. A content write sets both to the same instant, so it was coincidentally correct — until they diverge: if a metadata write bumps `updatedAt` past `content_updated_at`, the conflict path returned the larger `updatedAt`, so the relay's re-persist sent an If-Match the CAS (which checks `content_updated_at`) could never match → perpetual conflict → dropped reconciled edits. Return `contentUpdatedAt` in both paths so the relay's token always matches what it's checked against. * fix(collab-doc): defer persist whenever the version is missing; guard the content-version test - Empty-file CAS race (Medium): the unconditional-write carve-out for size===0 read `record.size` outside the write transaction, so a concurrent first content write could land after the check and be clobbered. With content_updated_at NOT NULL every existing file always has a real version, so a missing expectedVersion is always transient — always defer, never write unconditionally. Removes the TOCTOU hole. - Content-version test (Low): the merge-chokepoint test kept updatedAt == contentUpdatedAt, so it passed even if wired to the wrong field. Mock distinct values and assert contentUpdatedAt, so a regression to updatedAt now fails the test. * fix(collab-doc): don't reconcile a conflict the live doc already reflects (would wipe newer edits) flushPersist reconciled the durable body into the live doc on every conflict. But when the conflict comes from a racing self-persist (or an apply-edit the chokepoint already merged), the durable body is a STALE SUBSET of the live stream, and the incoming-wins updateYFragment merge moves the doc backward — wiping newer in-flight edits, which the retry then persists. Before reconciling, re-check the freshest synced version. If it already covers the conflict version, the live doc has already incorporated that content (or is ahead), so skip the reconcile and just retry with the freshest version as If-Match — the re-projection captures the current live stream, preserving every edit. Only a genuine out-of-band change the live doc hasn't incorporated (freshest < conflict version) is reconciled in. freshest never exceeds the durable version, so this can't loop. * fix(collab-doc): make content_updated_at monotonic per file; skip-reconcile can't loop The If-Match token was stamped with app-local new Date() on each content write, so cross-instance clock skew could stamp a later write with an EARLIER content_updated_at — breaking the version ordering the whole optimistic-concurrency scheme (and the skip-reconcile branch's freshest>=version assumption) depends on. Under skew the relay's monotonic syncedVersion could exceed the durable version, sticking the If-Match: persist conflicts forever, exhausts retries, drops the session's edits. - Stamp content_updated_at strictly after the current committed value (we hold the row's FOR UPDATE lock): new Date(max(now, currentFile.contentUpdatedAt + 1ms)). Monotonic per file regardless of clocks; also removes same-millisecond collisions. updatedAt stays plain wall-clock (display/sort). - Skip-reconcile branch retries with result.version (the durable value the CAS will match), never freshest (which could exceed it and loop). Belt-and-suspenders now that the version is monotonic. * refactor(collab-doc): drop the destructive in-persist reconcile; adopt-version-and-retry on conflict The in-persist reconcile projected the durable body back over the live doc via updateYFragment ("make the doc match"). That is destructive: when the live stream is already ahead — the common case, because the write chokepoint (mergeEditIntoLiveFileDoc) already merged the out-of-band change into the stream — it moved the doc backward and wiped newer in-flight edits. This produced a run of races (stale snapshot, wipe-newer-edits, version-lag skip miss) that a full-document reconcile fundamentally can't avoid, since deciding when it's safe relies on a laggy cross-task version token. Remove it. On conflict, adopt the durable version as the new If-Match and retry: captureState re-reads the current stream (which holds the out-of-band change AND the live edits), so the re-projection persists the converged result. The durable change reaches the live doc via the chokepoint, never here. Trade-off: the only unmerged out-of-band write is one whose chokepoint merge itself failed (rare, logged), which we accept over the frequent reconcile-wipes-edits race. - flushPersist: conflict -> ifMatch = result.version, retry (bounded). No applyMarkdownToLiveFileDoc. - conflict response drops `markdown` (contract + relay type + persist) — no body needed, saves a blob fetch. applyMarkdownToLiveFileDoc stays (still used by the apply-edit route / the chokepoint). * fix(collab-doc): don't let a last-leave conflict retry clobber via the stale local snapshot Regression from dropping the reconcile: on conflict the retry adopts result.version and re-reads captureState. But after single-pod last-leave teardown the room is already destroyed, so captureState falls back to the pre-teardown localState (which lacks the out-of-band change); the retry then CAS-passes and overwrites the committed external write — undoing the external-wins last-leave policy. Null localState on the first conflict, so the retry can only use freshly-read authoritative state (stream / live doc). When none is available (single-pod room gone, or a transient stream-read failure) captureState returns null and the retry stops, leaving durable content authoritative. Covers both the single-pod and multi-task-stream-unavailable variants of the stale-snapshot clobber. * fix(collab-doc): stop (don't re-persist) on a persist conflict — closes the commit-window clobber The conflict retry adopted the durable version and immediately re-persisted the current stream, assuming the stream already held the out-of-band change. But an external write commits durable BEFORE its chokepoint merge (mergeEditIntoLiveFileDoc) reaches the stream, so a persist landing in that window CAS-passed with a stream that still lacked the external content and clobbered the committed write — not just the rare merge-failed path, but a race on every external write, worst at last-leave flushes. Make persist a single attempt: on conflict, STOP and leave durable authoritative. The chokepoint merges the change into the stream and — only once it is actually there — advances the synced version via its own recordVersion; a later flush (debounced or final) then projects the converged stream with a matching token. The session's edits stay in the stream meanwhile. The conflict handler deliberately does NOT advance the synced version, or the next flush would clobber with a still-behind stream. Removes the retry loop and PERSIST_CONFLICT_RETRIES. * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run (#6094) * improvement(tables): fire the live-rows signal on async delete, run cancel, and column run These three table operations mutate row data but emitted no `rows` change signal, so open editors' grids stayed stale until a manual refresh (enrichment *results* already stream live via `cell` events; these are the bulk paths that don't emit per-cell events): - Async row delete (`runTableDelete`): signal as rows drop out (throttled with the existing progress event) and once more on completion — the `job` progress event only drives the delete meter, not the rows query. Covers the delete-async route and the copilot bulk-delete, since both share the runner. - Cancel runs (`cancel-runs` route): cancelling clears each affected row's exec state; the `dispatch: cancelled` events drop the run overlay but the client then renders authoritative DB state, so refetch. Only when something was actually cancelled. - Run column (`columns/run` route): starting a run bulk-clears the target group's cells to pending; refetch so the cleared cells show. Only when a dispatch was actually created. Guarded so no signal fires on a no-op/failure. Adds a delete-runner test asserting the completion signal. * fix(tables): guarantee the live-rows signal on every mutating path (review) - Delete runner (Greptile P1): a batch could commit and the job then cancel/supersede before the next throttled progress signal or `markJobReady`, bypassing both signals and leaving deleted rows on screen. Track `deletedAny` and fire the grid refetch in a `finally`, so it runs on EVERY exit — completion, cancel/supersede, mid-batch lock, or a rethrown error after a partial delete. - cancel-runs / columns/run routes (Cursor): the `cancelled > 0` / `if (dispatchId)` guards don't always reflect DB row changes — cancel tombstones exec state even when 0 dispatches were active, and a run bulk-clears cells then can return a null dispatchId. Signal unconditionally; a stale-but-harmless refetch beats a missed one. - Tests: assert the delete signal fires on the mid-run-cancel-after-delete path and NOT when nothing was deleted. * fix(tables): mark deletedAny before the page delete so a mid-page lock still refreshes the grid `deletePageByIds` commits in internal batches, so a delete lock landing mid-page can persist earlier batches and THEN throw TableLockedError — the catch returns without a count, so setting `deletedAny` from the return value missed it and the finally skipped the grid refetch. Set `deletedAny = true` before the call (any attempt may commit rows); an attempt that commits nothing only over-refetches (harmless). Adds a test asserting the signal fires when a page throws a mid-page lock. * fix(files): make embedded resource file view collaborative (#6095) * fix(files): make embedded resource file view collaborative The /chat resource panel rendered saved files through FileViewer without the collaborative opt-in, so a file open on the Files page and the same file open in the embedded panel never joined the same file-doc room — no live carets and no live content sync between the two surfaces. Pass collaborative on the EmbeddedFile FileViewer. Collaboration still self-gates on canEdit + non-streaming + workspace doc, so the agent token-stream preview (the dedicated streaming-file path, canEdit=false) is untouched. * fix(files): refcount file-doc room membership per shared socket Two collaborative surfaces in one tab (the Files editor and the embedded chat resource panel) share one Socket.IO connection, so both providers for the same file JOIN the same room over that socket. The server's LEAVE does socket.leave(name) with no membership refcount, so the first provider's destroy() would strand the second still-mounted one — no more live content or presence. Count live providers per file per socket (keyed by the stable Socket object, so it survives reconnects) and emit LEAVE only when the last provider for a file tears down. The single-provider path is unchanged (0->1->0). * feat(tables): propagate shared saved-view changes to collaborators live (#6100) Table views (named filter/sort/layout presets) are table-wide shared state — every reader sees every view — but view create/update/delete had no realtime signal, so a collaborator only saw another user's view changes on their own staleTime/focus refetch. Add a 'views' table event kind + signalTableViewsChanged, emitted from the views service (createTableView/updateTableView/deleteTableView, on real success only), and a client handler that invalidates the views query alone (no rows/definition refetch — a view is presentation state on the loaded table). Mirrors how row/schema/metadata changes already propagate. * test(tables): cover the views realtime signal (emit + on-success-only) (#6101) - events.test.ts: signalTableViewsChanged appends a single 'views' event carrying the tableId (through the real memory buffer). - views/service.test.ts: create/update/delete emit signalTableViewsChanged on real success, and DON'T on a no-op (a PATCH/DELETE targeting a missing view changes nothing, so it must not signal). Mirrors delete-runner's signal-path coverage; drives the DB via the shared dbChainMock. - Add tableViews to the comprehensive @sim/db/schema test mock so the service tests can queue the in-transaction existence row. * chore(ci): reconcile api-validation baselines after the staging merge The staging merge unioned realtime-rooms's own `as unknown as` cast (lib/collab-doc/converter.ts) with staging's zod-recursive-type cast (lib/api/contracts/tables.ts), so the non-test double-cast count is 9 — both casts pre-existed and were individually accepted on their branches. Also tighten rawJsonReads 6->5 to the true current count. Fixes the strict API contract boundary audit on realtime-rooms. * feat(copilot): stream file edits into the live collaborative Y.Doc (keep embedded view collaborative) (#6108) * feat(copilot): stream file edits into the live collaborative Y.Doc Copilot's file edits previously only reached the live doc once, at the final edit_content write, so a collaborative editor watching the file saw nothing until completion (streaming looked broken) and the client-side preview path was suppressed in collab mode. Make copilot a CRDT peer: as it streams append/update/patch content, merge the growing markdown into the file's live Y.Doc via the existing apply-edit path (a minimal updateYFragment diff, concurrent-edit-safe), throttled to ~250ms. version is omitted for these intermediate merges — they advance the live doc for viewers but are not durable checkpoints; the final edit_content write carries the real contentUpdatedAt and reconciles the durable file. Per the relay's persist gating, server-internal merges never schedule a persist, so a copilot-only stream produces zero intermediate file writes. - notify.ts: mergeEditIntoLiveFileDoc version is now optional (streaming omits it). - file-preview-adapter.ts: throttled live-doc merge at the edit_content stream hook. * fix(copilot): order + gate streaming live-doc merges; fast collab first render Harden the streaming merge (adversarial review): - Order + bound: dispatch through a per-file in-flight guard (drop-while-in-flight) so a stale out-of-order snapshot can never land after a newer one and regress the doc, and relay load is capped at one request per file regardless of rate. - No wipe: gate append/patch on the base file content having loaded — a base-less snapshot would diff to a delete-everything wipe of the seeded doc; update streams a full rewrite from scratch and needs no base. - Markdown-only gate: non-markdown files have no collaborative room, so skip the wasted relay round-trip. Fast collab first render (Issue 2): render the already-fetched markdown read-only via generateHTML while the collaborative doc seeds, with the editor mounted-but- hidden in the same layout box for a seamless swap on collabReady. Pure HTML — it never touches the Y.Doc (client seeding duplicates the doc), and generateHTML escapes text (raw-HTML snippets render escaped), so no XSS. * test(copilot): cover streaming file edits into the live collaborative Y.Doc Drives edit_content args_delta stream events through processFilePreviewStreamEvent and asserts the live-doc merge: fires with the growing FULL previewText and no version arg; is throttled (~250ms per file); is skipped for non-markdown files and for a base-less append (the delete-everything wipe guard); and runs at most one-in-flight per file. Verified to fail if any gate/guard is removed. * fix(collab-doc): coordinate live-doc merge ordering in one place; close durable-clobber race The second review found a residual: the durable edit_content write went through a different path than the adapter's in-flight guard, so a late straggler streaming merge could land after it and, via a persist, clobber the durable file's tail. Move the per-file coordination into mergeEditIntoLiveFileDoc (the one place both the streaming and durable paths call): a streaming (versionless) merge is dropped while one is in flight for the file; a durable (versioned) write instead WAITS for the in-flight streaming merge, so the final content is always the last merge applied and can't be regressed by a straggler. Simplifies the adapter (drops its Set + helper). Relocate the one-in-flight test to notify.test.ts (streaming-drops-while-busy + durable-waits-then-applies-last); the adapter test keeps throttle/gates/previewText. * fix(copilot): address review — order merges, exclude update, gate throttle, unhide stream Review round on #6108: - Greptile P1 (durable merges lose ordering): serialize ALL merges per file on one chain in mergeEditIntoLiveFileDoc (each chains after the current tail), so concurrent durable writes can't resume-and-fire out of order. notify now exposes isLiveDocMergeInFlight. - Cursor High (update stream blanks the doc): only append/patch stream — they build on the loaded base; update is a from-scratch rewrite whose partial snapshot would diff the full doc toward a fragment, so it applies atomically at the durable write. - Cursor Medium (throttle advances on a dropped merge): the adapter gates on !isLiveDocMergeInFlight, so the send throttle advances only on an actual dispatch — no lag, no backlog behind a slow relay. - Cursor Medium (placeholder hides a live stream): show the fast-render placeholder only when not streaming, so a stream that starts before the doc seeds shows through the editor. - Soften merge.ts/notify.ts comments per the lifecycle audit: only UNTOUCHED regions are preserved; a region the merge rewrites reconciles toward copilot's content. Tests updated: notify covers chain ordering + isLiveDocMergeInFlight; adapter covers append streaming, throttle, non-markdown/base-less/update skips, and the in-flight skip. * fix(collab-doc): reject stale durable merges at the relay (cross-process ordering) The in-process merge chain only orders merges within one apps/sim process. Two durable writes for the same file on DIFFERENT processes could reach the relay out of dispatch order; the relay recorded the version monotonically but still APPLIED the older markdown, regressing the live doc while the token stayed high (a later persist could then write the stale content back over the durable file). Enforce ordering at the relay — the single cross-process coordination point — using the existing Redis primitives: under the per-file Redis merge lock, read the cluster-wide synced version and SKIP a versioned merge that is not newer (a newer durable write already landed). Make recordVersion await setSyncedVersion so it is durable before the lock releases, so the next holder's staleness check reads a consistent value. Streaming (versionless) merges are unaffected — they carry no durable version and are ordered per-process by the caller. Adds a relay test asserting a stale/idempotent versioned merge returns 'stale' and never computes or publishes a diff. * fix(copilot): match durable path — detect markdown by MIME type + name at the stream gate The streaming gate checked isMarkdownFile with only the filename, while the durable merge uses type + name — so a text/markdown file without a .md extension was skipped mid-stream (it self-corrected at the durable write). Pass editIntent.contentType so streaming detects the same set of markdown files as the durable path. * test(copilot): assert throttle follow-through after an in-flight merge clears * fix(collab-doc): order streaming merges by streamedAt so a late snapshot can't regress a newer durable write * refactor(collab-doc): tidy merge-order docs + relay order object; cover multi-replica streaming stale-check * fix(collab-doc): order streaming merges by causal base version, not wall-clock A streaming snapshot now carries baseVersion (the durable contentUpdatedAt it was built from) instead of a wall-clock streamedAt. The relay drops the snapshot when a newer durable write landed since that base, so a concurrent human save can no longer be clobbered in the live doc and then persisted over the durable file. Skew-immune: both keys are DB-monotonic contentUpdatedAt values. * fix(collab-doc): derive streaming baseVersion as contentUpdatedAt ?? updatedAt Match the version line the seed/persist use so a legacy file with no content version still ships an ordered streaming snapshot instead of an unordered one. * fix(collab-doc): fail-closed on a streaming snapshot with no baseVersion The live-merge gate now requires a numeric baseVersion, not just loaded base content. A rare base with no file record (hence no version) would otherwise ship an unordered snapshot the relay can't stale-check, risking a clobber of a concurrent durable write. Skip the live merge instead; the durable write reconciles. * docs(collab-doc): document the accepted concurrent-independent-streams limitation * chore(ci): reconcile api-validation baseline after the staging merge The merge commit auto-merged the baseline at 1000; bump totalRoutes/zodRoutes to 1003 for staging's three new contract-bound routes (nonZodRoutes still 0). * test(files): update storage-accounting assertion to the mergeEditIntoLiveFileDoc options object * fix(collab-doc): trace the full yjs/tiptap external stack into the file-doc route bundles The seed/merge/persist internal routes run the collab-doc converter (markdown <-> Yjs via headless TipTap) server-side. Those deps are serverExternalPackages, and the standalone tracer only force-included jsdom — it does NOT follow yjs's ESM subpath imports of lib0 (lib0/logging, ...), so Docker/standalone builds shipped node_modules without them and the seed route 500'd (Cannot find module 'lib0/logging'). That left every collaborative document unseeded and permanently read-only on deployed envs. Force yjs, lib0, y-protocols, and @tiptap into the trace for all three routes. * fix(collab-doc): copy the full yjs/lib0 stack into the app image The seed/merge/persist routes run the converter (markdown <-> Yjs) server-side. yjs is a serverExternalPackage and the Next standalone tracer copies lib0 only partially — it drops the ESM subpath file lib0/logging.js that yjs.mjs imports via lib0's exports map, so the seed 500s ('Cannot find module lib0/logging') and every collaborative doc is stuck read-only. Verified in the running dev container: /app/node_modules/lib0 had 37/38 files, logging.js missing. outputFileTracingIncludes can't fix it — its globs resolve against apps/sim, but these deps hoist to the monorepo-root node_modules, so the glob matches nothing (my prior next.config attempt was a no-op; reverted). Instead COPY the complete lib0/yjs/y-protocols from the deps stage in the runner, overwriting the partial trace — the same pattern already used for isolated-vm. * feat(files): stream copilot edits into the collaborative doc smoothly (#6122) * feat(files): stream copilot edits into the collaborative doc smoothly - apply the agent stream client-side into the live Yjs binding as minimal updateYFragment diffs (like main's setContent, but incremental) so it renders smoothly AND broadcasts to every peer via CRDT — a collaborator on /files sees the stream for free - gate the apply on collabReady so diffs never land on an unseeded doc; keep the read-only placeholder visible until the seed swaps in - run streamed ops under a dedicated tx origin so they stay out of the user's undo stack - delete the throttled server-side streaming merge and the baseVersion ordering machinery it needed (relay + notify + session contract); the durable final write still reconciles open editors and seeds late joiners * fix(files): apply agent stream as a true CRDT peer + guard base-less snapshots Review round 1 (Greptile P1s): - apply the stream against a private shadow replica (seeded from the live doc at stream start) and relay only the agent's own delta into the shared doc, so a concurrent peer edit to a region the agent snapshot didn't include is no longer reverted (previously the whole-body reconcile deleted it) - gate append snapshots on "must extend the base": a base-less append fragment (emitted before the base loads) can no longer reconcile the seeded doc to a wipe; patch still legitimately replaces a mid-region - gate the apply on collabReady so diffs never land on an unseeded doc; keep the placeholder visible until the seed swaps in - plumb streamOperation through the preview surfaces to drive the append gate - add a peer-edit-preservation test (fails under whole-body reconcile) and refresh the undo-isolation + broadcast tests for the session API * fix(files): destroy the agent shadow deterministically on settle Cursor round 1 (Low): endAgentStream ran inside runOffRender, whose microtask is dropped when a rapid follow-up stream bumps the run token — leaking the shadow Y.Doc. Split it out into an unguarded microtask queued after the (droppable) final apply, so the shadow is always destroyed. * fix(files): agent stream frames skip the relay's durable persist Cursor round 1 (High): client-applied stream frames broadcast over the sync channel, so the relay stamped a socket origin and ran schedulePersist — durably writing partial agent content mid-stream, attributed to the watching user (the old server-merge applied with no origin and never did). Restore that behavior: - new FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST wire tag; the provider tags AGENT_STREAM_ORIGIN updates with it (normal user edits stay SYNC) - the relay applies it under an AgentSyncOrigin (carries the socket id for broadcast exclusion, but is not a plain string) so originSocketId() is null → no edited/schedulePersist/lastEditorUserId; excludeSocketId() still excludes the sender, and the update still publishes to the stream so peers converge - the copilot's final edit_content write remains the authoritative durable persist - tests: relay applies+fans-out but never persists a SYNC_NO_PERSIST frame (verified it fails if applied as a socket edit); provider tags agent edits * fix(files): open the stream shadow at start + private extend baseline Cursor round 2: - High (settle skips apply without session): the stream shadow is now opened on the first ready frame, BEFORE the extend gate — so an `update` rewrite (whose every frame is gated out until settle) and a stream that finishes before seed still get a session, and settle applies the final body via the reused-or-on-demand shadow instead of leaving the doc stale until the durable reconcile. - Medium (peer edits stall the stream): the extend gate now reads a private `lastStreamedBodyRef` (the agent's own last frame), snapshotted at stream start, not `lastSyncedBodyRef` which `onUpdate` clobbers on peer edits — so a collaborator typing can't make the growing snapshot stop prefixing the shown body and freeze it. - Medium (multi-replica over-persist): pre-existing, documented "safe over-persist" (a peer task tails the frame as REDIS_ORIGIN and marks edited) — refreshed the stale comment to describe the SYNC_NO_PERSIST source; copilot's edit_content write remains the authoritative durable persist. * fix(files): fail-close base-less previews + operation-based stream hold Cursor/Greptile round 3 (High + Medium) — remove the fragile string-prefix "extend gate", which was the root of both findings: - Server: `buildFilePreviewText` now fails closed for an `append` whose base content hasn't loaded (returns undefined, like patch/update), so a base-less fragment never reaches the client. This eliminates the base-less wipe at settle (Greptile P1) at the source; an empty file (existingContent === '') still previews normally. - Client: the collab streaming tick no longer string-prefixes the raw preview against the editor's canonical markdown (the '*' vs '-' / emphasis mismatch that froze every append frame — Cursor). The mid-stream hold is now purely operation-based: `update` waits for settle; append/patch/create apply each frame via the (peer-safe) shadow reconcile. lastStreamedBodyRef is now a plain dedup guard, not a prefix baseline. Keeps the shadow, durable write, and SYNC_NO_PERSIST unchanged. * fix(files): elect a single agent-stream writer across tabs Cursor round 4 (High): with the stream applied client-side, two tabs/windows on the same chat could each derive streamingContent (the reconnect/resume path re-consumes preview events) and each independently insert the stream under a different Yjs clientID, duplicating content until the durable reconcile. Fix — single-writer election via the file-doc awareness (new agent-stream-leader): - a client applying an agent stream announces `agentApplying` on its own awareness - only the leader (min clientID among announcers) applies mid-stream AND at settle; a non-leader renders the leader's ops via Yjs and does not apply (a non-leader applying the final body would re-insert the whole doc as a duplicate) - re-checked each frame, so it converges to one writer the moment awareness propagates; the sub-frame startup race is reconciled by the durable write - single-client (the common case) is unaffected: it is the only announcer, so it always leads * fix(files): gate the settle apply locally, not on a settle-time re-election Cursor round 5 (High): the settle recomputed leadership from live awareness and the leader cleared its announcement immediately, so a straggler peer that settled afterward became the sole announcer, self-elected, and applied finalBody through its base-seeded shadow — re-inserting the whole doc as a duplicate. Fix: gate the settle apply on a LOCAL didApplyStreamRef (set only when this client actually applied a mid-stream frame — i.e. it was the mid-stream leader whose shadow is up to date), not on a settle-time re-election. A client that never applied (non-leader, a held `update`, or a pre-seed stream) skips the final apply and converges via Yjs + the durable write. The mid-stream leader election (isAgentStreamLeader) is unchanged, so exactly one client's didApplyStreamRef is ever true. * fix(files): open the agent-stream shadow lazily on lead (no stale handoff) Greptile round 6 (P1): the leader race — (a) a mid-stream leadership handoff could apply from a stale pre-stream shadow, and (b) two tabs starting the same stream before awareness converges could both lead briefly. - (a) fixed: the shadow is now opened LAZILY in the tick, only when this client actually leads, seeded from the CURRENT doc — so a handoff successor diffs against the prior leader's ops (never a stale base) and a non-leader builds no shadow at all. Announce candidacy via a dedicated ref (decoupled from the shadow); settle still gates the final apply on didApplyStreamRef (leader-only). - (b) the pure startup race is inherent to eventually-consistent election. It is now the only residual: bounded to two tabs starting the SAME stream within the awareness-propagation window, transient (converges in a frame or two), and never persisted (SYNC_NO_PERSIST + the durable edit_content reconcile). Resumes are sequential, so the common multi-tab case elects cleanly. Documented inline; a server-granted lease would close it fully but at a round-trip cost on the common single-tab path, which isn't worth it. * fix(files): idempotent settle apply (update lands client-side; no straggler dup) Cursor round 6 (Medium): a lone client's `update` never applied client-side — held mid-stream, then skipped by the didApplyStreamRef settle gate — so the rewrite depended entirely on the durable merge (stale if delayed/failed). Root cause was over-correcting round 5. Now that the shadow is opened lazily in the tick (current-seeded), the round-5 base-shadow duplication is already gone, so didApplyStreamRef is unnecessary. Replaced it: settle applies the final body via `agentStreamSessionRef.current ?? beginAgentStream(editor)` — the leader reuses its up-to-date shadow (last throttled frame), while a client that never applied (non-leader, held `update`, pre-seed) opens a FRESH current-seeded shadow. Reconciling current->final is idempotent: a straggler that settles after another wrote the final reconciles to a noop. So a lone `update` applies at settle (no wait on the merge), and there's still no settle-time election or base-shadow dup. * fix(files): broadcast agent frames to the whole room (same-socket siblings) Cursor round 7 (Medium): SYNC_NO_PERSIST frames applied under an origin carrying the sender socket id, and excludeSocketId dropped that whole socket from the relay fan-out. A second FileDocProvider on the same socket (chat preview + Files editor) then missed all mid-stream ops and stayed stale until the durable reconcile — a regression from the old no-origin server merge, which reached both. Fix: the agent origin is now a plain AGENT_SYNC_ORIGIN symbol, and agent frames broadcast to the WHOLE room (no socket excluded), matching the old behavior — so a same-socket sibling provider stays live; the emitting provider no-ops on its own echo (the ops are already applied locally). originSocketId still returns null for the symbol, so it keeps skipping edited/schedulePersist. Removed excludeSocketId and the socket-carrying origin object. Updated the relay test to assert the whole-room broadcast (verified it fails if the sender is excluded). * fix(files): tag agent stream frames no-persist across replicas A peer task tailing an agent-streamed preview frame previously applied it as REDIS_ORIGIN, marking the seeded room edited and making a transient startup-race duplicate eligible for that task's last-disconnect flush. Mark agent frames with a stream field so peers apply them as REDIS_AGENT_ORIGIN, excluded from the edited/persist gate. The copilot's durable edit_content write stays the sole authority over file bytes. * fix(files): reseed agent shadow on lead regain + agent-only compaction Two multi-writer edge cases surfaced in review: - rich-markdown-editor: a client that led, lost leadership, then regained it reused its stale shadow (which never saw the interim leader's ops), re-emitting ops for content already present. Tear the shadow down when a client observes it is not the leader, so a regain rebuilds fresh from the current doc. - file-doc-store: compaction always stamped its snapshot REDIS_SNAPSHOT_ORIGIN (marks peers edited). A long agent-only stream crossing the threshold could fold preview content into a persist-eligible snapshot. Track whether a room integrated any real edit and stamp an agent-only snapshot REDIS_AGENT_ORIGIN so it stays no-persist. Both covered by falsification-verified tests. * fix(files): close realEdited data-loss race + elect a settle writer Independent audit surfaced two real gaps: - file-doc-store: realEdited was latched AFTER appendUpdate's awaits, but the edit already sits in room.doc synchronously. A concurrent agent-frame compaction could read realEdited=false, snapshot that real content, and stamp it a no-persist agent frame — a lost edit. Latch it synchronously (same tick as the doc mutation) before any await. Deterministic falsifiable test added. - rich-markdown-editor: at settle every tab applied the final body, and a non-leader's local microtask runs before the leader's final propagates, so both insert the tail (Yjs keeps both) -> duplicated tail. Elect a single settle writer (reliable — awareness is long converged by settle), reading leadership before clearing the announcement. Corrects the overclaiming idempotency comment and the handoff pick-up comment. Adds a y-tiptap internals upgrade-guardrail test. * fix(files): own presence per client id, not one-per-socket The shared workspace socket hosts one collaborative provider per mounted view, so the chat file preview and the standalone Files editor for the same file each bind their own Yjs client id over ONE socket. The relay owned a single client id per socket, so the later JOIN overwrote the earlier and dropped its awareness — which silently broke the single-writer agent-stream election (a peer stopped seeing the streaming provider's announcement and could self-elect, duplicating streamed text for the whole stream). Track ownership per (socket, client id): a socket owns a set of client ids; the awareness gate accepts a frame only if every id it carries is owned; cleanup drops all of a socket's ids; the roster stays one-entry-per-session. Reclaim and the same-user reconnect path evict just the reclaimed id, dropping the old socket only if it empties. Falsification-verified test added. * fix(files): make streamed file-preview accumulation replay-safe Guard deriveFilePreviewSession against re-delivered/replayed content events: apply a delta/snapshot only when previewVersion strictly advances, so a client re-render or stream replay can't double-append the tail (the duplicated-content bug) or regress on an older snapshot. * fix(files): fix new-file collab streaming latch and agent-edit duplication - Latch collab readiness so a new file's post-seed `synced` flap can no longer re-gate agent streaming (the stream previously showed only the seed and the rest appeared only on reload) - Relay defers the durable edit_content merge to an actively-streaming client: the client shadow stream and the server merge were both writing the same content into the live doc, duplicating it when the server ran ahead - Render the collaborator caret bar out of flow so a peer caret never nudges the surrounding text by ~1px - Remove dead code: unused FileDocMessageType alias, unnecessary LiveFileDocMergeOrder export Covered by tests: readiness latch (flap/offline/latch cases), relay merge deferral (single- and multi-replica), plus verified-failing guards. * test(copilot): fix loadWorkspaceFileTextForPreview mock to return { text } not a bare string The adapter reads previewBase.text to seed an append/patch base; the mock returned a bare '' so previewBase.text was undefined, making a base-less append fail closed (no file_preview_content). My PR's fail-close change exposed the wrong-shaped mock. --------- Co-authored-by: mzxchandra <129460234+mzxchandra@users.noreply.github.com>	2026-07-31 18:48:12 -07:00
Vikhyath MondretiandClaude	7798e83489	feat(function): custom sandboxes (#6071) ... * feat(sandboxes): workspace dependency sets for Function blocks Named package sets a Function block can import from. The server canonicalizes and hashes the list; E2B prebuilds a content-addressed template per set, Daytona installs per execution. Create/edit is gated to Max or Enterprise via the shared workspace entitlement check; execution is deliberately ungated, so a downgraded workspace keeps running what it already built. Also on this branch: - Extract the duplicated dropdown/combobox option-fetch lifecycle into use-fetched-options. Only combobox had the dependency-change reset, so every dropdown with dependsOn + fetchOptions cleared its list and never repopulated until reopened. - Collapse the repeated Max-tier entitlement check onto one hasMaxTierWorkspaceAccess, shared by inbox, live sync, and sandboxes. - Resolve a personal payer's block state through getEffectiveBillingStatus in getBillingEntityBlockStatus, so the client-side Max gates agree with the server-side ones when blockOrgMembers' fan-out is stale. - Carve the Daytona dependency install out of the caller's execution budget instead of stacking on top of it. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0273 Staging claimed 0271 and 0272 while this branch was out, so the hand-authored 0271_workspace_sandboxes was dropped before the merge and regenerated on top of the merged schema. Same DDL; drizzle emits plain CREATE TABLE/INDEX rather than the hand-added IF NOT EXISTS, which matches the repo default — that idempotent form is only needed for files with CONCURRENTLY ops below an embedded COMMIT. Regenerating also restores the meta snapshot the hand-authored migration never had. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging independently claims idx 0273, so remove ours before merging to avoid an add/add conflict on the drizzle migration index. Regenerated at the next free index once the merge lands. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0275 Staging took 0273 and 0274, so the sandboxes DDL lands at the next free index. The emitted SQL is byte-identical to the dropped 0273. Co-Authored-By: Claude <noreply@anthropic.com> * fix(billing): consolidate the Max-tier entitlement onto one predicate The Max tier was spelled five ways. The odd one out — `isMax`, defined as `isPro(plan) && credits >= 25000` — excluded both `team_25000` and `enterprise`, and it was the sole input to the personal-workspace cap. A delinquent Max-for-Teams org admin got 1 personal workspace while a delinquent Max individual got 10. Only free/pro_6000/pro_25000 were tested, so the two broken tiers were unpinned. Separately, the server gate and the client `hasUsableMaxAccess` were independent copies of the same rule. The settings sidebar renders Sandboxes and Sim Mailer from the client one while the API answers 403 from the server one, so any drift renders a feature unlocked that the API refuses. - `MAX_TIER_CREDITS` is derived from the `CREDIT_TIERS` table; `isMaxTier` in plan-helpers is now the single definition, shared by the server gates, the client derivation, `getPlanTypeForLimits`, `plan-view`, and the cap - `hasWorkspaceTierAccess(id, predicate, { intent, onMissingWorkspace })` becomes the one org-vs-personal payer fork. `intent: 'active-use'` means active and not billing-blocked; `'retention'` means active/past_due with block state ignored, so the inbox teardown guard keeps its fail-open semantics instead of implying them through a duplicated fork - `isWorkspaceOnEnterprisePlan`'s personal branch now applies the status and block checks its own org branch always had, and its TSDoc names its real consumer (copilot BYOK, not Access Control) - the client live-sync gate gained the server's `isHosted` branch, so a self-hosted deploy with billing on no longer locks an interval the API accepts. It reads both flags directly rather than taking one as a parameter the callers sourced from the same module - `sqlIsPro`/`sqlIsTeam` escape the `_` LIKE wildcard, matching the already correct hand-rolled filter in seat-drift - deletes the `TERMINAL_SUBSCRIPTION_STATUSES` and `ENTITLED_STATUSES` shadow constants, and corrects three test mocks that asserted `trialing` was entitled or usable `max-tier-parity.test.ts` asserts the client and server answers match for every plan name. Both new guards were checked against the old code: the parity test fails 3 assertions with the previous predicate, and the self-hosted test fails without the `isHosted` branch. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop the sandboxes migration ahead of the staging merge Staging has claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables) since the last merge, so our 0275_workspace_sandboxes collides on the index. Dropping ours first — the .sql, meta/0275_snapshot.json, and the journal entry — leaves packages/db/migrations byte-identical to the merge-base, so the merge sees no add/add conflict at all. Regenerated on the far side. Ours is the droppable side: plain additive DDL with no hand edits, which drizzle reproduces exactly. Staging's migrations are hand-written and must survive. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): regenerate the sandboxes migration as 0277 Staging claimed 0275 (table_views) and 0276 (drop_legacy_folder_tables), so the sandboxes migration dropped before the merge comes back on top as 0277. The emitted SQL is byte-identical to what was dropped — the original had no hand edits, so there is nothing to reapply. It is purely additive: two enums, sandbox_image and workspace_sandbox, their two FKs and six indexes. That it regenerated unchanged also confirms the schema.ts auto-merge was correct — had it lost staging's legacy-folder-table drops, drizzle would have emitted CREATE TABLE for them here. Snapshot chain is continuous (0273 -> 0277, each prevId matching the previous id) and the table counts track the DDL: 100 -> 101 (table_views) -> 99 (legacy folder tables dropped) -> 101 (the two sandbox tables). Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): gate on the enterprise feature flags, drop the rollout switch Sandboxes shipped behind `custom-sandboxes`, an AppConfig rollout flag falling back to a `CUSTOM_SANDBOXES` secret. That made it the only Max-gated surface with no self-hosted path: `INBOX_ENABLED` can force Sim Mailer on for an operator running their own billing, and `ENTERPRISE_ENABLED` turns on the other nine features at once, but neither reached sandboxes. A self-hoster had to find a separately-named variable that was not part of that family, and one running with billing enabled could not enable it at all. Sandboxes now joins the enterprise feature set and the rollout flag is gone: - `sandboxes` is an `EnterpriseFeature` with `SANDBOXES_ENABLED` and its `NEXT_PUBLIC_` twin, so the master switch and the per-feature override both reach it like every sibling - `hasWorkspaceSandboxAccess` takes the inbox's shape exactly — the override wins, then a deployment without billing is unrestricted, then the workspace payer needs usable Max or Enterprise - the settings nav gains `selfHostedOverride`, so the section resolves through the same path as Sim Mailer instead of a second entitlement AND-ed in - `custom-sandboxes`, the `CUSTOM_SANDBOXES` secret, the now-unreachable `SANDBOXES_UNAVAILABLE` 403 copy, and the route's kill-switch branch are deleted Its legacy default is `true`, matching `inbox`: the gate already returns true whenever billing is off, so `false` would leave the nav override disagreeing with the gate that answers the request. Self-hosted builds run on the operator's own E2B/Daytona credentials, so there is no Sim-side cost to withhold — the docs now say so, since enabling the feature without a provider configured is the obvious trap. The new gate tests run with billing enabled on purpose; the `!isBillingEnabled` bail would otherwise answer every case and hide whether the override is wired. Verified by deleting the override line — exactly the one assertion fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let the language menu match its trigger width `matchTriggerWidth={false}` exists for the opposite case — a narrow trigger whose option labels would truncate, letting the menu grow past it. The language field is a full-width form control with two short labels, so the override shrank the menu to "JavaScript" and pinned it to the right edge instead. The default (`true`) is correct here. Every other consumer passing `false` is a genuinely narrow trigger — a role picker in a member row, a table filter chip. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): re-queue a build when resolution finds the image unusable `ensureSandboxImage` only ran when a sandbox was saved, so resolution treated an unusable image as terminal and told the user to go fix a definition that was never wrong. Three states stuck permanently until someone re-saved in Settings: - a build that failed - a build whose worker died mid-flight, stranding the row in `building` - every sandbox created while the deployment ran a `runtime` provider, after a switch to a `prebuilt` one — `runtime` writes no image rows at all, so the whole fleet resolved to "no completed build" with nothing to repair it Resolution now re-queues through the registry's existing idempotent entry point before failing, and says a build is on its way instead of pointing at Settings. The conflict guard already claims only a `failed` row or a stale `pending`/ `building` one, so executions arriving during a healthy build enqueue nothing — no thundering herd from a hot workflow. The registry is imported dynamically for the same reason `sandboxDb` is: it pulls `@sim/db` into the static graph, which this module keeps out of the executor bundle. That also avoids a cycle, since the registry imports `invalidateSandboxResolution` from here. A repair that itself fails is logged and swallowed — it must never replace the build error naming the sandbox. Verified by deleting the repair call: exactly the three new assertions fail. Co-Authored-By: Claude <noreply@anthropic.com> * improvement(sandboxes): let the picker show just the sandbox name The label read "Test · Python · 1 package". The block's own list is already scoped to the language its sibling `language` subblock selects, so the language repeated on every row said nothing, and the package count is decoration next to the name that identifies the sandbox. The language stays for the one caller that cannot filter — agent tool-input renders this field under a synthetic id where the sibling `language` value is unreachable, so its list spans both languages and the name alone is ambiguous. That is the same missing value which disables filtering, so `showLanguage` is derived from it directly rather than passed independently and left to drift. A failed build is still marked: that suffix is the difference between a selection that runs and one that does not. Passing the flag also means dropping `.map(toSandboxOption)` for an explicit arrow — `Array.map` hands the index to the second parameter. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): show the sandbox name on the block card, not its uuid The card printed "443f4934-26ab-44ab-8...". `resolveDropdownLabel` only reads a subblock's static `options` array, and the sandbox picker is a `combobox` whose options load asynchronously, so its array is empty and the raw stored id fell through to the label. Resolved the same way skills and tools already are: a `resolveSandboxLabel` in the display layer, fed from the shared sandbox list query — the same cache entry the picker reads, so this adds no request. Two deliberate scopings: - the query is subscribed only for the sandbox row. `SubBlockRow` is memoized per subblock, and the list query polls while a build is in flight, so an unconditional hook would re-render every row on the canvas on each poll tick - the resolver matches the field id, not just the type. There is no dedicated subblock type for it, and matching `combobox` alone would relabel unrelated pickers An id with no matching sandbox resolves to null rather than a guess, so a deleted sandbox falls through to the caller's placeholder. The template preview surface is left alone: it is explicitly hook-free and passes empty lists for tools and skills too. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): hide the Sandboxes section with no provider configured Entitlement decides whether a workspace may author sandboxes; nothing decided whether anything could run one. A self-hosted deployment with SANDBOXES_ENABLED but no E2B or Daytona credentials got a fully functional tab whose output no Function block could select — the picker is gated on the provider vars, the tab was not. Both navigation planes now drop the section when neither NEXT_PUBLIC_SANDBOX_ENABLED nor the pre-Daytona NEXT_PUBLIC_E2B_ENABLED is set — the same pair the picker's `showWhenEnvSet` reads, so the two cannot disagree. Dropped rather than locked: an upgrade does not conjure a provider. The unified plane drops it in `buildUnifiedSettingsNavigation` rather than in the sidebar's filter, because the sidebar's `selfHostedOverride` short-circuit runs before its `requiresMax` check and would have revealed the tab anyway. It reads the browser twins, not the server's `isRemoteSandboxEnabled`, since this module renders on both sides. The predicate is a function, not a module constant, because the constant form was untestable and ambient: the env mock falls through to `process.env`, and `apps/sim/.env` (gitignored, so absent on CI) sets NEXT_PUBLIC_E2B_ENABLED=true. The nav tests passed locally and failed 6 assertions with the flag cleared. They now pin both flags, so the suite is identical with and without a local env file — verified by running it both ways. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct three claims the code no longer makes The Sandboxes section described behavior two commits on this branch changed, and led with an internal detail no reader needs. - entitlement is no longer Max/Enterprise only: self-hosted deployments unlock sandboxes with SANDBOXES_ENABLED, and the section is hidden outright when a deployment has no sandbox provider, which is the state a self-hoster is most likely to hit and least likely to diagnose - a build that is not Ready is no longer terminal. It is queued again on the next run, so the advice is to wait and re-run, not to go edit a package list that was never wrong - deleting a sandbox frees its build once nothing else references it. Builds are shared by content, so this is the one place a reader could reasonably assume deletion is immediate Dropped the `ModuleNotFoundError` aside: what the old code did instead is not something a reader needs to know to use the feature. The page is hand-written — `function` has category 'blocks' and is absent from `NATIVE_RESOURCE_BLOCK_TYPES`, so generate-docs skips it and these edits will not be overwritten. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): release the provider image when nothing references it Deleting a sandbox only removed its row, leaving the built template in E2B until the 30-day retention sweep — up to a month of paying to store an image nothing could select. Editing a package list had the same effect on the old content address, which is the more common case since every edit re-points the sandbox. `releaseSandboxImage(specHash)` now deletes the provider image and its row from both paths. It reuses the sweep's provider call and its ordering: image first, row second, so a refused delete leaves the row for the sweep to retry rather than orphaning a remote template nothing points at. Two guards make eager deletion safe: - builds are keyed by content, not by workspace, so two workspaces declaring the same package list share one image. The release no-ops while any sandbox still references the hash — otherwise one workspace's delete would break the other's - an in-flight build is left alone rather than raced; the sweep collects it once it settles Called detached from both routes. The row is already committed by then, so the user's action has succeeded whatever the provider says, and awaiting would hold a UI delete open on a remote call the sweep would retry anyway. Every failure inside is logged and swallowed for the same reason. E2B's delete verified against their API reference: DELETE /templates/{templateID} with X-API-Key, 204 on success. The existing implementation already matched, so this commit only adds the call sites and the guards. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rate-limit the automatic rebuild, drop the one-off status dot Two follow-ups to the resolution repair. The repair had no rate limit. `ensureSandboxImage` re-claims a `failed` row on sight, and a bad package name fails in seconds, so the in-flight guard never closed the window: a workflow on a one-minute schedule would enqueue a build a minute against a package list that will never resolve, each one real provider build compute. Before the repair existed resolution simply threw, so this was introduced with it. The two callers want different things, so the cooldown is opt-in. A save is a person explicitly asking for another attempt and still retries immediately; resolution passes `FAILED_BUILD_RETRY_COOLDOWN_MS` and gets at most one attempt per window no matter how often the workflow runs. Ten minutes: long enough that per-minute runs cannot drive per-minute builds, short enough that a transient registry outage clears within the hour. The status line loses its colour dot. `size-[6px] rounded-full` appeared in exactly one file in the repo, so it was a new primitive rather than a pattern, and it duplicated state the text colour already carries — the label now turns `--text-error` on a failed build, which is what every other status row in settings does. `ChipTag` was the wrong home for this: its variants are `mono`/`invite`, with no semantic tone, so a status version would have meant overriding its chrome from the consumer. Also corrects the docs line this changes: a failed build is retried periodically, and saving is the way to retry now, so "wait a moment and run again" no longer describes it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): claim the image row and its reference check in one statement Greptile P1. Reading references in one statement and deleting in another left a window — a wide one, since a provider delete is a network call — where a second workspace could declare the same package list, inherit the `ready` row, and have its next run fail against a template already on its way out. Content addressing is what makes that reachable: the image is shared, so one workspace's delete can strand another's sandbox. The reference check now lives in the conditional DELETE itself, so winning the delete is the proof that nothing referenced the hash. A workspace that adopts the hash first makes the delete match nothing and the release becomes a no-op. Claiming the row before the provider call would otherwise strand a template nothing points at if the provider then refused, so that path puts the row back and the retention sweep inherits the retry — the same property the previous ordering had. The sweep is deliberately left as it is: its equivalent window needs a hash unreferenced AND unused for 30 days, and its provider-first ordering encodes the documented retry-on-refusal behaviour this path now reproduces explicitly. No transaction is opened. The provider call sits between discrete statements rather than inside one, so no pooled connection is held across it — which is why this uses a conditional delete instead of the repo's `pg_advisory_xact_lock` pattern, whose lock only releases at commit. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): route the retention sweep through the same image claim Cursor and Greptile both flagged the sweep as still carrying the interleaving just fixed in releaseSandboxImage, and they are right — the reason given for leaving it alone last round does not survive scrutiny. That reason was that provider-first ordering encodes retry-on-refusal, so making the claim atomic would trade a race for an orphaned template. The release path already answers that: claim the row, and put it back if the provider refuses. The sweep can have both properties too. The rarity argument was also weaker than stated. The sweep nominates up to 200 candidates and then works through them eight network deletes at a time, so its check-to-delete gap is seconds to minutes — wider than the window that was just closed, not narrower. Both callers now share `claimAndDeleteImage`, which owns the whole contract: the unreferenced check lives inside the DELETE, the provider call runs only after the claim succeeds, and a refusal restores the row. Having written that ordering twice is what let the two paths drift, so it exists once now. The sweep's query becomes a nomination step only. Its retention cutoff is passed into the claim rather than trusted from the earlier read, so a candidate that stops qualifying mid-sweep fails its claim and is skipped instead of losing its image. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): rebuild a hash adopted while its image was being deleted Greptile's third pass on this path, and a case the previous two did not cover: the adopter starting a *fresh build* rather than inheriting a ready row. Claiming removes the registry row, so between that and the provider delete finishing, a workspace can declare the same package list, get a new row, and start a build under the same content-derived imageRef — which the in-flight delete then removes. The window itself is inherent. The registry row and the provider template are two systems with no shared transaction, so it can be narrowed but not closed. A Redis lock would not close it either: acquireLock returns true when Redis is absent, so it cannot be a correctness guarantee for self-hosted. Holding a Postgres advisory lock would, but only by pinning a pooled connection for the length of a provider call, which is a worse trade. What was avoidable is the adopter finding out the slow way. Its row is new and healthy-looking, so nothing noticed: resolution only repairs a row that is missing or failed, and a failed one waits out the retry cooldown first. The release path now re-checks after the delete and re-enqueues, so the rebuild starts immediately instead of one failed run plus a cooldown later. A build already in flight is left to the conflict guard, since it may still outlive the delete. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): reclaim a ready row whose image was deleted underneath it Greptile found the hole the previous commit left, and it is the case that made the claim in that commit's message wrong: this one is permanent, not transient. If a re-adopted hash reaches `ready` before the in-flight provider delete lands — plausible, since E2B layer caching can rebuild an identical spec in seconds — the row looks healthy while its imageRef points at nothing. Resolution repairs a row that is missing or failed, never one claiming to be ready, so nothing recovers it. The sandbox stays broken until someone re-saves it by hand. `rebuildIfReadopted` called `ensureSandboxImage` with no options, whose conflict guard reclaims only a failed or stale in-flight row, so it silently did nothing in exactly that case. The release path now passes `imageKnownGone`, which widens the re-claim to any settled row rather than only a failed one. It is the one caller that knows the image is gone regardless of what the row says. An in-flight build is still left alone: it either recreates the template it was building or fails into the normal repair path, and resetting it would only add a duplicate build. The three ways a settled row may be re-claimed now sit in one `settledRebuildBranch` helper — any settled row when the image is known gone, a failed one after the cooldown for an automatic caller, a failed one immediately for a person — because inlining the third case is what hid the gap. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): let a same-spec save retry a failed build Cursor Bugbot. `scheduleSandboxBuild` sat inside the changed-hash branch, so a save that did not alter the package list never reached the registry. The comment above it described the opposite — that an unchanged spec finds a ready row and enqueues nothing — which is what `ensureSandboxImage` does, but only if it is called. That made the docs wrong too. They tell a reader to save the sandbox again to retry a failed build immediately, and this branch is exactly why that did nothing: the only way to retry was to edit the package list into a different hash, which is not what someone recovering from a transient registry failure wants to do. The call is now unconditional and the registry decides what a save costs, which is what its conflict guard is for: a ready or in-flight row is left alone, a failed one is re-claimed at once. Releasing the previous image stays behind the hash check, since only a changed hash orphans one. Cache invalidation is unchanged — `scheduleSandboxBuild` already does it, which is why the else branch existed. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): correct the image cache's staleness invariant Cursor Bugbot found that a released image can still be served from another replica's cache. The finding is real, and the reason it went unnoticed is that the cache documented an invariant which eager release quietly broke. It claimed a `ready` row is terminal for its spec hash, so a cached hit could not go stale in a way that matters. That held while the only ways a row changed were an edit (new hash) or a delete (caught by the `workspace_sandbox` read). Releasing an image eagerly made a `ready` row disappear with the hash unchanged, so the premise no longer holds and the comment was actively misleading to the next reader. No behaviour change here — the exposure is bounded at IMAGE_TTL_MS on replicas other than the one that ran the release, and it self-heals once the entry expires and the row read finds nothing. Closing it properly needs cross-replica invalidation or a provider-error path that invalidates on "template not found", both of which are larger than a review fix; the comment now says so instead of implying the problem cannot exist. Co-Authored-By: Claude <noreply@anthropic.com> * docs(sandboxes): note that a JavaScript sandbox needs an import to apply Cursor Bugbot pointed out that `useRemoteSandbox` keys on detected static import/require and never on the selected sandbox, so JavaScript without one runs locally and the selection has no effect. Keeping the behaviour: honouring the selection would force those blocks remote, and the large-value-ref guard immediately below would then reject code that runs fine today. Documenting it instead, next to the picker, since a selection that silently does nothing is only surprising if nothing says so. Python is unaffected — it always runs remotely, so its sandbox always applies. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): stop create mode surviving a return to an open sandbox Cursor Bugbot. Create mode and having a sandbox open are mutually exclusive, but nothing enforced it, so both could be set at once — and the screen then lied about which sandbox its Delete pointed at. With `isCreating` true and `selectedId` restored, `baseline` is null, so the editor renders an empty "New sandbox" form, while the Delete action is built from `selected` and still targets the restored sandbox. An admin looking at a blank create form could delete a sandbox it never named. Two ways in, both closed: - Browser Forward after starting a new sandbox restores `selectedId` without going through `closeEditor`. The render-time sync that already drops a stale draft now also leaves create mode, which is the same class of correction and the reason that block exists. - "New sandbox" set `isCreating` without clearing `selectedId`, so the same contradiction was reachable without touching history at all. It now clears the selection, with `history: 'replace'` because switching mode is not a destination. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ci): pin the sandbox flag in the second nav catalog test, bump the chart Two CI failures, both mine. `app/workspace/[workspaceId]/settings/navigation.test.ts` asserts the unified catalog and was left on ambient env. Dropping the Sandboxes section without a sandbox provider made it 26 items instead of 27 on CI, which has no `apps/sim/.env` — the same trap already fixed in the sibling `components/settings/navigation.test.ts`, in the one file that was missed. Fixing it needs `vi.hoisted` rather than the sibling's `beforeEach`, because this file reads `allNavigationItems`, built once at module load; a hook would run after the value it is trying to influence already exists. The chart gate is separate: this branch adds sandbox settings to `helm/sim/values.yaml`, and the workflow requires a Chart.yaml bump whenever `helm/sim/**` changes. Additive config, so 1.3.0 -> 1.4.0 by SemVer. Verified by running the whole suite with the flags forced off, not just the two navigation files — no other test depends on a local env file. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): keep the row restore to a refused delete only Cursor and Greptile, independently, on the same code. `deleteImage` and `rebuildIfReadopted` shared one try/catch, so a rebuild failure after a *successful* provider delete was handled as if the provider had refused: the catch put the claimed row back, `ready` status and all, pointing at a template that no longer exists. That is the one state resolution cannot repair — it fixes a row that is missing or failed, never one claiming to be ready — so it reintroduced the permanent breakage an earlier commit had just closed, through the error path rather than the happy one. Restoring now belongs strictly to a refused delete. Once the template is gone the row stays gone, and the rebuild runs past that catch. The rebuild also swallows its own failures: it follows a delete that already succeeded, so it must not be reported as a failed release, and inside the sweep it must not reject the rest of its chunk. The adopter's next run still reaches the normal repair path. The regression test drives a rebuild failure and asserts no row is restored. It fails against the original shape — rebuild inside the shared try, no inner catch — which is what the two reviewers were describing. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): drop the dead row when a re-adopt rebuild cannot be scheduled Greptile, one layer under the previous fix. Making the post-delete rebuild swallow its own failures kept it from being reported as a failed release, but left the adopter's row claiming a `ready` image whose template is already deleted — the one state resolution cannot repair, since it rebuilds a row that is missing or failed and never one that says ready. So the row is now dropped when the rebuild does not take. That turns the adopter into the missing-row case, which the next execution repairs on its own, instead of a sandbox that stays broken until someone re-saves it by hand. A failure to drop it is logged at error, because at that point two writes in a row have failed and there is nothing further this path can do. Also gives the release tests a default "nothing re-adopted" select. Without it the rebuild threw on an unstubbed mock and the cleanup delete overwrote the predicate the claim assertions read, so two of them were passing on the wrong statement. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): repair a missing image at create, where the truth is observable Six review rounds narrowed the window between deleting a shared template and another workspace adopting its content hash, and each fix exposed the next facet. They all share a cause: the registry row and the provider template are two systems with no shared transaction, so any scheme that keeps them in step is guessing. Create is the one step that does not have to guess. It either gets a sandbox or it does not, so a `ready` row pointing at a deleted template now corrects itself the first time it is used, rather than needing someone to re-save the sandbox. - `SandboxImageBuilder.isMissingImage` asks the provider to classify its own failure. Prebuilt-only, because a runtime provider has no image to miss - E2B answers it off `NotFoundError`, which the SDK maps from a 404. The only resource a create names is the template, and the two subclasses that describe other calls — a missing file, an exited sandbox — are excluded. The classifier stays deliberately narrow: treating auth or rate-limit failures as a missing image would turn a provider outage into a build storm - `repairMissingSandboxImage` invalidates the cache, rebuilds with `imageKnownGone` (no cooldown, since this observed the image is gone rather than inferring it), and returns copy telling the author to run again - `ResolvedSandbox` carries `specHash` so the failing execution can name what to rebuild This subsumes the open facets rather than adding another guard beside them: the stale per-replica cache, an adopter left `ready` against a deleted ref, and a rebuild that never took all end at the same place — the next run repairs itself. Co-Authored-By: Claude <noreply@anthropic.com> * fix(sandboxes): key the build trigger by attempt, not by spec Cursor Bugbot. The Trigger.dev idempotency key was the content address alone, so a second attempt at the same spec was deduped against the first: the SDK returns the finished run instead of starting one, and the row that `ensureSandboxImage` just flipped to `pending` sits there with no worker. Nothing can re-claim a `pending` row until it goes stale, so a retry inside the 5-minute TTL did nothing for the next half hour. That silently disabled every repair path — save-to-retry, which the docs name explicitly, and both the resolution and create-time rebuilds. The key's own comment already said it exists "to collapse concurrent saves of the same spec into one build, not to suppress a retry after one failed". The conditional update above it is what actually collapses concurrent saves: only one caller gets a row back, so only one ever reaches the trigger. Keying by the claim's `updatedAt` keeps that property and makes each genuine attempt distinct, while a duplicate delivery of one attempt still collapses. Co-Authored-By: Claude <noreply@anthropic.com> * feat(sandboxes): create a sandbox from the picker, and fix three UI papercuts The Function block's sandbox field now pins a "Create Sandbox" row above its options, matching the "Create Skill" / "Create Tool" rows it sits beside, so authoring a package list no longer means leaving the workflow for Settings. The row is declared by the field (`createAction`) rather than hardcoded by id; block configs are read by the serializer and executor, so the name maps to a modal in the picker rather than carrying a component. Two things the modal has to get right. It seeds the new sandbox's language from the sibling the list is scoped by, or a sandbox created off a JavaScript block would land in the Python list and vanish. And the created option is held locally until a real fetch carries it, or the field would sit on a raw uuid until hydration answered. Also: - The Sandboxes icon was the Logs block's icon (`blocks/blocks/logs.ts`), in both the settings nav and the list rows. It is the Function block's now. - "Default image (no extra packages)" claimed something untrue: E2B and Daytona base images both ship with packages installed. - A new sandbox opened in Python while the Function block defaults to JavaScript. The test pins the two together rather than the literal. Draft shape and helpers moved out of the editor component into `utils.ts` — three consumers now, and it makes the defaults testable without a DOM. * feat(settings): one Max-plan wall, and give the create modal the same one The create-sandbox modal answered a non-Max workspace with a red line under a form it could never submit, and no way to act on it. It now renders the same wall the Settings > Sandboxes tab does — heading, one sentence on what the plan unlocks, and an Upgrade to Max chip — instead of the fields. That wall existed twice already (sandboxes and Sim Mailer), so this extracts it rather than adding a third copy. `SettingsUpgradeNotice` owns the copy rhythm and the route, and `compact` trades the page's full-height centering for a modal's. Both settings consumers now compose it; neither keeps its own markup. The action lands on billing, which `resolveSettingsHref` already redirects to the plan-comparison page for a member who cannot manage billing — so it is a route to explore plans, never a dead end. The chip stays hidden for non-admins, exactly as the settings pages had it. A non-admin on an entitled workspace gets the muted reason rather than the upgrade wall: buying a plan is not what is in their way. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-30 16:33:35 -07:00
Theodore Li	32293f4b55	feat(tables): typed predicate filter grammar, cursor pagination, and the v2 table surface (#6067)	2026-07-30 00:24:06 -07:00
Vikhyath MondretiandClaude	11c0d3b75d	feat(organizations): sweep a joiner's owned workspaces into the org on join, disclose it at accept, and add external workspace invites (#5918) ... * feat(invites): explicit external members * update docs * fix(organizations): atomic admin workspace sweep, removal-impact status in dialog, and preview-unavailable disclosure Review round 1: the v1 admin add-member now commits membership and the workspace sweep in one transaction; the remove-member dialog holds confirm while the credential-impact check loads and shows a caution when it fails; a failed join preview flags joinPreviewUnavailable so the accept screen falls back to a generic migration notice. Also aligns the invite test's react-query mock and repairs two pre-existing docs type errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): close the concurrent-workspace escape in the join sweep Personal workspace creation now serializes with organization joins on the user's billing-identity lock and re-verifies membership inside its transaction; both join paths (invite acceptance and the v1 admin add) re-read the owned-workspace set under that lock after the member insert and roll the whole join back when it diverged from the advisory-lock plan, so a workspace created mid-join can never land outside the organization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): fail stale-grant member joins and re-resolve the creation race client-side A member-role org acceptance whose grants all turned stale now rolls back with workspace-not-found instead of stranding a workspace-less member, and the workspace resolver treats the creation-vs-join 409 as a signal to re-resolve (the user is authenticated with org workspaces) rather than falling into the login path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): mirror the stale-grant gate in the join preview The accept-screen preview now returns no-join for a member-role org invite whose grants all left the stamped organization, matching the acceptance-side rollback so the disclosure never promises a migration that acceptance would refuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(workspaces): survive the join race on lazy default creation and gate removal on live impact data The workspace list GET now re-lists (returning the join sweep's workspaces) when lazy default creation loses the race to an organization join instead of failing with a 500, and the remove-member dialog gates its confirm on isFetching so a background refetch can never let an admin confirm against a stale credential-impact list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): surface the accept conflict message and refresh workspace caches post-accept The accept route now carries the human-readable message alongside the machine-readable error kind (the client prefers it for server-error), and a successful accept invalidates workspace queries so the swept workspaces appear immediately instead of after the stale window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(billing): sweep archived workspaces in Pro-to-Team conversion and org creation Every attach call site now passes includeArchived so the archived escape hatch is closed uniformly — join-attach, admin move, subscription-driven org provisioning, and manual org creation all sweep archived personal workspaces into the organization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): reject acceptance when the sweep set differs from the disclosed set The join preview now carries the workspace ids it disclosed, the accept screen echoes them back as a disclosure token, and acceptance rolls back with disclosure-outdated (409) whenever the set it would sweep no longer matches — a workspace created after the preview rendered can never move without the user seeing the refreshed notice first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): send the disclosure token for no-join previews too A preview that predicted no join still tells the user nothing moves — the empty disclosed set is now echoed on accept, so a join that becomes possible between preview and accept (left another org, billing turned usable, grants un-staled) conflicts with disclosure-outdated instead of sweeping workspaces without a rendered notice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): gate all-stale member joins before disclosure and always refetch removal impact The all-stale check for member-role org invites now runs before any mutation and before the disclosure comparison, so an invite whose grants all left the org fails with workspace-not-found instead of trapping owners of personal workspaces in a disclosure-outdated retry loop. The removal-impact query drops its stale window (staleTime 0): every dialog open refetches while the confirm is held on isFetching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(organizations): keep-external on org recovery and label archived move candidates Org creation/recovery now uses the keep-external collaborator policy (matching Pro-to-Team conversion) so different-org collaborators on archived workspaces cannot abort it with a conflict, and the admin workspace-move search and preflight expose an archived flag so internal tooling can label archived targets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(invitations): guard the reverse disclosure direction A will-join notice whose acceptance downgrades to no-join (stale escalation denial, concurrent other-org membership) now fails with disclosure-outdated instead of silently succeeding as an external grant — the disclosure token binds the outcome in both directions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address comments * fix * fix(invitations): one invite surface, coalesced grants, coherent seat model Consolidate the two invite modals into a single surface and fix the semantics the split had been hiding. Invite flow: - One InviteModal for all three entry points (workspace header, workspace settings, organization settings), with a workspace multi-select and an explicit Membership choice that states the seat consequence. - Coalesce grants instead of 500ing. A partial unique index allows one pending invitation per (email, organization), so inviting someone to a second workspace raised a raw 23505. New workspaces now merge into the pending invitation (with a retry for the concurrent-insert race) and the invitee gets one link covering everything. - External collaborators require their own paid plan, checked at invite time and re-checked on accept, since the invitation lives for 7 days. Imposed externality is exempt in both places: an invitee already in another organization is forced external regardless of the inviter's choice, so the plan gate must not apply to them. - Every invitation must grant at least one workspace, so accepting always lands somewhere. Enforced at creation for all roles. - Revocation is grant-scoped. Since one invitation can span workspaces, revoking from a workspace's member list withdraws only that grant and cancels the invitation just when the last one goes. Whole-invitation revocation now requires authority over all of it rather than admin on any single granted workspace. - The accept screen names every granted workspace and states whether the invitee joins as a member, an admin, or an external collaborator, and whether that uses a seat. Seats: - One rule for the pending-invitation predicate, seat capacity, and the derived figures; the three counting sites now share it. - Team seats are elastic (subscription.seats tracks the member count), so treating that as a cap reported negative headroom on any outstanding invite. Available seats are clamped and gates branch on whether the plan actually has a fixed cap. - POST /api/v1/admin/organizations/[id]/members could never succeed on Team: it validated N members against N seats. It now skips the cap for elastic plans, matching invitation acceptance, and reconciles seats after a committed add. Also surfaces the External label on the workspace Teammates list, which already received the flag and dropped it, and removes dead code: the grantless organization-invite route and contract, three unreferenced invitation helpers, and the unreachable ensureUserInOrganization/addUserToOrganization/validateMembershipAddition cluster. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): close the two accept-disclosure gaps Bugbot found Membership notice ignored the join preview. `buildMembershipNotice` keyed off the invitation's sent `membershipIntent`, but acceptance resolves an internal invite to external when the invitee already belongs to another organization, or when the granted workspace changed organizations after the invite went out. The screen therefore promised "you'll join as a member, which uses one of their seats" to people who would neither join nor consume a seat. It now keys on the preview's `willJoinOrganization` — the same signal the migration notice already used — and falls back to the sent intent only when no preview could be computed. In-app accept skipped the disclosure entirely. `useAcceptMyInvitation` posted an empty body, so `disclosedWorkspaceIds` was absent and the server's consent guard was skipped, and the pending-invitations modal never showed which owned workspaces would move. Accepting from the workspace switcher (including the desktop path) could silently sweep personal workspaces into the organization, bypassing the consent model this PR adds on /invite. The list endpoint now returns each invitation's join preview, the modal renders the same membership/migration disclosure as /invite, and accept echoes `disclosedWorkspaceIds` so the guard applies on both paths. Both notices moved into lib/invitations/disclosure-copy.ts and are consumed by /invite and the modal, so the two accept surfaces cannot drift into disclosing different outcomes for the same invitation — which is how this gap arose. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): carry the membership outcome in the disclosure token Empty disclosure skipped membership consent. The token was only the workspace-id list, so a no-join preview and a will-join preview for someone who owns nothing both echoed `[]`. Neither guard could tell those apart: the forward check compares sweep sets, and the reverse check required a non-empty disclosed set. An invitee who left their other organization between preview and accept would therefore be silently made a seat-consuming member after being told they would stay external, and the mirror case could silently demote a promised join. The accept body now also carries `disclosedWillJoinOrganization`, compared against the resolved outcome before any write, so consent covers the membership decision and not just the migration. Both accept surfaces send it. This was widened by the previous commit: keying the membership notice on the preview made the screen promise a join outcome the token never verified. In-app accept errors lacked copy. `getInvitationErrorMessage` omitted `external-requires-paid-plan`, `disclosure-outdated`, and `workspace-not-found`, so those failures fell through to the generic "may have expired" fallback. `disclosure-outdated` became newly reachable in-app the moment that path started sending the token, so the gap arrived with the fix for it. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): compare the join disclosure against new-membership creation The membership consent guard compared the disclosed outcome against `shouldJoinOrganization`, which stays true for an invitee who already belongs to the target organization — the invitation's intent is still internal. The join preview reports no-join for exactly that case, because nothing changes for them. Every such acceptance therefore failed `disclosure-outdated`, and the retry re-rendered the same preview, so the invitation became permanently unacceptable. The guard now compares against whether acceptance creates a NEW membership (`shouldJoinOrganization && !alreadyMemberOfTargetOrganization`), which is what the disclosure actually promises and what the preview reports. The already-a-member predicate is hoisted and shared with the join block below so the guard and the billing path cannot disagree about it. Regression test asserts a pre-existing member accepts with a no-join disclosure; it fails with `disclosure-outdated` against the previous comparison. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): tell an existing member their standing is unchanged The join preview reported the same no-join shape for two different outcomes: an external collaborator, and an invitee who already belongs to the organization acceptance lands in. `buildMembershipNotice` rendered both as "you'll join as an external collaborator ... everything you own stays yours", which is wrong for an existing member — they stay an internal member and simply gain the granted workspaces. The preview now reports `alreadyMemberOfOrganization` for that case (a membership in a DIFFERENCE organization is still the external path, since acceptance downgrades), and the notice states that standing is unchanged. This is the same conflation behind the accept loop fixed in 4eb7725f1a, now removed from the shape itself rather than worked around per consumer. Also re-verify the removal-impact disclosure at the moment of confirmation. `isFetching` only holds the confirm button while a request is in flight, so an identity-bound credential the member gained after the fetch settled would break on removal without ever being disclosed. Confirm now refetches and, if the set changed, keeps the dialog open on the refreshed warning instead of proceeding. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): serialize the join-preview reads, revert the removal refetch Two corrections to this branch's own review fixes. The pending-invitations list computed each row's join preview with Promise.all. Every preview issues several queries, and the endpoint is hit whenever the workspace switcher opens, so that held one pooled connection per pending invitation for as long as the slowest one took. The loop is sequential now; the list is a handful of rows, so the latency is not worth the pool pressure. The removal-impact refetch on confirm is reverted. It changed behaviour — a click could silently do nothing — and it did not actually close the window it targeted: the credential set can still change between the refetch and the independent DELETE, because the removal endpoint neither receives nor revalidates the disclosed set. Closing that properly means passing the disclosure to the endpoint and revalidating there, which is a feature rather than a review fix, so the prior behaviour stands until it is done deliberately. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): disclose the seat on a personal-workspace join Both accept surfaces scoped the membership notice on `organizationId` or the preview's `organizationName`. A personal-workspace invite has neither until acceptance runs — it creates the organization by converting the billed owner's Pro to Team — so the seat and membership disclosure was suppressed for exactly the case that creates the membership. They now also scope on the preview's `willJoinOrganization`, which is the authoritative signal. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): make the join preview a discriminated outcome The preview returned one no-join shape for five different results — external intent, already a member, a membership in another organization, a dead-grant rejection, and a billing rejection. Two accumulating booleans could not separate them, and the accept screen rendered the external copy for all of them: it told people whose acceptance would fail with `upgrade-required` or `workspace-not-found` that they were getting workspace access without a seat. It now reports one `outcome`: `will-join` (a seat is taken), `already-member` (only workspace access changes), `external` (never a seat), or `blocked` (acceptance fails, so nothing is promised). `blocked` renders no membership notice — silence is accurate where the external claim was false. The accept button is deliberately left enabled: those cases already fail closed with the correct error, and choosing what to actively tell someone whose organization's payment lapsed is a product decision, not a review fix. This also fixes a live mis-attribution the previous commit's guard introduced. The consent check ran before the gates that produce the real cause, so a blocked invitation returned `disclosure-outdated` — and the retry re-rendered the same preview, leaving the invitee looping with no explanation. The guard now sits after the dead-grant gate, and a disclosed `blocked` skips the comparison so the billing gate below can surface `upgrade-required` instead. The accept body carries `disclosedOutcome` in place of the boolean; the membership comparison is unchanged (`will-join` versus a new membership being created), so no acceptance that previously succeeded now fails. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): let billing-disabled personal invites be accepted The consent guard derived "a membership will be created" from `shouldJoinOrganization`, which is still true at that point — it is only cleared much later, after provisioning fails to yield a target organization. With billing disabled and no organization on the workspace there is nothing to provision and nothing to join, so the preview correctly reports `external` while the guard computed `will-join`, rejecting every personal and grandfathered workspace invite as `disclosure-outdated`. The retry rendered the same preview, so those invites could not be accepted at all on billing-disabled deployments. The predicate now mirrors the preview's own condition, so the two cannot drift. Regression test asserts acceptance succeeds with billing off; it fails with `disclosure-outdated` against the previous predicate. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): allow External with billing off, mark cross-org org invites blocked The External paid-plan requirement is seat economics — an external collaborator takes no seat, so somebody else must be paying for them. With billing disabled there are no seats and no subscription rows at all, so every account resolves as `free` and choosing External failed at send time and would have failed at accept. Member and Admin still worked, so a self-hosted deployment had no way to grant workspace-only access without an organization join and a workspace sweep. Both the invite-time and accept-time gates now short-circuit when billing is off. A route test asserted that rejection without setting `isBillingEnabled`, which the shared mock defaults to false — it passed only because the gate ignored the flag. It now opts in explicitly, since the rule it covers is billing-only. Separately, the preview reported `external` for any invitee already in a different organization, but acceptance only downgrades a workspace-kind invite with live grants; an organization-kind invite hard-fails with `already-in-organization`. Those now report `blocked`, so the screen stops promising external access that acceptance can never grant. Legacy organization-kind rows still exist and coalescing preserves that kind, so this is reachable. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): require org admin to grant org Admin, and two disclosure gaps Privilege escalation. `createWorkspaceInvitation` stamped organization role `admin` whenever the caller passed `membership: 'admin'`, but authorization only checked workspace admin access — and unifying the invite modals exposed the Admin option to any workspace admin, where it had previously been reachable only from organization settings. A workspace-scoped administrator could therefore invite someone who joins as an organization Admin, gaining admin on every workspace the organization owns plus member and billing management. The inviter must now already hold organization owner/admin, checked server-side because the batch endpoint is reachable without the modal, and the modal no longer offers Admin to anyone else. The preview promised external access without mirroring acceptance's `external-requires-paid-plan` gate, so a free invitee — one who cancelled Pro, or left the organization that forced the external invite — was told they had workspace access and then refused. It now mirrors that gate, including its exemptions (billing on, organization-owned workspace, externality not imposed), and reports `blocked`. The modal's Enterprise seat check counted every non-External email as a seat. The server does not: an existing organization member is granted access directly, and an invitee already in another organization is forced external. The hard block refused batches the API would have accepted, so it is advisory now — per-email failures already come back with reasons. Co-Authored-By: Claude <noreply@anthropic.com> * fix(invitations): restore the invite admin gate, hedge an unknown outcome Consolidating the modals dropped a permission check. The old workspace-header modal derived `canInviteMembers` from `userPermissions.canAdmin` internally and disabled its field and button; the shared modal takes `canInvite` as a prop that defaults to true, and no call site passed it. Non-admins therefore saw a fully enabled invite form and only learned otherwise when the server refused the send. All three entry points now supply it: workspace admin for the header, the existing `canManage` for the workspace Teammates page, and organization owner/admin for organization settings. When the join preview cannot be computed the outcome is unknown, and the callers send no disclosure token — so acceptance runs without the consent guards. The membership notice nevertheless asserted a seat-taking join from the sent intent, which acceptance may resolve to external, already-a-member, or a failure. It is conditional now ("If you're added to X as a member, that uses one of their seats"), so the consequence is still disclosed without being claimed as settled. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-29 20:27:55 -07:00
Theodore Li	985f9e6172	feat(tables): saved views with filter, sort, and column presets (#5961) ... * feat(tables): saved views with filter, sort, and column presets * fix(tables): views own column layout, preserve deep-linked sort, seed update cache * fix(tables): merge view layout writes server-side, keep layout on save-from-All * fix(tables): send view saves as a merge patch so concurrent writes can't clobber * fix(tables): persist explicit All selection, prune view state for deleted columns * fix(tables): key-order-stable dirty check, reset layout when switching to All * fix(tables): return 404 when patching a missing view * improvement(tables): order the bar filter, sort, columns and drop the hidden count * fix(tables): record newly created columns in the active view's order * fix(tables): don't write layout as a reader, clear dead sort, reset dead view id * feat(tables): add New view to the views menu, starting from All * chore(api): bump route-count baseline to 981 after staging merge * fix(tables): guard default demotion, use the applied filter for dirty/save * fix(tables): clear state when the active view is deleted externally * revert(tables): drop the ineffective rows-query gate for view resolution * feat(tables): enable saved views in the embedded mothership table * fix(tables): resolve the views flag on the chat route too * improvement(tables): right-align the embedded run/stop control * fix(tables): live layout on new views, prune stored config, order cache writes * fix(tables): live layout on save-as-view, reset it per view, ignore inherited param when embedded * fix(tables): route undo/redo column-layout writes to the active view * fix(tables): scope layout undo to the view that recorded it Layout is view-owned, but UndoEntry carried no owner, so the persistence sink — rebound every render to the active view — decided the target at undo time rather than at record time. Reordering in view A then undoing from view B wrote A's column order into B and left A unchanged. Entries are now stamped with the active view id, and the three layout-bearing action types (create-column, delete-column, reorder-columns) are pruned from both stacks when the active view changes. Row and schema actions are table-scoped and survive the switch untouched. Inert when views are disabled: the id is always null, so nothing is ever pruned. * fix(tables): send only the layout keys an action changes Pin, reorder, and insert-column each shipped a full columnWidths snapshot they never modified. Both sinks merge at top-level key granularity, so an earlier-issued patch landing after a concurrent resize replaced the newer width map with its stale copy. Resize, auto-resize, and delete-column still send columnWidths — those genuinely change it. * fix(tables): don't strand layout writes while views load, keep schema undos across views Two more instances of layout being written without a known owner. The sink was left unbound until a view resolved, so a resize/reorder/pin (or the column-append effect) during the views fetch fell through to the table's shared metadata — corrupting All for a table about to adopt a view, and losing the edit to the re-seed. The sink is now bound while the query is in flight and suppresses the write. An error counts as settled, so a failed views fetch falls back to All instead of suppressing layout writes for the session. Pruning was also too broad: create-column and delete-column are table-scoped schema ops that merely have a layout side-effect, so dropping them on a view switch made a deleted column unrecoverable. Only reorder-columns is purely layout and still prunes; the other two survive and have just their layout half suppressed at replay when the recorded view isn't active. * fix(tables): flush layout buffered during load when the owner settles to All Suppressing the write while the views query was in flight stopped All from being corrupted, but nothing resolved the buffer afterwards, so a resize during load looked applied and vanished on refresh. Settling on All re-seeds nothing (viewLayoutKey never changed), so the gesture is still on screen and is now persisted to shared metadata. Adopting a view re-seeds the grid from that view and has already replaced the gesture on screen, so the buffer is dropped to match. * fix(tables): resolve undo layout ownership at write time, not dispatch time The layout writes for column create/delete happen in mutation success callbacks, and persistLayoutRef is rebound every render. Resolving entryOwnsLayout up front meant the guard could still hold from before a view switch while the sink it guarded already pointed at the destination, writing the recorded view's layout into whichever view was now active. Now a function, so the guard and the sink are read at the same moment: switch away mid-mutation and the schema half still lands while the layout half is dropped, matching the rule that undo only ever affects the view on screen. * fix(tables): read layout from the grid instead of mirroring it The wrapper kept two shadow copies of the grid's column layout — liveLayoutRef and pendingLayoutRef — and all three of this round's findings were that mirror going stale: - liveLayoutRef was only cleared on activeView.id change, so widths buffered before the views query settled survived into All, where layout writes bypass the mirror entirely. Both create paths spread it last, so a saved view stored those snapped-back widths. - currentViewConfig memoized a spread of that ref, and mutating a ref doesn't re-run a memo, so Save as view sent the pre-gesture layout. - The flush effect keyed on activeView, but adoption writes the view id through the URL, so for one render the query had settled while activeView was still null — flushing to All in exactly the case that had to drop. The grid owns this state, so it now publishes a reader through a sink ref and the wrapper asks at the moment it needs a value. Nothing to keep in sync, so nothing to go stale. Only whether an unowned change happened is tracked, and the resolve effect — which is what actually picks the owner — decides to persist or drop. * fix(tables): flush unowned layout when the views fetch fails The resolve effect is the only caller of resolvePendingLayout and gated on isSuccess, which never becomes true on a query error — so layout touched during the load window was never persisted on the error path, even though the table had already settled to All and later writes worked. The error branch now flushes to shared metadata, matching the rest of the error path's fall-back-to-All behavior. * fix(tables): gate the on-screen layout restore by ownership, not just the persist The undo success callbacks applied the recorded view's order/widths/pinning to the grid unconditionally and only gated the PATCH, so switching views before a column create/delete mutation resolved left the destination displaying the origin view's layout until switched away and back. Each callback now checks ownership where its layout work begins: three are purely layout and return at the top; delete-column undo restores cell data first (row data, runs everywhere) and gates only the layout block below it. In a non-owning view the restored column still appears via the grid's append effect — at the end, leaving that view's layout untouched. * fix(tables): route all layout writes through one owner-aware sink, reconcile order at seed Two holes closed: The sink binding toggled on activeView, leaving a render-frame gap after the views query settled but before the resolve effect adopted a default — writes in that gap fell through to shared metadata. The sink is now always bound while views are enabled and handlePersistLayout is the single router, reading the owner at call time: unresolved buffers, a view patches the view, All writes metadata. The error branch stamps the owner so post-error writes stop buffering. The append effect only fires when the schema changes, so a column that arrived while another source owned the layout rendered via the displayColumns fallback but was never written into the adopted owner's stored order until a drag happened to heal it. Seeding now reconciles the incoming order against the schema and persists the appended tail through the current sink. * fix(tables): capture the layout owner when a schema action is dispatched Insert-column and the delete chain persist layout from mutation callbacks through updateMetadataRef, which always targets the current sink — so a view switch mid-flight wrote the origin view's order/widths/pins into the destination. Undo got this guard already; the live paths never did. Both now capture viewLayoutKey at dispatch and compare at the callback. On a mismatch the layout work is skipped: the destination re-seeded its own layout, the new column lands there via the append effect when the refetch arrives, and the deleted column's dangling keys are pruned on read. pushUndo also takes the captured owner as an override — stamped at callback time it would record the destination, letting a later undo apply the origin's layout to it. * fix(tables): resolve views on list availability, not query success A failed background refetch flips isError while the cached list stays usable, and every view mutation invalidates the views query — so one blip made the resolve effect treat views as terminally failed and stop applying switches until the next successful refetch. The axis is now whether a list exists: error with no list ever fetched settles to All; error with a cached list resolves normally against the cache; owner is unknown only while the initial fetch is in flight. * ci: raise Build App timeout to 25 minutes The build outgrew the 15-minute cap over the last two days of merges: 10m02 (c6acc62a07), then 14m44 after the folders/desktop/library batch (adc557a2f2, 16s under the limit), then two consecutive timeouts on 7b1af4121b after the outlook merge. Staging's own latest runs show the same signature (one cancelled). GitHub labels a job timeout "cancelled", which is why these read as cancellations.	2026-07-29 14:57:58 -04:00
mzxchandraandWaleed Latif	6ff6255900	feat(outlook): add Microsoft Graph calendar operations (#6041) ... * feat(outlook): add Calendars.ReadWrite and MailboxSettings.Read scopes Extend the shared outlook OAuth service with delegated Graph calendar scopes so one Microsoft connection covers mail and calendar. Existing connected users must reconnect to be granted the new scopes (noted in a code comment). Adds human-readable scope descriptions and test assertions. * feat(outlook): add Microsoft Graph calendar tools Six calendar operations against graph.microsoft.com/v1.0: list events (calendarView with nextLink paging), get, create, update (partial PATCH), delete, and respond to invites. Shared calendar-utils handles Graph's offset-less dateTime+timeZone shape, attendee normalization, and event flattening. All tools carry the Microsoft Graph error extractor and 429/backoff retry (honors Retry-After) for the mailbox concurrency limit. Registered in the tool registry and barrel. * feat(outlook): surface calendar operations in the Outlook block Add six calendar operations to the Outlook block operation dropdown with their conditional subBlocks, tool wiring, inputs, and event-shaped outputs. Mail operations are unchanged and backward compatible. * fix(outlook): address calendar review findings - Validate list-events pageToken origin with assertGraphNextPageUrl (matches the onedrive/microsoft_ad Graph paging guard) so a workflow-supplied URL can't receive the Outlook bearer token. - All-day create/update now normalize both bounds to midnight and force an exclusive end day (buildAllDayRange), instead of sending a zero-length same-midnight window that Graph rejects. - Drop the stale 'suggest meeting times' claim from the block longDescription. - Remove the unused MailboxSettings.Read scope (least privilege; no tool reads it). * feat(outlook): add calendar picker and harden calendar tools Validation pass over the new Microsoft Graph calendar operations against the v1.0 API docs, plus the calendar selection the tools were missing. - Add an `outlook.calendars` picker (GET /me/calendars) with basic selector + advanced manual ID, wired through the `calendarId` canonical param. List and create now target `/me/calendars/{id}/...`; get/update/delete/respond keep using `/me/events/{id}` since event IDs are mailbox-unique. - Fix all-day update rejecting when only one bound is supplied — both bounds are now normalized to midnight with an exclusive end day. - Fix "Send Response to Organizer" reading as OFF while Graph's default is to notify; it is now a dropdown defaulting to Yes, and the param is always sent. - Add timestamp wandConfig to the four calendar datetime fields and a list wandConfig to attendees. - Centralize Graph URL construction in calendar-utils; guard maxResults against non-numeric input and trim the calendarView time-window bounds. - Add three calendar templates and four calendar skills to OutlookBlockMeta. - Regenerate integration docs. * fix(outlook): scope online-meeting comments to what Graph documents onlineMeetingProvider is optional and defaults to unknown; the docs state that setting isOnlineMeeting alone initializes onlineMeeting. They do not document Graph substituting the calendar's defaultOnlineMeetingProvider, so the comments now claim only that, plus the real reason not to pin teamsForBusiness (mailboxes that disallow it via allowedOnlineMeetingProviders). * fix(outlook): allow calendar paging without re-supplying the time window Cursor Bugbot round: startDateTime/endDateTime were required tool params, so a paging call carrying only pageToken failed validateToolParameters before the request was built — even though the url builder short-circuits on pageToken and ignores both bounds. Relax them to optional and enforce the real invariant (pageToken OR both bounds) in the url builder, matching tools/sharepoint/list_sites.ts. Block subblocks stay required, so the normal editor flow is unchanged. Also correct the calendar_respond comment: Graph documents exactly two 400 conditions for accept/decline, both on proposedNewTime, which we never send. A non-empty comment alongside sendResponse=false is valid. * feat(outlook): add Calendars.ReadWrite.Shared for shared calendars The calendar picker lists /me/calendars, which can include calendars other users have shared with or delegated to the account. Calendars.ReadWrite covers only the user's own calendars, so selecting a shared team calendar would 403 on both read and write. Calendars.ReadWrite is kept alongside it, not replaced: Graph documents it as the sole accepted permission for creating and updating events and for accept/tentativelyAccept/decline (all list "Higher: Not available"), so the .Shared scope does not subsume it. Added now rather than later because this PR already forces existing Outlook users to re-consent for Calendars.ReadWrite; deferring would cost them a second reconnect. * fix(outlook): treat date-only bounds as all-day and guard partial all-day updates Cursor round 2: - Date-only bounds no longer produce a zero-length window. The param docs invited a date like 2025-06-03 for an all-day event, but buildAllDayRange only ran when isAllDay was explicitly true, so date-only input built a 00:00->00:00 timed window that Graph rejects. A date-only bound carries no time, so the only coherent reading is all-day; create/update now promote on that shape and the descriptions state it. - Converting an event to all-day with no bounds now fails with an actionable message instead of a Graph 400. Graph requires all-day events to have midnight start and end in the same zone, and those cannot be derived from a partial PATCH against an event whose existing bounds are timed. * docs(outlook): note that the calendar window fields are ignored when paging * fix(outlook): don't promote a lone date-only bound to all-day on update Regression from the previous round's date-only promotion. A single date-only startDateTime or endDateTime satisfied "all provided bounds are date-only", so the tool promoted to all-day and derived the missing side from the supplied one — turning a partial reschedule of a timed or multi-day event into a one-day all-day event and dropping the original other bound. Implicit promotion now requires BOTH bounds to be date-only, matching calendar_create_event. A lone date-only bound is ambiguous (convert to all-day, or just move that edge?) and a PATCH cannot read the event's existing bounds to disambiguate, so it stays on the timed path and leaves the other side untouched. Deriving a missing bound remains allowed when isAllDay is set explicitly, since that is stated intent rather than a guess. Also pins the explicit-isAllDay-false + date-only override with a regression test: the block always sends isAllDay for create, so an untouched switch arrives as false, and the data shape has to win or the fix would be unreachable from the UI. * revert(outlook): drop Calendars.ReadWrite.Shared from the outlook provider Reverts the scope I added two rounds ago. It was the wrong call. This provider is shared by work/school AND personal Outlook accounts, and the .Shared calendar scopes are not confirmed supported for personal Microsoft accounts. Requesting one risks failing consent for personal users — which would take mail access down with it, breaking functionality that works today. The PR already documents this exact reasoning as why findMeetingTimes was excluded, and that decision was made against a live personal mailbox. The evidence I added it on was a summarized read of the permissions reference claiming MSA support; a targeted follow-up could not confirm it for Calendars.ReadWrite.Shared specifically. Given the asymmetry — broken consent for all personal users vs. a shared-calendar feature gap — least privilege wins. Calendar operations therefore target calendars the account owns. The calendarId param descriptions now say a calendar shared by another user may return 403, and the scope list carries a comment explaining why .Shared must not be re-added. * fix(outlook): make retried event creates duplicate-safe via transactionId The retry config opts POSTs in via retryIdempotentOnly:false, but the executor's isRetryableFailure covers 429 AND 500-599 — not just the throttle the comment justified. A 5xx returned after Graph had already committed a create would be retried and produce a duplicate calendar event. create_event now sends a transactionId, which Graph documents for exactly this: it discards a repeat POST carrying an id it has already seen. The request body is built once per execution (formatRequestParams runs before the attempt loop), so the id is stable across retries of a call and unique between calls. The retry comment now describes what actually retries and why each non-idempotent method is safe: PATCH replays the same partial body as a no-op, and respond is state-idempotent though a post-commit retry can send the organizer a second notification — accepted deliberately, since Graph exposes no transactionId for accept/decline and failing outright under throttling is worse. * fix(outlook): tighten date-only detection and align online-meeting copy Final validation pass over the calendar integration. - isDateOnly matched "contains no T", so a space-separated datetime (2025-06-03 10:00) counted as date-only: the time was discarded and the value built as '2025-06-03 10:00T00:00:00', which Graph rejects. It now matches YYYY-MM-DD strictly, and buildGraphEventDateTime normalizes the space form to ISO rather than mangling it, so a natural input works instead of 400ing. - The isOnlineMeeting param descriptions still claimed Graph 'uses the mailbox default provider' — the same unverified mechanism already removed from the code comments. They now state only what the docs and the author's live testing support: the join URL depends on the providers the mailbox allows, and stays null on personal accounts. - Adds blocks/blocks/outlook.test.ts following the repo's per-block test convention: every calendar operation resolves to a registered tool in tools.access, supplies all required tool params, emits no params the tool cannot accept, maps one-to-one onto the calendar tools, and the calendarId canonical group and sendResponse default are pinned. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-29 10:20:13 -07:00
5 people	1d64b92b41	feat(desktop): desktop app (#5998) ... * top on a desk * fix auth stuff * intermediate state * update * local filesystem fixes * Huge * fix banner * ci: disable desktop release + e2e in CI for now The desktop-release reusable-workflow call requested contents: write, which ci.yml's permission grant (contents: read) rejects — invalidating the whole CI workflow. Desktop is tested locally for now; signed builds remain available manually via desktop-release.yml workflow_dispatch, and desktop e2e via its own workflow_dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: exempt electron from the release-age gate (time-boxed) electron@43.1.1 (published 2026-07-14) is exact-pinned for the desktop shell and blocked by minimumReleaseAge until 2026-07-21. Excluded with a drop-after date, following the vetted-typescript precedent. Verified the rest of the desktop dependency set clears the 7-day gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: brand app icon (packaged + dev Dock) - build/icon.icns regenerated from public/logo/primary/large.png on the Apple icon grid (824px body, r=185.4, centered on a transparent 1024 canvas), compiled with iconutil - dev runs set the same mark via app.dock.setIcon (static/dock-icon.png) — unpackaged Electron otherwise shows its default atom icon - un-ignore apps/desktop/build: it holds electron-builder INPUTS (icon, entitlements), which the /apps/**/build output rule was swallowing — the icns and entitlements were never actually tracked - revert resetAdHocDarwinSignature fuse: it corrupts the packaged binary signature (app killed at launch on arm64); the local ad-hoc deep-sign flow doesn't need it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * desktop: switch app icon to the b&w brand mark White rounded tile with the black sim wordmark (from public/logo/b&w/large.png), replacing the purple variant. Same Apple icon grid geometry (824px body, r=185.4, 1024 canvas). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix banner * Fix * clean up launcher * fix oauth * update desktop app * Improve browser use and consolidate desktop app * Desktop app ui cleanup * Updates * Updates * remove dev tool option * Browser updates * Fix electron bug * Browser shortcuts * lifecycle * feat(desktop): SSRF hardening + shared @sim/security/ssrf (re-home of #5763) (#5784) * feat: re-home @sim/security/ssrf + sim SSRF dedup onto dev (clean core) * feat(desktop): re-integrate SSRF guard + hardening onto rewritten dev Re-applies the browser-agent SSRF guard and hardening onto dev's evolved desktop files (dev rewrote session/driver/handoff/index and split out errors.ts/keyboard.ts): - session.ts: agent-partition onBeforeRequest is the SSRF choke point — DNS-resolving check (fail-closed) for document navigations, synchronous literal-IP backstop for subresources. - driver.ts: browser_navigate/browser_open_tab validate via checkAgentUrl for a clean model error; also adopt shared sleep/getErrorMessage and drop the local reimplementations + banner separators. - index.ts: local-only crashReporter (native minidumps, no upload) + CSP fallback wired into the app session. - window.ts: record the crash-dump dir on renderer_gone. - config.ts: drop the local LOCAL_HOSTNAMES set for the shared isLoopbackHostname (also removes the dead bare '::1'). - cdp.ts: per-WebContents callbacks so a background tab's events reach its own driver. - updater.ts: the manual check now surfaces network/manifest failures instead of silently swallowing them. - README: correct the App Sandbox / security-scoped-bookmark note. - electron-mock: webRequest.onBeforeRequest + crashReporter stubs. - api-validation: annotate dev's validated-envelope double-cast; bump the route-count baseline 964→965 for dev's already-merged route (ratchets stay tight; non-Zod and double-cast at baseline). Skipped as moot (dev already did them independently): launcher isVisible removal, decideStartRoute param drop, local-filesystem clear() removal. * chore(desktop): biome format install-local.ts (pre-existing dev lint failure) * refactor: apply audit cleanup (reuse + simplify) - domain-check: drop the redundant isIpLiteral guard (isLoopbackIp already validates and returns false for non-literals). - session.ts: use shared getErrorMessage instead of the local error ternary (the file already imports it). - tray.ts: use shared sleep() instead of a hand-rolled setTimeout promise. - updater.ts: distinguish the synchronous-throw log from the async-rejection log on the manual update check. * refactor: /simplify pass + review fixes - url-guard: bound the SSRF dns.lookup with a 5s deadline (fails closed on timeout) so a slow/hung resolver can't suspend the check and the onBeforeRequest callback indefinitely (Greptile P2); + test. - Finish the reuse consolidation the earlier pass missed: session.ts second error ternary → getErrorMessage; the bracket-strip idiom → unwrapIpv6Brackets in input-validation.ts, input-validation.server.ts (×2), onepassword/utils.ts (fixes the check:utils banned-pattern CI failure). - driver: document why the tool-level checkAgentUrl coexists with the onBeforeRequest enforcement seam (clean model error; loadURL rejection is swallowed). * fix(desktop): swallow late DNS rejection after the SSRF lookup timeout (Cursor) * refactor: split pure host helpers into @sim/security/hostnames (ipaddr-free) (#5787) unwrapIpv6Brackets + isLoopbackHostname move to a new ipaddr-free sub-export so client code can share them without pulling ipaddr.js into the browser bundle. ssrf.ts re-exports both, so its server/desktop consumers are unchanged. This eliminates the duplicate isLoopbackHostname in apps/sim/lib/core/utils/urls.ts: urls.ts and its three client importers (mcp queries, oauth probe, oauth url-validation) now use the single shared definition. * Desktop app fullscreen mode * fix(copilot): report closed browser session as a distinct terminal tool error A dead agent browser session used to answer every browser tool with an indistinguishable generic ~30s IPC timeout, which the model retried indefinitely (one turn: 59 minutes of failing browser_snapshot calls). - When the desktop app has reported the session closed, page-dependent browser tools fail immediately with an explicit session-closed message (and sessionClosed: true in the result data) instead of burning the full timeout per call. browser_navigate / browser_open_tab / browser_list_tabs still run, since they can start a new session. - A failure whose session died mid-call (e.g. during a takeover) gets the same tag appended, so the model learns the terminal cause rather than seeing a plain timeout. Companion to mothership's tool_failure_loop circuit breaker. * fix(desktop): route Cmd+W to focused browser tabs * fix(desktop): reserve macOS title bar safe area * fix(desktop): limit title bar safe area to login * fix install script * feat(desktop): improve local folder settings * feat(desktop): harden local capabilities and window chrome * fix(invitations): live refetches * fix(desktop): make manual update checks use updater state * fix(desktop): review fixes — OAuth error handling, query freshness, invitations Findings from an end-to-end review of the desktop work, fixed and verified. OAuth connect/login handoff: - Add a friendly /oauth-error landing page + onAPIError.errorURL so provider Cancel/Deny (which Better Auth redirects before the flow state is parsed) no longer dead-ends on a 404; re-initiating supersedes the idle loopback. - Stop a post-consent failure from reporting success (drop the baked-in errorCallbackURL param that collided with Better Auth's appended code; coerce an array error defensively on the complete page). - Guard the desktop connect listener with the same context-age check the web routers use, so an abandoned flow can't mislabel a later completion. - Clear an orphaned pending handoff when a loopback re-bind fails. Query freshness (desktop refetchOnWindowFocus): - Pin refetchOnWindowFocus off on queries that seed editable forms (environment/secrets, credential detail, schedules) so a background focus refetch can't drop an unsaved draft, and on the useWorkflowStates fan-out so returning to a large table doesn't fire N heavy envelope fetches. All no-ops on web (default already false). Invitations (in-app pending invitations): - Map accept/decline failures to friendly copy instead of raw machine codes. - Invalidate subscription + refresh session on accept (parity with the email path); reconcile the list on failure (onSettled) so dead rows drop. - Gate the modal's query on open so it no longer fetches on every app load. CI: - Wrap the latest-mac.yml update-feed route in withRouteHandler and allowlist it as a non-boundary route (input-less, YAML) so the contract audit passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * updates * fix(desktop): use workflow colors for environment icons * fix(desktop): use orange for dev icon border * fix(login): change one time token generation to GET * improvement(desktop): reveal local folders from settings Local-folder rows rendered their glyph at 20px inside the bordered credential tile — chrome meant for brand and logo icons — above a static subtitle that repeated what the section already said. The row now shows a plain 14px folder icon and the folder name alone. Clicking a row reveals the folder in the OS file manager through a new reveal_mount bridge op, which resolves the opaque localfs URI to a live grant and requires an active user gesture, matching the other grant mutations. The absolute host path still never crosses the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * improvement(desktop): row actions menu for folder grants, larger version text Revoke moves from an always-visible chip into the canonical RowActionsMenu, matching the MCP server rows. The version value moves off text-caption onto text-sm — it was rendering at the subtitle size, which also shrank the "x -> y on restart" line that matters most. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * feat(desktop): improve browser tab usability * fix(desktop): thicken environment icon borders * fix(desktop): strengthen environment icon borders * feat(desktop): support multiple windows and harden the agent browser Sim can now open many full windows in one process. The embedded browser is still a single native surface, so exactly one window owns it at a time. Ownership transfers only to the focused window: without that rule, two windows both showing the browser reclaim it on every bounds heartbeat and re-parent the native view back and forth roughly once a second while Sim sits in the background, where no window is focused. A destroyed owner is now forgotten rather than left rejecting updates from the window actually on screen, and a closing window's release is honoured even though Electron destroys it before emitting `closed` — previously that release was dropped and the next layout could re-parent the browser onto a window that never asked for it. The agent's password boundary is now enforced rather than assumed. It was treated as settled but had four ways through: `browser_press_key` sent trusted CDP keystrokes to whatever held focus, `clickElement` focused credential fields, `readActiveElementState` returned a preview of any focused value, and snapshots printed the contents of revealed password fields. Detection also used `instanceof HTMLInputElement`, which is realm-bound and returned false for inputs inside same-origin iframes — the nested login forms that need it most. Detection now matches on tagName/type/autocomplete, the keystroke guard runs in the driver where trusted CDP input is visible, and typing re-checks the real target before inserting, since login forms advance focus between the username and password steps. Signing out clears the embedded browser's profile. Its cookies, cache, pinned tabs, browsing trail, and reopen list all survived sign-out, so the next account on the machine inherited the previous user's live sessions. Partition hardening is keyed per session instead of a process-wide flag, which would have left a second partition with no permission handlers, no SSRF filtering, and no download blocking — silently, and still type-checking. Adds the first tests for page-functions.ts, including a serialization contract check: those functions ship to the page as String(fn), so a reference to module scope passes every other test and fails only against a real page. * fix(desktop): close clipboard, glob DoS, and authorization holes Found by a full audit of the desktop app against origin/staging. Each of these was measured or asserted rather than reasoned about. The agent could read the user's system clipboard. `browser_press_key('Cmd+V')` pasted it into a focused field and the next `browser_snapshot` returned it as an ordinary `value` — snapshots redact password fields, not pasted content, and clipboards routinely hold a password copied out of a manager. The credential guard added earlier did not catch it: `insertedTextFor` returns undefined whenever `meta` is set, so `Cmd+V` was classified as not text-inserting. `Control+V` reached the same place because the macOS normalizer rewrites it. Clipboard combos are now refused before dispatch rather than by withholding the CDP `commands` array, since off macOS these are Blink-native and a key event alone still performs them. Copy and cut go too — they clobber the user's clipboard as a side effect. A glob pattern could freeze the whole app. Micromatch compiles to a backtracking regex whose cost is exponential in wildcard count: measured against a single 46-character path with the options this code passes, ten wildcards took 2.7s and twelve took 43s, once per scanned entry, in one synchronous call that the surrounding abort checks never get to interrupt. That is the main process, so every window, the menu bar and the tray freeze with Force Quit as the only recourse, and the pattern is model-supplied. `safeRegex` reports the generated source as safe, so it was no defense. Patterns are now bounded at six wildcards, which keeps the worst case near 2ms while leaving headroom over real patterns (which top out around four). A timing probe backs it up, with a budget loose enough that JIT warmth and machine load cannot make it fire on a legitimate pattern — a tight budget proved flaky in both directions. The grep authorization guard compared `request.pattern !== args.pattern`, so a tool call carrying no pattern made that `undefined !== undefined` and the guard passed — grep then fell back to searching the renderer's own `query` across the whole grant. `include` and `query` were never bound at all, letting a renderer widen a search or silently narrow results the agent believes are complete. The sibling glob case already had the `typeof` check, which is what made the asymmetry clearly unintentional. The IPC sender gate used `startsWith`, the exact pattern `isAppOrigin` warns against 200 lines away ("that prefix-matches lookalike hosts"). It was safe only because of a trailing slash. It now uses that helper, which also fixes a false negative on an explicitly stated default port. * fix(desktop): stop double sign-out, stranded retries, and redundant writes Three correctness bugs from the same audit. Menu Sign Out tore down the session directly instead of going through the lifecycle coordinator, so it skipped the in-progress guard — and its own cookie removal then tripped the coordinator's cookie watcher into a second concurrent teardown, duplicating the sign_out event, the storage clear, and the /login load. Teardown also existed as two divergent copies. The coordinator now exposes `signOut()` and owns the single path; the menu just calls it. That `tearDownSession` is no longer imported in index.ts is the check that it landed. Offline recovery could strand permanently. The auto-retry loop stops itself before calling `retry()`, and `retry()` never re-armed the load watchdog, which is started once per window. So if a retried load hung — precisely what the watchdog is for — no load event fired and no timer remained anywhere; the user sat on the offline page until the window was closed. `retry()` now re-arms before loading. Pinned tabs were persisted on `did-navigate` and `did-navigate-in-page` for every tab, pinned or not, with no change check, and the settings store compares with `===` so a freshly built array never matched. Any single-page app therefore triggered a synchronous mkdir + write + rename of the whole settings file on the main thread on every route change — writing `[]` over `[]` when nothing was pinned. The list is now fingerprinted, seeded at restore from what is already on disk so the first navigation after launch is not a write either. * fix(desktop): leaked timers, silent grep failures, and crashed tabs Second pass on the audit backlog, all verified against tests that fail without the change. Every browser tool call leaked a timer. The watchdog raced the tool against `sleep()`, which cannot be cancelled, so when the tool won — the normal case — the timer stayed pending for the full window, up to two minutes, dozens deep during an agent run. Replaced with a cancellable timeout cleared in a `finally`; a test asserts the fake-timer count is unchanged across a call. An invalid grep regex reported "no matches". A SyntaxError from `new RegExp` returned an empty result set, which tells the model the string appears nowhere in the user's files — a factual claim it acts on, when the search never ran. It now fails as INVALID_REQUEST. The `safeRegex` guard moved out of the try while there, since it was only inside it to be re-thrown. A crashed tab wedged the session. Tabs left `tabs` only via close, so a dead renderer stayed forever: `activeTab()` filtered it out while `activeTabId` still named it, making `requireTab()` report "no page is open" with other tabs open, and the panel went blank with no recovery. `render-process-gone` now drops the tab, advances the active id, and reports session closure when it was the last. `probeSession` cleared its abort timer inline after the await, so a thrown fetch — the case the function exists for — skipped it. Moved to `finally`, which also brings the body read inside the deadline. One vanished file failed a whole directory listing: `Promise.all` over per-entry `lstat` turned a single ENOENT into NOT_FOUND for the directory. Churning directories like build output would intermittently fail to list. Removed the `session-lifecycle -> browser-agent/driver` import edge, which dragged the entire browser subsystem and its module-load `nativeTheme` listener into the auth path to reach one four-line function. `clearBrowserProfile` is now a required dependency wired from index.ts, which already owns both sides. Also deleted `attachSessionLifecycle`, a compatibility wrapper with zero callers. Added a channel-parity test between the preload bridge and the IPC table. They share ~20 channel names as bare string literals with nothing tying them together, so a typo on either side is a silently dead feature that type-checks and ships. Verified it fails on a one-character change. * fix(desktop): reach framed elements and harden the loopback sign-in Two behaviour fixes from the audit backlog. Interaction with same-origin iframes was broken. The snapshot deliberately walks into those frames and hands the model ids for what it finds, but every interaction then tested `instanceof HTMLInputElement` against the top frame's constructors — false for nodes owned by a frame, because element wrappers are realm-bound. So the driver reported a real `<input>` as "not a text input", which took out framed login forms and editors that put a contenteditable body in an iframe, such as TinyMCE. Framed selects reported "not a select" and framed clicks skipped focus entirely. Checks now compare `tagName` or duck-type the method being called, matching the realm-safe approach the credential guard already used. The native value setter is taken from the element's own realm: calling the top frame's setter on a frame's node throws "Illegal invocation". Snapshot value reporting follows the same rule, which is safe because the credential redaction above it is realm-safe and runs first. The loopback sign-in server could be cancelled by anything on the machine. It validated only the shape of the returned state, then tore the one-shot server down and dispatched, leaving the real constant-time comparison to the callback. So a request carrying any well-formed state killed an in-flight sign-in — and the port is reachable by any local process and by any page the user has open via a no-CORS GET, which cannot read the response but does not need to, since the side effect is the kill. The state is now checked before anything is torn down, and a Host that does not name the loopback is refused, which closes the DNS-rebinding shape. * refactor(desktop): drop duplicated helpers and stop logging query strings Net -3 lines, and one of them was a real leak. `navigation.ts` and `windows.ts` truncated URLs for their log lines with a bare `.slice(0, 200)`, which keeps the query string — the five other log sites in the app go through `scrubUrl` for exactly that reason. Tokens and signed parameters live in query strings, so a blocked-URL warning could write one to disk. Both now scrub. `local-filesystem.ts` carried a private `isRecord` byte-identical to `isRecordLike` in `@sim/utils/object`, and four more sites inlined the same check. All now use the shared helper, which also tightens three of them: the inline versions omitted the array exclusion, so an array satisfied a check that then cast it to a record. `tray.ts` hand-rolled slice-plus-ellipsis, the case `@sim/utils/string`'s `truncate` exists for. Titles between 58 and 60 characters now get an ellipsis where they previously did not — cosmetic, in a tray menu label. Removed the `getTabsState` passthrough in the driver, a one-line re-export of the session's own function, and renamed the session-level clear to `clearProfileStorage`. `clearBrowserProfile` existed twice under one name, the driver's being the composite that also clears the browsing-trail registry; index.ts was already aliasing at the import to tell them apart. Two things deliberately not done. The hand-rolled semver in updater.ts stays: replacing it needs `semver` plus `@types/semver` as new declared dependencies in the Electron main process, and the 90 lines it would delete are already covered by eight assertions that I verified match the library's behaviour case for case. Note the same prerelease comparison is duplicated in apps/sim/lib/desktop/min-version.ts, so a future consolidation should do both. No barrel for browser-agent either: routing `security-guards.ts` through one to reach a single leaf function would pull the whole browser subsystem into its module graph, which is the edge just removed from session-lifecycle. * refactor(desktop): move browser compositing out of the session module session.ts held five responsibilities in one flat namespace: 1,061 lines, 29 exports, 26 mutable module-level bindings. For contrast local-filesystem.ts is a comparable 1,125 lines with two exports and no ambient state — size was never the problem, the shared mutable namespace was. Compositing is the part worth isolating. Where the native view sits, when it is visible, which window owns it, the renderer bounds lease, and the occlusion snapshot are the most intricate logic in the browser and are almost entirely separable from tab bookkeeping. They now live in panel.ts (342 lines) and session.ts is 792, with 15 bindings instead of 26. The two modules were mutually dependent, which is what makes this kind of split go wrong. Rather than events or a shared store, panel.ts takes the four things it needs from the session through one PanelHost passed to initPanel — the same shape as the existing initSession — so the import graph is one-way and there is no new indirection to trace. Tab changes reach the panel by the session calling layout(), exactly as before. Two behaviours became explicit rather than implicit in the move: detachIfAttached replaces callers reading `attachedView` to decide whether a closing tab owns the surface, and isPanelVisible replaces `panelBounds !== null`. Nothing about the split is verified by the split itself, so the bounds lease got characterization tests first. It had none — there was not a single fake timer in the suite — despite being the mechanism that hides the view when the renderer crashes or wedges. Both tests were confirmed to fail against a broken lease before the refactor began. The other 47 tests were not rewritten: only the module their calls address changed, which is the useful signal that behaviour was preserved. Deliberately not split further. Focus tracking stays with tabs because it keys off tab ids, and profile teardown stays put; separating either would be taxonomy rather than decoupling. * refactor: drop the legacy local_* filesystem tool shim Granted folders are addressed through the ordinary VFS: the model calls read/grep/glob against paths under user-local/, exactly as it does for workspace files. A parallel local_read / local_grep / local_glob / local_list / local_stat / local_mount_directory / local_list_mounts / local_forget_mount / local_stage_file toolset existed alongside it, recognized but never advertised, so an in-flight checkpoint written by an older desktop build could still finish. There are no older desktop builds. apps/desktop is at version 0.0.0, the only artifacts are a local 0.0.0 build, MIN_DESKTOP_VERSION is '0.0.0' meaning no floor, and the app does not exist on staging at all — the v0.7.x tags are the web app's. Nothing can have persisted a checkpoint naming these tools, and nothing advertises them: they are absent from the generated tool catalog and from mothership's catalog. The shim was defending against a past that never happened. Removes the name table, the legacy request builder, the server-side LEGACY_READ_ONLY_TOOLS allowlist, the five local_* branches in the desktop authorization switch, and nine display labels. isDesktopFilesystemToolCall collapsed into isUserLocalVfsToolCall, which it had become a synonym for. Two tests went with it. One asserted that local_list_mounts routes to the desktop; the test immediately after it already covers the real path, an ordinary read against a user-local path. The other asserted that legacy names cannot open a folder picker, revoke a grant, or upload bytes — that property now holds because no such tool name exists, which is a stronger guarantee than refusing one. * refactor(copilot): remove the plan/changelog VFS artifacts and workflow aliases These beta surfaces are not a direction we are taking, so they come out rather than staying behind a flag. Gone: the workflow alias modules (path resolution, DB-backed resolver, .plans/.changelogs backing provisioning), the alias materialization in the copilot VFS, the alias write paths in resource-writer and workspace_file, the sandbox alias mounts in function_execute, the reserved backing-path guards across mkdir/mv/create, and the alias resolution in the chat home file picker. xlsx survives but changes owner. It was gated twice across the repo boundary: mothership's xlsx-writing flag gates the skill and prompt, while Sim gated the compile path on mothership-beta. Those live in separate AppConfig applications, so an operator had to flip two flags in two consoles, and off-hosted Sim fell back to the MOTHERSHIP_BETA_FEATURES secret while the mothership half stayed in Sim Cloud's AppConfig — split-brain across an ownership boundary. Mothership controls whether the model ever learns xlsx exists, so if it is never offered it is never requested and the second chokepoint only created a way for the two halves to disagree. Sim's gate is removed; xlsx-writing is now the single owner. With its last consumer gone, the mothership-beta flag and the MOTHERSHIP_BETA_FEATURES secret are deleted. The two entries in the infra repo are harmless until removed separately: they only inject an env var nothing reads, and createEnv runs with skipValidation. The reserved-system-file/folder concept goes with the aliases, since it existed only to hide the backing rows. includeReservedSystemFiles and includeReservedSystemFolders are removed rather than left as options every caller passes true to. backingVfsPath is removed for the same reason — nothing sets it once aliases are gone, so it was an always-undefined field on tool results. Test coverage is preserved rather than deleted with the feature. resource-writer.test.ts looked alias-only but three of its eleven cases cover the generic create path that survives; those are kept and the file retitled. Two open_resource tests and one output-path test used alias-shaped strings while asserting generic behavior; retargeted or dropped where a sibling already covers it. * refactor(copilot): remove the dead planArtifact column plumbing copilot_chats.plan_artifact has no writer and no reader that does anything with it. No client sends it, nothing renders it, and its whole history is fork-chat and duplicate-chat plumbing faithfully copying a column that is always null — the one change that might have populated it (mothership v0.8) was reverted. Removed from the schema, the copilot API contract, the chat lifecycle column sets, the fork route, superuser import, the data drain, the update-messages write path, and the legacy chat detail response. No migration here on purpose. The column stays in the database, orphaned and null; dropping it is a separate deliberate step rather than something that rides along with a code cleanup. Note that the next drizzle-kit generate will now want to emit the DROP COLUMN, and check-migrations-safety will ask for it to be annotated — that is the right moment to decide, not now. Mothership never saw this field; it is Sim-side only. * chore(copilot): sync the tool catalog for load_skill Picks up the new load_skill tool plus the grep description that dropped its stale reference to VFS "plans" entries. Generated from copilot/contracts/tool-catalog-v1.json. * refactor(copilot): follow the load_custom_tool rename to load_mcp_tool Mothership renamed the loader once it was clear MCP was the only catalog kind it could match, and dropped the single-valued `type` parameter. The two prompt strings that teach the model the call shape are updated to load_mcp_tool({ name }). load_custom_tool stays in the UI hide-list next to load_agent_skill so tool rows in historical transcripts keep rendering; nothing emits it any more. * chore(copilot): sync the tool catalog and hide load_skill in the UI load_integration_tool and list_integration_tools now publish route go/sync instead of sim/async. Nothing changes in Sim's behavior — they always ran in Go; the contract had been wrong. load_skill joins the hidden tools. It is the same shape as the other loaders already there: the agent pulling in a reference guide before doing the work is a step toward the action, not the action. Sim's display-coverage test caught that a newly added visible tool had no title or completed verb, which is the guard working. * fix(auth): handle session expiry in the app, not the desktop shell The workspace auth gate is a Server Component, so it only re-evaluates on a server render. A session that expired or was revoked mid-visit left the SPA mounted and silently 401ing every request, with nothing to redirect it. The desktop shell had grown its own detector for this: a 401 listener over /api/*, a session probe, and a native "your session has expired" prompt. It could only infer session state from cookie events and HTTP statuses, and it inferred wrong — it fired on ordinary sign-outs (in-flight requests 401 during teardown) and on launching already signed out (the window still shows the restored route while the web app redirects). Those were nearly all of its firings, since a 30-day sliding window means real expiry is rare. Generalizes the impersonation-expired screen instead, which already had the right shape: it keys off the session query settling to null after a session that was live. A signed-out visitor never arms it, and `error` is excluded so an offline blip cannot read as an expiry. The session query now refetches on focus for every session, not just impersonation ones, so returning to a window that slept through its session re-checks it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(copilot): port the scheduled-task and VFS fixes onto staging-v4 Replays the sim-side prompt-audit work on top of staging-v4. complete_scheduled_task was filtered out of the execute route's response payload, so an until_complete job could report completion and still be rescheduled; the post-run bookkeeping now also refuses to revive a job that already completed. Also clamps browser_wait_for's timeout the way the desktop agent does, and replaces the oversized-read error's offset/limit advice, which sent the model into a guaranteed retry loop. * feat(desktop): let the model actually see browser screenshots browser_screenshot captured an image and then threw it away. The renderer stripped the data URL and substituted a note, and the tool's own description told the model not to bother: "Dead end for perception." So the agent was blind to anything not expressible as DOM text — canvas, charts, maps, images, rendering and layout bugs. The copilot has carried the machinery for this all along. A tool result shaped as { content, attachment: { type: "image", source: { type: "base64", ... } } } is serialized into a real image content block, with the media type sniffed from the bytes rather than trusted from the declaration, and degraded to a text stub when the routed model has no vision so the provider never 400s. The screenshot result is now reshaped into that contract instead of discarded. A malformed data URL still falls back to a note rather than shipping an attachment the provider would reject. Captures are bounded to a 1024px longest edge at quality 70. CDP clip.scale is relative to CSS pixels, so this also sidesteps the device pixel ratio — an unclipped capture on a retina display returns a 2x image, which was several hundred kilobytes for no legibility the model could use. The description is rewritten to bias toward visual questions only: appearance, layout, rendering, charts, canvas. Reading content or finding something to click stays with browser_snapshot, which is cheaper and returns the element ids a screenshot cannot. That distinction is structural, not just advisory — having seen the page does not let the agent act on it. Companion change in mothership generalizes the tool-result inline-budget exemption from "the read tool" to "any result carrying a model attachment". Keyed on the tool name, an oversized screenshot fell through to the artifact branch: the image was replaced by a reference the model cannot open, and the result still reported success. Silent, and it would have hit almost every call. * fix(desktop): polish browser panel and environment tray icon * fix(desktop): enlarge environment tray markers * fix(desktop): smooth environment tray markers * refactor(copilot): consolidate resource mutation tools * chore(copilot): clean up VFS follow-ups * fix(desktop): round the dev tray marker * feat(desktop): add integrated terminal resources * Fix electron app resize causing glitchy browser frames * feat(copilot): add persistent tool permissions * fix(copilot): retire stale tool permission prompts * fix(desktop): keep terminal rendering responsive * fix(desktop): preserve resource rendering continuity * feat(desktop): add browser tab duplication actions * feat(desktop): add terminal tab context actions * fix(desktop): allow browser agent localhost navigation * feat(desktop): add tmux-backed terminal sessions * fix(desktop): restore terminal scrollback per view * chore(copilot): sync updated wait tool contract * poll terminal session state for non regular shells * add terminal right click menu * feat(desktop): add terminal handoff and key batching * fix(desktop): reserve the traffic-light lane from the platform macOS draws the window controls itself, at a fixed physical size, above all web content. The page renders full-bleed beneath them, so it has to reserve that lane — and it did so with five hardcoded CSS pixel values. CSS pixels scale with page zoom and the OS-drawn lights do not, so zooming out shrank the reservation until the lights were drawn over the sidebar toggle, and the header row below sat inside their band. Electron's `titleBarOverlay` publishes the controls' real geometry to the page as the `titlebar-area-*` env vars, which Chromium rescales per zoom so a reservation derived from them holds its physical size. Measured across zoom 0.58-1.2, the reserved area stays within ~0.6 DIP, the residual coming from env values being quantized to whole CSS pixels. Every lane length now derives from those vars, so the login route and the mothership content offset were fixed without being touched — they already read `--desktop-title-bar-height`. Two of the replaced constants were also simply wrong: the platform reports the lane at 38px and the safe area at 81px, against the hand-measured 36 and 83. The toggle keeps a constant physical size beside the lights, expressed as a proportion of the lane rather than in pixels: a px literal would scale with zoom, and calc cannot divide a length by a length to recover a scale factor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C54QHj4WPV777Fq2yRwkcb * fix(desktop): avoid transient terminal tab labels * feat(copilot): attach browser and terminal tab context * feat(desktop): close tmux panes from terminal tools * fix(desktop): keep terminal tab icons stable * add right click to browser and cleanup terminal right click options * fix(desktop): reduce hidden panel background work * perf(desktop): shrink browser panel snapshots * perf(desktop): reduce terminal main process overhead * perf(terminal): pause work for hidden sessions * fix session arch for desktop * fix(desktop): replace exited terminal sessions * feat(copilot): persist desktop resources across chats * fix(emcn): keep resource tab widths consistent * fix(copilot): restore active client panels * feat(desktop): import Chrome browser data * fix(copilot): close resources before chat creation * feat(desktop): suggest imported browser sites * fix(desktop): autofill identifier-first sign-ins * fix resizing issues + cookies source * fix visits marking * chore(db): drop branch migrations ahead of staging merge 0264/0265 on this branch collide with staging's 0264-0270 on both the journal idx slots and the meta snapshot filenames. Reverting the migration artifacts to the merge-base lets staging's chain merge cleanly; schema.ts keeps the copilot changes and drizzle-kit regenerates a single migration on top of 0270 after the merge. Co-Authored-By: Claude <noreply@anthropic.com> * feat(db): regenerate copilot tool-permission migration on top of staging Replaces the branch's old 0264/0265 (dropped pre-merge so staging's 0264-0270 chain could apply cleanly) with a single 0271 generated against staging's schema: the permission-decision enum, the two copilot_async_tool_calls decision columns, and copilot_chats.auto_allowed_tools. Deliberately does NOT drop copilot_chats.plan_artifact. The branch removed every reader, but the currently-deployed code still SELECTs that column, so dropping it in the same deploy breaks the old app version during blue/green overlap — `check:migrations` flags it for exactly this reason, and the honest fix is to defer rather than annotate around it. The column is retained in schema.ts marked @deprecated; drop it in a follow-up once this has rolled out. Also in this commit, all fallout from the merge itself: - pinned-fetch/revoke tests: their private-IP stub moved to @sim/security/ssrf alongside the source change. Worth noting the stub exists because the suite's 203.0.113.10 is TEST-NET-3, which the real classifier correctly calls reserved — the old stub had been quietly disagreeing with production. - materialize-file test: dropped the reserved-system-folder case, which covered the workflow-alias backing folders this branch deleted. - api-validation route ratchet 977 -> 983 (this branch's new routes). Co-Authored-By: Claude <noreply@anthropic.com> * add cmd f * review pass * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0271, so the snapshot and journal would conflict add/add. Ours is plain additive DDL that drizzle regenerates from schema.ts; staging's is a hand-written CONCURRENTLY index build that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * chore(db): drop branch migration ahead of staging merge Both sides independently claimed idx 0272, so the snapshot and journal would conflict add/add. Ours is plain additive DDL (one enum, two columns, one jsonb default) that drizzle regenerates from schema.ts; staging's is a hand-written migration with DO blocks and CONCURRENTLY index builds that cannot be regenerated. Dropping ours and re-generating on top of staging's is the only order that preserves both. schema.ts is deliberately untouched — it is the source of the regeneration. Co-Authored-By: Claude <noreply@anthropic.com> * style(db): biome-format the regenerated migration metadata drizzle-kit emits _journal.json and the snapshot with expanded arrays, which biome check rejects. The merge commit used --no-verify, so lint-staged never formatted them and CI's lint step failed on exactly these two files. Whitespace only — both files are byte-identical under `jq -S -c`. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): pin the platform in the OS-auth tests promptForSecret gates Touch ID on process.platform === 'darwin'. The suite mocked electron's systemPreferences but inherited the runner's real platform, so the eight biometric expectations passed on a Mac and failed on Linux CI, where every call fell through to the confirmation dialog instead. Pins the platform per-test and restores it after, and adds a case for the gate itself — the branch whose absence from the suite is what let this through. Co-Authored-By: Claude <noreply@anthropic.com> * fix(desktop): refine environment dock icons * fix(desktop): align packaged environment icons * fix(desktop): keep packaged dock icon rendering consistent --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theo@sim.ai>	2026-07-28 19:25:59 -07:00
Vikhyath Mondreti	c809845b99	improvement(self-host): enterprise features enabling (#6028) ... * improvement(self-host): enterprise features enabling * chore(helm): bump chart to 1.3.0 for the enterprise self-host values values.yaml gained the ENTERPRISE_ENABLED switch and INSTANCE_ORG_* keys, and the feature-flag envDefaults moved from "false" to empty so the master switch can resolve them. Additive and backward compatible, so a minor bump. * fix(self-host): address review findings on instance org and org delete Drop the per-process instance-org id cache. It went stale once the organization was deleted through the Admin API, and clearing it from the delete handler would only heal the replica that served that request. The lookup runs on the signup path against a single-row table, so re-reading costs nothing and keeps every replica self-correcting. Scope the org-delete subscription conflict to entitled statuses. Matching any row regardless of status let a canceled subscription — which bills nobody — permanently block deletion. * fix(admin): block org delete on any live subscription, not just entitled ones ENTITLED_SUBSCRIPTION_STATUSES excludes trialing, so a trial — which grants no entitlement but is a live Stripe subscription that will convert — slipped past the delete guard and could be stranded against a removed organization id. Adds TERMINAL_SUBSCRIPTION_STATUSES and inverts the predicate: block unless the row is finished. Expressed as the terminal set so a status Stripe adds later defaults to blocking, which is the safe direction for a destructive operation. * fix(self-host): resolve SSO and access-control in the UI, not the raw env var Nine client consumers still read NEXT_PUBLIC_SSO_ENABLED / NEXT_PUBLIC_ACCESS_CONTROL_ENABLED directly while the server gates and settings nav had moved to the resolver. With only ENTERPRISE_ENABLED set that produced dead ends: the SSO settings section appeared but ssoClient() was never registered and no login button rendered, and the Access Control section appeared but its page reported "not entitled". Points every consumer at isSsoEnabled / isAccessControlEnabled so visibility and capability come from one place. * fix(admin): validate retention workspace targets on the Admin API too retentionOverrides and per-workspace PII rules both name a workspace, and neither field is a foreign key. The settings UI rejected ids belonging to another organization; the Admin API did not, so the two paths could persist different data for the same org. Extracts the check as getForeignWorkspaceTargetsReason and points both routes at it, so they cannot drift apart again. * fix(self-host): close three review findings on admin routes and cleanup Make org delete atomic. detachOrganizationWorkspaces committed on its own, so a failed delete left workspaces detached and re-billed while the organization, its members, and its settings survived. Adds a Tx variant so both commit together. Gate the admin session-policy PATCH on entitlement, matching the settings UI. Without it the stored policy was inert — getSessionPolicy resolves to no-op when the feature is off, so the one eager clamp would be undone on the next refresh. Stop emitting plan-wide housekeeping when billing is off. It is keyed to the hosted free-tier 30-day window, the same default the per-workspace pass deliberately refuses to apply off-hosted. * fix(admin): gate whitelabel on entitlement and emit detach audits post-commit The Admin whitelabel PATCH skipped the entitlement check the settings UI runs, so an admin key could set branding the product had not granted the org. detachOrganizationWorkspacesTx also wrote its audit rows inside the caller's transaction, contradicting its own doc comment — a rolled-back delete would have left audit history describing detachments that never happened. It now returns the rows and callers emit them after commit. * fix(self-host): refuse instance-org resolution when the slug is ambiguous organization.slug has no unique constraint, and the lookup took the first of however many matched. The choice is unordered, so two replicas could resolve different organizations and split new signups between them. Resolution is now three-state. Ambiguity is distinct from absence, so it both declines to adopt an arbitrary organization and declines to provision another one on top of the duplicates.	2026-07-28 19:02:41 -07:00
Waleed	cb3611bad2	feat(folders): add resource pinning and generalize the folders contract (#6014) ... - Adds per-user pinning for workflows, files, knowledge bases, and tables: new `pinned_item` table, `/api/pinned-items` routes, React Query hooks, and a shared `PinButton` wired into Tables, Knowledge, and Files with pinned-first ordering - Adds the generic `folder` table with an idempotent, replay-safe, collision-aware backfill from `workflow_folder` and `workspace_file_folders`; no cutover yet, folder reads still go through a documented legacy adapter - Moves `folderSchema` to the generic vocabulary (`resourceType`, `deletedAt`) and drops the unused `color`/`isExpanded`	2026-07-28 14:42:11 -07:00
Waleed	ca13cc9c13	feat(api): add workflow export and import endpoints to the public v1 API (#5999) ... * feat(api): add workflow export and import endpoints to the public v1 API Adds GET /api/v1/workflows/[id]/export and POST /api/v1/workflows/import. The export envelope is accepted verbatim by import, so workflows round-trip between workspaces over the public API. Unlike the admin export, the public export is secret-sanitized: stored credentials and password fields are redacted while {{ENV_VAR}} references and block positions are preserved. Import regenerates block, edge, loop and parallel ids and de-duplicates the workflow name against the target folder. Also moves parseWorkflowVariables out of the admin types module into lib/workflows/variables/parse.ts so the public route does not import from the admin namespace. * fix(api): make workflow import atomic and clarify what export redacts Writes the imported graph and its variables in a single transaction and deletes the shell workflow row on any failure, so a caller that receives an error is never left with a partially imported workflow. Previously a throw from the variables update returned 500 while leaving the workflow behind with an empty variables map. Also narrows the export route's sanitization claim: workflow variables are emitted as stored, matching GET /api/v1/workflows/[id] and the in-app export. They are plaintext configuration readable at the same permission level this route requires; secrets belong in environment variables, which travel as unresolved references. * fix(api): close import defects found in audit and share one write pipeline Security: - Escape block names before interpolating them into a RegExp in updateValueReferences. Names reach it straight from imported workflow JSON and normalizeWorkflowBlockName preserves regex metacharacters, so a name like `a*a*a*a*b` compiled to a catastrophically backtracking pattern. A sub-kilobyte body blocked the event loop for 50s and grew exponentially. Also skip rename-to-itself, which is the entire map on the import path, so the scan no longer runs at all there. - Validate folder ownership before folder lock state, so a locked folder in another workspace can no longer be distinguished from a missing one. Correctness: - Gate the imported graph on workflowStateSchema, the same schema the canonical PUT /api/workflows/[id]/state path enforces. Without it a valid 201 could persist a block field of the wrong type, which then threw on every subsequent read and left a workflow nothing could open. - Guard the compensating delete so a failed rollback logs the orphaned id instead of vanishing into a generic 500. - Validate variable `type` against the enum and build the record on a null-prototype object, so a `__proto__` key no longer silently drops the variable. - Bound payload-derived names and descriptions to the same limits the contract declares for the explicit overrides. - Return the description as stored rather than coercing '' to null, matching GET /api/v1/workflows/[id]. Shared code, so the two write paths cannot drift: - Extract prepareWorkflowStateForPersistence and use it from both PUT /api/workflows/[id]/state and the v1 import route: agent-tool sanitization, block backfill, dangling-edge removal, and loop/parallel recomputation now have one implementation. - Persist inline custom tools on import, which the canonical path already did. - Move variable normalization into lib/workflows/variables and repoint the admin importer at it, removing the last duplicate. Docs: - OpenAPI: oneOf -> anyOf on the import body. WorkflowExport matches any object, so every valid object payload matched two branches and failed validation under any spec-driven validator. Document 423 and the loss of workspace-scoped bindings on export. Tests: prepare-state unit tests and a real export -> import round trip with no mocks of the sanitizer or parser, covering loop/parallel children and the regex-metacharacter payload. * fix(api): cap import names inside the bound and align the three import paths - `truncate` appends its suffix after slicing, so capping at the contract limit produced 203/2003-character values — past the very bound the cap exists to enforce, and into the headroom reserved for dedup suffixes. Reserve the ellipsis inside the limit. - Match `extractWorkflowName`'s candidate order (state.metadata.name before workflow.name) and trim, so the v1 API and the in-app importer resolve the same name for the same payload. Previously a hand-authored payload carrying both could yield two different names. - Run the admin importer through prepareWorkflowStateForPersistence too. It was writing raw parsed state, so a dangling edge tripped the workflow_edges foreign key and a block missing its backfilled columns could land unopenable — the same class this PR just closed on the v1 path.	2026-07-27 22:50:13 -07:00
Theodore Li	19c3b6f47d	feat(setup): setup wizard with browser-based Chat key handoff (#5911) ... * feat(setup): setup wizard with browser-based Chat key handoff Adds `bun run setup` and `bun run doctor` for local installs, and replaces the wizard's paste-your-Chat-key step with a browser handoff that never puts the key in a URL. * improvement(setup): drop the paste-a-key fallback, simplify consent copy The browser handoff is now the only path — the wizard waits on a spinner instead of racing a paste prompt. Consent card leads with "Connect your terminal" and moves the match-the-code disclaimer into the description. * fix(setup): pin kube context, keep secrets out of argv, validate reused keys Review findings from #5911: - helm/kubectl now run against the validated context instead of the ambient one - helm values are piped on stdin rather than passed as --set arguments - ENCRYPTION_KEY/API_ENCRYPTION_KEY are checked for the 64-hex format the app requires, not just length, so an unusable key is replaced rather than kept - the managed Redis container's published port is read back instead of assumed * refactor(copilot): one module for Chat API key operations list/generate/delete each repeated the same /api/validate-key envelope in their route. They now share callValidateKey in lib/copilot/server/api-keys.ts, which also keeps the display masking server-side so the full key can only ever leave at creation. * improvement(setup): reuse shared helpers, parallelize probes, drop dead code - PKCE verifier/state/pairing code now use generateSecureToken, generateRandomHex and generateShortId instead of hand-rolled randomBytes; the pairing loop's modulo was unbiased only because 256 % 32 == 0 - new sha256Base64Url in @sim/security/hash so both sides of the PKCE exchange derive the challenge from one implementation - isUsableSecret moved beside SECRET_KEYS so setup and doctor apply the same rule; doctor previously passed a key setup would replace - isTruthy narrowed to true/1, matching the app it claims to mirror — it accepted yes/on, so a flag could read on in doctor and off in the app - checkLive runs its five probes concurrently (~17s serial worst case) - detection overlaps the banner animation instead of queueing behind it - glyph.fail/glyph.warn at 13 sites that bypassed the constant; removed unused prompter exports, a dead ENV_PATHS re-export, and an unused export keyword * fix(setup): make doctor understand the compose env layout Compose writes a single root .env (what docker-compose reads via env_file) but the checks required the three per-app files, so a successful compose install was followed by doctor printing three failures and exiting 1 — and the whole coherence catalog was skipped because it keyed off apps/sim/.env existing. Layout is now derived from what's on disk and every check consults it: file and schema checks iterate the layout's targets, consistency reports skip when there's only one file to mirror, and coherence/live read the layout's primary file. The wizard's existing-config detection counts root for the same reason — a compose install used to read as unconfigured and re-run from scratch. * feat(cli-auth): device-authorization poll flow, drop the loopback listener The CLI no longer binds a local port. It generates a request id + poll secret, opens /cli/auth, and polls /api/cli/auth/poll over TLS while the user approves in the browser — so the flow works over SSH and inside containers, where the browser and terminal don't share a machine. - approve stores the approval keyed by request id (session-authed, userId from the session only); poll verifies the secret before an atomic claim, so an observer of the semi-public request id can neither mint nor cancel it - pairing code stays as the anti-phishing compare; no key ever crosses the browser; done page just confirms - removes the loopback listener, /token exchange, buildCliHandoffUrl, and validateCliCallbackUrl (+ its tests) — nothing hands a key to a URL anymore * fix(setup): reuse an existing managed Postgres container instead of colliding A running sim-postgres fell through to `docker run --name sim-postgres` and died on the name conflict; a stopped one failed with "no DATABASE_URL to reach it" because the generated password only lived in the env files a fresh clone lacks. Both facts are recoverable from Docker: the ladder now reads the published port and password back via `docker inspect` and reuses the container (starting it if stopped). A container that won't answer prompts before recreating, and never drops the data volume silently. * improvement(setup): audience-first run-mode hints Each run mode now names who it's for — compose for self-hosting/evaluating, dev for contributing to Sim, k8s for rehearsing a production deploy — with the live detection state (Docker/kube/VM) appended. * fix(cli-auth): retry a failed mint, port container/port fixes to Redis + k8s Review findings from #5911: - poll now reserves the mint with an atomic NX lock instead of deleting the approval up front, so a failed mint (e.g. mothership blip) is retried by the next poll instead of forcing a fresh browser approval; the lock still prevents a double-mint and its TTL frees the slot if the caller dies - setup reuses/recreates an unhealthy managed sim-redis instead of colliding on the name (Redis has no data volume, so it removes and recreates without a prompt) - k8s failure-path hints carry --context, matching the success-path hints, so a changed ambient context can't send diagnostics to the wrong cluster - compose port-free waits for a killed port to actually release before re-checking; SIGKILL is async, so the immediate re-check re-saw the port * fix(setup): harden mint cleanup, Windows browser, container detection, helm cwd Review findings from #5911: - a post-mint completeApproval failure no longer routes into releaseMint — the mint lock now outlives the approval (shared TTL), so a cleanup blip can't leave a re-mintable window and orphan a key; cleanup is best-effort after the key ships - compose doctor --fix writes the feature-flag twin to the layout's primary env (root .env on a compose install), not always apps/sim/.env - Windows opens the browser via `cmd /c start "" <url>` — `start` is a shell builtin, so spawning it directly ENOENT'd and the handoff never opened - managed-container detection filters loosely and pins the exact name in code; Docker's `name=^x$` anchor matches the internal `/x` form and often missed, skipping the reuse branch - the shared helm/kind run helper pins cwd to the repo root, matching helm test, so `helm upgrade --install ./helm/sim` works from any working directory * feat(chat-keys): standalone manage page, drop from settings nav, refresh README - Add /account/settings/chat-keys — a linkable page to view, create, and revoke Chat API keys - Remove Chat keys from the settings sidebar (account + unified nav) and its render branches - README: replace Docker Compose + Manual Setup with the bun run setup wizard; drop the manual COPILOT_API_KEY step, point to the manage page * fix(setup): per-key reason in the secret-replacement warning Cursor: the warn hardcoded '64-character hex key', but only ENCRYPTION_KEY/API_ENCRYPTION_KEY require that — BETTER_AUTH_SECRET/INTERNAL_API_SECRET only need length >= 32. Use the existing secretRequirement(key) helper so each replaced key reports its actual requirement. * fix(setup): compose doctor schema, cross-platform binary detection, quoted context hints - Doctor: for the compose (root) env layout, require only the secrets compose has no interpolation default for (BETTER_AUTH_SECRET/ENCRYPTION_KEY/INTERNAL_API_SECRET). DATABASE_URL/BETTER_AUTH_URL/NEXT_PUBLIC_APP_URL come from docker-compose ${VAR:-default}, so a healthy compose install no longer fails doctor. - Binary detection: use Bun.which instead of which (which is absent on Windows), so kubectl/helm/kind/docker resolve cross-platform. - k8s diagnostic hints: POSIX-quote the kube-context so a context with whitespace/metacharacters can't break or inject into a copied command. * fix(setup): quote kube-context in the helm uninstall tear-down hint too The tear-down hint used --kube-context ${context} raw while the sibling kubectl hints already used shq(); a context with whitespace/metacharacters could break or inject into the copied command. All copyable k8s hints now go through shq(context). * feat(setup): sim lifecycle CLI — start/stop/status/logs/down/reset Turn the setup entry into a 'sim' command umbrella so there's one place to run everything, not scattered docker/bun commands. Adds a global bin (bun link) + a bun run sim fallback. - Detects how you're running (compose file / managed dev containers / helm release) from disk + docker/helm state — no persisted mode. Ambiguous installs prompt. - start/stop/restart/logs work per mode; down removes containers (volumes kept); reset archives .env + wipes managed data; both destructive verbs confirm first. - status shows detected mode, container states, and app/realtime health. - Wizard outro + README now point at the sim commands and the one-time bun link. * feat(setup): 'bun run sim' is the primary entry; bare invocation prints help - Lead usage/wizard-outro/README with 'bun run sim <cmd>' (works with zero PATH setup); global bare 'sim' via bun link is an optional upgrade, with the ~/.bun/bin PATH caveat spelled out (Homebrew's bun omits it). - Bare 'sim' now prints help instead of launching the wizard; the wizard is 'sim setup'. The 'setup' npm script passes the keyword so 'bun run setup' is unchanged. * fix(setup): quote the auth URL for cmd /c start on Windows Cursor (High): cmd re-parses the command line and treats & in the query string as a command separator, so cmd /c start opened a URL truncated at the first &, breaking the key flow on win32 (the handoff URL always has request/challenge/pairing). Quote the URL and pass args verbatim so & stays literal. * fix(setup): verify kube-context is really local; lengthen CLI handoff wait - k8s: a context named like a local cluster (kind-*, docker-desktop) can actually point at a remote API server. Verify the server host is loopback/docker-internal before defaulting the 'use this context?' confirm to yes; otherwise warn and default to no, so generated secrets can't ship to a remote cluster on a blind Enter. - cli-auth: bump the device-flow wait from 3 to 15 minutes so first-time users have time to sign up, wait for the email OTP, and approve before the terminal stops polling. The server-side approval record keeps its own short TTL, so a longer client wait only costs cheap rate-limited polls. * fix(setup): only manage k8s lifecycle on a verified-local context Greptile: sim down/reset used the ambient kube-context, so switching context after setup could uninstall a same-named sim-dev release from the wrong cluster. Gate k8sInstall on the same locality check the wizard uses (API server is loopback/docker-internal) via a shared isLocalKubeContext helper — the wizard only ever deploys locally, so a remote current-context is never treated as a Sim install. * fix(setup): doctor skips placeholder secrets when seeding; reset names its target - checks: the missing-file autofix copied shared keys from apps/sim/.env whenever truthy, including .env.example placeholders — doctor --fix could seed unusable secrets into realtime/db env files. Skip placeholders, matching autofixForMissing. - lifecycle: reset now names the exact install (k8s context / compose file / dev containers) in its confirm, so a destructive reset can't silently hit the wrong same-named install after a context switch (down already names the context). * fix(cli-auth): size the poll rate limit to the poll cadence; honor Retry-After The poll route used the default public-IP bucket (10 burst, 5/min) but the CLI polls every 2s (30/min), so it 429'd within ~20s — worse behind a slow dev cold-compile. Give the endpoint a bucket matched to its cadence (60 burst, 60/min); it's not a brute-force surface (unknown request id returns pending, minting needs the 256-bit verifier). Also make the CLI honor Retry-After and back off on 429 so a shared-NAT per-IP limit degrades gracefully instead of hammering. * fix(setup): check ports before starting the dev server, not just compose Local dev auto-start spawned bun run dev:full with no port check, so it silently started a server that couldn't bind when 3000/3002 were already taken (e.g. another worktree's dev server). Extract compose's port-conflict resolver into a shared ensurePortsFree(ports) and run it before the dev start too — kill/recheck/leave, same as compose. Leaving the ports skips the auto-start with guidance instead of failing; compose still treats it as fatal. * fix(setup): verify the kube cluster is reachable, not just local A kubeconfig context can outlive its cluster — a kind cluster gets deleted or its Docker container stops (Docker/machine restart), but the context entry remains, pointing at a dead API-server port. The wizard checked the context looked local and handed it to helm, which failed with 'cluster unreachable'. Add a clusterReachable() liveness probe: only offer the current context when it actually answers; if a local context is dead, fall through to the kind path. There, if kind still knows 'sim' but it's stopped, start its node containers and wait for the API; if it's gone, create fresh. Either way the user gets a working cluster instead of a cryptic helm failure. * fix(helm): point appVersion at published image tags (v-prefixed, current) The chart's appVersion was "0.6.73", but CI publishes GHCR tags with a v prefix (its release-commit regex captures v0.7.45). Since sim.image defaults every image tag to Chart.AppVersion, a default helm install requested ghcr.io/simstudioai/{simstudio,realtime,migrations}:0.6.73 — a tag that has never existed — so app and realtime sat in ImagePullBackOff and helm --wait failed with 'progress deadline exceeded'. Any self-hoster installing with default values hit this, not just the setup wizard. Set appVersion to v0.7.45 (latest release on main; all three images verified present on ghcr) and bump the chart version to 1.1.1. Verified with helm lint, helm template (all images render as v0.7.45), and a live helm upgrade on a kind cluster where the new pods pull successfully while the old 0.6.73 pods remain in ImagePullBackOff. * Revert "fix(helm): point appVersion at published image tags (v-prefixed, current)" This reverts commit 28b6047d1d. * chore(api-validation): rebaseline route count to 977 after staging merge Staging moved the baseline to 975; this branch's two CLI-auth routes (approve, poll) make 977. The clean merge absorbed the earlier +2 adjustment. * fix(settings): don't highlight a sibling nav item on nested settings pages /account/settings/chat-keys is a real page but deliberately not a nav item, so the sidebar's parseSettingsPathSection fell through to defaultSection ('general') and highlighted General — the page read as though it lived inside General. Resolve the sidebar's active item with a null default so an unmatched nested route highlights nothing, and widen SettingsSidebar's activeSection to string | null. The section feeding the title/description provider keeps its default (pages override title/description anyway), and /account/settings/billing/credit-usage still correctly highlights Billing. * fix(setup,auth): manage explicitly-confirmed k8s contexts, fail loudly on reset, clear stale post-auth redirect - lifecycle: detection is now factual — a sim-dev release either exists on the current context or it doesn't. Gating on locality stranded a release the user explicitly confirmed during setup (status/start/stop/down/reset all claimed no k8s install). Locality is recorded instead and surfaced through describeInstall, which every destructive confirm renders, so acting on a non-local cluster is named and defaulted to no rather than silently blocked or silently allowed. - lifecycle: reset no longer discards helm uninstall's exit status. Env files are archived by that point, so claiming 'Reset complete' while the release still runs is the worst outcome — it now throws with retry/inspect commands. - auth: signup clears POST_AUTH_REDIRECT_STORAGE_KEY when it has no callbackUrl, and the verification-disabled path consumes it, so a stale CLI/invite destination can't leak into a later flow in the same tab.	2026-07-25 04:24:36 -04:00
Vikhyath Mondreti	fe184d3695	improvement(whatsapp): validate + improve integration skill for file inputs/outputs (#5942) ... * improvement(whatsapp): validate + improve integration skill for file inputs/outputs * fix lint * add whatsapp subblock migration	2026-07-24 16:14:27 -07:00
Waleed	513292f17b	feat(sso): DNS domain verification gating org SSO registration (#5909) ... * feat(sso): DNS domain verification gating org SSO registration Add org-scoped domain ownership verification (DNS TXT challenge) as the security precondition for configuring SSO. Closes the first-come domain-claim vuln where any org could wire another company's domain to its own IdP. - New sso_domain table + migration 0266; existing org SSO domains are grandfathered as verified so live tenants are unaffected - Verified-domains settings UI (enterprise-gated) with add/verify/remove - Register route now requires a verified domain for org-scoped registration; personal SSO and already-grandfathered domains are unaffected - Self-host register script writes the verified sso_domain row directly, so script-driven registration stays backwards compatible * fix(sso): harden domain verification against concurrency + fix CI lint Addresses review findings on state invariants under concurrent/failed writes: - Add unique index on (organization_id, domain) so concurrent claims can't create duplicate pending rows; POST re-reads and stays idempotent on conflict - Verify flips the row only if it's still the exact pending challenge checked (guards deletion/token-rotation mid-DNS-lookup) and maps the partial unique index violation to 409 instead of an unhandled 500 - Wrap the self-host script's provider write + verified-domain upsert in a transaction so a failed ownership write can't leave a provider committed - Format 0266 snapshot/journal with biome (fixes @sim/db lint:check) * fix(sso): re-check domain verification before provider write (TOCTOU) The register gate checked the verified sso_domain row only at handler entry, then ran OIDC discovery before writing the provider. A verified row removed during that window could still complete registration. Extract the check into a closure and call it both as an entry fast-fail and authoritatively right before registerSSOProvider, alongside the existing domain-conflict re-check. * fix(sso): stop rotating verification token on idempotent re-add Re-adding a pending domain rotated its verification token, which invalidated a TXT record the admin may have already published and — under two concurrent re-adds — could return a token the racing write had already superseded, so the admin's DNS record would never verify. Return the existing row unchanged instead; the pending token is always shown in the UI, so it is never lost. * fix(sso): close register TOCTOU with compensating delete + harden edges Audit-driven hardening: - Close the residual register TOCTOU: registerSSOProvider is create-only (throws if the providerId exists), so a compensating delete after the write is provably safe — it can only remove the just-created row. If verification was revoked during the write, roll the provider back and 403. - Verify is now idempotent under concurrency: a same-org row already flipped to verified by a racing request returns 200, not a confusing 409. - Grandfather backfill + self-host script now match normalizeSSODomain's dominant transforms (lower + trim + strip leading wildcard) so a non-canonical legacy domain can't miss the runtime gate's lookup. Prod backfill result is unchanged. - Cleanup: drop dead default export, align card radius to sibling convention. * fix(sso): redact domain tokens from non-admins + fix script stale-update Round-5 review findings: - GET /domains redacted the pending TXT verification token (a management secret) to any org member. Now only owner/admins read it; members see the list and status without it. Non-Enterprise orgs get an empty list (entitlement flag only), never the domains/tokens. - Self-host script decided update-vs-insert from a read taken OUTSIDE the transaction; a provider deleted mid-flight made the UPDATE match zero rows silently while the verified-domain upsert still committed (orphaned domain). The decision now happens inside the transaction from the UPDATE's row count. * docs(sso): drop unshipped enforce-SSO / auto-join copy from verified domains Verified domains currently only gate SSO configuration. Remove the forward-looking references to enforcing SSO and auto-joining members (deferred to a later release) from the docs, settings copy, nav description, and schema comment so we don't promise unshipped features. * fix(sso): guard rollback to new providers only + Enterprise-gate domain removal Round-6 review findings: - The compensating provider rollback now only fires when the provider did not exist before this request (providerExistedBefore). registerSSOProvider is create-only today so reaching the rollback already implies a fresh create, but this makes the safety local and future-proof: if Better Auth ever allowed updating an existing provider, a revoked-verification rollback must not delete that pre-existing row. - DELETE /domains now requires an Enterprise plan like add/list/verify, so all domain mutations share one entitlement (the UI already hides removal from non-Enterprise orgs). Adds a delete-route test. * fix(sso): roll back the SSO provider by row id, not logical keys The compensating rollback deleted by (providerId, orgId). providerId is unique, so if this request's row were deleted and recreated by a concurrent registration in the narrow window before the rollback, the logical-key delete would remove that other request's provider. Delete by the primary-key id registerSSOProvider returns instead, so only the exact row this request created is ever removed. * chore(sso): final-review polish — trim script read, unify copy, doc migration edge Cosmetic cleanup from a final 4-track adversarial review (no bugs found in the new logic): - Self-host script: narrow the pre-transaction existence read to select({ id }) instead of SELECT * (it only feeds a log line now). - Unify invalid-domain copy ("for example acme.com") and the verified-elsewhere 409 wording ("is already verified by another organization") across routes. - p-3 shorthand on the domain row card. - Document the migration's rare two-orgs-share-a-domain grandfather behavior (login unaffected; validated no such duplicates in prod). * fix(sso): apply attribute mapping + make SSO edit work; drop dead guard Two pre-existing SSO bugs the final review surfaced (prod has one SSO org, RVW, script-registered with the default mapping, so neither change affects it): - Attribute mapping was passed at the top level of the register payload, which Better Auth ignores — it reads oidcConfig.mapping / samlConfig.mapping. Nest it so custom mappings actually apply. (Default mapping is unchanged, so existing logins are unaffected.) - Editing an SSO provider was broken: registerSSOProvider is create-only and threw on the existing providerId → generic 500. Route now detects a provider the caller already owns and updates it via Better Auth's updateSSOProvider, and surfaces Better Auth's own error status/message instead of a blanket 500. Also drops the now-unnecessary providerExistedBefore guard (the rollback deletes by the created row's primary-key id and register is create-only) and the earlier final-review polish (script read, unified copy, migration edge note). Smoke-test SSO login + edit on staging before merge (auth-path change). * fix(sso): require null org on personal-mode provider lookups (gate bypass) The personal branch of both provider-ownership lookups keyed on (providerId, userId) without requiring organizationId IS NULL. Because org providers store userId = their creator and providerId is globally unique, an org admin could send a personal-mode request (no orgId) — which skips the membership check and the domain-verification gate — yet still match, and then via the new update path move, their org's provider to an unverified domain. Add isNull(organizationId) to the personal branch of both clauses so it can only match a genuinely personal provider, matching the route's own isOwnedByCaller. Found by an adversarial review of the update path added in 394bda9f7. * fix(sso): script updates the observed provider by id, not providerId Inside the registration transaction the script updated WHERE providerId — the logical key. If the observed provider was deregistered and a replacement created with the same providerId before the transaction ran, that update would clobber the replacement's config and ownership. Update the specific observed row by its primary-key id instead; if it's gone we insert, which fails cleanly on the providerId unique constraint rather than overwriting the replacement. * fix(sso): script upserts provider via delete-then-insert (no unique constraint) sso_provider.provider_id is a plain (non-unique) index and prod holds legitimate duplicates, so the previous "update by id, else insert" could create a duplicate provider when the observed row was deregistered and replaced before the transaction — the fallback insert would succeed. Delete every row for the providerId then insert exactly one, inside the transaction, so the providerId ends up as exactly this config atomically. Linked accounts key on the providerId string (not the row id), so existing logins are unaffected. * fix(sso): guard compensating-delete row id so rollback can't silently no-op * chore(sso): regenerate migration as 0268 after merging staging Staging landed migrations 0266/0267, colliding with our 0266. Removed our migration, merged staging, and regenerated cleanly with drizzle-kit as 0268_sso_domain_verification (identical sso_domain table + indexes), then re-appended the grandfather backfill. api-validation baseline reconciled to 973 (staging 970 + our 3 domain routes). Also make the register-route test's registerSSOProvider mock return an id so the guarded compensating delete runs. * refactor(sso): share normalizeSSODomain via @sim/utils so script matches gate The self-host script canonicalized SSO domains with a minimal inline transform (lower+trim+wildcard) that diverged from the app's full normalizeSSODomain (protocol, port, path, trailing dot, email local part) — equivalent spellings could store a different ownership key than the runtime gate looks up. Move normalizeSSODomain into @sim/utils/sso-domain (a pure function) so the register route, the domain-claim route, and the script all use the identical canonicalizer. The script now skips the verified-domain record when SSO_DOMAIN isn't a valid registrable domain instead of storing a malformed key.	2026-07-24 00:34:16 -07:00
3 people	6dcc65be89	feat(skills): add skill editors (#5705) ... * feat(skills): permissions layer * chore(db): drop skill_member migration 0261 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0262 on latest staging Same DDL as the dropped 0261 (skill_member table, enums, indexes, skill.workspace_shared) plus the hand-written write-user backfill, renumbered after staging's 0261. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0263 after staging merge Staging claimed 0262 (strong_storm); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(db): drop skill_member migration 0263 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0264 after staging merge Staging claimed 0263 (workflow_fork_sync_excluded); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make editing skills full page * fix disclaimer * edit access msg * fix lint * chore(db): drop skill_member migration 0264 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0265 after staging merge Staging claimed 0264 (fat_ikaris); same DDL plus the hand-written write-user backfill, renumbered on the merged snapshot chain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix tests * simplify system * fix * fix lint * add mship skills docs * chore(db): drop skill_member migration 0265 for regeneration on latest staging Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(db): regenerate skill_member migration as 0266 after staging merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deps): override zod to 4.3.6 to dedupe nested copies breaking type-check better-auth 1.6.23 and fumadocs-mdx resolve ^4.3.6 to a nested zod 4.4.3, which makes @sim/auth's inferred betterAuth types non-portable (TS2883) and split docs onto a second zod instance. Both ranges accept the repo-wide pinned 4.3.6, so a single hoisted copy satisfies everything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix lint * feat(skills,tools): fullscreen skill create + shared custom tool editor Moves the rich-markdown and custom-tool editing surfaces out of modals and onto full-page surfaces, and collapses the duplicated chrome behind shared components. Skills - Add /skills/new, a full-page create surface mirroring the skill detail page (CredentialDetailLayout + DetailSection + unsaved-changes guard). "Add to Sim" navigates there instead of opening a modal. - Import moves to a header action (SkillImportButton) backed by a shared readSkillFile helper; the GitHub-URL import and its /api/skills/import route are removed. - Skill name validation is now one shared validateSkillName, replacing three copies of the kebab-case rule and its messages. - The skill editor roster renders through the shared MemberRow instead of re-deriving its identity block, with a locked role control and a lock-reason tooltip explaining inherited workspace-admin access. Custom tools - Extract the canvas modal's schema/code editors into a shared custom-tool-editor module (fields, wand generation, schema helpers), cutting custom-tool-modal.tsx by ~900 lines. - Settings > Custom tools gains a full-page detail sub-view (SettingsPanel + SettingsSection + saveDiscardActions), deep-linkable via ?custom-tool-id. Rows are clickable; delete now lives only in the detail view. - Replace legacy Button/Input/Badge/Label with the chip family, move chip-field chrome into CodeEditor behind an error prop, and delete its dead wand button. Rich markdown field - maxHeight is now opt-in: omit it on a page and the editor grows with its content so the page owns the only scrollbar. Modals pass explicit caps. - The field variant drops to font-weight 400 to match adjacent chip fields. * fix(skills): address review round on create navigation, 409 copy, and editor audit - Skill create navigated using the first element of the upsert response, but that endpoint returns the caller's whole skill list (built-ins prepended) — match the new skill by its workspace-unique name instead. - The suggested-skill 409 toast claimed the skill existed but was not shared and told the user to ask a skill admin. Every workspace member can already see and use every skill, so a 409 only means the name is taken. - Adding an editor emitted the skill_shared event and SKILL_MEMBER_ADDED audit even when onConflictDoNothing skipped the insert on a concurrent add. Gate both on the insert actually returning a row. * chore: format skills-resolver test import * fix(skills,tools): audit fixes — autocomplete boundary, resize clipping, error routing Two real regressions introduced while simplifying the extracted editor: - The schema-param autocomplete's trigger was rewritten to match a trailing identifier, but the completion still split on separators. The two disagreed, so typing `data.ci` opened the menu and selecting replaced `data.ci` whole — eating the member-access prefix. Both now share one SCHEMA_PARAM_WORD regex. - The uncapped markdown field measured its height only on value change while always setting overflow-hidden, so any width change that re-wrapped lines clipped the tail with no scrollbar to reach it. Now re-measures via ResizeObserver. Also from the audit: - Generation writes bypass the code field's change handler, so an open autocomplete stayed over a disabled streaming editor; close it on busy. - Delete failures rendered in the Schema section's error slot on both custom tool surfaces; route them to a toast instead. - Skill create navigated away while still dirty, stranding the unsaved-changes guard's history sentinel so Back landed on an empty create form. - The Description field on skill create never received its error border. - Drop a double-applied opacity-50 (the editor already dims when disabled), a dead try/catch around a non-throwing call that also shadowed the error prop, and a stale reference to a /tools page that does not exist. - Docs still described the removed GitHub-URL import and the old Add Skill dialog; rewrite for the create page and file/paste import. * feat(tools): read-only tool detail, create lands on the new tool, drop dead wand prompt API - Viewers without edit rights could not open a custom tool at all, while the equivalent skill and custom-block surfaces both offer a read-only view. The detail page now takes `readOnly`: editors inert, no Save/Discard/Delete, no Generate. Creating still requires edit rights. - Creating a tool bounced back to the list while creating a skill lands on the new skill. Tools now do the same. The upsert returns the workspace's whole tool list (newest first) rather than just the new row, so the id is matched by title instead of by index — the same trap that produced the skill-create navigation bug. - Remove `openPrompt`/`closePrompt` from useWand. `closePrompt`'s last callers went away with the custom-tool-modal extraction and `openPrompt` had none before it; nothing reads `isPromptVisible` any more either. * fix(tools): read-only editors, design-system wrench, skills-matching tool identity - readOnly never reached the editors: the prop gated actions and Generate but the schema and code fields were still typable for viewers without edit rights. Wire disabled through both fields into CodeEditor. - The row icon used lucide's Wrench (strokeWidth 2) where @sim/emcn/icons ships one drawn for this system (1.55, tuned viewBox), and it inherited body text colour instead of --text-icon. Swap it. - Give the tool detail page the same identity heading as skill detail: tile, name, and description at the top left, instead of only a header title. - Extract ResourceTile so the skills and tools tiles share one definition (SkillTile now composes it), and add an opt-in `iconFilled` to SettingsResourceRow so the tools list tile matches the skills gallery. Both default to today's behaviour for every existing consumer. * fix(mentions): use the product's own glyph for every @ mention kind The `@` menu and the inserted chip mapped kinds to arbitrary lucide icons — `Sparkles` for a skill, a generic `File` for every file — while the rest of the product has a settled glyph per resource. Mirror CHAT_CONTEXT_KIND_REGISTRY, which Chat's `@` menu already renders from: - skill now uses AgentSkillsIcon, the same glyph SkillTile shows everywhere - workflow / folder / table / knowledge use the @sim/emcn/icons set the sidebar and the chat registry use - file derives its icon from the filename extension, so a .pdf and a .csv are distinguishable, matching the file list and Chat's context chips - integration keeps the block's brand icon from the registry Also drop the generic placeholder. `kind` is untrusted — the node schema defaults it to `''` and a hand-written `sim:` link can carry anything — but an unrecognized kind now yields no icon instead of a meaningless box, which is what the chat registry does. The menu already guarded a missing icon; the chip now does too, so this cannot crash on a malformed link. * chore(db): drop skill_member migration 0266 for regeneration on latest staging * feat(db): regenerate skill_member migration as 0267 after staging merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-07-23 19:52:00 -07:00
Waleed	2b5a92a3c8	feat(auth): org session policies — lifetime/idle limits, org-wide revocation (#5862) ... * feat(auth): org session policies — lifetime/idle limits, org-wide revocation, cookie-cache versioning * refactor(auth): consolidate session-policy clamp semantics, shared security-policy version module, canonical bounds, docs * polish(session-policy): cleanup pass — muted field labels, spinner reset, state tracker, response-seeded baseline, comment trims * fix(session-policy): govern member sessions by membership (closes revoke cookie-cache hole), normalize createdAt, remount on org switch, sync audit mock * fix(session-policy): clamp pre-join sessions on invite acceptance, normalize expiresAt, sync unified nav test * fix(session-policy): invalidate membership cache on removal/transfer, spare impersonator sessions in revoke-all, raise idle floor to 2x cookie window * fix(session-policy): resolve governing org by membership only — activeOrganizationId goes stale across transfer/leave * fix(session-policy): atomic policy save + eager clamp, asymmetric membership TTL, admin-add cache invalidation * fix(session-policy): org-scoped cookie version string, atomic revoke delete+bump * fix(session-policy): plan-gate effective policy so downgraded orgs stop enforcing automatically * chore(session-policy): drop dead bumpSecurityPolicyVersion helper — call sites bump transactionally * fix(session-policy): unify join paths on applySessionPolicyToNewMember; final audit polish (dead exports, response bound, test name)	2026-07-22 16:42:59 -07:00
WaleedandMarcus Chandra	6f33a9485b	feat(workflows): IDE-style reference viewer for workflows (#5854) ... * feat(workflows): add IDE-style reference viewer for workflows Adds a "Show references" viewer so you can see how workflows connect: which workflows call a given workflow ("Used by") and which workflows it calls ("Uses"), rendered as recursive, clickable trees. - Opened via Cmd/Ctrl+click on a sidebar workflow row and a "Show references" context-menu item. - Resolves references through both the workflow / workflow_input blocks (reusing isWorkflowBlockType) and published custom blocks (custom_block_* -> source workflow), scoped to the workspace. - Builds the whole workspace reference graph once from live workflow_blocks state; cycle-safe DFS marks A->B->A loops as (cycle) leaves. - Contract-bound GET /api/workflows/[id]/references with workspace-level authz; React Query hook gated to fetch only when the modal opens. - Unit tests for the pure graph/tree logic (cycles, self-refs, dangling drop, custom-block + workflow_input resolution) and route tests (401/400/403/200). * fix(workflows): correct reference resolution for active mode, cycles, cache, and graph size Addresses review findings on the reference viewer: - Resolve the workflow-block child via resolveActiveCanonicalValue (the shared SOT) instead of basic-first `||`, so an advanced-mode block whose old basic workflowId value lingers resolves to the active manual value. - Keep self-references (A -> A) and render them as a cycle leaf instead of dropping the edge, matching the cycle-safe viewer's purpose. - Set the references query staleTime to 0 so reopening the always-mounted modal refetches live editor state instead of serving a stale cached graph. - Bound converging paths: a node already expanded elsewhere in the tree is emitted once more as a plain leaf (edge stays visible) rather than re-expanded, so a densely reconverging graph can't grow exponentially. * improvement(workflows): align reference viewer auth, coverage, and UI with platform conventions - authorize via authorizeWorkflowByWorkspacePermission and derive the workspace server-side (404/403 semantics; drops the client-supplied workspaceId query param from the contract, hook, and modal) - add workflow-tool call edges: workflow_input tools inside tool-input sub-blocks now appear in both trees; non-call selector shapes stay deliberately excluded (documented against remap-internal-ids) - restore native cmd/ctrl+click open-in-new-tab on sidebar workflow rows; references stay reachable from the context menu - mount ReferencesModal on demand per row, deleting the prevIsOpen reset, the enabled knob, and the staleTime-0 workaround (now 30s) - align the tree with design tokens (--text-icon, --surface-hover, px-4 text gutter) and drop the hardcoded brand hex - escape LIKE wildcards in the custom_block_ prefix match; import MAX_CALL_CHAIN_DEPTH instead of mirroring it; remove dead fallbacks, the duplicate not-found scan, and the redundant custom-block row map * improvement(workflows): final polish on the reference viewer - drop the vestigial isOpen prop (conditional mount owns visibility) - unify on the emcn Workflow icon in tree rows - inline the static className and derive nodes without an annotation - remove one restating test comment * fix(workflows): resolve tool references by active canonical mode and keep the reference cache live - workflow_input tools inside tool-input now resolve basic/advanced via the index-scoped canonicalModes override, mirroring execution (Cursor finding) - staleTime back to 0: no mutation invalidates this key, so a reopen must background-refetch; on-demand mounting keeps the cached tree painting instantly (Greptile P1) - modal header uses the em-dash label-entity convention; tree items carry aria-level instead of a static aria-selected * fix(workflows): cover legacy workflow-typed tools and retry depth-truncated expansions - toolInputCallees matches both workflow tool type spellings via isWorkflowBlockType and passes the tool's own type as the legacy canonicalModes fallback, matching providers/utils resolution - a depth-capped expansion no longer poisons the expanded set, so a shallower path re-expands the node in full (Cursor finding) - the allowed-but-workspaceless auth branch now returns 403, not the authz result's 200 - tests: legacy tool type + per-tool index-scope isolation, diamond re-expansion with a real subtree, depth ceiling, shallow-path retry --------- Co-authored-by: Marcus Chandra <mzxchandra@gmail.com>	2026-07-22 14:03:07 -07:00
Waleed	93fbf584a2	feat(chat): soft-delete sidebar chats with restore from Recently Deleted (#5830) ... * feat(chat): soft-delete sidebar chats with restore from Recently Deleted * fix(chat): review round 1 — restore workspace authz, purge recheck, archived-list invalidation * test(chat): update SSE handler assertions for workspaceLists invalidation * fix(chat): bump updatedAt on restore, recheck retention cutoff in task cleanup * fix(chat): drop explicit feedback delete in task cleanup — chat FK cascade covers it * test(chat): cover restore route; guard legacy copilot delete from hard-deleting mothership chats * chore: revert unintended bun.lock drift from worktree install * fix(cleanup): recheck workflow archive cutoff on delete; export deletedAt in chat drain	2026-07-21 20:10:13 -07:00
Waleed	dd0e736d52	feat(managed-agents): add Claude Managed Agents workflow block (#5778) ... * feat(managed-agents): add Claude Managed Agents workflow block * improvement(managed-agents): complete session inputs/outputs; trim block templates - add memory instructions + file mount_path inputs; bound metadata (16 pairs) - surface cumulative token usage (inputTokens/outputTokens) as outputs - validate the full session-create schema against live docs - remove BlockMeta templates * improvement(managed-agents): select a Claude Platform credential instead of BYOK - register Claude Platform as a token-paste service-account credential (descriptor + validator) - add no-OAuth OAUTH_PROVIDERS entry; generalize the shared credential picker with a 'service-account' kind - block: oauth-input credential picker + dependsOn dropdowns; list route resolves the key server-side (audit-logged) - run via directExecution with the executor-injected key; drop the internal run route - remove the interim claude-platform BYOK provider * fix(managed-agents): harden reconnect loop; fix credential-picker regression - drive completion off terminal events + authoritative session status (drop the fragile busy-clock) - drain full event history so a long session's tail is never cut off - skip idless events in catch-up; require an id before replying to custom_tool_use - gate the shared credential-selector service-account lookup on credentialKind and use the non-throwing helper (was crashing multi-service OAuth pickers) - audit-log the list route's credential access; refresh stale tool docs * fix(managed-agents): address review round 2 - don't complete on idle status while a requires_action is still outstanding - vaults: combobox → dropdown so multiSelect actually attaches multiple vaults - auto-select a freshly-pasted service-account credential (onCreated through the connect modal) * fix(managed-agents): propagate cancellation, fix reconnect ordering, classify advanced fields - Thread the executor abort signal into directExecution (additive ToolConfig change) so a cancelled workflow stops the session immediately; best-effort user.interrupt releases the Anthropic session past cancel/wall-clock cap - Recompute the requires_action pending state from the chronological history so an older agent.message recovered on catch-up can't clear a newer pause - Retry a custom-tool error reply that failed to send instead of stranding the session (mark the event seen only once handled) - Mark optional fields (vaults, memory, files, metadata) mode: advanced - Title the session with the workflow id for Claude-console traceability * fix(managed-agents): stop leaked sessions on all give-up paths; harden event ordering and normalizers - Interrupt on sendUserMessage failure and on the reconnect-cap exit, matching the abort/wall-clock paths, so no give-up path leaves a session running - Order the events list by processed_at (list page order isn't guaranteed chronological) so catch-up accumulates text and reads the latest lifecycle event correctly regardless of API sort - Don't reset reconnect backoff on a failing custom-tool reply (stays unseen for retry) — prevents a no-delay reconnect storm - Treat only end_turn as a complete status_idle event; an unspecified idle defers to the sawActivity-gated status check (no empty completion pre-turn) - normalizeFiles parses a JSON-stringified table instead of dropping it; normalizeStringList returns [] on malformed JSON; metadata keeps scalar values - Declare the injected accessToken as a hidden param for convention parity * fix(managed-agents): interrupt the session on a mid-run stream/API failure The non-abort error catch path returned without stopping the session, so a mid-run network/API failure could leave the Anthropic session running against the workspace key. Interrupt on that path too — completing the invariant that every give-up exit (abort, cap, send failure, reconnect cap, stream error) stops the session. Still fails fast; no retry added. * fix(managed-agents): skip idless stream previews; resolve service-account provider icons - Skip idless events in the live SSE handler (mirroring catch-up). event_start/ event_delta previews carry no id, are never deduped, and final text always arrives as a persisted id-bearing agent.message — so appending previews could double the block's content output - Map serviceAccountProviderId to its base provider in PROVIDER_ID_TO_BASE_PROVIDER so parseProvider resolves 'claude-platform-service-account' to 'claude-platform' (a two-segment base the hyphen split can't recover), fixing the credential-row icon falling back to the generic external-link glyph * fix(managed-agents): keep idless terminals and preserve live pending state Refine the round-6 idless-event fix, which was too broad: - Process idless events again (revert the blanket stream skip) so an idless session.status_idle(end_turn)/session.error delivered only on the live stream still registers as terminal instead of reconnecting to a timeout. Only the agent.message TEXT append is now id-gated, so preview text still can't double - When catch-up history has no lifecycle event, restore the live-observed requires_action state instead of trusting a stale older agent.message that cleared it — prevents a false completion with partial output while a tool result is still pending * fix(managed-agents): route memory via metadata on self-hosted environments Live API testing revealed self-hosted environments reject the `resources` array with a 400 ("resources are not supported with self-hosted environments"), so the prior universal-resources[] payload would have failed any self-hosted session that attached a memory store or files. Restore env-type-aware routing: resolve the environment's config.type via a new getEnvironmentType() before session create, and for self_hosted send the memory store through metadata.memory_store_ids/memory_access (the worker consumes it) and drop file attachments. Cloud environments keep resources[]. Verified end-to- end against the live Managed Agents API (cloud resources[] 200, self-hosted metadata 200, self-hosted resources[] 400). * feat(managed-agents): environment-type selector; hide cloud-only fields on self-hosted Collapse what #5769 split into two blocks into one, natively: - Add an Environment type selector (Cloud / Self-hosted) that filters the environment list to the matching type and gates cloud-only fields - Memory store, memory access/instructions, and files are cloud-only (self- hosted rejects the resources[] attach — verified live, 400) and are now hidden on self-hosted instead of silently dropped. A self-hosted worker that uses a memory store reads its id from a Metadata key the author sets explicitly - Expose each environment's config.type on the list options so the picker can filter by mode; pass the selected type as a routing hint (server still re- resolves the authoritative type via getEnvironmentType) * fix(managed-agents): track requires_action by processed_at, not history position Persisted history can lag the live stream, so the last lifecycle event in the history array may be OLDER than a requires_action the stream already observed. Deriving the pending state from history position (findLastLifecycleEvent) could then clear a newer pause and let an idle snapshot complete a still-waiting session with partial text. Track requires_action from the NEWEST lifecycle event by processed_at across both the live stream and catch-up, so an older/lagging event can never override a newer pause. Removes the pendingBeforeCatchup snapshot and history-position recompute. New test covers lagging history holding only an older running event. * fix(managed-agents): treat missing processed_at as oldest, not newest A lifecycle event without processed_at mapped to +Infinity, poisoning the high-water mark: once seen, no later timestamped event could update the pending state (at >= Infinity always false), stranding a pause or clearing one wrongly. Map missing/unparseable processed_at to -Infinity so an untimestamped lifecycle event can never outrank a timestamped one in either direction — it neither blocks later real events nor clears a timestamped requires_action. Persisted lifecycle events always carry processed_at; this is purely defensive. New test covers a stray untimestamped running not clearing a timestamped pause.	2026-07-20 20:36:40 -07:00
Waleed	7e6aeb2b7a	feat(chat): favicon external links with secure link-preview tooltips (#5734) ... * feat(chat): favicon external links with secure link-preview tooltips * fix(link-preview): address review findings - allow http fetches to match advertised http(s) link support - fix meta content regex to handle apostrophes and either quote delimiter - hash Redis cache keys so sensitive URLs are not stored verbatim - add per-user rate limit to the outbound-fetching route - render siteName-only previews instead of falling back to the URL * fix(link-preview): redact full URLs from failure logs * improvement(link-preview): render-time preview fetch, cheerio parsing, cleanup pass - fetch previews when links render (emcn tooltip shows instantly — hover prefetch had no delay to race); tooltip reads the warmed cache, eliminating the URL-then-preview flash - parse OG metadata with cheerio (already used server-side) instead of hand-rolled regexes + entity decoding, fixing double-decode and quote-handling classes - drop the no-longer-needed prefetch hook; remove dead side prop on Tooltip.Content - extract ExternalLink to a sibling module per component-size guidelines; fix TSDoc placement * fix(link-preview): https-only previews and full-document parsing - drop allowHttp: plain-http fetches would reach the URL validator's self-host loopback exception; previews are now explicitly https-only on both server (early null) and client (query never fires for http) - parse the full capped document instead of truncating at the first <body> substring, which could match inside head scripts/comments and drop metadata * improvement(chat): render mailto links as plain text	2026-07-17 10:10:30 -07:00
Vikhyath Mondreti	86e6e1d26c	feat(forking): excluded workflows (#5727) ... * feat(forking): excluded workflows * improve sync preview	2026-07-16 20:23:26 -07:00
Waleed	68c4f28b77	feat(clickup): webhook triggers, hierarchy selectors, and Docs knowledge-base connector (#5708) ... * feat(clickup): webhook triggers with auto-managed subscriptions + hierarchy selectors * feat(clickup): KB connector (Docs v3), selector-route hardening, subblock migrations, registry-check regex fix * test(clickup): webhook provider handler tests + redact create-response secret in error logs * fix(clickup-connector): trim final page to maxDocs cap with precise listingCapped semantics * chore(clickup): lint formatting * fix(clickup): restore list-op location requiredness, split listSpaceId migration target, surface failed webhook rollback * fix(clickup): clickup.lists selector accepts listSpaceId context like clickup.folders * fix(clickup): integer-only maxDocs and location filters, depth-aware doc headings, most-specific-location hints	2026-07-16 10:17:02 -07:00
4 people	e2ea49ea7e	feat(instagram): add Instagram integration (#5568) ... * feat(instagram): add Instagram Login OAuth, tools, and block * feat(instagram): add Gmail-style media uploads for publish ops Resolve UserFiles to Meta-fetchable presigned HTTPS URLs (600s TTL) via internal publish routes, and fix OAuth scope storage plus connect-draft wiring so Instagram Login publishing is testable end-to-end. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(instagram): simplify messaging tools to direct requests, clean up types * fix(instagram): parallelize carousel child polling, enforce 2-10 items, extend poll window and insights periods, use canonical user_id in OAuth callback * fix(instagram): resolve user id from user_id only, accept numeric user_id Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): use form/query params for publish and comment endpoints, request message timestamps explicitly Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): normalize Graph ID outputs to strings so downstream .trim() calls are safe Co-authored-by: Cursor <cursoragent@cursor.com> * style(instagram): use brand gradient tile for the block icon Match the official Instagram look by filling the tile with the orange–pink–purple radial gradient so the white camera glyph sits on a full-bleed brand background. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): tighten publish defaults and cloud-storage upload UX Default Reel share-to-feed to Yes, drop unused media fields params, share publish transform helpers, and warn when cloud storage is missing for Meta-fetchable uploads. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): fail closed when cloud storage status is unknown Treat loading/error as blocked for requiresCloudStorage uploads, show the warning once the check finishes, and disable selecting local workspace files Meta cannot fetch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): proactively refresh long-lived tokens before expiry Meta only allows refreshing still-valid Instagram tokens, so refresh within 14 days of expiry (after the 24h age gate) instead of waiting until after accessTokenExpiresAt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(oauth): restore TikTok clientIdParamName JSDoc after merge The staging merge dropped the opening /** on ProviderAuthConfig.clientIdParamName, which broke TypeScript parse in CI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(api): Zod-contract storage-status and ratchet validation baseline Wire /api/files/storage-status through a shared route contract so the strict API validation audit stays at zero non-Zod routes after the new Instagram cloud-storage check. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): match Gmail advanced media placeholders Drop public-URL paste hints from advanced fields and the cloud-storage banner so the UI mirrors Gmail attachments. Co-authored-by: Cursor <cursoragent@cursor.com> * code review + hide from toolbar * address comments * fix(instagram): drop hidden Instagram from OAuth catalog pin test Instagram is hideFromToolbar so it is excluded from integrations.json; the pinned slug map must not expect it until the block is visible again. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(instagram): validate client ID before creating connect draft Avoid orphan pending credential drafts when INSTAGRAM_CLIENT_ID is missing, matching the Shopify authorize ordering. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-15 11:33:45 -07:00
Waleed	04535ee33b	feat(buffer): add Buffer integration with posts, channels, and ideas (#5637) ... * feat(buffer): add Buffer integration with posts, channels, and ideas * fix(buffer): return clear tool error when account lookup yields no account * fix(buffer): default schedulingType server-side so basic-mode blocks never fail validation * fix(buffer): probe Content-Type for extensionless media URLs so videos are not sent as images * feat(buffer): add get_ideas and get_idea_groups tools, harden media URL classification * fix(buffer): guard missing post on PostActionSuccess responses	2026-07-13 12:07:37 -07:00
Vikhyath Mondreti	ef7c8e24b2	feat(platform): settings permissions, admin, billing attribution (#5545) ... * fix(invites): preserve active organization for external access Keep organization activation server-owned so failed membership checks cannot clear a valid session context. * feat(admin, billing, settings): cleanup settings visibility, billing actor resolution, new admin routes * address comments * chore(db): reset pending migrations before staging merge Remove locally generated migrations so they can be regenerated against the latest staging schema without preserving stale snapshots or numbering. * regen migrations * address comments * chore(db): reset generated migrations before staging merge Remove this branch's generated migrations so they can be regenerated against the latest staging schema with fresh numbering. * upgrade global work * fix lint * address comments * legacy callbacks correctness * address comments * update * guardrail attribution	2026-07-13 00:53:49 -07:00
Theodore Li	7962236719	improvement(custom-blocks): hardened delete with usage count + per-input required option (#5575) ... * improvement(custom-blocks): usage visibility + type-to-confirm delete * feat(custom-blocks): per-input required option * improvement(custom-blocks): replace usage tab with delete-confirmation usage count * fix(custom-blocks): escape LIKE wildcards in usage scan + fresh count on delete modal * fix(custom-blocks): explicit ESCAPE clause on usage-scan LIKE prefilter	2026-07-10 20:16:53 -04:00
Theodore LiandClaude Opus 4.8	f4d47ed826	feat(slack): reusable custom bot credentials, slack_v2 block (preview), redesigned trigger (#5323) ... * feat(slack): enable assistant-agent tools via assistant:write scope Add assistant:write, app_mentions:read, and im:history to the Slack bot OAuth scopes so the Set Assistant Status / Title / Suggested Prompts tools (assistant.threads.*) work with users' existing Slack credentials — no new app or credentials required. Restore the action_assistant trigger capability (scope assistant:write) in the manifest generator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpeT8J5yVCrrNQB9Hzm9uS * Add slack trigger * fix channel picker in slack trigger * improvement(slack-trigger): reorder app type, gate account to sim mode, add channel-id input * fix(slack-trigger): drop unmapped events from filter, resolve oauth token for reaction text + file downloads * fix(slack-trigger): empty operation selection fires nothing; resolve token via credential owner not execution actor * fix(slack-trigger): ignore message edit/delete/system subtypes; prefer channel picker over stale manual ids * feat(slack-trigger): single-event model with contextual filters and full event catalog * fix(slack-trigger): apply event/channel/bot filters on custom-app path too * fix(slack-trigger): don't drop edit/delete events when channel_type is absent * feat(slack): reusable custom bot credentials, slack_v2 block, interactivity triggers - Custom bot as a workspace service-account credential (set up once, shared ingest URL /api/webhooks/slack/custom/{credentialId}, reused across triggers and actions) - slack_v2 action block: credential-based Custom Bot auth alongside Sim OAuth; v1 hidden from toolbar - Interaction triggers (block_actions / view_submission) with optional action/callback id filter; settings.interactivity in generated manifests - Setup wizard: name + description, full permissions by default with ChipDropdown customization; reconnect mode rotates secrets in place - Centralized service-account token resolution (unknown provider fails loudly) - Shared Slack webhook fan-out dispatcher for native + custom ingest routes * chore(api-validation): bump route baseline to 924 after staging merge * feat(slack): preview-gate slack_v2 and the custom-bot credential surfaces slack_v2 (block + hosted slack_oauth trigger) ships preview: true — hidden from all discovery until revealed via block-visibility AppConfig or PREVIEW_BLOCKS. v1 stays toolbar-visible with the legacy slack_webhook trigger until v2 GAs. The integrations-page custom-bot setup surface rides the same flag via isHiddenUnder(slack_v2); placed instances, existing credentials, and ingest/execution paths are never gated. * fix(slack): v1 keeps slack_webhook trigger subblocks; handle object-form event channels - v1 spread had been swapped to slack_oauth's trigger subblocks (shared with v2), leaving its slack_webhook deploy path without signing-secret config (Bugbot high). v1 now carries the legacy trigger set again; v2 swaps them for slack_oauth's. - resolveSlackEventChannel reads channel.id for channel_created/channel_rename payloads, so channel filters no longer drop every rename event. * fix(slack): default absent appType to custom at deploy; deactivate custom-bot webhooks on credential delete - appType is hidden and seeded 'custom' by value(), which only covers editor-created blocks; defaultValue now persists it via buildProviderConfig and the deploy fallback flips to custom (the only exposed mode this ship) - deleting a slack-custom-bot credential now also deactivates provider='slack' webhooks routed by that credential id, not just native slack_app rows * fix(slack): resolve credential owner for deploy-time team_id lookup A teammate deploying a trigger wired to a shared Slack credential isn't the credential owner; refreshAccessTokenIfNeeded only loads tokens for the owning user. Resolve the account owner first, mirroring the runtime formatInput path. * chore(slack): reconcile staging merge - nullable webhook.path coalesced at correlation/payload/tiktok boundaries - slack dispatch delegates to staging's dispatchResolvedWebhookTarget (shared preprocess/deployment/filter/enqueue lifecycle), keeping the skip-reason diagnostics; route tests reworked around that seam - api-validation route baseline 924 -> 926 * fix(slack): workspace-scope bot credentials at deploy; recreate webhooks on routing transitions - a bot credential id is semi-public (embedded in Slack Request URLs), so the custom deploy branch now rejects credentials outside the workflow's workspace - needsRecreation also compares path/routingKey, so a row from an older routing model can't survive redeploy as a stale delivery surface * test(slack): pin fail-closed behavior for empty/missing event selection * fix(slack): 409 on custom-bot name collision instead of silently returning the existing credential The service-account dedupe matches on displayName, which defaults to the Slack team name — shared by every bot in that workspace. A second unnamed bot create returned the first credential as success, orphaning the new id already pasted into the Slack Request URL. Same-id replays stay idempotent; different-id collisions now fail loudly so the wizard prompts for a distinct name. * fix(slack): reconnect surfaces Atlassian error codes and persists name/description edits - PUT credential route now returns the Atlassian provider code (providerErrorCode -> code) so reconnect failures map to specific token/domain messages, matching create - Google/Atlassian reconnect send + seed displayName/description (parity with Slack); edits are no longer silently discarded, and empty fields don't clobber existing values * fix(slack): require bot name; propagate rotated bot_user_id to webhooks on reconnect - the setup wizard now requires a bot name (canAdvance), so the credential name, manifest app name, and uniqueness key all use the user's choice instead of the shared Slack team-name fallback that collided for a second bot in one workspace - reconnect that changes the bot user id (recreated Slack app) now updates the bot_user_id cached in each bound webhook's providerConfig, so reaction self-drop keeps working instead of letting the bot's own reactions re-enter --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 19:43:52 -04:00
4 people	5d3809a0e3	feat(tiktok): add tiktok trigger, block (#5504) ... * feat(tiktok): add TikTok integration Adds TikTok as a full OAuth-based integration: provider registration (with TikTok's comma-separated scope and client_key requirements), 9 tools covering profile info, video listing/querying, creator info, direct video/photo posting (URL or file upload), inbox drafts, and post status polling, plus the TikTok block, icon, and generated docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): add avatarFile output to Get User Info Adds a file-typed avatarFile output (sourced from the largest available avatar URL) alongside the existing string avatar fields, so the profile picture can be materialized as a UserFile and chained into file-consuming blocks (e.g. attached to an email), per PR review feedback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): lower upload memory cap, drop redundant avatar string outputs Cap the file-upload video buffer at 250MB instead of TikTok's 4GB ceiling — relaying that much through this server's memory per request isn't safe under concurrent load, and larger files can still go through the PULL_FROM_URL path, which never buffers on our server. Also drop the now-redundant avatarUrl/avatarUrl100/avatarLargeUrl string outputs from Get User Info in favor of the file-typed avatarFile output alone, since the feature is unreleased and the raw URL is still reachable via avatarFile.url. Cover image URLs on List/Query Videos are confirmed to be signed, expiring TikTok CDN links; left as strings (no file-output conversion path exists for fields nested inside array items) but documented the expiry behavior more clearly. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ci): bump API validation route-count baseline for TikTok publish-video route The TikTok integration adds one new Zod-backed internal API route (app/api/tools/tiktok/publish-video), which trips the route-count ratchet in check-api-validation-contracts.ts. Bumping totalRoutes and zodRoutes from 917 to 918 (nonZodRoutes stays 0) to acknowledge the new route is properly validated. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): drop unused avatar_url_100 from default user fields After removing the avatar string outputs, avatar_url_100 was still requested from TikTok's user info endpoint but never surfaced anywhere. Removed it from the default field list and the field descriptions, and noted that avatar_url/avatar_large_url feed the avatarFile output. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop returning raw 'credential' subBlock id from tools.config.params The block's params function built a local `credential` variable from params.oauthCredential and returned it under the key `credential` in every switch case. That literal token is the raw subBlock id, which is deleted after canonical transformation into `oauthCredential` — the blocks.test.ts canonical-param-validation suite flags any params function that still references it. It was also redundant: oauthCredential is already part of the base resolved inputs, which the executor merges into the tool call before config.params overrides are applied, so the OAuth token resolution (which reads contextParams.oauthCredential) worked regardless. Removed the explicit credential plumbing, matching the convention already used by other OAuth blocks like dropbox.ts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): send empty JSON body on Query Creator Info POST query_creator_info had no request.body function, and formatRequestParams() only attaches a body when tool.request.body is defined at all — so despite sending Content-Type: application/json, the request went out with no body whatsoever. Added body: () => ({}), matching the convention already used by other parameterless-POST tools in this codebase (Google Vault, Supabase, Square, Gmail, etc.). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): stop dropping valid zero values in optional numeric fields cursor, photoCoverIndex, and videoCoverTimestampMs all used a truthy check (params.x && {...}) to decide whether to include an optional numeric override, which drops a legitimate 0 (first page has no cursor issue aside, photoCoverIndex 0 is TikTok's own default cover photo, and timestamp 0 is a valid first-frame cover). Switched to explicit undefined/empty-string checks, matching the !== undefined convention the underlying tools already use. In today's resolution pipeline these fields always arrive as strings (even chained block references get stringified by the template resolver), and a non-empty string like "0" is truthy, so this wasn't actively broken end-to-end - but it was relying on that subtlety rather than being correct by construction, and was inconsistent with the tools' own undefined checks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tiktok): accept newline-separated video IDs in Query Videos videoIds is a long-input (multiline textarea), the same widget used for the newline-separated photoImages field on this block, but its parser only split on commas. Entering one ID per line - the natural pattern for a multiline field, and the one already used elsewhere on this block - produced a single concatenated garbage string instead of an array, so TikTok's query would fail or return nothing. Now splits on commas or newlines, and updated the placeholder/description to reflect both formats. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(tiktok): add app-level webhook ingress and triggers * fix(tiktok): only count actually queued webhook executions Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tiktok): bump API validation baseline for staging merge Co-authored-by: Cursor <cursoragent@cursor.com> * cleanup code * fix type issues * misc code cleanup * remove photos and add upload for videos * move shared video output properties to types.ts so docs generation resolves them Co-authored-by: Cursor <cursoragent@cursor.com> * hide TikTok from toolbar and docs until the integration is ready to ship Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): ratchet API validation baseline to 924 after staging merge Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-07-10 15:08:24 -07:00
Theodore Li	5b7513f15d	feat(blocks): add block visibility gating (preview blocks + AppConfig reveals) (#5526) ... * fix(deps): install xlsx from @e965/xlsx npm mirror The dependency was pinned to a direct tarball on cdn.sheetjs.com, which now returns 403 (Cloudflare bot-challenge) to automated clients, breaking bun install in CI. npm's own xlsx is frozen at 0.18.5, so switch to the @e965/xlsx mirror which republishes the identical 0.20.3 CDN build to the npm registry. No code changes needed — all imports use bare 'xlsx'. * feat(blocks): add block visibility gating (preview blocks + AppConfig reveals) * fix(blocks): reset visibility to fail-closed empty state on workspace switch * fix(blocks): carry kill-switch entries across workspace-switch visibility resets * chore(deps): revert stray local xlsx-mirror commit (keep staging's pinned source) * chore(skills): rename gate-block skill to add-block-preview	2026-07-09 22:38:01 -04:00
Waleed	6e87d74a93	feat(jupyter): add Jupyter integration (contents, kernels, sessions) (#5527) ... * feat(jupyter): add Jupyter integration (contents, kernels, sessions) - 16 tools covering Contents, Kernels, Kernelspecs, and Sessions REST APIs - File upload/download via UserFile, following the Box upload pattern - Block with operation dropdown, token auth, and 8 catalog templates - Registered tools + block, generated docs, bumped API validation baseline * fix(jupyter): address Greptile review — SSRF guard, upload path ambiguity, silent notebook fallback - Route uploads through validateUrlWithDNS + secureFetchWithPinnedIP (matches Grafana/1Password pattern) instead of a raw fetch to the user-supplied server URL - Replace the upload path/trailing-slash heuristic with an unambiguous directory + filename split - create_file no longer silently writes an empty notebook when notebook content is malformed JSON — it now errors clearly * fix(jupyter): request content=1 when listing directory contents Without it, Jupyter Server returns directory metadata with content: null, so jupyter_list_contents always reported an empty items array. * fix(jupyter): reject path-traversal segments in Jupyter content paths encodeJupyterPath now rejects '.'/'..' segments across the whole path (shared by all 16 tools, not just upload); the upload route returns a clean 400 when it's hit. * fix(jupyter): close remaining path-traversal and redirect-credential gaps - extract the traversal check out of encodeJupyterPath into a shared assertion, and apply it to body-only path fields (rename newPath, copy copyFromPath, session path) that never flowed through URL encoding and so skipped the check - pass stripAuthOnRedirect to the upload route's secureFetchWithPinnedIP call so a malicious Jupyter server can't redirect the PUT to another origin and receive the caller's token * fix(jupyter): also reject percent-encoded traversal segments A segment like %2e%2e wouldn't match the literal '..' check. Now decodes each segment before comparing, in addition to the literal check, so an already-encoded traversal attempt is caught too. * fix(jupyter): route all 15 remaining tools through an internal proxy for HTTP/private-host support and no redirects The generic external tool executor blocks plain-HTTP and non-localhost private-IP hosts by default, so every non-upload Jupyter operation could fail against typical self-hosted setups (LAN IP, docker hostname, or even literal localhost on a hosted deployment) even though the upload route worked via its own internal route. Added /api/tools/jupyter/proxy (DNS-pinned, allowHttp, maxRedirects: 0) that mirrors the upstream Jupyter response verbatim, matching the established pattern for self-hosted-arbitrary-host integrations (Grafana, 1Password) instead of the generic executor path. Each tool's request block now posts to the proxy instead of building a direct external URL; transformResponse and outputs are unchanged since the proxy response mirrors upstream status/body exactly. Also switches the upload route from stripAuthOnRedirect to maxRedirects: 0 — stronger, since it stops the uploaded file body (not just the token) from ever reaching a redirect target. * fix(jupyter): validate proxy path at the trust boundary, reject path separators in upload filename - The proxy route now independently validates the incoming path field for traversal segments instead of only relying on tool-side validation before the request reaches it — the route is a shared internal boundary, not something only our own tool code can call - The upload route's fileName can come from an advanced override or the legacy fileContent path and could itself contain '/' or '\', silently nesting the upload deeper than the directory param specified. Now rejected outright before joining. * fix(jupyter): decode the whole path before splitting, not per-already-split segment A segment like foo%2f..%2fsecret has no literal slash, so splitting on literal '/' first and decoding each piece in isolation treats it as one opaque segment and never notices the '..' hiding behind the encoded slash. Decode the full path once, then split and check every segment the target server's own single URL-decode pass would see.	2026-07-08 19:35:57 -07:00
Vikhyath MondretiandSiddharth Ganesan	4a80374c03	improvement(forking): unlink, settings page migration, deployed chats, public apis, external mcp servers/tools, special subblocks (#5505) ... * bad checkpoint * stash * add unlink, move UI to settings pages * fix edge cases * remove dead code * remove migration 0255 ahead of staging merge (regenerated after) * regenerate workflow_mcp_server enum migration on top of staging (0257) * fix tests * more tests * address comments * consolidate fork migrations into 0257 (enum value + activity metadata indexes) * acquire MCP server locks before reads in attachment reconcile (TOCTOU) * move into ee folder + ui perm gates * use randomInt from @sim/utils/random for chat identifier suffix --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com>	2026-07-08 12:15:28 -07:00
3 people	8fcce5100c	fix(aws): align cloudwatch, cloudformation, athena, codepipeline with live API docs (#5483) ... * fix(aws): align cloudwatch, cloudformation, athena, codepipeline with live API docs - cloudwatch: fix list_metrics pagination (wasn't draining pages past 500), add MaxRecords cap validation to describe_alarms; add describe_alarm_history, filter_log_events, put_log_group_retention - cloudformation: fix get_template missing TemplateStage param, fix invalid ModuleTag in block metadata; add full stack lifecycle tools (create/update/delete/cancel_update_stack, create/describe/execute_change_set, get_template_summary) - athena: fix missing .trim() on query/named-query ID fields; add delete_named_query, batch_get_query_execution, list_databases, list_table_metadata - codepipeline: fix missing rollbackMetadata field in list_pipeline_executions response; add get_pipeline, list_action_executions, disable/enable_stage_transition * fix(aws): require template on cloudformation change-sets, bump route-count baseline - cloudformation create_change_set now rejects requests missing both templateBody and usePreviousTemplate, matching update_stack (Cursor Bugbot finding) - bump check:api-validation route-count baseline 906->917 to reflect the 19 new fully contract-bound routes added in this PR (0 boundary violations) * fix(aws): address Greptile round-1 review findings - cloudwatch describe_alarm_history: always request both MetricAlarm and CompositeAlarm types, even when alarmName is provided (was silently returning empty history for composite alarms queried by name) - cloudformation: add validateAwsRegion refinement to region field on the 7 new write-path contracts (update/delete/cancel-update-stack, create/describe/execute-change-set, get-template-summary), matching the pattern already used elsewhere - cloudformation: destroy the AWS SDK client in a finally block on the same 7 new write routes, matching the pattern used by every other new route in this PR * fix(aws/cloudformation): add missing region validation and client cleanup to create-stack - Add validateAwsRegion refinement to create-stack contract (completes P2 fix from Greptile review) - Wrap AWS SDK call in try/finally with client.destroy() (completes P2 fix from Greptile review) - Aligns create-stack with the pattern used across all 7 other new CloudFormation routes Co-authored-by: Waleed <waleedlatif1@users.noreply.github.com> * fix(aws): final validation pass — bound missing limits, close consistency gaps - cloudformation create_stack: add missing validateAwsRegion refine and client.destroy() finally block, matching sibling write routes - cloudwatch get_metric_statistics: cap statistics array at AWS's 5-item limit - cloudwatch get_log_events: cap limit at AWS's 10,000-record max - athena batch_get_query_execution: surface engineExecutionTimeInMillis/queryPlanningTimeInMillis/queryQueueTimeInMillis, matching the sibling get_query_execution tool's Statistics mapping * fix(aws/athena): destroy AWS SDK client on the 4 new athena routes Cursor Bugbot finding: batch_get_query_execution, delete_named_query, list_databases, and list_table_metadata created an AthenaClient but never called client.destroy(), unlike every other new route in this PR. Wrapped each in try/finally to match. * fix(aws/athena): add validateAwsRegion refinement to the 4 new contracts Greptile finding: delete_named_query, batch_get_query_execution, list_databases, and list_table_metadata accepted any non-empty string for region, unlike every other new contract in this PR. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Waleed <waleedlatif1@users.noreply.github.com>	2026-07-07 12:19:34 -07:00
Waleed	b686111082	feat(textract): migrate to AWS SDK, add AnalyzeExpense and AnalyzeID (#5456) ... * feat(textract): migrate to AWS SDK, add AnalyzeExpense and AnalyzeID - Replace hand-rolled AWS SigV4 signing with @aws-sdk/client-textract, matching sibling AWS integrations (secrets_manager, s3, sts) - Add Analyze Expense operation (invoice/receipt structured extraction) via AnalyzeExpense/StartExpenseAnalysis+GetExpenseAnalysis - Add Analyze Identity Document operation (AnalyzeID) with optional back-of-ID page - Add an operation selector to the textract_v2 block; defaults to the existing Analyze Document behavior for backward compatibility - Add tests for tool body/response mapping and route-level AWS response normalization * fix(textract): forward URL documents and fix ambiguous error status - textract_analyze_expense/textract_analyze_id now fall back to filePath/filePathBack when the document input is a URL string rather than an uploaded file object, so advanced "File reference" URL inputs actually reach the API (Cursor Bugbot) - mapTextractSdkError defaults to 500 (not 400) when the AWS SDK error has no HTTP status, since that implies a server-side/network failure rather than a bad request (Greptile) * fix(textract): stop stale processingMode from hiding ID document fields - Front-document fields (fileUpload/fileReference) are shared across all 3 operations; gate them with a values-aware condition so switching to Analyze Identity Document keeps them visible even if a stale processingMode='async' is left over from a previous operation - S3 URI field now also requires operation !== 'analyze_id', since that operation never supports S3 input * fix(textract): preserve first-page metadata across async pagination pollTextractJob's merge callbacks spread only the latest page, dropping any field (DocumentMetadata, model version) the first page had but a follow-up NextToken page omits. Merge accumulated first so later pages only override fields they actually return. * fix(textract): pass through the real upstream status for filePath fetch failures fetchDocumentBytes hardcoded 400 for any non-OK response from a document URL, masking transient 5xx failures from the document host as client errors and blocking tool-execution retries. Use the actual response status instead. * chore(textract): drop redundant inline comments	2026-07-06 19:42:36 -07:00
Vikhyath Mondreti	24ebba9acc	chore(credential-sets): cleanup feature (#5460)	2026-07-06 18:49:06 -07:00
Waleed	fb3f95d5dc	feat(aws): expand SES/STS/Secrets Manager tool coverage, fix API alignment gaps (#5450) ... * feat(aws): expand SES/STS/Secrets Manager tool coverage, fix API alignment gaps - SES: add suppression list management, email identity CRUD, template update, configuration set creation, custom verification email (10 new tools); fix silent httpsPolicy drop in create_configuration_set and unvalidated suppression reason enum in list_suppressed_destinations - STS: add AssumeRoleWithWebIdentity and AssumeRoleWithSAML (unsigned, no static credentials required); extend assume_role with policyArns/tags/transitiveTagKeys session params - Secrets Manager: add describe_secret, tag_resource, untag_resource, restore_secret, rotate_secret; fix list_secrets dropping rotation/version metadata fields; normalize tool versions to 1.0.0 and alphabetize registry entries All 31 tools verified param-by-param against live AWS API docs across two independent audit passes. * fix(aws): address review findings on SES config/identity and Secrets Manager rotation - ses_create_configuration_set: validate suppressedReasons against BOUNCE/COMPLAINT enum before calling AWS (was silently reaching AWS as a generic 500 for bad values); tags now a proper Zod array schema instead of a string with route-side JSON.parse - ses_create_email_identity: dkimSigningAttributes and tags now proper Zod object/array schemas instead of strings with route-side JSON.parse, matching the pattern used elsewhere (e.g. sts_assume_role tags, secrets_manager_tag_resource) - secrets_manager_rotate_secret: reject automaticallyAfterDays and scheduleExpression when both are supplied — AWS RotationRules accepts only one - sts createUnauthenticatedSTSClient: corrected a misleading comment claiming these calls are fully unsigned; the SDK still falls through its default credential provider chain * fix(ses): correct json input types for tags/dkimSigningAttributes The SES block declared the tags and dkimSigningAttributes block inputs as 'string' instead of 'json', so the generic block executor never parsed the JSON code-editor value before forwarding it — workflow runs sent a raw JSON string where the contract now expects a structured object/array, failing validation. Also corrected the corresponding tool param TypeScript types, which were still typed as string | null. * fix(ses): stop coercing switch string 'false' to true in create_configuration_set Boolean('false') evaluates to true, so turning off the reputationMetricsEnabled or sendingEnabled switch sent the opposite of the user's choice to SES. Match the established === 'true' string comparison pattern used elsewhere in the codebase. * fix(sts): stop double-parsing assume_role session tags input tags was declared as a 'json' block input, so the generic executor JSON.parse'd it before the switch-case handler ran — but that handler already converts the raw table-rows array (or a passthrough string) into the JSON string the sts_assume_role contract expects. Declaring it 'json' broke that conversion for non-string inputs. Reverted to 'string' so the handler's existing string/array disambiguation runs on the untouched raw value. * fix(sts): supply placeholder credentials to the unauthenticated client createUnauthenticatedSTSClient omitted credentials entirely, so the SDK's signing middleware fell through the default credential provider chain and threw CredentialsProviderError before the request was sent in any environment with no ambient AWS identity — even though AssumeRoleWithWebIdentity/AssumeRoleWithSAML never check the signature. Static placeholder credentials skip that resolution without granting or requiring any real IAM identity.	2026-07-06 17:26:44 -07:00
Theodore Li	82eff5435d	feat(custom-block): deploy a workflow as a reusable org-scoped block (#5407) ... * feat(custom-block): deploy a workflow as a reusable org-scoped block * fix(custom-block): reseed deploy form, guard duplicate publish, run child deployed * test(custom-block): isolate custom-block rows fetch in execution-core test * fix(custom-block): allow cross-workspace exec, org-scope authority, keep field ids, hide disabled * feat(custom-block): run child under source owner's identity, workspace, and env * fix(custom-block): bind publish authz to the source workflow's workspace * fix(custom-block): gate edit/delete on source-workspace admin, not org admin * chore(custom-block): rebaseline route count to 887 after staging merge * fix(custom-block): sanitize failure output so it can't leak source workflow internals * fix(custom-block): derive inputs and curated outputs from deployed state, not draft * fix(custom-block): hide disabled blocks from the toolbar palette too * fix(custom-block): bill nested + failed-run hosted cost; expose real inputs to the agent * fix(custom-block): enforce enterprise + flag gate at every consumption path	2026-07-04 15:57:24 -04:00
Waleed	759dddbf4c	feat(billing): dedicated Credit usage page with date-range filter and CSV export (#5405) ... * fix(billing): apportion per-row credit costs so they sum to the page total Cursor Bugbot (medium): each row rounded its own dollar cost to credits independently while the header total rounded the summed dollars once — over enough rows those two roundings can visibly disagree, the exact "line items don't add up to the total" class of bug apportionCredits was already built to prevent (used by the trace view / cost breakdown). Route now apportions each page's row credits against that page's dollar sum instead of rounding rows independently. Added a test with three sub-cent rows that would each independently round to 0 credits (but sum to 1) to prove the reconciliation holds. * fix(billing): dim stale credit usage rows while a new period loads Cursor Bugbot (medium): keepPreviousData kept the prior period's rows and total on screen while a newly selected period fetched, but the dropdown label updated immediately — so during the transition the displayed numbers were labeled under a period they didn't belong to. Now reads isPlaceholderData (the standard TanStack Query signal for "this data is a stale placeholder, not a fresh fetch for the current key") and dims the list while it's true, matching the same flag already used for this exact purpose in integration-skills-section.tsx. * fix(billing): show "<1 credit" for rows apportioned to 0 Cursor Bugbot (low): with apportioned per-row credits, a row with a real but sub-credit dollarCost can legitimately apportion to 0 credits once a sibling row absorbs the shared rounding remainder — rendering a flat "0 credits" reads as if nothing was charged, inconsistent with formatCreditCost's "<1 credit" wording used elsewhere in billing. Added dollarCost to the wire response (needed to distinguish a genuinely free row from a rounded-to-zero one) and a small formatRowCredits helper that only changes the label, not the underlying creditCost number, so the page-total reconciliation from the prior fix is unaffected. * fix(audit-logs): fix broken Custom range picker, trim time-range presets Custom range silently did nothing: the time-range trigger was a ChipSelect (Radix DropdownMenu, modal by default), and selecting "Custom range" opened the Calendar popover in the same tick the modal menu began its close/focus-lock cleanup, trapping the popover non-interactive. Swapped to ChipCombobox (Radix Popover, non-modal), mirroring the already-working pattern in the main Logs page exactly. Also trimmed the preset list from 11 to 8 entries (dropped Past 30 minutes/12 hours/14 days) so the menu fits without scrolling. * feat(billing): dedicated Credit usage page with date-range filter and CSV export Follow-up to #5391 per team feedback in Slack: move the credit usage list out of the inline Billing section into its own page, redesign rows to show source ("Chat", "Workflow: <name>") instead of a raw model description + badge, and add real date-range filtering and export. - Billing settings now shows a compact glance (30-day total + a "View usage logs" link) instead of the full inline list. - New /settings/billing/credit-usage page (sibling of [section], mirrors the secrets/[credentialId] detail-route pattern) with day presets (Today/7d/30d/All time) plus a working Custom range picker — the same ChipCombobox+Popover+Calendar wiring the audit-logs fix in this branch uses, not the broken ChipSelect pattern. - Rows show the humanized source label, or "Workflow: <name>" for workflow-sourced events (new server-side workflow-name lookup, batched per page). Dropped the redundant badge and raw model description. - CSV export of the currently-filtered logs via a new GET .../usage-logs/export route (mode: 'text' contract, synchronous single-response CSV — the dataset is a bounded per-user ledger, not a workspace-wide export, so no async job queue needed). Query-filter logic (date-range resolution, workflow-name lookup) is shared with the list route via shared.ts rather than duplicated. - period/startDate/endDate live in the URL via a co-located search-params.ts; the list query keeps keepPreviousData + isPlaceholderData dimming during filter transitions, matching the behavior already shipped in #5391. Verified live end-to-end: back link navigation, custom range picker opens and applies, day presets, CSV export downloads and matches the on-screen rows exactly (credits reconcile with the total), compact Billing summary + link. * refactor(billing): move workflow-name enrichment into getUserUsageLogs, dedup helpers /simplify pass over the credit-usage-page branch (4 parallel review angles: reuse, simplification, efficiency, altitude): - getUserUsageLogs now LEFT JOINs workflow and returns workflowName directly (matching lib/logs/list-logs.ts's established pattern), eliminating the route-layer resolveWorkflowNames query that both the list and export routes previously ran independently. - Added includeSummary (default true) to getUserUsageLogs so the export route's cursor loop can skip the cursor-independent SUM/GROUP BY aggregate it never reads — that aggregate was being recomputed on every page of a paginated export for no reason. - Fixed an off-by-one in the export's pagination loop: `<= MAX_EXPORT_ROWS` let it fetch one more full page past the cap only to discard it; `< MAX_EXPORT_ROWS` with a shrinking per-page limit never overshoots. - Deduplicated the SOURCE_LABELS map (was defined identically in both the page and the export route) into a shared, DB-free source-labels.ts both can import. - Export route now builds CSV rows via lib/table/export-format.ts's toCsvRow/formatCsvValue instead of a hand-rolled escaper. - Added formatApportionedCreditCost to conversion.ts so the page's row rendering shares its zero/sub-credit wording with formatCreditCost instead of re-deriving the same three-way branch. - Replaced the generic requireStartDateForCustomPeriod<Schema> contract helper (nontrivial generic bound for a single four-line refine used at two call sites) with a plain shared error-options object. - Removed the credit-usage page's dateRangeAppliedRef guard — a controlled Radix Popover never re-invokes onOpenChange in response to the parent's own setState call, so the guard was defending against a re-entrant close that can't happen. - Added a modal prop to ChipSelect (forwarded to the underlying DropdownMenu, which already supported it) so a future call site that hits the same "modal select traps a same-tick Popover" bug the audit-logs Custom range fix worked around has a real fix available instead of having to swap components again. Re-verified live end-to-end after the refactor: workflow-name resolution, credit reconciliation, and CSV export all still correct. * fix(billing): drop Dollar cost from the CSV export, strip inline comments We only surface credits to the user, not the underlying dollar figure — "Dollar cost" was the one place the export literally displayed a dollar amount (the rest of the codebase uses dollarCost purely as an internal signal to distinguish a sub-credit charge from a genuinely free event, never rendered as a "$" value). * fix(billing): export honors partial custom date range, surfaces truncation Greptile (P1) and Cursor Bugbot independently caught the same bug: handleExport only forwarded startDate/endDate when BOTH were truthy, but the list query and both API contracts treat endDate as optional for a custom period (defaults to now). A user landing on a bookmarked ?period=custom&startDate=... URL would see populated rows and an enabled Export button, then get a 400 on click since the export omitted the required startDate too. Fixed by forwarding each date independently, matching the list query's existing behavior. Also addressed Greptile's other two findings: - The export route now sets X-Export-Truncated so a 5,000-row-capped download is visible to the user (a toast), not just a server log. Reading that header meant switching the trigger from a plain anchor navigation to fetch+blob — an anchor can't inspect the response before the browser commits to the download. - resolveDateRange now throws explicitly when a custom period is missing startDate instead of silencing the null check with `as string`, which would have produced a silent Invalid Date if ever called without prior contract validation. * fix(billing): remove the export's arbitrary row cap, fix a cursor pagination bug it exposed A personal credit ledger doesn't have the same unbounded-growth problem a workspace table does — capping the export at 5,000 rows just meant long-tenured or high-usage accounts (exactly the ones most likely to need a full export to reconcile a billing question) got silently truncated. Replaced the cap with a 50,000-row circuit breaker that should never fire in normal use (logged as an error, not a warning, if it ever does) and bumped the page size from 500 to 1,000 to cut round trips. Removing the cap surfaced a real, pre-existing bug in getUserUsageLogs's cursor pagination: a raw `sql` template embedded a JS Date object directly as a bound parameter, which the postgres driver can't serialize (unlike drizzle's typed gte/lte operators, which already handle Date correctly elsewhere in the same function). It only ever manifested past the first page, which nothing before this export route's tight multi-page loop reliably exercised. Replaced the raw sql template with drizzle's typed lt/eq/or/and operators, matching the pattern already proven correct in this file. Verified live: seeded 6,000 rows (past the old cap) and confirmed the export downloads all of them in one request with credits reconciling exactly against the total. * perf(billing): skip the redundant cursor lookup when the caller already has it The export loop holds the previous page's rows in memory, so its next cursor's createdAt is already known — getUserUsageLogs was still re-resolving it via an extra DB round trip every page regardless. Added an optional cursorCreatedAt to skip that lookup when provided; the list route's existing callers are unaffected since they don't pass it. Verified live: zero cursor-lookup queries fired across a 3,500-row / 4-page export that previously issued one per page. * fix(billing): apportion credits over the whole filtered set, not per page/call Cursor Bugbot caught this: the list route apportioned each page's rows against only that page's own dollar total, while the export apportioned every exported row against the complete set's total. Since apportionment depends on the full set, the same log could show a different creditCost between the list and the export, or even between two pages of the same "Load more" list — and the sum of every loaded row could visibly drift from the "Total" header shown above them once more than one page had loaded. Extracted getUsageCreditsByLogId — a single, shared, whole-filter apportionment lookup both routes now call instead of each computing their own subset locally. The list route calls it once per page request (same cost profile as the summary aggregate it already pays for every page); the export calls it once before its pagination loop, not per page, keeping the round-trip count this session's earlier fix already reduced. Also extracted the condition-building shared by the main query, the summary aggregate, and this new lookup into one buildUsageLogConditions helper, removing a third copy of that logic. Verified live: summed every row across 4 "Load more" pages and confirmed it now matches the reported total exactly (previously could drift), and confirmed the list and the export produce byte-identical credit sequences for the same rows. * fix(billing): make custom-range startDate/endDate nullable, not '' defaulted startDate/endDate had no sensible static default (they're only ever meaningful mid-custom-range), so defaulting them to '' via .withDefault('') meant switching back to a preset left the URL carrying startDate=&endDate= instead of dropping the params entirely. Made them nullable (no .withDefault) instead, matching the identical fields in the main Logs page's own search-params.ts. Verified live — switching from a custom range back to a preset now clears both params from the URL completely. * feat(audit-logs): add CSV export, matching the Credit usage page pattern Adds an Export chip to the top-right of the Audit Logs page (via SettingsPanel's actions slot — the same header mechanism the Credit usage page uses), downloading every audit log matching the current search/type/date filters as CSV. - New GET /api/audit-logs/export route: same session + enterprise admin/owner gating as the existing list route, reuses the shared buildFilterConditions/buildOrgScopeCondition/queryAuditLogs helpers (already using drizzle's typed operators for cursor pagination, not the raw-sql-with-embedded-Date pattern fixed elsewhere this session), and the same fetch+blob+X-Export-Truncated pattern the Credit usage export already established. - Capped at 10,000 rows (not the 50,000 used for a personal credit ledger) — an org's audit trail can genuinely grow much larger than one user's usage history, so this is sized for "a reasonable audit review window," with truncation surfaced via a toast rather than silently dropped. - Bumped the API-validation-contract audit's route-count baseline for the new route. Verified live against a real enterprise org: switched to "All time," exported ~750 real audit log rows, confirmed formatting (quoted descriptions, actor email fallback) and correct filter scoping. * fix(billing): skip wasted credit apportionment on the summary fetch, block export during stale data Cursor Bugbot caught two real issues: 1. The compact Billing summary glance (limit=1) only ever reads summary.totalCredits, but the list route unconditionally ran getUsageCreditsByLogId's whole-filter scan on every call including this one — pure wasted work for a caller that discards the result. Added an includeCredits query flag (default true, using the shared booleanQueryFlagSchema) so useUsageSummary can opt out; the main paginated view keeps it on since it genuinely needs per-row values. 2. Export stayed enabled while useUsageLogs held stale rows via keepPreviousData mid-filter-transition — a user could change the period/range and click Export before the new data loaded, exporting against the new filter while the table still showed the old one. Export is now also disabled while isPlaceholderData is true. * fix(billing): deterministic apportionment order, block audit export during stale data Cursor Bugbot caught two more real issues on the latest push: 1. Same stale-export bug as the earlier Credit usage fix, this time in Audit Logs: Export stayed enabled while useAuditLogs held prior rows via keepPreviousData, so it could export against a just-changed filter while the table still showed the old one. Now also disabled while isPlaceholderData is true. 2. getUsageCreditsByLogId had no ORDER BY before apportionCredits's largest-remainder tie-break, so which row absorbed a tied remainder credit depended on undefined Postgres row order — the same event's displayed credit could flip between calls (list vs. export, or even two successive requests). Added the same `orderBy(desc(createdAt), desc(id))` the main list query already uses, making the tie-break reproducible. Verified live: 3 identically-costed rows produced the same tie-break winner across 3 repeated requests (previously order-dependent). * fix(billing): distinguish a failed summary fetch from zero usage The compact Billing glance only branched on isPending, so once useUsageSummary settled into an error state, totalCredits stayed undefined and formatCreditsLabel(0) rendered "0 credits" — visually identical to genuinely having no usage this period. Now shows the same neutral "—" placeholder for isError as it already does for isPending. * fix(billing): gate the credit-usage page server-side for enterprise accounts Greptile (P1) caught this: hiding the "View usage logs" link on the Billing page for enterprise accounts doesn't stop direct navigation — anyone with the URL (bookmark, shared link, browser history) could still reach the full page and its CSV export, which enterprise accounts were never supposed to see at all (billing is managed out-of-band for them). Added a server-side check in page.tsx before anything renders: resolve the session, look up the highest-priority subscription, and redirect to /settings/billing if it's enterprise — matching how getHighestPrioritySubscription is already used elsewhere for server-side plan checks, rather than relying on a client-side-only conditional the way the Billing page's inline section does. Also fixes loading.tsx: it was a Server Component (no directive) passing a raw icon function reference into the client Chip component, which fails RSC serialization. Added 'use client'. Verified live in a real browser against both an enterprise account (redirects to Billing before any credit-usage content renders) and a non-enterprise account (reaches the page normally).	2026-07-03 19:37:17 -07:00
Vikhyath Mondreti	b4b666bfbe	improvement(forking): fork time ux (#5348) ... * improvement(forking): fork time ux * add storage quota * address comments * merge latest staging * address comments	2026-07-02 12:12:30 -07:00
Waleed	3d1e8c4f84	chore(ci): bump API contract boundary audit route-count baseline to 884 (#5375) ... The pinned totalRoutes/zodRoutes baseline (883) went stale after a legitimate new Zod-contract-compliant route merged without the required ratchet bump, breaking check:api-validation:strict on staging HEAD itself. Actual count is 884 total / 884 zod / 0 non-zod routes — a pure headcount update, no policy violation.	2026-07-02 11:03:59 -07:00
Waleed	59d6b8a62e	fix(onepassword): validate integration against API docs, add file downloads (#5365) ... * fix(onepassword): validate integration against API docs, add file downloads - add onepassword_get_item_file tool + route for downloading item file attachments (SDK items.files.read / Connect files/{id}/content), backed by newly-exposed item.files metadata on get/create/replace/update item - fix update_item JSON Patch applying array indices instead of 1Password's documented field-ID addressing (/fields/{fieldId}/...), which silently dropped field edits in Service Account mode - fix Service Account mode's list-vaults/list-items filter to honor SCIM `eq` exact-match semantics instead of always substring-matching - expand the create-item category dropdown from 9 to 19 real, creatable 1Password categories (was missing SOFTWARE_LICENSE, EMAIL_ACCOUNT, MEMBERSHIP, PASSPORT, REWARD_PROGRAM, DRIVER_LICENSE, BANK_ACCOUNT, MEDICAL_RECORD, OUTDOOR_LICENSE, WIRELESS_ROUTER, SOCIAL_SECURITY_NUMBER) - replace the block's single opaque `response: json` output with typed, per-operation output fields matching repo convention - remove incorrect password-masking on the Vault ID field (not a secret) - re-export tool types from the onepassword barrel * fix(onepassword): honor SCIM attribute name in filter matcher matchesFilter always compared against name/title regardless of the attribute named in the eq expression, so `id eq "..."` incorrectly matched against the display name instead of the id. * fix(onepassword): close output-parity and doc-string gaps from final audit - restore a deprecated no-op 'response' output so pre-existing saved workflows referencing it fail soft (empty) instead of hard-erroring now that per-operation outputs replace it - add missing block outputs (urls, favorite, version, state, lastEditedBy) for get/create/replace/update item so all real FULL_ITEM fields are discoverable as <Block.field> references - hide Connect Server credential fields for Resolve Secret (Service Account only) instead of leaving them selectable and silently ignored - correct two doc-string enum lists that advertised values the API doesn't return (vault type TRANSFER, item state DELETED) * fix(onepassword): fix silent data loss in update_item (Service Account mode) update_item applied user JSON Patch ops (documented/typed against the Connect-shaped vocabulary get_item returns: label/type/section.id) directly onto the raw SDK item, whose vocabulary differs (title/ fieldType/sectionId, and SDK category enum strings vs Connect's SCREAMING_SNAKE_CASE). Most patches beyond /title, /tags/-, and /fields/{id}/value silently no-opped or could corrupt the item while still reporting success. Extracted the Connect->SDK item conversion already used by replace_item into a shared connectItemToSdkItem helper. update_item now normalizes the fetched item to Connect shape, applies patches to that, then converts back before calling items.put() -- matching create/replace's existing translation pattern. Found via an adversarial final-verification pass that traced concrete patch operations by hand against the SDK's actual field vocabulary. * fix(onepassword): preserve field metadata and empty-title fallback connectItemToSdkItem rebuilt every field as a bare object, dropping SDK-only metadata (e.g. password-generation details) that a raw patch/replace previously left untouched. Now merges onto the existing SDK field by id before applying the translated properties, and only starts fields bare when they're genuinely new. Also restored the || (not ??) fallback on title to match replace_item's prior behavior of treating an explicitly empty title as "not provided".	2026-07-02 10:23:48 -07:00
Waleed	af53eda5ac	feat(landing): reintroduce /contact page styled like /demo (#5315) ... * feat(landing): reintroduce /contact page styled like /demo - Restore the /contact page (removed in #5181) with a two-column layout mirroring /demo: value prop + trusted-by logos on the left, a message form card on the right, on the platform light tokens and chip components - Restore the contact contract, /api/contact route (rate-limit, honeypot, Turnstile, help-inbox notification + visitor confirmation), now fully contract-bound via parseRequest - Add a useSubmitContact React Query mutation hook - Link Contact from the footer Resources column and add it to the sitemap * fix(contact): server-authoritative captcha + review fixes - Make captcha server-authoritative: drop the client-trusted captchaUnavailable flag; a valid Turnstile token is the only way past the stricter fallback bucket, so callers can't opt out of the challenge - Re-execute the Turnstile widget on every submit (incl. after expiry) instead of falling into the no-captcha path once the token expires - Reset the pre-submit gate on mutation settle so rapid double-clicks can't fire a duplicate /api/contact request - Map only feature_request to its email type; every other topic resolves to a General Inquiry confirmation so support requests aren't labeled bug reports - Drop the confirmation-email promise from the success copy (it's best-effort) - Collapse the duplicated no-captcha rate-limit branch; hoist shared response constants; read the Turnstile site key as a module constant * fix(contact): drop redundant Turnstile hostname pin The Turnstile site key is already domain-bound in Cloudflare, so pinning expectedHostname to the marketing SITE_URL (www.sim.ai) only rejected valid tokens issued on self-hosted, preview, and apex-vs-www hosts. Remove the pin and rely on Cloudflare's own domain binding. * fix(contact): fail closed on the no-captcha rate-limit backstop checkRateLimitDirect fails open on limiter-storage errors so a limiter outage never takes down normal traffic. But the contact route's no-captcha bucket is the only throttle on token-less submits, so a fail-open there let uncaptcha'd requests reach the email path unthrottled during an outage. - Add an opt-in { failClosed } option to checkRateLimitDirect; default behavior (fail open) is unchanged - Use failClosed on the contact no-captcha backstop so an unenforceable limit rejects instead of admitting - Cover both fail-open and fail-closed paths with tests * refactor(contact): TSDoc over inline comments Move the captcha-design rationale into the route handler's TSDoc and drop the inline body/JSX comments, per the project's TSDoc-only comment convention.	2026-07-01 01:17:11 -07:00
6 people	ca34301d7f	fix(mailer): permissions entitlements for enabling/disabling (#5312) ... * v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * fix(mailer): permissions entitlements for enabling/disabling * fix lifecycle for agentmail infra --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 21:23:11 -07:00
5 people	ad19f7fc40	improvement(landing): refine hero and mothership visuals (#5181) ... * stash * feat(landing): mothership feature stages + pre-footer CTA Tell-then-show landing: the Mothership section defines the five capabilities (Mothership · Pod · Formation · Dispatch · Return); the Features section now shows each as a real Sim UI callout floating over a static, edge-faded platform backdrop (Linear's "callout over a faded platform" pattern). - FeatureStage template: copy + masked static LandingPreview + elevated callout - LandingPreview: static autoplay=false snapshots with per-stage view/workflowId - Callouts: Mothership chat, model picker, parallel-agents Formation graph, deploy targets, logs table - Pre-footer CTA set over the Mothership render; removed the old capabilities grid Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(landing): reusable platform-page + solutions-page layouts and routes Add config-driven, padding-safe layouts consumed by route pages: - platform-page: hero (shared CTA) + centered logos + N card rows (3|4), JSON-LD, single <h1>, server-only; Workflows route as first consumer. - solutions-page: structural mirror (kept separate to diverge later); IT, Engineering, Finance, Compliance, HR routes under /solutions. - Hoist shared LandingShell/HeroCta/Logos to components/ (top-level = shared); refactor hero to consume them. - Restructure all of (landing) to the workspace folder-per-component convention (each component in its own folder + index.ts barrel). * refactor(landing): convert hero-visual CSS-module keyframes to Tailwind Move the hero-visual + stage-home keyframe animations out of CSS modules into tailwind.config (matching the existing dash-animation pattern) and delete both module.css files. Components now use animate-hero-* utilities + arbitrary properties for the per-element delays, SVG stroke draw, and gradient shimmer; reduced-motion preserved via motion-reduce: variants. Upgrade the shimmer's hardcoded #b4b4b4 to the --text-subtle token. brand-tokens.module.css is intentionally kept: it reassigns --surface-*/ --text-* token VALUES via a doubled-class selector for specificity over .light, which Tailwind utilities cannot express. * refactor(landing): move brand palette from CSS module into LandingShell Replace brand-tokens.module.css with a BRAND_TOKENS constant of Tailwind arbitrary-property utilities applied on the LandingShell wrapper, so the brand hex lives in the component, not a stylesheet. They emit in the utilities layer and override .light (@layer base) by cascade order — verified the brand --text-primary (#121212) wins over .light (#1a1a1a). No more .module.css files remain in the landing. * chore(landing): remove Testimonials from the home page for now Drop <Testimonials /> from the landing composition (component kept for re-adding later). * feat(landing): hero send→loader→workflow animation + landing WIP Hero visual: clicking send zooms into the button, morphs the disc into the gooey thinking loader (held, then cycling), slides it straight across to a phrase indicator with the camera following (no zoom-out), then zooms back out as the reply types and the chat morphs into the GitHub→Agent→Jira workflow. The chat card holds a fixed size through the zoomed scene and the greeting reserves its space, so nothing drifts; the user bubble reveals only on zoom-out. Loader ink tweens dark→gradient via the thinking-loader stop-color/flood-color transition. Also folds in in-progress landing work: knowledge + integrations feature callouts, CTA chat, mothership + line-glyph, wordmark tweak; removes the ethos and testimonials sections. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(landing): responsive pass for iPad + mobile Make the landing page fully responsive while keeping the desktop layout byte-identical (desktop classes stay the unprefixed baseline; smaller screens layer max-* overrides on top). - Navbar: hide desktop clusters below lg, add MobileNav hamburger sheet (scroll-lock, Escape/tap close, reduced-motion aware) - Hero: collapse the absolute split (visual + logos) to a stacked column below xl so iPad-landscape avoids the headline/visual collision - Mothership: 4-col grid steps to 2 (tablet) then 1 (phone) - Features: drop the floating callout below md, show the un-masked backdrop preview full-width - CTA + Footer: scale type/padding; footer 7-col steps to 3 then 2 - Document the breakpoint strategy in the landing CLAUDE.md Also includes the in-progress mothership goo/iso brand marks and the marks-lab preview route the section depends on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(landing): align hero visual panel to text + logos extent Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(landing): delay hero user bubble until card finishes expanding The grey user bubble's fade-in raced the card's upward grow on send. Hold the bubble's reveal until after the parent-driven grow settles so the card expands fully before the bubble appears. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(landing): isolate new landing — remove dead old-folder code + --landing-* coupling - Delete dead (landing) auth-modal (a duplicate of (home)'s, still on the old --landing-* / dark tokens) — its removal drops the new landing's last styling tie to the old landing. - Delete 5 merge-orphaned, zero-consumer callouts (deploy-callout, mothership-chat-callout, mothership-chat-preview, workflow-graph-preview, model-picker-preview). - Relocate the one live preview (logs-table-preview) into its consumer features/components/ + add a barrel; dissolve the owner-less feature-callouts/ shell. - Fix stale --landing-bg-surface reference in landing-preview-mount. (landing) now has zero (home) imports and zero --landing-* token usage. * refactor(landing): token-map hex, fix a11y/SEO, align structure Styling (within (landing)): - Replace ~90 hardcoded hex colors with the in-scope brand tokens they already equal (--surface-*/--text-*/--border*); divider edges -> --border, field/card edges -> --border-1. Delete the redundant C color-palette mirrors in the landing-preview home/sidebar and route them through tokens. - Convert static inline SVG styles (display:block/outline:none) to Tailwind. - 6 un-tokenizable hexes remain (dark send-button fills, status-green dot) — no brand token exists; left as-is. a11y / SEO: - Decorative mothership goo/iso marks: role='img'+aria-label -> aria-hidden. - Preview chrome titles <h1> -> <span> (kills duplicate client-only H1s). - sitemap.ts: add /workflows and the five /solutions/* routes. Structure: - Folder the bare logo-mark/mobile-nav leaves + barrels; complete the navbar components barrel and consolidate navbar.tsx to a single barrel import. * style(landing): restore taller hero panel with border-shadow chip chrome - Revert the right visual panel to the previous full-height framing (top-8 bottom-8) — hero text (pt-[112px]) and the logos panel are untouched, so their positions and spacing are unchanged. - Apply the canonical border-shadow chip surface: --surface-2 fill + the shared chipBorderShadowRing (1px hairline ring + soft drop shadow) from emcn, the documented chrome for a landing media panel. * feat(landing): swap Volvo for thinkproject and reposition hero logos - Replace Volvo with the thinkproject wordmark (official SVG, tagline/descriptor cropped out, all paths unified to --text-primary #1a1a1a; aspect 6.01). - Reorder the shared 6-logo set so the 3x2 hero grid reads: Rivian|VW (top-left), eXp Realty (top-center), Russell (top-right); Artie (bottom-left), thinkproject (bottom-center), Mobile Health (bottom-right). - Enlarge Rivian|VW a touch (height 15 -> 17, same aspect). - eXp Realty, Artie, Russell, Mobile Health, Rivian|VW all retained. * style(landing): size hero description with the type scale (text-lg) Replace the arbitrary text-[20px]/text-[16px] on the hero description with named scale tokens — text-lg (18px) desktop, text-md (16px) on phones — a touch smaller and the canonical lead size (1.2x the platform's 15px base). * style(landing): hero headline "for AI automations" with break after "agent" Replace "solving automations" with the higher-intent "AI automations" and move the line break after "agent" so "for AI automations." sits on the second line. * style(landing): unify CTA radius and box hero logos in cards - HeroCta email bar: rounded-[13px] -> rounded-lg, so the bar, the inset Book-a-demo chip, the Sign-up chip, and the navbar chips all share one radius. - Hero logos: box each wordmark in a bordered --surface-1 card (platform card chrome: rounded-lg + --border-1, 100px tall) on a responsive 3-up grid (2-up on phones) at a consistent gap-5 rhythm. Wide marks scale to fit (max-w-full h-auto). The platform/solutions 'row' layout stays bare wordmarks. * style(landing): concentric CTA bar radius + tighter logo cards - HeroCta email bar back to rounded-[13px] (= inner chip 8px + ~5px inset) so the Book-a-demo chip's right corners nest concentrically inside the bar. - Logo cards: smaller and tighter — h-20 (80px), px-4, gap-3 (12px, the product UI card-grid rhythm). * style(landing): restore 100px logo cards, scale icons down 15% The 80px cards read too wide-for-their-height. Restore h-[100px] (keeping the tighter gap-3/px-4) and instead shrink the wordmarks to 0.85x their optical size in the grid via GRID_ICON_SCALE — row layout unchanged. * style(landing): match sign-up radius to email bar + shrink logo icons - Sign-up chip overridden to the email bar's rounded-[13px], so the two hero CTAs share one corner radius. - Logo icons: GRID_ICON_SCALE 0.85 -> 0.65 and card padding px-4 -> px-2; card dimensions (h-[100px], gap-3) unchanged. * style(landing): shrink hero logo cards Cards read massive — too tall (100px) and stretched to fill the panel. Drop to h-16 (64px), cap width at w-[150px], and make the grid w-fit so it hugs the cards instead of stretching. gap-3 and the 0.65 icon scale unchanged. * style(landing): upscale hero logo cards ~25% Cards read too small. Bump all dimensions: h-16->h-20 (80px), w-[150px]->w-[180px], px-2->px-3, and icon scale 0.65->0.8. Grid stays content-hugging at gap-3. * style(landing): taller logo cards, larger icons, reorder top row - Card height h-20 -> h-[88px] (width w-[180px] unchanged), icon scale 0.8 -> 0.85. - Top row reordered: eXp (left), Russell (center), Rivian|VW (right). * style(landing): more card height, swap top-row Rivian/eXp back - Card height h-[88px] -> h-24 (96px); width unchanged. - Top row: Rivian|VW (left), Russell (center), eXp (right). * feat(landing): add "Trusted by technical teams at" label above hero logos Top-left, gap-3 above the logo grid (matching the grid rhythm); text-sm (navbar text size) in --text-muted (the label token). * style(landing): recolor logos to --text-body, match label gap to hero rhythm - Recolor all six customer logo SVGs to #3b3b3b (--text-body light value), so they match the Sim navbar wordmark's color. Landing is light-only, so the hardcoded value always equals var(--text-body). - Trusted-by label gap gap-3 -> gap-[22px] (the hero's description->CTA spacing). * style(landing): scale hero CTA down a hair, drop radius to the nav chip's Sign-up read too round. Take the bar + Sign-up to h-[40px] / rounded-lg (8px, the navbar chip radius), and keep the inset Book-a-demo concentric: h-[2em] + rounded (4px) with a 4px inset (8 = 4 + 4). * style(landing): round Book-a-demo to rounded-md to match the bar curve rounded (4px) read too square next to the bar's rounded-lg (8px). Bump to rounded-md (6px) — echoes the bar's curvature, still inside the 4px inset. * style(landing): match Book-a-demo proportions to the navbar chip Restore h-[2.143em] (the chip's 30/14 height ratio); with px-[0.571em] (its 8/14 padding ratio) and the 16px label, Book-a-demo now shares the navbar chip's exact height/padding/text proportions. * style(landing): equal inset around Book-a-demo (h-[30px]) Button was h-[2.143em] (34.3px) -> only ~1.9px top/bottom vs 4px right inside the bar's 38px inner box (40px minus the 1px border). Drop to h-[30px] (the nav chip height) so it centers to an equal 4px inset on top, bottom, and right. * style(landing): enlarge Book-a-demo to h-[32px], tighten inset to 3px h-[30px] read too small/airy in the bar. Bump to h-[32px] and pr-[4px] -> pr-[3px] so the inset is an equal, snugger 3px on top, bottom, and right. * style(landing): lift hero logos off the bottom again (pb-20) Restore the 80px bottom padding so the logos rest 112px above the section bottom (mirroring the hero text's 112px top) instead of sitting flush with the visual panel's bottom. max-xl:pb-0 keeps the stacked layout tight. * improvement(landing): refine hero and mothership visuals * fix(landing): cap hero fold height so it doesn't stretch on huge monitors The section was min-h-[calc(100vh-62px)], so on very tall displays both absolute panels (top-8 bottom-8) stretched — the visual panel grew gigantic and the bottom-anchored logos sank to the very bottom. Cap the fold at 960px via h-[min(calc(100vh-62px),960px)] (min-height can't be capped by max-height): the whole hero stops growing, panels/logos stay proportioned like a large laptop, and the next section just starts below. Laptops (<=16in) are unaffected; max-xl:h-auto keeps the stacked layout below xl. * refactor(landing): session cleanup — DRY CTA label, drop dead grayscale Final tidy after this session's hero/CTA/logo iteration: - hero-cta: extract the duplicated 16px label knob (px-[0.571em] + text-[16px] + font-size:inherit) into a single CTA_LABEL constant, matching the 'single knob' the TSDoc already describes — used by both Book-a-demo and Sign-up. - logos: remove the grayscale filter (now a no-op — all wordmarks were recolored to a single #3b3b3b), inline the single-use LOGO_GAP_X, and flatten the nested cn() into plain layout ternaries (dropping the now-unused cn import). * improvement(landing): animate mothership illustrations * style(landing): solid-ink branding + hero cursor/loader polish Branding: drop the bespoke BRAND_TOKENS palette and bottom-reveal from LandingShell (use the platform's own light tokens); re-ink the wordmark, logo-mark, and hero loader from the gradient+glow to a solid --text-body so the marks read as one ink with the nav text. Add a `shimmer` prop to ThinkingLoader for a static --text-body label, and stroke the squeeze arcs with the shared gradient. Hero visual: the cursor now enters from below the field and chases the send button live through the zoom (retimed beats, no arrive-then-wait); the greeting fades in gently instead of shimmer-revealing; the click ring becomes a press-dip (hero-cursor-press replaces hero-click-ring and hero-greeting-reveal). Extract BlockHandles so the morphed GitHub card carries a real edge handle in scene space; seed the compose card at its true height; pop the sent bubble in immediately. * improvement(landing): update feature iso-marks to perfected geometry Re-author the four Mothership iso-mark illustrations (Integrate, Ingest, Build, Monitor) on the refined isometric geometry, keeping the existing animation vocabulary intact: hover line-draw plus per-mark auto-motion (integrate float, ingest pulse, monitor panel-separate, build grid-flow). Map the raw exports onto the shared token palette/line weight for consistency and tune per-mark sizes for one optical weight. Build is now pure CSS (grid-flow replaces the RAF wave), so it drops 'use client' and renders as a server component. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(landing): add pricing, privacy, terms, and changelog pages - New public /pricing page: Free/Pro/Max/Enterprise cards with the full comparison breakdown transposed from shared upgrade data + JSON-LD; prices, CTAs, and features derive from shared billing constants so they can't drift from the in-app upgrade page. - Migrate /privacy, /terms, and /changelog into the (landing) route group via a shared prose-page system (single source of truth for legal/prose chrome). - Landing polish: solid-ink iso-mark illustrations + footer/cta/features/ mothership spacing and token cleanups; sitemap adds /pricing. - Audit pass: crawlable ChipLink CTAs, correct heading hierarchy, structured-data featureList derived from the visible comparison data, legal plan name Team->Max. * large edits across landing finalization * feat(auth): port OAuth-only signup + Microsoft provider from staging Align auth-page logic with origin/staging (PR #5073) while keeping the new chip-styled UI: - Add Microsoft as a better-auth social sign-in provider (auth.ts) and surface it through the OAuth provider checker, providers API + contract, login/signup forms, SocialLoginButtons, and the landing auth modal. - Gate email/password signup behind the emailSignupEnabled server flag (DISABLE_EMAIL_SIGNUP) so signup becomes OAuth-only when configured. - Add DISABLE_MICROSOFT_AUTH / DISABLE_EMAIL_SIGNUP env + feature flags. * fix(icons): render brand icons legibly when bare and on light tiles (#5292) Monochrome brand icons hardcoded a single white or black fill matched to their colored tile, so they vanished when rendered bare on the home Suggested actions list (white-on-white in light mode, black-on-black in dark mode). Convert those marks to currentColor so they adapt to context, and make tile foregrounds contrast-aware via getTileIconColorClass instead of a hardcoded text-white. Also centralize all color math in apps/sim/lib/colors (perceived brightness, hex/rgb/hsl conversion, contrast-text) and route every consumer through it: the bare-icon audit, block tiles, logs trace view, whitelabeling theming, workspace presence, and the PPTX renderer no longer carry duplicate copies. Adds a bare-icon CI audit (scripts/check-bare-icons.ts) and authoring guidance. --------- Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu> Co-authored-by: andresdjasso <andresdjasso@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Waleed <walif6@gmail.com>	2026-06-30 13:50:04 -07:00
Waleed	d0aed14a0f	feat(integrations): wave-4 tool-depth (Slack/Asana/Jira/Google Docs/Trello/Monday) + context.dev validation (#5289) ... * fix(context_dev): validation pass — add search numResults/country, accuracy fixes Comprehensive /validate-integration of all 22 context.dev tools against the live API docs found the integration clean (no correctness bugs). Applied the actionable items: - search: expose numResults (10-100) + country inputs (API supported them; users were silently capped at 10 results) - accuracy: scrape_html type description (+doc/docx), map meta description (+sitemapsSkipped), brand links description (+contact) - robustness: trim string query values in appendParam * feat(integrations): wave-4 tool-depth — Slack, Asana, Jira, Google Docs, Trello, Monday Deepen six existing blocks with 38 new tools, no new OAuth scopes (all under already-granted scopes), additive/backwards-compatible: - Slack (7): schedule/list/delete scheduled messages; archive/rename/set-topic/set-purpose conversation - Asana (8): create/get project, list workspaces, create subtask, delete task, add followers, create/list sections (via internal routes + contracts) - Jira (5): list/get project, get transitions, list issue types, get fields - Google Docs (6): delete content range, named ranges, paragraph bullets, update paragraph style (documents.batchUpdate) - Trello (7): create board/list, get board/card, add checklist/label/member - Monday (5): change column value, create board/column, get groups, duplicate item Route baseline 873->881 for the 8 new Asana internal routes. * fix(integrations): wave-4 validation pass — fix alignment enum, GraphQL input-object, scope/UI gaps Comprehensive /validate-integration of all 6 modified integrations (existing + new tools) vs live API docs. Fixes: - google_docs: CRITICAL alignment enum LEFT/RIGHT/JUSTIFY -> API enum START/END/JUSTIFIED (mapped); namedStyleType 'unchanged' option; 'zero-based' index wording - monday: CRITICAL search_items columns now emits GraphQL input-object with unquoted keys (was always failing the non-cursor branch) - slack: schedule_message DMs via user-id-as-channel; add channels:manage/groups:write/reactions:read scope descriptions; nextCursor optional - jira: list_projects expand=lead so lead outputs populate (was always null) - trello: get_actions limit now applies to the card path too - asana: add missing 'completed' + 'projects' subBlocks (were unsettable in UI); request permalink_url via opt_fields on create routes * fix(integrations): clamp context.dev search bounds; precise Google Docs index wording - context_dev/search: clamp numResults to the documented 10-100 range; normalize country to trimmed uppercase - google_docs: replace ambiguous '1-based'/'zero-based' index wording with the concrete fact (the document body starts at index 1), matching buildInsertLocation (index<1 appends) and buildContentRange * fix(integrations): validate context.dev country (ISO-2); regenerate google_docs docs - context_dev/search: reject non-2-letter country values with a clear error instead of forwarding them - docs: regenerate google_docs.mdx so the public index-contract wording matches the updated tool descriptions (body starts at index 1) * fix(asana): omit completed unless explicitly set (don't send false on unchecked) The new completion checkbox mapped an unchecked/untouched state to completed:false, which made update_task silently un-complete tasks and search_tasks filter to incomplete. Now only sends completed when the box is checked (undefined otherwise). * fix(slack): expose Destination toggle for schedule_message so DM scheduling is reachable The mapper already routes schedule_message DMs (user-id-as-channel); add schedule_message to the destinationType condition so users can deliberately choose Channel vs DM instead of it only triggering via leftover state.	2026-06-30 11:39:11 -07:00