743 Commits

Include renames

Author	SHA1	Message	Date
Vikhyath Mondreti	fa394e5e07	fix(execution): resolve secrets against the acting principal, not the workflow owner (#6690) ... * fix(execution): resolve secrets against the acting principal, not the workflow owner * fix(execution): resolve anonymous public-API runs as the workspace billing account * fix(execution): propagate run identity across dispatch paths and scope public runs to workspace secrets	2026-08-13 18:50:15 -07:00
4 people	1424809614	feat(netsuite): add Oracle NetSuite integration (#6476) ... * revise netsuite integration * fix(netsuite): align selector route with snowflake * test(netsuite): remove selector route coverage * test(netsuite): align coverage with snowflake * fix(netsuite): complete integration validation * refactor(netsuite): align integration with codebase patterns * test(netsuite): correct async job citation * fix(netsuite): address final audit findings * fix(netsuite): surface upsert/transform Location, relax task link check Oracle documents the Location response header for create and update, and both tools already require it. Upsert and transform also produce a record but Oracle documents no response headers for either, so they dropped the header entirely and the new record's ID was unreachable. Add a `resource-optional` location mode that captures Location when NetSuite sends it and never fails when it is absent, and wire it to upsert and transform along with their tool and block outputs. Async task discovery rejected the whole response if any task link carried a rel other than `self`, collapsing the picker into a 502. Oracle documents a `self` link per task but never guarantees it is the only one, so skip other relationships and fail only when no self link exists. Also use the shared `truncate` helper in the error sanitizer per the repo convention instead of an inline slice. * fix(netsuite): validate SuiteQL pages against their documented shape The shared collection-page validator required links, items, count, hasMore, offset, and totalResults on every 200, and a missing field turns a successful call into a reported failure. Oracle documents all six for record collections and SuiteAnalytics dataset pages, but its SuiteQL reference lists only links, count, offset, totalResults, and items. A documented SuiteQL response that omits hasMore would therefore have been rejected. Split out a suiteql-page validator that requires the five documented SuiteQL fields and type-checks hasMore only when the account returns it. Record collections and dataset pages keep requiring all six. * chore(netsuite): regenerate tool metadata after rebase on staging The rebase conflicted only in the generated tool-id, tool-metadata, and tool-output artifacts, which NetSuite and the newly landed LogRocket integration both extend. Regenerated from the merged registries: the result is staging's catalog plus the 27 NetSuite tools, with LogRocket's entries intact and no other tool changed. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-13 15:47:03 -07:00
Waleed	58b5ee9337	feat(logrocket): add LogRocket integration (#6678) ... * feat(logrocket): add LogRocket integration * improvement(logrocket): fail loudly on a non-numeric identify timestamp and cover releases in the catalog copy * fix(logrocket): require a highlights identity and treat pagination cursors as opaque strings * fix(logrocket): trim request fields so whitespace-only input fails validation * fix(logrocket): declare the release version in the block inputs map	2026-08-13 15:01:22 -07:00
Waleed	6de8ba2504	fix(v2): close the correctness gaps an end-to-end audit found (#6655) ... * fix(v2): stop a third-party tool description from 500ing MCP discovery `v2McpToolInputSchema` declared `description: z.string().optional()` inside a `.catchall(z.unknown())` object, and a declared key beats the catchall. The MCP SDK's own `ToolSchema.inputSchema` does not declare `description` at all, so any value — including the JSON `null` a Python server emits for an absent one — passes its validation and reaches Sim unchecked. The builder's outbound `.parse()` then threw, and the discovery error policy correctly declines to classify a Sim-side schema defect, so the endpoint that completes MCP onboarding answered a bare 500. The key is dropped and left to the catchall; `type`, `properties`, and `required` stay pinned because the SDK enforces those at least as tightly. Also in the v2 resources family: - The single-resource query schemas for MCP servers, skills, custom tools, and secrets are now `.strict()`, matching every list in the same family. A mistyped flag was silently ignored behind a 200. - `openapi/resources.ts` re-derived `RESOURCE_ERRORS` and `RESOURCE_CONFLICT_ERRORS` inline in 21 of 22 operations. They now import the shared constants; the generated spec is unchanged, which is the point. - The internal MCP refresh route stamped `updatedAt` alongside `lastToolsRefresh`. `updatedAt` means "configuration last changed" and is a public keyset sort, so a refresh moved rows out from under an in-flight page. `updateServerStatus` already held that invariant; the route now matches it. - The discovery cooldown is a typed `McpServerCooldownError` rather than a substring search for `cooldown`. `McpConnectionError` interpolates the server's display name into its message, so a server named after the word was reported as a transient cooldown when its connection had genuinely failed. * fix(v2): close correctness gaps in the workflows deployment surface Deploy and rollback bodies were plain objects, so a misspelled key was stripped rather than rejected. On rollback that is silent misbehavior: an omitted `version` legitimately means "reactivate the preceding version", so `{"versoin": 5}` rolled back somewhere else and answered 200. Both v2 bodies, the run-read query, and the versions cursor are now strict. Deployment versions are an `integer` column, but the path param, the versions cursor, and the v1 body each bounded it differently or not at all — an out-of-range value overflowed the comparison into an unclassifiable 500. One exported bound now covers all three. Resume admission raised bare `Error`s for a stale contextId or an already-resumed run, which the resume surfaces could not classify and reported as 500. They now use the sibling `ResumeAdmissionError` already in that file, carrying 404/409/400 and whether an automatic retry can clear the refusal. Docs corrections: rollback publishes the 409 its webhook-path conflict already produces; deploy/undeploy/rollback reject a workspace key with 403, not the concealed 404 they documented; the workflows OpenAPI module imports the shared error sets instead of re-deriving them; import and the folder ops explain their folder-tree 413. The export route is marked `headSafe: false` so a HEAD probe stops filing a WORKFLOW_EXPORTED audit event for an export that never happened. `runId` is one bounded schema across the run and log resources. * fix(v2): conceal knowledge upload existence, tighten knowledge/files bounds Security: the four knowledge document-upload routes rendered a bare upload error policy with no resource concealment, while every sibling knowledge route uses one. Because the use case resolves the knowledge-base context before workspace authorization, the unconcealed 403 told any valid API-key holder that a knowledge base exists in a workspace it cannot reach — the exact signal GET /api/v2/knowledge/{id} withholds by answering 404 either way. All four now use the composed concealing policy, which also renders the 415/402/413 the route-local renderer already handled; that duplicate renderer is deleted. Contracts: - POST /knowledge/search is strict. It was the only non-strict v2 request body, so a mis-cased rerankerEnabled or topK returned 200 with the key stripped, changing what the caller was billed and silently disabling reranking. - The document list takes limit, cursor, and search from the shared v2 schemas. search was an unbounded, empty-accepting v1 string, so ?search= answered 200 with a full page here and 400 on GET /knowledge, and the term reached an unindexed filename LIKE scan with no ceiling. - The 16 non-strict single-field workspace query slices across both families are strict, matching GET /knowledge/{id}/tags. - GET /audit-logs takes workspaceIdSchema instead of a bare string (?workspaceId= was forwarded as a filter and returned zero rows) and the shared run-window bounds for startDate/endDate. Documentation: - listAuditLogs drops the 404 it has no code path to emit. - upsertFileShare describes its workspace-key refusal as the 403 it renders; the operation denies the key by principal kind, which the concealment policy does not rewrite. - The 12 body-reading knowledge and files operations publish the 413 their pre-validation body read raises, and the file list publishes the folder-tree 413 its now-capped path index raises. Correctness: queryWorkspaceFilePage loads its folder path index under MAX_FOLDERS_PER_WORKSPACE like the workflow, table, and knowledge lists. An uncapped index does not fail on truncation, so a real folder outside the read rows resolved to undefined and answered "Folder not found". * fix(v2): publish the reachable 413 on body-carrying resources ops `parseRequest` buffers a JSON body through `parseJsonBody` under `DEFAULT_MAX_JSON_BODY_BYTES` before any schema runs, and the v2 builders supply `V2_PARSE_DEFAULTS.payloadTooLargeResponse`, so every operation whose contract declares a body already answers 413 above the cap. The resources family published it on none of them. A status a caller cannot see in the spec is a status they will not handle. Adds `RESOURCE_BODY_ERRORS` and `RESOURCE_CONFLICT_BODY_ERRORS` to the shared sets and applies them to the seven affected operations: createMcpServer, updateMcpServer, createSkill, updateSkill, createCustomTool, updateCustomTool, and setSecret. All seven are `defineV2JsonRoute` handlers on non-GET methods with no `parseOptions` override, so the 413 is genuinely reachable on each. The new sets are opt-in rather than folded into the base sets precisely because reachability is not automatic — an operation with no body, or one whose payload reaches it through an uncapped path, would be publishing a response that can never arrive. A sweep test pins the invariant across the resources, billing, and logs documents. It is one-directional by construction: several bodyless operations publish 413 for their own folder-tree and render ceilings, so the converse would flag correct documentation. Also completes the shared-constant consolidation started in cd3efefab9: `openapi/billing.ts` and `openapi/logs.ts` each re-derived `RESOURCE_ERRORS` inline in two operations. Both now import it, and both regenerate byte-identical. * fix(v2): head-safe binary downloads, coded 403s, and truthful surface docs Adds `headSafe` to `defineV2BinaryRoute`, mirroring the JSON builder: a HEAD on a route that declares itself unsafe is authenticated and rate-limited, then answered bodiless before parsing or executing. `GET /api/v2/files/{fileId}` is the one binary v2 route and it records a `FILE_DOWNLOADED` audit event, so a HEAD probe used to fabricate a download that never happened. Names the cause of five refusals that reached the wire as codeless 403s (billing principal-kind, personal-keys-disabled and role, secret admin and write, the workspace table quota, and public sharing), adding three members to the closed `FORBIDDEN_DETAIL_CODES` set. The billing cross-tenant refusal is concealed as a 404 instead of coded, and the credential-list and knowledge file-ownership refusals stay codeless deliberately, documented at the site. Makes `WORKSPACE_KEY_OPERATION_NOT_PERMITTED` reachable: an operation that denies workspace keys also omits them from `principalKinds`, so the kind guard always fired first and callers got `PRINCIPAL_KIND_NOT_PERMITTED` instead of the published code. Drops the unused 410 response, shares one `order` schema between the two run reads so both specs spell the enum the same way, and corrects the false statements about 403 codes, 413 causes, cursor schemes, and full-set lists in the conventions skill and the contract TSDoc. * fix(tables): close the v2 tables correctness and contract gaps - updateColumnOptions was the only column mutator with no lock assert: an options-only PATCH applied on a schema-locked table, and an option REMOVAL cleared cells on a delete-locked one. Assert schema always, escalate to the destructive gate only when options are dropped. - GET/DELETE /tables/imports/{id} 500'd on a first-party import job (null payload) or an unrepresentable status. Both now read as absent, so the answer is the 404 it always was. - Offset cursors stamped the sort but not the filters, so a page-2 cursor replayed under a different predicate paged an unrelated sequence silently. Offsets now carry a filter fingerprint and refuse a mismatch. - Publish 413 on every tables operation that accepts a request body: the v2 JSON builder reads the body under a byte ceiling before validation, so the status is reachable on all of them. Derived at document assembly so a new route cannot regress it. - Enforce MAX_VIEWS_PER_TABLE on view create, making the list contract's "small bounded set" claim true. - Accept the upload control token on the import read, so an upload-backed import is readable during the phase its own 201 reported; drop the `queued` status the reads can never return. - Declare the Find search-term cap, the Find match cap, and the run row-id ceiling the domain already enforces. - Uniform 201 on the row and column creates. * docs(v2): record why the two migrate-on-read GETs stay head-safe An enumeration of side-effecting v2 GETs flagged these two for issuing a workflow_blocks update. The write is convergent and would be issued by the next ordinary read, and headSafe: false answers 200 unconditionally, so declaring it would cost HEAD its existence check to prevent nothing. * fix(api): classify the caller input that reached the driver unvalidated Four families of caller-reachable 500s share one shape: a value the contract admits, the application forwards, and the database rejects. An unclassified driver throw renders as INTERNAL_ERROR, so a bad request came back as a server fault — on pure reads as well as writes. NUL bytes are rejected at the contract boundary, in parseRequest, not per field. A shared string primitive only protects the fields somebody remembers to build on it, and it cannot protect the values that have no string schema at all: a table cell and a predicate value are z.unknown() because their type belongs to the column, not the wire, and those are exactly the values found reaching the driver. One scan over the already validated params/query/body covers every field including the ones nobody has enumerated. Only U+0000 is rejected; every other control character is ordinary content that Postgres stores verbatim. Date bounds on a filter are now parsed, not merely type-checked, with the same normalizer the date column type uses to store cells — so the filter grammar and the storage grammar agree, and gt/gte/lt/lte on both JSONB date columns and the createdAt/updatedAt system columns answer an unparseable bound with 400 instead of an invalid-input-syntax 500. An afterRowId/beforeRowId anchor that does not exist is a classified not-found rather than a bare Error, and a zero-byte knowledge document is refused at admission: every parser rejects an empty buffer outright, so the upload could only ever consume storage and quota on its way to processingStatus failed. * fix(v2): stop six endpoints from returning a confident untruth Six defects that share a shape: a 200 that misrepresents what happened, which is the one class a caller cannot detect from the response. Knowledge search silently degraded. Reranking is implemented and does run, but a deployment with no Cohere credential, a provider error, or a timeout was swallowed into a warning log and answered 200 with plain vector ordering and no `rerankerScore` anywhere — indistinguishable from a reranker that ran and agreed with the vector order. The fallback stays (an outage should not take search down) and is now reported: `rerankerStatus` is required on every search response. v2 also omitted the `rerankerModel` default the internal contract supplies, so `rerankerEnabled: true` alone failed the use case's model guard and returned unreranked results after paying for the widened candidate retrieval; it now defaults like its sibling. `GET /billing/logs` accepted `startDate`/`endDate` with any relative period and dropped them, answering over the default 30-day window — a caller reconciling charges got real rows that were not the rows it asked for. Both bounds are now rejected outside `period=custom`, take the same strict UTC form as `GET /logs` via the shared `v2RunWindowBoundSchema`, and reject an inverted window instead of returning an empty page. MCP registration stamped `connectionStatus: 'connected'` and `lastConnected: now` at insert without contacting the endpoint, and did the same on any non-OAuth re-registration while leaving `lastError` stale. `tool-validation` gates tool availability on that column, so an unreachable server read as healthy. Both paths now leave the columns at their honest defaults for `mcpService.updateServerStatus` to move after a real discovery; the client-side optimistic copy matches. `skills.create` allowed a workspace API key while every other skill write denies one, so a key could only ever accumulate skills it could never remove — and the row it left was attributed to the workspace's billing owner, minting an editor grant for a human who did not act. Creation now denies a workspace key, making the lifecycle symmetric on the per-skill editor model that authorizes the rest of it. `runCount` counts successful non-paused runs and is never decremented by retention, so it disagrees with the runs list in both directions; the description now says so rather than claiming "total recorded runs". Run retention itself was undocumented — free-plan runs are hard-deleted after 30 days, which is why a workflow reports runs beside an empty list — and is now stated on both reads over the execution-log table. * fix(tables): refuse the writes v2 was silently discarding - Uncoercible cell values were stored as null under a 200 on any optional column: "abc"/true/[1] into number, "yes"/1/{} into boolean, "not-a-date" into date, an undeclared option into select, an object into string. The read side already 400s on the same mismatch in a predicate, so the two halves of the API disagreed about the same value. `coerceRowValues` / `coerceRowToSchema` now take an explicit policy and default to `reject`; `null` is passed only where a machine produced the value for a cell no caller typed — a computed (workflow/enrichment) write and a CSV import, neither of which has anyone to answer with a 400. - A multi-select coerced `["green"]` to `[]` — the drop was inside the registry, so no policy above it could see it. It now refuses any part that matches no option, which is what the single branch and the bulk retype gate already did. - A bare number in a date cell was read as epoch milliseconds, so the far more common Unix-seconds shape stored a timestamp 50 years early. The unit is not recoverable from the value and both readings are in range, so a bare number is refused in both directions and the retype gate no longer needs an override to be stricter than the write path. - Unknown column names were dropped by the name→id remap: an insert of {"nosuchcol":"x"} created an empty row under a 201, and a patch of {"zzz":"x"} answered updatedCount:0, indistinguishable from an empty match. The v2 row boundary now names them and refuses. - The table ceiling was enforced only inside createTable, which for an upload-backed import does not run until the CSV has crossed the wire: a full workspace got a 201 and a presigned PUT for up to 5 GiB, then a 403 at complete with an orphaned object left behind. The advisory check now runs when the session is created; the authoritative one stays in the transaction because the quota can move mid-upload. - Cap workflow groups per table. GET /tables/{id}/groups is published as a full-set list, and the group count had no bound of its own — the indirect one does not survive an update path that adds no columns. - Present a group's outputs/dependencies/inputMappings by column NAME. They are created by name, stored by id, and were read back as ids on a surface that is otherwise name-keyed, so a group could not be round-tripped. - Publish the predicate grammar: the operator set, the per-type restrictions, and that `*` — not `%` — is the wildcard. It was true only in the SQL builder's own comments, so the natural guess matched zero rows under a 200. - Stop advertising a `workflowId` default of "" on group create; a manual group that omits it has always been refused. * fix(v2): bind every paged list's cursor to its filters, not just its sort A v2 cursor names a position in one sequence, and a list decides that sequence from its sort AND its filters. Only the sort was stamped on the shared keyset codec, so a cursor from an unfiltered walk was accepted under a changed `search`, `scope`, `deployedOnly`, or folder and answered from a sequence the caller never asked for. The two offset lists already stamped both; nothing else did. The failure differs by scheme but is silent in both. An offset lands at an unrelated ordinal. A keyset stays internally coherent — correctly ordered, duplicate-free — and drops every match sorting before its position, which a caller holding an opaque token reads as "almost nothing matched". One mechanism, shared with the table-row codec: canonical JSON plus a SHA-256 fingerprint (`lib/api/cursor-binding.ts`), stamped by `cursorFilterScope` alongside `cursorSortKey`. The two stamps stay separate so the 400 names which half changed. `limit` is never bound — it selects how much of the sequence to return, not what it is. The three lists whose token is minted by a domain codec (`/logs`, `/audit-logs`, `/billing/logs`) get the same binding by wrapping that token in a query-stamped envelope; the domain cursor is untouched. `present` now also receives the parsed request, so a presenter reads the filters it stamps straight from the query instead of the use case carrying an HTTP cursor concern back out — the `cursorSort`/`cursorScope` round-trips through three application services are removed. `list-pagination.test.ts` now declares each paged list's binding and checks it against the contract in both directions, so a new list, or a new filter on an existing one, fails until its binding is decided. * fix(v2): authorize HEAD probes and declare every v2 query schema Two ways the v2 surface answered a request it had not checked. `headSafe: false` exists so a HEAD cannot fire the side effect its GET performs — an outbound MCP discovery, a FILE_DOWNLOADED audit event, a WORKFLOW_EXPORTED audit event. The short-circuit sat between admission and parsing, so it returned a bodiless 200 before resource authorization ran at all: authorization lives inside the use case, and the use case was exactly what the short-circuit skipped. Any valid API key drew 200 for a denied principal kind, a nonexistent id, another tenant's workspace, and a request missing a required param, while the GET beside it answered 403 or 404. That is an existence oracle over MCP server ids, file ids, and workflow ids. `OperationUseCase` gains an optional `authorize()` that runs the phase before the business transaction — allowed-principal check, canonical load, asserted-scope comparison, current access check — and stops. `defineAuthorizedWorkspaceUseCase` shares one implementation between it and `execute`, so the two cannot answer differently. A HEAD on a not-head-safe route is now admitted, parsed, and authorized like the GET, rendering refusals through the route's own error policy, then answered bodiless. The builders refuse at definition time to pair `headSafe: false` with a use case that has no `authorize`, so the next such route is a boot failure rather than a silent 200. Separately, `parseRequest` validates the query slice only when the contract declares one, so an omitted `query` means "never look at the query string" rather than "takes no query params". 69 v2 contracts omitted it and accepted anything: `?bogus=1` was a 200 on `GET /workflows/{id}` and a 400 on every list. They now declare `noInputSchema`, and 8 more contracts that declared a query without `.strict()` are tightened. A sweep over the contracts tree is the enforcement — a compile-time gate on `defineRouteContract` was tried and reverted because the required intersection collapses inference of the sibling generics. Four route tests appended `?workspaceId=` to a PATCH/PUT that reads it from the body; that copy was being silently dropped and is now a 400. The generated specs are byte-identical: the OpenAPI generator learns that a slice declaring no keys publishes no parameters. * test(tables): pin the multiselect paste on the refusal, not the silent empty cleanCellValue runs the same registry coercion the server does, so tightening multiselect on the server changed this helper too. The case asserting an empty array was pinning the silent-drop the tightening removed. * docs(v2): make the API-key security description render as plain prose The description was already published on every spec but did not appear in the rendered Authorization block. It carried a raw > and backticks, which the markdown pass in the docs renderer does not survive; the operation description on the same page renders fine. Reworded to plain prose with the same substance. * fix(v2): bind the query cursor to its filter on every shape Two agents each fixed half of this: the shared list codecs gained filter binding, and the table codec gained a fingerprint, but the pure-keyset shape stamped it on neither encode nor decode. A keyset position is absolute in (order_key, id), which is why it was left unbound — but absolute ordering is not completeness. Replaying the cursor under a wider filter silently omits every match sorting before it, so paging predicate A then B returned rows 7,9 where the full B sequence is 1,3,5,7,9. Also answers a lost create race with the conflict it already documents, and shortens three descriptions that dwarfed their siblings — the forbidden-code catalogue now lives on the error envelope's details field, published once per document instead of on all 135 operations. * fix(tables): make a saved view's column references survive the write A view config stores every column reference as a stable column id, but two things wrote it in different vocabularies and nothing translated between them. `config.sort` was pruned on read against the live column ID set while the contract defines `sort[].field` as a column NAME, so every name-keyed sort — the only kind the v2 surface can express — pruned to nothing and the view came back with `sort: null`, on both create and PATCH, with no warning. The same prune dropped a sort on `createdAt`/`updatedAt`/`id`, which are sortable row columns that simply are not in `schema.columns`. `config.filter` had the opposite failure: it was stored verbatim, so a predicate naming a column that does not exist saved happily and then 400'd on every `/query`, `/query/count`, and `/rows/find` that tried to use it. The write path now canonicalizes a config before storing it: every column reference (layout keys, `sort[].field`, each `filter` leaf `field`) is resolved to the column's stable id, and `filter`/`sort` are validated against the live schema so a reference that can never resolve is refused instead of saved. The v2 read presents the config back keyed by column name, matching `presentV2WorkflowGroup` and every other v2 row/data surface — a caller never sees a `col_…` id, and what it wrote is what it reads. Resolution is a lookup with pass-through, so the id-keyed first-party UI is unaffected. Column LAYOUT stays unvalidated on write and pruned on read: it auto-saves as the user drags, so racing a column delete must self-heal, not fail the drag. The read path still never prunes a predicate, for the reason already documented there — a pruned condition silently widens the view's row set. * fix(storage): validate at the decode and multipart boundaries, bound derived keys Four caller-reachable 500s shared one shape: input passed boundary validation, then failed in the storage/key layer. Each is fixed at the boundary that owns the transformation, not at the call sites. Percent-encoded NUL in a canonical folder path. `parseRequest`'s NUL scan sees `%00` as three ordinary characters; the NUL only exists after `parseFolderPath` decodes it. Reads survived as 404s, writers carried the decoded name into an INSERT and the driver threw. The rejection now lives in `encodeFolderPathSegment`, the single chokepoint both building and parsing funnel through, so it covers every escape a caller can spell. NUL in a multipart field. A multipart route declares no body contract, so its fields never reach contract validation at all — the knowledge-document key was sanitized while `original_name` was not, and the object landed in storage before the insert threw. `readFormDataWithLimit` is the shared multipart reader every such route already funnels through, so the scan goes there and runs before a caller holds a File to upload, which removes the orphan rather than cleaning it up. Storage-key overflow at 225 characters. Every generator embedded the file name in a path component it also prefixed with a timestamp and a uniquifier, so the effective limit was 255 minus that prefix while the contract advertised 255 — a 225-character name produced a 256-byte component and ENAMETOOLONG from local storage, and the upload session handed out a transfer URL that could never succeed. `buildStorageKeySegment` reserves the prefix out of the component's budget, making the key independent of name length and the declared limit honest. The NUL predicate is now shared from `@sim/utils/string` by all three boundaries instead of being restated at each. * docs(v2): make the published spec describe the API it has Three descriptions asserted behavior the code no longer has, and three rules the code enforces were published as unconstrained strings. `downloadFile` and `listMcpServerTools` still told callers a `HEAD` on a not-head-safe route "is answered with an empty 200 ... reports only that the endpoint exists and the caller is authorized". That was true of the old short-circuit, which sat between admission and parsing and therefore returned 200 for an id the same caller's `GET` refused. The builders now authorize a HEAD exactly as the GET, so the spec said the opposite of a security fix. One `HEAD_MIRRORS_GET` constant replaces both sentences and is added to `exportWorkflow`, whose `headSafe: false` was never documented at all. A test walks the `app/api/v2` tree for the declaration and fails on any operation that carries it without the sentence, or that resurrects the old claim. `createMcpServer` promised that re-registering an existing URL "rewrites the configuration and returns the server to the same unverified state"; it is a 409 pointing at PATCH. `authType` claimed Sim "detects it from the server when omitted" — registration deliberately never contacts the server, and the column defaults to `headers`. The default stays: `headers` and `none` are behaviourally identical (only `oauth` branches), so changing it is a migration with no caller-visible payoff, while the sentence was simply false. `predicate` was the API's most consequential gap: a `pipe` over `z.unknown()` documents from its input, so the leaf keys `field`/`op`/`value` appeared nowhere in the contract and `{column, operator, value}` was a 400 a caller could not correct against. Both predicate schemas now publish a real recursive JSON Schema through `.meta()`, self-referencing so the recursion resolves from one `$defs` entry, with every bound read from the constant that enforces it. Also published: the canonical folder-path rule and its 4096-byte cap on the four path components (the `superRefine` contributed nothing to JSON Schema); the closed 12-value `recursive` vocabulary on a destructive delete; and the null-matching behaviour of the negating operators. The clamping `limit` branch drops `minimum`/`maximum`, which in JSON Schema mean "rejected outside" and made SDKs refuse locally what the server clamps. `deleteFile` stops publishing a 409 nothing in its path can raise. `restoreFile` and `abortFileUpload` keep theirs — the report called them unemittable, but restore raises `FileConflictError` after exhausting its rename retries and abort refuses a completed session. Description tail, across the seven specs: p99 733 to 465, max operation 1643 to 1114, over 700 chars 31 to 13, over 400 70 to 61. Constraints moved from operation prose onto the fields they constrain rather than being deleted. * fix(v2): make upload completion, blank query values, search, and folder filters answer correctly Four defects on the v2 surface, each reproduced before it was fixed. Upload completion dispatched document indexing from inside the completion transaction, so a queue or processing failure returned 500 after the object was stored, the document row was created, and the session was marked completed — and the only recovery, replaying the request, answered 200. The dispatch is now a follow-on step that runs after the session is durably completed and is logged rather than raised. Its outcome stays visible on the document itself (`failed` with an error, or `pending` when it was never picked up), and the recovery path re-queues a `pending` registration instead of keying off a message left on the session. A query parameter sent with no value was read as `0`, `false`, or the parameter default: `?limit=` became `LIMIT 1` on the three lists that clamp, and `?minCost=` on `/logs` became a live `cost >= 0` filter. `search` and `cursor` already rejected a blank and documented "omit the parameter instead"; that rule now applies to every v2 parameter, enforced on the raw query before coercion so a parameter added later inherits it. The document list matched `_` and `%` in `search` as live LIKE wildcards while every sibling list escaped them through `searchFilter`, so the documented substring match returned everything for `a_itest`. It now uses the same helper. A `folderPath`/`folderPaths` naming no folder answered 404 on `/logs`, `/files`, `/workflows`, `/tables`, and `/knowledge`, while every other filter answers an empty page and the sibling folder lists already do. All five now return an empty page. Mutations keep their 404. * chore(v2): regenerate the specs from the merged sources The four spec conflicts in the wave-3 merge were resolved by taking one side, which left them describing neither branch. Regenerated so the published documents match the contracts they are built from. * docs(v2): give a built-in skill's id its real form The contract said a built-in skill uses its name as the id. The ids are `builtin-` plus the name, so a client following the description asks for /skills/research and gets a 404 where the spec promises the skill. * fix(uploads): keep local upload artifacts inside NAME_MAX `POST /api/v2/files/uploads` accepted a name of up to 255 characters, returned 201, and handed back a transfer URL that could never succeed: the PUT against it 500'd and `complete` then reported the object missing. The local provider named its staged object after the destination — `{key}.{uploadId}-{uuid}.tmp` plus a `.upload-metadata.json` sidecar — so the staged component was the key's length plus ~99 bytes of fixed overhead. Past roughly 125 characters of name that crossed POSIX `NAME_MAX`, and `ENAMETOOLONG` is not a `LocalUploadBodyError`, so it escaped as a 500. Multipart `complete` built the same name and failed the same way. Only local storage is affected; S3, Azure, and GCS have no per-component limit. `buildStorageKeySegment` already budgeted the key to 255, one layer above where the overflow happened. Two changes close it at the layers that own each suffix: - Staged artifacts move to a `.staging` root and are named from the upload id alone. A name derived from the destination inherits its length and then adds to it; a fixed-width one removes the arithmetic instead of re-budgeting it, so no suffix added here later can depend on the caller's file name. The staging root is a cleanup sweep root, which also reclaims artifacts that used to be orphaned beside the destination. - The durable sidecar is reserved out of the key budget centrally. `LOCAL_UPLOAD_METADATA_SUFFIX` moves next to the budget that must account for it, and the budget is derived from a list of sidecar suffixes, so adding one shrinks every key builder at once. The declared `maxLength: 255` stays honest: a 255-character name now completes PUT and `complete` end to end. * fix(uploads): budget every key built from a caller-supplied name Auditing the rest of the codebase for the shape that broke the upload-session PUT found five more key builders that put an unbounded name into a path component local storage writes directly. Three are on the same route as the original bug: `table_import`, `profile_picture`, and `workspace_logo` built their key inline with `sanitizeFileName`, which maps characters and never truncates, while their sibling purposes went through `buildStorageKeySegment`. A 255-character name broke `table_import` at the metadata sidecar and the other two at the object write itself. The other two are local-storage writers reached from elsewhere: knowledge-base connector sync capped the document title at 200 and then appended a timestamp, a uuid and `.txt` on top of the cap, landing at exactly 255 with no room for the sidecar; the Mistral-OCR staging and chunk keys inlined the sanitizer with no bound at all; and inbound email attachments went into a key with neither sanitizer nor bound, on a file name an outside sender chooses. All now derive their component through `buildStorageKeySegment`, so the reservation is stated once. The upload-session test asserts it for every purpose the contract admits, which is what keeps a newly added purpose from reintroducing the hand-built form. * fix(v2): stop the logs and billing reads answering 500 or a silent restart Four caller-reachable failures on `GET /logs`, `GET /logs/{runId}`, and `GET /billing/logs`, each fixed at the layer that owns the guarantee. `minDurationMs`/`maxDurationMs` were published as `number` against an `integer` column, so `1.5`, `-0.5`, `2147483648`, and `1e30` all reached Postgres as bind parameters it refuses to parse. They are now whole milliseconds bounded to int4, and the generated spec says so. `0000-01-01T00:00:00Z` satisfies the published `date-time` pattern but names no instant Postgres can store, since the proleptic Gregorian calendar has no year zero. `v2RunWindowBoundSchema` now rejects it, which covers both log families and the files-audit read that share the schema. A scoped cursor whose inner token was the empty string passed the `typeof === 'string'` envelope check and then read as falsy in every domain reader, so both lists silently served page one again with a `nextCursor` inviting another lap — the exact failure `UNKNOWN_CURSOR_MESSAGE` exists to make visible. An empty inner is now unreadable, and the sibling `decodePublicLogCursor` gets the same treatment for its `id` half. The rejection message no longer names `sortBy`/`sortOrder`, which neither operation accepts. `GET /logs/{runId}` reported `folderPath: null` for both a workflow at the workspace root and a folder it could not resolve, so a caller could distinguish neither, and `null` is not a value `folderPaths` takes back as a filter. The root is now `/`, matching the workflow resources. Also, from the same audit: comma lists reject an empty entry the way `folderPaths` already did instead of dropping it; a query param sent twice is named as duplicated rather than reported absent; and the `triggers=all` sentinel, the detail-level promotion by `includeTraceSpans`/`includeFinalOutput`, and the 403/404 split against the billing family are documented where each is decided. * fix(v2): pin naive timestamps to UTC and close six contract divergences Application-written timestamps reached the wire as a local wall clock labelled `Z`. Every column in `schema.ts` is `timestamp without time zone`, so the instant a value denotes was decided by whoever wrote it and whoever read it, and the writers disagreed: `now()` renders in the session's TimeZone, drizzle's `mapToDriverValue` is `toISOString()`, and a raw `Date` bound through postgres.js is cast down in the session's TimeZone. The read side disagreed the same way — postgres.js parses oid 1114 with `new Date(x)`, which is the process's local zone, while a value it hands back as a string is read as UTC by drizzle. The result passes every `date-time` check, so it silently corrupts sorts and range predicates and can place `updatedAt` before its own `createdAt`. `packages/db/timestamps.ts` removes the ambiguity at the driver boundary rather than at the call sites: the session TimeZone is pinned to UTC so all three write paths store the same wall clock, and oid 1114 is parsed as UTC so every read path recovers that instant. `withUtcTimestamps` merges both into a client's options, because `connection` is nested and a pool setting its own `application_name` would otherwise drop the TimeZone. Production already runs both in UTC, so nothing changes there; every other environment now behaves the way production does. Alongside it, six places where the published contract and the code disagreed: - Multi-select `ncontains` was documented as "the exception" that excludes nulls. It never did, and no test claimed it did — `data` is never NULL, so containment is false for an absent key and the negation is true, exactly like every other negation. The sentence was wrong. - `recursive` published twelve lowercase spellings while `z.stringbool()` folded case, so the server honoured `recursive=True` as a destructive recursive delete that a generated client would have refused to send. Narrowed to case-sensitive: accept exactly what is published. - The upload data plane answered with a bare `{ error: string }`. Being absent from the OpenAPI documents is a statement about addressability, not about behaviour; both PUTs now use the canonical envelope, and what the transfer step promises is published on `transfer.url`. - Full-set lists told callers to "send it back as `cursor`" on a `.strict()` query that rejects `cursor`. `v2CursorListResponse` now takes `paged`. - A `HEAD` on a download skips the read that produces `Content-Length`, so it cannot size a download; the description says so. - The upsert conflict-target rejection echoed the storage id a name-keyed surface had already translated to, and the scoped-cursor 400 named `sortBy`/`sortOrder` params `/audit-logs` does not accept. * improvement(v2): cut the extraneous half out of the published descriptions The v2 spec's description median was already healthy at 42 characters; the tail was not. 174 descriptions ran past 200 characters and 13 past 700, almost all of it rationale, cross-references, and constraints restated on the wrong object. Trim the shared error, folder-path, retention, pagination, and workspace-key constants first, since each is published on between two and twenty-seven operations. `FOLDER_TREE_TOO_LARGE` dropped the clause explaining why the tree has to load, `FULL_SET_LIST` dropped a second sentence restating its first, `RUN_RETENTION` dropped the `runCount` caveat that already lives on `runCount`, and the 503 and 499 descriptions dropped the paragraphs narrating why they are documented at all. That reasoning belongs in the TSDoc beside each constant, which is where it now is. Then the operations. Execute Workflow and List Runs each restated a rule their own parameters already carry — the `X-Run-Id` uniqueness claim and the `order` sort deviation — so both moved to the parameter that owns them. The run-status enum sent a caller to `paused.automaticResumeWaitingReason` and then explained that field in place of describing it; the explanation moved onto the field, which previously said only that it was "the reason automatic resume is waiting". Align the parameter vocabulary a caller meets in every family. One `cursor` description had forked on the table row query, one `sortBy` on knowledge documents, and the table row `limit` published neither its bounds nor its default. `nameSortCollation` is now a function of the column it names, so the knowledge document list can state the caveat about `filename` without claiming a `name` field it does not have. `scripts/openapi/documents.test.ts` pins `cursor` and `sortOrder` to one string each, and the retention window to both reads that publish it. Distribution over the seven documents: mean 71 to 67, p95 223 to 199, p99 453 to 370. Over 200 characters 174 to 147, over 300 94 to 59, over 400 54 to 20, over 700 13 to 9. The median is unchanged at 42. * fix(v2): keep one unreadable-cursor message Two branches each added the constant, in cursor-binding and list-query. It belongs beside its sibling REFILTERED_CURSOR_MESSAGE, so the list-query copy and its importers move there. * fix(v2): bind a cursor to what a set filter means, not how it was spelled workflowIds, triggers and folderPaths are comma lists the query treats as unordered sets, and tagFilters is an object whose key order carries no meaning. Fingerprinting the raw spelling bound the cursor to the spelling, so a caller who reordered an equivalent filter mid-walk got a 400 for a page that was genuinely the next one. * fix(v2, db): make two unfalsifiable tests observable and document strict query Three follow-ups on the w5 policy work: one decision recorded, two tests that could not fail. The `query: noInputSchema` sweep is kept. It is a real tightening — 69 v2 operations that ignored an unknown query param now answer 400 — so it was weighed rather than assumed. The v2 body slice on those same endpoints was already `.strict()`, and every v2 list already rejected `?bogus=1`, so the split was arbitrary rather than a promise: the same typo was a 400 on `GET /workflows` and a silent 200 on `GET /workflows/{id}`. A parameter the server drops without saying so is the bug class the lists' rule already exists to prevent. No first-party caller is affected — the two SDKs send only `includeOutput`/`selectedOutputs`, both declared; the UI and the desktop app make no v2 calls at all; `requestJson` appends nothing implicitly and no v2 cache buster exists; every docs example uses a declared param. A third-party caller appending a tracking tag does break, which is why the behavior is now documented in the API reference with the exact 400 body rather than left to be discovered, and why the reasoning sits in the v2 conventions skill next to the rule instead of only in a commit message. `packages/db/timestamps.test.ts` asserted that `withUtcTimestamps` registers a UTC parser on oid 1114 by reading it off a bare postgres.js client. Every real client is then handed to `drizzle()`, which overwrites that entry with a transparent parser, so the assertion held whether or not the parser had any effect. The mechanism is fine and stays: drizzle's own `PgTimestamp` mapper appends `+0000`, so the read is UTC-correct either way and the session `TimeZone` pin — the write-side fix — is untouched by `drizzle()`. The test now resolves the parser both before and after `drizzle()`, pins the clobbering it depends on, and asserts the instant recovered through the full composition, so a regression in either layer is red. `timestamps.ts` records why the inert entry is kept. `nul-byte-boundary.test.ts` embedded a raw U+0000, so git classified it binary and rendered it as `Bin 0 -> 4102 bytes` — the test proving the NUL hardening works was the one file a reviewer could not read. The escape is byte-for-byte equivalent at runtime. Two older files had the same defect and are fixed the same way. `check:source-text` now fails the build on a raw NUL in any tracked source file, and `.gitattributes` forces source files to diff as text so the next one is visible in review rather than hidden by it. * fix(w5): narrow three fixes that reached past the harm they were fixing The workflow-create `23505` handler answered for the whole transaction, which also runs `saveWorkflowToNormalizedTables`. `workflow_blocks.id` is a global primary key, so a block-id collision — an integrity fault already seen in production — surfaced as `A workflow named "X" already exists in this folder`. Match on the constraint name; any other unique violation propagates unchanged. Moving the knowledge dispatch out of the completion transaction was right, but a dispatch failure then committed the session as `completed` and left the document at `pending`, which nothing sweeps and `retryProcessing` refuses. Record the failure on the document instead, so it lands on the existing failed-document path, and describe what the code does rather than a recovery branch that cannot fire for this state. The MCP re-registration reset stopped a registration claiming a connection it never made, but reset for any re-registration. `isServerEligibleForDiscovery` skips an OAuth row that is not `connected`, so a rename removed every tool the server published with no path back. Scope the reset to url, transport, headers, auth type, OAuth credentials, and revival. * fix(tables): confine the write-policy tightening to what the caller sent The null-policy work made `reject` the default for caller-supplied writes, which is right, but it landed on the wrong values. - A partial update coerces the MERGED row, so an untouched legacy cell failed an unrelated column's update — and failed a paged bulk job after its earlier pages had committed. The merged-row callers now name the patch's keys; every other key follows the `null` policy, in the in-memory copy only (the write sends the patched keys alone). - A multiselect whose members do not all resolve returned `{ok:false}`, which on the machine paths that pass `'null'` — CSV import, computed writes, the cell-write snapshot — erased the whole cell. Those paths now consult a new `salvage` hook and keep the members that do resolve; a caller-supplied write still 400s on an unknown option. - Refusing a bare number in `date.coerce` reached the executor, v1, copilot and the grid. The refusal stays where there is a caller to tell, and `salvage` restores the milliseconds reading where the only other answer is a blank cell. Also: the cursor docblocks claimed pure-keyset cursors were left unbound while the code and its tests bind them; a saved-view create took the table's SCHEMA advisory lock, so it queued behind column rewrites whose statement timeouts run past its 3s lock_timeout, and now takes a views-scoped lock instead; and a view whose column was deleted could not be saved at all, because the Save chip always resends the filter — references the stored config already carries are now exempt while a newly introduced one is still refused. The cursor version is deliberately not bumped: the stamp is additive, unfiltered in-flight tokens keep working, and a filtered one fails with the accurate "restart paging without the cursor" rather than a generic unreadable-cursor 400. * test(db): narrow the mapped timestamp to Date mapFromDriverValue is typed unknown, so the composition assertions did not type-check outside the test's own runner. * fix(v2): correct four stale contracts and clear the merge debris behind them Five of the reported defects were real and four of them were documentation that had stopped describing its own code. `cleanCellValue` said only "coerce a raw input value"; it also answers `null` for anything the column type refuses, and since the multiselect write path started refusing partial matches that is the difference between a paste storing one option and blanking the cell. It deliberately does not consult `salvage`, which would read the same paste as the option that did resolve — that reading is for writes with no caller to answer, and a typed cell has one. The pairing is now asserted, so a future helper that "improves" the paste by salvaging it fails. `EXECUTE_OPTION_CONSTRAINTS` carried two stacked TSDoc blocks, the second explaining that the enumeration had moved onto the fields; the body schema still told a reader the six combinations were enumerated in the constant. The deployment route's second block orphaned the endpoint documentation above it, and `list-query.ts` kept the TSDoc for a cursor message that now lives, with its own rewritten doc, in `cursor-binding.ts`. Two agents left near-identical essays arguing the same 400-vs-403-vs-409 question about the table ceilings and concluding that neither status changes; the decision is recorded once, in `billing.ts`, and `service.ts` points at it. The credentials use case echoed `sortBy`/`sortOrder` back with a TSDoc explaining that the presenter needs them, which it no longer does — it reads `query.*`. The local upload roots move from the data-plane provider to `core/storage-key.ts`, beside the sidecar suffix, so the cleanup sweep can name what it reclaims without importing the transport that writes it. `documents.test.ts` justified sweeping only knowledge and files for the 413 by saying the same sweep over the other five documents still reported gaps. It does not: widened to all seven, every body-carrying operation publishes it. Three reports did not survive checking, and the evidence is recorded where the next reader will look. An empty rerank result is not the reranker matching nothing — `rerank` asks for `top_n` over a non-empty document list, so an empty array means the response carried nothing usable, which is what `unavailable` already promises. The zero-byte knowledge document is refused on the upload-session path too, by `validateFile`, under both boundary contracts; that parity is now pinned, and it fails if the guard is removed. The MCP re-registration reports exactly the connection fields its SET clause writes, and the create mutation already drops both caches — what lags is the status badge, not the tools, because discovery is gated on `connected` for OAuth rows only. * fix(v2): de-duplicate a set filter before fingerprinting it The filters compile to inArray, which is set membership, so workflowIds=A,A,B selects exactly what A,B does. Sorting alone still bound them to different pages, so an equivalent filter with a repeated member 400d mid-walk. * fix(w6): close a head-authorization hole, a TZ leak, and five tests that could not fail Six risks an adversarial read of this week's diff raised, verified one at a time. Two of the six were already correct and are reported as such rather than changed. `v2HeadAuthorizationResponse` optional-called the use case's authorization phase, so a use case without one would have answered the bodiless 200 that `headSafe: false` exists to prevent. The definition-time guard does cover both builders that reach it — they are its only callers — but an optional call turns a missing phase into that leak silently, so the responder now refuses instead of skipping. `packages/db/timestamps.test.ts` assigned `process.env.TZ` at module scope and never restored it. `TZ` is process state: a worker running files back to back carried Asia/Tokyo into every file that followed, and only when the ordering put it after this one. The zone is now set and restored around the file, with both properties the suite depends on intact. Upload publication moved its staging area out of the destination's own directory into a shared `.staging` root, which makes the publishing `link` a cross-subtree one. A volume mounted under part of the uploads tree puts the two on different devices and `link` answers `EXDEV`, which the same-directory link could not. Publication now copies onto the destination's device and links from there, keeping the create-or-fail step that stops a replay from overwriting a stored object. Five tests that passed regardless of the code: - `resolveFolderPathFilter` was only ever exercised through hand-written reimplementations in the suites that mock it out, so widening a miss to unfiltered — every filtered list answering with the whole workspace — left them all green. The real helper is now tested where it lives. - The only measurement of `generateWorkspaceFileKey` asserted the key's last component against `NAME_MAX` rather than the component plus the sidecar written beside it, so it passed with the sidecar reservation removed. - `GET /logs` asserted only that a rejected cursor does NOT name `sortBy`, which almost any wording satisfies, including one saying nothing at all. - The skills lifecycle test asserted that the four writes agree on a workspace-key policy, which a lifecycle uniformly allowing one also satisfies; it now pins the policy they agree on and the kinds they admit. - The v2 skills create test lost `expect(capture).not.toHaveBeenCalled()` when the create path moved to a personal key. The behaviour it pinned is gone — the workspace-key create is refused now — so it is re-homed as the refusal reaching the caller as a 403 with no analytics behind it. Two claims did not hold. `CURSOR_VERSION` is correctly left at 1: the filter stamp is additive, a pre-stamp token still decodes, an unfiltered read still resumes, and only a filtered replay fails — with a conflict that names the filter, where a version bump would answer a generic unreadable-cursor 400 to every in-flight token. Tests pin all three, plus the minted version itself. And the upload-session key-budget cases do exercise the real shared budget through the real segment builder; only the workspace-key prefix is the stub's, which is now stated where the stub is declared. * refactor(v2): collapse two names for the cursor scope key onto one helper `cursorFilterScope` in the v2 response module was a one-line pass-through to `cursorScopeKey` in `lib/api/cursor-binding`, so the same function was reachable under two names from two modules. Routes now call `cursorScopeKey` directly, the way they already import `unorderedScopePart` and the cursor messages from that module, and the wrapper plus its duplicated doc comment are gone. Also folds the `id -> name` column map in the v2 tables presenter onto `buildColumnNameById`, which the same file already imports and calls thirteen lines above; restores two doc comments that had drifted onto the wrong declaration; and replaces three `as Date` casts in the timestamp test with `toEqual(new Date(...))`, which needs no cast and additionally fails when the mapped value is not a Date at all. * refactor: delete three pieces of surface this branch added with no consumer `v2CursorSchema` had one caller, `v2PaginationFields`, in the same file, and its only parameter was a default nobody overrode — so the export and the parameter were both unreachable. Inlined into the pair it belongs to; the emitted schema and its description are byte-identical, so the generated OpenAPI does not move. `PatchedKeys` was declared `ReadonlySet<string> | readonly string[]`, but all four callers pass `Object.keys(...)` and no test passes a set, which left the `instanceof Set` arm of `policyResolver` unreachable. Narrowed to the array form the callers actually use. `NUL_CHARACTER` was exported from `@sim/utils/string` and imported by nobody — every boundary imports `containsNulCharacter` instead. Kept as the module-local constant the predicate reads, dropped from the package surface. * docs(v2): state why the local upload data-plane routes bypass the builders Both local-storage PUT routes use raw `withRouteHandler`. The global rule allows that only for documented protocol or lifecycle exceptions, and their TSDoc explained the OpenAPI exemption and the error envelope but never the builder bypass itself. Record the actual reason: a signed `upload-token` is the credential, so there is no API key, `Principal`, or semantic operation for a builder to authenticate and authorize against, and the body streams straight to storage rather than being parsed. * test(v2): pin cursor-to-filter binding on the tables and runs lists The branch binds every paged cursor to the filters it was minted under, but the binding was enforced end-to-end on only 4 of 16 paged lists. The contract-level CURSOR_BINDINGS sweep looks like the safety net and is not: it checks each contract against a hand-maintained map of param names, never against what a route actually stamps into cursorScopeKey, so it stays green for a route that dropped the stamp entirely. Confirmed by deletion. Removing tableCursorFilters from both call sites on GET /v2/tables left all 8 tests passing, and the runs route was worse — its one relevant assertion was weakened from toEqual to toMatchObject in this same branch, leaving the new filter field unpinned. Adds a mint-then-replay test to each: a cursor minted under one filter set and replayed under another is a 400 that never reaches the use case, with a same-filter resume case as the control so the 400 cannot be satisfied by blanket rejection. Restores toEqual on the runs cursor payload, pinning that a filter is stamped without hardcoding the fingerprint. Both new guards were verified to fail: removing the binding reddens the refiltered test on tables, and both the refiltered and the re-armed toEqual test on runs. * fix(tables): keep the v2 write strictness inside v2 The write-path tightening on this branch changed shared code that every first-party surface reaches, so the workspace grid, the internal `/api/table` routes, `/api/v1`, the Copilot table tools, and the executor's Table block all inherited a contract only `/api/v2` publishes. Each of them now behaves exactly as it does on staging again, and v2 keeps the strictness by opting into it. - `coerceRowValues`/`coerceRowToSchema` default to the `null` policy again — an uncoercible optional cell is blanked and the row is written. `reject` is reached through `RowWriteOptions.uncoercibleValues`, which the v2 row routes set via `strictWrite` on the application input. - The same `strictWrite` scopes the unknown-column refusal to v2. Copilot feeds the model's raw arguments in unfiltered, so a hallucinated key, an echoed `id`, or a name left over from a rename had begun refusing the whole write. - Multiselect and bare-epoch values land again for first-party callers through the registry's existing `salvage` hook, which the `null` policy already consults; the grid's `cleanCellValue` consults it too, so a paste naming one live option and one deleted one keeps the live one instead of erasing the cell. - The saved-view name→id remap no longer rewrites a ref that already means something else, so a user column named `id`/`createdAt`/`updatedAt` cannot hijack a view's system-column sort or filter. - `createTableView` tolerates the refs its own config carries unless the caller is strict, so "Save as view" stops 400ing on a dangling filter the Save chip accepts. - The bulk update runner is byte-identical to staging again. The 100-view cap stays: the list read is unpaginated, so the promise it makes only holds if the write side enforces it, and it refuses a new view rather than an existing config. * test(v2): pin cursor-to-filter binding on seven more paged lists Extends the mint-then-replay guard from tables and workflow runs to the remaining paged v2 lists the audit found with no route-level coverage: credentials, audit-logs, custom-tools, mcp-servers, secrets, knowledge bases, and knowledge documents. Each gets a cursor minted by driving GET under one filter and replayed under another, asserting a 400 carrying REFILTERED_CURSOR_MESSAGE that never reaches the use case, plus a same-filter resume control so the 400 cannot be satisfied by blanket rejection. The three cursor schemes are all covered: keyset (readSortedCursor), the scoped wrapper audit-logs uses for its domain token, and the offset cursor on knowledge documents. The documents suite had no GET coverage at all, so its list use case gains a real mock and the route's GET export a describe block. All fourteen were verified to fail: dropping the cursor-filter argument from both call sites on each route reddens exactly that route's refiltered test and leaves every other assertion in the file green, which is the failure mode the contract-level CURSOR_BINDINGS sweep cannot see. * test: cover four untested behaviors and drop five tests that cannot fail Adds coverage that goes red when the behavior is reverted: - `rejectDuplicateQueryValues` through `parseRequest`, not just the pure helper — the existing blank-query tests stay green even when parseRequest ignores the flag entirely. - `failUndispatchedDocumentProcessing`'s pending + not-deleted WHERE guard, asserted on the condition tree so removing it fails. - The widened `present(result, request)` signature, so dropping the second argument stops being a silent no-op. - The NUL scan on `readFormDataWithLimit`'s content-length branch — the branch every ordinary browser and curl upload takes, and the one the existing multipart tests never reached. Removes tests verified incapable of failing: the credentials projection row (the outbound `.parse()` strips unknown keys either way), the per-document 413 sweep (vacuous on two of three documents, subsumed by the sweep in scripts/openapi/documents.test.ts), the two upload-session rows that assert their own `generateWorkspaceFileKey` stub, the storage-key row whose 20-byte name never reaches the budget, and the views-lock assertion against a function `views/service.ts` does not import. * fix(v2): parse a bound list filter once, so the scope matches the query The logs list fingerprinted `workflowIds`, `triggers`, and `folderPaths` through unorderedScopePart, which trims each member, then split the same raw values itself with `.split(',').filter(Boolean)`, which does not. So `?workflowIds=A,B` and `?workflowIds=A, B` produced one fingerprint and two different result sets: the second selects on a member with a leading space that matches no row. A cursor minted under one was accepted under the other, which is the exact failure the filter binding exists to refuse. Extracts parseUnorderedList as the single parse. unorderedScopePart now derives from it, and the route passes the array to the query and the joined form to the scope, so the members fingerprinted are by construction the members filtered on. Also drops three inline splits. Reported by Greptile. * fix(v2): bind an AND-conjoined filter array as a set, not a sequence The knowledge documents list fingerprinted tagFilters through canonicalJson, which sorts object keys but preserves array order. Each filter compiles to a condition in and(...whereConditions), and AND is commutative, so the same clauses written in a different order select the same documents — and got a different fingerprint, refusing a cursor for a page that was genuinely the next one. Adds unorderedJsonScopePart beside parseUnorderedList: members are canonicalized, de-duplicated, and sorted, so `A AND A` binds like `A` and clause order stops mattering. A non-array or unparseable value still binds by its raw spelling, since that request fails validation anyway. Replaces the route-local canonicalTagFilters, and corrects the claim on canonicalJson that array order only ever costs a restart — for a set-valued filter it costs a spurious 400. Reported by Greptile. * fix(v2): bind list filters by the value the query acts on, not its spelling Third report of one root cause, so this fixes the cause rather than the case. A cursor scope must fingerprint what the query filters on; every place it fingerprinted the caller's raw text instead, two spellings of one filter got two scopes and a valid next page got a 400. Knowledge documents: tagFilters bound the raw query text while the route already parsed it two lines below for the use case. The schema defaults operator to 'eq', so {tagName,value} and {tagName,value,operator:'eq'} are one filter to the query and were two scopes to the cursor. The scope now binds the parser's output, which also subsumes the clause-order fix — both route tests go red against the raw-text form. Logs and workflow runs: startDate/endDate bound the raw text, but z.string().datetime() admits every sub-second spelling of one instant, so `…00Z` and `…00.000Z` name one window and got two scopes. New instantScopePart binds the parsed instant. Replaces unorderedJsonScopePart, which took raw text and could not see a schema default, with unorderedScopeOf over the parsed value. Swept all fourteen routes that build a cursor scope for the same divergence; these were the only ones where a scope part is derived differently from the value reaching the use case. Reported by Greptile. * fix(v2): bind the audit and billing window bounds by instant The previous sweep for this defect looked for a transform in mapInput, so it missed the two routes that pass their raw bounds to a use case that parses them deeper. Both fingerprinted startDate/endDate as text while their predicates convert to a Date, so `…00Z` and `…00.000Z` name one window and got two scopes, refusing the genuine next page. Billing keeps stamping the raw params rather than resolveDateRange's output, for the reason already recorded there: a relative `period` resolves against the clock, so hashing the resolved window would reject every next page. Normalizing the explicit bounds is compatible — instantScopePart is a pure function of the caller's own text and resolves nothing. Re-swept all fourteen cursor-scope routes by scope part rather than by transform site. Every temporal and structured part now binds canonically; the rest are enums and identifiers with one spelling per value. Reported by Greptile. * fix(v2): drop an inert field from the document tag-filter scope resolveKnowledgeTagFilters builds every structured filter with the stored definition's fieldType and never reads the caller's — not for resolution, not for validation, not in its output. Fingerprinting it made a field the query ignores decide whether a cursor resumes, so adding or removing a matching fieldType refused a page that had not moved. Swept the other twelve cursor-scope routes for the same shape. No scope part is absent from its mapInput, this was the only scope carrying a structure resolved against stored state, and knowledge/search has no cursor at all. Reported by Greptile. * refactor(v2): derive the body 413 from the contract in every document Two mechanisms encoded one rule. `withRequestBodyErrors` derived the 413 from `route.contract.body` for the tables document, while the resources document hand-picked RESOURCE_BODY_ERRORS / RESOURCE_CONFLICT_BODY_ERRORS at nine sites. The cross-document sweep caught drift, but only after the fact: a new body operation that forgot the _BODY_ variant published a reachable 413 nowhere until a test failed. Hoists the mapper to openapi/shared.ts and applies it in both documents, so the rule is derived rather than remembered. The two hand-picked sets and their shared TSDoc are gone. Regenerating all seven specs produces zero drift, which is the proof the two mechanisms were computing the same thing. * refactor(v2): collapse duplicated cursor and validation mechanisms, drop dead exports One rule, one implementation: - `parseRequest` hand-inlined the "caller envelope or default" validation-error projection four times. Extract `projectValidationError` and route all four through it. - Nine keyset lists hand-rolled the `present` half of the cursor pair that `readSortedCursor` already owns the read half of. Add the symmetric `writeSortedCursor` and use it everywhere. - `GET /workflows/{id}/runs` re-derived `readSortedCursor`'s invalid/refiltered ladder from `decodeSortedCursor`; it now calls the shared reader and keeps only the key-arity check that is genuinely its own. Files and exports that no longer earn their place: - Inline `credentials/utils.ts` into its single consumer. - Delete symbols with zero references repo-wide: `v2CustomToolWriteError`, `secretCredentialTypes`, `v2CursorList`, `v2WorkspaceAccessError`, `resolveFolderPathIdentity`, `folderPathForId`, `v2FolderPathMutationError`, and seven of twelve `tables/utils.ts` exports. - Drop `export` from symbols used only inside their own module. No behavior change; every response body and error message is byte-identical. * docs(v2): cut duplicated and non-load-bearing comment prose Five rationales were written three to five times each by parallel agents that could not see one another. Each now has one home and the rest point at it: - HEAD existence oracle -> the headSafe option on defineV2JsonRoute - cursor query binding -> cursorScopeKey in lib/api/cursor-binding.ts - storage-key prefix budget -> buildStorageKeySegment - NUL / U+0000 -> the containsNulCharacter predicate - blank and duplicate query values -> their own implementations Also drops changelog-in-source (prose narrating what the code used to do), anchorless module headers attached to no declaration, rejected- alternative essays, and @param tags that only restate the signature. Comments only: the diff contains no executable-code change. * fix(v2): name the undecodable-cursor failure on the two sortless lists GET /workflows/{id}/versions and GET /workspaces/{id}/members threw a bare 'Invalid cursor' literal where every other v2 list uses a shared constant. The right one is UNREADABLE_CURSOR_MESSAGE, not INVALID_CURSOR_MESSAGE: both lists take only limit and cursor, so naming sortBy/sortOrder would answer one 400 with advice that earns a second. Their missing filter scope is correct and stays. Neither contract accepts a filter — v2PaginationFields is the whole query — so there is nothing to bind, and limit is excluded from a scope by design. Pins the message on the versions route, verified to fail against the literal. * test(openapi): give the determinism check a chosen timeout `serializes all documents deterministically` serializes all seven published documents twice — roughly 2MB of JSON — under vitest's 5s default, which is not a budget anyone picked for it. The published specs grew 3.3% on this branch (961KB -> 993KB) from richer descriptions, which is far too small to move a comfortable test and is enough to tip one already sitting just under the cap. Measured at 5.1s in isolation with nothing else running. Raises it to 30s for the openapi suite rather than trimming a real assertion. * fix(v2): make the NUL path scan linear, and force a write surface to choose Two findings from a simplify pass, both in code this branch added. findNulBytePath copied `[...path, key]` per child, which is O(nodes x depth). A caller controls that depth directly: v2 row cell values are `z.unknown()`, so nesting passes Zod untouched and reaches the scan. Measured on Node 22 -- JSON.parse accepts a 200KB body nested 100k deep in 9.8ms, and the scan then blocked the event loop for 27.7s. Frames now carry a parent link and the path is materialized once, for the node actually reported: 27.7s -> 5ms, with byte-identical paths across nested arrays, records, NUL keys and clean input. The always-run first pass drops Object.entries for Object.keys, which halves its cost on large bodies by not allocating a pair array per object. `strictWrite` was optional with the lenient default, so a v2 write route added tomorrow would silently inherit first-party behavior -- unknown column dropped under a 201, uncoercible cell stored as null -- defended by nothing but five copies of a literal. It is now required on the five write-shaped inputs, so omission is a compile error. The type-checker named every caller: the five v2 routes already passed true, and the three Copilot sites now say false explicitly, which is the behavior they already had. * refactor(v2): apply the body-413 mapper to every OpenAPI document The earlier unification wired withRequestBodyErrors into two of the five content documents and left files-audit, knowledge and workflows hand-writing the entry, so the helper's own claim that "a new body route cannot forget it" held on 40% of the surface while reading as global. Regenerating all seven specs produces zero drift, which is the useful proof: the mapper agrees with every hand-written entry today, so the gap was never a missing 413 — it was a missing guarantee for the next body route added to those three documents. The existing hand-written entries stay. The mapper is one-directional and several bodyless folder reads publish 413 for the folder-tree ceiling, so stripping them by hand would risk removing one the mapper cannot restore. * refactor(v2): fold the v2 validation renderer into the shared parse defaults V2_PARSE_DEFAULTS calls itself "the parse failures every v2 route renders the same way", but the option deciding how a v2 validation failure renders sat outside it and was re-stated at seven sites. A raw route that spread the defaults and stopped emitted a non-v2 error envelope. Removes the redundant line from the five sites that only restated it. The two builders keep theirs: theirs sits after `...options.parseOptions`, so it is a deliberate override that stops a caller swapping the v2 renderer, not a copy. Also adopts the mandated `filterUndefined` in cursorScopeKey in place of the Object.fromEntries/Object.entries form CLAUDE.md forbids, and collapses a one-element `as const` array plus a Math.max over it to the single `.length` they computed. * test(persistence): keep the wire round trip without tripping the utils audit check:utils forbids `JSON.parse(JSON.stringify(...))` and points at structuredClone, which is right for a deep clone and wrong here: this test exists to prove the schema accepts a `deployedAt` that arrived over HTTP as a string as well as an in-process `Date`. structuredClone preserves the `Date`, so adopting it would leave the test asserting nothing about the wire form. Splits the serialize and the parse into two statements. The round trip stays lossy — verified `JSON.parse(JSON.stringify(...))` yields a string where structuredClone yields a Date — and the pattern the audit matches is gone. Arrived from staging in #6660, so `check:audits` is red on origin/staging too, not only here.	2026-08-13 10:52:20 -07:00
Waleed	1fa40b8118	feat(v2): complete and align the v2 API surface (#6643) ... * fix(v2): close four validation holes in the logs and billing surfaces Each of these answered a caller-supplied value with a 500 or a silently wrong result instead of a 400. - `GET /api/v2/logs` accepted any string as `startDate`/`endDate`. The route constructs a `Date` from it, so `?startDate=abc` reached the driver's timestamp mapper as an `Invalid Date` and 500'd. Both bounds now carry `.datetime()`, matching the sibling run list so one timestamp works on both collections. This narrows the accepted set: a date without a time and an offset-bearing timestamp are now rejected, and the field descriptions say "UTC ISO 8601" rather than overpromising "ISO 8601". - `v2BillingStatusQuerySchema` was the only non-strict query schema in its family, so a mis-cased `workspaceID` was stripped and the caller got account-scope billing in place of the workspace scope it asked for — a wrong answer about money, served as a 200. - An unresolvable `cursor` on `/api/v2/billing/logs` applied no cursor condition and restarted the sequence at page 1 while still reporting `hasMore`, so a pager holding a cursor across a deploy loops over the first page and counts the same credits on every lap. It is now a 400. The message does not reuse `INVALID_CURSOR_MESSAGE`, which names `sortBy`/`sortOrder` params this collection does not accept. - The logs `status` field disagrees with the run resources for the same run: the run projection overlays `paused` from `paused_executions`, so an ordinary human-in-the-loop pause reads `paused` there and `pending` here. Reconciling would mean joining `paused_executions` in this read and silently moving live runs between two buckets of a shipped field, so the divergence is documented on the contract instead. * feat(v2): expose the MCP tool plane and page the MCP server list Registering an MCP server through v2 dead-ended: nothing on the public surface ever ran tool discovery, so connectionStatus, toolCount, lastError, and lastToolsRefresh stayed at their registration defaults and there was no way to read a server's tools without opening the UI. Adds GET /api/v2/mcp-servers/{id}/tools over a thin use case composed from the existing mcp_servers.tools.discover operation, resolveServerContext, and mcpService.discoverServerTools. It is personal-API-key-only — discovery resolves the acting user's own OAuth credentials, which a workspace key cannot supply — and the contract says so rather than letting callers meet an unexplained 403. Discovery failures are classified instead of collapsing into a 500: an unreachable or cooling-down server is a retryable 503, a stale OAuth grant is a 401. Also pages GET /api/v2/mcp-servers. It was the one unbounded list on the v2 surface, classified full-set on a bounded-by-construction rationale that only holds for folder lists; nothing caps how many servers a workspace registers. * feat(v2/tables): strict row bodies, a filtered row count, and round-trippable required columns Three tables gaps from the v2 capability evaluation. Strictness. Every v2 tables request body is now `.strict()`. The row family was the whole hole: `POST /query` sent v1's `filter` key answered 200 with a fully unfiltered page, because Zod strips unknown keys unless told not to. The same laxity covered the row create/update/delete/upsert/find bodies, the run and cancel-runs bodies, the enrichment body, and — outside the row family but the same class — the column delete, view create/update, and export bodies. A contract sweep now walks every body-bearing tables contract and fails if one of them stops rejecting an unrecognized key. Filtered row count. `POST /api/v2/tables/{tableId}/query/count` answers the question v1's `includeTotal`/`totalCount` answered and the `{data, nextCursor}` envelope has nowhere to put: how many rows a predicate matches. It binds the existing `queryTableRows` use case with `includeTotal: true, limit: 1` — no new domain logic and the same `tables.rows.query` read policy. The use case types `totalCount` as nullable because paged callers can decline it; this route always asks for it, so a null is treated as a broken invariant rather than presented as a fabricated zero. Required columns. `required` is accepted on create-table, add-column, and update-column, matching v1. v2 emitted the flag on every read while stripping it from every write, so a column could not round-trip. Enforcement was already complete: turning it on over rows with null, missing, or empty cells is rejected by the domain. * test(skills): pin the workspace-API-key split as structural, not accidental A workspace API key can create a skill it can then never update or delete, which no sibling resource does — so the asymmetry reads like an oversight worth widening. It is not. Skill edits are authorized by the per-skill editor row belonging to the acting user, which is why update/upsert/delete declare a 'read' floor rather than 'write': workspace role is not the authority. A workspace key carries no user subject, so allowing one replaces a 403 with an unclassified PrincipalSubjectUserRequiredError that the v2 surface renders as a caller-reachable 500. Records the reason on the registry and pins it, so the next reader finds the argument instead of flipping the flag. * feat(v2): read deployment state, and undo a file delete Two v2 reads that existed only as a side effect of a mutation. `GET /api/v2/workflows/{id}/deployment` publishes the state the deploy, undeploy, and rollback responses carry, plus `needsRedeployment` — which those responses structurally cannot carry, because they answer at the moment the draft and the live version are equal. A caller that lost the mutation response, or that polls from another process, had no way to ask. Reuses `readWorkflowDeploymentStatus` behind `workflows.read`, the same use case the internal status and deploy GETs already adapt. `DELETE /api/v2/files/{fileId}` was a soft delete with no way to see what it archived and no way to reverse it. `GET /api/v2/files?scope=archived` pages the archived set and `deletedAt` on the file resource dates each one; `POST /api/v2/files/{fileId}/restore` reverses the delete through the existing `files.restore` operation. Restore is not a pure undo — it returns the file to the root and renames it on a collision — so the use case now reads the file back and both the response and the OpenAPI description say what actually came back rather than what was deleted. `scope=all` is rejected on the list for the reason the internal contract already gives: it drops the `deleted_at` predicate and cannot use the partial index. `scope=archived` combined with `folderPath` 404s when the containing folder was archived too, which the contract documents. * fix(v2): keep the unresolvable-cursor rejection a 400 on every surface The cursor rejection lived in shared billing core but was an OrchestrationError only, which the session-only GET /api/users/me/usage-logs cannot project: that route is raw withRouteHandler and readTypedError matches instanceof HttpError, so any signed-in caller typing ?cursor=x got a 500. UnknownUsageCursorError is an HttpError carrying the OrchestrationError as its cause, so the v2 route still renders BAD_REQUEST off the cause chain and the internal route answers 400. Also closes the other half of the run-list parity: an inverted window on GET /api/v2/logs is now a 400 instead of a silently empty page. * fix(v2/tables): sweep union bodies per member and name the shapes on a rows 400 Review follow-ups on the strictness work. The sweep was vacuous on the one union body it covers. Parsing `{ notAContractField: true }` against `v2CreateTableRowsBodySchema` and looking for `unrecognized_keys` anywhere in the issue tree is satisfied by either member alone, so dropping `.strict()` from the single-row branch shipped green — reproduced, 36/36 passing with the regression in place. The sweep now flattens a union body into its members and asserts each one separately; removing `.strict()` from either branch now fails a case that names it. `POST /rows` answered an unknown key with `Invalid input`, the exact message the v2 conventions name as failing the actionable-error rule, because a union surfaces `invalid_union` first. The union now carries a message naming both accepted shapes; the per-member failures still ride along in `details`. Two TSDoc corrections. The `required` docstring claimed the domain rejects turning the flag on over rows with empty cells — true of the update path, false of add-column, which applies the flag as given (the same shape `unique` already had here). And `.strict()` binds the top level only, so the view `config` object and the shared sort-spec elements still strip unknown keys; both docstrings now say so instead of implying full coverage. * fix(v2): classify MCP discovery failures by type, not by substring The tool-discovery error policy consumed categorizeError's status, whose fallback is a substring match on the upstream message. Three consequences, all caller-visible: - A ZodError from the builder's own response `.parse` contains `invalid_type`, so a Sim-side response-schema defect answered 400 "Invalid request parameters" and suppressed the builder's 500 and its unhandled-error log. - An upstream `Invalid params` or `not found` became the caller's 400/404 on a request the contract had already validated. - A stale OAuth grant to the third-party server answered 401, the status this surface reserves for a missing or invalid Sim API key, so a client would rotate a credential that was never the problem. The policy now dispatches on the MCP error families and returns null for anything else. Reauthorization is a 409 carrying `details.code: MCP_SERVER_REAUTHORIZATION_REQUIRED`; an unreachable, slow, or cooling-down server is a 503 with a constant message. Also: widen the shared server path-param description now that it covers tool listing, map the list query explicitly so no undeclared `cursor` reaches the use-case input, and document the endpoint's write side effects. * merge: bring in the MCP tool plane workstream * feat(v2): make knowledge tags usable and let documents be updated v2 accepted tag slots on upload and filtered search by tag display name, but no response ever returned a tag value and nothing listed the vocabulary, so a shipped feature dead-ended in the public API. A document that failed processing could only be deleted and re-uploaded, and retiring 500 documents cost 500 requests. - GET /api/v2/knowledge/{id}/tags returns the vocabulary (display name, slot, field type) as a full-set list. - Document list and detail responses carry `tags`, keyed by display name exactly as search keys its result metadata. Writes stay slot-keyed; the tags endpoint is the mapping and the contract documents the split. - PATCH /api/v2/knowledge/{id}/documents/{documentId} renames, enables, disables, retags, or requeues processing. Derived indexing state is not writable: asserting `processingStatus` on an unindexed document would corrupt search. A retry may not ride along with field updates. - PATCH /api/v2/knowledge/{id}/documents bulk-enables or bulk-disables. Bulk delete is deliberately absent — that operation records no semantic audit, and a public bulk delete would empty a knowledge base leaving no DOCUMENT_DELETED entries. - The document list accepts the same name-based `tagFilters` as search; the name-to-slot resolver moves out of search into a shared helper, and the filters are stamped into the offset cursor scope so a replayed cursor cannot cross a filter change. - Search accepts `rerankerEnabled`, `rerankerModel`, `rerankerInputCount` and returns `rerankerScore`; `rerankerApiKey` and `skipUsageBilling` stay unexposed. Every result now names its `knowledgeBaseId`. knowledge.tags.list flips from workspaceApiKey 'deny' to 'allow' (and gains the workspace_api_key principal kind) so it matches the sibling reads knowledge.documents.list / read / search. The vocabulary is required input for two operations a workspace key can already perform. Every tag write stays human-delegated. * fix(v2): name every 403 cause, unfork boolean params, close nested strictness holes Four cross-cutting consistency gaps on the v2 public surface. **403s now carry a machine-readable cause.** The conventions skill mandated `error.details.code` on 403 and nothing emitted one, so a client had to string-match prose to tell "raise this member's role" from "this workspace refuses personal keys" from "buy an enterprise plan" — four different remedies behind one status, and every message reword a silent break. The vocabulary is a closed set, `FORBIDDEN_DETAIL_CODES`, with a `Record` of descriptions beside it that the generated OpenAPI 403 description is built from, so a code cannot reach the wire unpublished. Refusals throw `ForbiddenOperationError` in the domain and `v2CaughtOrchestrationError` — the function every v2 error policy falls through to — attaches the code, so a route cannot forget it. The audit-log resolver distinguished four causes and collapsed them into one; it now names each. Cross-tenant refusals deliberately get no code: they are concealed as 404 and naming their cause would hand back the existence signal the concealment withholds. **Two boolean query params rejoin the majority.** `?includeDeparted` and `?includeOutput` were `'true'`/`'false'` string enums inherited from the internal shapes they reused, while four sibling params were real booleans. Both move to `booleanQueryFlagSchema`, which still coerces both strings — a strict widening, so an existing caller is unaffected, and the spec stops telling callers to send a string. **Two nested strictness holes close.** `.strict()` binds the top level only, so `sort: [{ field, direction, nulls: 'last' }]` was answered 200 with the null-ordering request dropped, and an unknown key inside a saved view's `config` was accepted and discarded — the headline `filter` bug one level down. `sortSpecSchema`'s element and both view-config schemas are now strict. Safe on the read side because `normalizeStoredViewConfig` projects the schemaless stored blob onto the declared keys first, so a legacy row cannot turn into a 500. The two sort dialects stay as they are. `/logs` and `/workflows/{id}/runs` have one sortable column, so there is no `sortBy` to pair with; renaming `order` breaks every caller and an alias is a second spelling of one thing with undefined precedence. Both contracts and the skill now state the rule. * style: format the files the workspace-scoped lint gate does not reach `turbo run lint:check` runs `biome check .` per workspace, so `scripts/` at the repo root is outside the graph and four changed files were unformatted — one of them a merge artifact from reconciling the route baseline across branches. * fix(v2): collapse the four knowledge document projections onto one null-tolerant summary Extracts toV2DocumentSummary in app/api/v2/knowledge/utils.ts and composes the list, upload-acknowledgement and detail presenters from it. toV2TaggedDocument serialized uploadedAt with a bare .toISOString(), so a document with no upload timestamp threw where every sibling returned null and the contract declares the field nullable. Also consolidates the two Zod strictness walkers onto one shared introspection helper that unwraps wrappers and expands unions, closing the hole where a union-shaped schema answered null and was skipped by the pagination sweep. * fix(v2): stop HEAD driving MCP discovery, and unbreak the updatedAt keyset page B1: Next aliases HEAD onto GET, which RFC 9110 permits only because GET is safe. The MCP tool-discovery GET is not: it opens a live connection to the registered endpoint and writes the outcome onto the server row. The v2 JSON builder gains a headSafe option, default true, and the discovery route declares itself unsafe — a HEAD is authenticated and rate-limited, then answered bodiless. B2: a discovery status write stamped updatedAt, which this branch added as a keyset sort, so any concurrent discovery duplicated and skipped servers across a caller's pages. Discovery liveness already has lastConnected, lastToolsRefresh, lastError and statusConfig. B4: a public refresh now skips the positive cache but keeps the failure cooldown, so it cannot be used to drive a connection attempt per request at a failing endpoint. An explicit user action on their own server keeps the full bypass. B6: the consecutive-failure counter is incremented SQL-side rather than read, incremented and written back, and the success branch carries the same workspace, liveness and staleness guard the failure branch already had. * fix(v2): bound the bulk update echo, close the search leak, and make the docs true B3: a selectAll bulk document update echoed every changed identifier, which the request does not bound — a 100k-document knowledge base produced a multi-megabyte array, materialized and then element-wise validated. The use case now reports whether the selection was unbounded and the presenter omits the echo. A1: the knowledge search presenter spread the whole use-case result, which also carries userId, workspaceId, a cost breakdown and a live secret-trace registry. Only Zod's default key-stripping kept them off the wire. Projected explicitly. P1-a: GET /knowledge/{id}/tags advertised all 17 slots while the document PATCH accepted only the seven text ones. The writer already coerces every slot type, so the PATCH now takes all 17 in their declared types, with a 400 where a malformed value used to silently clear the tag. P1-b: both new PATCHes deny workspace API keys and now say so. P1-c: the two table query reads declare maxBodyBytes and now document the 413. P1-d: getWorkflowDeploymentV2 loses its legacy suffix. C3: deletes two orchestration error mappers with no callers that mapped 'forbidden' with no details. D2: a stored null in table_views.config survived the pick and failed the response schema. Also folds the six 'bounded set' paraphrases onto one FULL_SET_LIST constant, shares the run-window date bound between the logs and runs lists so their documented parity is enforced rather than asserted, adds the missing barrel export for FORBIDDEN_DETAIL_CODE_DESCRIPTIONS, and strictens two response schemas whose peers were already strict. Migrates 40 v2 route tests onto the shared @sim/testing harness: 26 asserted a rateLimitSubjectIds shape v2 auth never returns, 26 asserted the wrong refillRate, 33 could not exercise their 401 path at all, and 6 hard-wired the rollout gate to null. * fix(mcp): bound the connect handshake, and stop the 403 description over-claiming B5: the connect clamp was getMaxExecutionTimeout(), the workflow ceiling of seven days, so the real bound became the server row's own timeout — which the registration contract permits up to 300s — times the connect retries. A slow server could hold a Node request for roughly twenty minutes. Connecting is not a workflow run, so the handshake now shares the one-minute ceiling tools/list already applies to itself. C2: the generated 403 description asserted that error.details.code names the cause on every 403. Nine domain refusals still throw a bare forbidden OrchestrationError and reach the wire codeless, so the wording now says 'where the cause is one a caller can act on'. Reparenting those throws is left as a deliberate change: one of them is a cross-tenant refusal that belongs in the codeless class and would change its status. * chore: reconcile the route ratchet with staging * style: sort imports and format the three files biome flagged * fix(openapi): import the forbidden-code constants from their module, not the application barrel The barrel also re-exports the authorized use-case layer, which loads @sim/db at import time. That pulled a database connection into the OpenAPI spec check, so check:audits failed wherever DATABASE_URL is absent, including CI.	2026-08-12 15:04:24 -07:00
3 people	2805a8def9	feat(windchill): add document integration (#6577) ... * feat(windchill): add document integration * fix(windchill): align tool contracts and docs * fix(windchill): use official integration icon * fix(windchill): correct response and paging semantics * fix(windchill): align execution and API contracts * refactor(windchill): inline route authentication * fix(windchill): correct OData query encoding, content download, and cleared-field handling Validated the integration end to end against PTC Windchill REST Services 2.7 documentation and fixed every divergence found. Protocol correctness: - Encode OData query spaces as %20 rather than the form-encoded `+` that URLSearchParams emits. Every multi-token $filter and $orderby reached Windchill as a literal `+` and could not match. - Download content through the documented typed navigation `<content>/PTC.ApplicationData/Content/URL`, which returns a signed vault URL, instead of a `$value` segment that WRS does not implement. The resolved URL is pinned to the configured HTTPS origin. - Terminate every Stage 2 CacheDescriptor_array entry with `;` to match the documented grammar. - Raise the $top bound to Windchill's documented 2000 maximum, keeping 200 as the default page size. Cleared-field handling: - The executor merges raw block inputs before the block's param transform, so omitting a key could not clear it. A cleared numeric or boolean field reached the URL builder as '' and threw, and cleared optional strings failed contract validation. Coercions now emit an explicit undefined, and the internal-route body strips blanks centrally. Robustness and contracts: - Bound the document-structure walk to the depth actually requested. - Loosen response schemas that re-applied request-side bounds to provider-returned values, which turned committed mutations into opaque parse failures. - Return contract-shaped bodies for oversized, malformed, and unhandled request failures. - Normalize downloaded content types and drop charset parameters. Presentation and docs: - Square the icon to a centred tile on white. - Replace WT.Document and PATCH-compatible jargon with plain language. - Fix canvas sentence noun stutters on the bulk operations. - Correct the revision skill's unverified working-copy claim to read the OID back rather than assume it, and add retirement and stale-checkout skills. - Add a manual intro section to the integration docs page. * fix(windchill): align tool copy with the docs page and rebase the route baseline Tool descriptions feed both the integration catalog and the generated docs page, so the plain-language pass had to reach them too: drop WT.Document and PATCH-compatible from the operation copy, and correct the $top bound the descriptions still advertised as 200. Correct the docs intro's attachment wording, gloss OData on first use, and attribute the bulk-atomicity claim to PTC's documented behavior. Raise the API route-count baseline, which staging advanced while this branch was behind. * feat(windchill): add update common properties Name, Number, and Organization are rejected by the PATCH-based update operation, and the rejection message told users to reach for Windchill's UpdateCommonProperties action that the integration did not expose. Add it. PTC documents UpdateCommonProperties as a bound DocMgmt action taking an Updates wrapper, available when hasCommonProperties is set on the Documents entity, and refused while the document is checked out. The subblock and param descriptions carry that constraint, and the rejection message now names the operation that does the job. * test(windchill): assert block and tool params stay aligned for every operation Validating the new operation surfaced that nothing enforced the block-to-tool alignment the review process had been checking by hand. Assert it for all 27 operations instead: every required tool param has a required, non-advanced input under that operation's condition, and no operation shows an input its tool cannot accept. Both fail on a deliberately broken condition or a dropped required flag. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-12 13:18:31 -07:00
Waleed	9dfd9db3b0	feat(xai): wire reasoning effort through the Grok adapter (#6627) ... * feat(xai): wire reasoning effort through the Grok adapter The catalog never declared reasoningEffort for xAI and the adapter never sent reasoning_effort, so the flag was dead for every Grok model. Values are per-model and verified against the live API rather than the docs, which are wrong in three places: grok-4.5 does accept xhigh, grok-4.3 supports the parameter at all (undocumented) including none, and grok-4.20-0309-reasoning rejects it outright despite being a reasoning model. Also corrects grok-4.5's missing cachedInput and drops an inline comment the new provider TSDoc now covers. * test(xai): type the provider test helper instead of casting to any * fix(agent): correct reasoning-effort copy that still claimed GPT-5 only	2026-08-12 11:08:27 -07:00
Waleed	47f143016e	fix(docs): restore api-reference URL continuity and fix translated SDK bodies (#6617) ... Two docs-only defects introduced by #5273 (`263e3ca67e`), which re-founded the public API reference on the v2 surface. 1. Ten translated SDK snippets produce a deterministic 400. The streaming example in the five translated `api-reference/typescript.mdx` and `python.mdx` pages was repointed from `/api/workflows/{id}/execute` to `/api/v2/workflows/{id}/execute` and nothing else was changed — fr/ja/zh typescript.mdx are literally one-line diffs. `message` stayed at the body root. That was correct against v1, whose route treats the whole non-control body as workflow input, but `v2ExecuteWorkflowBodySchema` ends in `.strict()` and the route parses before executing, so every copied snippet returns `400 Unrecognized key: "message"`. The same commit fixed the English bodies to `input: { … }`, so this is an oversight, not a decision. The ten fences now match `en/api-reference/typescript.mdx:959` and `python.mdx:681`. Not relaxing `.strict()`: it is deliberate house style across the v2 contract and is what makes a typo'd option fail loudly instead of silently. 2. Thirty-two published operation pages 404 with no redirect. Replacing the single v1 `openapi.json` with seven v2-only specs changes page identity, because fumadocs derives every generated page as `slugify(tag)/operationId` from the specs at build time. Re-deriving both sets gives 52 old slugs and 128 new ones: 32 disappear and 20 keep their URL while silently retargeting v1 -> v2 (`knowledge-bases/updateKnowledgeBase` also flips PUT -> PATCH). All 52 are in the live sitemap — parsing `<loc>` from docs.sim.ai/sitemap.xml gives 458 URLs of which 56 are `/api-reference/`: the four static pages plus all 52 generated ones by name, including every one of the 32 that die. They are 200 today under an allow-all robots.txt. The spec swap itself is deliberate and CI-enforced (`check-openapi-specs.ts` requires every published operation under `/api/v2/`), so restoring the v1 operations is not an option. The missing piece is the redirect map, in a file that already carried 56 such rules from earlier doc moves. `permanent: true` (308) is used only for a true 1:1 successor — same operation, renamed. A 308 is cached indefinitely and effectively unrecallable, so anything that collapses two pages onto one, changes the identifier model, or lands on a merely adjacent operation is `permanent: false` (307). That splits 21/11. Four destinations differ from the mapping proposed in review, each on evidence from the specs rather than from the operation names: - `workflows/getJobStatus` is not destination-less. The v2 queued-execution receipt (`QueuedWorkflowRun`) returns `statusUrl` `/api/v2/workflows/{id}/runs/{runId}`, so `workflow-runs/getWorkflowRunV2` is the successor poll target — far better than a generic landing page. - The three HITL read operations go to `getWorkflowRunV2`, not to the resume page: `WorkflowRunStatus` carries a `paused` object with `contextId`, `pausedAt`, and `pauseKind`. Pointing a GET doc at a POST doc would be wrong. - `human-in-the-loop/listPausedExecutions` goes to `listWorkflowRunsV2`, whose `status` filter includes `paused`. - `tables/batchUpdateRows` is 307, not 308. v2 `updateTableRows` is "Update Rows by Filter" — the successor of v1 `updateRows` (PUT, predicate-based), which keeps its 308. v2 has no by-id batch update at all, so batchUpdateRows lands on a genuinely different operation. 3. A guard, so the map cannot rot silently. `scripts/openapi/docs-redirects.test.ts` recomputes the generated slug set the way fumadocs does and asserts no `/api-reference/` source shadows a live page and every destination resolves. Nothing else in the repo reads docs URLs, so a future spec regeneration would otherwise break the map with no signal. It needs no wiring: `check-openapi.ts` already runs this vitest config. The redirect array moves to `apps/docs/lib/redirects.ts` because the guard cannot import `next.config.ts` — `createMDX()` runs the fumadocs-mdx generator at import time, which made vitest emit an unhandled build error and warn about false positives. The 56 pre-existing rules are byte-identical to before, verified programmatically; `next.config.ts` keeps the same public shape and Next's own `checkCustomRoutes` accepts all 88 rules. Open question for the owner, larger than the redirects: all 82 `/api/v1` route files survive on staging, so a live public API now ships with zero reference docs, while the documented `/api/v2` surface returns 404 for any caller outside the off-by-default `v2-api` flag cohort. Is that the intended end state or transitional?	2026-08-12 01:49:59 -07:00
Waleed	34d65df7d6	fix(sdk): make the 0.2.0 SDK release safe to publish (#6616) ... The v2 SDK migration (#5273, #6564) shipped five breaking changes in both SDKs but got the release mechanics wrong in three separate ways, and left one of the two rewrites unable to complete a single successful call. Versions. packages/ts-sdk/package.json read 0.1.3 -- a patch digit added inside an unrelated compatibility commit, never deliberated. npm expands ^0.1.2 to >=0.1.2 <0.2.0, so every existing consumer would have picked the break up on a lockfile refresh: AsyncExecutionResult.jobId renamed to runId, executionId dropped from that interface, a failed sync run now throwing instead of resolving {success:false}, the request body reshaped, and the endpoint moved to /api/v2 with no fallback. 0.2.0 excludes every existing range, so the upgrade becomes opt-in. packages/python-sdk carries the identical break and was never bumped at all, so its publish job would have skipped green at the "version already exists" gate and left the repo and PyPI silently divergent; it moves 0.1.2 -> 0.2.0 in lockstep, along with the __version__ string in simstudio/__init__.py, which tracks pyproject and would otherwise have started lying. setup.py is left at 0.1.1: it is unchanged from main and demonstrably unread (0.1.2 published from pyproject while setup.py already said 0.1.1). It wants deleting, in its own commit. A 404 fallback was considered and rejected. The legacy 202 body's statusUrl points at /api/jobs/{jobId}, so mapping jobId onto runId would hand the caller an id that getWorkflowRun cannot resolve against that same old server -- a successful execute followed by an inexplicable failure on the next call is a worse contract than a clean 404. Both READMEs instead state the minimum server version and name the endpoint to check for. Cancelled runs. packages/python-sdk computed success as status != 'failed', so a run cancelled out of band reported success=True. The TypeScript SDK uses a closed whitelist and reports False, and before the migration both SDKs read the server's own value, which was False -- so this was a Python regression, not merely an inconsistency. Fixed by mirroring the whitelist. The v2 contract enumerates exactly completed|failed|paused|cancelled, so narrowing the blacklist to a whitelist cannot drop a live value, and a status added later now defaults to "not successful" rather than silently reporting True. WorkflowExecutionResult gains a status field because Python, unlike TypeScript, does not throw on 'failed' -- so success=False alone is ambiguous there in a way it is not in the TypeScript SDK, which is why status is not added to both. Rate-limit header. Found while auditing the two SDKs for further divergence, and the reason the Python bump could not have shipped as it stood: every authenticated v2 response now carries X-RateLimit-Reset as an ISO 8601 timestamp (recorded by v2RateLimits.publicApi, stamped by withRouteHandler). The Python SDK parsed it with int(), raising a bare ValueError that no handler in execute_workflow catches -- so every successful v2 execution raised instead of returning. None of the legacy endpoints the SDK previously called record a rate-limit snapshot, which is why the latent int() survived until the v2 move. The TypeScript SDK already branches on the format; _parse_reset_header mirrors it, including degrading an unrecognised value to 0, because a quota hint must not take down the call it rode in on. Timing metadata. The v2 rewrite stopped forwarding startedAt/endedAt, which main passed through and the TypeScript SDK still reports; restored under the same startTime/endTime keys the TypeScript SDK uses. Tests: cancelled/failed/paused status coverage, the ISO reset header, and the restored metadata keys, each verified red against the unfixed line first. The TypeScript suite gains matching cancelled/paused and ISO-reset pins -- they pass against today's source by design, and were confirmed to fail against a deliberately degraded copy so they are not toothless. Deliberately not included: a CI guard failing a PR that changes SDK source without a version bump. It would have caught this twice over, but it is a new script and workflow rather than a fix to the defect at hand. Review revision. bun.lock recorded packages/ts-sdk at 0.1.3 and was left stale by the first pass, so the repo asserted two versions for the same workspace package -- in a change whose whole thesis is that the version strings had diverged. It does not break CI (bun 1.3.14 accepts the mismatch under --frozen-lockfile, confirmed here), but 092311ea68 bumped the lock in lockstep with package.json, and the next unfrozen install would otherwise drop the line into an unrelated PR. _parse_reset_header gated the numeric branch on str.isdigit(), which accepts characters int() rejects ('²'.isdigit() is True, int('²') raises) -- and that int() sits outside the try, so the one function added to stop a quota hint raising could still raise, contradicting its own docstring. str.isdecimal() is exactly the set int() accepts. The tolerates-unparseable test is parametrized over both forms and was confirmed red on '²' against isdigit. Docs and docstrings: apps/docs api-reference/python.mdx mirrors the README's dataclass block and was the only copy left without the new status field. RateLimitInfo now names its units, because reset is epoch seconds for the legacy integer and milliseconds for the ISO form that v2 sends. execute_workflow's Args entry still described the pre-v2 body shape ("spread at root level"); every input is nested under input now, and this is the commit that ships that help() text to PyPI. The "declared last so positional construction keeps working" sentence was a maintainer's note that belongs in this message, not in every user's help(WorkflowExecutionResult).	2026-08-12 01:48:37 -07:00
Waleed	892401a8d0	fix(uploads): sign Azure upload URLs create-only (#6607) ... getBlobPresignedUploadUrl signed its SAS with BlobSASPermissions.parse('w'). Per Azure's service-SAS reference, `w` is "create or write content" and permits overwriting an existing blob; `c` is "write a new blob" and does not. main signed `c` before this signer moved out of core/storage-service.ts, so Azure deployments lost create-only enforcement in the move. The `If-None-Match: '*'` the signer returns cannot carry the guarantee on its own: an Azure service-SAS string-to-sign covers the resource, times, permissions and the five rsc* response-header overrides, never request headers, so a client is free to drop it. The header is signed on the other two providers -- inside the PutObjectCommand on S3, and as x-goog-if-generation-match in signed extensionHeaders on GCS -- which is why only Azure regressed. Without this, a signed upload URL stayed a plain overwrite grant on the final key for its full hour. A caller could replace the object after complete had already verified size and content type, written the workspace file row and metered storage from that verified HEAD, leaving durable metadata and billing describing content that no longer exists. The multipart block-staging signer keeps `w`: block staging is overwrite-shaped and matches main. The existing test asserted parse('w'), so it locked the defect in; it now asserts create-only, and its name states the guarantee so a future flip reads as deleting a security property rather than adjusting a value.	2026-08-11 23:10:42 -07:00
Justin Blumencranz	a72457b9f5	fix(docs): preserve items response fields (#6587)	2026-08-11 19:43:58 -07:00
Waleed	7c05e36049	fix(api): close five defects found auditing the v2 migration against main (#6575) ... * fix(tables): stop a column retype from nulling empty-string cells A type conversion rewrote every cell holding '' to null. Main only nulled a blank the target type could not read; '' is a real stored value that both string and json columns accept, so string->json and json->string silently destroyed those cells. Worse on a required target: countEmptyCells matches only a missing key, SQL NULL, or '[]', so '' passes the required guard and the rewrite then wrote null behind a constraint that had just succeeded. The per-cell decision is now the pure retypeCellRewrite, restoring main's rule: null a blank only when the target cannot read it, otherwise coerce. * fix(execution): release the concurrency slot when a group cancel is refused The stop-the-work effects (durable Redis abort record, queue-job cancel, in-process abort) all fire before the workflow-group sidecar is consulted, and none can be undone. When the sidecar refuses the claim we throw a conflict, which skipped releaseExecutionSlot and stranded the plan concurrency reservation until it expired. Every conflict return is a terminal-or-absent state - a missing log row, a log already completed or errored, or a terminal cell - so a refusal never means the run is still executing. The slot is released before the throw, keeping the exact success && !isPausedCancellationPath predicate rather than a blanket finally that would free reservations for live runs. * fix(uploads): recover an ambiguous PUT instead of discarding the object Main recovered an upload whose bytes committed but whose response was lost, via a verify endpoint. The session client retries the PUT instead, but every provider now signs a create-only precondition, so the retry returns 409/412, is classified non-retryable, and the session aborts - deleting the object that had already landed. A transient blip on the final ack cost the whole upload. A conflict on a retry attempt is now treated as our own earlier PUT having committed, and completion proceeds. That is safe because completeUploadSession independently verifies the object through assertObjectIdentity, which rejects on uploadId mismatch before anything durable is registered. A first-attempt conflict still fails loudly. * fix(folders): enforce the workspace folder ceiling on the create path Readers bound the active path index at MAX_FOLDERS_PER_WORKSPACE and throw once a workspace exceeds it, but POST /api/folders reached createFolder, which has no maxFolderRows field and never counts. A workspace could therefore be driven past the ceiling, after which the 27 capped read sites failed on a state the product had allowed. createFolder now asserts room inside its transaction, right after the mutation lock, so the count cannot be raced. The refusal is a typed conflict rendering 409 with an actionable message rather than a 500. The check counts rows directly instead of loading the path index, so an already-over-cap workspace gets a clean refusal rather than a read error, and no reader gained a cap. folderMutationStatus also gained the payload_too_large mapping it was missing, which had been rendering a delete-cascade cap breach as an unexplained 500. * fix(skills): route internal skill writes through the shared use cases The internal route made the workspace authorization decision itself, never consulting the skills operation policy, never loading canonical workspace context, and recording an audit entry with no operation id or actor projection. v2 and Copilot already went through the use cases; only this surface did not. GET/POST/DELETE now authenticate, parse, call the shared use case, and present. Request and response shapes are unchanged. Two behavior changes fall out: a write against a deleted workspace is now refused with 404 rather than accepted, and permission-denial text matches the rest of the platform. Legacy internal-JWT auth is dropped because no principal kind expresses that caller and nothing calls it: the whole repo references /api/skills only in two comments, no tool declares an internalRoute to it, and the executor reads skills through a direct listSkills call rather than over HTTP. * fix(folders): enforce the workspace ceiling on the remaining create paths Folder duplication, admin workspace import, and workspace forking all inserted folders without consulting the ceiling that 27 read sites enforce, so any of them could leave a workspace whose reads then fail. Each now asserts room for the rows it is about to add rather than one at a time: duplication measures the whole subtree up front, forking counts its bulk insert, and import counts per segment because that is genuinely one row. assertFolderCollectionHasRoom gained an additionalRows notion for the bulk case, and short-circuits when nothing is being added so an over-cap workspace still reads and still syncs. Duplication deliberately does not take the folder mutation lock. Holding it across the copy would block folder creation workspace-wide for an unbounded time - there is no cap on workflows per subtree and duplicateWorkflow runs sequentially - and narrowing it is impossible because an advisory transaction lock cannot be released early; splitting the transaction would leave a half-copied tree on failure. A rare few-row overshoot near the ceiling is the better trade, and it matches what forking already does. A test asserts the lock is absent so re-adding it is a visible decision. Admin import gained the transaction and lock it never had. Its folder-full refusal escapes the per-workflow result list, because a full tree is a property of the workspace and would otherwise be buried as N failures behind a 200. The fork and promote routes had no catch at all, and withRouteHandler only classifies HttpError, so a refusal rendered as an opaque 500 - twice over, since drizzle wraps the throw. Both now project a classified conflict as 409 and rethrow anything unclassified. * fix(uploads): bound the signed PUT lifetime and advertise its real expiry A single-PUT transfer was signed for the whole 24h upload-session TTL, because expiresAt was reused as both the session lifetime and the signing lifetime. Multipart part URLs in the same file kept 1h, and the pre-migration presigned route signed every PUT for 1h, so the widening was unintended rather than a policy change. No provider clamps below 24h. The PUT presign is now clamped at the provider boundary by a shared UPLOAD_URL_TTL_MS, which the part-URL path also uses so the two cannot drift. An expired PUT URL is deliberately not recoverable: unlike multipart, which re-signs per part call because its progress is durable, a PUT is not resumable, so an expired URL and an interrupted PUT have identical recovery. Nothing leaks, since every provider signs a create-only precondition. Clamping alone would have made the contract lie: the URL would die an hour before the session's advertised expiresAt, with nothing telling an integrator why the 403 happened. The PUT transfer now carries its own expiresAt, mirroring the multipart part-URL field. It is provider-dependent on purpose - cloud transfers report the clamped signature expiry, while the local data plane has no signature and admits against the session, so reporting an hour there would have been a new inaccuracy in the other direction. * chore: delete the dead presigned-upload and skill-adapter paths The presigned upload routes and the internal skills adapters were both replaced during the v2 migration, leaving their implementations behind with no callers. Removed generatePresignedUploadUrl and verifyPresignedUploadReceipt with their three provider helpers, QUOTA_EXEMPT_STORAGE_CONTEXTS and the types it orphaned, and the performCreateSkill/performUpdateSkill/performDeleteSkill adapters with recordSkillEvent and statusForSkillOrchestrationError. Each was verified unreachable across apps, packages, scripts and ee, including barrel re-exports and string access, not just direct imports. recordSkillEvent needed the closest look, since deleting an audit writer can silently drop coverage. The use cases declare the same action, resource, and description, and the framework adds the operation and actor the old helper lacked; recordAudit back-fills actorName and actorEmail from the user table when both are omitted, so the one field the helper passed is not lost. The self-hosting architecture doc described a directUploadSupported flag on an endpoint that no longer exists, and now describes the upload-session flow that replaced it. * refactor(folders): keep the cheap resource facts out of the schema graph Reading a folder resource type's label or its lock support meant importing folderResourceConfig, which imports the db schema for every table it serves and from there reaches lib/table/service, the executor, and the tool registry. That mattered as soon as lib/folders/queries needed a label: queries is reached from workspace-file-manager, which is reached from the files and chat pages, so one import edge put roughly 4,700 modules into those page graphs and broke the tool-registry boundary audit. Labels and lock support now live in a leaf module that imports only a type, and config composes them so there is still one source of truth. The three folder routes that pulled the whole config in for a single boolean read the leaf instead. * fix(skills): apply an upsert batch in one transaction The internal skills route looped the batch, calling an independently committing use case per item. A rejection on a later item left the earlier ones written and audited while the request reported failure - the compound-mutation rule in CLAUDE.md exists for exactly this. No new transaction plumbing was needed: upsertSkills already wraps its whole item loop in one db.transaction, so the partial commit came from calling it N times rather than once. upsertSkillBatch now validates and per-skill authorizes every item before issuing a single write, and createSkill and updateSkill became thin wrappers over it so v2 and Copilot keep one authority for the rules. The compound operation declares the read floor that skills.update already used, and the use case additionally authorizes skills.create when any item lacks an id, still ahead of every write. A read-only member who is a skill editor keeps their edit, and creates are not authorized more loosely than before. Audit projects one entry per committed skill, and analytics moved after the commit so nothing is reported for a rolled-back item. Note metadata.operation for these writes is now skills.upsert rather than skills.create/update; the action field still carries the distinction. * fix(security): close two disclosure gaps and finish the slot-leak fix The payer-pool gate only covered the workspace branch. A personal API key that omits workspaceId takes the account branch, where getHighestPrioritySubscription resolves an organization subscription from any member row regardless of role - so a plain member read the organization-wide credit and storage pool by dropping one query parameter. The account branch is now gated by the same authority, and the storage pool is not queried when it may not be disclosed. Forcing that branch self-scoped instead would have downgraded plan, period, and status, which is what a member needs to see whether the org is blocked. GET /api/v1/logs/executions/[executionId] emitted the workflow snapshot raw, carrying password sub-block values and oauth-input credential ids. It now shares the sanitizer the v2 read already used, extracted so there is one implementation rather than two. Env-var references are still preserved. cancelWorkflowGroupExecution itself was unguarded, so an unexpected throw from its transaction escaped ahead of every release site - the same reservation leak this branch set out to close, still open on the adjacent path. It now releases through the shared predicate and rethrows, because a failed transition means the cell state is unknown and a success-shaped answer would be a lie. The comment claiming the abort record cannot be taken back was false and now states the real reason: a refusal is always a terminal-or-absent state. * fix(v2-api): cover every persisted run status and every capped body The workflow-runs endpoints carried the same omission the logs contract had: the execution logger persists redacting, the run schemas did not list it, and because validation is whole-response one such row returned 500 for an entire page. Both schemas now derive from PersistedWorkflowExecutionStatus behind the same AssertNever gate, so a future status is a type error rather than a production 500. The single-run read keeps its extra queued value, which only it can observe. That schema was also serving as the run-list status filter. Widening it would have accepted a filter value the application input cannot express, so the reported set and the accepted filter are now separate schemas. Routes declaring maxBodyBytes without payloadTooLargeResponse fell back to a bare string with no error code and no private cache header. Rather than patch the four, the default moved into the builders - all three had the hole - which covers 58 body-bearing handlers, and a route override still wins. The five per-route overrides that merely restated the default are gone. Also documents the 413 on the one knowledge route that has a real body cap, adds the rollout gate's 404 to the last v2 operation missing it, and rewords the nextCursor description, which read as though every list were a full-set list. * fix(api): make the shared traits and lifetimes single-sourced The folder resource-traits leaf composed labels into config but restated lock support independently, so the routes reading the trait and the orchestration reading the config could disagree about which resources lock. Config now composes both, and supportsLocking is required rather than optional so a new resource type cannot silently omit it. The upload commit claimed the PUT clamp and the multipart part URLs shared a constant and could not drift. That was true only of the advertised expiry - the three provider signers each hardcoded an hour, so changing the constant would have moved what we advertise while leaving what we sign, recreating exactly the mismatch the clamp removed. Each provider now receives the lifetime in its own unit from the one constant. Table restore hand-rolled its status map and returned the driver message verbatim at 500, leaking the failed statement and its bound parameters - the same defect this branch closed at nine other sites. It and import-csv now use the shared projection; import-csv's result type also had to carry the lock the classifier already set, so a 423 can name it. Deletes v2RowWriteError, which had no callers and would have rendered a locked table as 400 by discarding the 423 it was handed.	2026-08-11 19:37:40 -07:00
Waleed	be5db68644	fix(agiloft): repoint the block at the alrest surface and fix EWLogin (#6562) ... * fix(agiloft): make the block work, and align it with the REST documentation The native Agiloft block could not authenticate against any instance. A customer reported it; production traces for their workspace confirm every failure mode verbatim. Fixing that exposed a second, larger problem, and a per-endpoint audit against the full published documentation found the rest. Authentication - EWLogin sent only $KB/$login/$password as query parameters. A live instance answers `400 EWWrongDataException ... One has to specify $table, $KB, $lang parameters`. $table is required even though only $KB/$login/$password/$lang are documented. Parameters now travel in a form-encoded body, which the docs permit and which keeps the password out of URLs and access logs. - The authentication scheme is read from the login response and trimmed; Agiloft returns it as "Bearer " with a trailing space. - EWLogout was missing $lang. Surfaces - Record create, read, update, search and saved-search now use the endpoints that accept the token EWLogin issues; the legacy operations authenticate from inline credentials, which is what that surface expects. Nothing sends both forms at once — the documented 400 for doing so is what the original report had run into. - EWSelect passes credentials in a POST body, one of the five operations documented to support it. - Attachment retrieval uses the documented EWRetrieve endpoint, with filePosition rather than position, and no longer needs a login/logout pair. Defects found in the audit - remove_attachment reported zero on every call: its body is the EWREST assignment form but the route ran JSON.parse then Number(), yielding NaN. - The EWREST parser could not read EWActionButton's documented response, which puts both assignments on one line. - EWLock treated any 200 as success, including the documented {error, error_description} envelope, and invented an 'UNKNOWN' status. - EWTable discarded the linked-field details, required flag and text field type it had asked for, making includeLinkedInfo inert. - select_records had no result ceiling at all; both it and search now cap and report a truncated flag rather than reporting a capped length as a total. - Optional string inputs rejected null, so a blank Page field failed validation before any request was made. - Upsert treated the documented 202 async acknowledgement as a missing-ID failure, and returned no callback ID for the caller to poll. - Every response contract required an output that the 401 and 500 paths never return. Coverage added - Table and field discovery (EWTable), upsert (EWUpsert), async status (EWAsyncStatus), natural language search (EWNLPSearch), action buttons (EWActionButton), the REPLACE_WITH_ANOTHER delete rule with its substitute records, $async on upsert, and <fieldName>$overwrite on attach. - Reads with a named field list go through the search projection; an unfiltered contract record runs to roughly 184KB and swamps downstream agent context. - Errors are readable: Agiloft wraps failures in HTML around a typed exception and an internal task id, and the JSON endpoints now request real status codes rather than a 200 the caller has to interpret. Not implemented: $searchSQL and $operationHints=NOLOCK are EWRead/EWUpdate parameters and those operations do not run on that surface here; EWQuestion, EWHotlinks, EWOData, EWBroadcast and webhook registration have no documentation beyond their names. Verified against the published documentation, not against a live instance. * fix(agiloft): give natural language search a sentence that paints check:canvas-sentences failed: the nlp_search card resolved to nothing on an untouched canvas, so it painted empty. Its only basic-mode field was the long-input query, and the field list is advanced, so every segment dropped. The sentence now leads with the knowledge base, matching the shape List Tables already uses — both operations are knowledge-base scoped rather than table-scoped, so it also reads more accurately. * fix(agiloft): stop retrying refusals, and expose the outputs the new operations return Five findings from review that had gone unanswered. An Agiloft refusal was surfacing as HTTP 500. readAlrestJson throws when the envelope reports success:false, the route catch mapped that to 500, and the tool runner retries 500s — so a create the server had already rejected could be retried and duplicate the record. Refusals now return a settled failure with the message intact; genuine faults still 500. list_tables could not run in its primary mode. EWTable is knowledge-base scoped, but some instances reject EWLogin without a $table, so whole-knowledge-base discovery failed at login with nothing to fall back to. It now says what the caller can do about it rather than surfacing the raw login error. Upsert corrupted structured values. Every field went through String(), so a multi-value field collapsed into one joined string instead of the documented repeated key/value pairs, and an object silently wrote "[object Object]" into the record. Arrays now encode as repeated pairs and objects are refused, since Agiloft documents no encoding for them. Two outputs were invisible in the editor. `records` was conditioned on search_records alone, so natural language search results could not be chained, and `callbackId` on run_action_button alone, so a queued upsert's callback could not be wired into Async Status even though both values exist at runtime.	2026-08-11 16:31:32 -07:00
Waleed	81e04a8e41	fix(agiloft): align the integration with the documented ewws REST interface (#6556) ... * fix(agiloft): align the integration with the documented ewws REST interface The CRUD tools targeted /ewws/REST/{kb}/{table}/{id} with JSON bodies and guessed at the response by probing `data.result ?? data` and `id ?? ID`. Agiloft documents that path as a URL convention only -- no method table, no example call, and no response shape -- and no known client uses it. The EW* operation family is specified end to end, including exact response bodies, so every operation now goes through it and parses the documented `EWREST_key='value';` assignment format. - EWCreate/EWRead/EWUpdate/EWDelete/EWSearch/EWSelect/EWGetChoiceLineId are form-encoded and parsed via a shared EWREST parser; the /.json suffix is kept only on EWAttachInfo, the one operation with a published JSON sample - EWDelete now sends the deleteRule the docs require, defaulting to ERROR_IF_DEPENDANTS so a delete fails rather than cascading - EWRemoveAttachment uses GET; it does not accept DELETE - EWSearch accepts the documented `search` saved-search label, so saved searches are reachable for the first time - Search query help taught AND/OR; Agiloft uses && and || - Add run_action_button (POST /ewws/async/EWActionButton) for approvals and send-for-signature steps - Drop saved_search: EWSavedSearch has no doc page, so neither its URL nor its response could be verified and it could only ever return an empty list - Add force on unlock, filter read fields locally since $fields is undocumented, correct lock status to LOCKED/NO_LOCK, and stop reporting a fabricated page size of 25 * fix(agiloft): fail loudly on non-EWREST bodies and keep the retired tool resolvable - EWSearch and EWSelect report an empty result set as `EWREST_id_length = '0';`, so a body with no assignments at all is a refusal Agiloft returned with HTTP 200, not an empty result. Both routes now surface it as an error instead of a successful empty list. - Re-register agiloft_saved_search as a retired tool. Removing it outright left workflows saved with operation='saved_search' deriving a tool id the registry no longer provided, which throws "Tool not found" at execution. It now fails through directExecution with a message pointing at the Search Records operation's Saved Search field, without issuing an undocumented request. It stays out of the operation dropdown so it cannot be chosen for new blocks. - Guard EWCreate and EWUpdate against oversized record data. Those operations carry field values in the query string, so a large payload hits the request line limit; the tool now explains that rather than surfacing an opaque 414.	2026-08-11 13:27:20 -07:00
Waleed	c365b14f73	feat(calendly): extend tools with booking, availability, no-shows, and routing forms (#6545) ... * feat(calendly): extend tools with booking, availability, no-shows, and routing forms Adds 12 tools verified against the Calendly OpenAPI spec: get_user, get_event_invitee, create_event_invitee, list_event_type_available_times, list_user_busy_times, list_user_availability_schedules, create_scheduling_link, create/delete_invitee_no_show, list_organization_memberships, list_routing_forms, and list_routing_form_submissions. Also fixes issues found while validating the existing tools: - list_webhooks dropped the scope query param the API requires, so every call with scope unset returned 400 - list_event_types could only send active=true, making inactive event types unlistable - user and organization filters now accept a bare UUID or a full URI consistently across every operation - json array params (eventGuests, events) are normalized whether they arrive as an array or a JSON string * improvement(calendly): type the block/tool alignment test instead of using any * fix(calendly): normalize webhook organization and user identifiers	2026-08-11 11:22:28 -07:00
6 people	263e3ca67e	improvement(external-endpoints): v2 versions with clean signatures + updated docs based on openapi spec (#5273) ... * v0.6.29: login improvements, posthog telemetry (#4026) * feat(posthog): Add tracking on mothership abort (#4023) Co-authored-by: Theodore Li <theo@sim.ai> * fix(login): fix captcha headers for manual login (#4025) * fix(signup): fix turnstile key loading * fix(login): fix captcha header passing * Catch user already exists, remove login form captcha * improvement(external-endpoints): v2 versions with clean signatures + updated docs * feat(usage): accept X-API-Key on usage-logs list + export /api/users/me/usage-logs and /export now use checkHybridAuth — the same auth /api/users/me/usage-limits already accepts — so external monitors can read summary.bySourceCredits (the source breakdown of usage-limits' aggregate currentPeriodCost) instead of estimating Copilot spend by subtraction. Workspace-scoped keys are pinned to their own workspace's slice of the ledger: the filter defaults to the key's workspace and an explicit mismatch 403s. Both endpoints documented in openapi-core.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(billing): dedicated v2 usage endpoints; keep internal usage routes session-only Replaces the earlier X-API-Key enablement on /api/users/me/usage-logs with a dedicated public surface, so the internal Billing-settings endpoints can evolve with the UI while external monitors get a stable versioned contract: - GET /api/v2/billing/usage — current-billing-period summary with bySourceCredits (the source breakdown external monitors need to watch e.g. Copilot consumption without estimating by subtraction), plus limitCredits and plan - GET /api/v2/billing/usage/logs — cursor-paged credit ledger in the v2 envelope - workspace-scoped keys are pinned to their own workspace's slice; personal keys read the account ledger The public wire is credits-only: usage-logs rows now carry a hasCost boolean instead of dollarCost (the Billing UI only needed the >0 signal), and the rateLimit block is removed from the usage-limits response and docs (deploy-modal tab relabeled accordingly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(docs): validate OpenAPI specs against the Zod contracts in CI The specs in apps/docs are hand-authored because they carry what Zod never defines — error envelopes, status codes, prose, examples — so they can't be generated; check:openapi validates them instead: - spec integrity: $refs resolve, operationIds unique, 2xx documented, no orphaned component schemas - v2 conventions: every /api/v2 operation documents 401 + 429 and every 4xx/5xx resolves to the canonical { error: { code, message } } envelope - contract cross-check: contracts are auto-discovered from lib/api/contracts/v2 (each carries its method + path); doc<->contract coverage both ways, query/body/response field diffs via z.toJSONSchema - examples: documented request/response examples must parse with the matching contract's actual Zod schemas First run caught real drift, fixed here: 16 stale orphaned schemas in the core spec, the v2 billing ops referencing v1-shaped error components, deploy/rollback examples missing the required nullable lifecycle keys, CreateTableBody missing folderId, a legacy-grammar delete-rows example, and four knowledge document ops missing their required workspaceId query param. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(docs): recursive field diff in check:openapi + the deep drift it found A mutation test showed the doc<->contract field diff only compared top-level properties, so a typo inside the { data } envelope passed. The diff now descends through matching object properties and array items (both sides must expose a property set — passthrough contracts and prose-only docs end the descent instead of false-positive), with the Zod JSON-schema root doubling as the $defs context. Deep drift it immediately caught, fixed here: select-column config (options/multiple) missing from every tables column schema, AddColumnBody hand-rolling a third column shape (now composed from ColumnInput, with position/workflowGroupId as the per-op extensions the contracts actually admit), chunking strategyOptions undocumented, and the deployment lifecycle fields (activeDeployment/latestDeploymentAttempt) missing from DeploymentState. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(security): close the triggerType rate-limit bypass on workflow execute Caller-supplied triggerType flowed unchecked into preprocessExecution, whose checkRateLimit default turns OFF for 'manual'/'chat' — so any API-key caller, and any anonymous public-API caller billed to the workspace owner, could execute unthrottled by sending {"triggerType":"manual"} (async runs also skipped the worker-side check via admissionCompleted). External callers may now only send the redundant 'api' value; internal JWT callers ('workflow'/'mcp') are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * refactor(execution): extract enqueue/status/cancel into shared libs Prepares the v2 execution surface: handleAsyncExecution's queue logic moves to lib/workflows/executor/enqueue-execution.ts (slot/claim semantics encoded in a discriminated outcome, not HTTP statuses), the execution-status read to execution-status.ts, and the order-sensitive cancel machinery to lib/execution/cancel-workflow-execution.ts. The v1 routes re-render identically — their suites pass unmodified. Also: preprocessExecution gains rateLimitCounter ('sync'|'async') and its 429 now carries code RATE_LIMIT_EXCEEDED + retryAfterMs (previously indistinguishable from the concurrency 429 and Retry-After was discarded); and the duplicate cancel contract in contracts/logs.ts is unified on the full 5-value reason enum — its narrower copy made requestJson throw a client ZodError when cancelling a paused HITL run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): callable execution service + structured error classifier executeWorkflowService composes the same libs the v1 route holds inline (call-chain guard, execution-id claim, LoggingSession, preprocessing, deployed-state load + file-field processing, timeout-bound executeWorkflowCore, output hydration/compaction) for the deployed-state caller class — the seam the v2 execute route and in-process internal callers share, making the HTTP endpoint syntactic sugar. classifyExecutionError stops discarding the block context that buildBlockExecutionError already attaches at throw sites: failed runs now yield {message, code, blockId, blockName, blockType} with a stable append-only code enum (TIMEOUT/CANCELLED/USAGE_LIMIT_EXCEEDED/ INVALID_INPUT/BLOCK_EXECUTION_FAILED/CHILD_WORKFLOW_FAILED/ OUTPUT_TOO_LARGE/EXECUTION_FAILED), so callers route on error class instead of substring-matching messages — the single place raw errors are interpreted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): POST /api/v2/workflows/[id]/execute Thin route over executeWorkflowService: X-API-Key or anonymous public-API auth (sync/stream only for anonymous), strict body with body-flag async (no mode headers on v2), SSE passthrough for stream, and the execution resource response — executionId always present, in-band run failures are status:'failed' with the structured {message, code, blockId, blockName, blockType} error, sync timeout is status:'failed' + TIMEOUT instead of v1's 408, and a Response block's payload stays inside output (authors never control response status/headers on this origin). Async debits the async bucket and the 202 statusUrl points at the v2 executions resource. Adds CLIENT_CLOSED_REQUEST/SERVICE_UNAVAILABLE to the v2 error codes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): v2 executions status + cancel with queued backfill GET /api/v2/workflows/[id]/executions/[executionId] is the single status URL for sync and async runs: before the async worker writes the durable log row, status is backfilled from the job queue (deterministic job id) as 'queued'/'running' — closing v1's 202-to-pickup 404 window — and failed runs carry the structured error object. POST .../cancel renders the shared cancellation lib in the v2 envelope with the tightened 5-value reason enum. Both authenticate via the shared resolveV2WorkflowAccess (X-API-Key, authz masked as 404, allowPersonalApiKeys honored). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(execution): workflow tool + MCP bridge run in-process workflow_executor (workflow-as-agent-tool) short-circuits in executeTool through WorkflowBlockHandler — the same invocation boundary canvas child workflows use — mirroring the deployed_block_executor precedent. The MCP serve bridge calls executeWorkflowService directly instead of fetching its own execute endpoint; deployment-version pinning, MCP response-size rejection, and the actor override become typed options instead of header sniffing. Both callers drop the double admission slot and duplicate top-level log row the HTTP hop cost, and failed child runs now surface the structured error + child executionId so parents and MCP clients can route on error class and hand providers a reproducible handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(infra): CORS + CSP coverage for the v2 execute path /api/v2/workflows/:id/execute gets the same wildcard-origin, credential-free CORS policy as v1 (the default credentialed policy would block browser API-key calls and open a cookie CSRF surface) with X-Sim-Stream-Protocol allowed and no X-Execution-Mode (async is body-selected on v2), plus the COEP/COOP/CSP header block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(ui): deploy modal + copilot advertise the v2 execute surface All 20 API-tab snippets move to POST /api/v2/workflows/{id}/execute with the nested {"input": ...} body, async as the "async": true body flag (X-Execution-Mode gone), status polling against the v2 executions resource, the third tab renamed Usage and pointed at /api/v2/billing/usage, and {data} envelope unwraps in the printed responses. Fixes the latent baseUrl derivation (endpoint.split('/api/workflows/')) that would have silently built garbage URLs under a v2 endpoint, and deletes dead code (exampleCommand across 3 sites, getAsyncExampleTitle). Copilot deploy/manage/serializer endpoint builders and the api_trigger bestPractices example follow (the latter also drops its hardcoded staging host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * docs(api): document the v2 execution surface Adds execute, execution status, and cancel to openapi-v2-workflows.json with the structured ExecutionError schema (append-only code enum + block attribution) and the ExecutionResource contract, documenting the rules that differ from v1: modes are body-selected, a failed run is HTTP 200 with status 'failed', an executionId always means data (never the error envelope), queued status is visible immediately, and Response-block payloads stay inside output. Registers the three pages in the generated workflows meta.json and bumps the route-count baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * feat(api): gate the whole /api/v2 surface behind one flag; UI stays on v1 Every v2 route now runs exactly one check immediately after auth — v2ApiGateError — and answers 404 when the `v2-api` flag is off, so the surface is invisible until it is deliberately rolled out. The gate is keyed on userId only: a workspace/org-keyed check would have to read membership for a caller-supplied id before authorization runs, and its 404-vs-403 split would leak cohort membership (the trap the per-domain table gate worked around by running late). The two executions routes inherit it from the shared access resolver; the tables-specific gate is removed so no route checks twice. `tables-v2-api` stays, now gating only the internal predicate-grammar route /api/table/[tableId]/query — note v2 tables routes move to the unified flag, so enabling them is a `v2-api` decision now. Reverts the deploy modal, copilot handlers, and api_trigger example to the v1 execute endpoint: v1 works unchanged, and the UI must not advertise a surface most users would get a 404 from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(executor): restore child-cost aggregation dropped by the staging merge Staging's custom-block rewrite deleted `aggregateChildCost` from workflow-handler.ts, and git merged that file cleanly — but this branch's workflow-tool-runner.ts, added for the v2 execute migration, still imports it. A silent semantic conflict: no marker, broken build. Taking staging's rewrite is correct, so the helper is defined locally in its one remaining consumer rather than resurrected in the file staging just rewrote. Same four lines over the still-exported `calculateCostSummary`, so a failed child workflow keeps billing the hosted-key spend it consumed instead of reporting $0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation (#6134) * refactor(orchestration): move the shared error contract out of lib/workflows OrchestrationErrorCode and statusForOrchestrationError are the contract every lib/[resource]/orchestration module returns against, but they lived inside the workflows module, so resource-neutral code (lib/folders) already had to import from a workflow path. Moved to lib/core/orchestration/types. Adds a 'locked' class mapping to 423. Both tables and workflows have a lock that forbids a mutation, and each caller was translating that to a status itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): make lib/table/orchestration the single implementation Column update was implemented four times — the UI route, v1, v2, and the copilot table tool — each calling the same column services but owning its own guards, error mapping, and audit. The copies had drifted, and the drift was the bug: v2 was missing both guards, only the copilot copy minted stable option ids, and only v1/v2 audited. performUpdateTableColumn, performDeleteTable, and performDeleteTableRow now own that logic; all ten call sites reduce to auth, parse, call, render. The guards are asserted once in lib/table/orchestration rather than four times against four routes. Behavior this consolidates, previously true on only some paths: - The typeChanging guard. updateColumnType early-returns on an unchanged type and drops any options sent with it, so restating the current type alongside new options silently discarded them. v2 had no guard at all and, since its contract shares v1's body schema, accepted options and ignored them. - The select-unique guard. Each write is its own locked transaction, so a rename or type change paired with a constraint write that is going to fail commits first and then throws, half-applying the schema change. - Stable select-option ids. Cells reference the option id, so an edit that re-sends an option by name has to reuse it or every cell holding it is orphaned. Only the copilot path did this; normalizeSelectOptionsInput moves to lib/table/select-options and now covers every caller. It preserves a supplied id, so it is a no-op for the fully-formed options the HTTP contracts accept. - required forwarded into the type and options writes, so a conversion validates against the constraint the same request is setting. - An audit on every successful update. The UI route and the copilot tool emitted none. - Single-row delete through the row service. v2 did a raw db.delete, skipping assertRowDelete and deleteOrderedRow, so a delete-locked table returned 200 and the row-count bookkeeping never ran. - The delete actor handed to deleteTable, which audits only when a row was actually archived. v1 and v2 omitted it and audited themselves outside that check, emitting TABLE_DELETED for a no-op delete of an archived table. Failure classes come back as OrchestrationErrorCode; v2 renders them through a new v2ErrorForOrchestration, mirroring statusForOrchestrationError on the v1 and UI surfaces, so a given failure maps to the same status everywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(tables): bind the column-update tests to the orchestration function The base's route tests assert which column service each payload reaches — the behavior that now lives in performUpdateTableColumn. They mocked the `@/lib/table` barrel; the orchestration module imports the service directly, so they mock that too and keep asserting the same thing through the extracted implementation. The orchestration tests move onto the base's semantics: writes address the stable column id, a rename rides inside the write it accompanies rather than running first, and the currency guards replace the non-select options guard the service now owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(copilot): drop the column-type import the delegation made dead Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(tables): move the audit log out of the table service `lib/table/service.ts` wrote its own audit rows, so whether an operation was audited depended on which function a caller reached for rather than on a user having performed it. That is what let v1 and v2 audit a no-op delete, and what made `deleteTable`'s optional `actingUserId` double as an audit opt-out flag. Worse, most sites fell back to `actingUserId ?? createdBy`, so an unattributed call was logged against the table's *creator*. The copilot `mv` path passed no actor at all: renaming someone else's table recorded them as the renamer. Audit now lives in the orchestration functions — performDeleteTable, performRenameTable, performMoveTableToFolder, performUpdateTableLocks — and the services just write. Internal callers (folder cascade, import rollback) keep calling the service and are silent by construction rather than by remembering to omit an argument. Two services now return what the audit needs: `deleteTable` reports whether it actually archived a row, so a repeat delete logs nothing; `updateTableLocks` returns the before/after locks, since only the locked write can observe the transition its description names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(tables): restore audit provenance and conflict status in orchestration Moving the audits into the orchestration functions dropped three things the routes had been carrying, and added one the orchestration now owns twice. - The v1 and v2 column-update routes passed `request` to `recordAudit`, so their audit rows recorded the caller's IP and user-agent. The orchestration function had no way to receive it. Every table orchestration function now takes an optional `OrchestrationRequestContext` and every HTTP route forwards it; the copilot and VFS callers, which have no request, omit it. - `classifyTableMutation` matched `TableConflictError` on "already exists" appearing in the message and reported it as `validation`, turning the UI route's 409 on a duplicate table rename into a 400. It now matches the type, the way `performRestoreTable` already did. - `captureServerEvent` ran on every delete while the audit was gated on a row actually being archived, so a repeat delete of an archived table still reported `table_deleted`. Both now hang off the same evidence. - The copilot delete path kept its own `captureServerEvent` from when the service did not emit one, double-counting every copilot table delete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * fix(tables): say which type a no-op column update restated A copilot `update_column` payload whose only content was the column's current type used to return success with the live schema, while the v1, v2, and UI routes rejected the same payload with "No updates specified". Delegating to `performUpdateTableColumn` unified them onto the routes' rejection — correct, but the message tells the caller its request was empty when it named a type. The orchestration function now reports the same thing `updateColumnType` reports when it loses this race concurrently: the column is already that type, re-issue without the type change. An empty payload still reads "No updates specified". Drops the copilot's `outcome.table ?? tableForUpdate` fallback with it — the comment described the no-op that can no longer reach that line, and a success always carries a table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a * refactor(tables): classify failures by type instead of by message text The table module decided HTTP statuses by searching error messages for phrases. `VALIDATION_MESSAGE_FRAGMENTS` and `ROW_WRITE_ERROR_PATTERNS` held 32 substrings between them, and fifteen more lists were inlined in routes — 83 matchers over 17 files, each its own copy of the guesswork and already drifted apart. It made message wording load-bearing: `TableRowLimitError`'s own doc comment noted that its text had to contain "row limit" for a route to answer 400, and adding "already exists" to a rename message silently demoted a 409 to a 400 (the bug fixed one commit ago, by adding another special case). Services now throw `OrchestrationError`, which carries the transport-neutral `OrchestrationErrorCode` the layers above already speak. Classification is one `instanceof` in `orchestrationErrorResponse` (UI + v1) and `v2CaughtOrchestrationError` (v2). Every pattern list is gone. Wording is free to change; an unclassified error still becomes a generic 500, which is what an unexpected fault should be. `asOrchestrationError` walks the `cause` chain rather than testing the caught value directly: drizzle wraps a throw raised inside a transaction callback in a `DrizzleQueryError` whose own message is the failed SQL, so a bare `instanceof` would drop every failure raised inside `withLockedTable`. That is the same reason `rootErrorMessage` had to dig for a root cause before. Three throws stay bare `Error` deliberately — `Table ID mismatch`, `Workspace ID mismatch`, and `Failed to build upsert conflict predicate` are internal invariants no consumer classified, and they keep falling through to a 500. `Insufficient capacity` was in the pattern list with no producer anywhere in the codebase. Status changes, all deliberate: - `'forbidden'` joins the code union so the table-row-limit ceiling keeps its 403; without it this refactor would have flattened it to 400. - import-async's table-limit rejection: 400 -> 403, matching the two other create routes it had drifted from. - Renaming a table to an invalid name: 500 -> 400. `validateTableName` messages don't contain "Invalid", so no matcher ever caught them. - Restoring a table that isn't archived, or into an archived workspace: 500 -> 400. - A duplicate *column* name stays `validation`/400 rather than becoming a 409 like a duplicate table name. Both v1 and the orchestration have always answered 400 for it; changing a published status is not this refactor's job. The twelve tests that changed were asserting the substring mechanism itself, constructing plain `Error`s with magic strings. They now assert the real contract, plus new cases pinning that identical wording carrying no classification stays internal and keeps its message off the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGzbVDZpe2dEALbu2BUU8a --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials (#6150) * feat(api): add v2 endpoints for MCP servers, skills, custom tools, folders, and credentials * fix(api): correct credential role, skill permission bar, MCP url identity, and custom-tool conflict mapping * fix(api): align credential mutation gating, provider-outage status, and unique-violation conflicts * fix(api): close unique-violation, revival, orphan-write, and env-rename gaps * fix(api): treat every provider-outage code as unavailable on create and update * fix(credentials): use the shared outage predicate on the session update path * fix(contracts): anchor the predicate double-cast annotation to the cast `check:api-validation:strict` counted 9 unannotated double-casts against a baseline of 8, failing CI. The predicate leaf schema was annotated, but the annotation sat above the declaration while the checker anchors on the line carrying the cast — five lines below, at the close of the object literal. The scanner walks back at most three lines and stops at the first non-comment one, so it hit `value: z.unknown().optional(),` and never saw the reason. Splitting the object schema from the cast puts them adjacent, so the existing reason binds. No behavior change — the cast, the schema, and the reasoning are unchanged. Also lowers the rawJsonReads ratchet 6 -> 5 to match the current count, which had drifted down; leaving it high lets a removed raw read silently come back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(skills): point the orchestration error contract at its moved module #6150 branched before #6134, so skill-lifecycle.ts imports @/lib/workflows/orchestration/types — the module #6134 moved to @/lib/core/orchestration/types. Git merged a file deletion on one side with a new file referencing it on the other: no textual conflict, broken build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(knowledge): make lib/knowledge/orchestration the single implementation (#6154) * refactor(knowledge): make lib/knowledge/orchestration the single implementation Knowledge base create was implemented four times — the internal route, v1, v2, and the copilot tool — and the orchestration around the shared write had drifted. Extract it the same way lib/table/orchestration was: services write, orchestration decides which writes run, guards them, audits them, and returns a transport-neutral failure. Behavior converged, not preserved: - One chunking default (DEFAULT_CHUNKING_CONFIG). The agent defaulted minSize to 1 against the API's 100, so identical input produced differently-chunked knowledge bases depending on who created it. The agent path now chunks at 100. - Every successful mutation is audited inside the orchestration function. The copilot tool called recordAudit zero times, so agent-created knowledge bases, document uploads, updates and deletes left no audit trail at all. - Failures classify by class, not by message text. The knowledge service errors are OrchestrationError subclasses and storage-quota rejections throw a shared StorageLimitExceededError, replacing four separate message greps for "already exists" / "does not have permission" / "storage limit". delete_connector reported the opposite of what happened. It reached the route through an internal HTTP self-call that sent no query string, so the route's keep-documents default always applied while the agent told the user the documents had been removed. The self-call is gone — all four connector operations run in-process — and the orchestration returns the real counts. Also: - OrchestrationErrorCode gains 'payload_too_large' (413 / PAYLOAD_TOO_LARGE). Without it, dropping the storage-limit message match would have regressed the documented 413 on knowledge base create and document upload to a 500. - messageForOrchestrationError renders a route's own wording for an unclassified fault, so a driver's message no longer reaches the client on a 500. - v1 and v2 knowledge base update now forward actorUserId, which the service requires for a workspace move; both omitted it. - The connector DELETE route reads deleteDocuments through parseRequest. Its contract declared z.boolean(), which would have rejected the string a query param actually is. - Drop the 409 from POST /api/v2/knowledge/{id}/documents in the OpenAPI spec. Nothing on the upload path throws a conflict; it was only ever reachable by the message match this change removes. Behavior change worth noting: a v1/v2 PUT carrying only the workspaceId scope field and no actual updates now returns 400 rather than 200 with the unchanged knowledge base. Deliberately deferred: document update remains internal-only. Extracting performUpdateKnowledgeDocument makes exposing it on v1/v2 a contract and a route away, but that is a new public surface rather than part of this consolidation. * fix(knowledge): make connector create atomic and stop flattening failures Review round 1 on #6154. - Resolve the billing payer before the connector is committed, not after. A malformed attribution header rejected post-commit left a live connector behind a 500, and a retry created a duplicate plus duplicate sync work. Manual sync resolves before writing its audit for the same reason. - Let the source-config validator carry its own failure class. Collapsing every rejection to `validation` flattened the connector PATCH route's 401 (stale stored credential) and 409 (missing workspace context) into a 400. - Add `unauthorized` to OrchestrationErrorCode. It is the class that 401 was already expressing on this route, and the v2 vocabulary already had UNAUTHORIZED; only the shared union was missing it. - Report a knowledge base that exists but failed to archive as failed, with the reason, rather than as not found. The copilot delete loop folded every non-not-found failure into `notFound`, telling the user it was never there. - Route copilot failures through the same message helper the HTTP surfaces use, so an unclassified fault's raw text (a driver's failed SQL) no longer reaches the agent verbatim while the UI and public APIs get the generic wording. * feat(api): expand the public v2 files surface (#6160) * feat(api): expand the public v2 files surface Adds folder support, rename/restore, move, bulk archive, share, and content replace to /api/v2/files, so managing files by API no longer stops at upload + download + archive-one. Routes are thin: auth -> parse -> perform* -> serialize. Share and content replace get their orchestration extracted first so the session routes and the public ones cannot diverge on the effective-authType resolution, the EE public-sharing gate, or the storage-quota classification. Presigned upload stays session-only: presign does an advisory quota check and the real debit happens in the separate register step, so a caller that never registers leaves unaccounted bytes with no reaper. The buffered multipart path debits inside uploadWorkspaceFile's own transaction. * fix(files): classify folder and content failures instead of 500ing them Bugbot round 1. The v2 routes map errorCode straight to a status, so every manager failure that arrived unclassified became a 500 for what is really a caller-fixable 400 or 404. - Folder manager throws OrchestrationError: missing target/folder -> not_found, reparent cycle / self-parent / restore-into-archived-workspace -> validation. - File manager does the same for the in-transaction 'File not found' paths that the earlier pass missed. - updateWorkspaceFileContent's outer catch re-wrapped everything in a bare Error, which stripped the class off StorageLimitExceededError and the new not_found alike. It now rethrows a classified failure untouched and attaches cause to the generic wrap, so asOrchestrationError can still walk the chain. - Every remaining perform* gained the asOrchestrationError branch. - renameWorkspaceFile returned the pre-update read, so the v2 PATCH reported a stale updatedAt; it now returns the timestamp it actually wrote. Docs: upload auto-suffixes a duplicate name rather than rejecting it, matching the in-app uploader. The description claimed 409 and was simply wrong. * fix(files): surface a failed upload read-back as the real error getWorkspaceFile swallows a query failure and returns null unless throwOnError is set, so a transient blip on the post-upload read reported as 'file could not be read back'. Distinguish the two: a real null after a just-committed write is an invariant break, a query failure is itself. * revert(api): drop the dedicated v2 file-folder routes File folders already live in the shared folder table as resourceType 'file' (#6045 cut them over, #6051 dropped workspace_file_folders), and the remaining file-specific folder machinery is being folded into the generic folder engine. Publishing /api/v2/files/folders/** would pin that transitional split into a public contract we'd then have to keep or break. Files stay folder-aware — folderId/folderPath on the projection, folderId on upload, and the move route — because a folder id is a folder.id and survives the unification untouched. Folder management belongs on /api/v2/folders once that surface serves resourceType 'file'; until then there is no v2 way to enumerate file folders, which is the deliberate gap. The orchestration classification fixes stay: the internal routes and the copilot file-folder tools still call those perform* functions. * fix(files): classify upload failures instead of matching their wording Bugbot round 2. uploadWorkspaceFile had the same outer-catch rewrap that updateWorkspaceFileContent did, so a blown storage quota reached the route as a bare Error and the v2 handler recovered the status by substring-matching the message. Any rewording silently demoted a 413 to a 500. - uploadWorkspaceFile rethrows a classified failure untouched and attaches cause to the generic wrap. - FileConflictError is now an OrchestrationError('conflict'), so a duplicate name classifies like every other conflict. Its 'FILE_EXISTS' discriminator had no readers and is gone; the instanceof checks elsewhere still hold. - The v2 upload handler uses v2CaughtOrchestrationError, dropping all three string matches. Also documents that bulk-archive is best-effort: unknown or already-archived ids are skipped rather than failing the call, and deletedItems is what actually happened. That asymmetry with the single-id DELETE was undocumented. * feat(api): add search, filtering, and sorting to the v2 list endpoints (#6189) * feat(api): add search, filtering, and sorting to the v2 list endpoints One convention across every v2 list, documented on lib/api/contracts/v2/shared.ts: `search` (case-insensitive substring on the resource's natural name field), `sortBy` + `sortOrder` (per-resource enum, never a free string), and enumerated resource-specific filters. Reuses the sortBy/sortOrder pair v2 logs and v2 knowledge-documents already ship rather than inventing a third dialect alongside the Logs filters and the Tables predicate grammar. Every filter and sort is pushed into SQL. GET /api/v2/files previously read the whole scope and sorted/sliced it in JS; it now goes through a new queryWorkspaceFiles that filters, orders, and bounds the page in one query. Cursors are stamped with the sort they were minted under, so replaying one under a different sort is a 400 instead of silently duplicated or skipped rows. * fix(api): validate v2 cursor key values and compare timestamps at ms precision Two review findings, fixed at the root by making a keyset key own its cursor codec instead of hand-writing a decoder per sort. Cursor key values are caller-controlled, and matching the sort stamp and key count was not enough: an unparseable timestamp or a non-numeric size reached the query as an Invalid Date or NaN and surfaced as a 500. Each key now type- checks its own value and rejects a cursor it cannot hold, which both routes render as the documented 400. Timestamp keys now order and compare on date_trunc('milliseconds', col). Postgres keeps microseconds and defaultNow() populates them, but a cursor value round-trips through a millisecond-only JS Date — comparing the raw column against the truncated value re-admitted the page's own last row, duplicating it and stalling pagination outright at a page size of one. Reachable today via workspace_files.updated_at, which insertFileMetadata leaves to defaultNow(). * feat(api): complete the v2 workflows resource with versions and CRUD (#6184) * feat(api): complete the v2 workflows resource with versions and CRUD Adds version listing/detail plus create, update, and delete to the v2 workflows surface, which previously covered only execution and deployment. - GET /api/v2/workflows/[id]/versions — cursor-paginated, newest first - GET /api/v2/workflows/[id]/versions/[version] — version + pinned state - POST /api/v2/workflows, PATCH and DELETE /api/v2/workflows/[id] All six delegate to the existing orchestration and persistence helpers; no new domain logic. * fix(api): check folder containment before lock state; reject malformed version cursors assertFolderMutable walks a folder's ancestor chain without filtering on workspace, so inspecting it before containment let a caller tell a locked folder in someone else's workspace (423) from a nonexistent one (400). Create and update now assert containment first, matching the ordering import-workflow.ts already uses. A version cursor that decodes to JSON without a numeric version filtered every row out and returned an empty page with nextCursor null, which reads as a clean end-of-list. Malformed cursors are now a 400. * refactor(api): page workflow versions in the persistence helper listWorkflowVersions read every version row and the route filtered and sliced the result in memory, so the response was bounded but the query was not. It now takes optional limit/afterVersion, turning the cursor into a real keyset query; the route asks for limit + 1 and only trims the has-more probe. Both params are optional, so the internal, v1 admin, and copilot callers are unchanged. Also restores the untouched GET handler in [id]/route.ts to its original formatting — collapsing its signature had re-indented the whole body and buried the actual additions in whitespace churn. * feat(api): expand v2 tables with stateless multipart transfers (#6188) * feat(api): expand the public v2 tables surface Adds 16 operations so a v2 caller can do what the internal surface can: rename/move/lock a table, restore it, manage saved views, run enrichment columns, look up rows, and import/export with observable job control. Extracts lib/table/orchestration/import.ts (performTableCsvImport, performCreateTableFromCsv) and lib/table/export-stream.ts from the first-party routes, then repoints those routes at them, so v1 and v2 cannot drift on what an import or export actually does. events/stream, metadata and dispatches stay internal — they are editor state, not public API. * fix(api): make v2 table PATCH all-or-nothing and name the lock in every 423 Greptile P1: PATCH applied locks, rename and move as three sequential transactions, so a folder rejected mid-request left the earlier writes persisted while the response reported failure — and the schema-changed signal was skipped, leaving open clients on stale state. Every rejectable condition now runs before the first write, and the signal fires whenever anything did land. Cursor: v2TableLockError dropped the lock kind, so async import, column run, enrichment and table mutations returned a bare LOCKED. A table has four independent locks, so the caller could not tell which to clear. * fix(api): report the lock kind on classified 423s too, not just thrown ones The previous commit named the lock only where the rejection was thrown and caught at the route boundary. Where it instead arrives as a classified `errorCode: 'locked'` outcome — delete table, delete row, update column, and the table mutations — the kind was dropped, so those 423s stayed unactionable while their neighbours improved. The orchestration results now carry `lock`, and a shared `v2TableOrchestrationError` renders both arrival paths into the same `{ code, message, details: { lock } }` body. `details` is omitted rather than sent null when the kind is unknown, so a caller branching on it sees absence instead of a phantom value. * fix(api): make async table imports observable, not just startable `POST /import-async` pointed callers at `GET /api/v2/tables/jobs` to track progress, but that endpoint filters to `type = 'export'` — imports are derived onto the table itself, one write job at a time, and exports get a separate list precisely because they are excluded from that derivation. The public Table shape omitted those derived fields, so an async import could be started and cancelled but never observed to completion, failure, or progress. That is the gap the import/export/job-control set was meant to close. Table now carries `job` — id, type, status, rowsProcessed, error, or null when idle — and the import-async docs point at the table rather than the export list. * feat(api): make v2 table PATCH state which operations landed on failure Greptile held the PR at 4/5 on the residual non-atomicity and named two acceptable resolutions: make PATCH atomic, or have the contract adopt and expose partial-success explicitly. Atomicity would mean threading one transaction through renameTable, moveTableToFolder and updateTableLocks — three shared service functions with four non-test callers including the first-party route and two copilot tools — and deferring their per-operation audits to commit time. That is a refactor of shared write paths well outside this PR. So the contract states it instead. Every rejectable condition is already pre-validated, so a failure here is a genuine fault; when one follows a successful operation the error now carries `details.applied` listing what is live. Absent when nothing applied, so its presence always means "these changes took effect despite the error". Documented on the operation. `v2ErrorForOrchestration` gained the optional `details` this needs. * fix(api): make table lock flags read-only on the public v2 surface The new PATCH /api/v2/tables/[tableId] accepted a `locks` object, gated on workspace admin plus the table-locks feature. That still lets an API key clear the guard placed there to stop it: `write` is the floor for the endpoint, and admin keys are ordinary API keys, so a lock is no longer a boundary the key cannot cross. Locks stay readable on the table resource and enforcement is unchanged (a locked verb still returns 423). Changing one is now a first-party admin action only. The v2 body is declared here rather than reusing the first-party updateTableBodySchema, which keeps its `locks` field so the UI can still toggle them. It is .strict(), so a request carrying `locks` is rejected with a 400 naming the field instead of silently succeeding without applying it. * fix(api): keep reporting applied operations when the PATCH re-read fails The composite table PATCH promises that `error.details.applied` names the operations that are live despite an error, but `applied` was scoped inside the try. A rename or move that committed and was then followed by a throw in the final re-read — or a re-read finding the table archived — returned a bare 500/404 with no details, telling the caller nothing had landed. It would then retry into a duplicate-name conflict or repeat the move. `applied` is now function-scoped so every post-write exit carries it: the 404 on a missing re-read, a thrown lock error, a classified orchestration error, and the generic 500. `v2TableLockError` gains the same `extraDetails` parameter `v2TableOrchestrationError` already had. * feat(api): add workflow group writes to the v2 tables surface v2 exposed GET /groups but none of the writes, so the public API could run an enrichment or workflow column and read its binding, but never create one. A caller could add a plain data column and trigger the machine; wiring the two together still required the UI. Adds POST/PATCH/DELETE on /api/v2/tables/[tableId]/groups. The group is the unit that fills columns — one group feeds several — so creating one creates its output columns in the same call, matching the first-party shape rather than inverting it onto the column endpoint. Four departures from the first-party body, all public-surface concerns: - group.id is optional and server-generated. The UI mints an id to render optimistically; a public caller has no such need and a client-chosen id is a collision waiting to happen. - outputColumns[].workflowGroupId is dropped from the body and stamped from the resolved group, so it cannot disagree with it. - autoRun defaults to false. First-party defaults true so a UI add fills cells immediately; here it would make one POST fan out a metered run across every existing row. - A group naming neither a workflowId (type manual) nor an enrichmentId (type enrichment) is a 400 rather than a half-specified group the route has to guess about. Also rejects an outputColumns entry no group output feeds — the two arrays are joined by column name, and the first-party client builds both from one picker so it cannot desync, but a public caller can. Workspace containment on workflowId is asserted before it is persisted, on create and on any update that re-points the group; without it a table becomes a way to invoke workflows the key cannot otherwise reach. * improvement(api): make v2 table import and export async-only Drops the three synchronous entry points: POST /tables/[tableId]/import, POST /tables/import-csv, and GET /tables/[tableId]/export. Sync import tied a write to the lifetime of an HTTP request. The body *was* the data, so it carried a 10 MB cap that Next silently truncates past — a partial import reporting success. It also had no job, so a timeout mid-write left rows in place with nothing to poll and nothing to cancel. The async path reads the file from storage instead: upload via POST /api/v2/files for a key, start with POST /import-async, watch GET /tables/[tableId] -> job, stop with POST /job/cancel. Sync export carried no such hazard, but one shape per operation beats two: with both removed the surface has exactly one way to move a table in or out, and the CLI wraps the extra calls. This also removes the last multipart handling in v2 tables. Those were the only routes bypassing parseRequest — form fields were parsed by hand against separate form schemas, outside the contract system every other v2 write goes through. Create-a-table-from-CSV is now two calls: POST /tables, then /import-async with createColumns. csvImportModeSchema is append|replace, so there is no single-call create. Route baseline 1064 -> 1061. * docs(api): correct the import-async note about upload size limits The docstring claimed there is no synchronous upload endpoint and so no request-body size cliff. Both are wrong: POST /api/v2/files is a synchronous multipart upload with a 100 MB cap, and it is the only v2 upload path (presigned is deliberately absent). What async-only actually bought: the cap went 10 MB -> 100 MB, it fails on an explicit size check and a bounded body read rather than a proxy cap that silently truncates, authorization completes before any body is buffered, and the table write is a job that can be watched and cancelled. * feat(api): unify file and table transfers * improvement(api): make multipart transfers stateless * fix(api): make table import completion retries idempotent * feat(v2-tables): paginate the table list `GET /api/v2/tables` returned every table in the workspace in one response — it used the cursor envelope but hardcoded `nextCursor: null`, and had no `limit`. That was defensible when tables were only created through the UI; `POST /api/v2/tables` is public now, so a script can create them in bulk and the list has no way to ask for less. Adds `queryTables` alongside `listTables` rather than changing it, so the internal callers that genuinely want the whole scope are untouched — the same split `queryWorkspaceFiles` / `listWorkspaceFiles` already uses. Filter, order and slice all run in the query, so a `search` never costs a full-workspace read. A cursor whose values don't bind raises a validation error instead of being coerced to "no filter", which would have silently served page 1 under a resumed cursor. The keyset closes on `id` so a page boundary inside a run of equal names or timestamps stays stable. The shared `LimitQuery` doc component said "Maximum rows to return"; it now serves the table list too, so the wording is resource-neutral. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(api): add multipart knowledge document uploads * fix(api): keep usage admission at knowledge upload session creation * feat(knowledge): wire knowledge base uploads to multipart sessions * fix(knowledge): refuse to abort an upload once a document is bound * fix(uploads): prevent multipart cleanup races * Unify file creation and signed upload sessions (#6264) * feat(uploads): unify signed upload sessions * fix(uploads): preserve attachment storage semantics * feat(files): add authored file creation * fix(uploads): omit hoisted S3 metadata headers * feat(api): add file metadata endpoint * improvement(api): scope folders to resource paths (#6284) * improvement(api): scope folders to resource paths * fix(files): serialize folder resolution with uploads * fix(files): release folder lock before upload setup * fix(api): normalize folder paths and unblock resource mutations * fix(api): make resource cleanup and metadata consistent * improvement(uploads): persist multipart sessions in postgres * fix(db): store table row trigger timestamps in UTC * improvement(api): default folder deletion to non-recursive * fix(billing): unify chat usage source * improvement(logs): expose trace spans on log detail * fix(logs): parse list trace spans * improvement(api): replace workflow jobs with execution resources (#6294) * improvement(api): replace workflow jobs with execution resources * fix(api): preserve legacy jobs while preferring v2 executions * fix(api): make execution polling resume-aware * fix(ui): hide async examples for public workflows * fix(api): bridge resume queue visibility lag * feat(api): add v2 workflow resume endpoint * fix(api): project pending resume attempts * fix(api): prefer terminal logs over stale resumes * improvement(api): unify v2 resource query layers (#6319) * improvement(api): unify v2 resource query layers * fix(api): address v2 review findings * fix(api): preserve cancelled queue status * fix(api): guard cancelled job transitions * fix(api): close v2 resume and log gaps * feat(api): rename v2 executions to runs * feat(api): split credentials and secrets * feat(api): add workspace metadata and email attribution * improvement(api): consolidate public v2 route handling * improvement(files): centralize operations across APIs and Copilot (#6392) * improvement(files): unify rename authorization * chore(skills): add file operation migration guide * improvement(files): consolidate file operation authorization * improvement(files): extract shared operation foundation * improvement(api): simplify internal route declarations * improvement(files): centralize application authorization * refactor(api): share workspace file name validation * refactor(files): centralize copilot application calls * docs(skills): generalize application operation migration * improvement(api): centralize remaining v2 resource operations (#6412) * improvement(api): centralize v2 resource operations * fix(api): preserve custom tool conflict errors * improvement(api): migrate policy-sensitive v2 reads (#6410) * improvement(workflows): centralize v2 application operations (#6411) * refactor(api): migrate v2 knowledge operations (#6413) * refactor(api): migrate v2 knowledge operations * fix(knowledge): fail upload completion on dispatch errors * fix(knowledge): preserve upload retry and VFS errors * improvement(tables): centralize v2 application operations (#6414) * improvement(tables): centralize v2 application operations * fix(tables): preserve run validation and signals * feat(auth): add scoped internal executor delegation (#6459) * feat(auth): add scoped internal executor delegation * fix(auth): derive delegation lifetime from one timestamp * Include share status in file metadata * feat(auth): centralize delegated identity policy (#6462) * improvement(copilot): consolidate application adapters (#6450) * improvement(api): harden application route boundaries (#6451) * improvement(api): harden application route boundaries * fix(folders): reject creates at workspace cap * fix(knowledge): enforce trusted workspace scope (#6452) * fix(knowledge): enforce trusted workspace scope * refactor(knowledge): declare v2 body lifecycle * finish knowledge application migration * refactor(knowledge): compose copilot batch commands * fix(knowledge): parse connector query flags * fix(knowledge): finalize partial batch effects * fix(knowledge): align merged application boundaries * fix(knowledge): close application boundary review gaps * style(knowledge): satisfy branch biome checks * fix(knowledge): page connector documents in editor * refactor: enforce Copilot table application boundary (#6453) * refactor: enforce copilot table application boundary * fix(tables): finish application boundary migration * fix(tables): restore scoped copilot imports * fix(tables): compose copilot commands atomically * fix(tables): preserve workflow group scheduling * fix(tables): complete fixed copilot composition * fix(tables): reject enrichment output mutation * fix(tables): complete authorized application boundary * fix(workflows): migrate Copilot application boundary (#6455) * fix(workflows): migrate Copilot application boundary * fix(workflows): finish delegated application migration * fix(workflows): encode VFS folder aliases * fix(workflows): close application composition gaps * fix(workflows): preserve VFS validation errors * fix(workflows): complete application boundary migration * test(workflows): format canonical binding coverage * fix(workflows): scope executor metadata reads * fix(workflows): bind executor metadata targets * improvement(skills): align application operation guidance (#6532) * feat(api): expose v2 resource owners * fix(api): distinguish visible resource authorization failures (#6537) * feat(api): generate v2 OpenAPI from contracts (#6509) * feat(api): generate v2 OpenAPI from contracts * fix(api): preserve string boolean wire defaults * fix(api): document file download headers * fix(docs): use TypeScript CLI with Next.js * fix(docs): avoid client-rendered theme script * fix(api): document departed audit default * feat(api): replace legacy core docs with v2 * feat(api): generate v2 OpenAPI from contracts * feat(api): refine generated v2 OpenAPI docs * fix(docs): align localized v2 execution examples * fix(ci): restore Helm diff and sync audit mock * fix CI regressions after staging merge --------- Co-authored-by: Waleed <walif6@gmail.com> Co-authored-by: Theodore Li <theodoreqili@gmail.com> Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com> Co-authored-by: Theodore Li <theo@sim.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-11 05:45:25 -04:00
Waleed	a64ce49af9	feat(incidentio): add on-call, alert, catalog, and team tools (#6529) ... * feat(incidentio): add on-call, alert, catalog, and team tools Adds 19 tools to the incident.io block, taking it from 46 to 65 operations. Every endpoint, param, and response field is taken from the official OpenAPI spec at api.incident.io/v1/openapiV3.json. Who is on call had no reachable answer before: the data lives in ScheduleV2.current_shifts, and both schedules_list and schedules_show returned it but never declared it. The new incidentio_on_call_now tool flattens current and upcoming shifts to one row per person, and the two existing schedule tools now declare the fields they were already returning. Also fixes two pre-existing wiring bugs: the block declared an output named schedule_override while the tool emits override, and the on-call handoff skill described a lookup the integration could not perform. * fix(incidentio): stop the alert filter sentinel reaching the API The has_notes and include_maintenance_window dropdowns default to the string "any", meaning "do not filter". The params transform skipped the key in that case, but the executor merges its output over the raw inputs (`{ ...inputs, ...transformedParams }`), so the sentinel survived and the tool sent has_notes[is]=any, which incident.io rejects. The transform now always assigns the key, mapping "any" to undefined so it overwrites the sentinel instead of leaving it in place. The tool also only serializes these filters when it actually has a boolean. Adds tests covering the sentinel, both real boolean values, and the documented bracket-operator filter syntax.	2026-08-10 22:02:50 -07:00
Theodore Li	5478a690cc	improvement(setup): complete knowledge and update flows (#6521)	2026-08-10 23:38:47 -04:00
Waleed	daac4f38d4	docs(salesforce): correct the setup steps that would strand an admin (#6526) ... * docs(salesforce): correct the setup steps that would strand an admin Verified the guide against Salesforce's current UI and docs. Most of it holds; these do not: - The Client Credentials step told admins to check "Enable Client Credentials Flow" under OAuth Policies. On an External Client App that checkbox is under Edit Settings → OAuth Settings; the Policies page holds only the Run As picker, so anyone following it literally hunts for a control that is not on the screen. The FAQ answer inherited the same conflation. - Pre-authorizing the app must go through the profile or a SECOND permission set. A permission set backed by the Salesforce API Integration license cannot hold an Assigned Connected Apps section at all, so the app can never be assigned from the same permission set that grants object access — which produces exactly the "user hasn't approved this consumer" failure that step exists to prevent. This is the likeliest way a JWT setup fails. - Salesforce requires an RSA key of at least 2048 bits; an ECDSA key is silently rejected, and the certificate must stay under 4 KB. - The JWT toggle does not appear until Enable OAuth is on, and the control is "Upload Files". Also scopes the capability promise for the API-only license: SOQL and CRUD on standard objects are supported, reports and dashboards are genuinely unverified in either direction, and Apex Class Access is a permission this license cannot hold, so Tooling API calls touching ApexClass will fail. * docs(salesforce): align the Developer Edition host in the FAQ with the setup section The setup section was corrected to make the `-dev-ed` suffix conditional, but the FAQ still presented it as mandatory — so an admin whose Developer Edition domain lacks the generated suffix would read two contradictory formats on the same page and validate against a host that does not exist.	2026-08-10 20:33:21 -07:00
Waleed	ba5dfb1f89	feat(salesforce): add JWT bearer flow and sandbox OAuth support (#6508) ... * feat(salesforce): add JWT bearer flow and sandbox OAuth support Salesforce integration users could only authenticate through interactive OAuth, which an API-only integration user cannot complete — there is no UI for them to log in to. Adds the JWT Bearer Flow as a second grant on the existing service-account provider, and registers sandbox as its own authorization server so sandbox orgs can connect at all. The assertion is audienced at the org's My Domain URL rather than login/test.salesforce.com: Salesforce ended legacy hostname redirections in Spring '26 and External Client Apps now reject the generic sandbox host with app_not_found. My Domain is valid for Connected Apps and External Client Apps, production and sandbox alike, and is what the Salesforce CLI recommends — so the stored host alone determines the environment. Sandbox credentials are stored under their own provider id, mapped back to the one Salesforce service via additionalProviderIds on OAuthServiceConfig. That is threaded through every resolution point, including the two SQL filters that would otherwise have hidden sandbox credentials from the block picker entirely. Also fixes three latent bugs surfaced along the way: Zoom interpolated an undefined client secret into its Basic auth header, sandbox refresh tokens would have been posted to the production endpoint, and the sandbox connector would have been silently dropped as unconfigured. * fix(salesforce): canonicalize connected provider ids in the copilot credential tool A credential stored under an alternate authorization server was recorded in `connectedProviderIds` under its own id, while the not-connected list compares against the service's canonical id — so a sandbox-only Salesforce user was reported as both connected and not connected. Record the canonical id instead. Also types the JWT test's assertion decoder instead of returning `any`. * fix(salesforce): close reconnect, instance-URL, and key-handling gaps An independent audit swarm found four real defects in the JWT bearer work: - The credential update hook rebuilt its request body from a hand-written allowlist, so `authMethod`, `privateKey`, and `username` were silently dropped. A JWT private key could never be rotated through the UI, and switching grants failed with a generic error. Forwards the whole contract body instead, so a field added to the contract later cannot be lost again. - `getInstanceUrl` guarded only the `sub` claim against login-host origins, so a sandbox id token whose `profile` was rooted at test.salesforce.com yielded the login host as the org's API base. Both claims are now guarded, and a guarded-away `profile` falls through to `sub` instead of ending the lookup. - `canonicalizeServiceProviderId` replaces the previous fold, which also matched family-wide service-account ids and so dropped one arbitrary sibling product (Gmail, Confluence) from the copilot's not-connected list. - The private key was collected in a plain textarea, leaving browser spell check and autofill free to ship it to third parties. Also restores the explicit https check on the userinfo-derived instance URL, anchors the scope marker, caps the accepted RSA modulus, and stops single-grant providers paying for a stored-blob decrypt on every reconnect. Docs: the JWT path no longer tells readers to enable the Client Credentials Flow, and calls out the my.salesforce-setup.com host as the likely wrong paste. Adds coverage for the paths the audit proved untested: partitionClientCredentialFields, credentialProviderMatchesService's alternate-server clause, reconnect carry-forward, the typographic-apostrophe error branch, and the passphrase hint. * fix(salesforce): handle the Government Cloud JWT audience and unassigned-profile errors Verification against Salesforce's own sfdx-core surfaced two gaps: - `gs1` Government Cloud orgs have ordinary *.my.salesforce.com hosts, but Salesforce requires `https://gs1.salesforce.com` as the JWT audience. The host regex accepted them, so they would have failed with an opaque audience error. The token still posts to the org's own host; only `aud` differs. - `invalid_app_access` — Permitted Users is set to admin-pre-authorized but the run-as user's profile was never assigned to the app — is the likeliest misconfiguration and had no hint at all. Also sends `iat`, matching sfdx-core and every mainstream implementation, and softens two TSDoc claims that were stronger than the evidence: Salesforce does not hard-reject a far-future `exp` (its own CLI ships one), and My Domain is the right audience for commercial orgs rather than universally. * fix(salesforce): match sandbox credentials in Chat and the connect draft Two more surfaces resolved a credential to its service by exact provider id: - `credentialsForTarget` compared only `providerId`/`baseProviderId`, so a sandbox-only user's Salesforce chip in Chat read as disconnected and re-prompted them to connect. The alternate ids are passed in by the caller rather than resolved in the module, which is `'use client'` and would otherwise pull the OAuth provider registry into the chat bundle. - `createConnectDraft` resolved the service name by exact id, so a sandbox connect defaulted to the label "My salesforce-sandbox". * fix(salesforce): carry alternate provider ids through chat connect verification The chip's live target was widened to match a sandbox credential, but the post-connect verification leg re-reads the STORED attempt, which did not carry the ids — so completing a sandbox connect from Chat was detected as a failure and the chip was marked failed. The attempt now persists them; attempts written before this simply match as they did, and they expire within 15 minutes. Also marks the auth-method picker required while it is the field blocking submit on a reconnect, so the greyed button has a visible cause. * fix(salesforce): send reauthorize to the server that issued the credential "Update access" derived its provider from the service id, which always yields the primary authorization server. A sandbox credential missing a scope sent the user to login.salesforce.com — where a sandbox-only user cannot sign in at all, and where a user who can sign in creates an orphan production account while the banner never clears. Both credential selectors now pass the selected credential's own provider id, which the connect modal already honours. Also names the alternate provider ids explicitly in the disconnect sweep. That branch is unreachable today (every caller sends an accountId), but it was catching them only by the `{base}-` prefix accident. * fix(salesforce): make the Government Cloud audience check exact, not a prefix `startsWith('gs1-')` was invented from a paraphrase of sfdx-core and would have misrouted an ordinary org like gs1-widgets.my.salesforce.com to the GovCloud audience — breaking a setup that works today. sfdx-core's host signal is the literal gs1.my.salesforce.com; its other signal is the org's createdOrgInstance, which we never see. Matching exactly means a miss falls back to My Domain, which is the behaviour before the branch existed, while a false positive cannot happen. Also replaces the hand-rolled origin regex in getInstanceUrl with URL parsing, which normalizes userinfo, ports, and case before the login-host comparison, and drops two error hints that had no evidence behind them.	2026-08-10 19:34:17 -07:00
Theodore Li	56910c002e	feat(embeddings): add OpenRouter support (#6396) ... * feat(knowledge): add OpenRouter embedding fallback * fix(knowledge): preserve successful embedding batches * feat(embeddings): add OpenRouter provider * fix(knowledge): bill only platform embedding tokens * test(embeddings): include OpenRouter provider * feat(embeddings): load OpenRouter model catalog * fix(embeddings): preserve legacy provider default * fix(embeddings): batch OpenRouter requests * fix(embeddings): reset stale OpenRouter model	2026-08-10 15:15:18 -04:00
Vikhyath Mondreti	156ee3ebf2	fix(provenance): feature flagged, inexact sidecars (#6491)	2026-08-10 09:26:19 -07:00
Waleed	4b2412b752	fix(auth): stop offering account creation when registration is disabled (#6484) ... DISABLE_REGISTRATION blocks /signup server-side, but the invite flow, the login form, the SSO form, and the CLI handoff all kept routing people there, stranding invited users on a dead end. The flag also never covered OAuth account creation, so social sign-in still minted accounts for unknown identities.	2026-08-09 22:02:42 -07:00
Waleed	303986f45f	feat(snowflake): credential-based auth, object pickers, and 9 new operations (#6474) ... * feat(snowflake): credential-based auth, object pickers, and 9 new operations Replace the per-block host + PAT fields with a Snowflake service-account credential, move the credential picker to the top of the block, back the object fields with metadata-only pickers, and add nine operations. - credential: snowflake-service-account token service account (account host + programmatic access token), verified against the SQL API with the same headers the tools use - selectors: database, schema, table, warehouse, execution role, file format and procedure pickers behind one /api/tools/snowflake/objects route - new operations: unload_data, list_databases, list_schemas, list_tables, alter_warehouse, resume_task, suspend_task, list_query_history, list_copy_history * fix(snowflake): migrate renamed subblock IDs and authenticate before parsing - add SUBBLOCK_ID_MIGRATIONS entries so the renamed object fields map onto their pickers and the removed host/apiKey values are parked - authenticate the caller before contract validation in the selector route, per the API route convention * fix(snowflake): close unload-query breakouts, drop parked secrets, correct docs - assertBalancedQuery now skips // line comments, $$ dollar quoting and rejects ambiguous nested block comments; each hid a paren that let an injected OVERWRITE = TRUE escape the derived table - always emit OVERWRITE so an injected duplicate is rejected by Snowflake rather than silently replacing staged files - _removed_ migration targets now drop the stored value instead of parking it under a dead key, where export scrubbing (which walks the block config) would never clear it - 403 falls back to the shared invalid-credentials message, which names the network policy and SQL API causes Snowflake does not distinguish in the body - correct the network-policy-by-user-type claim: only SERVICE_AGENT is exempt - correct MAX_FILE_SIZE and errorOnly tool descriptions to match the fixed code * fix(snowflake): stop untouched switches emitting clauses; retarget migration - an untouched switch serializes as null, and advanced mode emits every advanced subblock, so alter_warehouse silently sent AUTO_RESUME = FALSE and permanently disabled auto-resume on the warehouse; normalize optional booleans to undefined in tools.config.params - point the subblock migration at the advanced text members: a migrated block has no credential, so a picker cannot hydrate a stored name, and legacy fileFormat values were qualified while the picker lists bare names - add the missing json-object wand type and scope the SQL wand prompt, which promised bindings that unload_data does not accept * fix(migrations): sweep already-parked subblock values; align picker 403 - an earlier version of this migration renamed retired fields into _removed_* keys instead of deleting them, so deployed workflows still hold those values; they match no oldId, so a dedicated sweep clears them for every block type - the picker now treats a Snowflake 403 like a 401: it means a network policy or a disabled SQL API, which the credential validator already reports as a credential problem rather than a bad request * fix(wand): add json-array generation type for array-contract fields The json-object reinforcement tells the model the response must start with { and end with }, which fights any field whose contract is an array. Snowflake's rows, matchColumns and procedureArguments all ask for arrays, so they were being steered toward an object that the JSON parse would then reject. Adds a sibling json-array type that strips fences the same way but reinforces brackets, and points the three array fields at it. bindings and filters are genuine objects and stay on json-object. * fix(snowflake): unload a table, not an inline query The COPY INTO grammar places the source immediately before its copy options, so an inlined query sits one parenthesis from being able to rewrite them. Guarding that means matching Snowflake's tokenizer exactly, and three successive versions of the guard were each defeated: // line comments, $$ dollar quoting, and a bare carriage return, which the scanner did not treat as a line terminator but Snowflake does. Each fix was a guess at a lexer the public docs do not specify. Removes the inline-query source instead of guessing a fourth time. A table name goes through qualifiedIdentifier, which is provably safe. Exporting a query result now means materializing it first — a view, or CREATE TABLE AS SELECT via Execute SQL — which the tool description, the block skill and the docs all say. Also from the final audit: - optionalBoolean accepts the string forms a direct tool call delivers, matching the other boolean readers on this block, and its TSDoc no longer states the serializer rule backwards - the five JSON editors declare language: 'json', so invalid JSON is caught inline instead of at execution - bound the RESULT_SCAN read in SQL, not only by rows_per_resultset - pin every migration target to a live subblock id, for all blocks	2026-08-09 00:20:09 -07:00
Waleed	e6485f522d	fix(dynatrace): send the only unmute reason the API accepts and request the detail fields the tools map (#6463) ... Unmute forwarded the shared muteReason dropdown's FALSE_POSITIVE default, but Dynatrace accepts exactly one unmute reason, AFFECTED. The tool-level fallback never fired because a truthy invalid reason was already supplied, so unmute failed from the block unless the reason was changed by hand. The vulnerability, problem, and attack detail endpoints omit every optional property unless it is named in `fields`, so the descriptions, remediation guidance, affected entities, root-cause evidence, and attacker details those tools map were always null.	2026-08-08 17:10:58 -07:00
Waleed	76b535f676	chore(snowflake): drop the local write caps in favor of the shared limit (#6461) ... The 1000-row and 1 MB bound-value caps duplicated the shared 10MB tool request body gate on the same axis, at a lower threshold. Nothing fails between the two thresholds: Snowflake's 1 MB guidance is a recommendation about metadata retention rather than a hard limit, and statements above it still execute. The row array is already resident before the statement is built, so the caps also bounded no allocation the request body did not already bound. These were removed once before in 624b7373f5 and should have stayed removed.	2026-08-08 16:52:09 -07:00
4 people	3096de846e	feat(integrations): add Snowflake PAT integration (#6407) ... * feat(integrations): add Snowflake PAT integration * fix(snowflake): scope block params by operation * refactor(integrations): simplify Snowflake safeguards * refactor(snowflake): isolate statement capabilities * fix(snowflake): localize required user agent * chore(snowflake): limit changes to integration scope * fix(snowflake): correct SQL generation, transport, and param conventions Address defects found by validation against the Snowflake SQL API v2 and SQL reference docs. SQL generation: - lift PARSE_JSON out of the VALUES clause into a projecting SELECT; the previous form is rejected for any object or array value - escape backslashes as well as quotes in string literals, closing a COPY option injection through the user-or-llm stagePath and pattern fields - reject "--" in stage paths, which commented out every following clause - emit COPY INTO clauses in the documented positional order - exclude only view types in introspect_schema so temporary, external, and event tables are visible - use plain equality in MERGE and reject null or duplicate match keys - bound rows and bound-value bytes for every statement, measured in UTF-8 - reject qualified task names, which TASK_HISTORY silently ignores - replace a raw NUL byte in the source with its escape sequence Transport: - read DML stats from the documented top-level ResultSet property - drop Link-header and 391908 paging, which belong to the retired API, and report partition completeness as unknown rather than falsely complete - require a 2xx status before trusting a success SQLSTATE - cap response bodies and fail closed on invalid session context names Conventions: - inline shared params into each tool instead of cross-file spreads, which also lets the docs generator emit host and apiKey - use the official Snowflake brand mark on a white tile * fix(snowflake): emit task history time bounds as literals TASK_HISTORY only accepts bind variables for RESULT_LIMIT and TASK_NAME per BCR-1410, and that change explicitly excludes a bind passed through another function first. A bind in SCHEDULED_TIME_RANGE_START/END is therefore dropped without an error, so the requested window became a no-op and the function fell back to returning the most recent runs. Emit validated literals instead, which also restores Snowflake's seven-day range error. Also reject a fractional skip-file percentage at the block boundary rather than in the builder, and correct the cancel description: a cancelled child marks the task graph run failed, so downstream tasks are skipped rather than continuing. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-08 16:26:57 -07:00
Waleed	56200177e6	feat(mintlify): add Mintlify integration (#6457) ... * feat(mintlify): add Mintlify integration Adds all 18 documented Mintlify REST API endpoints across deployment, automation, agent jobs, prose detection, docs search/assistant, and analytics export. * refactor(mintlify): tighten payload typing and test import	2026-08-08 15:46:29 -07:00
Waleed	d317607d2c	feat(dynatrace): add the write and configuration surfaces (#6398) ... * feat(dynatrace): add the write and configuration surfaces Takes the block from 22 operations to 47. The original PR shipped the read paths plus a few ingests; this closes the gaps that made those reads dead-end. The one that was a real defect: security was read-only. The audit-vulnerabilities skill promised "a remediation queue" and then gave you no way to act on it, even though muting is the single most common triage action. Adds mute and unmute, singly and in bulk, plus the remediation items behind a third-party finding, plus the Attacks API so an exploited vulnerability can be traced to the request that exploited it. The rest, by how much they unblock: - Custom tags (read/add/delete). Entity tags already drive every selector in the block; being able to write them closes a loop that was half open. - Settings objects (schemas, list, get, create, update, delete). This is how maintenance windows, alerting profiles, and management zones are configured in modern Dynatrace, so "open a maintenance window before the deploy" was simply unreachable before. The value is a schema-defined blob, so the tool is honestly opaque rather than falsely typed; the docs tell you to mirror an existing object. Update and delete carry the updateToken so a concurrent change fails instead of being overwritten. - Synthetic monitors and on-demand batch execution, which pairs with the deploy-marker tool to gate a release on a smoke test. - Problem comment get/update/delete, and SLO create/update/delete, completing CRUD that was previously half-built. Two structural notes. Synthetic monitors are the only endpoints still on Environment API v1, so `buildDynatraceUrl` grew a v1 sibling and the shared base-URL normalizer now strips either version; the query builder also learned to repeat a param per value, which Synthetic's `tag` needs. And creating an SLO returns 201 with an empty body and the new ID in the Location header, so that tool reads the header rather than parsing nothing. Deliberately excluded: the Grail/DQL query API. It is the long-term successor to the deprecated logs/search endpoint, but it authenticates with a platform token rather than an Api-Token, so it is a second auth path and belongs in its own change. * fix(dynatrace): drill the documented JSON shapes, and require the tag selector Two problems, one found in review and one worth more than it was given. The tag operations could run without an entity selector. All three tag tools declare `entitySelector` required, but the shared block field was only marked required for List Entities, so the block let a workflow reach those tools with an invalid configuration and let Dynatrace do the rejecting. My own structural auditor missed it because it only checked that *some* visible subBlock existed for a required param, not that the specific one was required — that check is now precise, and it confirms these three were the only instances across all 47 operations. The larger one: outputs were declaring `type: 'json'` for shapes the API reference documents in full. Thirty-five of them. The top-level entities were mapped properly, but nested payloads — a problem's evidence and impact analysis, a vulnerability's risk assessment and global counts, an attack's attacker, request, entry point and exploited vulnerability, a remediation item's assessment and mute state, the synthetic execution and failure records, the metric ingest error envelope, the DQL translation — were passed through as anonymous blobs. A downstream block could not reference `attacker.sourceIp` without knowing to guess it. All of those now carry their fields. What stays opaque is now only what genuinely is, and each says why in its description: a settings object's schema-defined value, an entity's type-dependent property bag and relationship keys, caller-supplied synthetic metadata, an audit log's JSON patch, the undocumented partial-success body of log ingestion, and the handful of security-detail shapes the reference names without expanding. * fix(dynatrace): make the synthetic enabled filter tri-state Review catch. `enabled` on List Synthetic Monitors is a three-way filter — enabled, disabled, or either — and I had it as a switch. The URL builder deliberately serializes `false` (there is a test pinning that `evaluate=false` survives), so leaving "Enabled Only" unchecked sent `enabled=false` and returned only the disabled monitors: exactly backwards. Made it a dropdown with Any / Enabled only / Disabled only, matching the monitorType field directly above it, which had the same shape and already used an empty-id "Any" option. The params mapper sends nothing for "Any". Checked the other nine switches rather than assuming. None share the bug: for each of them off genuinely means false, and false is Dynatrace's own default, so serializing it is correct. A test now pins that list so the trap cannot be re-introduced by converting one of them, alongside a test covering all three states of the filter.	2026-08-07 19:39:55 -07:00
Waleed	2e7e5ae804	feat(dynatrace): add the Dynatrace integration (#6393) ... * feat(dynatrace): add the Dynatrace integration Adds a Dynatrace block backed by 22 Environment API v2 tools, covering the surfaces an observability workflow actually reaches for: - Problems: list, get, close, list comments, add comment - Metrics: query data points, list and get descriptors, ingest line protocol - Entities: list, get, list entity types - Events: list, get, ingest - Logs: search, ingest - SLOs: list, get - Application Security: list and get security problems - Audit log: read Every request path, query parameter, and response mapping is taken from the published Dynatrace API reference — no inferred fields. Auth is an access token sent as `Authorization: Api-Token ...` against a user-supplied environment URL, so SaaS, Managed, and environment ActiveGate all work. Two details worth knowing: `ingest_event` exposes Dynatrace's event timeout as `eventTimeout`, not `timeout`. The tool transport reserves `params.timeout` for the HTTP request deadline, so the obvious name would have silently retargeted the wrong knob. `get_metric` encodes its path segment with `encodeDynatracePathSegment` rather than `encodeURIComponent`, which leaves the `:` separators in metric keys and transformation operators intact, matching the docs' own examples. * fix(dynatrace): close the gaps a validation pass turned up Three real defects and one usability gap, all found by auditing the tools against the Dynatrace API reference a second time. `ingest_logs` double-encoded its payload. `logs` is a `json` param, and a `json` param arrives as a *string* whenever it comes from a long-input field or an LLM tool call — only a block-to-block reference hands over a parsed value. `JSON.stringify` on that string produced `"[{...}]"`, so Dynatrace received a quoted string where it expected an array. The block hid this in the UI path by pre-parsing, but the parse lived in `tools.config.params` and *threw* on malformed input, and it never covered the direct tool-call path at all. Both tools now normalize through the shared `parseJsonParam`, so the tool is correct regardless of who calls it, and the block just forwards the raw value. `ingest_event.properties` had the identical bug. Path identifiers were not trimmed. A problem or entity ID pasted with a trailing newline became `%0A` in the URL and 404'd with nothing to suggest whitespace was the cause. Errors dropped the part that matters. Dynatrace's ErrorEnvelope carries `constraintViolations[]`, which names the offending selector or parameter; the generic `nested-error-object` extractor returns only `error.message` ("Constraints violated."), and which extractor won was left to fallback order. Adds a `dynatrace-errors` extractor that folds the violations into the message and pins it on all 22 tools. It sits after `nested-error-object` in the chain, which already matches this shape, so no other service's error handling changes. Adds 21 tests covering URL construction for SaaS/Managed/ActiveGate, cursor pagination dropping sibling filters, identifier trimming, metric-key colon preservation, both JSON-param paths, the `eventTimeout` -> `timeout` mapping, EntityStub flattening, the audit log's dotted `dt.settings.*` keys, and the 204/200 split on log ingestion. * docs(dynatrace): add the page intro, and pin every response key in tests Adds a MANUAL-CONTENT:intro block to the generated integration page covering what the block reaches, how to get an environment URL and a scoped token for SaaS vs Managed, how selectors work, and how cursor pagination behaves. Verified it survives `generate-docs.ts` byte-identically. Also closes the last silent-failure gap the validation pass left open. A wrong top-level response key does not throw — it maps to an empty array and reads as "no results", which is indistinguishable from a genuinely empty environment. Dynatrace is unusually easy to get wrong here: the SLO list returns `slo` (singular) and the metric query returns `result` (singular). Adds a table-driven test asserting the documented key for all ten list endpoints plus the scalar keys of the ingest and single-entity responses. Confirmed it bites by flipping `data.slo` to `data.slos` and watching only that row fail. * chore(dynatrace): type the shared param map as unknown Review follow-up. `Record<string, any>` in the block's params builder dropped compile-time checking from every operation's shared params; `unknown` is enough here since the values flow straight into the tool param maps. Matches .claude/rules/sim-typescript.md, which sibling blocks (Datadog, Grafana) still violate. * fix(dynatrace): stop three silent failures found in a final read-through All three turn a failed call into something that looks like a successful empty one, which is the worst shape for an observability integration — you cannot tell "nothing is wrong" from "the call did not work". `readJsonBody` swallowed any unparseable body and returned `{}`. A gateway HTML page, a captive-portal interstitial, or a truncated payload therefore mapped every field to null and read as "no problems found". Only genuinely empty bodies are tolerated now (201 from add-comment, 204 from log ingest); anything else that will not parse raises with a truncated preview. `ingest_logs` sent `[]` when the payload was missing or empty. Dynatrace answers 204 to that, so the tool reported `accepted: true` for a call that shipped no logs. It now fails loudly instead. `encodeDynatracePathSegment` percent-encoded the whole metric key and then regex-unescaped `%3A` back to `:`. Same output, but it undoes the encoder's work and hides the intent. Colons are structural in a metric key, so it now splits on them, encodes each part, and rejoins — which says that directly. Each fix has a test, and each test was confirmed to fail in isolation with only its own fix reverted.	2026-08-07 17:35:49 -07:00
Waleed	1c0e82a4e2	perf(ci): parallelize repo audits, guard env-dependent tests, and fix the docs generator (#6358) ... * perf(ci): parallelize the repo audits and guard env-dependent tests The 21 independent audits ran as 21 sequential CI steps, each a single-threaded read-only walk of the tree. scripts/run-audits.ts runs them concurrently: 28s serial -> 5.0s wall locally at 13-way. It buffers each audit's output and replays only failures, so a green run stays quiet and a red one still names the audit and shows why. Audits needing a git base ref (block registry, migration safety) or that write files (drizzle generate) stay as their own steps. Also fixes 5 tests that fail for every macOS dev and are invisible in CI. They shell out to python3 using `match` statements and 3.12 f-string nesting, which need >= 3.10; stock macOS ships 3.9.6, so `bun run test` produced raw Python SyntaxErrors with no guard and nothing tying them to a missing tool. One also needs ripgrep, which CI installs and a Mac usually does not. @sim/testing/environment detects both and the tests skip with a reason via vitest's ctx.skip(). Under CI it throws instead: these suites deliberately run the real helper rather than a mock -- the cloud-review path/read-size bounds and the placeholder compiler's generated Python are only observable that way -- so a missing tool in CI means a security boundary silently stopped being covered, which is worse than a red build. Drops the Codecov upload. The workflow already documented it as a dead path: nothing generates apps/sim/coverage, vitest runs without --coverage, and fail_ci_if_error hides it, so it reported green having uploaded nothing. * fix(ci): raise the python floor to 3.12 and stop the bridge audit serializing the batch Two review findings, both real. MIN_PYTHON was 3.10, chosen for the `match` statements the compiler suite generates. But two of the three guarded tests also use PEP 701 f-strings -- reusing the outer quote, and embedding `#` -- which are 3.12. Verified on a real 3.11 interpreter: the match-guard test passes, the other two fail with `f-string: unmatched '('` and `f-string expression part cannot include '#'`, which is exactly the raw SyntaxError the guard exists to prevent. A 3.10 floor let them through and failed anyway. The audit parallelization did not speed CI up -- it slowed it down. Serially the 21 audits took ~31s; concurrently the batch took 39.2s wall, because check:desktop-bridge went from 1s to 39.2s and became the entire wall clock while the other 20 finished in 9s. It is the only audit that shells out through `bunx`, which re-resolves the package against the shared install cache -- a network-backed sticky-disk mount on CI. Cheap when it runs alone, serialized behind the others when they run together. Spawning the resolved compiler entry point directly removes that layer. Verified the audit still fails on a breaking bridge change rather than passing faster by doing less. * fix(docs): unbreak the MDX build and read trigger config from the registry The docs build has been failing on staging since the Smartlead merge: ./apps/docs/content/docs/en/integrations/smartlead.mdx Expected a closing tag for `<original>` before the end of `paragraph` Tool descriptions are emitted as prose, and that path escaped only braces -- every table-cell path already escaped angle brackets. MDX reads `<` as the start of a JSX tag, so a description like 'The copy is named "<original> - copy"' fails the build outright. escapeMdxProse handles the MDX-hostile characters and leaves pipes, parens and brackets alone, which are legal in prose and whose escaping would mangle markdown links. Trigger configuration now comes from the evaluated registry instead of regex over source. Static parsing silently dropped every field whose builder assembled its array imperatively or took a description as a parameter -- all ten Jira triggers lost `webhookSecret` and `jqlFilter` that way, and Monday lost its config too, so regenerating the docs was destructive. Reading real objects also deletes 232 lines of parsing. Note `required` may be a condition object rather than `true`; only an unconditional `true` renders as Required, matching the previous behavior. Tool headings now show the tool's name ("A2A Send Message") rather than its id (`a2a_send_message`), unformatted, across 241 generated pages. Names come from tools/generated/tool-metadata.ts, which CI keeps in sync. These headings feed each page's table of contents. a2a.mdx is hand-written, so its headings were updated directly. Also consolidates five hand-inlined copies of the escape chain into the escapeMdxCell that already existed, and drops 44 comments that restated the line below them. Generator: 4306 -> 4069 lines. Every refactor step was verified against a golden manifest of all 289 generated files -- proven deterministic across runs and proven to catch a one-character change -- so the only output differences are the intended ones. KNOWN GAP: extractTriggerOutputs still parses source and has the same blind spot; it already drops one Jira output section on main. Regenerating is now safe for trigger config but still lossy for trigger outputs. * refactor(ci): derive the audit list and stop shelling out through bunx Review pass over the audit runner and the tool guards. The audit list was hand-maintained alongside package.json with nothing linking them, and it had already drifted: check:cron-parity exists, passes, and ran in no CI step at all. The list is now derived from the check:* scripts with an explicit exclusion map, so a new audit is opted out deliberately rather than forgotten. That picks up cron-parity — 22 audits now, not 21. check-realtime-prune-graph.ts still shelled out through `bunx turbo`, the same pattern that took the bridge audit from 1s to 39s once the audits ran concurrently. Both now go through scripts/local-bin.ts, which resolves node_modules/.bin — the same path check:native-typecheck asserts is the native TypeScript 7 compiler, so the one guarded path is the one that runs. Audits are spawned as their script rather than `bun run <name>`, which started a bun process only to read package.json and start a second one. Tool detection is memoized per process; it was re-spawning python3 on each of the 5 call sites, in every vitest worker. The CI throw is deliberately NOT memoized — memoizing it would turn every call after the first into a silent skip, which is the failure mode the guard exists to prevent. Verified it still throws for all three guarded tests, not just the first. Also: dropped the environment module from the @sim/testing barrel so node:child_process stays out of unrelated consumers' module graphs, restored the per-audit reporting the 21 separate steps used to give (collapsible groups, error annotations, and a timing table they never had), and trimmed comments that restated their code or duplicated the runner's own docs. * fix(devin): give the 11 Devin tools real display names Every Devin tool had its id as its `name` (`list_session_messages`), so the generated docs rendered `### list_session_messages` where every other integration renders a human name. It was the only integration doing this -- 11 of 4427 tools. Names take the service prefix, matching the majority convention (3200 of 4416 names start with their service). Also points the ship skill at check:audits instead of hand-listing the audits. That copy had drifted five behind package.json: cron-parity, import-specifiers, sql-date-binding, trigger-block-cycle and native-typecheck were all missing, so shipping never ran them. It was the third copy of that list; there is now one. * fix(docs): read trigger outputs from the registry too Closes the gap left by the config fix: extractTriggerOutputs still parsed source, so triggers whose outputs come from a builder call lost their tables. jira_webhook had no output section at all. The registry was not a drop-in, which is why the naive swap deleted 10,298 lines earlier. The two sides encode nesting differently. A TriggerOutput marks a group by OMITTING type and holding children as sibling keys: issue: { id: { type: 'number' }, title: { type: 'string' } } while the renderer walks the JSON-Schema-ish shape the parser used to synthesize: issue: { type: 'object', properties: { id: …, title: … } } formatOutputStructure only descends into .properties, so handing it the raw registry value collapsed every nested group to one untyped row and dropped its children. normalizeTriggerOutputs converts between the two, preserving leaves that already declare properties/items and merging the 13 hybrid nodes that carry both a type and inline children. Measured across all 368 triggers before changing anything: 155 identical, 213 divergent, and the divergence was purely the nesting encoding — no node has a non-string type, and a group never carries its own string description, so leaf-vs-group classification is unambiguous. That is what makes a nested property literally named 'description' (42 of them) survive. Deletes the static path: extractTriggerOutputs, resolveTriggerBuilderFunction, resolveTriggerOutputsConstant, readTriggerSiblingModules, getWebhookProviderConstants, plus resolveConstStringValue and matchQuotedProperty which the config fix had already stranded. 20 output sections recovered (linear 79->93, tiktok 6->11, jira 44->45) and 1698 rows. Verified independently: zero sections lost across all 289 generated files, no file lost rows, output deterministic across regeneration. The 96 deletions are all corrections, not losses. 70 are confluence fields the parser flattened out of `comment: { ...buildContentEntityFields(), parent: {…} }` and rendered as top-level trigger outputs; they reappear nested under their parent in the same hunk. 8 are greenhouse key ordering, 6 are intercom descriptions the parser had dropped, 1 is a vercel row moving position. Generator: 4069 -> 3903 lines. * chore(test): silence vite 8 deprecation warnings in the sim vitest config @vitejs/plugin-react v4 targets pre-rolldown Vite: it sets `esbuild.jsx` and `optimizeDeps.rollupOptions`, both deprecated under Vite 8's oxc pipeline, and self-reports that plugin-react-oxc should be used instead. v6 is that plugin merged back under the original name — it requires Vite ^8, drops Babel entirely, and emits none of those options. Vite 8 also resolves tsconfig paths natively, so vite-tsconfig-paths is replaced by `resolve.tsconfigPaths`. Full apps/sim suite unchanged: 1483 passed / 2 skipped files, 20415 passed / 30 skipped tests. * refactor(docs): drop 33 more comments that restated their code Second pass over the generator, e.g. `// Copy icons from sim app to docs app` above `copyIconsFile()`. Kept the multi-line runs (those carry reasoning), the ones with concrete examples, and the one marking a deliberate empty catch. Verified byte-identical output across all 289 generated files. Generator: 3903 -> 3870 lines, 4306 at the start of this branch. * refactor(ci): read package.json once in the audit runner auditScripts() re-read the manifest the module body had already loaded. * fix(pdl): name the tools directory after the tool ids People Data Labs declared `pdl_*` tool ids under `tools/peopledatalabs/`. Every other integration names the directory after its id prefix -- 259 of 260 before this, and PDL was the only exception. The docs generator locates a tool's definition by deriving the directory from the id prefix, so it looked in `tools/pdl/`, found nothing, and returned null for all 11 tools. peopledatalabs.mdx rendered eleven bare `###` headings with no description, no Input table and no Output table. Renaming the directory rather than the ids: tool ids are persisted in saved workflows, so renaming those would break existing users. The directory is internal -- 15 files' imports. Fixed at the source rather than teaching the generator a fallback. A special case would have left the invariant broken and the next integration free to break it again; now 260 of 260 hold, and the generator needs no exception. peopledatalabs.mdx: 11 empty headings -> 456 lines. Repo-wide: zero pages with an empty action body.	2026-08-06 19:08:32 -07:00
Waleed	aae9ce62e7	feat(smartlead): add Smartlead integration (#6352) ... * feat(smartlead): add Smartlead integration Adds a Smartlead block with 22 tools covering campaigns, sequences, leads, analytics, and webhooks. Every request path, parameter, enum, and response mapping was verified against the live Smartlead API rather than its documentation, which proved unreliable: - `POST /campaigns/new` (documented) 404s; the real path is `/campaigns/create` - `GET /campaigns/{id}` and `/sequences` return bare payloads, not the documented `{success, data}` envelopes - `/statistics` returns paginated per-email rows, not the documented aggregate - `POST /campaigns/{id}/leads` returns import counters under entirely different field names than documented - documented `/leads/{id}`, `/top-level-analytics`, `/all-leads-activities`, `/lead-lists/`, and `/lead-tags/` all 404 Enum values (campaign status, track settings, stop-lead settings, webhook event types, engagement status) were probed value-by-value against the API. Notes on the API's shape, encoded in the mappers: - string-encoded numbers (`total_leads: "1"`, `sent_count: "0"`) are normalized to numbers so a field never changes type between operations - `seq_delay_details` is read as `delayInDays` but written as `delay_in_days` - webhook writes echo `event_type_map`/`category_id_map` objects while the list endpoint returns `event_types`/`categories` arrays; both map to arrays - `track_settings` reads back in a vocabulary it will not accept on write Statistics rows and lead message-history entries pass through unmapped: no account could produce a non-empty sample, so no field names were invented. Email-account tools and a webhook trigger are omitted for the same reason. Adds a `smartlead-errors` extractor since the API's 400s put the useful text in `message` while `error` is only "Bad Request". * feat(smartlead): expand to the core workflow surface and fix review findings Grows the block from 22 to 47 tools and fixes every defect found in review. New tools (all executed against the live API end to end): campaign email accounts (list/add/remove), duplicate, delete, CSV lead export, webhook delete + delivery summary, lead + mailbox statistics, top-level analytics by date, lead activities, get lead by id, unsubscribe from campaign, unsubscribe globally, mark complete, delete from campaign, master-inbox replies, lead lists (list/get/create/update/delete), email accounts, clients. The endpoint inventory was rebuilt by extracting method+path from all 212 reference pages, which corrected several earlier conclusions: get-lead-by-id is `/leads/{id}` (not under `/campaigns/`), lead lists are `/lead-list/` (singular), and lead activities are `/campaigns/all-leads-activities` with no campaign segment. More documented paths that 404 in reality: lead tags at `/crm/leads/tags`, and webhook delete at `/campaigns/{id}/webhooks/{id}` — deletion actually takes the id in the body. Shapes the docs got wrong again, caught live: `GET /leads/{id}` wraps the lead in a single-element `data` array; `DELETE .../leads/{id}` answers with the bare string `success`, not JSON; duplicate returns `newCampaignId`; create/update lead list take `listName`, and mark-complete takes `campaign_lead_map_id` where its siblings take `lead.id`. Review fixes: - get_campaign, get_campaign_analytics and get_lead_by_email reported an all-null success for a missing resource, because Smartlead answers HTTP 200 with `{}` (or an empty body) instead of 404. They now fail closed. - update_campaign_settings silently reset stop_lead_settings and send_as_plain_text: their dropdown defaults are materialized at block creation, so every settings update carried them. Both now default to "Leave unchanged". - Malformed JSON in Leads/Sequences/Custom Fields resolved to `undefined`, which overwrote the raw string the executor falls back on and dropped the field silently. Parsing now raises, and is scoped to the operation that consumes the field so a stale hidden value cannot fail an unrelated one. - The four documented import overrides (block/unsubscribe/duplicate/bounce lists) had no field, so the block's own skill instructions were unexecutable. - leadId did not distinguish lead.id from campaign_lead_map_id; passing the latter 404s, and list_campaign_leads surfaces it first. - Path ids are trimmed and escaped; dead code and a hand-rolled id mapper removed. Unverified and called out rather than guessed: add/remove email accounts to a campaign (no mailbox could be connected, so only their error shape was seen), and the row shapes for statistics, message history, inbox replies, email accounts and clients — every one of those collections was empty on the verification account, so their rows pass through unmapped. * fix(smartlead): correct request params and outputs found in re-validation Three tools sent a parameter Smartlead's validator rejects outright with 400, so the affected operations failed whenever the field was filled in: - get_campaign_lead_statistics paginated with `skip`; the endpoint accepts `offset` and only echoes it back as `skip`. - list_lead_activities and list_inbox_replies both sent a campaign filter. `campaign_id`, `campaignId`, `campaign_ids` and `email_campaign_id` are all rejected, so the filter is gone rather than advertised and broken. mark_lead_complete reported `next_sequence: null` on every call, including when a step remained: `status.nextSequence` is an object, not a number. It now maps to `next_sequence_id` and `next_sequence_delay_in_days` — verified live returning step 10093171 rather than null. get_lead_by_id reused the by-email mapper, so it always claimed the lead belongs to zero campaigns; `GET /leads/{id}` omits `lead_campaign_data` entirely. It now declares the narrower shape it actually returns. A stale advanced `clientId` leaked into list_email_accounts: advanced subblocks serialize without evaluating their condition, and that tool consumes `clientId` while sitting outside its condition list. The field is now offered for that operation too, so the value is visible wherever it is sent. Two dropdowns had defaults that act on their own. `status` defaulted to PAUSED, so choosing Update Campaign Status and never opening the dropdown paused the campaign; it now requires an explicit choice. `pauseLead` sent `false` on every categorization, which risks resuming a paused lead; it now defaults to leaving the state alone. Also counts CSV export rows with a quote-aware scan so a newline inside a name, location, or custom field no longer inflates the count, and fills in the block output declarations for the fields the 47 tools actually return. * fix(smartlead): preserve a zero-day next-sequence delay and render enum values in docs A next sequence scheduled to send immediately reported no delay at all: `Number(next.delayInDays) || null` mapped a legitimate 0 to null. Tool descriptions built enum lists with template literals. The runtime value and the LLM-facing tool metadata were correct, but the docs generator reads the description statically, so the public page rendered `${SMARTLEAD_CAMPAIGN_STATUSES.join(...)}` instead of START, PAUSED, STOPPED. The five affected descriptions now spell the values out. * fix(smartlead): stop email-account tools from emitting mailbox credentials Connecting a real mailbox to the verification account made the email-account response shapes observable for the first time, and they carry the stored credentials: `GET /email-accounts/{id}/` and the campaign route return `password` in plaintext, the list route returns it base64-encoded, and both carry `imap_password`. Both tools passed rows through unmapped, so those values would have reached workflow output, execution logs, and model context. They now select fields explicitly and omit the credentials. Verified against the live API: the API response contains the password while the tool output does not, for both tools. Also fills in the real email-account fields, which were previously an opaque array — id, sender identity, SMTP/IMAP host and port, verification state and last error, sending caps, warmup status, and tags. * fix(smartlead): remove the dead campaign field that could target the wrong campaign Removing the campaign filters from list_lead_activities and list_inbox_replies left their `activityCampaignId` subblock, its params mapping, and its inputs entry behind. Two problems, the second serious: - On those two operations the field promised campaign scoping the API cannot do. Smartlead rejects every candidate key (`campaign_id`, `campaignId`, `campaign_ids`, `email_campaign_id`), so the value was silently discarded and account-wide results were reported as scoped. - Worse, the field is `mode: 'advanced'`, and advanced subblocks serialize without evaluating their condition. A value left over from listing activities therefore fed `campaignId` on all 32 campaign operations through the `params.campaignId || params.activityCampaignId` fallback. Configuring List Lead Activities with campaign 111, then switching the block to Delete Campaign and leaving Campaign ID blank, would have passed required-validation and deleted campaign 111. Both list tools now also say plainly that Smartlead exposes no campaign filter, rather than advertising one in their descriptions. Also: route mark_lead_complete's next-sequence id through the shared numeric coercion, since Smartlead string-encodes numbers inconsistently and its sibling field already arrives as a string; re-bind the two enum constants that lost their last consumer so the literal descriptions cannot drift undetected; and declare the 17 tool output keys the block was missing — `accounts` most importantly, which is the entire payload of both email-account tools.	2026-08-06 17:40:20 -07:00
mzxchandraandWaleed Latif	dc5bab6e54	feat(embeddings): multi-provider Embeddings block on a shared core (#6317) ... * feat(embeddings): multi-provider Embeddings block on a shared core The Embeddings block was OpenAI-only with a bare fetch: no batching, no retry, no metering, and no hosted-key support. Meanwhile the knowledge-base indexing path already had a real multi-provider engine. Nothing bridged the two, so the block could not reach Gemini and the KB engine could not be reached from a workflow. Extract the shared core into lib/embeddings/ first, then build breadth on top of it, so both the KB path and the block resolve models and providers from one catalog and one set of adapters instead of a third parallel implementation. - lib/embeddings/: catalog, client, key resolution, batching, L2 normalization, and adapters for OpenAI, Azure OpenAI, Gemini, Cohere, and Mistral - lib/knowledge/embeddings.ts becomes a thin KB wrapper with its exported signatures unchanged; the 1536-dimension vector invariant does not move - one tool per provider from a shared factory, behind a single /api/tools/embeddings route and contract - new `embeddings` block type; the `openai` block is left functionally untouched and only leaves the discovery surfaces via hideFromToolbar plus sunset.replacedBy, so placed instances keep working unmigrated - openai_embeddings is now an alias of embeddings_openai, so legacy instances pick up batching, retry, and metering with no visible change * fix(embeddings): report an unsupported dimension as a client error The route validated the model and the provider match up front but left `dimensions` to be checked inside embed(), where resolveDimensions throws and the generic catch maps it to 502. A typo in the block's dimension field, or a reference expression resolving to an out-of-range value, was reported as an upstream gateway failure rather than bad input. Resolve dimensions in the route alongside the other boundary checks and return 400. The throw stays the single source of the message, so the two call sites cannot drift. Adds route tests covering auth, the response shape, each boundary rejection, input normalization, and the 502 path for genuine provider failures. * fix(embeddings): only send a dimension when the caller asked to reduce resolveDimensions() returns the model's native size when no reduction is requested, and that resolved value was handed straight to the adapter. The adapters guard on `dimensions !== undefined`, so the field was always populated and always sent. Models that support Matryoshka reduction accept their own native size, so this was invisible for text-embedding-3-*, gemini-embedding-001, embed-v4.0, and codestral-embed. Models that do not support the parameter at all reject it outright: every unreduced request to text-embedding-ada-002 and mistral-embed failed with a 400, which is both of the models whose catalog entry has no supportedDimensions. Track the caller's explicit reduction separately from the resolved dimensionality. The resolved value still drives reporting and billing; only the requested one reaches the wire. Found by driving the live provider matrix against all four providers. * test(knowledge): de-flake the sync-engine suite Every test dynamically imported the module under test, so the first one to run paid the whole cold-load cost inside its own 10s timeout and failed intermittently under load. The dynamic imports were working around a hoisting problem: mockMapTags is a top-level const read by a vi.mock factory, and vi.mock is hoisted above it, so a static import of the module under test crashes with a use-before-initialization error. Declaring the mock through vi.hoisted() removes that constraint, which is the pattern the testing guidelines already call for. One static import replaces 42 dynamic ones. The file drops from ~15s to ~2s and passed 5 consecutive runs. * fix(embeddings): drop a capability the selected model no longer offers The per-model Dimensions and Task Type dropdowns each share one subblock id, and nothing clears a stored subblock value when its dependsOn fields change — dependsOn only feeds rendering. A choice made for one model therefore outlives a switch to another. Picking 3072 on text-embedding-3-large and switching to -3-small left 3072 stored while the dropdown offered at most 1536, and the block forwarded it. Same for a task type: 'similarity' chosen on Gemini survived a switch to Cohere, which has no equivalent input type. The guards only checked that the model declared the capability at all, not that the value was one it lists. Check membership so a stale value falls back to the model's native size, or is omitted, instead of being sent and rejected. The user cannot have deliberately chosen an option the dropdown stopped presenting. * feat(embeddings): use the latent-constellation mark for the block icon Replaces the scatter-plot-on-axes placeholder with a centre node, four neighbours, and the rays between them — a point and its nearest neighbours in embedding space, which is what the block actually produces. The axes mark read as a generic chart and said nothing specific to embeddings. Nodes are filled so they hold their shape at small sizes. The rays carry less weight than the nodes to keep the hierarchy, but at 1.6/0.9 rather than the 1.4/0.75 they were drawn at, so they do not thin out to loose dots in the 14px block-search row. Kept byte-identical between the app and docs icon sets. * fix(embeddings): declare the outputs the legacy openai block returns openai_embeddings became an alias of embeddings_openai, so the legacy block's runtime payload gained `provider` and `dimensions`. Its declared outputs still listed only embeddings/model/usage, so the tag picker never offered two fields every run demonstrably returns, and downstream blocks could not reference them. Declaring them is additive and does not touch execution. Asserts the legacy block's output keys match the replacement's, since both run the same tool and neither should expose fields the other lacks. * fix(copilot): resolve same-id subblock variants before validating A block may declare one field id several times, each variant conditioned on another field — the embeddings block declares model, dimensions, and taskType once per provider, and the image and video generators do the same. Validation keyed a map by id alone, so whichever variant was declared last silently became the validator for every write to that field. Programmatic edits to an embeddings block were therefore checked against Mistral's option lists whatever the saved provider: `text-embedding-3-small` was rejected as not one of mistral-embed/codestral-embed, and dimensions valid only elsewhere (3072, 768) could not be set at all. Values that happened to overlap the last variant passed, so automation saw partial success rather than a clean failure. Keep every candidate per id and pick the one whose condition holds, evaluating against the mutation's inputs merged over the block's saved values so a partial write still resolves. When no condition matches, fall back to the union of all variants' options rather than guessing. Conditions still never gate whether a field may be written — that was a deliberate choice and a hidden field stays writable. They only select which definition describes the field, and an unresolved condition widens the accepted set instead of narrowing it. * fix(copilot): prefer a conditioned variant over an unconditioned catch-all An unconditioned same-id variant matches every set of values, so it would shadow a genuinely selected variant purely by being declared first. Prefer a variant that actually asserted something about the current values. No block in the registry currently declares a catch-all ahead of a conditioned variant on a field where it would change validation, so this is a guard against the pattern rather than a fix for a live case. * chore(embeddings): scope this branch to the multi-provider block Two changes made while building the Embeddings block are not part of it and ship separately, so their files are restored to staging here: - copilot edit-workflow validation resolving same-id conditional subblock variants. The embeddings block surfaced it, but it is a platform fix affecting ~20 blocks that declare a field id more than once, and it narrows what programmatic edits accept — that deserves its own review. - the sync-engine test de-flake, which is unrelated test hygiene. Both are preserved in full on feat/embeddings-full-snapshot. Note this restores the reported bug where a programmatic edit to an embeddings block validates model/dimensions against the last-declared provider variant. The block is unaffected in the editor and at runtime. * fix(embeddings): honor per-model token limits and bound the JSON input path Review round 1. Batching used one 8,000-token constant for every model, inherited from the knowledge-base engine this branch extracted. `batchByTokenLimit` truncates any single text above the limit it is given, so that constant both sent oversized input to models with a lower ceiling and silently dropped content models with a higher one accept: - Gemini declares 2,048, so a 3,000-token text passed through whole and the provider rejected it, surfacing as a 502. This also affected knowledge-base indexing on staging, which uses the same constant. - Cohere declares 128,000, so anything past 8,000 was truncated for no reason. Batch against the selected model's own `maxInputTokens` instead. Using the per-input ceiling as the per-batch budget also keeps every individual text within it. The contract bounds the array arm of `input`, but a JSON-encoded array arrives as a plain string and `normalizeInput` only expands it after validation — so neither the 1,000-input cap nor the non-empty checks applied to the reference-expression path the route was written to accept. `"[]"` also reported success with no vectors. Re-check the normalized list so the bounds hold for both shapes. * chore(embeddings): regenerate tool metadata for the new embedding tools CI's tool-metadata:check gate failed: registering embeddings_openai, embeddings_gemini, embeddings_cohere, and embeddings_mistral left the generated tool-ids/metadata/outputs artifacts stale. * fix(embeddings): project before batching, and keep the sunset block's docs icon Review round 2. Projection ran inside callEmbeddingAPI, after batchByTokenLimit had already measured and truncated the original text. The projector rewrites resolved secrets to placeholders, which changes length, so batching sized against a string that was never sent: a lengthening projection then pushed input past the model's ceiling and the provider rejected it, and a shortening one discarded document content that would have fit. Project once up front, then batch the projected text, so truncation measures what actually goes to the provider. This also keeps projection to exactly one call per embed(), so no retry can re-project. Separately, marking the legacy openai block hideFromToolbar dropped it from the generated docs icon map, which only retains hidden blocks when they are versioned. integrations/openai.mdx is deliberately kept — docsLink is baked into every placed instance — so BlockInfoCard lost its icon and fell back to a text tile. A sunset block keeps its docs page for the same reason a hidden versioned block does, so the generator now treats it the same way. The sim-side integrations map still omits it, which is intended: that feeds the discovery page a sunset block should not appear on, and placed blocks render from the registry's own icon reference. * fix(embeddings): override stale block params instead of omitting them Review round 3. The generic handler merges the params() result over the original inputs (`{ ...inputs, ...transformedParams }`), so omitting a key leaves the stale value in place. The previous round dropped an unsupported taskType or dimensions by omission, which was therefore a no-op through the executor path: a reduction or task type chosen for one model still reached the tool after a model switch. Rewrite each stale field to an explicit `undefined`, which does override in a spread. Same class of bug for `model` itself, which was forwarded whenever present without checking it belongs to the selected provider. Every provider's model dropdown shares the `model` id, so switching provider kept the previous provider's model and failed at the route as a mismatch. It now falls back to the provider's default unless the saved model actually belongs to it. Tests assert the merged result rather than the returned object, since the return shape alone cannot distinguish an omitted key from an overridden one — which is exactly why the previous fix looked correct and was not. * fix(embeddings): discount the batch ceiling when the tokenizer is foreign Review round 4. Batching measures with tiktoken, which only has encodings for OpenAI models — every other id falls back to cl100k_base. Gemini's 2048, Cohere's 128k, and Mistral's 8192 were therefore enforced in OpenAI token units, so an input near one of those ceilings could still be rejected upstream or trimmed more than needed. A true fix needs per-provider tokenizers, which the repo does not have: estimateTokenCount is a chars-per-token heuristic, and truncation needs a real encode/decode pair to slice on a token boundary. So the ceiling is discounted for foreign tokenizers rather than trusted exactly. The discount is one-sided on purpose. Overshooting means the provider rejects the whole request; undershooting only trims a text that was already at the limit, so the margin errs toward the second. resolveBatchTokenCeiling is a pure function tested directly, rather than inferred from truncation behavior, so the guarantee holds per model as the catalog grows. * fix(embeddings): keep the batch ceiling exact and warn before truncating Review round 5. Reverts the safety margin from round 4. The two review findings were in direct tension: round 4 flagged that a foreign model's ceiling is measured in tiktoken units, and the margin added to absorb that error reintroduced the round 3 harm — valid content truncated below the provider's declared limit. The margin was the wrong trade. It swapped a loud failure for a silent one: an undercount surfaces as a provider rejection the caller can see and act on, while shortening an embedding's input produces a degraded vector that is indistinguishable from a good one at every layer above it. Silent quality loss in a retrieval index is the worse outcome, and it is also the harder one to ever notice. So the declared ceiling is applied exactly, and truncation is no longer silent: an input above the limit now logs a warning naming the model, the limit, and whether the count was approximate. hasApproximateTokenCount records which models are counted with a foreign tokenizer without being used to shrink anything. The tokenizer imprecision itself remains, and cannot be fixed without per-provider BPE the repo does not have — estimateTokenCount is a chars-per-token heuristic, and truncation needs a real encode/decode pair to slice on a token boundary. * refactor(embeddings): drop dead surface and enforce OpenAI's item cap Audit follow-ups on the multi-provider embeddings work: - Enforce OpenAI's documented 2048-entry `input` array cap in the OpenAI and Azure adapters. Nothing bounded item count on the OpenAI path — batching bounds tokens per request, so a batch of many short inputs could exceed it. - Make the provider item cap single-source. It was declared both on the catalog entry and on the adapter, read through a `??`; the adapter is the wire-protocol owner, so the catalog copy is gone. - Have the knowledge-base view call `getKbEligibleModels()` instead of re-deriving the same `kbEligible` filter inline. - Remove dead surface: the unused `EMBEDDING_TASK_TYPES` constant, `EmbeddingToolDefinition`, `HOSTED_KEY_PROVIDERS`, and the five request-body fields (`workspaceId`, `workflowId`, `executionId`, `userId`, `useHostedCostTracking`) the route never reads. - Trim `@/lib/embeddings` to what callers outside the module use. - Drop the route's manual request-id plumbing; `withRouteHandler` supplies it. - Fix two comments that had drifted onto the wrong declaration. * fix(embeddings): normalize reduced Cohere output; correct OpenAI token ceiling Second validation pass against provider documentation. - Cohere: normalize locally when `output_dimension` reduces below native. Cohere documents the parameter as Matryoshka truncation but never states that it renormalizes, and an unnormalized vector silently skews cosine similarity. `l2Normalize` is idempotent, so this is a no-op if Cohere already returns unit vectors and a correctness fix if it does not. Covered by a test that fails without it. - OpenAI: raise the per-input ceiling from 8191 to the 8192 the API reference documents, so a maximal input is no longer truncated by one token. - Share the OpenAI response type with the Azure adapter instead of declaring an identical copy, mirroring how the mail providers share `_nodemailer`. - Rewrite the Gemini item-cap comment to say the 100-item limit is observed rather than documented, which is what Google's reference actually supports. Docs: add a manual intro to the Embeddings page covering providers, models, inputs, outputs, and comparability rules. The generated Input tables are empty because `createEmbeddingTool` builds params programmatically and the docs generator only reads literals, so the manual section carries that reference. * fix(embeddings): split per-input and per-request token limits; close provider gaps Four gaps found in the validation pass. Gemini token counts were estimated, not measured. `BatchEmbedContentsResponse` carries `usageMetadata.promptTokenCount`; without reading it the client fell back to tiktoken, which has no Gemini encoding and silently used `cl100k_base` — the wrong tokenizer on a count knowledge-base runs bill against. `maxInputTokens` was doing two jobs: the per-input ceiling that decides truncation, and the per-request budget that decides how many inputs share a batch. These are different provider limits, and conflating them meant Cohere packed batches against its 128k per-document ceiling while OpenAI's documented 300,000-token request cap went unenforced. They are now separate fields. Truncation moves out of `batchByTokenLimit` and into `embed`, so it happens once, against the per-input ceiling, and always logs. The request budget is floored at that ceiling — a budget below it would truncate inputs the provider accepts. Batch sizes are unchanged everywhere except Gemini, which rises from 2048 to the 8192 the other providers already used. codestral-embed now offers its documented 3072 maximum. Its API default is 1536, so the offered sizes straddle the default; the catalog invariant relaxes from "native size first" to "native size present", which is what the block relies on. The Mistral API-key field no longer differs from the other three. Sim stocks `MISTRAL_API_KEY` — `mistral_parse` already hides its key field on hosted — so one field with `hideWhenHosted` replaces the conditional pair. Docs: correct the API-key row, which described the old Mistral-only behavior. * refactor(embeddings): derive block options from the catalog; use shared helpers Findings from a four-angle quality review. Reuse: `splitByItemLimit` and `processWithConcurrency` were reimplementations of `chunkArray` (`@sim/utils`) and `mapWithConcurrency` (`@/lib/core/utils/concurrency`), so `lib/embeddings/batching.ts` is gone. That helper's doc forbade a throwing mapper; embedding legitimately wants a failed batch to fail the call, since a partial vector set is not a usable result, so the contract is reworded to cover both intents rather than forked. The block no longer hand-copies the catalog. Its model, task-type, and dimension dropdowns are derived from `EMBEDDING_MODELS`, which deletes roughly 150 lines of literals that had to be kept in step by a drift test. The comment claiming this was impossible was wrong: `generate-docs.ts` only reads `subBlocks` looking for an `id: 'operation'` entry, which this block does not have. Verified by regenerating — `embeddings.mdx` and `integrations.json` come out byte-identical. Single-sourced two maps that were stated twice: BYOK provider ids (which encode the non-obvious gemini -> google mapping) and the per-provider default model. The route previously took its default from `getModelsForProvider(provider)[0]`, which silently depended on catalog key order. Azure's `endpoint` and `apiVersion` are required on their own context type instead of optional on the shared one, so the adapter can no longer be built without them and emit an `undefined/...` URL. Also: contract enums now `satisfies` the catalog unions so they cannot drift, the barrel exports only what callers outside the module use, the redundant `requestedDimensions` field is a parameter, the bare `getEmbeddingModelInfo()` call is a named `assertKbEmbeddingModel`, and the route checks payload size before scanning entries rather than copying the body first. * docs(embeddings): correct comments that drifted from the code A comment pass over the feature found four that no longer matched what they sat on, all introduced by earlier rounds of this work. The contract's `satisfies` note promised that adding a catalog provider could not leave the wire enum stale. It cannot deliver that: `satisfies` proves every listed member is valid, not that the list is exhaustive, so an addition stays silently absent. Reworded to say what it does and does not catch. The client cited Gemini as a provider that omits usage, which the Gemini adapter now contradicts — it reads `usageMetadata.promptTokenCount`. Every adapter defines `parseTokens`, so the fallback is about a response lacking a usage block, not about a particular provider. `l2Normalize` documented only Gemini, though Cohere now calls it for a different and stronger reason, and "normalizes in place" read as mutation when the function returns a copy. The route's new size-guard comment claimed it avoids copying the payload; nothing there copies. The real reason is that summing lengths gates before the per-entry character scan. Also: split the derived-sub-block TSDoc so both constants carry hover text, gave the payload cap its own doc, dropped one comment that restated a signature, and tightened two long blocks without losing a fact. * fix(docs): generate tool inputs for factory-built tools The four embeddings tools rendered header-only Input tables. `extractToolInfo` finds a tool's `params` by regex over the tool's own file, and these files hold nothing but a `createEmbeddingTool({...})` call — the params live in the factory's module. There was already a fallback for a same-file `...spread` base, so this adds the cross-module equivalent: follow the factory's import and read `params` from there. Two things surfaced once the tables populated. `hosting` was not in the set of keys that terminate the `params` capture, so the non-greedy match ran past it to `request:` and swallowed the whole hosting block. Every tool with a `hosting:` section between `params:` and `request:` was publishing `pricing` and `rateLimit` as if they were user-facing inputs — this drops those rows from eight unrelated integration pages as well. The shared apiKey description was a template literal, which the regex emitted verbatim as `${name} API key`. It is now a static string, matching how every other tool in the repo declares one. Docs: the Embeddings page keeps a prose intro in its MANUAL-CONTENT block like other integrations, with the hand-written input/output tables removed now that the generated ones are correct. The sunset `openai` page loses its `encodingFormat` row — page generation skips hidden blocks, so that page is frozen and would otherwise keep advertising a parameter the aliased tool no longer accepts. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-06 15:25:17 -07:00
Waleed	c3da54470b	docs(sso): correct callback host and issuer guidance, document Entra SAML and IdP-initiated behavior (#6334) ... * docs(sso): use the deployed host in callback and entity ID examples * docs(sso): correct host, issuer, and provider-id guidance; document Entra SAML and IdP-initiated behavior * docs(sso): send Entra federation metadata to the field that reads it	2026-08-06 12:01:45 -07:00
Waleed	ab257555d8	fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs (#6311) ... * fix(sso): link Entra sign-ins to existing accounts and enforce unique provider IDs Better Auth 1.6.23 calls the account-linking handler with trustProviderByName: false, which disables the trustedProviders allowlist for SSO entirely. Trust now comes only from the provider's domainVerified flag, which Sim never set — so any user who already had a Sim account was stranded on "account not linked". Entra never sends email_verified, so this hit every Microsoft tenant. Sim already proves domain ownership via sso_domain before a provider can be registered, so the register route mirrors that decision onto domainVerified. The column defaults to true so existing providers keep signing in across the deploy, since enabling the option turns sign-in into a hard gate. Also enforces the providerId uniqueness Better Auth already assumes: it rejects any id that exists in any tenant and resolves providers by that column alone, so a second customer picking "azure-ad" could not register at all and got an opaque 422. Sim now returns a 409 naming a free id, and a unique index makes the duplicate-row state unreachable. * fix(sso): revoke domain trust when verification is removed mid-update The create path re-checks domain ownership after Better Auth persists the provider and rolls the row back if the verified sso_domain row disappeared in that window. The update path had no equivalent, so deleting the verified domain while updateSSOProvider was in flight still set domainVerified, restoring same-email account-linking trust for a domain the org no longer proves it owns. The update path has no newly-created row to roll back, so it clears the flag instead: that denies linking and blocks sign-in on the provider until the domain is verified again. * fix(sso): make domain-trust grants atomic and propagate revocation Greptile flagged that the ownership check and the domainVerified write were separate statements, so a domain deleted between them still ended with trust granted. Two changes close it from both sides. The grant now folds the ownership test into the UPDATE's WHERE clause, so Postgres evaluates both in one statement and the write matches nothing once the proof is gone. Removing a verified domain now clears domainVerified for providers on that domain, in the same transaction as the delete. This was a standing gap, not just a race: deleting a domain previously left linking trust set indefinitely. Together the provider cannot end up trusted without current ownership in either commit order — if the grant lands first the delete clears it, and if the delete lands first the grant no-ops. * fix(sso): report a refused domain-trust grant instead of returning success The conditional grant could match zero rows if the verified domain was deleted between the pre-write check and the write. The route ignored that and returned 200, leaving a provider that cannot sign anyone in while telling the admin it saved. The grant now reports whether it matched, and that result is the single decision point on both paths: the create path rolls the provider back, the update path clears the flag, and both return SSO_DOMAIN_NOT_VERIFIED. This also drops the separate post-write ownership read, since the UPDATE re-tests ownership itself. * feat(sso): let admins map IdP claims, and trim setup comments Identity providers disagree on which claim carries each value — Entra can send the address as `upn` rather than `email` — and the mapping was hardcoded, so a mismatch had no fix in the UI at all. Adds an Attribute mapping section for both protocols, defaulting to each protocol's standard claim names shown as placeholders, so the common case still needs no input. Editing an existing provider now loads its stored mapping and only treats a value as an override when it differs from the default, so a saved custom mapping is never silently rewritten. * feat(sso): expose the standard enterprise IdP options in the setup form Rounds out the form with the options Better Auth already accepts but the UI hid, so a non-standard IdP no longer dead-ends at a field that cannot be set. SAML gains signature algorithm, digest algorithm and NameID format. Only SHA-256 and stronger are offered: Better Auth warns on SHA-1 as deprecated and rejects anything outside its secure set, so weaker choices would only produce failed saves. SAML also surfaces the SP Entity ID beside the ACS URL. IdP admins are usually handed a vendor metadata document; Sim does not publish one, and these are the two values it would carry. OIDC gains authorization, token and JWKS endpoint overrides for providers whose discovery document is incomplete or unreachable. Discovery still fills them in when they are left blank. All of these load from the stored config when editing, so re-saving a provider cannot quietly drop them. * fix(sso): withhold domain trust from personal providers on the hosted deployment A personal (org-less) provider has no verified domain behind it, but the trust grant treated it as authoritative anyway. On the multi-tenant deployment that is an account-takeover primitive: anyone able to register one could claim a domain they do not own, point it at their own IdP, and have a sign-in auto-link to an existing account on that domain. Sim's UI always registers org-scoped, so this only reaches direct API callers. Self-hosted deployments are single-tenant — the operator is the only tenant — so the org-less path keeps working there. Also clears the attribute mapping when the protocol changes: claim names are protocol-specific, so an OIDC override carried into a SAML config would save a mapping the IdP cannot resolve. * docs(sso): correct the personal-provider trust note after the hosted gating * fix(sso): drop the inert SAML algorithm selects, make NameID format clearable The signature and digest algorithm selects were placebo controls. Tracing @better-auth/sso 1.6.23, those two values are only read by validateConfigAlgorithms, mergeSAMLConfig and sanitizeProvider — createSP and createIdP never pass them to samlify, so nothing they select reaches the SAML exchange. Bugbot separately noted they could not be cleared, since Better Auth merges with `??` and omitting a key keeps the stored value. A control that neither applies nor clears should not exist, so both are removed. NameID format is genuinely wired (createSP passes it as nameIDFormat) and is kept, but is now always sent rather than omitted when set to the provider default. samlify falsy-guards the value, so an empty string reads as unset and "Provider default" can actually clear a stored override. The read-only provider view also now shows the SP Entity ID and the ACS label for SAML — admins land there after saving and need the same two values the form says their IdP requires. * docs(sso): tighten the personal-provider note to the self-host path it describes * fix(sso): forward an empty SAML NameID format so the provider default can be restored The form sends an empty identifierFormat when the admin selects "Provider default", but the route dropped it with a truthiness check. Better Auth merges SAML config with `??`, so an omitted key retains the stored value — the selection appeared to apply and silently did not. Forwarding the empty string makes it reach the merge, and samlify falsy-guards nameIDFormat, so it reads as unset. Selecting the provider default now actually clears a stored override. * fix(sso): revoke trust for providers whose domain is spelled with a wildcard Migration 0268 grandfathered providers by normalizing their domain with lower + btrim + a stripped leading `*.`, so sso_provider.domain can hold `*.acme.com` while its verified sso_domain row holds `acme.com`. The revoke on domain deletion compared the raw column, so such a provider matched nothing and kept domainVerified after its ownership proof was gone. The comparison now applies the same normalization 0268 used, so a grandfathered row is matched the way it was written. * fix(db): give the SSO index migration the concurrent-build convention it skipped packages/db/scripts/migrate.ts documents the required shape for CONCURRENTLY statements, and the previous migration follows it. 0284 did not, and the omission is silently destructive. migrate.ts sets a session lock_timeout of 5s, which survives the embedded COMMIT. CREATE INDEX CONCURRENTLY waits on every concurrent write transaction in the database — not only ones touching this table — so on a busy database the build is cancelled with 55P03 and leaves an INVALID index. The retry then replays the file, IF NOT EXISTS skips the invalid index, and DROP INDEX removes the only working index on provider_id. The migration journals as applied and exits 0 with provider_id unindexed and uniqueness unenforced, reopening the cross-tenant provider resolution this migration exists to close. Adds SET lock_timeout = 0 around the concurrent statements, a pre-drop of the target index name so a replay rebuilds rather than skips, and restores the 5s timeout afterwards. Verified by stranding an INVALID index and replaying: the end state is a valid unique index with uniqueness enforced. Also corrects the sso() comment that claimed domainVerified confines linking to matching email domains. link-account.mjs blocks on `!isTrustedProvider && !userInfo.emailVerified`, so an IdP asserting email_verified links regardless of domain — the flag narrows nothing on its own. * fix(sso): give the Enter shortcut the same guard as the Add domain button The Enter handler called handleAdd unconditionally while the button was disabled during an in-flight add, so repeated presses could issue overlapping requests. Both now read one canAddDomain flag rather than duplicating the condition. * fix(sso): stop the provider ID being editable after it is saved Renaming it was never useful and always destructive. The value forms the redirect URL registered with the identity provider, so changing it breaks sign-in until the IdP is updated. Worse, the register route selects create-vs-update by (providerId, organizationId), so a renamed id misses and registers a SECOND provider; the settings page renders providers[0], so the duplicate is invisible, there is no delete action to remove it, and existing account rows still reference the old id. Editing now shows it as a read-only copyable value, and the create form says up front that it cannot be changed later. Also hoists the suggestion list to module scope — it was rebuilding 44 objects on every keystroke anywhere in the form. * fix(sso): stop persisting generated IdP metadata so SAML cert rotation works The route stored an IdP metadata document built from cert + entryPoint even when the admin supplied none. The form loads that document back into its optional metadata field and resends it, and on the next save it wins over the certificate — so rotating a SAML signing certificate through the form appeared to succeed and changed nothing. Only metadata the admin actually pasted is persisted now. With none stored, Better Auth's createIdP builds the IdP from issuer, entryPoint and cert, which are the fields the form edits. No SAML providers exist in production, so this changes no live tenant. * fix(sso): always write SAML IdP metadata so clearing it takes effect on update Not storing generated metadata fixed new providers but not existing ones: Better Auth merges SAML config with `??`, so omitting the key let a previously stored document survive and keep overriding the certificate. The key is now always written, empty when the admin supplied none. createIdP falsy-guards it and falls back to issuer/entryPoint/cert, so clearing the field actually clears it. * fix(sso): hold the domain proof under a row lock while granting trust Two fixes from review. The trust grant folded the ownership test into the UPDATE's WHERE clause, but under READ COMMITTED the EXISTS subquery is evaluated against the statement's original snapshot. A delete committing while the UPDATE waited on the provider row could therefore still see the removed sso_domain row and grant trust after ownership was gone. The grant now selects the proof FOR SHARE inside a transaction before writing, so the delete blocks until it commits, and if the delete committed first the select finds nothing and no trust is written. Editing a SAML provider also broke on configs written by the previous commit: hydration used `config.idpMetadata?.metadata || config.idpMetadata`, and `{ metadata: '' }` is falsy at the property but truthy as an object, so an object landed in a string field and failed validation on save. It now narrows on the type and handles both the object and legacy bare-string shapes. * refactor(sso): write the two merge-sensitive SAML fields the same way idpMetadata and identifierFormat both exist to defeat Better Auth's `??` merge, which silently keeps a stored value when a key is omitted, but they were written differently — one always, one only when defined. Both are now always written, empty when unset, under one comment explaining why and noting that each is falsy-guarded downstream. Also drops a redundant saveDisabled: false; the prop already defaults to false. * fix(sso): report the row the trust grant actually matched The grant returned true once it found the proof, without checking that the provider UPDATE matched anything, so its boolean did not always mean what callers read it to mean. It now reports the matched row. * fix(sso): restore provider domain trust when a domain is re-verified * fix(sso): correct the domain-removal warning now that it disables sign-in * chore(sso): trim verbose comments * fix(sso): revert a rejected SSO update instead of leaving it stored	2026-08-06 00:52:39 -07:00
Vikhyath Mondreti	c530d276b7	fix(env): flag combinations for sandboxes (#6308) ... * fix(env): flag combinations for sandboxes * more changes	2026-08-05 20:22:36 -07:00
Vikhyath MondretiandSiddharth Ganesan	117fe3137b	feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift, workflow exec cancellations (#6247) ... * feat(code): cli sandboxes, enterprise timeouts, secrets projections, resolver lift * fix(execution): harden compatibility and secret diagnostics * fix(execution): harden generated JavaScript literals * fix(execution): align timeout cleanup semantics * fix(tables): decouple stale job cleanup * fix(execution): drain stale workflow backlog * test(sandbox): make deadline assertions timing-safe * fix(execution): lock cleanup candidate batches * fix(execution): preserve cleanup failure metrics * cancel route fixes * separate out mship template and func template * fix * fix(execution): harden secret projection and block runs * fix(workflow): validate draft execution state * run from block ui disabling * feat(copilot): expose Sim sandboxes to mothership * feat(copilot): expose sandbox capability catalog in VFS * Updates * fix legacy logs showing up * fix(copilot): keep sandbox config visible * fix model provenance issues * fix lint' * more lint * more * test(files): align provenance copy query order * consolidate migrations, rollout compat * integration projections * update skills * fix * add provenance linters * fix: address review and compatibility regressions * fix: make tool boundary audit Bun 1.3 compatible --------- Co-authored-by: Siddharth Ganesan <siddharthganesan@gmail.com>	2026-08-05 19:22:04 -07:00
Waleed	dcaa118752	improvement(docs): restructure sidebar, align chrome, rename Mothership to Chat (#6296) ... Sidebar: 11 separator groups become 5, with each module a collapsible folder that auto-opens on the active page. 61 always-visible rows drop to 16. Groups mirror the app's own nav (Chats/Workspace/Workflows) rather than inventing a taxonomy; Enterprise and Self-Hosting are hoisted out of Platform. Chrome: register the `hover-hover` variant, without which every @sim/emcn hover state silently compiled to nothing; restore the sidebar's Geist font stack; add 11 emcn tokens that were falling back to currentColor; adopt the named type scale; align row geometry, hover tokens and group labels with the app. Rename: mothership/ -> chat/ with redirects for the old URLs. Asset paths, the @mothership.sim.ai domain and the `mothership` log-trigger enum value are deliberately left alone -- they are CDN objects, a real domain, and a live product value. Also removes the page-type badge, drops the "Next" heading from the ToC, and lets FAQ rows open independently so expanding one no longer shifts the page.	2026-08-05 13:49:17 -07:00
Waleed	a3a887ad5a	fix(docs): point the service-account guides at the real connect flow (#6277) ... * fix(docs): point the service-account guides at the real connect flow Fourteen of the twenty service-account guides sent admins to a workspace Settings → Integrations tab that does not exist — Integrations is a top-level workspace route, and there is no integrations section in the settings navigation at all. The same fourteen then told them to search the catalog for "<Service> Service Account", a name no catalog entry has: the list is derived from blocks, so the entries are "Airtable", "Monday", "Wealthbox", and search matches only name and description. Both steps now match the six guides that were already correct (Box, Zoho Desk, Zoom, Salesforce, Pipedrive, Atlassian), so all twenty describe one flow. The per-guide connect labels were already right and are untouched. Google needed a third fix: its final step said Click **Save**, but the modal's primary button is `Add {connectNoun}`, which for Google falls back to "Add service account". It also has no catalog entry of its own, so the search now points at Google Drive with a note that any Google integration works. Also adds `invalidCredentialsHelp` for Wealthbox. Its validator rejects a token that works only over Wealthbox's documented ACCESS_TOKEN header, because Sim's tools authenticate with Bearer — a deliberate, documented limitation whose reason reached the server log and never the user, who saw only "Double-check it". * fix(docs): make the Wealthbox rejection copy true for every failure path `invalidCredentialsHelp` replaces the generic message for every `invalid_credentials` rejection, and the Wealthbox validator raises that code on three paths: a 402 expired trial, a 401/403 where both header styles fail, and a 401/403 where Bearer fails but the ACCESS_TOKEN probe succeeds. The copy described only the third, so two of the three told the user their token was valid and pointed at remediation that could not help. Now leads with what is checkable in all three cases and makes the Bearer note conditional on the one signal that distinguishes it — the token working elsewhere.	2026-08-04 20:07:03 -07:00
Waleed	0ab44c5b44	improvement(zoho-desk): pick the data center from a dropdown and trim service-account help text (#6271) ... * improvement(zoho-desk): pick the data center from a dropdown and trim service-account help text The Zoho Desk Self Client modal rendered a paragraph of setup steps as the hint under Client secret, duplicating both the setup guide and two of its own field hints. Cut it to the one caveat that isn't derivable from the form, and moved it to the org-identifier field the caveats actually qualify. Data center is now a dropdown sourced from ZOHO_DESK_DATA_CENTERS. Same editorial pass across the other service accounts: Zoom, Salesforce, Shopify, Webflow, Trello and Cal.com dropped setup steps in favor of caveats. Also adds the documented Zoho Desk params that were missing (list_tickets assignee/channel/receivedInDays, list_comments and list_threads sortBy, get_contact and get_thread include), each gated per operation so a stale subBlock value can't leak into an endpoint that reads the same param name. * fix(zoho-desk): let an unsupported receivedInDays reach the tool's validation The block mapper filtered on shape before forwarding, so a fractional or non-numeric value was dropped and List Tickets then ran with no window at all — returning the whole queue as though the requested filter had applied. The tool owns that validation, so the mapper now passes the value straight through. Adds a block-to-tool seam test: neither side's own tests could catch a value lost between them. * fix(zoho-desk): overwrite operation-scoped params instead of omitting them The block mapper scoped params by destructuring them out of the spread, on the assumption that a key left out of the return value never reaches the tool. It does: both call sites merge the mapper's output on top of the original inputs (`{ ...inputs, ...transformedParams }`), so an omitted key is restored. The serializer is what actually held this together, and it has a gap — an advanced subBlock with a retained value is emitted for every operation while the block's advanced toggle is off, because that branch returns on isNonEmptyValue without evaluating the subBlock's condition. So a Sort By set on List Tickets reached List Comments, and a ticket Include reached Get Contact, each rejected by Zoho. Out-of-range from/limit reached the wire for the same reason. Every scoped param is now assigned unconditionally, undefined included, so the merge cannot resurrect a stale value. Also fixes a crash this branch introduced: clearing the Departments multi-select stores [], which reached the comma-list normalizer and threw on .split. The helper now takes arrays, which is what that subBlock actually stores. The block-to-tool tests now model the real merge rather than the mapper's return value alone — the previous version passed while production threw on the same input. Corrects two comments that misstated where Zoho documents customFields and errorMessage, and splits the shared include subBlock, since Get Ticket accepts contract and skills and List Tickets does not. * fix(zoho-desk): do not scope params on the agent-tool path The previous commit made the mapper assign every operation-scoped param unconditionally, so the merge could not resurrect a stale value. That is right on the canvas path and wrong on the agent-tool path, where `operation` is a sibling of the tool call rather than a member of params: the mapper saw `operation === undefined`, every gate resolved to undefined, and the merge then overwrote the model's own arguments with it. A Zoho Desk tool called by an agent lost every parameter the model supplied. That path needs no scoping — the tool is already chosen, and the model addresses tool params by their real names — so it now returns early. Custom fields are still coerced there, since parsing JSON is a type fix rather than an operation gate, and that parsing is now shared by both paths. * fix(zoho-desk): keep the legacy include working on Get Ticket Splitting the shared `include` subBlock into `include` and `ticketInclude` left workflows saved before the split reading an empty field, so their Get Ticket calls silently stopped embedding what they asked for. Get Ticket now reads `ticketInclude ?? include`. The fallback only goes that direction: Get Ticket accepts every value List Tickets does plus `contract` and `skills`, so a legacy value is always valid there, while List Tickets still reads only `include` and can never receive the two extra tokens it does not document.	2026-08-04 18:28:50 -07:00
Theodore Li	35fd4ef42f	improvement(self-host): simplify capability setup configuration (#6230) ... * feat(self-host): add capability-aware setup * fix(self-host): preserve capability compatibility * fix(copilot): honor preview availability server-side * improvement(self-host): centralize capability resolution * fix(self-host): preserve integration availability paths * fix(testing): align capability-aware config mocks * improvement(self-host): simplify capability setup configuration * fix(setup): preserve unowned storage overrides * fix(self-host): reconcile storage and allowlists * fix(integrations): preserve connect deep links	2026-08-04 15:47:36 -04:00
Emir KarabegandWaleed Latif	9b9da81a27	improvement(platform): drop lucide-react for the in-house icon set, flatten the type and border scales, and retire scheduled tasks and workflow references (#6241) ... * border styling * improvement(platform): migrate off lucide-react, flatten the font-weight scale, and retire scheduled tasks and workflow references * chore(platform): drop the dead schedule client layer and repair stale rule and skill docs Follow-up cleanup for the platform commit, which removed the workspace scheduled-tasks surface and migrated off lucide-react. Both left dead tails that type-check clean, so nothing flagged them. Six mutation hooks in hooks/queries/schedules.ts lost their only consumer when the scheduled-tasks page was deleted: useDisableSchedule, useResumeSchedule, useDeleteSchedule, useExcludeOccurrence, useUpdateSchedule, useCreateSchedule. They are removed along with the three contract objects that served only them — disableScheduleContract, excludeOccurrenceContract, deleteScheduleContract. disableScheduleBodySchema and excludeOccurrenceBodySchema are deliberately kept: both are members of scheduleUpdateSchema, the discriminated union the live PUT /api/schedules/[id] route parses. Dropping them would collapse the union and 400 the disable and exclude_occurrence actions. The schedule-calendar tree and its utils stay unmounted for later reuse. Its TSDoc now says so, since it has no importer and would otherwise read as dead code on the next sweep. The add-enrichment skill templated an import from lucide-react, a dependency the platform commit deleted, so running it produced an unresolvable import. It now points at @sim/emcn/icons, matching all five shipped enrichments. The emcn-design-review skill and several rule files still pointed at apps/sim/components/emcn/**, which moved to packages/emcn/**. Also corrects the documented Chip variant list — it advertised a ghost variant that never existed and omitted border — repoints the sim-url-state date-parser example at an inline snippet now that its source file is gone, and normalizes the one strokeWidth the icon migration left at 1.5 in bubble-chat-delay. * fix(platform): mark the resource chrome as client components `skills/page.tsx` is a Server Component, and this branch moved its `IntegrationTabsHeader` import onto the `@/app/workspace/[workspaceId]/components` barrel. That barrel re-exports `SortDropdown` from `resource-options`, which calls `useState`, so the server graph now reaches a client-only module and `next build` fails. `resource-header` has the same latent problem (`useState`, `useEffect`, `useRef`). Both files are genuinely client components, so they get the directive rather than the page dropping the barrel import — local feature barrels are the convention here. Also drops a stale `lucide-react` mention now that the dependency is gone. * chore(scheduled-tasks): remove the scheduled-task logic Scheduled tasks are retired. This removes the `sourceType = 'job'` half of `workflow_schedule` from the application, leaving the workflow Schedule trigger (`sourceType = 'workflow'`) untouched. Gone: - the job orchestration layer (`lib/workflows/schedules/orchestration.ts`) and the agent-job runner in `background/schedule-execution.ts` - the job claim/dispatch half of the schedules execute tick - POST /api/schedules (job creation) and the job branches of GET /api/schedules and PUT/DELETE /api/schedules/[id] - the copilot job tools and handlers, the `scheduledtask` resource type and chat-context kind, and the VFS `jobs/` materialization - the scheduled-task analytics events and the job variant of the schedule-disabled email Kept on purpose: `scheduled-tasks/components/schedule-calendar/**` and `scheduled-tasks/utils/**`, which the agents module will reuse. `packages/db/schema.ts` is deliberately untouched — the columns stay for now and come out in a follow-up with a proper expand/contract migration. The generated copilot catalog and VFS snapshot types are regenerated from the matching copilot PR, which removes the tools and the `jobs` snapshot field at the source. Verified: 23/23 type-check, biome, api-validation, production build, and the full vitest suite (18361 passing; the one failure in executor/handlers/pi/cloud-review-tools.test.ts predates this branch). * fix(sidebar): derive the settings and switcher widths from SIDEBAR_WIDTH This branch moved `SIDEBAR_WIDTH.DEFAULT` from 248 to 238 but left two hardcoded `248px` chrome widths behind, so both sat 10px wider than the live sidebar: - the workspace-switcher menu, which is meant to line up with the sidebar column it drops out of - the standalone settings sidebar, whose own comment says to keep it in step with the in-workspace chrome Both now read `SIDEBAR_WIDTH.DEFAULT` directly rather than repeating the number, so the next change to the constant cannot leave them stale again. * fix(schedules): stop the API accepting actions it no longer handles Adversarial pass on the scheduled-task removal found a real regression in PUT /api/schedules/[id]. Removing the job-only `update` and `exclude_occurrence` handlers left them in `scheduleUpdateSchema`, so those bodies still parsed. The handler chain is `disable` first and then an unguarded fall-through to reactivate, so an `action: 'update'` request would have silently REACTIVATED the schedule instead of being rejected. Both actions are dropped from the discriminated union, so `parseRequest` now rejects them with a 400. Their bodies, response types and the orphaned `createScheduleContract` (its POST route is gone, and nothing imported it) go with them. * chore(landing): retire the scheduled-tasks marketing surface The feature is gone from the product, so the marketing pages stop selling it. - deletes the `/scheduled-tasks` landing page and its calendar-loop hero, and the `LandingPreviewScheduledTasks` panel - drops the view from the landing preview: the `SidebarView` member, the nav entry and its now-unused Calendar icon, the callout label, both render branches, and the staged chat copy in `workflow-data` - removes the navbar and footer links and the sitemap entry - removes the route from `LANDING_ROUTES`, the COEP exemption list that must list every `app/(landing)` route `/scheduled-tasks` is indexed, so it 301s to `/workflows` rather than starting to 404 — that is the surface that still carries scheduled execution via the workflow Schedule trigger. Left alone deliberately: `demo-scheduler` is the Cal.com booking embed for the demo page, unrelated to this feature, and the scheduling library article is a generic SEO piece that never pitched it. * perf(chat): stop the resource picker fetching schedules it no longer shows Dropping the `scheduledtask` group from the add-resource dropdown left `useWorkspaceSchedules` behind, so the picker still issued a workspace schedules request whose result never reached a group. Worse than a wasted request: `schedulesPending` was still in the hydration gate, so the whole picker waited on that response before it could settle, and `schedules` was still a `useMemo` dependency, re-running the group build when it resolved. The hook and its route stay — `/api/schedules?workspaceId=` still correctly lists workflow schedules, unlike `createScheduleContract`, whose route this branch removed. * chore(scheduled-tasks): drop the leftovers the removal stranded An independent audit of the branch turned up dead code and stale docs that the compiler cannot see — nothing behavioural, but all of it rots silently. - README still sold the feature: the "Scheduled tasks" tile, the prose listing it as a workspace surface, and the now-unreferenced screenshot. The landing surface went in c61770a8c; this tile was missed. - `resource-content.tsx`: `SCHEDULE_STATUS_LABEL`, `formatScheduleInstant` and `ScheduledTaskField` were orphaned when the schedule render branch went. - `computeNextRunAt`: zero callers, including tests — its only consumer was the removed agent-job runner. - `applyScheduleUpdate`'s `allowCompleted` option: no call site passes it, and its comment described self-completion, which no longer exists. The guard stays (legacy `sourceType='job'` rows still carry `status='completed'` until the DB follow-up); it is simply unconditional now. - Three TSDoc blocks still described a create-job route and "opening a scheduled-task artifact". Type-check re-run with --force, since a cached turbo replay is not a check. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-04 10:28:00 -07:00
Waleed	3de63c94e3	feat(self-host): align Docker Compose with Helm and overhaul self-hosting docs (#6225) ... * feat(self-host): align Docker Compose with Helm and overhaul self-hosting docs Docker Compose shipped no scheduler, so scheduled workflows, every polling trigger, connector syncs, the outbox, and data drains silently never ran. Adds a cron service running the same 18 jobs the Helm chart schedules as CronJobs, and closes the remaining behavioral gaps between the two paths: bundled Redis in the chart, no hosted plan caps in chart defaults, pinned image tags, and fail-fast secrets. A CI check keeps the schedulers in sync. Also rewrites the self-hosting docs: 14 new pages, 8 updated, reorganized into Install / Configure / Operate. * fix(self-host): drop bun install from chart CI, remove air-gapped and backup docs The scheduler-parity check pulled a full dependency install into the chart-validation job, which fails building isolated-vm on that runner. Rewritten to use only node builtins so the job installs nothing. Also removes the air-gapped and backup/restore pages, and stops pinning a concrete release in the docs so the examples do not go stale each release. * fix(helm): bundle Redis in secret-manager modes unless the URL is supplied Suppressing Redis whenever a secret mode was active left those deployments with no Redis at all — REDIS_URL is optional there and both shipped examples omit it. The chart now steps aside only on a detectable signal: an explicit app.env.REDIS_URL, an ESO remoteRefs.app.REDIS_URL mapping, or the new redis.provideUrl=false opt-out for a pre-created Secret it cannot read. * fix(compose): derive realtime BETTER_AUTH_URL from NEXT_PUBLIC_APP_URL realtime read BETTER_AUTH_URL directly and fell back to localhost while simstudio derived it from NEXT_PUBLIC_APP_URL, so setting only the public origin left realtime authenticating against http://localhost:3000. * fix(helm): deliver bundled REDIS_URL via ConfigMap so an operator value always wins Injecting REDIS_URL as an inline container env made it beat every envFrom source, so a REDIS_URL held in a pre-created Secret or synced by External Secrets was silently shadowed and traffic moved to a fresh in-cluster Redis. Kubernetes resolves duplicate envFrom keys by letting the last source win, so the bundled URL now ships as a ConfigMap listed before the app Secret. Any operator-supplied value overrides it without the chart needing to read it, which also removes the redis.provideUrl flag the previous attempt required. * docs(helm): spell out the egress rule external datastores need The default NetworkPolicy allows 443 plus the bundled Postgres and Redis by pod selector. Anything you run outside the chart on another port needs its own rule, which is easiest to miss when REDIS_URL arrives via a Secret the chart cannot inspect. Adds a copyable example to the production checklist and the security guide. * feat(helm): add networkPolicy.allowExternalEgress for managed datastores The default policy allows 443 plus the bundled Postgres and Redis by pod selector, so a managed datastore on another port needs a hand-written CIDR rule — awkward when REDIS_URL arrives via a Secret the chart cannot inspect. Adds an opt-in switch that drops the port restriction while still blocking the cloud metadata endpoints. Defaults to false, keeping this chart stricter than the common chart default of unrestricted egress.	2026-08-03 14:52:48 -07:00
Vikhyath Mondreti	b8ec114382	improvement(chat): secrets mounting / exposure improvements and controls (#6191) ... * fix(copilot): secrets injection into sandbox * improvement(chat): secrets mounting / exposure improvements and controls * fix(secrets): simplify copilot mounting flow * test(secrets): preserve standard tool permissions * fix(copilot): bind workflow tool completions * fix(secrets): preserve own environment keys * fix(copilot): release failed workflow claims * fix(copilot): trust compacted workflow completion	2026-08-02 10:21:27 -07:00
Waleed	25e609167f	fix(search): disambiguate tables and knowledge bases by folder (#6192) ... * fix(search): disambiguate tables and knowledge bases by folder The Cmd-K search modal listed tables and knowledge bases without the folder breadcrumb workflows and files already showed, and the table, knowledge base, and search-and-replace pickers in the workflow editor rendered bare names -- so two resources sharing a name in different folders were indistinguishable. Extracts the disambiguation the workflow selector already did into shared collectDuplicateNames + disambiguateLabelByFolder, and shares the search row's folder breadcrumb and its memo comparator, which were duplicated between the workflow and file rows. Also routes folder text through filterAndCap's secondary-rank parameter rather than concatenating it into the name, so an exact name match can no longer be outranked by a folder that happens to fuzzy-match. * fix(zoho-desk): use the real Zoho Desk mark on a white tile The icon was a generic headset placeholder drawn in currentColor, so it never resembled Zoho at all. Replaces it with the mark from Zoho's official logo -- wordmark stripped, viewBox set to the mark's own bounding box so it centers -- and moves the tile to white, matching the other brand-mark integrations.	2026-08-02 00:04:35 -07:00
mzxchandraandWaleed Latif	87aeca6f0c	feat(zoho-desk): add Zoho Desk integration (#6157) ... * feat(zoho-desk): add Zoho Desk integration Add a full Zoho Desk integration: tools, block, icon, and a webhook trigger. Tools (tools/zoho_desk): list/get/update tickets, list/add comments, list/get threads, get contact, list organizations, and download attachments as UserFiles via an internal route. Registered in tools/registry.ts. Block (blocks/blocks/zoho-desk.ts): operation dropdown, OAuth credential, an organization selector backed by GET /organizations, per-operation fields, and BlockMeta templates. Wires the Zoho Desk trigger. OAuth (zoho-desk provider): authorize/token at accounts.zoho.com with access_type=offline + prompt=consent; the Desk REST base is derived from the token response api_domain and persisted so calls honor data residency instead of assuming desk.zoho.com. Every call sends Authorization: Zoho-oauthtoken and the orgId header. Trigger + webhook handler (triggers/zoho_desk, lib/webhooks/providers/zoho-desk.ts): Sim creates and tears down the Zoho Desk webhook subscription. Inbound events are verified with JWT RS256 (X-ZDesk-JWT) against the data-center JWKS, ACKed via the durable queue to meet Zoho's 5s deadline, and fail loudly on Free/Standard editions that cannot create webhooks. * fix(zoho-desk): OAuth PKCE, DC scope-marker parsing, SSRF, and e2e fixes OAuth: forward code_verifier in the custom getToken (PKCE is enabled, so the exchange must echo the verifier or Zoho rejects the request with invalid_request). Surface Zoho's error/error_description, which it returns in the JSON body with HTTP 200, instead of collapsing every failure into "no access token". Data-center base parsing: better-auth persists Zoho's scopes comma-joined with no spaces, so the greedy \S+ marker regex swallowed the whole scope list into the host. Stop the capture at a comma or whitespace in both read sites (token route and webhook handler), so apiDomain resolves to the real Desk host. Attachment SSRF: replace the permissive host regex (which accepted attacker domains like zoho.attacker.com) with a strict Zoho-apex suffix allowlist. Block: guard Number() pagination so a non-numeric typo can't send NaN; add the ignoreSourceId -> sourceId loop-guard header to update_ticket (matching add_comment). Organizations route: surface fetch/Zoho failures with a real status instead of a 200 with an empty list, so the org selector no longer fails silently. * fix(zoho-desk): webhook creation, attachment naming, and HTML content handling Webhook trigger (verified end-to-end against a live Enterprise org): - Omit ignoreSourceId; Zoho rejects a non-Zoho UUID with INVALID_DATA. Drop the generateId() fallback and its providerConfig persistence. - Answer Zoho's create-time notification-URL probe via the existing pending webhook verification mechanism (GET/HEAD matchers) so subscription creation no longer 405s. - mapZohoWebhookError now surfaces Zoho's real errorCode / message / field errors instead of a catch-all edition message, and attaches an HTTP status so 4xx flow through NonRetryableDeploymentError while 429/5xx stay retryable. - Propagate the real status through deploy.ts so failed creates don't retry-loop. get_attachment polish: - Return the downloaded file's name under `name` (ToolFileData key) instead of `filename`, and derive it (explicit -> Content-Disposition -> URL segment -> fallback) so attachments are no longer stored as "untitled". - Gate the add_comment-only `contentType` param so it isn't sent to get_attachment. HTML content handling (Zoho content fields emit raw HTML): - Add a Zoho-local html-to-text converter mirroring the Outlook dual-field pattern: when contentType is 'html', derive a plain-text `contentText` alongside the untouched raw `content` + `contentType`; plainText mirrors. - Apply to comments (list/add), threads (list/get), the ticket description (descriptionText), and the webhook trigger payload. Trigger org selector: Organization is now a credential-scoped combobox that lists the connected account's Zoho Desk organizations. * fix(zoho-desk): review round - DC-base derivation, org-loader resilience, batched-event visibility - deriveZohoDeskBaseFromApiDomain: preserve an already-regional desk.zoho.<tld> api_domain instead of falling back to the US (.com) data center, and map the DC TLD from any zoho(apis).<tld> host - keeps Desk calls in the right data center for residency. - fetchZohoDeskOrganizationOptions: wrap the token/org fetch in try/catch and degrade to an empty list (the org field is a free-text combobox, so manual entry still works) instead of hard-failing the selector on token/DC/network errors. - formatInput: warn (not silently drop) if Zoho ever delivers more than one event in a single payload. * fix(zoho-desk): harden attachment download against redirect-based SSRF/token leak Replace the raw fetch in the attachment route with secureFetchWithValidation (the same guarded fetch the copilot file-download tool uses). The download URL is user/LLM-influenced and Zoho may redirect, so auto-following redirects could send the OAuth token / orgId to an untrusted or internal host. The guarded fetch pins the resolved IP, blocks private/reserved targets on every hop, drops the Authorization header if a redirect leaves the origin (stripAuthOnRedirect), and enforces the 50MB cap while streaming. The strict Zoho apex allowlist still gates the initial origin as defense in depth. * fix(zoho-desk): only add the edition hint when Zoho's error indicates it mapZohoWebhookError appended the "requires Professional edition or higher" guidance to every 403, but a 403 can also mean a wrong org, a missing scope, or a bad token. Gate the hint on Zoho's own errorCode / message matching the permission/edition pattern instead of the bare status, so unrelated 403s surface Zoho's real reason without the misleading suffix. Adds a test for the non-edition 403 path. * fix(zoho-desk): stop duplicating /api/v1 when resolving a relative attachment href A relative attachment href that already starts with `api/v1` (as Zoho's hrefs often do) was concatenated onto getZohoDeskApiBase (which ends in /api/v1), producing `/api/v1/api/v1/...` and a failing download. Extract a tested resolveZohoAttachmentUrl helper that uses absolute hrefs as-is and strips a leading slash + `api/v1/` prefix from relative ones before joining, so the path is correct for absolute, root-relative, and api/v1-prefixed hrefs alike. * fix(zoho-desk): reject an empty update_ticket PATCH with a clear error update_ticket built its PATCH body from optional fields via filterUndefined, so a call with no fields set sent `{}` and surfaced an opaque Zoho failure. Guard the body builder to throw an actionable "provide at least one field" error before the request. Adds a test for the empty and populated body paths. * fix(zoho-desk): fall back to the credential Desk domain in webhook JWT verify verifyAuth chose the JWKS host from providerConfig.apiDomain and otherwise defaulted to the US host (desk.zoho.com), so a non-US webhook row missing apiDomain would verify against the wrong JWKS and reject legitimate events. When apiDomain is absent, resolve it from the OAuth credential's __zoho_domain__ scope marker (mirroring deleteSubscription). The persisted-apiDomain fast path stays DB-free to respect the 5s delivery deadline. Adds tests for both paths. * fix(zoho-desk): apply the Zoho host allowlist to the organizations route The organizations route built its URL from the client-supplied apiDomain and attached the OAuth token without the https-Zoho-host allowlist the attachment route already enforced, so a session-access caller could point the server at an arbitrary origin and leak the token. Extract the shared isZohoHost allowlist and an assertZohoUrl guard into tools/zoho_desk/utils (two consumers now), guard the organizations URL before fetching, and refactor the attachment route to reuse the shared helper. Adds tests for the allowlist and guard. * fix(zoho-desk): propagate provider 4xx in the stable webhook prepare path The v2 stable deploy preparation flattened every registration failure (except path conflicts) to HTTP 500, so a provider-attached permanent 4xx - e.g. Zoho's edition/validation failures from createSubscription - retried instead of failing the deploy terminally. Propagate the attached status (`?? 500`), matching the legacy save path's status-aware mapping so both deploy paths route 4xx through NonRetryableDeploymentError. * fix(zoho-desk): make createSubscription config failures non-retryable createSubscription threw plain Errors (no status) for missing orgId, event type, or credentials, and for a Zoho success with no webhook id - so the deploy outbox mapped them to 500 and retried permanent configuration failures. Attach a 4xx via statusError (400 for missing config/credentials; 422 for the no-id anomaly, where a retry risks duplicate webhooks) so they fail the deploy terminally like the mapped Zoho API 4xx responses. Tests assert the 400 status on the guard paths. * fix(zoho-desk): enrich prevState with contentText symmetrically with payload formatInput derived plain-text contentText only on payload, so an update event for a comment/thread left prevState as raw HTML while payload carried contentText - inconsistent shapes for before/after comparisons. Apply withDerivedContentText to prevState too. Test asserts both are enriched. * docs(zoho-desk): regenerate integration docs Regenerate zoho_desk.mdx from the current tool definitions: removes the stale add_comment `ignoreSourceId` input row (the field was dropped because Zoho rejects arbitrary values) and adds the derived `contentText` / `descriptionText` plain-text fields on comments, threads, and tickets. * fix(zoho-desk): validate the persisted Desk base against the strict host allowlist deriveZohoDeskBaseFromApiDomain trusted any host matching `desk.zoho.[a-z.]+`, so a crafted api_domain like `desk.zoho.com.attacker.com` passed and was persisted as the credential's `__zoho_domain__` REST base - later receiving the OAuth token on every Desk tool/webhook call. Gate the derivation on the strict isZohoHost apex allowlist (which rejects that lookalike), extracted with assertZohoUrl into a dependency-free host-allowlist module so the auth token-exchange path validates hosts without pulling in the tool utilities. The attachment and organizations routes now import the shared guard from there. Also: formatInput now emits the normalized null trigger shape for an empty/ malformed event array instead of leaking a raw `[]` to downstream steps. Tests cover the empty-array shape and the lookalike-host rejection. * fix(zoho-desk): correct API field names, scopes, and host validation Validation pass against Zoho's published Desk API surfaced six defects that typecheck, lint, and the existing suite all passed over, because each one fails silently against the live API rather than erroring. Wire-name mismatches (Zoho ignores unknown keys, so all three were silent): - update_ticket sent `customFields`; the ticket PATCH body names it `cf`. `customFields` exists only as a deprecated alias on other Desk resources and on the separate validate-field-updates endpoint, so updates reported success and applied nothing. - ZOHO_DESK_TICKET_PROPERTIES and ZOHO_DESK_CONTACT_PROPERTIES advertised a `customFields` output; both resources return `cf`. The declared field always resolved undefined and the real one was undeclared. - list_tickets sent `departmentId`; the query param is `departmentIds`, so the department filter was dropped and every department's tickets came back. Content handling: - deriveZohoContentText matched `contentType === 'html'`, but Zoho spells the discriminator per resource: comments use `html`, threads use the MIME form `text/html`. Every thread's `contentText` was therefore raw markup - the exact opposite of the field's purpose. Now normalized across both spellings, parameterized values, and casing, with regression tests. Scopes (least privilege): - Desk.tickets.ALL -> Desk.tickets.READ + Desk.tickets.UPDATE. No tool creates or deletes a ticket; ALL additionally granted ticket DELETE. - Dropped Desk.search.READ (no search tool exists) and Desk.webhooks.READ / .UPDATE (the provider only creates and deletes), plus their orphaned SCOPE_DESCRIPTIONS entries. Host validation - the webhook provider was the only token-carrying path not anchored to the Zoho apex allowlist, including the JWKS fetch, where an unrecognized host would have stood in as the JWT issuer: - createSubscription, deleteSubscription, and verifyAuth now route their base through a shared allowlist check. - getZohoDeskApiBase validates rather than trusting injection precedence. - The organizations route uses secureFetchWithValidation with stripAuthOnRedirect, matching the attachment route it had diverged from. Block and trigger: - The trigger's department field is renamed `triggerDepartmentIds`; sharing the `departmentIds` id let a value typed as a list_tickets filter become the webhook subscription's filter when switching modes. - `isPublic` no longer serializes onto all ten operations, matching the existing gating for `contentType`. - from/limit reject negatives and fractions instead of forwarding them. - update_ticket gains description, resolution, and classification (all already declared as outputs), and a departmentId input so a ticket can be moved. Accuracy corrections to user-facing text, all against the published parameter tables: `from` is 0-based (0-4999, default 0), not 1-based; per-endpoint limits are tickets 1-100/10, comments 1-100/50, threads 1-200/100; sortBy lists Zoho's actual allowed values; the two `include` sets genuinely differ per endpoint; status and priority accept comma-separated lists. Also: path IDs are trimmed via requireZohoDeskId so a pasted trailing space fails with a clear message instead of a %20 404; comment `commenter` and thread `status`/`isDescriptionThread`/`visibility`/`canReply` are now declared; ZOHO_CLIENT_ID/SECRET added to the oauth test env; docs page gains a MANUAL-CONTENT intro covering capabilities, the Professional-edition webhook requirement, and the US-data-center limitation. Not verified from documentation, needs a live account before merge: - the OAuth scope for the attachment content sub-path (Zoho publishes none, and there is an unanswered SCOPE_MISMATCH report against it) - 12 of the 17 offered webhook event ids (5 are confirmed); Ticket_Delete is documented but not offered - the ticket `descriptionContentType` key, and the POST /api/v1/webhooks body shape, neither of which appears in any reachable Zoho reference * chore(zoho-desk): regenerate tool metadata The param and description corrections in the previous commit changed the generated tool surface, so tool-metadata:check failed in CI. Regenerated; the diff is two Zoho-only lines. * fix(zoho-desk): stop posting null for untouched update_ticket fields `filterUndefined` strips only `undefined`, but an untouched subBlock never arrives as `undefined`: the workflow serializer initializes every subBlock value to `null` (stores/workflows/utils.ts) and extractBlockParams writes those nulls straight into tool params, with nothing between the serializer and request.body filtering them. Reproduced against the real serializer and block with only `status` set: basic {"subject":null,"status":"Closed"} advanced {"subject":null,"status":"Closed","priority":null,...,"cf":null} `subject` leaks even in basic mode because it declares no `mode`, so shouldSerializeSubBlock never drops it. Zoho documents subject as a writable field, so every status-only edit either failed the PATCH or blanked the ticket's subject; in advanced mode the whole update surface nulled out, including `cf`. Two things hid this. The empty-PATCH guard was unreachable from the block (the body always carried at least `subject`), and the existing test called buildBody with fields *absent* rather than null - the shape the block never produces - so it could not fail on the real path. Replaces filterUndefined with a local omitUnset that drops undefined, null, and '' (a cleared input means "leave unchanged", not "set to empty"). Adds three tests using the real serializer shape, all verified to fail before the fix. Also fixes the same null-blindness in the block's param mapping, where Number(null) === 0 injected from=0 on every operation, and corrects the shared limit placeholder, which claimed max 100 while list_threads allows 200. * feat(zoho-desk): add Self Client service-account credential Adds a second way to connect Zoho Desk, alongside the interactive OAuth flow: a Zoho Self Client, pasted as client id + client secret + organization id. Built on the existing client-credential-accounts framework rather than a new credential path, so it behaves like the Zoom Server-to-Server and Box CCG accounts already in the repo - a short-lived token minted on demand, no refresh token. Two Zoho behaviors the generic framework does not cover: - `scope` must be COMMA-separated on Zoho's token endpoint; a space-separated list is rejected as an invalid scope. The list comes from getCanonicalScopesForProvider('zoho-desk'), so the Self Client and the OAuth flow can never drift apart on scopes. - Zoho reports OAuth failures in the JSON body, frequently with HTTP 200 (e.g. {"error":"invalid_client"}), so the success body is inspected for an `error` field before the token is read - a status-only check would accept a failed mint. deriveZohoDeskBaseFromApiDomain moves out of auth.ts into the dependency-free host-allowlist module so the minter and the OAuth path share one derivation instead of duplicating it, and the mint response's api_domain now flows through to tools as `apiDomain` (the SA branch of the token route previously returned none, so SA calls would have assumed desk.zoho.com). Docs: hand-authored zoho-desk-service-account.mdx following the existing *-service-account.mdx pages, registered in meta.json and in the generator's keep-list so stale-page cleanup does not delete it. Known limitation, documented in the descriptor helpText and the docs page: webhook triggers still require an OAuth connection. Webhook provisioning resolves credentials through getCredentialOwner/refreshAccessTokenIfNeeded, which is OAuth-account-only for every provider in the repo - not a Zoho-specific gap. Unverified from documentation, needs a live Zoho org before merge: - the `ZohoDesk.` soid prefix. Zoho documents only the syntax {servicename}.{zsoid} with a single CRM example; no first-party doc states the Desk prefix. normalizeZohoDeskSoid passes through any value already containing a '.', so an operator can paste a corrected full soid without a code change. - whether zsoid is the same identifier as the Desk orgId header value. - whether the client-credentials endpoint accepts Desk.webhooks.CREATE/DELETE for a Self Client. - whether the mint response populates api_domain for Desk (documented for CRM); if absent the derivation falls back to the US Desk host. * fix(zoho-desk): derive descriptionText for ticket-shaped payloads Cursor Bugbot: webhook ticket events reached workflows as raw HTML with no plain-text sibling. `withDerivedContentText` only looked at `content` / `contentType`, but ticket resources carry their body on `description` / `descriptionContentType`, so trigger output disagreed with get_ticket. The helper now derives both, which also removed two inconsistencies on the tool side: get_ticket had its own inline copy of the derivation (now one shared implementation that cannot drift), and update_ticket returned its PATCH response raw despite the shared output map declaring descriptionText. `descriptionContentType` remains the one field name unconfirmed in any Zoho reference. It degrades safely - an absent key makes deriveZohoContentText return the value unchanged, so descriptionText mirrors description rather than breaking, exactly as get_ticket already behaved - and it is now one helper to correct if Zoho names it differently. * feat(zoho-desk): let the service account pick its data center Zoho's accounts server is per region, and the integration pinned every call to the US host. For the interactive OAuth flow that is currently unavoidable - better-auth's authorize/token URLs are static per provider - but the service account mints its own token, so the region can simply be chosen. This makes the Self Client the only way a non-US Zoho org can connect. Adds an optional `dataCenter` field to the client-credential framework. Optional matters: ClientCredentialAccountFieldId and ClientCredentialAccountFields are shared with Zoom, Box and Salesforce, whose descriptors and minters are unchanged. Blank keeps the previous behavior (US), so existing credentials are unaffected. Only us/eu/in/au are offered - the four regions where both the accounts server and the Desk REST host are confirmed. CA is deliberately absent: Zoho's accounts docs say accounts.zohocloud.ca while Zoho's own Desk SDK says accounts.zoho.ca, and the two cannot both be right. JP/SA/CN/UK lack a confirmed Desk host. The Desk base is now derived from the selected region rather than inferred from the mint response, which also removes a dependency on `api_domain` being populated for Desk (Zoho documents it for CRM only). When `api_domain` IS present and disagrees with the region, it wins - it is authoritative about where the token actually works - and the mismatch is logged so a mis-selected region is diagnosable. deriveZohoDeskBaseFromApiDomain gains a `try` variant returning undefined so an untrusted api_domain can no longer masquerade as an authoritative US answer and silently override a correct region. A wrong region fails loudly rather than silently: the minter runs as verification on both create and reconnect, so the credential is never persisted in a broken state. Because Zoho reports it as `invalid_client` - a Self Client only exists on its own region's accounts server - the operator hint for that code now names the data center as a candidate cause. Copy is scoped per path rather than blanket "US only": the OAuth service description, trigger setup instructions, and the docs intro now say which path each limitation applies to, and the service-account page documents the four regions with a sign-in-domain to region-code table. * fix(zoho-desk): strip ticket description HTML, classify body-reported refresh failures Final validation pass findings. descriptionText never stripped anything. It was gated on a `descriptionContentType` discriminator that Zoho does not send: the Ticket_Add webhook sample ships `"description": "<div>Description</div>"` with no such key, and the ticket GET/PATCH response field lists have no content-type sibling either. So get_ticket, update_ticket, and every webhook ticket payload emitted descriptionText as a byte-identical copy of the raw HTML, while the declared output promised stripped text. The tests did not catch it because they fabricated the shape - both fixtures constructed `descriptionContentType: 'html'`, a key Zoho never emits, proving the branch works without proving it is ever taken. Ticket descriptions are HTML by convention, so the strip is now unconditional (html-to-text is a near-identity on genuinely plain text), an explicit descriptionContentType is still honored if Zoho ever adds one, and the fixtures now use Zoho's real shape with no content-type key anywhere. A body-reported refresh failure was unclassified. Zoho answers a revoked refresh token with HTTP 200 and `{"error":"invalid_client"}`; refreshOAuthToken only checked `data.ok === false` (a Slack-ism), so the request fell through to the "no access token" guard and returned no errorCode. isTerminalRefreshError could therefore never recognize invalid_client as terminal, the credential was never marked dead, and every later execution retried a refresh that cannot succeed - with the user shown "No access token in refresh response" instead of a reconnect prompt. The body is now classified before the status is trusted, matching what the token exchange and the service-account mint already did. That guard also stopped logging the whole response body, which carries live tokens on a partial success. Also: an unrecognized dataCenter now fails with a named error instead of quietly resolving to US and surfacing as an opaque invalid_client (blank still means US); the webhook JWKS cache is bounded, since its key derives from a providerConfig field that SYSTEM_MANAGED_FIELDS protects from diffing but not from being written; and the attachment `size` output no longer asserts bytes, a unit Zoho documents as KB. * feat(zoho-desk): canonical selectors and BlockMeta skills The block picked its organization with an ad-hoc `combobox` + `fetchOptions`. Only five blocks in the repo did that, and the other four are core blocks (agent/credential/function/logs) - no other OAuth integration used it. Every other resource a user has to identify was a bare short-input taking an opaque numeric id. Zoho Desk now uses the same machinery as the other 25 selector providers: hooks/selectors/providers/zoho-desk/selectors.ts registered in the selector registry, consumed from the block as basic selector + advanced manual input sharing one canonicalParamId, for organization, update-ticket department, and the list-tickets department filter. The trigger's org field moves to the same selector. zoho-desk-org-options.ts is deleted rather than left beside the new path, so blocks/ has zero fetchOptions usages outside the core blocks. Wire params are unchanged (orgId, departmentId, departmentIds, assigneeId, ticketId, contactId) - this is a UI change, not an API change. The organizations route now resolves the credential server-side. It previously had the browser fetch an access token and POST it back, which an earlier audit flagged as the one place a Zoho token left the server; the new selector-credential resolver keeps it server-side for both the OAuth and service-account credential types and re-anchors every outbound host to the Zoho apex allowlist. No agents selector: the endpoint is documented but its OAuth scope is not, and the nearest evidence points at Desk.agents.READ, which we do not request. Adding it would force every existing Zoho Desk user to reconnect for a convenience field, so assigneeId stays a manual input until the scope can be confirmed against a live org. Adds the skills array BlockMeta was missing - 227 of 300 blocks declare one and this did not. Seven skills, each grounded in a use case Zoho or the ecosystem actually advertises (auto-triage, SLA escalation, digest, AI draft reply, customer context, engineering handoff, knowledge-gap report) and each exercising only tools in tools.access. CSAT surveys, ticket creation, dedup and keyword search were deliberately left out: the integration has no tool for them, and a skill implying an unsupported action is worse than a shorter list. * feat(zoho-desk): agents selector and free-text trigger organization Three improvements that were previously deferred only to avoid forcing existing users to reconnect or orphaning saved workflows. This integration is unmerged and has no users, so the constraint does not apply and the better option wins. assigneeId was the last field still asking for an opaque numeric id. It is now a canonical selector pair backed by a new zoho_desk.agents selector, which required adding the Desk.agents.READ scope - the reason it was skipped before. Route follows the departments one exactly: auth before parseRequest, host anchored to the Zoho apex allowlist, secureFetchWithValidation with stripAuthOnRedirect, and a page drain capped at 20 pages with 204 treated as end-of-list. Scope caveat: Zoho publishes no explicit scope line for the list-all GET /api/v1/agents. Every other endpoint in the Agents module documents Desk.agents.READ (get by id, get by email, roles/{id}/agents), and it is the only agents-module scope Zoho defines, so that is the basis. Inference across a module rather than a direct quote - worth one live call before merge, same as the existing attachment-scope note. The trigger regained free-text organization entry, lost when the org field became a selector. The earlier concern - that a manual value would land under its raw subBlock id and never reach the provider - turned out not to hold: buildProviderConfig already collapses canonical pairs and writes the active member under the canonical key. The real gap is narrower and does exist: when canonicalModes pins the group to basic while only the manual field has a value, the collapse deletes the canonical key even though the required-field check passes, so the deploy succeeds and then fails at subscription time. resolveConfigOrgId closes that, with a test. The block/trigger `orgId` id overlap stays shared, now with a comment. Two earlier audits disagreed; renaming turns out to be the wrong call. buildCanonicalIndex has an explicit guard for trigger-mode reuse and blocks.test.ts codifies it as a valid pattern, orgId means the same portal in both modes (unlike departmentIds, which is correctly distinct), and a separate triggerManualOrgId would put two advanced members in one canonical group - getCanonicalValues takes the first non-empty, so a stale tool-mode value could silently supply the trigger's organization. * fix(zoho-desk): make the attachment cap reachable, unbreak selector paging Final audit round. The 50 MB attachment ceiling could never be hit. This route returns the file as base64 inside its JSON body, and the executor reads internal tool responses through readToolResponseBody, capped at 10 MB. Base64 inflates 4/3, so ~7.5 MB of raw bytes is the real ceiling - and the old limit meant a larger attachment was downloaded, encoded and serialized in full (peaking near 250 MB of live allocation, with nothing bounding concurrent downloads) purely to be rejected afterwards. The cap is now the reachable size, so the limit enforces itself while the bytes are still streaming, and an overflow returns 413 with the actual ceiling instead of a generic 500. Raising it properly means uploading in the route and returning a file reference, as the WhatsApp media route does - not a bigger constant. Selector paging assumed a 0-based `from`. Zoho's docs contradict themselves: the pagination section says "range 0-4999, default 0" while the listing examples read as 1-based ("from=5 and limit=50 retrieves records 5 to 54"). Under the 1-based reading, stepping by exactly the page size re-fetches the boundary record and the dropdown shows a duplicate per page. Rather than pick a base that cannot be confirmed without a live tenant, the department and agent drains dedupe by id, which is correct under either reading. The organization list was unpaginated, and Zoho's listing APIs default to ten per page. An account with more accessible portals silently got a truncated dropdown, and since every other selector and every tool call is gated on orgId, a missing portal was unreachable except through the advanced manual field. Both the selector route and list_organizations now request the documented maximum. Docs: regenerated so the trigger table includes manualOrgId, and two service-account claims are hedged to match what the code already says it cannot verify - that zsoid equals the Desk orgId header value, and that every tool works under the requested scopes (Zoho publishes no scope for the attachment content sub-path). Also: status and priority move out of advanced mode - they are the fields most often changed on a ticket update; the custom-fields wand prompt now ends with the required "Return ONLY" clause; and the shared-orgId rationale comment cites the mechanism that actually applies (buildCanonicalIndex dedupe plus the first-non- empty rule in getCanonicalValues) rather than a blocks.test.ts branch that never evaluates this pair. * fix(zoho-desk): five-audit round - serializer trigger-advanced leak, scopes, paging Five independent audits (OAuth/scopes, tools-vs-docs, block/selectors, blast-radius, /validate-trigger). Findings, most severe first. A trigger-mode field was a live tool-mode required param. `shouldSerializeSubBlock` excluded `mode: 'trigger'` but not `'trigger-advanced'`, so the trigger's required `manualOrgId` validated on every tool operation. Reproduced against the real serializer: with the Organization field pinned to advanced, running List Organizations failed with "Missing required fields: Organization ID" - a field that operation does not even render, and which the user could not clear without switching operations. Fixed in the serializer rather than locally, because the Google Sheets/Drive/Calendar pollers have the identical shape. `limit=200` on /organizations was an undocumented parameter I added by extrapolating from /departments and /agents. Zoho documents NO parameters for that endpoint and its sample is a bare GET; the other siblings cap at 100 and Zoho answers out-of-range with 422. Since orgId gates every tool and both other selectors, a 422 there would have made the whole integration unreachable. Reverted to Zoho's documented shape. `descriptionText` was HTML-stripping plain text. The previous round made the strip unconditional after finding Zoho sends no `descriptionContentType`, but Zoho's REST samples show plain descriptions while only the webhook payload is HTML - and the webhook path runs this over contact/account/department bodies too. html-to-text is not identity on plain text: it decodes entities and deletes tag-shaped content ("a < b > c", XML snippets). Now sniffs for markup first. `omitUnset` made every documented field-clear impossible. Zoho's own PATCH sample uses `"classification": ""` and `"productId": ""` to clear. Dropping `''` meant no scalar field could be cleared. Now drops only undefined/null - the serializer-null case it was written for - and forwards `''`. status/priority leaked between operations. One shared subBlock served both the list_tickets filter and the update_ticket value, and subBlock values survive an operation switch, so a filter of "Open,On Hold" could be PATCHed onto a ticket and an update value could silently filter a later list. Split per operation. Auth: `invalid_code` added to TERMINAL_ERRORS - it is Zoho's code for a revoked refresh token, so without it the previous round's refresh fix never actually dead-flagged the credential it was written for. The shared refresh body-error branch now also requires `!data.access_token`, so no provider can have a successful refresh misclassified. The token route now uses the validating `extractZohoDeskBaseFromScope` instead of a private regex with no https/allowlist check - that value is injected into every tool call. Scope list falls back to the requested scopes when Zoho omits `scope`, which would otherwise flag every credential as needing reconnect. The Self Client mint no longer sends `aaaserver.profile.READ`, a scope that grant never uses. Trigger: `includePrevState` now set for every *_Update event, not just tickets - it defaults to false, so prevState was permanently null for contact/agent/task/ article updates while the trigger advertised it. `departmentIds` is only sent for events Zoho documents as accepting it, and the field is conditioned accordingly. Empty filters serialize as `null`, matching Zoho's examples, rather than `{}`. JWKS fetch bounded to 1.5s - jose's default is 5000ms, exactly Zoho's whole delivery deadline, and Zoho publishes no retry. The create-time validation POST fallback is now matched by the pending-verification probe. Ticket_Delete added. All 17 webhook event ids, the POST /api/v1/webhooks body contract, and the JWT claim/JWKS specifics are now confirmed verbatim against Zoho's webhook documentation - previously 12 of 17 events and the entire subscription contract were unverified. * revert(zoho-desk): back out both shared lib/oauth changes Reverting two changes to shared OAuth code because their premise is inferred rather than proven, and neither meets the bar for touching a path every provider runs. `refreshOAuthToken` body-error branch. The premise was that Zoho reports refresh failures with HTTP 200 and an `error` body. That is documented and empirically confirmed for the authorization-code EXCHANGE (see the comment on getToken in auth.ts), but I never confirmed it for the REFRESH grant specifically - and if Zoho returns a proper 4xx there, the existing `!response.ok` path already classifies it via extractErrorCode, making the branch dead code that every one of the ~34 providers still executes on each refresh. A shared branch whose only justification is an unverified inference about one provider is not worth its blast radius. `invalid_code` in TERMINAL_ERRORS. Same problem, worse downside: the code is sourced from a Zoho community post rather than official docs, TERMINAL_ERRORS is consulted for every provider, and a false positive marks a credential dead for an hour. Not adding it simply preserves today's behavior (retry rather than dead-flag), so reverting costs nothing that was previously working. Both are cheap to reinstate, correctly scoped, once a live Zoho account shows what a revoked refresh token actually returns. Kept: the token-redaction on the "no access token" warn, which is an unambiguous improvement independent of Zoho. Also kept, deliberately, is the serializer `trigger-advanced` exclusion - that one rests on a reproduced bug rather than an inference, and it aligns the serializer with the convention the rest of the codebase already follows (blocks.test.ts treats `trigger` and `trigger-advanced` identically in six places, as does the copilot block-metadata tool, and blocks/types.ts documents trigger-advanced as "the advanced side of a trigger field"). * fix(zoho-desk): carry the stored data center through a credential reconnect A reconnect rebuilds the service-account secret blob from the submitted fields only, and the connect modal never prefills - correctly, since for every other field in this family the stored value is a secret the admin must retype. The data center is the first non-secret member of that set, so it was being silently dropped: rotating a client secret on an EU/IN/AU credential moved it back to the US accounts server, where the next mint fails with an opaque invalid_client. performUpdateCredential now reads the stored dataCenter out of the existing blob when the caller does not supply one. The read is failure-tolerant - an undecryptable or unparseable blob yields undefined rather than throwing, so it can never block a reconnect, and the provider default applies as before. Raised independently by three reviewers; I twice argued it was acceptable because the mint fails loudly rather than corrupting silently. That was true and beside the point - the operator still had to guess why. * fix(zoho-desk): delta-audit findings - prevState scope, status leak, HTML sniffer An audit of the commits the earlier five audits never saw. All four findings are in code written as fixes for those audits, which is where this branch has repeatedly introduced new problems. `includePrevState` was sent for Ticket_Comment_Update. The previous commit gated it on an `_Update` suffix and claimed Zoho supports it on every update event. Zoho's webhook doc lists the attribute on Ticket/Contact/Agent/Task/Article update events but NOT on Ticket_Comment_Update, which documents only `departmentIds`. That made it an undocumented filter key on a live subscription create - the same class of risk the same commit reverted `limit=200` for, so it failed that commit's own stated bar. Now an explicit set rather than a suffix rule. The status/priority split did not stop the leak it was written for. The mapping used `operation === 'list_tickets' ? filterValue : updateValue`, whose bare else covers all eight other operations - so a stale Update Ticket status was forwarded into get_ticket, list_comments and the rest. Harmless on the wire (those tools ignore it) but exactly the stale-value pattern the neighbouring gates exist to prevent. Both fields are now scoped to the two operations that declare them. The HTML sniffer destroyed plain text. `/<[a-z!\/][^>]*>/` fires on any `<` followed by a letter with a later `>`, so realistic ticket bodies lost content: "if x<y then z>0" became "if x0", and "replace <username> with the real name" lost the placeholder. It now requires a real element - a paired tag, a self-closing tag, a comment/doctype - or an entity, and the entity arm covers hex references it previously missed. Regression tests verified by reverting to the loose pattern and watching them go red. The reconnect data-center carry-forward is scoped to client-credential providers. As written it added a DB read plus a decrypt to every service-account reconnect for every provider - Slack, Atlassian, all token-paste providers - to carry a field only Zoho has. Also: the JWKS cache-bound TSDoc had been orphaned onto the wrong constant by an earlier insertion, and `cooldownDuration` was dropped since it restated jose's default while only `timeoutDuration` needed justifying. * test(zoho-desk): cover the webhook subscription filter rules The subscription filter logic had no test coverage at all, and it is where the last two rounds both found bugs - includePrevState on an event Zoho does not document it for, and departmentIds sent to events that accept no filters. Adds six cases against the real createSubscription: includePrevState is set for each of the five documented update events and NOT for Ticket_Comment_Update, departmentIds is kept for a filterable event and dropped for one that is not, and an event with no filters serializes as null rather than an empty object. Verified the guard bites: reverting PREV_STATE_EVENTS to the `endsWith('_Update')` rule turns the Ticket_Comment_Update case red. The Ticket_Comment_Update assertion checks the with-departments case as well as the bare one - asserting only `not.toHaveProperty` on the bare filter would pass vacuously, since that filter is legitimately null. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>	2026-08-01 21:55:00 -07:00
Waleed	9064039c19	improvement(logfire): scope block outputs per operation and refresh brand chrome (#6178) ... * improvement(logfire): scope block outputs per operation and refresh brand chrome - gate each block output on the operations that actually return it - swap in the official Logfire mark, black tile with brand-magenta bare icon - move host to advanced mode and alphabetize the tool registry entries - add track-logfire-llm-cost and verify-logfire-token-target skills * fix(logfire): honor numeric-string limits and surface token validity fields - accept a numeric-string limit so agent-invoked calls stop silently falling back to Logfire's 100-row default - keep an hour-only UTC offset intact instead of producing +05Z - surface expiresAt and spendingCapReachedAt on Get Token Info - document pending_span as a fourth record kind * chore(logfire): regenerate tool metadata and document the step in the skill - regenerate apps/sim/tools/generated/tool-outputs.ts, which CI's tool-metadata:check requires after a tool output change - add the regeneration step and artifact-diff guidance to the validate-integration skill so the gate stops being missed * chore(skills): sync validate-integration projections	2026-08-01 18:16:54 -07:00
3 people	18214158b9	fix(sanitization): secret exposure in function and agent trace spans (#6000) ... * fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>	2026-08-01 16:37:08 -07:00
Waleed	19b0312a04	feat(managed-agent): add session lifecycle operations (#6140) ... * feat(managed-agent): add session lifecycle operations Adds an operation selector to the Claude Managed Agents block, backed by nine new tools alongside the existing run-session behavior: - create session (non-blocking, seeds initial_events) - send message to an existing session - get session (surfaces tool calls awaiting approval) - list events - update session (title/metadata) - interrupt session - respond to tool confirmation (allow/deny) - archive session - delete session Run Session stays the default, so blocks saved before the selector existed keep their exact behavior and field layout. * fix(managed-agent): address review findings on session lifecycle ops - List Events kept the OLDEST slice when capped, dropping the agent's most recent reply. Paging is exhaustive again and the cap now keeps the newest N after ordering, with a `truncated` flag so callers know it is a tail. - listPaginated returned whole pages past maxItems; it now trims to the exact cap. - A whitespace-only title passed the update guard and would have cleared an existing session title. Blank is now treated as not provided. - Interrupt had no request timeout and could hang; bounded at 15s while still honoring the workflow signal. - Custom-tool gates were surfaced by Get Session but could not be answered, since they need user.custom_tool_result rather than a confirmation. Adds a Respond To Custom Tool operation and a `kind` on each pending gate so a workflow routes to the right one. - Docs rendered a raw ${DEFAULT_EVENT_LIMIT} placeholder for the default. * fix(managed-agent): correct truncation flag, gate lookup, custom tool result - `truncated` was true whenever the history size equalled the limit, even though nothing was dropped. Event reads now report the untrimmed total and the flag compares against that. - Pending-gate enrichment capped its read, which keeps the OLDEST events in page order — the opposite of where blocking gates live. It now filters to the ids being looked up as pages arrive, which is both correct regardless of page order and bounded by the id count. Paging continues on the raw page so a fully-filtered page is not mistaken for the end of the list. - Respond To Custom Tool applied one result to every id, so multiple pending tools would all receive the same output. It now answers a single call per invocation. * fix(managed-agent): stop fractional event limits reading unbounded A limit below 1 passed the positivity check and then floored to 0, which made `slice(-0)` hand back the ENTIRE history flagged as complete — the opposite of the requested bound. The limit is now floored before it is validated, so anything that does not resolve to a positive integer falls back to the default. Also hardened the library: a zero or negative cap short-circuits to an empty result instead of falling through to `slice(-0)`, so no future caller can hit the same trap. * fix(managed-agent): stop gate lookup scanning the full tool history The id filter keeps the collected array tiny, so `maxItems` never trips and the walk continued to the end of a session's tool history even after every blocking id had been found. `listPaginated` now takes a `stopWhen` predicate and the gate lookup ends as soon as it has all the ids it came for. Also makes a blocked-but-unnamed session observable: when a session reports `requires_action` with no blocking event ids, `requiresAction` stays true — reporting false would tell a workflow the session is fine while it is parked indefinitely — and the dead end is logged and documented instead. * fix(managed-agent): floor event cap and make metadata clearing explicit - A `maxItems` between 0 and 1 slipped past the zero guard and became `slice(-0)` — the whole history — because slice truncates its index toward zero. The cap is now floored at the library boundary, so no caller can hit it whatever they pass. - Update Session documented full metadata replacement but could not express a clear: an empty map normalizes to "absent". Inferring the clear from emptiness would be worse, since an untouched table is also empty and would wipe metadata on every title-only update. Adds an explicit `clearMetadata` instead, and corrects the parameter's documentation. * test(managed-agent): pin the HTTP shape of every session endpoint Method, URL, and beta header for all 11 calls, plus the SSE accept header, the separate memory-store beta (combining the two is a documented 400), and content-type only on requests that carry a body. These are the details types cannot catch and that break silently when a path is "tidied".	2026-07-31 17:59:20 -07:00
Theodore LiandClaude Opus 5	c5cc6ce26c	feat(chat): hide the Chat module when NEXT_PUBLIC_CHAT_DISABLED is set (#6137) ... * feat(chat): hide the Chat module when CHAT_ENABLED is unset A self-hosted deployment that skipped the chat key still rendered the full mothership Chat UI, landing on the composer and 401ing on every message. Gate it behind a CHAT_ENABLED / NEXT_PUBLIC_CHAT_ENABLED twin, written by the setup wizard alongside COPILOT_API_KEY and validated by the existing FLAG_TWINS doctor check. The flag resolves at module scope on both render passes, so no chat surface renders then disappears. With Chat off the workspace lands on its first workflow (resolved server-side, behind the cached host-context check so no workflow id leaks to non-members), and the chats list, scheduled tasks, editor Chat panel, and chat CTAs are absent. Routes are gated rather than deleted: /home redirects because it is baked into delivered invitation emails and the accept contract. Also fixes two bugs the gate exposed: a persisted activeTab of 'copilot' left the workflow panel blank from first paint, and the panel's handoff listener claimed MOTHERSHIP_SEND_MESSAGE events outside its own gate, silently swallowing "Fix in Chat" messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * refactor(chat): gate the UI on NEXT_PUBLIC_CHAT_DISABLED, not an opt-in flag CHAT_ENABLED made Chat opt-in, so every existing deployment that already had COPILOT_API_KEY would have lost the module until it set a new variable. Invert to an opt-out so nothing changes for them. That also collapses the twin. The only reason the flag needed a server/client pair was that it projected a secret; NEXT_PUBLIC_CHAT_DISABLED is not one, so getEnv resolves the same value from process.env on the server and window.__ENV in the browser. Gone with it: the FLAG_TWINS entry and its doctor sync check, the two-variable wizard write, and the boot-time throw, whose contradiction (flag on, key absent) can no longer be expressed. Presentation and capability are now separate concerns. NEXT_PUBLIC_CHAT_DISABLED decides whether the surfaces render; COPILOT_API_KEY decides whether the work can run, and gates the paths that need it — the Sim Chat block, prompt-job claims, and inbox access — each failing on its own terms. The wizard writes the opt-out when you skip the chat key, which is the case this started from: a fresh self-host that never configured Chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * feat(setup): prompt for the chat key in k8s mode The dev and compose flows minted a chat key and wrote the Chat opt-out alongside it; k8s did neither, so a cluster install with no COPILOT_API_KEY in its Helm values rendered a Chat module that rejects every message. Prompt with the same flow and feed both values into `app.env`, which the chart already renders as arbitrary container env. Reading the previous release's key matters here in a way it does not for the file-based modes: `helm upgrade` without `--reuse-values` keeps only what this document carries, so a key the user elects to keep has to be re-supplied or it is silently dropped. Splits the release-values read from the secret-reuse check so both the key and the secrets come from one `helm get values` call, and carries the mothership override across for the same mint-here-validate-there reason the other modes document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): write app-behavior flags to every env file the app can start from The wizard wrote the Chat opt-out only to the env file its own mode owns, so choosing compose put it in the root `.env` while `bun run dev` reads `apps/sim/.env` and never saw it. Skipping the chat key appeared to do nothing. Mirror values that change how the app behaves — as opposed to where it connects — across both targets. Connection settings deliberately do not go through this: DATABASE_URL and friends differ between the compose stack and a local dev run, which is why this takes an explicit set of values rather than the whole batch. The mirrored file is written even when absent, since missing is exactly the case that stranded the flag, but with seeding suppressed so a compose run leaves a one-line apps/sim/.env instead of a full .env.example for a stack the user is not running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(compose): forward NEXT_PUBLIC_CHAT_DISABLED to the app container The wizard wrote the flag into the root .env, but compose only passes through variables the service's `environment` block names — and that block listed COPILOT_API_KEY without its companion. Skipping the chat key on a Docker install therefore did nothing: the value sat in .env and never reached the container. Add the passthrough to all four compose files. Reverts the previous commit's mirroring into apps/sim/.env, which treated the symptom — each mode writes only the env file it owns, and that file is now wired correctly. k8s needs no equivalent: its values flow into `app.env`, which the chart renders key by key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): resolve the landing route without blocking on the database Server-resolving the first workflow meant a session lookup, an access check and a query had to finish before anything rendered. A slow or unreachable database left the user on a blank page under a populated sidebar — worse than the instant redirect it replaced, and with no signal that anything was wrong. Redirect straight to `/w` instead and let it pick from the workflow list the layout already prefetches, so the choice costs no round trip and cannot hang. Repoints the sidebar's primary action rather than hiding it: the slot that offered "New chat" now offers "New workflow" and creates one, since with Chat off there is no composer to open but the intent is the same. Sends the CLI key handoff to signup rather than login. It is reached from a terminal — usually the setup wizard standing up a fresh self-host — where the visitor has no account yet. Both auth pages cross-link carrying the callback, so a returning user is one click from login with their destination intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * improvement(chat): address cleanup-pass findings on the Chat gate Effects: the panel's auto-select effect read the copilot chat list while the list query was deliberately skipped, took "empty" for "deleted in another tab", and cleared the user's selection — latching a ref that stopped it ever being restored. Guarded on the same condition as the handoff listener. Memo: `/w` filtered workflows through a useMemo whose array dependency was a fresh `[]` on every render while the query had no data — the exact window the page exists for — so it memoized nothing and re-fired the redirect effect. Keyed on the workflow id instead. Same unstable-default problem on the sidebar's chat list, where it invalidated five downstream memos; given a stable empty constant. Callback: `handleCreateWorkflow` listed the whole mutation object in its deps, which TanStack recreates every render. Harmless until this branch wired it into the top nav, where it defeated `memo(SidebarNavItem)`. React Query: Recently Deleted still fetched archived chats unconditionally and offered restores into routes that now 404. Also surfaces an error state on `/w` — it is the landing route now, so a failed list fetch would otherwise spin forever behind a log line — fixes a spinner using a token undefined in dark mode, and trims comments that restated code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(chat): gate workflow creation on write access, pin the key in schedule tests The zero-workflow landing offered "Create workflow" to every member. Creation navigates optimistically, so a read-only member was sent to a workflow the server had already refused to create, with the failure never surfaced. Gate both entry points — the empty state and the sidebar's "New workflow" row — on the same `canEdit` check the rest of the sidebar uses, and tell read-only members who can make one instead of offering an action that cannot succeed. The schedule-execution tests only passed locally because vitest loads the developer's own `.env`, which supplied COPILOT_API_KEY; CI has none, so the prompt-job claim guard skipped the claims those cases assert on. Pin the key through the env mock so the suite states its own preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha * fix(setup): name both variables in the chat-key failure hint The caller writes the Chat opt-out whenever the prompt returns no key, so the hint's "or set COPILOT_API_KEY yourself" restored capability while leaving the module hidden — the one path where following setup's own advice does not work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ErcRgvi7VQBeKDQ3MBMha --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 19:23:24 -04:00