mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-12 23:33:47 +08:00
ed17bb2bacccbc004d94109fca260e1363cfd0dd
5836
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Waleed
|
ed17bb2bac |
chore(deps): upgrade next to 16.3.0 and clean up the TypeScript toolchain (#6235)
Bumps next, @next/env and the @next/swc-* optional deps to 16.3.0 across the root overrides, apps/sim, apps/docs and packages/emcn. Two config notes worth keeping: - `experimental.turbopackFileSystemCacheForBuild`'s default flipped false -> true for stable in 16.3.0, so our explicit `false` is now load-bearing rather than defensive. Without it this bump would have silently re-enabled a build cache measured 3.2x slower on this codebase (#6078). Comment updated to say so. - `experimental.useTypeScriptCli: true` is now pinned. TypeScript 7 ships no JavaScript compiler API until 7.1, so Next's default checker cannot run and needs the project-local `tsc` CLI instead. 16.2.12 was silently skipping build-time type checking entirely because it detected @typescript/native-preview and short-circuited the stage ("Finished TypeScript in 138ms"); pinning the flag keeps that from drifting back. TypeScript toolchain cleanup that the upgrade makes possible: - Drop @typescript/native-preview from apps/sim and apps/docs. It was the pre-release channel for TS 7 and is superseded by typescript@7 (nightlies now ship as typescript@next), and its presence is what suppressed build type checks. - Align packages/browser-protocol and packages/terminal-protocol from typescript ^5.7.3 to ^7.0.2 so every workspace is on one compiler version. apps/sim keeps @typescript/typescript6 as a production dependency: the function sandbox at app/api/function/execute dynamically imports the TS 6 compiler API to transpile user code, and TS 7 has no API to replace it yet. Build and dev were benchmarked 3x per version on a byte-identical tree; the upgrade is performance-neutral (build median 100s -> 99s, dev:full warm 10s -> 9s). |
||
|
Waleed
|
f210a6edc0 |
fix(execution): offload buffered event values under budget pressure (#6231)
* fix(execution): offload buffered event values under budget pressure An execution buffers EVENT_LIMIT events inside a per-execution byte budget, so a full ring only fits if events average under budget/EVENT_LIMIT. Values were only offloaded to object storage at the shared 8 MiB cap, far above that, so a run emitting large block outputs exhausted its budget within a few dozen events and stayed pinned at its ceiling for the rest of its life. Applying that ceiling to every run would be worse than the problem: the SSE stream carries the compacted event and the terminal renders a ref only as a preview, so ordinary block outputs would stop being readable live, and every value would cost an object-storage write on the hot path. Engage the tight ceiling only once a run has actually buffered past half its budget. A short run keeps full-fidelity output and pays nothing; a runaway one stops accumulating. Both bounds derive from the existing budget rather than being asserted, and preserved UserFile base64 is exempt — it is an explicit request for inline delivery, already bounded by its own cap and the strip-and-recompact fallback. Also stop a failed resume-path buffer write from failing the run: it was awaited bare, so the rejection propagated into the executor callback and failed work that had already completed. The buffer only backs reconnect replay, so degrade to live-only delivery the way the execute route does. * fix(execution): measure pressure at write time and keep terminal status last Pressure was read from bytes counted once a flush succeeded, but a burst is compacted long before the scheduled flush runs — so the very batch that exhausts the budget went through at the loose ceiling and was dropped instead of offloaded. Count bytes as each event is compacted. Separately, the terminal-alone retry stamped terminal status while entries queued ahead of it were still unwritten. Terminal status is the reader's end-of-run signal: a reconnecting client drains what is in Redis and closes, so those entries were stranded behind a stream it had already finished with. Drain the backlog first, then publish the terminal event. * fix(execution): do not lose the backlog or publish terminal status early Draining the backlog ahead of the terminal event left the terminal status armed, so whichever chunk emptied the queue stamped the run complete before its terminal event was written — the inverse of the ordering the drain was added to guarantee. Disarm the status for the drain and restore it afterwards. The drain's result was also discarded: a transient Redis failure requeues its batch, and the unconditional reassignment that followed dropped those events even though the budget never rejected them. Keep whatever could not be persisted, and publish the terminal event alone only once nothing earlier is still queued — failing otherwise lets the caller degrade, which records the status without claiming the missing events arrived. Leave eventId unset on a failed resume-path write. Assigning 0 was persisted by clients as a reconnect cursor and rewound them to the start of the run. * fix(execution): keep an event in the buffer when a pressure offload fails Durable compaction runs before an event is queued, so a storage or metadata failure dropped it from replay entirely — a reconnecting client would never see it, even though the live path carried on. Offloading under pressure is only an optimization that keeps a heavy run from exhausting its budget, so when the value cannot be persisted, fall back to buffering it inline: exactly what the run would have done before pressure engaged. |
||
|
Waleed
|
41592dfbf1 |
fix(execution): stop the event buffer retaining a run-length backlog (#6229)
The Redis byte-budget branch in doFlush requeued the rejected batch and rethrew, skipping the MAX_PENDING_EVENTS trim every other failure path applies. The backlog then grew for the rest of the run and each retry re-serialized it, so a wide parallel fan-out could drive unbounded heap growth and stall the event loop. Drop rejected chunks instead of requeueing, pace retries through the existing backoff, and split batches that exceed the single-write cap so an oversized batch can make progress instead of stalling forever. Terminal status is now writer-scoped, since a concurrent scheduled flush can be the loop that drains the final chunk, and a terminal event whose batch was dropped is retried on its own rather than lost with it. Record terminal stream meta when the terminal event cannot be buffered, so reconnecting readers stop polling an active stream until their deadline. Drop the unused reserve/release budget helpers. |
||
|
Siddharth Ganesan
|
5ab5f2c7ed |
feat(browser, terminal): implement browser driver, password manager, terminal features (#6196)
* icon styling * feat(desktop): isolate chat browser and terminal sessions * feat(desktop): uncap browser and terminal tabs * feat(desktop): polish browser and terminal resources * improvement desktop * fixes * updates * fixes * fix * update tests |
||
|
Waleed
|
3de63c94e3 |
feat(self-host): align Docker Compose with Helm and overhaul self-hosting docs (#6225)
* feat(self-host): align Docker Compose with Helm and overhaul self-hosting docs Docker Compose shipped no scheduler, so scheduled workflows, every polling trigger, connector syncs, the outbox, and data drains silently never ran. Adds a cron service running the same 18 jobs the Helm chart schedules as CronJobs, and closes the remaining behavioral gaps between the two paths: bundled Redis in the chart, no hosted plan caps in chart defaults, pinned image tags, and fail-fast secrets. A CI check keeps the schedulers in sync. Also rewrites the self-hosting docs: 14 new pages, 8 updated, reorganized into Install / Configure / Operate. * fix(self-host): drop bun install from chart CI, remove air-gapped and backup docs The scheduler-parity check pulled a full dependency install into the chart-validation job, which fails building isolated-vm on that runner. Rewritten to use only node builtins so the job installs nothing. Also removes the air-gapped and backup/restore pages, and stops pinning a concrete release in the docs so the examples do not go stale each release. * fix(helm): bundle Redis in secret-manager modes unless the URL is supplied Suppressing Redis whenever a secret mode was active left those deployments with no Redis at all — REDIS_URL is optional there and both shipped examples omit it. The chart now steps aside only on a detectable signal: an explicit app.env.REDIS_URL, an ESO remoteRefs.app.REDIS_URL mapping, or the new redis.provideUrl=false opt-out for a pre-created Secret it cannot read. * fix(compose): derive realtime BETTER_AUTH_URL from NEXT_PUBLIC_APP_URL realtime read BETTER_AUTH_URL directly and fell back to localhost while simstudio derived it from NEXT_PUBLIC_APP_URL, so setting only the public origin left realtime authenticating against http://localhost:3000. * fix(helm): deliver bundled REDIS_URL via ConfigMap so an operator value always wins Injecting REDIS_URL as an inline container env made it beat every envFrom source, so a REDIS_URL held in a pre-created Secret or synced by External Secrets was silently shadowed and traffic moved to a fresh in-cluster Redis. Kubernetes resolves duplicate envFrom keys by letting the last source win, so the bundled URL now ships as a ConfigMap listed before the app Secret. Any operator-supplied value overrides it without the chart needing to read it, which also removes the redis.provideUrl flag the previous attempt required. * docs(helm): spell out the egress rule external datastores need The default NetworkPolicy allows 443 plus the bundled Postgres and Redis by pod selector. Anything you run outside the chart on another port needs its own rule, which is easiest to miss when REDIS_URL arrives via a Secret the chart cannot inspect. Adds a copyable example to the production checklist and the security guide. * feat(helm): add networkPolicy.allowExternalEgress for managed datastores The default policy allows 443 plus the bundled Postgres and Redis by pod selector, so a managed datastore on another port needs a hand-written CIDR rule — awkward when REDIS_URL arrives via a Secret the chart cannot inspect. Adds an opt-in switch that drops the port restriction while still blocking the cloud metadata endpoints. Defaults to false, keeping this chart stricter than the common chart default of unrestricted egress. |
||
|
Waleed
|
1242301bd3 |
improvement(emcn): share one emails/domains chip input across share and deploy modals (#6226)
* improvement(emcn): share one emails/domains chip input across share and deploy modals
Extracts the emails chip lifecycle out of ChipModalField type='emails' into a
standalone ChipEmailsInput, and points both the file share modal and the deploy
modal's chat tab at it instead of their hand-rolled TagInput wiring.
- add ChipEmailsInput (dedupe, normalize, format gate, paste, per-chip errors)
with an allowDomains opt-in for bare @domain.tld entries
- share modal and deploy modal chat tab now use it; drop both hand-rolled
add/remove/validate implementations and the dead emailError state
- move the shared allowlist policy into validateAllowlistEntry
- drop the "Add specific emails or whole domains" hint text
- give OutputSelect a size prop; the deploy modal chat tab uses the 30px chip
trigger so it lines up with the Title field above it
- drop overflow-y-auto from the chat deploy form, which was promoting overflow-x
to auto and rendering a stray horizontal scrollbar
* improvement(utils): one email syntax gate, drop the backtracking placeholder regex
Audit follow-ups on the emails chip input.
- move EMAIL_SYNTAX_REGEX and the new @domain pattern into @sim/utils/string as
isValidEmailSyntax, so emcn and lib/messaging/email/validation.ts stop keeping
byte-identical copies of the RFC 5322 regex
- allow single-label domains (@intranet) again — requiring a dot rejected
entries the old startsWith('@') check accepted, which self-hosted
deployments use. A lone @ and malformed labels stay rejected
- replace derivePlaceholderWithTags' /^Enter\s+(.+?)s?$/i with string ops;
CodeQL flagged it as polynomial backtracking (js/redos). Verified identical
output across the placeholder shapes in use
- forward the emails control's props explicitly instead of underscore-discard
destructuring, matching ChipModalFileControl in the same file
* test(email): pin the allowlist entry rules
Covers isValidEmailSyntax's allowDomains branch (single-label domains stay
valid, malformed bare domains that the old startsWith('@') check accepted do
not), the 254-character cap, the DNS label limit, and validateAllowlistEntry
waiving address-level policy for bare domains. Verified both new rules fail
when the behavior is reverted.
|
||
|
Waleed
|
67e355b1ec | chore(landing): remove the Russell Investments logo (#6228) | ||
|
Waleed
|
c759a889d8 |
refactor(voice): load STT availability through React Query (#6224)
* refactor(voice): load STT availability through React Query useSpeechToText fetched `/api/settings/voice` inside an effect and stored the result in useState behind a hand-rolled mountedRef guard: no cache, no dedupe across mounts, and no AbortSignal, so the response was fetched and parsed even after unmount. Two simultaneously mounted consumers issued two requests. It also bypassed hooks/queries/**, which is where every other server read in the app lives — and it escaped `check:react-query`, whose audit only covers useQuery/useMutation call sites. The value is server env read at request time, so it cannot change within a session; the new hook uses an infinite staleTime and a caller-controlled `enabled` so clients without the audio APIs never issue the request. Hydration is unchanged: SSR renders unavailable, and the first client render still resolves unavailable because `data` is undefined until the fetch settles. No initialData, deliberately — adding it would break that. mountedRef stays; it is still load-bearing for the streaming lifecycle. * test(queries): unmount rendered roots between tests renderHookWithClient created a React root per test but never tore it down, so trees stayed mounted with live QueryClient observers until worker teardown and async notifications could cross test boundaries. Audited every test in the repo using createRoot: 51 of 53 already unmount. The two that did not were both mine — voice.test.tsx here and chats.test.tsx from #6223 — so both are fixed and the pattern is now uniform. * fix(voice): let a failed STT probe recover on a later mount The app QueryClient sets retryOnMount: false and retry: 1, and refetchOnWindowFocus only refetches stale queries — which an infinite staleTime never becomes. So one transient failure cached the error for the life of the client and hid the mic until a full page reload. The effect this replaced refetched on every run, so retryOnMount: true restores parity: no refetch after success, a retry per mount after failure. Test asserts recovery under the app's real query defaults and fails without the override. |
||
|
Theodore Li
|
3f70096841 |
improvement(self-host): gate email verification on a mail provider, add self-host settings, land setup on signup (#6216)
* fix(auth): skip email verification when no mail provider is configured Signup pushed /verify unconditionally, stranding self-hosted deployments with no mail provider on a screen no email could ever satisfy. Derive one server-side effective value (verification enabled AND deliverable) and read it from Better Auth enforcement, signup routing, and the verify page. * feat(settings): add a self-host section with the managed Chat keys link Self-hosters had no in-app pointer to the managed service that issues their Chat keys. New Settings > System > Self-host section, gated on `requiresSelfHosted` so it is absent on hosted Sim, containing only that link. * improvement(setup): land the wizard handoff on signup A freshly provisioned deployment has no accounts and / renders the marketing landing page, so the bare origin left operators hunting for the CTA. Single-source the URLs and point every open-Sim handoff at /signup across all three modes. * improvement(settings): mark the self-host section with a sprout Server was already doing double duty for MCP servers and Mothership, and the icon set ships no botanical glyph, so the mark is a text emoji. * improvement(settings): draw the sprout as an emcn line icon, move to Platform The emoji rendered in the platform's own colors, so it was the one glyph in the nav that ignored --text-icon. Replaced with a hand-drawn emcn Sprout (24 grid, 1.55 stroke, currentColor) matching the house style, renamed the tab to Self hosting, and regrouped it under Platform — self-hosting is deployment-wide, not per-workspace. Still self-hosted-only. * improvement(settings): drop the section header from self hosting One row does not need a section label, and removing it takes the divider with it. The body is now the Chat keys row and its managed-keys link, nothing else. |
||
|
Waleed
|
1708173970 |
refactor(chat): clean up the deployed chat surface (#6220)
* refactor(chat): clean up the deployed chat surface Eight-angle cleanup pass over the full contents of the chat surface and the speech code that survived the voice-mode removal. Dead code - enforceChatRateLimit: added for the TTS relay in #6212, orphaned when #6215 deleted that route. Zero consumers. - ChatToolCallStatus, ChatErrorType, and six unused CHAT_ERROR_MESSAGES keys (only GENERIC_ERROR and CHAT_UNAVAILABLE are read). - scrollToMessage was declared and destructured by ChatMessageContainer but never used in its body; removing the prop also made the scrollToShowOnlyMessage branch unreachable, since the sole caller passed true. - permissionState and the language prop on useSpeechToText: both write-only across the repo. - The image branch in ChatFileDownload's renderIcon returned the same DefaultFileIcon at the same size as the fallback. - chatKeys.status/detail: aliases of deploymentKeys nothing imported, and misleading since they root under a different key namespace. Redundant state - password-auth and email-auth each kept a boolean in lockstep with `errors.length > 0`; email-auth also validated on every keystroke and then immediately hid the result. - file-download tracked hover in state to drive one opacity class; now group-hover. Verified emcn Button sets no `group` class of its own. Memoization - ChatMessageContainer's memo() could never bail: chat.tsx passes an inline arrow for scrollToBottom and displayMessages is a fresh array. Four of the five things that re-render ChatClient are its props anyway, so the memo is dropped rather than propped up. - ClientChatMessage keeps its memo — it blocks markdown re-parsing — but loses the custom comparator, which compared proxies (a key:status fingerprint, files by length) and ignored attachments and type entirely. Default shallow compare on its single prop is both simpler and stricter. - Six useCallbacks whose consumers are native DOM handlers or inline arrows, so nothing observed their identity. Effects - The scroll listener attached in an effect keyed on [chatConfig, authRequired] — values it never reads, standing in for "the container has mounted". It now attaches via a ref callback, so it no longer re-attaches on every config refetch. Design system and a11y - z-[100] -> z-[var(--z-dropdown)] (same value), shadow-lg -> shadow-medium, list styles from inline style to Tailwind classes, hover: -> hover-hover: on touch-reachable targets, Check sourced from emcn alongside its Duplicate pair. - Accessible names on the remove-attachment, stop, and send buttons, which announced only as "button". - Dropped a keyboard handler on a role='group' div with no tabIndex, where target === currentTarget was unreachable, and the Tooltip Provider wrappers and delayDuration, which emcn documents as no-op passthroughs. * fix(chat): restore markdown list markers The design-system pass swapped inline `listStyleType` for Tailwind classes, but the edit that added `list-disc`/`list-decimal` silently did not apply while the one removing the inline style did. With Preflight setting `list-style: none`, every bullet and number in an assistant response disappeared. `list-item` on the `li` sets display only, not the marker type. |
||
|
Waleed
|
b0491f1161 |
fix(chat): invalidate deployment queries after a chat mutation (#6223)
PATCH /api/chat/manage/[id] calls performFullDeploy when the workflow has drifted from its active deployment, so editing a chat can mint a new deployment version. useUpdateChat invalidated only chatStatus and chatDetail, leaving the deployment panel showing the previous version and a stale "needs redeployment" indicator until the staleTime expired. Both mutations now route through invalidateDeploymentQueries, the shared helper the rest of the deployment surface uses. That also picks up deployedState, which useCreateChat's hand-rolled list had omitted even though performChatDeploy replaces the deployed workflow state. Tests cover both mutations and were verified to fail against the previous invalidation. |
||
|
Waleed
|
36aace765e |
fix(auth): correct callback URL resolution across SSR and hydration (#6217)
* fix(sso): derive the SSO callback URL during render
`callbackUrl` was seeded into `useState('/workspace')` and overwritten from
a `useEffect` that read `searchParams`, so the first painted frame always
carried the default. On any deep link with `?callbackUrl=`, the "Sign in
with email" and "Sign up" links briefly pointed at `/workspace` instead of
the requested destination, and a click landing in that window navigated to
the wrong place.
- derive `callbackUrl` from `searchParams` during render; the validation
gate is unchanged, so an off-origin or malformed value still falls back
to `/workspace`
- keep the warning for a rejected value in an effect, now keyed on the
param itself rather than the `searchParams` object, so it fires once per
actual change instead of once per identity change
- add first-frame tests via `renderToString`, which runs no effects and so
pins exactly the window the old code got wrong
* fix(auth): resolve callback URLs against the app origin server-side
`validateCallbackUrl` compared against a sentinel base
(`https://callback-url-validator.invalid`) when `window` was undefined, so
the server rejected every absolute URL — including the same-origin ones the
function documents as valid. A component deriving a callback URL during
render therefore produced one destination in the SSR markup and a different
one after hydration.
The exposure was not new to the SSO form: `login-form.tsx` and
`signup-form.tsx` already derive their callback URL during render on
`force-dynamic` pages, so both carried the same divergence.
- resolve against the deployment's own origin server-side, so the server
reaches the same verdict the browser will after hydration
- fall back to the sentinel when the app URL is unset or unparseable, which
keeps the server fail-closed: absolute URLs are rejected, as before
- cover the absolute same-origin case and the unset-app-URL fallback in the
existing suite; all 15 open-redirect rejection cases are unchanged
* fix(env): drop the getBaseUrl browser-origin fallback
The fallback was added in #6214 as a safety net while the real cause — the
hosted env script losing its `beforeInteractive` strategy — was fixed in the
same PR. With the injection ordering restored, `window.__ENV` is populated
before hydration, so the fallback is unreachable in any correctly configured
deployment.
Guessing the origin was also unsafe in the one case it could still fire. An
opaque origin — a sandboxed iframe, and `/chat/*` is deliberately embeddable
— serializes to the string `'null'`, which is truthy, so `getBaseUrl()` would
have returned `'null'` and every call site would have silently built
`null/api/...`. A throw surfaces the misconfiguration instead of encoding it
into request URLs.
- restore the unconditional throw when NEXT_PUBLIC_APP_URL is unset or blank
- mirror it back in the shared testing mock
- flip the two fallback tests to assert the throw, keeping whitespace-only
coverage
Server-side behavior is unchanged: there was never a `window` to fall back
to, so callers that already guard `getBaseUrl()` (`getBaseDomain`,
`validateCallbackUrl`) keep their existing fail-closed paths.
|
||
|
Waleed
|
09eba8a0f9 |
improvement(ci): move CodeQL off default setup onto Blacksmith (#6219)
* improvement(ci): move CodeQL off default setup onto Blacksmith Default setup pinned every scan to a 4-vCPU GitHub-hosted runner with no cancel-in-progress: PR scans ran 30-125 min and re-ran on every push (#6183 burned six overlapping runs). None of that is reachable from the settings UI, so the config moves into the repo. - Blacksmith 8-vCPU via the same CI_PROVIDER escape hatch as ci.yml - cancel-in-progress scoped to pull_request so push/schedule scans finish - push to main + PR to main/staging + nightly safety net - paths filter so doc-only PRs skip the run entirely - paths-ignore config drops tests/mocks/fixtures: 12,716 -> 11,128 files - languages: javascript-typescript + actions; python dropped (7 files in tree) Default setup has been disabled; the two cannot both be active. * fix(ci): restore CodeQL coverage of the data-drain test route Review round 1. - Drop the '**/test/**' and '**/tests/**' globs. A `test` directory is a routable Next.js path segment, and those globs excluded apps/sim/app/api/organizations/[id]/data-drains/[drainId]/test/route.ts — a POST handler that authorizes, decrypts destination credentials and makes an outbound request. CodeQL paths-ignore has no `!` negation to carve it back out, and the globs only covered 76 of 12,716 files. - Add `ready_for_review` to the pull_request activity types. It is not a default type, so a PR opened as a draft and later marked ready was skipped by the draft guard and never rescanned until the next push. |
||
|
Waleed
|
83988c1ebb |
refactor(chat): drop code the voice removal left unreachable (#6218)
Full-file cleanup pass over the files voice mode was removed from. - /api/speech/token: with the chatId branch gone, billingAttribution is always resolved, so four `billingAttribution ? ... : ...` guards and the checkActorUsageLimits fallback were unreachable. Removed, along with the now-unused imports and test mock. - chat.tsx: the inputValue state had one remaining writer (`''`) and no reader — ChatInput has owned its own input value since it went uncontrolled, and removing the voice-transcript caller left handleSendMessage always receiving an explicit message. - use-chat-streaming: messageIdMap was written in three frame handlers and never read, and setIsStreamingResponse was returned but never destructured by the only consumer. - Comments: dropped JSX section labels that restate the element beneath them and TSDoc that restates the identifier; narrowed the speech contract's workspaceId doc, which existed to contrast with the removed chatId field. |
||
|
Waleed
|
78740c05b6 |
chore(chat): remove deployed-chat voice mode (#6215)
Removes the voice-first interface and TTS playback from the deployed chat, keeping workspace dictation, which is a separate feature. - Deletes the VoiceInterface UI and its particles canvas, the chat mic input, the TTS audio-streaming hook, the /api/proxy/tts/stream relay and its contract, and the voice-settings query hook. - Unpicks the voice wiring in chat.tsx and use-chat-streaming: the audio stream handler, sentence-splitting for speech, voice-first mode state, and the isVoiceInput plumbing through ChatInput. - Drops the now-dead chatId branch from /api/speech/token. It was the anonymous public-chat path; with no caller left it would have stayed an unauthenticated relay spending the platform key. That leaves resolveDeployedChatCaller unused, so it goes too. - Removes code the above orphaned: MAX_CHAT_SESSION_MS, the noop util, the audio/position refs in use-chat-streaming that were only ever written, and the chatId field on the speech contract. Keeps /api/settings/voice, lib/speech and use-speech-to-text: the workspace home input still uses them. Keeps the voice-output usage source, enum and label — Postgres cannot drop an enum value, and historical usage_log rows still need a label to render. |
||
|
Waleed
|
dd7951563e |
fix(env): restore beforeInteractive on the hosted public env script (#6214)
The hosted `<PublicEnvScript>` rendered a plain `<script>`, which lands at the end of `<head>` — after the ~40 `<script async>` chunk tags Next emits at the top of the document. An async script runs as soon as its fetch resolves, so on a warm cache a Next chunk could execute (and hydration begin) before the parser reached the env tag, leaving `window.__ENV` undefined for the first render. That surfaced as "Something went wrong" on the workflow page, since `getBaseUrl()` throws during the deploy modal's render, and as the socket falling back to the page origin instead of NEXT_PUBLIC_SOCKET_URL. Regressed in #5522, which replaced next-runtime-env's PublicEnvScript (to avoid its unstable_noStore forcing dynamic rendering) with a static equivalent that dropped the beforeInteractive strategy. - render the library's own `<EnvScript>`, which defaults to beforeInteractive and does not call unstable_noStore — hosted and self-hosted now share one implementation and one loading strategy - drop the hand-rolled serialization and `<` escaping; Next's beforeInteractive path already runs the payload through htmlEscapeJsonString, which escapes `& > < U+2028 U+2029` - fall back to the browser origin in getBaseUrl() rather than throwing, so a missing injected env can never tear down a page through the error boundary; server-side callers still fail loudly - read NEXT_PUBLIC_EMAIL_PASSWORD_SIGNUP_ENABLED via getEnv() in the SSO form, matching login/signup/auth-modal — `env.X` returns the build-time placeholder, not the runtime value |
||
|
Waleed
|
0bc4fb4656 |
fix(security): meter and throttle the deployed-chat TTS relay (#6212)
* fix(security): meter and throttle the deployed-chat TTS relay POST /api/proxy/tts/stream treated "a live public chat exists" as authorization to spend the platform ElevenLabs key. A public chat id is handed to every visitor, so any anonymous caller could synthesize speech with no length cap, no rate limit and no usage accounting. Bring the relay in line with its STT sibling (/api/speech/token): - Resolve the chat's workspace and bill synthesized characters to that payer via a new `voice-output` usage source, so spend is attributable and counts against the plan's usage limit (402 once exceeded). - Throttle per IP before any database work, and per chat afterwards, to bound both one caller hammering many chats and many callers hammering one chat. - Cap `text` at 2000 characters and allowlist `voiceId`/`modelId`, so the caller can no longer choose an unbounded charge, a premium or cloned voice, or the billing model. - Drop `Access-Control-Allow-Origin: *`, which let any third-party page read the audio; deployed chat and the Office embed are same-origin. * fix(security): correct TTS metering, pricing and body cap Follow-up review of the previous commit found five defects in it: - Usage rows collided. `usage_log.event_key` is unique and inserts are conflict-do-nothing, and the key is derived from the entry's stable fields. With no explicit sourceReference, two synthesis calls of equal character count in the same workspace produced the same key, so every repeat length went unbilled — defeating the metering this change is for. Each call now carries a unique sourceReference. - Priced at $0.10 per 1k characters, twice the published ElevenLabs Flash/Turbo rate of $0.05, which would have overcharged customers 2x. - No body cap, so an anonymous caller could make the route buffer up to the shared 50 MB default before validation. Now 16 KB, as the STT sibling does. - Threshold settlement ran per sentence: several queries and a possible Stripe call on a realtime path. The workflow execution that produced the text already settles the payer. - The per-IP bucket was described as preventing database amplification. getClientIp trusts the leftmost X-Forwarded-For, so an attacker rotates past it; the comment now says the per-chat bucket is load-bearing. * refactor(chat): share the deployed-chat auth gate across voice routes Review of the previous commits surfaced duplication and one more gap: - The TTS and STT routes had grown near-identical copies of the chat auth + payer lookup. Extracted to resolveDeployedChatCaller, so the gate and the payer resolve together and cannot drift per route — that duplication is how the unmetered TTS path shipped in the first place. - Neither copy filtered chat.archivedAt, so an archived chat could still authorize spend against its former owner's workspace. The shared lookup now filters it, fixing both routes at once. Note: not covered by a test — the db chain mock does not evaluate WHERE clauses, so an assertion here could not fail. - Replaced the route's hand-rolled 429 builder with the existing enforceIpRateLimit helper, and added enforceChatRateLimit alongside the per-user/IP/workspace helpers. Gains the standard Retry-After and X-RateLimit-Reset headers plus throttle logging. - Dropped a test that asserted a module the route no longer imports was never called: it could not fail. - Narrowed the contract: unexported the single-use allowlists and dropped .passthrough() now that the body is a closed shape. * fix(security): fail closed when voice-output usage cannot be recorded Review round 1 findings: - A ledger write failure previously logged and streamed the audio anyway, leaving the spend unrecorded and the payer's usage understated. The caller is anonymous, so serving audio we could not charge for is the unmetered spend this route exists to prevent — it now returns 500. - Use generateId() from @sim/utils/id rather than crypto.randomUUID, per the AGENTS.md ID rule. generateId returns a full UUID v4, so the per-call uniqueness the usage_log event_key depends on is unchanged. * fix(chat): split long TTS text so the relay cap cannot drop audio The client sentence-splits on Western `.!?` only, so text that never matches — CJK punctuation, or a list with no terminal punctuation — accumulates and is flushed as one block at the end of the stream. Against the new 2000-character relay cap that block is rejected and the whole message plays no audio, a regression introduced by adding the cap. Split to cap-sized pieces at the single point that enqueues synthesis, so both the per-sentence path and the end-of-stream flush are covered. Prefers a whitespace or CJK punctuation boundary, falling back to a hard cut when a block has none. The server cap stays as the enforcement point. * fix(security): release the vendor stream when metering rejects the request The fail-closed branch returned 500 with the ElevenLabs response body still open, so synthesis and download kept consuming vendor and runtime resources for a caller that was already rejected. Cancel it before returning, and assert the cancellation in the test. |
||
|
Waleed
|
806ea0c5bb |
fix(tooltip): stop text blurring for ~150ms every time a tooltip appears (#6211)
* fix(tooltip): remove velocity skew/scale that blurred text on every appear The tooltip animated a fractional scale() + skew() over 150ms on the element containing its text. Chrome promotes the bubble to a compositor layer for the transition, rasterizes the text once at the pre-transition scale, then GPU-resamples that bitmap for the duration — so text rendered blurry until the transition settled and the layer re-rasterized at 1:1. It fired on every appear: a pointer entering a trigger is by definition moving, so the first pointermove after pointerenter always set a non-zero skew and a fractional scale. - drop the velocity-reactive skew/scale flourish and the pointer-velocity bookkeeping that existed only to feed it - round tooltip position to whole pixels; clientX/clientY are fractional on HiDPI/zoomed displays, leaving the bubble on a subpixel boundary - drop the dead `filter` from the transition list — nothing ever set a filter - skip the state update when the rounded position is unchanged, so pointer jitter no longer re-renders every Tooltip.Trigger/Content consumer The 150ms ease-out translate is kept, so the bubble still trails the cursor. * improvement(tooltip): keep the velocity flourish, drive it without a CSS transition Restores the velocity-reactive skew/scale removed in the previous commit. The flourish was never the problem on its own — handing it to a CSS transition was. An interpolated fractional scale makes the compositor rasterize the tooltip's text once and resample that bitmap for the duration, which is what read as blur. Applied as a static value per pointer event instead, so every frame is rasterized at its own scale: - split the transform across the individual `translate`, `scale`, and `transform: skew()` properties, and transition only `translate` — position still eases toward the cursor, the flourish no longer interpolates - smooth the pointer velocity in JS (low-pass filter) to replace the smoothing the CSS transition used to provide, so the squish still ramps rather than snapping between raw per-event velocities - quantize the flourish to 3 decimals so jitter below the visible threshold settles instead of re-rendering every consumer Whole-pixel position rounding and the redundant-update bail-out are unchanged. * fix(tooltip): don't seed pointer velocity from the trigger box on focus The previous commit routed `onFocus` through a shared reveal helper that seeds `lastPointerRef` from the coordinates it is given. For focus those are the trigger's box center, not the pointer — so if the pointer already happened to be over the trigger, the next `pointermove` measured the box-to-cursor delta as velocity and spiked the skew/scale flourish. Split the helper in two: reveal-from-pointer seeds velocity tracking, reveal-from-element leaves it cleared. Restores the pre-PR behavior, where focus explicitly nulled the pointer snapshot. Caught by Cursor Bugbot. * fix(tooltip): make the flourish smoothing frame-rate independent The velocity low-pass filter applied a fixed coefficient per pointer event, so how fast the squish settled depended on how fast the device emitted events — 233ms at 30Hz down to 29ms at 240Hz, an 8x spread for the same gesture. It was also far snappier than the 150ms CSS ease-out it replaced, so the flourish read as twitchier than before. Derive the coefficient from the real elapsed time instead (1 - exp(-dt / tau), tau = 50ms). Settling is now flat at ~150ms from 60Hz upward, matching the duration of the transition this stands in for. Also separates the smoothing delta from the velocity-normalization delta: the latter is still floored at one frame to keep a 1ms event from reporting an enormous velocity, but flooring the former was itself a source of frame-rate dependence below 16ms. Verified against Chrome's documented re-raster behavior: a layer is re-rastered at its new scale when the scale changes via script, but not when a declarative animation interpolates it, which is why the flourish must stay out of the transition list. https://developer.chrome.com/blog/re-rastering-composite |
||
|
Waleed
|
6806fc2f41 |
fix(resources): sort folders and resources as one list so pinned items reach the top (#6213)
* fix(resources): stop hoisting folders so pinned items reach the top Folders and their sibling resources were sorted as two lists and concatenated folders-first, so a pinned file, table, or knowledge base could never rank above an unpinned folder — pinning only reordered within each partition. Sort them as one list instead: pinned -> sort key -> name, in a shared `sortResources` comparator. Rows with no value for the active column (a folder has no row count or token count) sort last in both directions, and the name tiebreak is never inverted by `desc`. Also aligns the surfaces that browse the same resources: the mothership resource trees, the collapsed sidebar flyout (whose nested levels partitioned folders-first while its own root interleaved), the sidebar file tree, and the recently-deleted tiebreak. Files' sort params were nullable only to encode "folders name/asc, files updated/desc"; with one list that state is gone, so they default to updated/desc like Tables and Knowledge. * improvement(resources): prefetch pinned ids and members with the resource lists Files, Tables, and Knowledge already prefetched their items and folders, but not the two lists a complete row needs. Pinned ids are the list's primary sort key now, so a page that painted before they arrived rendered the whole list in the wrong order and then visibly re-sorted. Members back the Owner column, which painted empty and filled in after. Both now hydrate alongside the lists via `prefetchResourceListChrome`. Pinned keys move to `hooks/queries/utils/pinned-item-keys` so the server prefetch can address them without pulling the contracts barrel and the optimistic-mutation machinery into the route. * chore(knowledge): drop the dead pre-redesign knowledge base sort `sortKnowledgeBases` was a second, complete ordering for knowledge bases (name/createdAt/updatedAt/docCount) left over from the old sort dropdown, with no consumers since the shared Resource sort menu replaced it — the kind of duplicate that silently diverges from the page it shadows. Its `SortOption`/`SortOrder` types and the `SORT_OPTIONS` list that fed it were dead with it. `utils/sort.ts` now holds only a filter, so it is renamed to `utils/filter.ts`. * fix(resources): sort unknown owners last and align the knowledge sort menu An owner id that resolves to no workspace member renders an empty cell, but its sort key was `''`, so those rows floated to the TOP of an ascending Owner sort while every other valueless cell sorted last. They now key `null` and follow the same nulls-last rule on all three pages. The Knowledge sort menu also listed Owner after Last Updated while its column order — and Tables' menu — put Owner before it. * chore(knowledge): delete the dead base-card grid view `BaseCard`/`BaseCardSkeleton`/`BaseCardSkeletonGrid` were the pre-redesign grid-card knowledge base view, kept alive only by the barrel re-export — no call sites since the Resource list replaced it. Removing it orphans `components/constants.ts` entirely (its sort types went with the dead sort; its three class-name constants had no consumers left), so that goes too. |
||
|
Waleed
|
feaddc437a |
improvement(headers): one action cluster and one ordering app-wide (#6210)
Extends the settings-header ordering past detail pages. Five headers hand-rolled their own action wrapper. `resource-header` used `flex shrink-0 items-center` — no height, no gap, so chips on tables, files, knowledge, logs and scheduled tasks sat flush against each other; the integrations tab strip and the integration block detail each used `ml-auto flex items-center`. Only the settings shell and credential detail wore the intended `flex h-[30px] items-center gap-1`. That string is now HEADER_ACTION_CLUSTER, next to PAGE_HEADER_BAR, and all five compose it. `Resource.Header` now ranks its actions through orderHeaderActions too, so the resource pages inherit the same order as settings rather than rendering their array verbatim. `ResourceAction` gains `id`, which was the only field keeping it from being a subset of `SettingsAction`. Delete is now ranked by its `id` rather than by where the caller put it. That matters for a header with no primary action: the file detail listed `Download → Share → Delete`, leaving a destructive chip in the slot a primary would occupy. Tagging it `id:'delete'` fixes that without inventing a primary. |
||
|
Waleed
|
b741176d4b |
improvement(settings): one header action order across detail pages (#6206)
Detail headers disagreed on where Delete sits. The skills page reads
`Share → Delete → Discard → Save`, but every SettingsPanel page spread
saveDiscardActions() first and appended Delete, rendering it to the RIGHT of
the primary chip: sandboxes, custom tools, custom blocks, permission groups and
data retention all did this.
Fixed in the shell rather than at nine callsites. orderHeaderActions() ranks
actions — secondary, then `id:'discard'`, then `variant:'primary'` — stably
within each band, so writing the array the natural way now produces the right
header and a new detail page cannot get it wrong. This generalizes past Save:
workflow MCP servers' `Add workflows` primary is now right-most with Delete
before it, instead of the reverse.
The ranking has to survive one trap. The shell routes onSelect through
configRef.current.actions[index] to avoid stale closures, so reordering the
render without preserving the source index would bind every chip to the wrong
handler — clicking Delete would Save. orderHeaderActions carries {action,index}
pairs; settings-header-shell.test.tsx pins that at the render level, including
the conditional-Discard case where a missing action shifts every index.
Delete is also now a plain chip on the nine resource-detail headers, matching
skills, each with a stable `id:'delete'` (three lacked one, so the chip
remounted when its label flipped to Deleting...). `variant:'destructive'` is
kept for actions destructive at scale — Delete all passwords, Clear all
browsing data, Sign out all members — which the confirm modal does not cover
the way it covers removing the single resource you are looking at.
|
||
|
|
03333ce30d |
feat(library): AEO vs GEO: What Answer Engine and Generative Engine Optimization Actually Mean (#6204)
* feat(library): AEO vs GEO: What Answer Engine and Generative Engine Optimization Actually Mean * feat(library): add generated cover for AEO vs GEO post --------- Co-authored-by: Sim Pi Agent <pi@sim.ai> Co-authored-by: Waleed Latif <walif6@gmail.com> |
||
|
Vikhyath Mondreti
|
f07160aec7 | fix(trigger): deps to include e2b and daytona (#6205) | ||
|
Waleed
|
030c4e2a5e |
refactor(auth): extract connector definitions out of auth.ts (#6203)
auth.ts had grown to 3,927 lines, of which ~2,340 were the genericOAuth
connector list — the OAuth apps a workspace connects tools to, as distinct from
the handful of providers used to sign in to Sim. Adding a connector meant
editing the same file that configures sessions, database hooks and Stripe.
Moves that list to lib/auth/connectors/providers.ts behind
buildConnectorProviders(), and relocates getMicrosoftUserInfoFromIdToken to
lib/oauth/microsoft.ts alongside the three Microsoft helpers it already depends
on. auth.ts drops to 1,489 lines and reads as auth configuration again.
Pure move, verified mechanically: the connector array is token-identical after
stripping whitespace, and all 179 template literals emit byte-identical strings
(the one apparent diff was reindentation inside a ${} expression, not text).
Behavior, evaluation order and log scopes are unchanged; the array is still
built once, when betterAuth() runs.
The explicit GenericOAuthConfig[] return type is required, not cosmetic —
inline, the entries were contextually typed by the config property. Without it
prompt: 'consent' widens to string and every getUserInfo parameter becomes
implicitly any.
|
||
|
Waleed
|
14d9542f2a |
fix(credentials): capture the correct provider identity on connect and rotate (#6201)
* fix(credentials): capture the correct provider identity on connect and rotate Attio OAuth recorded an arbitrary workspace member instead of the authorizing user, so two members connecting under one Sim user collapsed into a single account row via the stale-sibling dedupe. Notion read `profile.person.email`, which never exists on a bot token. Synthetic connector emails were minted on live third-party domains. Google service-account rotation left the credential labeled with the old key's client_email and skipped audit metadata entirely. Box and Salesforce identity lookups failed silently with no logger in either file. Service-account principals are now a single ServiceAccountPrincipal union (user / tenant / lookup_failed / null) mirrored centrally into both audit and stored metadata, so a principal can no longer be captured and forgotten, and "which account is this credential?" is answerable from SQL. * fix(credentials): keep the provider-reported name when identity lookup degrades Box and Salesforce returned early on a missing user id, discarding a `name` or `login` the response did carry and relabeling the credential to the enterprise or host fallback. Only the principal should degrade; the human label still beats an id-derived string. Also notes the Salesforce `openid` scope in the connect help text. The client credentials minter sends no scope parameter — effective scopes come from the customer's Connected App — so without `openid` the userinfo lookup can 403 and the run-as user silently never reaches the audit record. |
||
|
Waleed
|
786c854e04 |
improvement(settings): consolidate resource UI onto shared primitives (#6202)
* improvement(settings): consolidate resource UI onto shared primitives Sandboxes, MCP tools, and MCP servers each modeled their list rows and detail headers differently. Collapse them — and the surfaces they drifted from — onto one set of primitives. SettingsResourceRow now owns the row entirely: tile, title/subtitle tokens, padding and bleed, hover band, hit area, focus ring, and the one navigation chevron. Adds onClick/href (a stretched overlay, so interactive trailing controls keep their clicks), navigable, iconVariant='custom', and a badge slot for decoration that must not swallow row clicks. Rows that open a detail page get a chevron and a whole-row click; flat records keep the "..." menu. Delete moves to the detail header as a destructive chip behind a confirm modal — sandboxes previously deleted with no confirmation at all. Also folds in SettingsField (new), SettingsEmptyState tone='error', RESOURCE_LIST_STACK/GRID, RESOURCE_TILE_PLAIN, and a shared MemberAvatar; deletes DetailIconTile (byte-identical to ResourceTile); and standardizes on the emcn ArrowRight, which is a visibly different glyph from lucide's. * fix(settings): correct row bleed, delete-confirm binding, and avatar seeding Follow-up from review of the consolidation. The sandbox delete confirmation was boolean-only state. Browser Back unmounts the modal without closing it, so opening another sandbox re-opened it already confirmed — against the wrong sandbox. Reset it where the draft guard already handles the same history hazard. RESOURCE_LIST_GRID kept an 8px column gap after the bleed moved onto the row, so neighbouring cards overlapped by exactly the gutter and the right-hand card's stretched hit area won it: clicking between two cards opened the wrong one. Budget the gap for the bleed instead. Also: restore the `group` the template-icon hover outlines depend on; add a `flush` row for headings and overflow boxes; seed MemberAvatar identically on every surface; keep the MCP transport label visible in all row states; give the Delete chip a stable id so it doesn't remount mid-delete; and stop an empty subtitle rendering a phantom line. Docs: widen the rule's path globs to the surfaces it now governs, drop the `text-[14px]` example that contradicted the token rule, and add a Mode C for migrating rows onto the primitive. * revert(settings): keep the dense member roster avatar Consolidating the Teammates/Organization avatar onto the credential member row's was a redesign, not a deduplication. The two encode different things: the roster is a dense list keyed on email where the avatar is a 14px neutral marker, while a member management row carries a name, an email, and a role control and earns a 36px hashed avatar. Merging them made those rows ~70% taller, gave every workspace in a permission group a colour-hashed "avatar" seeded on its name, and cut the Add Members picker from ~7 visible rows to ~5 inside its fixed-height box. Restores both avatars and the containers that owned their bleed, and records in the rule why they stay separate. Keeps only the unrelated fix in that area: the picker row was the one settings row rounded at `sm`. * fix(settings): wire flush, drop the lone skeleton, close review gaps Final review round. Two fixes the previous commit claimed but did not land: `CredentialDetailHeading` never passed `flush`, so every credential detail heading wore list-row padding, and the empty-subtitle guard was never applied. Both were scripted replacements that silently no-op'd. RESOURCE_LIST_GRID also dropped to one column 32px earlier than the grid it replaced — `auto-fit` measures tracks, not margin boxes, so widening the gap for the rows' bleed moved the breakpoint. Track minimum now budgets for it. Removes the BYOK skeleton rather than maintaining a second copy of the row: it was the only skeleton in settings, and it had already desynced from the row it imitates. Its peers render nothing while loading. Also: unify the glyph-tile treatment across MCP/sandboxes/workflow-MCP with custom tools; move decoration out of `trailing` in verified-domains and recently-deleted; convert the last hand-rolled row and empty states in workflow-MCP, api-keys, copilot and group-detail; give copilot's delete the same `...` affordance as api-keys; announce the row description via aria-describedby, which the stretched overlay had silenced; and let SettingsField render its own value so callers stop restating type tokens. Docs: correct claims that predate this PR — `aside` does not exist, the navigation source of truth is under components/, beforeunload mounts in the layouts, `getSettingsSectionMeta` takes two args — and stop asserting a literal-pixel grep returns zero when display type legitimately uses it. |
||
|
Waleed
|
2ccda18f3e |
test(files): collaborative agent-streaming coverage (two-writer, multi-editor, undo, persist round-trip) (#6199)
* test(files): two-writer concurrent-editing regression coverage for agent streaming * test(files): multi-editor + undo + persist round-trip + late-joiner streaming integration coverage * test(files): make full-rewrite two-writer test actually exercise the concurrent peer edit * test(files): assert every peer edit lands (no false-green from a no-op peerInsertNear) |
||
|
Waleed
|
5460133e07 |
fix(files): collapse blank-line runs to markdown standard (no empty-paragraph explosion / reflow) (#6198)
parseMarkdownToDoc now strips ALL top-level empty paragraphs (leading, interior, trailing), not just trailing. A run of blank lines between blocks is insignificant in markdown (CommonMark collapses it), but @tiptap/markdown reconstructs one empty paragraph per blank line — which made the mounted editor render vertical gaps that exist nowhere else the file is viewed (GitHub, download, our own static preview), let a pathological blank run explode into thousands of empty nodes, and caused the visible reflow on open (static preview collapses empty <p>; the live editor gives each a trailing-break line). Collapsing on parse keeps normal one-blank-line spacing, matches every standard renderer, and stays idempotent so the round-trip-safety probe still reaches a fixed point (files stay editable; existing files normalize on next cold-open + save). Serializer is intentionally NOT changed — a global blank-run collapse there would corrupt blank lines inside fenced code blocks. |
||
|
Vikhyath Mondreti
|
746756d8a9 |
improvement(misc): settings organization, chat agent deploy request keys (#6195)
* improvement(misc): settings organization, chat agent deploy request keys * fix subtitle * adjust perms |
||
|
Vikhyath Mondreti
|
b8ec114382 |
improvement(chat): secrets mounting / exposure improvements and controls (#6191)
* fix(copilot): secrets injection into sandbox * improvement(chat): secrets mounting / exposure improvements and controls * fix(secrets): simplify copilot mounting flow * test(secrets): preserve standard tool permissions * fix(copilot): bind workflow tool completions * fix(secrets): preserve own environment keys * fix(copilot): release failed workflow claims * fix(copilot): trust compacted workflow completion |
||
|
Waleed
|
25e609167f |
fix(search): disambiguate tables and knowledge bases by folder (#6192)
* fix(search): disambiguate tables and knowledge bases by folder The Cmd-K search modal listed tables and knowledge bases without the folder breadcrumb workflows and files already showed, and the table, knowledge base, and search-and-replace pickers in the workflow editor rendered bare names -- so two resources sharing a name in different folders were indistinguishable. Extracts the disambiguation the workflow selector already did into shared collectDuplicateNames + disambiguateLabelByFolder, and shares the search row's folder breadcrumb and its memo comparator, which were duplicated between the workflow and file rows. Also routes folder text through filterAndCap's secondary-rank parameter rather than concatenating it into the name, so an exact name match can no longer be outranked by a folder that happens to fuzzy-match. * fix(zoho-desk): use the real Zoho Desk mark on a white tile The icon was a generic headset placeholder drawn in currentColor, so it never resembled Zoho at all. Replaces it with the mark from Zoho's official logo -- wordmark stripped, viewBox set to the mark's own bounding box so it centers -- and moves the tile to white, matching the other brand-mark integrations. |
||
|
Waleed
|
f113b0bc24 |
fix(chat): show table and knowledge base folders in the resource menus (#6190)
Tables and knowledge bases gained folder support, but the chat attachment menu and the embedded resource menu still listed them flat. Both now render the same hierarchy the Tables and Knowledge pages show. Unifies the two near-duplicate workflow/file tree builders into one shared buildResourceFolderTree plus a spec-driven useResourceTreeSections, so all four foldered families go through one code path. Items whose folder no longer resolves now surface at the root instead of vanishing. |
||
|
mzxchandraandWaleed Latif
|
87aeca6f0c |
feat(zoho-desk): add Zoho Desk integration (#6157)
* feat(zoho-desk): add Zoho Desk integration
Add a full Zoho Desk integration: tools, block, icon, and a webhook trigger.
Tools (tools/zoho_desk): list/get/update tickets, list/add comments,
list/get threads, get contact, list organizations, and download attachments
as UserFiles via an internal route. Registered in tools/registry.ts.
Block (blocks/blocks/zoho-desk.ts): operation dropdown, OAuth credential,
an organization selector backed by GET /organizations, per-operation fields,
and BlockMeta templates. Wires the Zoho Desk trigger.
OAuth (zoho-desk provider): authorize/token at accounts.zoho.com with
access_type=offline + prompt=consent; the Desk REST base is derived from the
token response api_domain and persisted so calls honor data residency instead
of assuming desk.zoho.com. Every call sends Authorization: Zoho-oauthtoken and
the orgId header.
Trigger + webhook handler (triggers/zoho_desk, lib/webhooks/providers/zoho-desk.ts):
Sim creates and tears down the Zoho Desk webhook subscription. Inbound events
are verified with JWT RS256 (X-ZDesk-JWT) against the data-center JWKS, ACKed
via the durable queue to meet Zoho's 5s deadline, and fail loudly on
Free/Standard editions that cannot create webhooks.
* fix(zoho-desk): OAuth PKCE, DC scope-marker parsing, SSRF, and e2e fixes
OAuth: forward code_verifier in the custom getToken (PKCE is enabled, so the
exchange must echo the verifier or Zoho rejects the request with invalid_request).
Surface Zoho's error/error_description, which it returns in the JSON body with
HTTP 200, instead of collapsing every failure into "no access token".
Data-center base parsing: better-auth persists Zoho's scopes comma-joined with no
spaces, so the greedy \S+ marker regex swallowed the whole scope list into the
host. Stop the capture at a comma or whitespace in both read sites (token route
and webhook handler), so apiDomain resolves to the real Desk host.
Attachment SSRF: replace the permissive host regex (which accepted attacker
domains like zoho.attacker.com) with a strict Zoho-apex suffix allowlist.
Block: guard Number() pagination so a non-numeric typo can't send NaN; add the
ignoreSourceId -> sourceId loop-guard header to update_ticket (matching add_comment).
Organizations route: surface fetch/Zoho failures with a real status instead of a
200 with an empty list, so the org selector no longer fails silently.
* fix(zoho-desk): webhook creation, attachment naming, and HTML content handling
Webhook trigger (verified end-to-end against a live Enterprise org):
- Omit ignoreSourceId; Zoho rejects a non-Zoho UUID with INVALID_DATA. Drop
the generateId() fallback and its providerConfig persistence.
- Answer Zoho's create-time notification-URL probe via the existing pending
webhook verification mechanism (GET/HEAD matchers) so subscription creation
no longer 405s.
- mapZohoWebhookError now surfaces Zoho's real errorCode / message / field
errors instead of a catch-all edition message, and attaches an HTTP status so
4xx flow through NonRetryableDeploymentError while 429/5xx stay retryable.
- Propagate the real status through deploy.ts so failed creates don't retry-loop.
get_attachment polish:
- Return the downloaded file's name under `name` (ToolFileData key) instead of
`filename`, and derive it (explicit -> Content-Disposition -> URL segment ->
fallback) so attachments are no longer stored as "untitled".
- Gate the add_comment-only `contentType` param so it isn't sent to get_attachment.
HTML content handling (Zoho content fields emit raw HTML):
- Add a Zoho-local html-to-text converter mirroring the Outlook dual-field
pattern: when contentType is 'html', derive a plain-text `contentText`
alongside the untouched raw `content` + `contentType`; plainText mirrors.
- Apply to comments (list/add), threads (list/get), the ticket description
(descriptionText), and the webhook trigger payload.
Trigger org selector: Organization is now a credential-scoped combobox that
lists the connected account's Zoho Desk organizations.
* fix(zoho-desk): review round - DC-base derivation, org-loader resilience, batched-event visibility
- deriveZohoDeskBaseFromApiDomain: preserve an already-regional desk.zoho.<tld>
api_domain instead of falling back to the US (.com) data center, and map the
DC TLD from any zoho(apis).<tld> host - keeps Desk calls in the right data
center for residency.
- fetchZohoDeskOrganizationOptions: wrap the token/org fetch in try/catch and
degrade to an empty list (the org field is a free-text combobox, so manual
entry still works) instead of hard-failing the selector on token/DC/network
errors.
- formatInput: warn (not silently drop) if Zoho ever delivers more than one
event in a single payload.
* fix(zoho-desk): harden attachment download against redirect-based SSRF/token leak
Replace the raw fetch in the attachment route with secureFetchWithValidation
(the same guarded fetch the copilot file-download tool uses). The download URL
is user/LLM-influenced and Zoho may redirect, so auto-following redirects could
send the OAuth token / orgId to an untrusted or internal host. The guarded fetch
pins the resolved IP, blocks private/reserved targets on every hop, drops the
Authorization header if a redirect leaves the origin (stripAuthOnRedirect), and
enforces the 50MB cap while streaming. The strict Zoho apex allowlist still
gates the initial origin as defense in depth.
* fix(zoho-desk): only add the edition hint when Zoho's error indicates it
mapZohoWebhookError appended the "requires Professional edition or higher"
guidance to every 403, but a 403 can also mean a wrong org, a missing scope, or
a bad token. Gate the hint on Zoho's own errorCode / message matching the
permission/edition pattern instead of the bare status, so unrelated 403s surface
Zoho's real reason without the misleading suffix. Adds a test for the
non-edition 403 path.
* fix(zoho-desk): stop duplicating /api/v1 when resolving a relative attachment href
A relative attachment href that already starts with `api/v1` (as Zoho's hrefs
often do) was concatenated onto getZohoDeskApiBase (which ends in /api/v1),
producing `/api/v1/api/v1/...` and a failing download. Extract a tested
resolveZohoAttachmentUrl helper that uses absolute hrefs as-is and strips a
leading slash + `api/v1/` prefix from relative ones before joining, so the path
is correct for absolute, root-relative, and api/v1-prefixed hrefs alike.
* fix(zoho-desk): reject an empty update_ticket PATCH with a clear error
update_ticket built its PATCH body from optional fields via filterUndefined, so
a call with no fields set sent `{}` and surfaced an opaque Zoho failure. Guard
the body builder to throw an actionable "provide at least one field" error
before the request. Adds a test for the empty and populated body paths.
* fix(zoho-desk): fall back to the credential Desk domain in webhook JWT verify
verifyAuth chose the JWKS host from providerConfig.apiDomain and otherwise
defaulted to the US host (desk.zoho.com), so a non-US webhook row missing
apiDomain would verify against the wrong JWKS and reject legitimate events. When
apiDomain is absent, resolve it from the OAuth credential's __zoho_domain__ scope
marker (mirroring deleteSubscription). The persisted-apiDomain fast path stays
DB-free to respect the 5s delivery deadline. Adds tests for both paths.
* fix(zoho-desk): apply the Zoho host allowlist to the organizations route
The organizations route built its URL from the client-supplied apiDomain and
attached the OAuth token without the https-Zoho-host allowlist the attachment
route already enforced, so a session-access caller could point the server at an
arbitrary origin and leak the token. Extract the shared isZohoHost allowlist and
an assertZohoUrl guard into tools/zoho_desk/utils (two consumers now), guard the
organizations URL before fetching, and refactor the attachment route to reuse
the shared helper. Adds tests for the allowlist and guard.
* fix(zoho-desk): propagate provider 4xx in the stable webhook prepare path
The v2 stable deploy preparation flattened every registration failure (except
path conflicts) to HTTP 500, so a provider-attached permanent 4xx - e.g. Zoho's
edition/validation failures from createSubscription - retried instead of failing
the deploy terminally. Propagate the attached status (`?? 500`), matching the
legacy save path's status-aware mapping so both deploy paths route 4xx through
NonRetryableDeploymentError.
* fix(zoho-desk): make createSubscription config failures non-retryable
createSubscription threw plain Errors (no status) for missing orgId, event type,
or credentials, and for a Zoho success with no webhook id - so the deploy outbox
mapped them to 500 and retried permanent configuration failures. Attach a 4xx
via statusError (400 for missing config/credentials; 422 for the no-id anomaly,
where a retry risks duplicate webhooks) so they fail the deploy terminally like
the mapped Zoho API 4xx responses. Tests assert the 400 status on the guard paths.
* fix(zoho-desk): enrich prevState with contentText symmetrically with payload
formatInput derived plain-text contentText only on payload, so an update event
for a comment/thread left prevState as raw HTML while payload carried
contentText - inconsistent shapes for before/after comparisons. Apply
withDerivedContentText to prevState too. Test asserts both are enriched.
* docs(zoho-desk): regenerate integration docs
Regenerate zoho_desk.mdx from the current tool definitions: removes the stale
add_comment `ignoreSourceId` input row (the field was dropped because Zoho
rejects arbitrary values) and adds the derived `contentText` / `descriptionText`
plain-text fields on comments, threads, and tickets.
* fix(zoho-desk): validate the persisted Desk base against the strict host allowlist
deriveZohoDeskBaseFromApiDomain trusted any host matching `desk.zoho.[a-z.]+`,
so a crafted api_domain like `desk.zoho.com.attacker.com` passed and was
persisted as the credential's `__zoho_domain__` REST base - later receiving the
OAuth token on every Desk tool/webhook call. Gate the derivation on the strict
isZohoHost apex allowlist (which rejects that lookalike), extracted with
assertZohoUrl into a dependency-free host-allowlist module so the auth
token-exchange path validates hosts without pulling in the tool utilities. The
attachment and organizations routes now import the shared guard from there.
Also: formatInput now emits the normalized null trigger shape for an empty/
malformed event array instead of leaking a raw `[]` to downstream steps. Tests
cover the empty-array shape and the lookalike-host rejection.
* fix(zoho-desk): correct API field names, scopes, and host validation
Validation pass against Zoho's published Desk API surfaced six defects that
typecheck, lint, and the existing suite all passed over, because each one fails
silently against the live API rather than erroring.
Wire-name mismatches (Zoho ignores unknown keys, so all three were silent):
- update_ticket sent `customFields`; the ticket PATCH body names it `cf`.
`customFields` exists only as a deprecated alias on other Desk resources and
on the separate validate-field-updates endpoint, so updates reported success
and applied nothing.
- ZOHO_DESK_TICKET_PROPERTIES and ZOHO_DESK_CONTACT_PROPERTIES advertised a
`customFields` output; both resources return `cf`. The declared field always
resolved undefined and the real one was undeclared.
- list_tickets sent `departmentId`; the query param is `departmentIds`, so the
department filter was dropped and every department's tickets came back.
Content handling:
- deriveZohoContentText matched `contentType === 'html'`, but Zoho spells the
discriminator per resource: comments use `html`, threads use the MIME form
`text/html`. Every thread's `contentText` was therefore raw markup - the exact
opposite of the field's purpose. Now normalized across both spellings,
parameterized values, and casing, with regression tests.
Scopes (least privilege):
- Desk.tickets.ALL -> Desk.tickets.READ + Desk.tickets.UPDATE. No tool creates
or deletes a ticket; ALL additionally granted ticket DELETE.
- Dropped Desk.search.READ (no search tool exists) and Desk.webhooks.READ /
.UPDATE (the provider only creates and deletes), plus their orphaned
SCOPE_DESCRIPTIONS entries.
Host validation - the webhook provider was the only token-carrying path not
anchored to the Zoho apex allowlist, including the JWKS fetch, where an
unrecognized host would have stood in as the JWT issuer:
- createSubscription, deleteSubscription, and verifyAuth now route their base
through a shared allowlist check.
- getZohoDeskApiBase validates rather than trusting injection precedence.
- The organizations route uses secureFetchWithValidation with
stripAuthOnRedirect, matching the attachment route it had diverged from.
Block and trigger:
- The trigger's department field is renamed `triggerDepartmentIds`; sharing the
`departmentIds` id let a value typed as a list_tickets filter become the
webhook subscription's filter when switching modes.
- `isPublic` no longer serializes onto all ten operations, matching the existing
gating for `contentType`.
- from/limit reject negatives and fractions instead of forwarding them.
- update_ticket gains description, resolution, and classification (all already
declared as outputs), and a departmentId input so a ticket can be moved.
Accuracy corrections to user-facing text, all against the published parameter
tables: `from` is 0-based (0-4999, default 0), not 1-based; per-endpoint limits
are tickets 1-100/10, comments 1-100/50, threads 1-200/100; sortBy lists Zoho's
actual allowed values; the two `include` sets genuinely differ per endpoint;
status and priority accept comma-separated lists.
Also: path IDs are trimmed via requireZohoDeskId so a pasted trailing space
fails with a clear message instead of a %20 404; comment `commenter` and thread
`status`/`isDescriptionThread`/`visibility`/`canReply` are now declared;
ZOHO_CLIENT_ID/SECRET added to the oauth test env; docs page gains a
MANUAL-CONTENT intro covering capabilities, the Professional-edition webhook
requirement, and the US-data-center limitation.
Not verified from documentation, needs a live account before merge:
- the OAuth scope for the attachment content sub-path (Zoho publishes none, and
there is an unanswered SCOPE_MISMATCH report against it)
- 12 of the 17 offered webhook event ids (5 are confirmed); Ticket_Delete is
documented but not offered
- the ticket `descriptionContentType` key, and the POST /api/v1/webhooks body
shape, neither of which appears in any reachable Zoho reference
* chore(zoho-desk): regenerate tool metadata
The param and description corrections in the previous commit changed the
generated tool surface, so tool-metadata:check failed in CI. Regenerated;
the diff is two Zoho-only lines.
* fix(zoho-desk): stop posting null for untouched update_ticket fields
`filterUndefined` strips only `undefined`, but an untouched subBlock never
arrives as `undefined`: the workflow serializer initializes every subBlock value
to `null` (stores/workflows/utils.ts) and extractBlockParams writes those nulls
straight into tool params, with nothing between the serializer and request.body
filtering them.
Reproduced against the real serializer and block with only `status` set:
basic {"subject":null,"status":"Closed"}
advanced {"subject":null,"status":"Closed","priority":null,...,"cf":null}
`subject` leaks even in basic mode because it declares no `mode`, so
shouldSerializeSubBlock never drops it. Zoho documents subject as a writable
field, so every status-only edit either failed the PATCH or blanked the ticket's
subject; in advanced mode the whole update surface nulled out, including `cf`.
Two things hid this. The empty-PATCH guard was unreachable from the block (the
body always carried at least `subject`), and the existing test called buildBody
with fields *absent* rather than null - the shape the block never produces - so
it could not fail on the real path.
Replaces filterUndefined with a local omitUnset that drops undefined, null, and
'' (a cleared input means "leave unchanged", not "set to empty"). Adds three
tests using the real serializer shape, all verified to fail before the fix.
Also fixes the same null-blindness in the block's param mapping, where
Number(null) === 0 injected from=0 on every operation, and corrects the shared
limit placeholder, which claimed max 100 while list_threads allows 200.
* feat(zoho-desk): add Self Client service-account credential
Adds a second way to connect Zoho Desk, alongside the interactive OAuth flow: a
Zoho Self Client, pasted as client id + client secret + organization id. Built
on the existing client-credential-accounts framework rather than a new credential
path, so it behaves like the Zoom Server-to-Server and Box CCG accounts already
in the repo - a short-lived token minted on demand, no refresh token.
Two Zoho behaviors the generic framework does not cover:
- `scope` must be COMMA-separated on Zoho's token endpoint; a space-separated
list is rejected as an invalid scope. The list comes from
getCanonicalScopesForProvider('zoho-desk'), so the Self Client and the OAuth
flow can never drift apart on scopes.
- Zoho reports OAuth failures in the JSON body, frequently with HTTP 200
(e.g. {"error":"invalid_client"}), so the success body is inspected for an
`error` field before the token is read - a status-only check would accept a
failed mint.
deriveZohoDeskBaseFromApiDomain moves out of auth.ts into the dependency-free
host-allowlist module so the minter and the OAuth path share one derivation
instead of duplicating it, and the mint response's api_domain now flows through
to tools as `apiDomain` (the SA branch of the token route previously returned
none, so SA calls would have assumed desk.zoho.com).
Docs: hand-authored zoho-desk-service-account.mdx following the existing
*-service-account.mdx pages, registered in meta.json and in the generator's
keep-list so stale-page cleanup does not delete it.
Known limitation, documented in the descriptor helpText and the docs page:
webhook triggers still require an OAuth connection. Webhook provisioning resolves
credentials through getCredentialOwner/refreshAccessTokenIfNeeded, which is
OAuth-account-only for every provider in the repo - not a Zoho-specific gap.
Unverified from documentation, needs a live Zoho org before merge:
- the `ZohoDesk.` soid prefix. Zoho documents only the syntax
{servicename}.{zsoid} with a single CRM example; no first-party doc states the
Desk prefix. normalizeZohoDeskSoid passes through any value already containing
a '.', so an operator can paste a corrected full soid without a code change.
- whether zsoid is the same identifier as the Desk orgId header value.
- whether the client-credentials endpoint accepts Desk.webhooks.CREATE/DELETE
for a Self Client.
- whether the mint response populates api_domain for Desk (documented for CRM);
if absent the derivation falls back to the US Desk host.
* fix(zoho-desk): derive descriptionText for ticket-shaped payloads
Cursor Bugbot: webhook ticket events reached workflows as raw HTML with no
plain-text sibling. `withDerivedContentText` only looked at `content` /
`contentType`, but ticket resources carry their body on `description` /
`descriptionContentType`, so trigger output disagreed with get_ticket.
The helper now derives both, which also removed two inconsistencies on the tool
side: get_ticket had its own inline copy of the derivation (now one shared
implementation that cannot drift), and update_ticket returned its PATCH response
raw despite the shared output map declaring descriptionText.
`descriptionContentType` remains the one field name unconfirmed in any Zoho
reference. It degrades safely - an absent key makes deriveZohoContentText return
the value unchanged, so descriptionText mirrors description rather than breaking,
exactly as get_ticket already behaved - and it is now one helper to correct if
Zoho names it differently.
* feat(zoho-desk): let the service account pick its data center
Zoho's accounts server is per region, and the integration pinned every call to
the US host. For the interactive OAuth flow that is currently unavoidable -
better-auth's authorize/token URLs are static per provider - but the service
account mints its own token, so the region can simply be chosen. This makes the
Self Client the only way a non-US Zoho org can connect.
Adds an optional `dataCenter` field to the client-credential framework. Optional
matters: ClientCredentialAccountFieldId and ClientCredentialAccountFields are
shared with Zoom, Box and Salesforce, whose descriptors and minters are
unchanged. Blank keeps the previous behavior (US), so existing credentials are
unaffected.
Only us/eu/in/au are offered - the four regions where both the accounts server
and the Desk REST host are confirmed. CA is deliberately absent: Zoho's accounts
docs say accounts.zohocloud.ca while Zoho's own Desk SDK says accounts.zoho.ca,
and the two cannot both be right. JP/SA/CN/UK lack a confirmed Desk host.
The Desk base is now derived from the selected region rather than inferred from
the mint response, which also removes a dependency on `api_domain` being
populated for Desk (Zoho documents it for CRM only). When `api_domain` IS present
and disagrees with the region, it wins - it is authoritative about where the
token actually works - and the mismatch is logged so a mis-selected region is
diagnosable. deriveZohoDeskBaseFromApiDomain gains a `try` variant returning
undefined so an untrusted api_domain can no longer masquerade as an authoritative
US answer and silently override a correct region.
A wrong region fails loudly rather than silently: the minter runs as verification
on both create and reconnect, so the credential is never persisted in a broken
state. Because Zoho reports it as `invalid_client` - a Self Client only exists on
its own region's accounts server - the operator hint for that code now names the
data center as a candidate cause.
Copy is scoped per path rather than blanket "US only": the OAuth service
description, trigger setup instructions, and the docs intro now say which path
each limitation applies to, and the service-account page documents the four
regions with a sign-in-domain to region-code table.
* fix(zoho-desk): strip ticket description HTML, classify body-reported refresh failures
Final validation pass findings.
descriptionText never stripped anything. It was gated on a
`descriptionContentType` discriminator that Zoho does not send: the Ticket_Add
webhook sample ships `"description": "<div>Description</div>"` with no such key,
and the ticket GET/PATCH response field lists have no content-type sibling
either. So get_ticket, update_ticket, and every webhook ticket payload emitted
descriptionText as a byte-identical copy of the raw HTML, while the declared
output promised stripped text.
The tests did not catch it because they fabricated the shape - both fixtures
constructed `descriptionContentType: 'html'`, a key Zoho never emits, proving the
branch works without proving it is ever taken. Ticket descriptions are HTML by
convention, so the strip is now unconditional (html-to-text is a near-identity on
genuinely plain text), an explicit descriptionContentType is still honored if
Zoho ever adds one, and the fixtures now use Zoho's real shape with no
content-type key anywhere.
A body-reported refresh failure was unclassified. Zoho answers a revoked refresh
token with HTTP 200 and `{"error":"invalid_client"}`; refreshOAuthToken only
checked `data.ok === false` (a Slack-ism), so the request fell through to the
"no access token" guard and returned no errorCode. isTerminalRefreshError could
therefore never recognize invalid_client as terminal, the credential was never
marked dead, and every later execution retried a refresh that cannot succeed -
with the user shown "No access token in refresh response" instead of a reconnect
prompt. The body is now classified before the status is trusted, matching what
the token exchange and the service-account mint already did. That guard also
stopped logging the whole response body, which carries live tokens on a partial
success.
Also: an unrecognized dataCenter now fails with a named error instead of quietly
resolving to US and surfacing as an opaque invalid_client (blank still means US);
the webhook JWKS cache is bounded, since its key derives from a providerConfig
field that SYSTEM_MANAGED_FIELDS protects from diffing but not from being
written; and the attachment `size` output no longer asserts bytes, a unit Zoho
documents as KB.
* feat(zoho-desk): canonical selectors and BlockMeta skills
The block picked its organization with an ad-hoc `combobox` + `fetchOptions`.
Only five blocks in the repo did that, and the other four are core blocks
(agent/credential/function/logs) - no other OAuth integration used it. Every
other resource a user has to identify was a bare short-input taking an opaque
numeric id.
Zoho Desk now uses the same machinery as the other 25 selector providers:
hooks/selectors/providers/zoho-desk/selectors.ts registered in the selector
registry, consumed from the block as basic selector + advanced manual input
sharing one canonicalParamId, for organization, update-ticket department, and
the list-tickets department filter. The trigger's org field moves to the same
selector. zoho-desk-org-options.ts is deleted rather than left beside the new
path, so blocks/ has zero fetchOptions usages outside the core blocks.
Wire params are unchanged (orgId, departmentId, departmentIds, assigneeId,
ticketId, contactId) - this is a UI change, not an API change.
The organizations route now resolves the credential server-side. It previously
had the browser fetch an access token and POST it back, which an earlier audit
flagged as the one place a Zoho token left the server; the new selector-credential
resolver keeps it server-side for both the OAuth and service-account credential
types and re-anchors every outbound host to the Zoho apex allowlist.
No agents selector: the endpoint is documented but its OAuth scope is not, and
the nearest evidence points at Desk.agents.READ, which we do not request. Adding
it would force every existing Zoho Desk user to reconnect for a convenience
field, so assigneeId stays a manual input until the scope can be confirmed
against a live org.
Adds the skills array BlockMeta was missing - 227 of 300 blocks declare one and
this did not. Seven skills, each grounded in a use case Zoho or the ecosystem
actually advertises (auto-triage, SLA escalation, digest, AI draft reply,
customer context, engineering handoff, knowledge-gap report) and each exercising
only tools in tools.access. CSAT surveys, ticket creation, dedup and keyword
search were deliberately left out: the integration has no tool for them, and a
skill implying an unsupported action is worse than a shorter list.
* feat(zoho-desk): agents selector and free-text trigger organization
Three improvements that were previously deferred only to avoid forcing existing
users to reconnect or orphaning saved workflows. This integration is unmerged and
has no users, so the constraint does not apply and the better option wins.
assigneeId was the last field still asking for an opaque numeric id. It is now a
canonical selector pair backed by a new zoho_desk.agents selector, which required
adding the Desk.agents.READ scope - the reason it was skipped before. Route
follows the departments one exactly: auth before parseRequest, host anchored to
the Zoho apex allowlist, secureFetchWithValidation with stripAuthOnRedirect, and
a page drain capped at 20 pages with 204 treated as end-of-list.
Scope caveat: Zoho publishes no explicit scope line for the list-all
GET /api/v1/agents. Every other endpoint in the Agents module documents
Desk.agents.READ (get by id, get by email, roles/{id}/agents), and it is the only
agents-module scope Zoho defines, so that is the basis. Inference across a module
rather than a direct quote - worth one live call before merge, same as the
existing attachment-scope note.
The trigger regained free-text organization entry, lost when the org field became
a selector. The earlier concern - that a manual value would land under its raw
subBlock id and never reach the provider - turned out not to hold: buildProviderConfig
already collapses canonical pairs and writes the active member under the canonical
key. The real gap is narrower and does exist: when canonicalModes pins the group
to basic while only the manual field has a value, the collapse deletes the
canonical key even though the required-field check passes, so the deploy succeeds
and then fails at subscription time. resolveConfigOrgId closes that, with a test.
The block/trigger `orgId` id overlap stays shared, now with a comment. Two earlier
audits disagreed; renaming turns out to be the wrong call. buildCanonicalIndex has
an explicit guard for trigger-mode reuse and blocks.test.ts codifies it as a valid
pattern, orgId means the same portal in both modes (unlike departmentIds, which is
correctly distinct), and a separate triggerManualOrgId would put two advanced
members in one canonical group - getCanonicalValues takes the first non-empty, so
a stale tool-mode value could silently supply the trigger's organization.
* fix(zoho-desk): make the attachment cap reachable, unbreak selector paging
Final audit round.
The 50 MB attachment ceiling could never be hit. This route returns the file as
base64 inside its JSON body, and the executor reads internal tool responses
through readToolResponseBody, capped at 10 MB. Base64 inflates 4/3, so ~7.5 MB
of raw bytes is the real ceiling - and the old limit meant a larger attachment
was downloaded, encoded and serialized in full (peaking near 250 MB of live
allocation, with nothing bounding concurrent downloads) purely to be rejected
afterwards. The cap is now the reachable size, so the limit enforces itself while
the bytes are still streaming, and an overflow returns 413 with the actual
ceiling instead of a generic 500. Raising it properly means uploading in the
route and returning a file reference, as the WhatsApp media route does - not a
bigger constant.
Selector paging assumed a 0-based `from`. Zoho's docs contradict themselves:
the pagination section says "range 0-4999, default 0" while the listing examples
read as 1-based ("from=5 and limit=50 retrieves records 5 to 54"). Under the
1-based reading, stepping by exactly the page size re-fetches the boundary record
and the dropdown shows a duplicate per page. Rather than pick a base that cannot
be confirmed without a live tenant, the department and agent drains dedupe by id,
which is correct under either reading.
The organization list was unpaginated, and Zoho's listing APIs default to ten per
page. An account with more accessible portals silently got a truncated dropdown,
and since every other selector and every tool call is gated on orgId, a missing
portal was unreachable except through the advanced manual field. Both the
selector route and list_organizations now request the documented maximum.
Docs: regenerated so the trigger table includes manualOrgId, and two
service-account claims are hedged to match what the code already says it cannot
verify - that zsoid equals the Desk orgId header value, and that every tool works
under the requested scopes (Zoho publishes no scope for the attachment content
sub-path).
Also: status and priority move out of advanced mode - they are the fields most
often changed on a ticket update; the custom-fields wand prompt now ends with the
required "Return ONLY" clause; and the shared-orgId rationale comment cites the
mechanism that actually applies (buildCanonicalIndex dedupe plus the first-non-
empty rule in getCanonicalValues) rather than a blocks.test.ts branch that never
evaluates this pair.
* fix(zoho-desk): five-audit round - serializer trigger-advanced leak, scopes, paging
Five independent audits (OAuth/scopes, tools-vs-docs, block/selectors,
blast-radius, /validate-trigger). Findings, most severe first.
A trigger-mode field was a live tool-mode required param. `shouldSerializeSubBlock`
excluded `mode: 'trigger'` but not `'trigger-advanced'`, so the trigger's required
`manualOrgId` validated on every tool operation. Reproduced against the real
serializer: with the Organization field pinned to advanced, running
List Organizations failed with "Missing required fields: Organization ID" - a
field that operation does not even render, and which the user could not clear
without switching operations. Fixed in the serializer rather than locally,
because the Google Sheets/Drive/Calendar pollers have the identical shape.
`limit=200` on /organizations was an undocumented parameter I added by
extrapolating from /departments and /agents. Zoho documents NO parameters for
that endpoint and its sample is a bare GET; the other siblings cap at 100 and
Zoho answers out-of-range with 422. Since orgId gates every tool and both other
selectors, a 422 there would have made the whole integration unreachable. Reverted
to Zoho's documented shape.
`descriptionText` was HTML-stripping plain text. The previous round made the strip
unconditional after finding Zoho sends no `descriptionContentType`, but Zoho's REST
samples show plain descriptions while only the webhook payload is HTML - and the
webhook path runs this over contact/account/department bodies too. html-to-text is
not identity on plain text: it decodes entities and deletes tag-shaped content
("a < b > c", XML snippets). Now sniffs for markup first.
`omitUnset` made every documented field-clear impossible. Zoho's own PATCH sample
uses `"classification": ""` and `"productId": ""` to clear. Dropping `''` meant no
scalar field could be cleared. Now drops only undefined/null - the serializer-null
case it was written for - and forwards `''`.
status/priority leaked between operations. One shared subBlock served both the
list_tickets filter and the update_ticket value, and subBlock values survive an
operation switch, so a filter of "Open,On Hold" could be PATCHed onto a ticket and
an update value could silently filter a later list. Split per operation.
Auth: `invalid_code` added to TERMINAL_ERRORS - it is Zoho's code for a revoked
refresh token, so without it the previous round's refresh fix never actually
dead-flagged the credential it was written for. The shared refresh body-error
branch now also requires `!data.access_token`, so no provider can have a
successful refresh misclassified. The token route now uses the validating
`extractZohoDeskBaseFromScope` instead of a private regex with no https/allowlist
check - that value is injected into every tool call. Scope list falls back to the
requested scopes when Zoho omits `scope`, which would otherwise flag every
credential as needing reconnect. The Self Client mint no longer sends
`aaaserver.profile.READ`, a scope that grant never uses.
Trigger: `includePrevState` now set for every *_Update event, not just tickets -
it defaults to false, so prevState was permanently null for contact/agent/task/
article updates while the trigger advertised it. `departmentIds` is only sent for
events Zoho documents as accepting it, and the field is conditioned accordingly.
Empty filters serialize as `null`, matching Zoho's examples, rather than `{}`.
JWKS fetch bounded to 1.5s - jose's default is 5000ms, exactly Zoho's whole
delivery deadline, and Zoho publishes no retry. The create-time validation POST
fallback is now matched by the pending-verification probe. Ticket_Delete added.
All 17 webhook event ids, the POST /api/v1/webhooks body contract, and the JWT
claim/JWKS specifics are now confirmed verbatim against Zoho's webhook
documentation - previously 12 of 17 events and the entire subscription contract
were unverified.
* revert(zoho-desk): back out both shared lib/oauth changes
Reverting two changes to shared OAuth code because their premise is inferred
rather than proven, and neither meets the bar for touching a path every provider
runs.
`refreshOAuthToken` body-error branch. The premise was that Zoho reports refresh
failures with HTTP 200 and an `error` body. That is documented and empirically
confirmed for the authorization-code EXCHANGE (see the comment on getToken in
auth.ts), but I never confirmed it for the REFRESH grant specifically - and if
Zoho returns a proper 4xx there, the existing `!response.ok` path already
classifies it via extractErrorCode, making the branch dead code that every one
of the ~34 providers still executes on each refresh. A shared branch whose only
justification is an unverified inference about one provider is not worth its
blast radius.
`invalid_code` in TERMINAL_ERRORS. Same problem, worse downside: the code is
sourced from a Zoho community post rather than official docs, TERMINAL_ERRORS is
consulted for every provider, and a false positive marks a credential dead for an
hour. Not adding it simply preserves today's behavior (retry rather than
dead-flag), so reverting costs nothing that was previously working.
Both are cheap to reinstate, correctly scoped, once a live Zoho account shows
what a revoked refresh token actually returns.
Kept: the token-redaction on the "no access token" warn, which is an unambiguous
improvement independent of Zoho.
Also kept, deliberately, is the serializer `trigger-advanced` exclusion - that one
rests on a reproduced bug rather than an inference, and it aligns the serializer
with the convention the rest of the codebase already follows (blocks.test.ts
treats `trigger` and `trigger-advanced` identically in six places, as does the
copilot block-metadata tool, and blocks/types.ts documents trigger-advanced as
"the advanced side of a trigger field").
* fix(zoho-desk): carry the stored data center through a credential reconnect
A reconnect rebuilds the service-account secret blob from the submitted fields
only, and the connect modal never prefills - correctly, since for every other
field in this family the stored value is a secret the admin must retype. The
data center is the first non-secret member of that set, so it was being silently
dropped: rotating a client secret on an EU/IN/AU credential moved it back to the
US accounts server, where the next mint fails with an opaque invalid_client.
performUpdateCredential now reads the stored dataCenter out of the existing blob
when the caller does not supply one. The read is failure-tolerant - an
undecryptable or unparseable blob yields undefined rather than throwing, so it
can never block a reconnect, and the provider default applies as before.
Raised independently by three reviewers; I twice argued it was acceptable because
the mint fails loudly rather than corrupting silently. That was true and beside
the point - the operator still had to guess why.
* fix(zoho-desk): delta-audit findings - prevState scope, status leak, HTML sniffer
An audit of the commits the earlier five audits never saw. All four findings are
in code written as fixes for those audits, which is where this branch has
repeatedly introduced new problems.
`includePrevState` was sent for Ticket_Comment_Update. The previous commit gated
it on an `_Update` suffix and claimed Zoho supports it on every update event.
Zoho's webhook doc lists the attribute on Ticket/Contact/Agent/Task/Article update
events but NOT on Ticket_Comment_Update, which documents only `departmentIds`.
That made it an undocumented filter key on a live subscription create - the same
class of risk the same commit reverted `limit=200` for, so it failed that commit's
own stated bar. Now an explicit set rather than a suffix rule.
The status/priority split did not stop the leak it was written for. The mapping
used `operation === 'list_tickets' ? filterValue : updateValue`, whose bare else
covers all eight other operations - so a stale Update Ticket status was forwarded
into get_ticket, list_comments and the rest. Harmless on the wire (those tools
ignore it) but exactly the stale-value pattern the neighbouring gates exist to
prevent. Both fields are now scoped to the two operations that declare them.
The HTML sniffer destroyed plain text. `/<[a-z!\/][^>]*>/` fires on any `<`
followed by a letter with a later `>`, so realistic ticket bodies lost content:
"if x<y then z>0" became "if x0", and "replace <username> with the real name"
lost the placeholder. It now requires a real element - a paired tag, a
self-closing tag, a comment/doctype - or an entity, and the entity arm covers hex
references it previously missed. Regression tests verified by reverting to the
loose pattern and watching them go red.
The reconnect data-center carry-forward is scoped to client-credential providers.
As written it added a DB read plus a decrypt to every service-account reconnect
for every provider - Slack, Atlassian, all token-paste providers - to carry a
field only Zoho has.
Also: the JWKS cache-bound TSDoc had been orphaned onto the wrong constant by an
earlier insertion, and `cooldownDuration` was dropped since it restated jose's
default while only `timeoutDuration` needed justifying.
* test(zoho-desk): cover the webhook subscription filter rules
The subscription filter logic had no test coverage at all, and it is where the
last two rounds both found bugs - includePrevState on an event Zoho does not
document it for, and departmentIds sent to events that accept no filters.
Adds six cases against the real createSubscription: includePrevState is set for
each of the five documented update events and NOT for Ticket_Comment_Update,
departmentIds is kept for a filterable event and dropped for one that is not, and
an event with no filters serializes as null rather than an empty object.
Verified the guard bites: reverting PREV_STATE_EVENTS to the `endsWith('_Update')`
rule turns the Ticket_Comment_Update case red.
The Ticket_Comment_Update assertion checks the with-departments case as well as
the bare one - asserting only `not.toHaveProperty` on the bare filter would pass
vacuously, since that filter is legitimately null.
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
|
||
|
Waleed
|
69289d26e7 |
fix(execution): stop run-buffer failures from stranding resumed executions (#6187)
* fix(resume): stop a failed run-buffer publish from stranding a resumed execution The run buffer is a replay convenience for stream readers; the durable execution record is authoritative. A failed terminal event publish was deciding the outcome of work that had already run: it threw, the resume was marked failed, and the paused execution was left paused forever. That path is also not retryable — the workflow had already executed — so the next resume attempt re-ran its side effects. Degrade instead. Record the terminal status on the stream meta so readers are not left polling an 'active' stream, log the failure, and let the resume settle on its real result. The same treatment applies when the terminal event is never published at all, which previously synthesized an error for the same stranding effect. Removes the now-unreachable TERMINAL_PUBLISH_ERROR constant and its branch. Pre-execution buffer failures stay fatal and retryable, since no work has happened yet. * fix(execution): give per-user Redis budget keys a fixed window The per-user byte budget refreshed its TTL on every accepted write, so for any user who never went a full TTL without writing, the key never expired. The per-execution data it accounted for kept expiring underneath it, so the counter accrued bytes Redis had already dropped and drifted toward the ceiling. On reaching it, every subsequent write for that user was rejected until they stopped writing for a full TTL — and since a rejected write does not refresh the TTL, it recovered on its own and then refilled. User keys now get a fixed window: the TTL is set when the key is created and never extended. Applied to all five Lua scripts that touch the key so the writers cannot drift apart. Execution-scoped keys keep sliding — they are refreshed on the same schedule as the data they account for, so their counter and bytes stay in step. * fix(execution): heal a user budget key that somehow has no expiry The no-op branch of the flush script dropped the user-key EXPIRE outright rather than guarding it like every other site. No current path can create the key without an expiry, but if one ever did, that branch was the one place that would never give it one — and a user counter with no expiry is the unbounded version of the bug this series fixes. Guard it instead, so the branch heals such a key rather than skipping it, and so all five scripts read the same way. * fix(stream): end a replay cleanly when terminal metadata has no terminal event Terminal metadata is the authoritative end-of-run signal — the happy path writes it atomically with the terminal event, and a run whose terminal event could not be buffered records the status on its own. The reader required both, so the degraded case threw and turned a replay that was merely incomplete into a broken one. Metadata without a matching event means the buffer degraded, not that the run is still going. Log it and close the stream: the reader has already received every event that was buffered, and the durable record is unaffected either way. |
||
|
|
13772565a3 |
fix(chat): show deployment passwords to admins (#6177)
* fix(chat): show deployment passwords to admins
* fix(chat): reject whitespace-only passwords
* fix(chat): preserve password visibility on regenerate
* fix(chat): harden password reveal and close deployment lockout paths
Follow-ups from a security review of the password reveal endpoint. The
permission model itself was correct — the reveal is gated on workspace
admin via the canonical resolver, so derived org-admin access is honored.
These address secret handling and validation around it.
- Cap set-path passwords at the same 1024 chars the chat login accepts.
Neither the input nor the schema bounded length, so a longer password
saved fine and then failed the login POST on length before auth ran,
locking every visitor out permanently.
- Discard the revealed password when the field is hidden. It previously
stayed in state and in the input's DOM value with Copy still armed, so
the field read as hidden while still handing out the plaintext.
- Evict the decrypted password from the mutation cache on unmount, and
correct the TSDoc claiming it was never retained — it sat in the
MutationCache for the default five minutes after the modal closed.
- Validate the password inside performChatDeploy, the writer both callers
must use. The copilot deploy_chat tool bypasses the route contract and
could still store a whitespace-only or over-long password, or create a
password-protected chat with no password at all.
- Stop echoing raw decryption errors from the reveal endpoint.
- Only persist a new password when the chat ends up password-protected;
PATCH { authType: 'email', password } used to re-arm the secret that the
auth-type branch had just cleared.
Also replaces the hand-rolled copy state with useCopyToClipboard, which
fixes an unawaited clipboard write that surfaced as an unhandled rejection
and a "Copied" confirmation shown even when the write failed.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(chat): correct password-change confirmation gate and stale reveal error
Addresses both open Bugbot findings.
- shouldConfirmPasswordChange keyed on "a chat exists" rather than "a password
exists", so switching a public chat to password protection for the first time
asked the admin to confirm changing a password that was never set. It now
takes the existing-password signal the component already computes.
- A failed reveal left "Failed to load the current password" on screen while the
admin typed or generated a replacement, because the mutation only drops its
error on the next attempt. Editing or regenerating now resets it.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Waleed
|
797f781855 |
chore(deps): bump @opentelemetry/core to 2.8.0 for CVE-2026-54285 (#6182)
W3CBaggagePropagator.extract() had no size limits on inbound baggage headers. Bumps the OTel core packages to 2.8.0 and the exporter/sdk-node packages to 0.219.0, which pin core 2.8.0 so no vulnerable copy is nested under them. |
||
|
Waleed
|
a76b928771 |
fix(security): bind copilot chat attachment keys to their owner (#6179)
* fix(security): bind copilot chat attachment keys to their owner `POST /api/copilot/chat` accepted a client-supplied `fileAttachments[].key` and passed it straight into `trackChatUpload`, which wrote it into the `workspace_files` ownership binding with no permission check and no key-ownership validation. That binding is what `verifyFileAccess` and the Files feature resolve authorization from, so any workspace member — including read-only — could hand in another member's key and have their file re-parented to a private chat: removed from the Files listing, from folders and from download-by-id, and destroyable through the chat-delete FK cascade. The sibling register route already enforced these invariants; the copilot path did not. - `trackChatUpload` now rejects keys that do not address the target workspace, only re-links a chat-upload row the caller already owns (matched by row id, not by raw key), and only mints a new binding when the key has no prior record at all — including soft-deleted ones, which the partial active-key unique index would otherwise let it insert over. - Minting a new binding verifies the object exists in storage, matching `registerUploadedWorkspaceFile`. - `buildCopilotRequestPayload` gates attachment tracking on write/admin, the same grant the upload routes that issue these keys require. * fix(security): make the chat-upload ownership check atomic with its write The ownership lookup and the binding UPDATE were separate statements, and the UPDATE matched on the captured row id alone. A concurrent `materialize_file` sets `context='workspace'` and clears `chatId` on that same row, so the tracking write still matched and dragged the saved file back into chat scope — hiding it from every workspace-file listing and re-exposing it to the chat-delete cascade, with materialize's storage-usage increment left stranded. Re-assert every ownership predicate in the UPDATE so the statement is itself the atomic check. The lookup now only decides UPDATE-vs-INSERT and is no longer load-bearing for authorization, which also makes the existing `updated.length === 0` fail-closed branch correct rather than dead. * fix(uploads): tolerate transient storage probes and reject unowned keys with 403 Follow-ups from a backward-compatibility audit of the attachment-key hardening. The existence probe is hygiene, not authorization — the key-format and no-prior-record guards already carry that, and a binding to a nonexistent object grants nothing readable. But `headObject` rethrows non-404 provider errors, so a transient 5xx or throttle would drop a legitimate attachment. Only a definitive not-found now rejects; a thrown error logs and proceeds on the ownership guards. This path is reached solely by >50MB multipart uploads, the one flow that persists no metadata row at upload time. The stage route mapped an ownership rejection to a 500. It is a client error; return 403 instead. * fix(security): compare-and-swap the chat binding on chat uploads Two overlapping chat requests could both observe the same claimable row with `chat_id IS NULL` and both satisfy the update predicate, so the later write silently moved the upload to its own chat — taking over the delete-cascade lifecycle of a file the first chat had already bound. Scope the update to `chat_id IS NULL OR chat_id = <target>` so the statement is a compare-and-swap: the loser matches zero rows and fails closed. The resolver applies the same rule so the update-vs-insert decision stays coherent. This also makes an upload bind to exactly one chat, matching the 409 the sibling `local-files/stage` route already returns for the same case; verified no client flow relinks a key across chats (drafts are per-chat, every retry path replays under a pinned chat id, forking copies blobs to fresh keys). * refactor(copilot): gate attachment tracking with the shared permission predicate `permissionSatisfies` is the documented single source of truth for permission comparisons and exists to replace hand-written `=== 'admin' || === 'write'` ladders. `userPermission` is typed `string` for legacy reasons, so narrow it with `isPermissionType` first — an unrecognized value fails the gate instead of ranking below every level. Behavior is unchanged for all three levels. Imported from the dependency-free `/predicates` subpath rather than `/workspace`, which would pull `@sim/db` onto the chat request path. |
||
|
Vikhyath MondretiandClaude
|
e90378a5c1 |
improvement(permissions): confine workspace role changes to existing members (#6180)
* improvement(permissions): confine workspace role changes to existing members * improvement(permissions): retry lock timeouts and audit admin upserts truthfully Bounding the row-lock wait with `lock_timeout` made a competing writer raise 55P03, which the retry set excluded — so the bounded wait failed the request where the unbounded one would have waited. 55P03 is now retried like the other contention aborts, the timeout is shortened since each attempt releases its pooled connection, and contention that outlives every attempt answers with a busy conflict instead of a generic server error. The admin member upsert recorded MEMBER_ADDED even when the conflict branch amended an existing membership, putting a join that never happened in the workspace audit trail. It now records a role change, matching the action the response already reported. Adds the missing test coverage for withTransactionRetry. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Waleed
|
9064039c19 |
improvement(logfire): scope block outputs per operation and refresh brand chrome (#6178)
* improvement(logfire): scope block outputs per operation and refresh brand chrome - gate each block output on the operations that actually return it - swap in the official Logfire mark, black tile with brand-magenta bare icon - move host to advanced mode and alphabetize the tool registry entries - add track-logfire-llm-cost and verify-logfire-token-target skills * fix(logfire): honor numeric-string limits and surface token validity fields - accept a numeric-string limit so agent-invoked calls stop silently falling back to Logfire's 100-row default - keep an hour-only UTC offset intact instead of producing +05Z - surface expiresAt and spendingCapReachedAt on Get Token Info - document pending_span as a fourth record kind * chore(logfire): regenerate tool metadata and document the step in the skill - regenerate apps/sim/tools/generated/tool-outputs.ts, which CI's tool-metadata:check requires after a tool output change - add the regeneration step and artifact-diff guidance to the validate-integration skill so the gate stops being missed * chore(skills): sync validate-integration projections |
||
|
Theodore LiandClaude Opus 5
|
8348592d38 |
fix(copilot): close fail-open write gates in agent tools (#6132)
* fix(copilot): make tool write gates fail closed The three handler-map management tools (manage_custom_tool, manage_mcp_tool, manage_skill) gated writes with `context.userPermission && perm !== 'write' && perm !== 'admin'`. userPermission is optional on the execution context, so an absent value skipped the check entirely and the write proceeded unguarded — while the server-tool router's equivalent gate is fail-closed. Both paths now share copilotToolCanWrite, built on the canonical permissionSatisfies (null/undefined never satisfies). manage_custom_tool additionally resolved its target as `params.workspaceId || context.workspaceId`, so a model-supplied workspace id won while the permission check was resolved for the context workspace — and upsertCustomTools performs no authz of its own. It now uses the server-set context only, matching its two siblings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiHhAk2R1NryaS3R8n2yFz * fix(copilot): gate materialize_file writes and enforce the deploy mutation lock materialize_file's save/extract/import all create workspace resources, but the handler-map path has no central permission check, so a read-only member could create files and workflows through the agent. Gate on write access after param validation. assertWorkflowMutable moves into performFullDeploy / performFullUndeploy / performActivateVersion, where performRevertToVersion already had it. The check previously lived only in the deploy routes, so the copilot deploy tools — which call the orchestration functions directly — could deploy, undeploy, and activate versions of a locked workflow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(copilot): enforce secret-admin on env writes and gate KB indexing on usage Two more paths where the agent is a weaker route to the same write than the UI. upsertWorkspaceEnvVars was a weakened copy of the environment route's write: no per-key secret-admin check, no advisory lock, no audit row. Its only caller is the set_environment_variables copilot tool, which gates on workspace 'write' alone — so any write-level member could have the agent overwrite a workspace secret they do not administer, with nothing in the audit log and a lost-update race against the route's locked transaction. The gate now lives in the function so every caller inherits it, reusing getWorkspaceEnvKeyAdminAccess rather than restating the route's logic. knowledge_base add_file computed a billing attribution and then indexed without calling checkAttributedUsageLimits, which every upload route applies before accepting indexing work — an over-quota workspace could index without limit through the agent. The same file's query operation already gated correctly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(copilot): return lock denials and stop bootstrapping a legacy secret's ACL Two defects in the previous commits, both found by review. assertWorkflowMutable throws, and the three orchestration entry points awaited it outside any try/catch, so a locked workflow escaped as an exception instead of the { success: false } result the callers consume — performChatDeploy and the copilot deploy tools would have surfaced a generic 500 rather than a lock denial. performRevertToVersion already converted it; the three now do too, through a shared helper. upsertWorkspaceEnvVars derived newKeys for createWorkspaceEnvCredentials from the credential rows rather than the stored variables. A secret written before credential rows existed has no ACL, so overwriting it looked like adding a new key: it minted a credential and made the caller that secret's admin. The environment route derives newKeys from the locked jsonb read for exactly this reason, and now so does this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style(env): collapse the merged test import onto one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCXbU36FwJBmtJKaH83BUm * fix(env): give the workspace env denial its own write-access message WorkspaceEnvAccessError reported "must be an admin of these secrets" for both denials, so a caller lacking workspace write to ADD a key was told to get secret-admin on a key that does not exist yet. Carry the reason and mirror the route's two messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCXbU36FwJBmtJKaH83BUm --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Waleed
|
30820fcbca |
fix(copilot): guard document-style extraction against zip bombs (#6176)
* fix(copilot): guard document-style extraction against zip bombs extractDocumentStyle handed an attacker-controlled archive straight to JSZip and inflated named parts (word/theme/theme1.xml, word/styles.xml, ppt/presentation.xml, ppt/slideMasters/slideMaster1.xml) with no size bound, reachable from GET /api/workspaces/[id]/files/[fileId]/style and from the workspace VFS. Reading only a few entries is no protection: the bomb just has to live at one of those paths. It now calls assertOoxmlArchiveWithinLimits, the same guard the document parsers use, and the hand-rolled ZIP_MAGIC check is replaced by isZipShaped from that module — zip-guard is already shared this way by lib/uploads/archive.ts and lib/copilot/tools/handlers/upload-file-reader.ts. Checking each entry's size through JSZip instead would have been cheaper, since only a handful of parts are read, but JSZip reports the size the archive declares — the same attacker-controlled field a bomb lies about — so it would need to re-derive the guard's verification to be sound. The guard sits inside the existing try, so a rejection logs and returns null: the route already answers 422 and the VFS already returns null when no summary can be produced, and neither caller changes. * test(copilot): declare the ratio-test expansion instead of carrying it The compression-ratio case built a real 400 MiB string and deflated it synchronously, costing the parallel test runner memory and CPU for no added coverage. The guard reads the total the archive declares, so declaring the expansion exercises the same ratio path with a few-hundred-byte fixture. Still fails when the guard call is removed, and the file now runs in 286 ms instead of seconds. Caught by Greptile review. |
||
|
Waleed
|
51dd0df4ca |
fix(files): authorize presigned upload contexts per type (#6175)
The batch presign endpoint validated its type param against the shared seven-value upload enum while only authorizing knowledge-base, so any authenticated user could mint an S3 presigned PUT into workspace-logos, profile-pictures, execution, mothership, chat and copilot prefixes — objects that are then served unauthenticated from the app origin with a one-year public cache. The single presign endpoint had the same gap for chat, which has no authorization predicate and no client. - Batch presign now accepts only knowledge-base, and always requires a workspaceId the caller has write/admin on before anything is minted - Single presign drops chat from its accepted contexts; every remaining context has a per-context predicate - Both allowlists live in the contract module so the enforced enum and the documented one cannot drift apart again |
||
|
Waleed
|
e5d2f7d80a |
fix(realtime): stop read-only members persisting block positions (#6174)
* fix(realtime): stop read-only members persisting block positions The socket operation ACL granted the `read` role block.updatePosition and blocks.batchUpdatePositions on the premise that they were ephemeral cursor sync. They are not: both are followed by persistWorkflowOperation, which UPDATEs workflow_blocks.positionX/positionY and bumps workflow.updatedAt. A member holding only `read` on a workspace could therefore permanently rewrite the coordinates of every block of every workflow in it — a write across the read/write boundary. Live cursors ride their own `cursor-update` event, and the smooth-drag broadcast is the UNCOMMITTED position path, which returns before persisting and never consults the role table — so the read role needs no grant at all. * test(realtime): assert the role ACL against production, not a fixture The shared ROLE_ALLOWED_OPERATIONS fixture still listed the two position operations for the read role, and three tests compared that fixture against itself — so they certified whatever it said, including the grants this PR removes. They now assert checkRolePermission over the protocol's complete operation list (the fixture's copy omits subblock/variable/admin-only ops), plus one test that pins the fixture to the production ACL so the two cannot drift apart again. |
||
|
Waleed
|
ccc2ec9507 |
fix(file-parsers): guard .doc uploads against zip-bomb memory exhaustion (#6166)
* fix(file-parsers): guard .doc uploads against zip-bomb memory exhaustion DocParser handed the raw upload straight to officeparser and then mammoth, both of which inflate every ZIP entry into memory before any app-level size cap applies. The extension is only a routing hint, so a bomb-bearing OOXML archive renamed to .doc selected the one parser that skipped the guard its docx/pptx/xlsx siblings all call. Adds assertOoxmlArchiveWithinLimits to DocParser.parseBuffer, and centrally in file-parsers parseBuffer so a future parser cannot silently opt out. The guard reads the central directory's declared sizes without decompressing, and no-ops for non-ZIP buffers, so legacy OLE .doc files are unaffected. * fix(file-parsers): verify actual inflation, not just declared ZIP sizes The declared uncompressed sizes in a ZIP central directory are attacker- controlled, so a bomb can under-report them and pass the size and ratio checks untouched. officeparser and mammoth only detect the mismatch after inflating the entry in full: a 498 KB archive declaring 1000 bytes per entry drove 559 MB resident through the .doc parser and 538 MB through .docx, then failed. SheetJS and officeparser reject the container earlier, so xlsx/pptx were not affected, but doc and docx both were. Each entry is now inflated during verification under a maxOutputLength bound equal to the size it declared. Node's zlib aborts the moment output would exceed that bound, so a lying entry costs only its declared size and the inflated bytes are discarded immediately; both bomb variants now reject at +0 MB across every extension. Stored entries are checked against their own compressed size, and unsupported compression methods fail closed. Verification walks the contiguous run of central-directory records rather than the EOCD's declared entry count, since that run is what a decompression library allocates per entry — a lied-down count must not hide an entry from verification. Cost is ~0.45 ms per MB of uncompressed content (22 ms for a 50 MB archive), against parse times an order of magnitude larger. All 17 real Word-produced .docx fixtures in mammoth's test data are still accepted. * fix(file-parsers): require central and local ZIP headers to agree The parsers disagree about which header to trust. JSZip skips the local header outright and decompresses using the central directory's method, while SheetJS's parse_local_file switches on the local header's method and inflates from there. An entry claiming STORED centrally and DEFLATE locally therefore took the guard's stored branch, skipping bounded inflation, and was still expanded downstream — a 398 KB archive hiding a 400 MB deflate payload. Verification now rejects any entry whose two headers disagree on compression method, and on declared sizes when the local header carries them (the data-descriptor flag and ZIP64 sentinels legitimately omit them, and those entries stay covered by the bounded inflate). Caught by Greptile review. All 17 real Word-produced .docx fixtures in mammoth's test data are still accepted. * fix(file-parsers): charge hidden central-directory entries against the cap sumDeclaredUncompressedSize walked only the entry count the EOCD declares, while verification walks the contiguous run of records. JSZip's readCentralDir loops on the record signature and keeps every entry it finds — a count mismatch is explicitly not an error there — so an archive that under-reported its count could hide honestly-large entries from the total-size cap and still have the parser expand them. The sum now walks the same contiguous run as the verification pass and readZipCentralDirectoryStats, and fails closed when the run is shorter than the declared count. Caught by Cursor Bugbot review. |
||
|
Waleed
|
5686b7bb83 |
fix(realtime): enforce room access continuously, not only at join (#6170)
* fix(realtime): enforce room access continuously, not only at join File-doc, table, and workspace-list rooms authorized once at JOIN and never again, so a member whose workspace access was revoked or downgraded kept live collaborative write access — including durable Yjs document writes — for the whole lifetime of an already-open socket. The access-revalidation sweep explicitly skipped every non-workflow room. - sweep every room type, authorizing each against its own resource - share one membership policy (ROOM_MEMBERSHIP_ACTIONS) between the join check and the sweep, so a file-doc room keeps requiring write in both - gate file-doc document frames and table cell selections on the cached permission, evicting on a confirmed loss of access - re-check the cached decision before a join commits, so a join that authorized just before a revocation cannot re-enter the room - never let a join's own cache write clobber a revocation recorded mid-flight - surface room-access-revoked to clients; the file-doc editor falls back to read-only instead of accepting keystrokes that go nowhere * refactor(realtime): one shared eviction path for revoked room access The sweep and both per-frame gates each open-coded emit + leave + local-state cleanup. Route them all through evictSocketFromRoom so they cannot diverge on what eviction means; workflow keeps its historical access-revoked payload. * fix(realtime): re-check access before workspace-list room joins too The workspace-files / workspace-tables joins committed straight from their authorize result, so a join that authorized just before a revocation could put the socket back in a room the sweep had already evicted it from. Mirrors the guard the file-doc and table joins already had. * fix(realtime): order role-cache writes by read start, not write time Two authorizations can start in one order and finish in the other, so the decision written last can come from the older read. A join that authorized before a revocation but returned after the sweep's denial would bury it, handing the socket another full cache TTL of access. Every writer now takes a monotonic ticket before it queries and yields only to a later-started read. * chore(realtime): drop the test-only unguarded role-cache writer Tests can express the same setup with commitRoomPermission + a read ticket, so the cache has exactly one write path and no export without a production caller. * fix(realtime): keep handler-initiated table eviction retryable Evicting leaves the Socket.IO room synchronously, which is also how the sweep discovers work — so a presence removal failing in the per-frame path could never be retried and left a ghost collaborator until disconnect. Failed (or unconfirmed) removals now hand off to the sweep's existing cleanup lane instead of a second retry loop. * fix(realtime): re-resolve access at join commit instead of peeking the cache The pre-commit recheck peeked the role cache, which reports an EXPIRED entry as unknown and fails open — so a join stalled longer than the cache TTL could re-enter a room the sweep had already evicted it from, including a file-doc room where the next cold-cache frame is accepted as a durable write. All three joins now re-resolve the way the workflow join always has; it is normally a cache hit, since the join's own authorize just warmed it. * fix(realtime): keep the join generation guard after the access re-check The access re-resolve added in the previous commit sat AFTER the generation / superseded guard in the table and workspace-list joins, so a leave or a newer join landing during that await no longer cancelled the stale join — it would go on to leave the room the client had switched to and commit the abandoned one. The guard is now the last thing before the commit in all three handlers, as it already was for file-doc and workflow. * test(realtime): use the shared sleep helper in the new join tests check:utils bans the inline new Promise(setTimeout) form; the two stalled-join tests were the only new offenders. * fix(realtime): leave the prior table room only once the join is certain A table switch left the previous room before the access re-check ran, so a denial there aborted the join and left the client in no table room at all — silently dropped from one it may still be allowed to occupy. The leave now happens after the re-check, matching the file-doc and workspace-list joins. * fix(realtime): close the table join window between re-check and commit Moving the prior-room leave after the access re-check left Redis awaits between that check and socket.join, and superseded() only watches the join generation — so a sweep revocation landing in that window could still put a revoked socket back in the room. A synchronous cache peek immediately before the commit closes it without reintroducing the await; the authoritative resolve moments earlier wrote a fresh entry, so a differing read IS the revocation being guarded. |
||
|
Waleed
|
bf78c4c1d6 |
improvement(settings): unify save/discard across editable surfaces (#6173)
Save is now always rendered (primary, disabled until there is something to save) and Discard only appears once there is something to discard. Previously sandboxes showed no action at all until dirty — so a new sandbox had no visible Create button — while skills always showed a disabled Save and no Discard. - saveDiscardActions moves to @/components/settings and owns the one rule - adds SettingsActionChip / SettingsActionChips so the settings shell and the ReactNode-slot headers render actions through the same chip path - skills, secrets, connected credentials, custom tools and the secrets manager drop their hand-rolled Save chips for the shared helper - credential-detail drafts seed once per credential instead of re-seeding from every refetch, which was acting as an implicit discard |
||
|
Waleed
|
be8a93de81 |
fix(security): validate cloud region/project inputs and bind vertex credentials to the executing workspace (#6167)
* fix(security): validate cloud region/project inputs and bind vertex credentials to the executing workspace
Two credential-exposure bugs on the Vertex AI path.
1. `vertexLocation` reached `new GoogleGenAI({ location })` unvalidated. The
SDK interpolates that value straight into the API hostname
(`https://${location}-aiplatform.googleapis.com/`), so a value like
`attacker.tld/x` terminates the authority component and relocates the
request — with the workspace's GCP bearer token attached by the auth
client — to an arbitrary host. Reachable from agent/router/evaluator
blocks and from `POST /api/guardrails/validate` with only a session
cookie. `vertexProject` (interpolated into the URL path) and the Bedrock
region (interpolated into the endpoint hostname) had the same shape of
problem.
Adds `validateGoogleCloudLocation` / `validateGoogleCloudProject` to the
shared input-validation module and applies them, plus the existing
`validateAwsRegion`, at the provider chokepoints. Every path to the SDK
goes through `executeRequest`, so no caller can bypass them.
`azureEndpoint` already routes through the DNS-pinning SSRF guard.
2. `resolveVertexCredential` enforced only the user↔credential predicate and
never the workflow-workspace↔credential predicate that
`authorizeCredentialUse` applies on the HTTP path. A credential held in
workspace B could be pasted into a workflow in workspace A and consumed
by workspace-A principals with no access to B — including on deployed
runs, where `enforceCredentialAccess` is false and `ctx.userId` is the
workflow owner rather than the trigger caller. Service-account
credentials mint a `cloud-platform`-scoped token, so this handed
workspace A the use of workspace B's GCP identity.
The resolver now takes the executing `workspaceId` and rejects a
credential belonging to a different workspace, mirroring
`credential-access.ts`. All four executor call sites pass
`ctx.workspaceId`.
* fix(providers): normalize vertex location case before validating
Hostnames are case-insensitive, so a mixed-case location like US-Central1
reaches Google today. Lowercase it before the region check rather than
rejecting an input that currently works.
* fix(security): accept AWS European Sovereign Cloud regions in validateAwsRegion
eusc-de-east-1 is a real Bedrock-enabled region that the existing pattern
did not cover, so applying the validator to bedrockRegion would have
rejected a config that worked before. Adds the eusc-<country> partition.
|
||
|
mzxchandraandWaleed Latif
|
cd50a445fd |
feat(chat): highlight-to-chat — reference file/table selections in Chat (#6087)
* feat(chat): highlight-to-chat for file and table selections
Adds an IDE-style "add to chat" affordance so the Sim agent can reference an
exact passage of a file or a specific set of table rows/cells instead of a whole
resource.
- Two new ChatContext kinds: file_selection (inline selected text + line range)
and table_selection (authoritative row ids, optional column ids; the server
re-fetches current rows by id). Chip registry, client serializer, boundary
contract, and server resolver all extend along their existing switch(kind)
seams.
- Producers: Monaco context menu, Tiptap bubble menu, and the table grid context
menu, all using the Sim Chat block icon. Adding a selection opens the split
slideover (chat + resource) with the chip dropped straight into the input; from
a standalone Files/Tables page the context is stashed and drained on chat mount.
- Cmd+C / Cmd+V: a selection copied from a file/table rides a custom
text/x-sim-selection clipboard MIME and pastes into chat as the same reference
chip; the chat input round-trips a sole selection chip on copy/cut too.
- Guards: selection payloads are length/row/column bounded and truncated within
the schema bound; labels carry a deterministic key so distinct same-size
selections don't collide; a shared resource tab closes only once no remaining
chip references it; stale cell-range column ids drop the context rather than
dumping the full table.
* fix(chat): widen selection code fence so embedded backticks can't truncate it
Cursor: resolveFileSelectionResource wrapped the selected passage in a fixed ```
fence, so a selection that itself contained a fenced code block closed the outer
fence early and the agent received a truncated snippet. The fence is now one
backtick longer than the longest backtick run in the content (floored at three),
matching CommonMark's close rule.
* fix(chat): carry the selection chip on a column-header copy
Cursor: a column-header Cmd+C always took the async paged clipboard path, which
replaces the whole clipboard with text/plain only and can't carry a custom MIME
- so pasting a column copy into Chat couldn't rebuild the table_selection chip
that Add-to-chat produces for the same selection. When every row is loaded and
within the chat-selection cap, the column copy now does a synchronous event
write so the scoped table_selection rides alongside text/plain (mirroring the
row-'some' and cell-range paths); oversized/partially-loaded columns keep the
async plain-text path.
* refactor(chat): clean up highlight-to-chat selections
Correctness:
- Stop reporting fabricated line numbers for rich-markdown selections.
`doc.textBetween` counts ProseMirror block boundaries, not markdown source
lines, so the chip label and the agent prompt both claimed line ranges that
don't exist in the file. Line info is now emitted only by Monaco.
- Bound a table_selection's rendered markdown by characters, not just row and
column counts — 500 wide rows dwarfed the 20k-char file-selection budget.
Rows are emitted until the budget is spent, and the content says what was
omitted.
- Complete `areContextsEqual` for both selection kinds, so re-adding the same
selection dedupes while a different passage of an already-referenced file
registers as new.
Consistency:
- Carry the resource display name on the context instead of recovering it by
regex from the chip label, deleting fileNameFromSelectionLabel and
tableNameFromSelectionLabel.
- Replace the user-visible `#k3f9` hash disambiguator with a readable ordinal
applied at insert time (`Sales (3 rows) (2)`), via a shared
prepareContextForInsert used by both the add-to-chat and paste paths.
- Fold MothershipPendingContextStorage into MothershipHandoffStorage as a
chip-only handoff (optional message), removing the parallel storage class,
the second drain effect, and its StrictMode guard ref.
- Replace `window.location.assign` with `router.push`, matching the existing
"Troubleshoot in Chat" handoff.
- Collapse three near-duplicate synchronous copy branches in table-grid into
shared buildTableSelectionContext / writeLoadedRowsWithChip helpers, also
reused by the add-to-chat handler.
- Trim multi-paragraph inline comments to TSDoc stating each reason once.
Tests: new coverage for the character budget, the label ordinal, selection
equality, and chip-only handoff accumulation; each verified to fail when its
fix is reverted.
* refactor(chat): tighten table copy fallback and helper placement
- writeLoadedRowsWithChip now requires a chip to carry; with none it falls
through to the canonical paged path (preserving its row loading and
truncation notice) instead of doing a bare synchronous write.
- Reuse selectedColumnIds in the context-menu memo.
- Restore resolveTableSelectionResource's TSDoc, orphaned when renderTableCell
was extracted between the doc and its function.
* fix(chat): apply chip handoffs as one batch; widen table copy chip path
Cursor Bugbot: a multi-context chip handoff dispatched one event per context,
and insertContextChip resolved label collisions against selectedContexts read
through a ref that only refreshes on render. Each dispatch therefore saw the
same stale list, so a second same-label selection was never ordinalized and
addContext dropped it while its @token still landed in the text. The event now
carries the whole batch and insertContextChips threads each resolved context
forward as it goes.
Greptile: an explicit multi-row ('some') selection no longer requires every
selected row to be loaded before taking the chip-carrying sync path. That
gate assumed the paged fall-through would copy more, but its loadRows returns
rowsRef.current unchanged for 'some' — the same rows, minus the chip. Renamed
the parameter to to say what it actually gates on.
Remaining chip-less cases are inherent to the async Clipboard API, which
replaces the whole clipboard and cannot hold a custom MIME: a filtered
select-all (must page in more rows) and selections past the 500-row chip cap.
'Add to chat' covers both — it is not gesture-bound and drains to the cap.
* fix(chat): distinguish a line-less file-selection label from the whole file
Cursor Bugbot: a rich-markdown selection labelled itself with the bare file
name, which is exactly the whole-file chip's label. Menu-driven inserts reject
any context whose label is already taken (isContextAlreadySelected closes the
menu silently), so once a markdown selection was attached, mentioning that same
file was quietly dropped. One-way: the reverse order works because programmatic
inserts ordinalize through prepareContextForInsert.
Fallen out of dropping the fabricated line range from this editor — with no
range left, the label collapsed onto the file name. Fixed at the single source:
buildFileSelectionLabel now returns 'notes.md (selection)' when there is no line
range, so it stays honest about location while remaining distinguishable.
* fix(chat): don't revive an aged-out chip handoff when accumulating
Cursor Bugbot: pendingContexts merged a prior chip-only handoff without
checking its age, while store stamps a fresh timestamp on every write. An
abandoned handoff that had already passed max-age would therefore ride along on
the next 'Add to chat' and fire on the following navigation as if current.
Introduced by the accumulation behavior added earlier in this PR. Applied the
same freshness bar consume uses, and hoisted the 60s window into a single
MAX_AGE_MS constant so the two paths cannot drift.
* fix(chat): reference every selected row in a table chip, not just loaded ones
Cursor Bugbot: for a 'some' row selection the chip's rowIds came from the
loaded-page intersection (currentRows filtered by the selection) rather than
rowSel.ids. The chip carries ids and the server re-fetches them via
getRowsByIds, so a selected row that simply had not been paged into the grid
was silently dropped from the agent's context — select 600 rows with 200
loaded and the agent saw 200. Add to chat had the same loaded-only narrowing
through contextMenuRowIds.
Both now send the full selection, still bounded by MAX_TABLE_SELECTION_ROWS.
Only the pasted text stays limited to loaded rows, which is inherent — there
are no cell values to serialize for a row that has not been fetched.
* docs(chat): scope the table copy 'complete' comment to the text path
It read as though the whole selection were complete when ids are unloaded,
which is now only true of the serialized text — the chip deliberately carries
every selected id for the server to re-fetch.
* fix(chat): compare selection ids as sets, not sequences
Cursor Bugbot: sameIds compared rowIds/columnIds by index, but a table
selection's ids iterate in click order (they come from a Set), so the same rows
picked in a different order — or reached via a cell range rather than the
gutter — compared unequal. prepareContextForInsert then added a second
ordinalized chip pointing at rows already referenced instead of no-opping.
More reachable since the previous commit started sourcing rowIds from
rowSel.ids directly, where insertion order tracks the user's clicks.
* fix(chat): enforce the table selection budget over the whole rendered content
Cursor Bugbot: the budget subtracted only the header and divider before packing
rows, then prepended the 'Selected ...' prose and the newlines afterward, so the
final content could exceed MAX_TABLE_SELECTION_CONTENT_LENGTH whenever the last
accepted row left less slack than the prefix needed. The cap the TSDoc promises
was not actually enforced.
The prior test passed while missing this: its rows were wide enough that packing
stopped far short of the limit, so the boundary was never exercised. Replaced
with rows sized to fill the budget almost exactly, asserting both that the
content stays within the cap and that it still approaches it (so the assertion
can't pass by emitting an empty table).
Also capitalize Chat in the three 'Add to Chat' menu labels, matching the
constitution's module naming and the existing 'Fix in Chat' / 'Troubleshoot in
Chat' UI strings.
* fix(chat): don't swallow a paste whose selection chip is already attached
Cursor Bugbot: the selection-paste path called preventDefault before
prepareContextForInsert, so when that returned null (the same selection is
already a chip) the handler returned having claimed the event — no chip
inserted and no text/plain pasted either. Cmd+V did nothing.
preventDefault now waits until there is a chip to insert; a duplicate falls
through to the plain-text paste below, which is the reasonable reading of the
gesture.
* fix(chat): derive the budget reserve from the same clause it reserves for
Cursor Bugbot: worstCaseSizeClause built its own string that omitted the
row/rows word the real clause always carries, so the reserve ran ~5 characters
short and a tightly packed selection could still exceed
MAX_TABLE_SELECTION_CONTENT_LENGTH. A bug in the previous fix, from duplicating
the format instead of sharing it.
Replaced with one sizeClause(shown, omitted) used for both the up-front reserve
and the final prose, so the two cannot describe the count differently. The
reserve passes (rows.length, rows.length) — max digits on both counts and the
plural forced — which is an upper bound on any real clause.
The earlier tight-packing test could not see this: a single cell width leaves
whatever remainder it leaves, and 100 left more than 5 characters. Added a
sweep over widths 60-75 that collects overflows so a failure names the width;
it catches the reported bug at width 74 (20002 vs 20000).
* fix(chat): align MothershipChat's onContextRemove with the surface contract
The remaining-contexts argument was added to ChatSurfaceContextValue but not to
MothershipChat's own prop type, which forwards straight into it. Nothing passes
the handler there today so it typechecked (fewer params is assignable), but the
two declarations of the same wiring disagreed.
Does not change behavior: home still wires the remove handler only to the
empty-state surface, as it did before this PR.
* refactor(chat): apply cleanup pass findings
emcn design: the table context menu built its Add-to-Chat label in the parent
and never pluralized it, so right-clicking a single row read 'Add rows to Chat'
directly above 'Delete row'. Derive it inside ContextMenu from selectedRowCount
like every sibling label, with an addToChatCellScoped boolean mirroring
workflowCellScoped. Also more correct: selectedRowCount accounts for a
select-all beyond the loaded page, which contextMenuRowIds.length does not.
callbacks: drop two useCallback wrappers whose identity nothing observes — both
handleAddSelectionToChat handlers feed unmemoized components (one through an
inline arrow). buildSelectionContext stays wrapped; it is a real dep of the
copy-bridge effect.
comments: fold a duplicated rationale paragraph in handleAddSelectionToChat
left by two commits stacking, drop a {@link MothershipHandoff} that resolves to
nothing (the type is not imported there), and trim a rowIds doc that restated
its own type.
Skipped, deliberately: the effects pass proposed moving buildContext out of
useSelectionCopyBridge's deps behind a latest-ref. buildSelectionContext is
already stable, so churn is near-zero, and it would leave two sibling
useCallbacks purposeless.
* fix(chat): don't attach a selection chip to a copy from a nested input
Cursor Bugbot: a copy from a field inside the editor — Monaco's find box being
the common one — bubbles to the container while the document still holds a
highlight, so the bridge attached the editor selection to text the user never
copied. Chat paste then prefers the custom MIME and inserts a reference chip
instead of the search term.
Skips INPUT targets only. Copying the table grid's INPUT/TEXTAREA guard would
have suppressed the chip on the main copy path this hook exists for: Monaco's
own editing surface is a hidden textarea, unlike the grid's cell editors, which
really are form fields.
Tests cover both directions — chip attached from the textarea surface, skipped
from a nested input — and were verified to fail against the missing guard and
against the INPUT+TEXTAREA variant.
* fix(chat): persist the source names a selection chip renders from
Cursor Bugbot: fileName/tableName were mapped into the optimistic message and
accepted by the API schema, but PersistedMessageContext, buildPersistedUserMessage
and toDisplayContexts never carried them. After a reload a file_selection chip
fell back to its label for getDocumentIcon — and the label carries a location
suffix ('notes.md:12-40'), so extension detection broke.
Persists only the two names the display path reads. The rest of the payload
(text, rowIds, columnIds, line numbers) stays unpersisted on purpose: it exists
to resolve the selection server-side at send time, is never re-read when
rendering a past message, and would put a selection-sized blob — up to the 20k
char cap — in every stored message.
Test asserts both halves of that, and was verified to fail with the mapping
removed.
* refactor(chat): remove a needless alias and correct an eslint-disable reason
Self-audit for shortcuts, prompted by 'nothing hacky':
- handlePaste kept `const prepared = preparedSelection`, an alias added only to
avoid renaming two downstream lines. Uses the real name now.
- The home.tsx drain's eslint-disable claimed handleContextAdd is 'a stable body
function'. It is a body function, so it is a NEW value every render — the
justification was false. Replaced with the actual reason: it is omitted to keep
the drain one-shot, and doing so is harmless because consume() clears
atomically, so a re-run would find nothing.
Audited the rest of the diff for suppressions, casts and swallowed errors. The
three catch blocks are documented graceful degradations with explicit fallbacks
(row-drain failure, browsers rejecting a custom clipboard MIME mid-gesture,
malformed clipboard JSON); the one double cast is a DataTransfer stub in a test.
* fix(chat): bound the sync copy path by the text limit, not the chip cap
Cursor Bugbot: writeLoadedRowsWithChip bailed once loaded rows exceeded
MAX_TABLE_SELECTION_ROWS, but buildTableSelectionContext already slices rowIds
to that cap. So selecting more than 500 loaded rows fell through to the async
path, which cannot carry a custom MIME, and silently lost the chip — while Add
to Chat on the very same selection still produced a 500-row chip.
The two limits govern different things: the chip's cap is how many rows a
table_selection can reference, the text's is TABLE_LIMITS.MAX_COPY_ROWS. Gate on
the latter. Past it the paged path still takes over, because it owns truncation
and the accompanying notice.
* refactor(tables): extract selection-to-chip helpers into utils so they are testable
Follow-up I owed on the previous round: the copy path's eligibility rule and the
context builder were module-private in a ~4,600-line component with no test file,
so the last two fixes to them were reasoned rather than covered — and both were
wrong on the first attempt.
Moves selectedColumnIds and buildTableSelectionContext to the existing
table-grid/utils.ts (which already owns RowSelection, DisplayColumn and
getColumnId), and extracts the copy decision as canWriteRowsWithChip.
writeLoadedRowsWithChip keeps only the clipboard and toast effects, so the pure
rule can be tested without a DOM. utils.ts stays free of side effects.
New utils.test.ts covers the two limits that were conflated — a selection past
the chip cap still qualifies (the context caps its own rowIds), one past
MAX_COPY_ROWS defers to the paged path — plus the all-columns collapse and both
caps. Verified to fail against the old chip-cap gate.
* fix(chat): scope selections to the columns actually picked, and stop under-counting rows
Three findings from one Bugbot round.
Hidden columns widened cell ranges (reported twice). buildTableSelectionContext
and contextMenuColumnIds collapsed a range to an open scope when it covered
'every column', comparing against displayColumns.length — which drops hidden
columns AND expands workflow groups, so it never meant 'the whole schema'.
Selecting every visible column therefore cleared columnIds and the server
re-fetched columns the user had hidden. The collapse is removed rather than
re-based on a schema count: no count available to a caller describes the schema,
and an explicit column list is what the user actually selected. totalColumnCount
is gone from the signature.
Add-to-chat label undercounted rows. The menu derived its count from
selectedRowCount (loaded rows only) while the chip was built from the full
rowSel.ids set, so the label could promise fewer rows than were sent. Both now
read one addToChatRowIds memo, with the count passed through explicitly since it
legitimately differs from the count the delete/run labels use.
Monaco line range was off by one. A full-line highlight ends at column 1 of the
FOLLOWING line, so endLineNumber named a line that contributed no text — the
chip label and the agent prompt both claimed an extra line.
The collapse test I added last round asserted the buggy behavior as correct; it
now pins the opposite, and fails if the collapse returns.
* fix(tables): cap the Add-to-Chat label at the rows a chip can carry
Cursor Bugbot: last round's fix for the label undercounting rows introduced the
opposite error. addToChatRowCount passed the raw selection size, but
buildTableSelectionContext caps rowIds at MAX_TABLE_SELECTION_ROWS, so a
2,000-row selection advertised 2,000 while the chip referenced 500 — breaking
the same invariant the fix claimed to establish.
Routes the count through a chipRowCount helper next to the builder that applies
the cap, so the label cannot drift from the payload again.
utils.test.ts now asserts the invariant directly rather than the formula:
across 1, 42, 500, 750 and 50,000 requested rows, chipRowCount equals the
rowIds length the context actually carries. Verified to fail if the cap is
dropped.
* fix(chat): stop a removed chip lingering when its label prefixes another
Cursor Bugbot: the mention sync tests each label with a lookahead that rejects
only word characters, so '-', ')' and space all let a shorter label match INSIDE
a longer token. '@notes.md:12' matches within '@notes.md:12-40', and
'@Sales (3 rows)' within '@Sales (3 rows) (2)'. Deleting the shorter chip left
its context attached, and it was still sent with the message.
The label class is pre-existing, but this PR made it routine: line ranges and
uniqueContextLabel ordinals generate prefix pairs for any two selections of the
same file or table.
Fixed at the sync rather than by reshaping labels to dodge the prefix — a label
format chosen to avoid a matcher bug would just relocate it. Contexts are now
tested longest-label-first, and each matched token is blanked before shorter
labels are tested, so every context is judged against text its own token owns.
Blanked in place, not removed, so the (^|\s) boundary of whatever sits next to
it is preserved; prev order is still what's returned.
Shared with the workflow copilot input, so tests cover both directions: the two
prefix pairs are dropped when only the longer token remains, both survive when
both tokens are present, order is preserved, and trailing punctuation after a
mention still keeps its chip.
* fix(chat): include the line range in file-selection equality
Cursor Bugbot: areContextsEqual compared only fileId and text for
file_selection, while the comment directly above claimed equality was the
selected range. A line that occurs twice in a file — a repeated import, a
closing brace — highlighted at both places produced identical text, so
prepareContextForInsert called the second a duplicate and dropped its chip,
even though the labels (notes.md:12 vs notes.md:50) were plainly different.
Comparing startLine/endLine as well makes the code match what the comment
promised. The rich-markdown editor omits the range, so both sides are undefined
there and identical text still dedupes — correct, since two identical passages
are genuinely indistinguishable without line numbers.
Tests cover both: distinct ranges stay distinct, an exact repeat still dedupes,
and the no-line-number path still dedupes. Verified the first fails against
text-only equality.
* fix(tables): drain past the cap so exclusions can't shrink a select-all chip
Cursor Bugbot: for a gutter select-all the menu count comes from
selectedRowCount (capped), but the chip was built by loading exactly
MAX_TABLE_SELECTION_ROWS and filtering exclusions AFTER. Any excluded row inside
that prefix left the chip short of the advertised count.
Third appearance of the same invariant — label vs payload — this time in the
select-all path specifically, which the earlier fixes did not touch.
Loads the cap plus the exclusion count, which covers the worst case where every
exclusion falls inside the prefix, so the filtered result still reaches the cap
whenever the table has the rows. Extracted as drainTargetForChip so the
compensation is stated and tested rather than an inline arithmetic detail.
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
|
||
|
Waleed
|
d89ab4a8ee |
fix(security): bound response bodies in secureFetchWithPinnedIP by default (#6169)
secureFetchWithPinnedIP only capped a response body when the caller passed maxResponseBytes; with the option absent the body streamed into an unbounded Buffer.concat. Several tool proxies whose target host is user-supplied (Jupyter, ClickHouse, Grafana, 1Password Connect) and the RSS poller called it without that option, so an attacker-controlled server answering with an endless chunked body could grow the shared process heap until it was OOM-killed. Make the cap fail-safe: default to 100MB (and treat a non-positive value as the default) so there is no unlimited mode, then pass tighter explicit caps at the user-supplied-host sites. Responses that carry no body (HEAD, 204, 304) are exempted from the content-length pre-check — they advertise the resource size as metadata, which would otherwise spuriously fail a HEAD probe of a large file or an RSS conditional-GET 304. |
||
|
Waleed
|
47e8f1eb5b |
fix(ssh/sftp): cap remote file reads on received bytes, not stat() size (#6168)
The SFTP/SSH download routes buffered a remote file into memory with the only size guard being sftp.stat().size — a value the caller-supplied SSH server controls. A server that reports a tiny size and then streams endlessly drove unbounded heap growth until OOM. Read through the existing readNodeStreamToBufferWithLimit limiter, which enforces the cap on bytes actually received and destroys the stream on breach, via a small readSftpFileCapped wrapper in sftp/utils. All four SFTP-reading tool routes now share it — download, download-file, read-file-content, and the append path of write-file-content, which had no cap at all. The wrapper keeps a no-op error listener on the stream for its whole life: ssh2 rejects still-pending SFTP requests when the channel closes, which arrives as a late error event after the limiter has detached its own handlers, and an error event with no listener would take the process down. Too-large responses now return 413 to match how the rest of the routes surface PayloadSizeLimitError, and responses report the actual byte count rather than the server-reported stat size. |