mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
e4cedcb91f8ea705ccf8a003fde49cdbb5bdf221
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Theodore Li
|
334c7c81bc |
fix(security): suppress the CodeQL password-hash alert on sha256Hex (#5982)
CodeQL fails #5963 (the v0.7.46 release PR) with 2 high-severity js/insufficient-password-hash alerts on packages/security/src/hash.ts. Both are false positives that surfaced now because
|
||
|
Theodore Li
|
19c3b6f47d |
feat(setup): setup wizard with browser-based Chat key handoff (#5911)
* feat(setup): setup wizard with browser-based Chat key handoff
Adds `bun run setup` and `bun run doctor` for local installs, and replaces
the wizard's paste-your-Chat-key step with a browser handoff that never puts
the key in a URL.
* improvement(setup): drop the paste-a-key fallback, simplify consent copy
The browser handoff is now the only path — the wizard waits on a spinner
instead of racing a paste prompt. Consent card leads with "Connect your
terminal" and moves the match-the-code disclaimer into the description.
* fix(setup): pin kube context, keep secrets out of argv, validate reused keys
Review findings from #5911:
- helm/kubectl now run against the validated context instead of the ambient one
- helm values are piped on stdin rather than passed as --set arguments
- ENCRYPTION_KEY/API_ENCRYPTION_KEY are checked for the 64-hex format the app
requires, not just length, so an unusable key is replaced rather than kept
- the managed Redis container's published port is read back instead of assumed
* refactor(copilot): one module for Chat API key operations
list/generate/delete each repeated the same /api/validate-key envelope in
their route. They now share callValidateKey in lib/copilot/server/api-keys.ts,
which also keeps the display masking server-side so the full key can only ever
leave at creation.
* improvement(setup): reuse shared helpers, parallelize probes, drop dead code
- PKCE verifier/state/pairing code now use generateSecureToken, generateRandomHex
and generateShortId instead of hand-rolled randomBytes; the pairing loop's
modulo was unbiased only because 256 % 32 == 0
- new sha256Base64Url in @sim/security/hash so both sides of the PKCE exchange
derive the challenge from one implementation
- isUsableSecret moved beside SECRET_KEYS so setup and doctor apply the same
rule; doctor previously passed a key setup would replace
- isTruthy narrowed to true/1, matching the app it claims to mirror — it accepted
yes/on, so a flag could read on in doctor and off in the app
- checkLive runs its five probes concurrently (~17s serial worst case)
- detection overlaps the banner animation instead of queueing behind it
- glyph.fail/glyph.warn at 13 sites that bypassed the constant; removed unused
prompter exports, a dead ENV_PATHS re-export, and an unused export keyword
* fix(setup): make doctor understand the compose env layout
Compose writes a single root .env (what docker-compose reads via env_file) but
the checks required the three per-app files, so a successful compose install was
followed by doctor printing three failures and exiting 1 — and the whole
coherence catalog was skipped because it keyed off apps/sim/.env existing.
Layout is now derived from what's on disk and every check consults it: file and
schema checks iterate the layout's targets, consistency reports skip when
there's only one file to mirror, and coherence/live read the layout's primary
file. The wizard's existing-config detection counts root for the same reason —
a compose install used to read as unconfigured and re-run from scratch.
* feat(cli-auth): device-authorization poll flow, drop the loopback listener
The CLI no longer binds a local port. It generates a request id + poll secret,
opens /cli/auth, and polls /api/cli/auth/poll over TLS while the user approves
in the browser — so the flow works over SSH and inside containers, where the
browser and terminal don't share a machine.
- approve stores the approval keyed by request id (session-authed, userId from
the session only); poll verifies the secret before an atomic claim, so an
observer of the semi-public request id can neither mint nor cancel it
- pairing code stays as the anti-phishing compare; no key ever crosses the
browser; done page just confirms
- removes the loopback listener, /token exchange, buildCliHandoffUrl, and
validateCliCallbackUrl (+ its tests) — nothing hands a key to a URL anymore
* fix(setup): reuse an existing managed Postgres container instead of colliding
A running sim-postgres fell through to `docker run --name sim-postgres` and died
on the name conflict; a stopped one failed with "no DATABASE_URL to reach it"
because the generated password only lived in the env files a fresh clone lacks.
Both facts are recoverable from Docker: the ladder now reads the published port
and password back via `docker inspect` and reuses the container (starting it if
stopped). A container that won't answer prompts before recreating, and never
drops the data volume silently.
* improvement(setup): audience-first run-mode hints
Each run mode now names who it's for — compose for self-hosting/evaluating, dev
for contributing to Sim, k8s for rehearsing a production deploy — with the live
detection state (Docker/kube/VM) appended.
* fix(cli-auth): retry a failed mint, port container/port fixes to Redis + k8s
Review findings from #5911:
- poll now reserves the mint with an atomic NX lock instead of deleting the
approval up front, so a failed mint (e.g. mothership blip) is retried by the
next poll instead of forcing a fresh browser approval; the lock still prevents
a double-mint and its TTL frees the slot if the caller dies
- setup reuses/recreates an unhealthy managed sim-redis instead of colliding on
the name (Redis has no data volume, so it removes and recreates without a prompt)
- k8s failure-path hints carry --context, matching the success-path hints, so a
changed ambient context can't send diagnostics to the wrong cluster
- compose port-free waits for a killed port to actually release before
re-checking; SIGKILL is async, so the immediate re-check re-saw the port
* fix(setup): harden mint cleanup, Windows browser, container detection, helm cwd
Review findings from #5911:
- a post-mint completeApproval failure no longer routes into releaseMint — the
mint lock now outlives the approval (shared TTL), so a cleanup blip can't leave
a re-mintable window and orphan a key; cleanup is best-effort after the key ships
- compose doctor --fix writes the feature-flag twin to the layout's primary env
(root .env on a compose install), not always apps/sim/.env
- Windows opens the browser via `cmd /c start "" <url>` — `start` is a shell
builtin, so spawning it directly ENOENT'd and the handoff never opened
- managed-container detection filters loosely and pins the exact name in code;
Docker's `name=^x$` anchor matches the internal `/x` form and often missed,
skipping the reuse branch
- the shared helm/kind run helper pins cwd to the repo root, matching helm test,
so `helm upgrade --install ./helm/sim` works from any working directory
* feat(chat-keys): standalone manage page, drop from settings nav, refresh README
- Add /account/settings/chat-keys — a linkable page to view, create, and revoke Chat API keys
- Remove Chat keys from the settings sidebar (account + unified nav) and its render branches
- README: replace Docker Compose + Manual Setup with the bun run setup wizard; drop the manual COPILOT_API_KEY step, point to the manage page
* fix(setup): per-key reason in the secret-replacement warning
Cursor: the warn hardcoded '64-character hex key', but only ENCRYPTION_KEY/API_ENCRYPTION_KEY require that — BETTER_AUTH_SECRET/INTERNAL_API_SECRET only need length >= 32. Use the existing secretRequirement(key) helper so each replaced key reports its actual requirement.
* fix(setup): compose doctor schema, cross-platform binary detection, quoted context hints
- Doctor: for the compose (root) env layout, require only the secrets compose has no interpolation default for (BETTER_AUTH_SECRET/ENCRYPTION_KEY/INTERNAL_API_SECRET). DATABASE_URL/BETTER_AUTH_URL/NEXT_PUBLIC_APP_URL come from docker-compose ${VAR:-default}, so a healthy compose install no longer fails doctor.
- Binary detection: use Bun.which instead of which (which is absent on Windows), so kubectl/helm/kind/docker resolve cross-platform.
- k8s diagnostic hints: POSIX-quote the kube-context so a context with whitespace/metacharacters can't break or inject into a copied command.
* fix(setup): quote kube-context in the helm uninstall tear-down hint too
The tear-down hint used --kube-context ${context} raw while the sibling kubectl hints already used shq(); a context with whitespace/metacharacters could break or inject into the copied command. All copyable k8s hints now go through shq(context).
* feat(setup): sim lifecycle CLI — start/stop/status/logs/down/reset
Turn the setup entry into a 'sim' command umbrella so there's one place to run everything, not scattered docker/bun commands. Adds a global bin (bun link) + a bun run sim fallback.
- Detects how you're running (compose file / managed dev containers / helm release) from disk + docker/helm state — no persisted mode. Ambiguous installs prompt.
- start/stop/restart/logs work per mode; down removes containers (volumes kept); reset archives .env + wipes managed data; both destructive verbs confirm first.
- status shows detected mode, container states, and app/realtime health.
- Wizard outro + README now point at the sim commands and the one-time bun link.
* feat(setup): 'bun run sim' is the primary entry; bare invocation prints help
- Lead usage/wizard-outro/README with 'bun run sim <cmd>' (works with zero PATH setup); global bare 'sim' via bun link is an optional upgrade, with the ~/.bun/bin PATH caveat spelled out (Homebrew's bun omits it).
- Bare 'sim' now prints help instead of launching the wizard; the wizard is 'sim setup'. The 'setup' npm script passes the keyword so 'bun run setup' is unchanged.
* fix(setup): quote the auth URL for cmd /c start on Windows
Cursor (High): cmd re-parses the command line and treats & in the query string as a command separator, so cmd /c start opened a URL truncated at the first &, breaking the key flow on win32 (the handoff URL always has request/challenge/pairing). Quote the URL and pass args verbatim so & stays literal.
* fix(setup): verify kube-context is really local; lengthen CLI handoff wait
- k8s: a context named like a local cluster (kind-*, docker-desktop) can actually point at a remote API server. Verify the server host is loopback/docker-internal before defaulting the 'use this context?' confirm to yes; otherwise warn and default to no, so generated secrets can't ship to a remote cluster on a blind Enter.
- cli-auth: bump the device-flow wait from 3 to 15 minutes so first-time users have time to sign up, wait for the email OTP, and approve before the terminal stops polling. The server-side approval record keeps its own short TTL, so a longer client wait only costs cheap rate-limited polls.
* fix(setup): only manage k8s lifecycle on a verified-local context
Greptile: sim down/reset used the ambient kube-context, so switching context after setup could uninstall a same-named sim-dev release from the wrong cluster. Gate k8sInstall on the same locality check the wizard uses (API server is loopback/docker-internal) via a shared isLocalKubeContext helper — the wizard only ever deploys locally, so a remote current-context is never treated as a Sim install.
* fix(setup): doctor skips placeholder secrets when seeding; reset names its target
- checks: the missing-file autofix copied shared keys from apps/sim/.env whenever truthy, including .env.example placeholders — doctor --fix could seed unusable secrets into realtime/db env files. Skip placeholders, matching autofixForMissing.
- lifecycle: reset now names the exact install (k8s context / compose file / dev containers) in its confirm, so a destructive reset can't silently hit the wrong same-named install after a context switch (down already names the context).
* fix(cli-auth): size the poll rate limit to the poll cadence; honor Retry-After
The poll route used the default public-IP bucket (10 burst, 5/min) but the CLI polls every 2s (30/min), so it 429'd within ~20s — worse behind a slow dev cold-compile. Give the endpoint a bucket matched to its cadence (60 burst, 60/min); it's not a brute-force surface (unknown request id returns pending, minting needs the 256-bit verifier). Also make the CLI honor Retry-After and back off on 429 so a shared-NAT per-IP limit degrades gracefully instead of hammering.
* fix(setup): check ports before starting the dev server, not just compose
Local dev auto-start spawned bun run dev:full with no port check, so it silently started a server that couldn't bind when 3000/3002 were already taken (e.g. another worktree's dev server). Extract compose's port-conflict resolver into a shared ensurePortsFree(ports) and run it before the dev start too — kill/recheck/leave, same as compose. Leaving the ports skips the auto-start with guidance instead of failing; compose still treats it as fatal.
* fix(setup): verify the kube cluster is reachable, not just local
A kubeconfig context can outlive its cluster — a kind cluster gets deleted or its Docker container stops (Docker/machine restart), but the context entry remains, pointing at a dead API-server port. The wizard checked the context looked local and handed it to helm, which failed with 'cluster unreachable'.
Add a clusterReachable() liveness probe: only offer the current context when it actually answers; if a local context is dead, fall through to the kind path. There, if kind still knows 'sim' but it's stopped, start its node containers and wait for the API; if it's gone, create fresh. Either way the user gets a working cluster instead of a cryptic helm failure.
* fix(helm): point appVersion at published image tags (v-prefixed, current)
The chart's appVersion was "0.6.73", but CI publishes GHCR tags with a v prefix (its release-commit regex captures v0.7.45). Since sim.image defaults every image tag to Chart.AppVersion, a default helm install requested ghcr.io/simstudioai/{simstudio,realtime,migrations}:0.6.73 — a tag that has never existed — so app and realtime sat in ImagePullBackOff and helm --wait failed with 'progress deadline exceeded'. Any self-hoster installing with default values hit this, not just the setup wizard.
Set appVersion to v0.7.45 (latest release on main; all three images verified present on ghcr) and bump the chart version to 1.1.1. Verified with helm lint, helm template (all images render as v0.7.45), and a live helm upgrade on a kind cluster where the new pods pull successfully while the old 0.6.73 pods remain in ImagePullBackOff.
* Revert "fix(helm): point appVersion at published image tags (v-prefixed, current)"
This reverts commit
|
||
|
|
f03b4337fa |
feat(mothership): mixture of models, search agent, persistent subagents, fork chat, inline questions (mothership v0.8) (#5410)
* feat(scout): add scout agent * fix(contracts): update contracts to include scout agent * feat(copilot): search agent (research+scout merge) + read-only table/KB tool handlers Mirrors mothership dev f90f9b05: - regenerated tool-catalog/tool-schemas mirrors (search trigger replaces research + scout; QueryUserTable / SearchKnowledgeBase entries) - queryUserTableServerTool / searchKnowledgeBaseServerTool: read-only wrappers delegating to the full user_table / knowledge_base handlers with hard operation allowlists (and outputPath export rejection on query_user_table) - display maps: 'search' agent label/title/icon added; research + scout entries retained so historical transcripts keep rendering - Search.id replaces Research.id in LONG_RUNNING_TOOL_IDS (it inherits research's long crawls) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(copilot): run_code compute-only handler; docs lint fix Mirrors mothership dev db60da94: run_code is the compute-only variant of function_execute for the search agent — same sandbox and inputs, no outputs.files / outputTable, so it cannot create or overwrite workspace resources. Wrapper handler hard-rejects the write vectors and delegates to executeFunctionExecute; run_code is deliberately absent from OUTPUT_PATH_TOOLS and the table output post-processor, so the name gating blocks writes even for leaked args. Added to LONG_RUNNING_TOOL_IDS, display title/icon maps, and the regenerated catalog/schema mirrors. Also removes two ineffective biome suppression comments in the docs workflow-preview (the rule doesn't fire in the docs app config). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(copilot): failed tool calls must surface their error in terminal data A failed handler result that carried a defined-but-empty output (the app-tool executor's 'Tool not found' ships output: {}) won the priority race in getToolCallTerminalData, so the resume payload's data — the only thing the model reads — was a bare {} with the error text dropped. The search agent retried run_code 20+ times blind against a stale server because every failure rendered as empty instead of 'Tool not found'. Failed calls now always carry error in their terminal data: merged into object outputs, wrapped alongside non-object outputs, preserved when the output already has an error field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(chat): render inline question tags from the agent in chat * fix(chat): let inert multi-step questions browse all prompts * improvement(chat): guard question answer formatting against sparse arrays * chore(copilot): drop user_memory from generated contracts and tool display Companion to mothership 8ae32e97 (user_memory tool removed — the feature no longer exists). Regenerates the mothership contract mirrors via generate-mship-contracts.ts, which also picks up the pending telemetry contract additions (gen_ai.agent.name labels, llm.client.context_tokens, llm.client.compactions, llm.request.compaction_trigger, llm.compaction.pause, gen_ai.usage.context_tokens), and removes the user_memory display title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): answered question card becomes the user turn; two select types only UI ordering: answering a question card no longer echoes a duplicate user bubble. The combined answer still goes on the wire as a user message, but the chat pairs it back to its card (strict 'Prompt — Answer' match, now uniform for single questions too) and renders the card as the answered recap — the card IS the user turn, and the next assistant message streams below it. The pairing is derived from the transcript, so live and reloaded renders are identical; a dismissed card followed by an unrelated typed message does not match and renders normally. Messages ending with a question card also drop the copy/thumbs actions row — the card is an input surface, not a reactable assistant turn. Question types are now single_select and multi_select only: text is removed (the free-text 'Something else' row covers it) and confirm collapses into single_select with Yes/No options. multi_select rows toggle with a check and the free-text row's arrow submits the step; answers are comma-joined labels plus any typed entry. Agent-supplied catch-all options ('Other', 'Something else', 'None of the above') are stripped at parse — the card always provides its own free-text row; a question left with no real options is invalid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): question cards are single_select only Removes multi_select (and its toggle/check UI). The card is one shape: pick one option or type into the always-present 'Something else' row. Catch-all stripping and the transcript pairing/recap behavior are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * improvement(chat): bring back multi_select question cards Re-adds multi_select with a reworked interaction: option rows carry real checkboxes (emcn Checkbox chrome) instead of numbers and arrows, an option-styled Submit row confirms the step, and the "Something else" row reads as a plain option until clicked — then it becomes the focused text box, auto-checks, and can be unchecked without losing the typed text (blur with nothing typed reverts it). single_select behavior, catch-all stripping, and the transcript pairing/recap format are unchanged; multi_select answers are the checked labels comma-joined. * chore(copilot): regenerate mothership contract mirror (chat blob span attrs) * chore(copilot): regenerate mothership contract mirror (chat blob metrics) * feat(secrets): make output of generate api key a secret * feat(cli): add mkdir, mv, cp to mship tool set * feat(fork-chat): add fork chat to mothership * fix(fork-chat): fix messageid handling in fork chat * feat(credentials): agent-initiated oauth credential reconnect (#5488) * feat(credentials): agent-initiated oauth credential reconnect * fix(credentials): address reconnect review findings * improvement(credentials): log when connect draft name lookups degrade * fix(conflicts): remove migration * fix(conflicts): fix conflicts * fix(fork-chat): add migrations back * fix(ci): fix lint * fix(ci): fix bad import * fix(vfs): fix 500 char limit in vfs for skills and custom tools * feat(copilot): gate user skills to explicit slash-attach (#5536) Stop the mothership from adopting a workspace user-skill on its own: - Remove the load_user_skill tool and its three payload callers (chat payload, mothership execute route, inbox executor); delete lib/mothership/skills.ts + its test. Skills no longer autoload as the agent's own instructions. - Rename the workspace "## Skills" inventory to "## Agent Block Skills — NOT FOR YOU" with a one-line guardrail so a skill's description (e.g. "respond like a pirate") is not treated as an instruction. Skills reach the model as behavior only via explicit /-attach. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(lots-of-things): lots of things * feat(subagents): add persistent subagents * fix(copilot): let edit_workflow set knowledge-base tag filters, and stop it clearing them (#5546) * fix(copilot): persist KB tag subblocks as JSON strings from edit_workflow The edit_workflow tool normalizes array-with-id subblocks (via normalizeArrayWithIds) but only re-stringifies the keys listed in JSON_STRING_SUBBLOCK_KEYS. `tagFilters` (knowledge-tag-filters) and `documentTags` (document-tag-entry) were missing, so agent-authored tag filters were stored as raw JSON arrays while those UI components read their value with JSON.parse (expecting a string). The result: an agent edit to a Knowledge block's tag filter persisted correctly but rendered as an empty filter in the editor (JSON.parse on an array throws -> []). - Add `tagFilters` and `documentTags` to JSON_STRING_SUBBLOCK_KEYS so edit_workflow stores them in the same shape the UI writes. - Make both components' parsers tolerate an already-parsed array on read, self-healing values already persisted in the broken (array) shape. Search execution was unaffected (parseTagFilters accepts arrays), so the value was never lost — only the editor render and round-trip were broken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): expose KB tag definitions in VFS meta.json Surface each knowledge base's defined tags (displayName -> tagSlot) inline in its meta.json via serializeKBMeta, loaded in one batched query (loadKbTagDefinitions), so the agent can bind a knowledge-tag filter to a real tag slot instead of guessing a tag name it cannot otherwise see. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stringify KB tag subblocks on the nested-node edit path The nested-node merge path normalized array-with-id subblocks but never re-serialized the JSON_STRING_SUBBLOCK_KEYS, so editing a block nested in a loop/parallel container still persisted tagFilters/documentTags (and conditions/routes) as raw arrays -- the exact shape the subblock components cannot JSON.parse. Route all four write paths through a single normalizeSubblockValue helper so the normalize and re-stringify steps cannot drift apart again, and extract the duplicated string-or-array read logic into parseJsonArrayValue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): tighten subblock serialization helpers Derive KbTagDefinitionSummary from the canonical TagDefinition instead of restating its fields, make parseJsonArrayValue generic so callers drop their `as T[]` casts, and unexport the three builders helpers that no longer have consumers outside the module now that normalizeSubblockValue fronts them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): stop stripping tagFilters/documentTags from the agent's workflow view sanitizeForCopilot dropped `tagFilters` and `documentTags` from the workflow state the agent reads (workflows/{name}/state.json), while edit_workflow is allowed to write both. The field was therefore write-only: on a follow-up edit the agent read back an absent field, concluded no filter was set, and cleared the user's tag filter. The redaction was introduced for workflow *export* (#1628) and is already enforced there by sanitizeWorkflowForSharing's key list. The duplicate in the copilot-only sanitizeSubBlocks was redundant for export and destructive for the agent. Removes it and pins the contract with a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): reject malformed KB tag values instead of clearing the filter `knowledge-tag-filters` and `document-tag-entry` had no arm in the `edit_workflow` input validator, so they fell through to the pass-through default. Any non-array value the agent supplied -- a double-encoded JSON string, an object, an unparseable string -- reached `normalizeSubblockValue`, where `normalizeArrayWithIds` coerces unparseable input to `[]`. The write path then persisted `"[]"` over the tag filter the user had configured. `condition-input` and `router-input` already guard against exactly this and return an actionable error to the model. Extend that arm to cover the two KB subblock types. It keys on subblock type, so the unrelated `tagFilters` short-input on the Algolia block is unaffected. `null`/`undefined` and empty arrays still clear the field, so intentional clears keep working. Also wrap `loadKbTagDefinitions` in try/catch. Tag definitions are an optional meta.json enrichment, but the query ran inside the top-level `Promise.all`, so a transient failure would reject the entire workspace VFS materialize and leave the agent unable to read any file. Now it degrades to a meta.json without tag definitions, matching the sibling materializers. Adds regression tests for both, plus the first tests for `parseJsonArrayValue`, the helper that keeps pre-fix raw-array rows readable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(copilot): collapse duplicate JSON-array parsing in edit-workflow builders `normalizeArrayWithIds` and `normalizeConditionRouterIds` each hand-rolled the same "accept a raw array or the JSON string these subblocks persist" parse. Extract `parseJsonArray`, which returns null when the value is neither, so each caller keeps its own distinct fallback: `[]` for the former, the untouched original value for the latter. Behavior-preserving. An empty array is truthy, so `[]` and `"[]"` still parse through rather than hitting either fallback. `validation.ts` has a third copy, but `builders.ts` already imports from it, so sharing the helper across the two would introduce an import cycle. Left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): specify tag name and legal operators in KB meta.json `tagDefinitions` exposed `displayName`, but a `tagFilters` entry must carry the key `tagName`. An entry written with `displayName` passes validation and persists, then filters nothing -- a silent failure. Rename the field at the serializer boundary; the DB column is untouched. Also emit the operators legal for each tag's `fieldType`, reusing `getOperatorsForFieldType`. `between` is valid for number and date but not for text or boolean, and the agent has no way to infer that. An unrecognized fieldType yields an empty list rather than throwing. Still unspecified, and deliberately out of scope: a filter entry's value key is `tagValue` (but `value` on documentTags), and `between` needs `valueTo`. Those describe the subblock entry shape, not the knowledge base, so meta.json is the wrong place for them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): pass a nullish subblock clear through instead of serializing "[]" `validateValueForSubBlockType` accepts null as an explicit clear, but `normalizeSubblockValue` then ran it through `normalizeArrayWithIds`, which coerces any non-array to `[]`, and persisted the string "[]". No data is lost either way -- "[]" and an absent field both mean "no filters". But it left the field present when the caller asked for it to be unset, so `sanitizeForCopilot` showed the agent an empty filter rather than an absent one, contradicting the absent-means-unset invariant the sanitizer documents. It also made Algolia's `if (params.tagFilters)` see a set value, since "[]" is truthy. An explicitly empty array still serializes to "[]" -- clearing with a value is distinct from clearing by omission. Reported by Cursor Bugbot on #5546. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(changes): huge changes * fix(subagents): lanes * fix(mship): transcript stuff * fix(subagents): thinking lanes * fix(superagent): fix superagent tools and checkpoints * fix(scope): scope subagent tools * fix(lint): fix lint * chore(db): regenerate workspace_files.message_id migration as 0260 on staging base * fix(superagent): fix superagent integration tools * improvement(questions): make something else a placeholder * chore(copilot): regenerate mothership contract mirror after staging rebase * feat(mship): add external mcps to mship * fix(ci): fix dev build * fix(stream): show thinking text * fix(ci): force redeploy * fix(mothership): keep chat forks outside workspace storage billing Preserve the product invariant that Mothership chat files are not charged as workspace file storage after the billing storage merge. * fix(uploads): restore listWorkspaceFiles throwOnError option dropped in rebase * fix(subagent-streaming): remove italics * fix(mothership): treat subagent lanes closed by subagent_end as settled so the between-steps thinking indicator isn't suppressed * fix(ui): thinking loader and rool names * fix(ui): add file * fix(thinking): show thinking during subagents * fix(chat): drop dead thinking-channel ternary after lanes skip thinking blocks * fix(thinking): remove thinking text * improvement(function execute): add timeout to function execute and stop showing text in subagents * fix(subagents): hide thinking text * fix(ff): move ff to go * improvement(superagent): nuke superagent * feat(main-agent): superagent into main agent * chore(db): regenerate workspace_files.message_id migration as 0262 on staging base * fix(credentials): restore reconnect params on shared createConnectDraft * fix(migrations): rebase with staging --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Emir Karabeg <emirkarabeg@berkeley.edu> Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> |
||
|
Waleed
|
d078c84ee6 |
chore(typescript): upgrade to TypeScript 7 (native Go compiler) (#5521)
* chore(typescript): upgrade to TypeScript 7 (native Go compiler) Bumps typescript to ^7.0.2 across every workspace package. Full bun run type-check/lint/build/test all pass; apps/sim's type-check (the one needing an 8GB heap bump) drops from ~55s to ~7s wall time. Migration fixes required by TS7's stricter defaults: - baseUrl removed: drop it from 5 tsconfigs (paths already resolved relative to tsconfig dir, so behavior is unchanged) and prefix the one bare (non-relative) paths entry each in apps/sim and apps/realtime with './' - moduleResolution=node10 removed: switch packages/cli and packages/ts-sdk to "bundler", matching the rest of the monorepo - types now defaults to [] instead of auto-including every @types/* package: add "types": ["node"] to the shared base tsconfig (this is fundamentally a Node monorepo, so this restores prior behavior in one place instead of duplicating it per-package), add explicit @types/node deps to packages that now rely on it transitively via @sim/db/@sim/logger, and add "declare module '*.css'" to the two packages with plain (non-module) CSS side-effect imports that TS7's stricter checker now flags - packages/logger's isomorphic `typeof window` check no longer needs DOM lib in every consumer: replaced with `'window' in globalThis` - packages/testing and apps/realtime's fetch/DOM mocks need DOM lib where they're compiled, since they model the browser Fetch API - the `typescript` npm package no longer exports the classic Compiler API from its main entry (moved to unstable/ast subpaths); apps/sim's Function-block route used it at runtime to strip import statements from user code, so that one call site now uses Microsoft's official transition package, @typescript/typescript6 - Next.js 16.2.6's own TypeScript-detection heuristic hardcodes a path TS7 no longer ships, and its auto-install fallback assumes npm/pnpm; added @typescript/native-preview as a devDependency to apps/sim and apps/docs so Next detects a valid native compiler instead of trying (and failing) to auto-install one Not merging yet: TS 7.0.2 published today and is still inside this repo's bunfig.toml minimumReleaseAge (7-day) supply-chain gate, so `bun install` will fail for everyone until 2026-07-15. Opening this now to get it through review; hold the actual merge until then. * fix(typescript): address Greptile review findings on TS7 upgrade - packages/logger: 'window' in globalThis treats a shim that leaves globalThis.window explicitly undefined as browser-only, silently dropping production server logs. Restore the original typeof !== 'undefined' semantics via an inline cast instead, so it stays correct without requiring DOM lib in every consumer. - packages/ts-sdk, packages/cli: both are tsc-built, published as Node ESM (package.json "type": "module" with an "exports" map). "moduleResolution": "bundler" is too permissive for that target - it accepts import patterns (e.g. extensionless relative imports) that Node's actual ESM resolver rejects at runtime. Switch both to "module"/"moduleResolution": "nodenext", the correct pairing for a published Node ESM package. Verified real tsc builds (not just --noEmit) still succeed for both. * chore(bunfig): temporarily disable minimumReleaseAge gate for TS7 install TS 7.0.2 published today, still inside the 7-day gate. Lowering to 0 to unblock this merge; will restore to 604800 in an immediate follow-up commit right after merging. |
||
|
Waleed
|
a7b0bd311d |
fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) (#4837)
* fix(deps): upgrade vitest to ^4.1.0 to patch critical Vitest UI advisory (GHSA-5xrq-8626-4rwp) - Bump vitest and @vitest/coverage-v8 to ^4.1.0 across all workspaces (only patched release for the critical 'Vitest UI server arbitrary file read/execute' advisory; no 3.x backport exists) - Widen @sim/testing peer range to ^3.0.0 || ^4.0.0 - Migrate constructor mocks to class expressions: vitest 4 uses Reflect.construct for mocks invoked with new, and arrow/function implementations are not constructable (function expressions also get reverted to arrows by biome's useArrowFunction) - Remove deprecated test.poolOptions from apps/sim/vitest.config.ts (options are now top-level in vitest 4) * fix(deps): exclude vulnerable vitest 4.0.x from @sim/testing peer range Tighten the v4 arm of the peer range to >=4.1.0 <5.0.0 so the peer requirement cannot be satisfied by the unpatched 4.0.x builds that GHSA-5xrq-8626-4rwp affects. * fix(testing): make vitest 4 constructor mocks type-check cleanly - logging-session & mcp-oauth mocks: a class passed to mockImplementation has a construct signature that isn't assignable to its (...args) => any parameter, failing tsc. Use named function declarations instead (constructable via Reflect.construct, assignable to mockImplementation, and not rewritten to arrows by biome's useArrowFunction). - database.mock.ts: vitest 4's generic vi.fn typings no longer break the self-referential cycle on the transaction callback's tx param; loosen tx and annotate the callback's return type to resolve the implicit-any errors. * test(isolated-vm): de-flake queue-capacity scheduler tests The 'queue is full' and 'per-owner queued limit' tests relied on 'await sleep(1)' to assume the first request had reached the queue before submitting the overflow request. The first request only enqueues after an async spawn-failure chain (acquireWorker -> spawn exit -> resolve null -> enqueue), which isn't guaranteed within 1ms under CI load — the overflow request then found an empty queue and hit the 200ms queue-wait timeout instead of the capacity rejection. Replace the wall-clock barrier with a deterministic, event-driven one: hold the single global concurrency slot (IVM_MAX_CONCURRENT=1) with an active worker and await an explicit 'dispatched' signal (fired when the worker receives its execute message, after the scheduler counts it active). The follow-up requests then deterministically hit the synchronous enqueue path. Also drops the queue-wait timeout from 200ms to 50ms, so the tests run faster. |
||
|
|
5f0f0edd63 |
improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped * improvements * chore(repo): rebase fixes and quality improvements for realtime split Addresses merge-time issues and gaps from the realtime app split: - Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks - Restore README branding, fix AGENTS.md script reference - Restore TSDoc on workflow-persistence subblocks helpers - Use toError() from @sim/utils/errors in save.ts - Add vitest config + local mocks so @sim/audit tests run standalone - Move socket.io-client to devDependencies in apps/realtime - Add missing package COPY steps to docker/app.Dockerfile - Add check:boundaries/check:realtime-prune scripts and wire into CI Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(security): consolidate crypto primitives into @sim/security Move general-purpose crypto primitives out of apps/sim into the @sim/security package so both apps/sim and apps/realtime can share them. @sim/security exports (all pure, dependency-free): ./compare safeCompare (constant-time HMAC-wrapped equality) ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format) ./hash sha256Hex ./tokens generateSecureToken (base64url) Migrate apps/sim call sites to use these + @sim/utils helpers: crypto.randomUUID() -> generateId() from @sim/utils/id createHash('sha256').digest -> sha256Hex timingSafeEqual on hashed hex -> safeCompare new Promise(setTimeout) -> sleep from @sim/utils/helpers No behavior change: encryption format, digest output, and token length are preserved exactly. * refactor(copilot): use toError in remaining otel/finalize sites Replace the last two `error instanceof Error ? error : new Error(String(error))` patterns with toError from @sim/utils/errors. Completes the sweep of clean candidates — no behavior change. * refactor(security): consolidate HMAC-SHA256 primitives into @sim/security Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates 15 webhook providers plus 5 other hot paths (deployment token signing, outbound webhook requests, workspace notification delivery, notification test route, Shopify OAuth callback) off bare `createHmac` calls. Secret parameter accepts `string | Buffer` to cover base64-decoded Svix-style secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and Textract tools intentionally retains direct `createHmac` usage — its multi-step key derivation chain doesn't fit a generic helper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(packages): post-audit test + packaging polish - Add safeCompare unit tests (identity, length mismatch, hex-nibble diff). - Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract. - Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports. - Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained. - Document the data-field invariant in load.ts for loop/parallel subflow patching. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): address PR review feedback - Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage (env.PORT already defaults to 3002 via zod schema). - Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over the schema default for PORT; keeps SOCKET_PORT functional as an override instead of dead code. - Add dedicated type-check CI step for @sim/realtime so TS errors surface pre-deploy (the Dockerfile runs source TS via Bun and has no implicit build-time type check). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(realtime): remove unused SOCKET_PORT env var SOCKET_PORT has lived in the socket server since the June 2025 refactor but was never actually set in any deploy config — docker-compose.prod, helm values/templates, .env.example, and docs all use PORT or the 3002 default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so removing it is safe. Simplifies realtime port resolution to `env.PORT` (zod-validated with a 3002 default) and drops the orphaned sim-side schema entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Waleed Latif <walif6@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |