mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-31 03:47:12 +08:00
Waleed
4508ec75d2
* fix(cli): resolve blockers and majors from a full command-surface audit Audit of all 222 CLI leaves against a live deployment, plus fixes for every defect it confirmed. Blockers: - An unrecognized --profile resolved to built-in defaults, so a typo silently targeted production and transmitted the API key there. - sim logs follow sent an undeclared query key and failed on every invocation. - sim workflows run exited 0 on a failed run, so CI reported success. - knowledge connectors documents update matched rows already in the target state, making exclude and restore permanent no-ops. - PDF text layers below the OCR threshold are transcribed by a model and stored verbatim with no record that it happened. Majors include: rollback --version was swallowed by the program-level flag and silently did nothing; nullable string flags could not send null despite their help promising it; six protocol commands discarded excess arguments, dropping files on upload; sim chat crashed with EPIPE when piped to head; tables import dropped malformed CSV rows without reporting them; audit-logs required an organization id no API surface exposed; secrets could not opt out of redaction or read a value from a file; bulk deletes and moves exited 0 having done nothing; and MCP registrations were destroyed by undeploy rather than restored. Adds extraction_method to documents so OCR output is distinguishable from parsed text, and reports the applied scope on billing logs so the two ledger questions are no longer indistinguishable. * fix(mcp): bound MCP restore by server and re-check uniqueness under the lock Two gaps in the archive/restore lifecycle this branch introduced. The candidate query bounded archived rows and deduplicated to one per server afterwards, so several archived generations stacked on one server consumed the whole budget and every other server the workflow had been published on fell out of the result with no warning. Deduplication moves into SQL so the bound applies to servers, preserving most-recently-updated-per-server. The live-registration check ran before the server lock was acquired, so a concurrent tool create could land in between and the restore would un-archive a second live row for the same server and workflow, violating the partial unique index and rolling back the whole deployment. That check now runs under the lock alongside the tool-name, capacity, and metadata-budget checks it belongs with. * fix(review): address round-three review findings across CLI and server Restore now picks candidate servers by recency: DISTINCT ON forces its own key to lead the sort, so bounding on that statement kept the lexicographically lowest server ids and left a workflow's most recently used servers archived. Deduplication and bounding are now separate stages. CLI: a total miss on tables move reported only in notFound exited 0; unsetting a key or removing a profile mutated the first duplicate INI block while reads merged later ones, so the removal appeared to succeed and did nothing; an import that rejected cells but no rows showed a clean progress line; and the --run-id help implied idempotency it does not provide. Server: a run with no recorded output projection let block-name selectors past the new validation; the billing window comparison still fired on a bound that parsed but failed the shared schema; CSV rejection accounting reached only the streaming path, so buffered and synchronous imports still dropped records silently, through to the Copilot tool that reports them; case-insensitive tag name uniqueness now serializes on the knowledge-base row the delete paths already lock; and a failed sync claim reports the lifecycle reason rather than always claiming a sync is in progress. Reverts an over-scrub from the previous commit: workspace-file-imports is consumed only by Copilot, so naming save_upload and glob there is the correct remediation rather than a leak, and the sweep that guards against leaks now exempts it explicitly. Corrects two contract descriptions that promised a bulk tag save would rename or relocate an occupied slot, which it deliberately no longer does. * fix(cli): remove flags a caller cannot use, and correct three that misled Removes surface that should not have shipped: - `files uploads get` is hidden. Its `--upload-token` was required, and the token is minted and consumed inside a single `files upload`, which completes or aborts its session before returning. Nothing in the CLI could produce the value, so the command answered every invocation by asking for something unobtainable. The same flag is dropped from the two table-import commands, where a CLI-created import is already queryable without it. - The `--no-<flag>` companion that sent JSON null is gone. `--no-X` means "send boolean false" on thirty-seven other flags, and one spelling should not carry two meanings. `--description ''` already clears the displayed value, and the help now warns that the literal word null is stored as text rather than suggesting a substitute, because on the OAuth client fields null revokes a stored grant and an empty string does not. - The document extraction method column and its contract field are reverted. Nothing read them, they were null for every existing document, and the name collided with the parser metadata field that already exists. Corrects flags that misled: the workflow move destination is `--to`, matching its two siblings rather than meaning the opposite of `--folder` one command over; `files list --recursive` is a bare flag like the four folder deletes rather than a twelve-alias string; the dispatch row cap takes a count instead of its wire object; `--yes` no longer claims to be required on commands that accept `--dry-run`; cancelling every run on a table is confirm-gated; and the retry-processing negation, which the route rejects, is suppressed. Extends the guard that missed all of this: it swept only `--x-` prefixes over generated commands, so a header spelled without one was invisible to it. It now derives every header name from the operation table and sweeps the assembled program. * fix(mcp): budget MCP restore against the workflow's live server fanout Restore bounded its candidates at the per-workflow server limit counting archived rows only, never subtracting the memberships the workflow already holds live. The fanout validation a few lines later in the same deploy transaction counts live servers against that same limit, so a workflow with both live and archived registrations could restore past it and roll the whole deployment back. The candidate query now bounds on the remaining headroom, and the budget is re-checked under the server locks and spent once per accepted candidate, so a create that lands between the count and the unarchive cannot push it over. Candidates that do not fit are dropped by recency, matching how the set is already selected, and stay archived with a warning naming the workflow, the server, and the reason — restore still never throws inside the deploy transaction. One residual is left open deliberately: a create on a server outside the candidate set is serialized by neither the locks nor the recount. Closing it would need a workflow-level lock, which would change the ordering every other writer here depends on, and the create path runs its own limit check. * fix(mcp): stop restore spending its budget on servers it cannot restore A server can hold both a live registration and archived ones for the same workflow: the partial unique index constrains only the live row. Such a server was counted twice — once shrinking the restore budget, once consuming one of its slots — before the liveness check under the lock skipped it. While the bound was the full server limit that waste was invisible; once the bound became the remaining headroom, every slot spent that way cost a registration that could have been restored. The candidate query now excludes servers the workflow is already live on, so the budget and the candidate set agree. The exclusion sits on the inner stage, before deduplication and the bound, and is a pre-lock optimisation only: the check under the server lock stays authoritative, because the query can go stale between reading and unarchiving. Candidates rejected for a tool-name collision, the per-server cap, or the metadata budget are still not replaced. That case is only knowable under the lock, so replacing it would mean fetching past the bound and locking servers outside the candidate set, widening an ordering every writer here relies on.
Integration documentation generator
generate-docs.ts compiles the per-service integration pages under
apps/docs/content/docs/en/integrations/ from the block/tool/trigger registry in
apps/sim. The ontology it encodes: everything is a block, and an integration is one
block that has Actions and, optionally, a Trigger.
Golden rule: the generated
.mdxfiles are derived artifacts, not the source of truth. Do not hand-edit them — your changes are overwritten on the next run. The only editable region is theMANUAL-CONTENTblock (see below). To change what a page says, edit the TypeScript inapps/simand regenerate.
Where an integration lives canonically
For a service like Gmail, three TS sources define it:
| Source | What it is | What it feeds in the page |
|---|---|---|
apps/sim/blocks/blocks/<service>.ts |
The block: type, name, category (tools for integrations), bgColor, config sub-blocks, tools.access (which actions it exposes), an optional triggers capability, outputs |
Header / BlockInfoCard, Usage Instructions, and which actions + trigger appear |
apps/sim/tools/<service>/*.ts |
Each action's params + outputs | Every ### <action> → #### Input / #### Output under ## Actions |
apps/sim/triggers/<provider>/ |
The trigger's config fields + outputs | The ## Triggers section |
apps/sim/components/icons.tsx |
The brand glyph | The page icon |
The block references actions by id in tools.access; the generator looks each one up in
apps/sim/tools/.
What the generator does
Run with cd apps/sim && bun run generate-docs (or bun run scripts/generate-docs.ts
from the repo root). One pass (generateAllBlockDocs):
- Copies icons
apps/sim/components/icons.tsx→apps/docs/components/icons.tsxand buildsapps/docs/components/ui/icon-mapping.ts. - Block pass — for each integration block (
category: 'tools', plus thememory/knowledge/tableexceptions), writesintegrations/<service>.mdx:BlockInfoCard+ Usage Instructions +## Actions. - Trigger pass (
generateAllTriggerDocs) — readsapps/sim/triggers/<provider>/and appends a## Triggerssection to that service's page, or writes a standalone page for trigger-only services. - Writes
integrations/meta.jsonand regenerates the landing page'sintegrations.json.
Hand-written pages it never touches
Core block pages (blocks/*), the native trigger pages (triggers/{start,schedule,webhook,rss,table}),
the integrations overview (integrations/index.mdx), and the service-account pages are
fully hand-written. The generator skips them via HANDWRITTEN_INTEGRATION_DOCS,
HANDWRITTEN_TRIGGER_DOCS, and SKIP_TRIGGER_PROVIDERS. Add a page name to those sets if
you hand-author a page the generator would otherwise produce.
Manual content (the one editable region)
Each generated page may carry hand-written prose inside marker comments. The generator preserves anything between the markers and overwrites everything else, so this survives every regeneration:
{/* MANUAL-CONTENT-START:intro */}
[AgentMail](https://agentmail.to/) is an API-first email platform…
{/* MANUAL-CONTENT-END */}
Supported section names: intro (after the BlockInfoCard — the most common),
usage, configuration, outputs, notes. The merge is by marker name
(extractManualContent + mergeWithManualContent), so a section is re-inserted at the
matching spot in the freshly generated structure.
If you move the output folder, reseed manual content from the old location first — the generator only preserves markers it finds in the existing output file, so a fresh folder starts with none.
Practical: to change…
- An action's params/outputs, a trigger, or to add a service → edit
apps/sim/{blocks,tools,triggers}and re-run the generator. - A page's prose intro → edit its
MANUAL-CONTENT:introblock directly; it survives regen. - The overview / service-account / core-block / native-trigger pages → hand-edit freely.
Gotchas
- Never hand-edit
apps/docs/components/icons.tsx— step 1 overwrites it from the sim app. Components that need an icon the sim app lacks should define it locally or use@sim/emcn/icons(seecomponents/workflow-preview/block-icons.tsx). - The generator is the source of truth for
integrations/and itsmeta.json; manual edits there are transient.
CI
The generator runs in CI on pushes to the main branch and commits the regenerated docs
back. Keep block/tool/trigger metadata accurate in apps/sim and the docs follow.