mirror of
https://github.com/simstudioai/sim.git
synced 2026-08-31 01:11:53 +08:00
Vikhyath Mondreti
fe0b92bb6d
* feat(secrets): show where a secret is referenced, beside its usage log
"See usage" answered who has run something with a key. It could not answer the
question a rotation actually starts from — where is this wired in — because a
secret four blocks depend on but nothing has executed yet has no usage rows at
all, so the panel read "This secret has not been used yet" for a live key.
The usage view now carries two tabs. Logs is the existing trail, unchanged and
still the default, since that is what the header action has always opened.
References is new: the blocks that name the secret as {{KEY}}, grouped under
their workflow, then the custom tools and MCP servers whose own bodies carry it.
Detection is the workspace-fork remapper's. remapSubBlocks already walks nested
tool-input params, resolves canonical basic/advanced pairs, and skips dormant
and condition-hidden members, so calling it per block inherits every rule a
fork already obeys. Only the aggregation is new: scanWorkflowReferences
collapses its output to unique (kind, sourceId) pairs and discards the workflow
— right for building a mapping table, wrong for locating a key. Nothing under
ee/workspace-forking changed.
- Candidates come from strpos(sub_blocks::text, name) > 0, deliberately not
LIKE: `_` is a LIKE single-character wildcard and nearly every env key
contains one, so SB_ACTION_ROUTER_SECRET would match text it does not occur
in. The prefilter can over-match but never under-match; the scanner decides.
The plan is an index scan on workflow by workspace, nested-looped into
workflow_blocks, so cost tracks the workspace rather than the table.
- Scope gates the read but does not narrow it. A {{KEY}} names a key, not a
scope, so the same sites answer for a workspace secret and the personal one it
shadows; narrowing here would report a personal secret as unreferenced the
moment a workspace variable of the same name existed.
- References reports one field per block, not a list. The remapper dedupes a
block's references by (kind, sourceId), so a block naming the secret twice
yields one entry — the type says so and a test pins it, because the row
renders that field as its whole description.
- Reads live state, not deployed: a draft workflow referencing the key must
show. Blocks are capped and the cap is reported as `truncated` rather than
silently trimming the list.
- Authorization is the existing usage gate, renamed requireSecretTrailReadAccess
and shared verbatim, so the two tabs can never disagree about who may look.
UI is existing primitives only — ChipModalTabs for the strip, DetailSection per
workflow over RESOURCE_LIST_STACK rows, IntegrationTile for the block glyph so a
block reads here as it does on an integrations row, SettingsEmptyState for the
gates. No new component, no new class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): close the reference-scan scope bypass and bound its output
Review round 1.
- use-cases.ts: `scope` was a caller-controlled assertion the reference scan never
narrowed by, so `scope=personal` returned from the shared gate before any check
and handed any workspace member the admin-gated reference map for any workspace
secret. The usage trail can trust that scope because it filters the read by
`secretOwnerUserId`; a name-based workspace-wide scan cannot. References now
authorize on what the NAME resolves to — a workspace secret under that name is
admin-gated outright, and absent one the caller must actually hold a personal
secret of that name, which also stops a member enumerating arbitrary names.
`scope` is dropped from the input, the contract, the hook and the query key
rather than merely ignored: a parameter that does not exist cannot be asserted.
The trail gate keeps its old name and a note saying why only a scope-narrowed
read may reuse it.
- scan.ts: the prefilter matched the bare name, so `API_KEY` also read every block
holding `{{API_KEY_TEST}}` or the words "the API_KEY value" — and those false
positives counted against the row cap, so on a workspace with enough of them
genuine references sorted later were never read at all. It now matches the
reference syntax (`{{name}}`, with the whitespace ENV_REF_PATTERN allows), so a
candidate is a real occurrence and the cap means what it says. A name outside the
env-key charset short-circuits, which is also what makes it safe to inline into
the regex unescaped. Verified against a real workspace: the exact key still
returns its 16 blocks, its prefix now returns 0 where it previously matched all
16, and a metacharacter name touches no query.
- scan.ts: capping tool and server ROWS did not bound the output — one MCP server
emits an entry per matching header plus one for its url, so 200 rows could expand
past the contract's 400-entry bound and make the route reject its own response,
turning a successful scan into a 500 and the tab into "Could not load
references." Emission now stops at the bound and reports `truncated`.
- secret-references-panel.tsx: the empty-state early return preceded the truncation
banner, so a capped scan that filtered everything out claimed the secret was
unreferenced. Both paths now share one note, and silence from a capped scan reads
as absence of evidence rather than evidence of absence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): cover unicode whitespace, legacy keys, and the shadowed-personal tab
Review round 2 — all three follow from round 1's own fixes.
- scan.ts: the syntax prefilter anchored on `[[:space:]]`, but the two engines
disagree about what whitespace is. `ENV_REF_PATTERN`'s `\s` accepts U+00A0,
U+202F and U+3000; Postgres `[[:space:]]` matches only the ASCII set. So a value
pasted with a non-breaking space inside the braces is a reference the executor
resolves and the prefilter silently dropped — the one failure direction this
feature must never take, since the answer it gives is "unused, safe to delete".
Anchoring on `[^[:alnum:]_]` instead accepts every whitespace encoding while
still rejecting a longer key on either side, and needs no code-point list that
could drift. It can admit a non-reference like `{{-NAME-}}`; that costs one
candidate row, and the scanner re-checks every candidate regardless. Erring loose
here is deliberate. (Greptile's `{{\tAPI_KEY\t}}` example was already handled —
tab is ASCII — but the unicode half of the finding was real.)
- use-cases.ts: the gate read `keyAccess.knownKeys` as "a workspace secret exists
under this name", but that set only covers names with an `env_workspace`
credential row. A legacy value written before the ACL existed has no row and
still wins at run time, so it fell through to the personal branch and handed a
non-admin the reference map for exactly the oldest keys. It now reads the
authoritative `workspace_environment.variables` map through a new
`hasWorkspaceEnvValue`, which is documented against `knownKeys` so the two are
not confused again. `getWorkspaceEnvKeyAdminAccess` keeps its existing contract —
its `knownKeys` still answers the ACL question its other callers ask.
- secret-references-panel.tsx: a personal secret shadowed by a same-named workspace
variable could open the view (its owner may read their own Logs) but References
always hit the workspace refusal and rendered a generic load error — a tab
offered in a state where it cannot succeed. The refusal is correct; the tab now
states the shadowing instead of asking for a map it will be denied, reusing the
wording the detail page already shows. No request is made in that state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): re-check the reference gate's volatile input after the scan
Review round 3.
The name-resolution gate reads whether a workspace value exists, then scans. A
workspace secret created between the two makes the map now in hand admin-gated,
so a personal owner could receive it without workspace-secret administration.
The window is small and the data is derivable — a workspace member can already
open every workflow and read its `{{KEY}}` references — but the gate's stated
contract is that references follow the same predicate as revealing the value, and
a point-in-time check that can be overtaken does not honour that. An advisory lock
or a snapshot transaction would serialize a read-only view against secret writes
for it, which is the wrong trade.
Instead the one volatile input is re-read after the scan and the request fails
closed if it flipped. `requireSecretReferencesReadAccess` now reports which branch
authorized: an `admin` grant holds however the name resolves and pays nothing,
while a `personal` grant — the only one resting on absence — is re-checked. A
non-admin loses nothing they were entitled to keep; the request is refused the way
it would have been a moment later.
Adds a `listSecretReferencesUseCase` suite covering both denial paths, the legacy
value, the personal owner, the admin short-circuit, and the race itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): accept JSON-escaped whitespace in the reference prefilter
Review round 4.
The prefilter reads a `::text` rendering of a JSON column, and `jsonb::text`
renders a real tab inside a string value as the literal pair `\` `t`. `t` is
alphanumeric, so `[^[:alnum:]_]` could not consume it and the row was discarded
before `ENV_REF_PATTERN` ever ran — the References tab omitting a live reference
and reporting `truncated: false` while doing it.
Round 3's fix was verified against a raw text value rather than the JSON
rendering, which is exactly why it looked correct: `E'{{\tAPI_KEY\t}}'` matches,
`jsonb_build_object('v', E'{{\tAPI_KEY\t}}')::text` does not.
The gap between `{{` and the name now accepts three encodings at once — raw
characters (covering every Unicode space, which Postgres `[[:space:]]` misses),
JSON two-character escapes, and `\uXXXX` (how a vertical tab survives the same
rendering). Verified against the real jsonb rendering: tab, newline, carriage
return, vertical tab and form feed all recover, U+00A0 / U+3000 / space / plain
keep matching, and `{{API_KEY_TEST}}`, `{{MY_API_KEY}}` and prose are still
rejected — so the row cap keeps meaning what it says.
Plain-text columns (`custom_tools.code`, `mcp_servers.url`) carry no JSON
escaping, but tool code is JavaScript source and can contain the same escape
sequences literally, so the one predicate is right for every column.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(secrets): land the References link on the block, and name its field
Feedback round.
- Logs leads the tab strip. It was already the default tab; the order now says so.
- The usage view drops its resource heading for a plain "Usage" title. The back
chip already names the secret, so the tile and the subtitle underneath were
saying it a second time. `CredentialDetailLayout` gains an optional `title`
that renders the same element, class and column position the settings shell
gives `SettingsPanel` — which is how the sibling Forks "Activity" view titles
itself. Existing callers pass nothing and are unchanged.
- A block row now lands on the block instead of the workflow's default framing.
The editor had no URL params at all, so `?block=` is its first: read once on
arrival, acted on, and stripped. It is a navigation signal rather than canvas
state — the carve-out in sim-url-state.md is about pan, zoom, selection and
drag, which are socket-synced or high-frequency; this is neither, and it rides
in the link so a middle-click or reload keeps it where an in-memory handoff
could not.
The consuming effect mirrors the note-search reveal in the same file, including
the three details that make that one work: read from `displayNodes` so a target
arriving before its node mounts is retried on the mounting commit, route
selection through `resolveSelectionConflicts`, and latch in a ref. It also
claims `userFocusedWorkflowIdRef` the way a node click does, because `onInit`
re-reads that inside its own rAF and would otherwise `fitView` over the camera —
and that ref is reset by exactly the `workflowIdParam` change a deep link causes.
The panel opens for free: `syncPanelWithSelection` already follows selection.
`useSearchParams` needs a Suspense boundary and the editor's ancestry has none,
so the read lives in a leaf under its own `fallback={null}` rather than wrapping
the editor and adding a `loading.tsx` to its mount path. `next build` passes.
- A tool-input reference showed `tools-tool-0-code`. Those
`{subBlockId}-tool-{index}-{paramId}` keys are documented as an ephemeral,
client-only projection of the canonical `tool-input` value and are not meant to
be persisted, but older rows carry them — so the scanner reported whichever the
record yielded last. They are dropped before scanning, which is right even where
the two disagree: `tool.params` is what executes, so a mirror the canonical no
longer matches describes a reference that no longer runs.
- The row now shows the field's label from the block config rather than its
storage id — "API Key", "Tools", "Code", "Bot Token" — falling back to the id
when the block or field is unregistered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): make the reference prefilter exactly as tight as the authority
Review round 6.
The gap between `{{` and the name accepted any non-word character, so
`{{-API_KEY-}}` and `{{"API_KEY"}}` matched in SQL while `ENV_REF_PATTERN`
rejects them. The previous commit called that free — "costs a candidate row and
nothing else" — which was wrong: a candidate row is a slot under
BLOCK_SCAN_LIMIT, so enough near-misses sorted earlier exhaust the cap before a
genuine reference is read, and the tab reports a live key as unused. That is the
same failure the tightening in round 1 was meant to remove, reintroduced by the
round 4 loosening that fixed JSON-escaped whitespace.
The gap now enumerates exactly the whitespace `\s` accepts, in each encoding it
can arrive in: `[[:space:]]` for raw ASCII, `\\[tnrf]` and `\\u000[bB]` for the
JSON escapes, and an explicit class for the Unicode spaces Postgres emits
verbatim but `[[:space:]]` does not match.
That class is generated from a code-point table rather than written literally.
Writing it by hand put a run of invisible characters in the source — a reviewer
cannot check them, and a formatter or editor can silently mangle them. The table
is the readable form and `toPgEscape` renders it.
Verified against the real jsonb rendering, 17 cases: raw space, tab, newline,
carriage return, vertical tab, form feed, U+00A0, U+202F, U+3000 and an embedded
reference all match; `{{-API_KEY-}}`, `{{"API_KEY"}}`, `{{API_KEY_TEST}}`,
`{{MY_API_KEY}}`, prose and an across-braces span all do not. Every candidate the
SQL admits is now a real occurrence, so the cap counts references and nothing
else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): cap the reference scan on results, not candidates
Review round 7.
The prefilter now matches reference syntax exactly, but `remapSubBlocks` filters
further on semantics SQL cannot see: it drops dormant canonical members and
condition-hidden fields. So a block whose only `{{KEY}}` sits in a hidden field is
a genuine candidate that yields nothing, and with the cap counting candidates,
enough of those sorted earlier displaced active references out of the answer.
Unlike the previous two rounds this is not fixable by tightening the prefilter —
no SQL predicate can evaluate canonical modes or field conditions. So the cap
moves to what it should have counted all along: blocks REPORTED. Candidates are
read a page at a time up to a ceiling far above the result limit, so filtered rows
are absorbed as extra reads instead of taking result slots.
Paging rather than one large read because the alternative is holding every
candidate block's `sub_blocks` in memory at once; peak memory is now one page.
`blockId` joins the ordering as a final tiebreak, since OFFSET paging over a
non-unique sort can repeat or skip rows across pages — which here would
double-report a block or silently lose one.
This does not make the scan unconditionally complete, and the ceiling says so:
bounded work and guaranteed completeness cannot both hold, so the only real
choice is where the bound sits and whether it counts something the reader can
see. It now counts results.
Query plan re-checked with the OFFSET in place: still an index scan on workflow by
workspace, nested-looped into workflow_blocks. Test added that pins the fix — 2,500
prose candidates sorted ahead of one real reference, which the previous cap
dropped entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): drop the paging that caused drift, and stop false truncation
Review round 8.
- Paging removed. It bought headroom and paid with drift: `OFFSET` is positional,
so a block renamed, inserted or deleted between page queries shifts the result
set, and the scan skips a live reference or reports one twice. That is a worse
failure than the one paging was added to fix, and it was self-inflicted last
round. Candidates are read in one statement again — one statement is one
snapshot, so neither skew nor duplication is possible — with the ceiling
lowered to 4,000 so a single read stays a sane amount of memory. Result-capping
survives, which was the actual point: filtered rows are still absorbed as extra
reads rather than taking result slots.
- `truncated` no longer fires on an exact landing. The block path now uses the
limit-plus-one read and strict `>` the resource paths already used, so a scan
that ends precisely on a bound reports complete instead of warning about
references that were all returned.
- The deep-link target is released when its block does not exist. It was cleared
only on a match, so a link to a since-deleted block left the id set with the
param already stripped: the effect re-checked on every canvas update forever and
shadowed a later link to the same block. Once any node has mounted the canvas is
populated, so an id still absent is gone and the target is dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(secrets): gate the deep-link release on the workflow being ready
Review round 9.
Round 8 released a deep-link target once `displayNodes` was non-empty, reading
that as "the canvas is populated, so a missing id is deleted". It is not: arriving
from another workflow the store still holds that graph, so nodes are present while
the linked workflow is still hydrating — and a valid `?block=` target was dropped
before its own blocks ever mounted.
The file already had the right predicate. `isWorkflowReady` pins
`hydration.phase === 'ready'`, `hydration.workflowId === workflowIdParam` and
`activeWorkflowId === workflowIdParam`, which is exactly "the graph now loaded is
this workflow's". Absence is only conclusive under that, and a node count never
was — it says something mounted, not whose.
This is the second fix to this release condition in two rounds, both from guessing
at a readiness signal instead of using the one the component already computes for
the same question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Integration documentation generator
generate-docs.ts compiles the per-service integration pages under
apps/docs/content/docs/en/integrations/ from the block/tool/trigger registry in
apps/sim. The ontology it encodes: everything is a block, and an integration is one
block that has Actions and, optionally, a Trigger.
Golden rule: the generated
.mdxfiles are derived artifacts, not the source of truth. Do not hand-edit them — your changes are overwritten on the next run. The only editable region is theMANUAL-CONTENTblock (see below). To change what a page says, edit the TypeScript inapps/simand regenerate.
Where an integration lives canonically
For a service like Gmail, three TS sources define it:
| Source | What it is | What it feeds in the page |
|---|---|---|
apps/sim/blocks/blocks/<service>.ts |
The block: type, name, category (tools for integrations), bgColor, config sub-blocks, tools.access (which actions it exposes), an optional triggers capability, outputs |
Header / BlockInfoCard, Usage Instructions, and which actions + trigger appear |
apps/sim/tools/<service>/*.ts |
Each action's params + outputs | Every ### <action> → #### Input / #### Output under ## Actions |
apps/sim/triggers/<provider>/ |
The trigger's config fields + outputs | The ## Triggers section |
apps/sim/components/icons.tsx |
The brand glyph | The page icon |
The block references actions by id in tools.access; the generator looks each one up in
apps/sim/tools/.
What the generator does
Run with cd apps/sim && bun run generate-docs (or bun run scripts/generate-docs.ts
from the repo root). One pass (generateAllBlockDocs):
- Copies icons
apps/sim/components/icons.tsx→apps/docs/components/icons.tsxand buildsapps/docs/components/ui/icon-mapping.ts. - Block pass — for each integration block (
category: 'tools', plus thememory/knowledge/tableexceptions), writesintegrations/<service>.mdx:BlockInfoCard+ Usage Instructions +## Actions. - Trigger pass (
generateAllTriggerDocs) — readsapps/sim/triggers/<provider>/and appends a## Triggerssection to that service's page, or writes a standalone page for trigger-only services. - Writes
integrations/meta.jsonand regenerates the landing page'sintegrations.json.
Hand-written pages it never touches
Core block pages (blocks/*), the native trigger pages (triggers/{start,schedule,webhook,rss,table}),
the integrations overview (integrations/index.mdx), and the service-account pages are
fully hand-written. The generator skips them via HANDWRITTEN_INTEGRATION_DOCS,
HANDWRITTEN_TRIGGER_DOCS, and SKIP_TRIGGER_PROVIDERS. Add a page name to those sets if
you hand-author a page the generator would otherwise produce.
Manual content (the one editable region)
Each generated page may carry hand-written prose inside marker comments. The generator preserves anything between the markers and overwrites everything else, so this survives every regeneration:
{/* MANUAL-CONTENT-START:intro */}
[AgentMail](https://agentmail.to/) is an API-first email platform…
{/* MANUAL-CONTENT-END */}
Supported section names: intro (after the BlockInfoCard — the most common),
usage, configuration, outputs, notes. The merge is by marker name
(extractManualContent + mergeWithManualContent), so a section is re-inserted at the
matching spot in the freshly generated structure.
If you move the output folder, reseed manual content from the old location first — the generator only preserves markers it finds in the existing output file, so a fresh folder starts with none.
Practical: to change…
- An action's params/outputs, a trigger, or to add a service → edit
apps/sim/{blocks,tools,triggers}and re-run the generator. - A page's prose intro → edit its
MANUAL-CONTENT:introblock directly; it survives regen. - The overview / service-account / core-block / native-trigger pages → hand-edit freely.
Gotchas
- Never hand-edit
apps/docs/components/icons.tsx— step 1 overwrites it from the sim app. Components that need an icon the sim app lacks should define it locally or use@sim/emcn/icons(seecomponents/workflow-preview/block-icons.tsx). - The generator is the source of truth for
integrations/and itsmeta.json; manual edits there are transient.
CI
The generator runs in CI on pushes to the main branch and commits the regenerated docs
back. Keep block/tool/trigger metadata accurate in apps/sim and the docs follow.