Files
sim/scripts
Theodore Li 8f312d299b feat(guardrails): PII redaction via Presidio sidecar (native VIN, per-rule language) (#5174)
* fix(logs): run PII redaction over HTTP and fix Presidio provisioning

- resolve the guardrails venv via candidate paths and fail fast instead of
  silently falling back to system python3 (the misleading "Presidio not
  installed" that broke redaction and the guardrails block in deployed runtimes)
- install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile
- route log redaction through an internal /api/guardrails/mask-batch endpoint
  so Presidio always runs in the app container, including async executions that
  persist inside the trigger.dev runtime

* fix(guardrails): chunk + time-bound internal PII mask requests

- chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large
  executions split across requests and never hit the contract's 100k cap
- add AbortSignal.timeout(45s) per request so a slow/unreachable app container
  aborts and the caller scrubs, instead of hanging the trigger.dev job
- catch maskPIIBatch failures in the route: log and return a structured 500
  (broken venv fails loudly server-side; caller still scrubs, no leak)
- add mask-client tests (order across chunks, count split, non-2xx, empty)

* fix(guardrails): mint internal token per mask request

A single token (5min TTL) could expire mid-batch when a large execution
fans out into many sequential chunk requests; mint one per request instead.

* feat(guardrails): run PII via Presidio sidecars + TS recognizer registry

- replace the per-call python3 subprocess (cold spaCy load every call) with
  two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP;
  the app image no longer carries Python/Presidio/venv
- add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL
- move VIN out of Python into a TS recognizer (check-digit validated) behind a
  CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is
  handled uniformly by the anonymizer
- drive the guardrails block's PII type picker from the shared pii-entities
  catalog (adds VIN, fixes drift) so block + Data Retention never diverge
- delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step

* fix(guardrails): bound-parallelize mask batch; refresh stale comments

- maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via
  mapWithConcurrency, so a chunk of many small leaves finishes within the 45s
  request timeout instead of aborting and scrubbing; order + fail-on-error kept
- drop stale comments referencing the deleted Python venv / 30s subprocess timeout

* refactor(guardrails): single Presidio image, native VIN, per-rule redaction language

- collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image
  serves /analyze + /anonymize)
- remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native +
  multi-language in the image; validate_pii is a thin analyze→anonymize client
- trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image)
- add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum,
  the Data Retention rule modal, and the guardrails block dropdown; resolver +
  logger thread it through to maskPIIBatch (default en), so non-English entity
  rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en

* fix(guardrails): correct sidecar port (5001) + README for combined image

The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize
on a single port 5001 with native VIN + multi-language recognizers. Fix the
PRESIDIO_URL default (was 5002) and rewrite the README, which still described two
stock containers and a TS VIN recognizer.

* fix(guardrails): coerce stored redaction language in the resolver

The persist-path resolver accepted any stored language string, so a stale/invalid
code (e.g. a dropped locale) would reach Presidio and scrub the log even though the
admin UI shows English. Coerce against the supported set via a shared
coercePiiLanguage helper (now reused by the data-retention route too), falling back
to en for unknown values.

* fix(guardrails): rename PRESIDIO_URL env var to PII_URL

Match the infra taskdef, which sets PII_URL on the app container for the
combined Presidio sidecar.
2026-06-23 05:29:01 -04:00
..

Integration documentation generator

generate-docs.ts compiles the per-service integration pages under apps/docs/content/docs/en/integrations/ from the block/tool/trigger registry in apps/sim. The ontology it encodes: everything is a block, and an integration is one block that has Actions and, optionally, a Trigger.

Golden rule: the generated .mdx files are derived artifacts, not the source of truth. Do not hand-edit them — your changes are overwritten on the next run. The only editable region is the MANUAL-CONTENT block (see below). To change what a page says, edit the TypeScript in apps/sim and regenerate.

Where an integration lives canonically

For a service like Gmail, three TS sources define it:

Source What it is What it feeds in the page
apps/sim/blocks/blocks/<service>.ts The block: type, name, category (tools for integrations), bgColor, config sub-blocks, tools.access (which actions it exposes), an optional triggers capability, outputs Header / BlockInfoCard, Usage Instructions, and which actions + trigger appear
apps/sim/tools/<service>/*.ts Each action's params + outputs Every ### <action>#### Input / #### Output under ## Actions
apps/sim/triggers/<provider>/ The trigger's config fields + outputs The ## Triggers section
apps/sim/components/icons.tsx The brand glyph The page icon

The block references actions by id in tools.access; the generator looks each one up in apps/sim/tools/.

What the generator does

Run with cd apps/sim && bun run generate-docs (or bun run scripts/generate-docs.ts from the repo root). One pass (generateAllBlockDocs):

  1. Copies icons apps/sim/components/icons.tsxapps/docs/components/icons.tsx and builds apps/docs/components/ui/icon-mapping.ts.
  2. Block pass — for each integration block (category: 'tools', plus the memory / knowledge / table exceptions), writes integrations/<service>.mdx: BlockInfoCard + Usage Instructions + ## Actions.
  3. Trigger pass (generateAllTriggerDocs) — reads apps/sim/triggers/<provider>/ and appends a ## Triggers section to that service's page, or writes a standalone page for trigger-only services.
  4. Writes integrations/meta.json and regenerates the landing page's integrations.json.

Hand-written pages it never touches

Core block pages (blocks/*), the native trigger pages (triggers/{start,schedule,webhook,rss,table}), the integrations overview (integrations/index.mdx), and the service-account pages are fully hand-written. The generator skips them via HANDWRITTEN_INTEGRATION_DOCS, HANDWRITTEN_TRIGGER_DOCS, and SKIP_TRIGGER_PROVIDERS. Add a page name to those sets if you hand-author a page the generator would otherwise produce.

Manual content (the one editable region)

Each generated page may carry hand-written prose inside marker comments. The generator preserves anything between the markers and overwrites everything else, so this survives every regeneration:

{/* MANUAL-CONTENT-START:intro */}
[AgentMail](https://agentmail.to/) is an API-first email platform…
{/* MANUAL-CONTENT-END */}

Supported section names: intro (after the BlockInfoCard — the most common), usage, configuration, outputs, notes. The merge is by marker name (extractManualContent + mergeWithManualContent), so a section is re-inserted at the matching spot in the freshly generated structure.

If you move the output folder, reseed manual content from the old location first — the generator only preserves markers it finds in the existing output file, so a fresh folder starts with none.

Practical: to change…

  • An action's params/outputs, a trigger, or to add a service → edit apps/sim/{blocks,tools,triggers} and re-run the generator.
  • A page's prose intro → edit its MANUAL-CONTENT:intro block directly; it survives regen.
  • The overview / service-account / core-block / native-trigger pages → hand-edit freely.

Gotchas

  • Never hand-edit apps/docs/components/icons.tsx — step 1 overwrites it from the sim app. Components that need an icon the sim app lacks should define it locally or use lucide-react (see components/workflow-preview/block-icons.tsx).
  • The generator is the source of truth for integrations/ and its meta.json; manual edits there are transient.

CI

The generator runs in CI on pushes to the main branch and commits the regenerated docs back. Keep block/tool/trigger metadata accurate in apps/sim and the docs follow.