* feat(data-retention): granular PII redaction stages (input + block outputs)
* fix(data-retention): propagate block-output redaction into child workflows
* fix(data-retention): close block-output redaction gaps on streaming + resume
* fix(data-retention): drain+mask streamed output, resolve PII policy unconditionally (no fail-open)
* test(testing): support leftJoin().where().limit() in shared db mock
* fix(data-retention): mask agent/Pi memory writes under block-output redaction
* fix(data-retention): guard partial PII stages in GET normalize
* fix(data-retention): mask seeded memory messages under block-output redaction
* fix(guardrails): fail closed on misaligned Presidio batch responses
* fix(data-retention): enabled stage with no entity types redacts all (no fail-open)
* fix(data-retention): reject enabled stage with no entity types; empty = off everywhere
* docs(data-retention): note resume remask covers inline values only
* fix(data-retention): scrub offloaded large-value refs from logs when block-output redaction is off
* fix(data-retention): hydrate, mask, and re-store large-value refs in logs (preserve redacted content)
* fix(data-retention): always apply logs policy to large-value refs when logs stage is on
* perf(data-retention): drop redaction byte ceiling, parallelize chunks (env-tunable), remove request timeouts, sync large-value walk
* feat(data-retention): gate granular PII stages behind pii-granular-redaction flag
- New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION),
layered on pii-redaction, gating the execution-altering input + block-output stages
- Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off
- UI shows only the Logs stage tab unless the flag is on; clamps active stage
- Drop the per-search Select all toggle; add a Deselect all action to the PII section header
* docs(pii): describe Presidio as a standalone service, not a sidecar
Presidio now runs as its own ECS service (and, in Helm, its own Deployment +
Service) reached over the network via PII_URL — not a sidecar in the app task.
Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to
match, and note the deploy requirement that PII_URL must be reachable.
* fix(data-retention): re-mask offloaded large-value refs on resume + don't lock out granular saves
- Resume/run-from-block restore now hydrates → masks → re-stores large-value refs
in restored blockStates (not just inline strings), so a value offloaded before the
block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast.
- pii-large-values: add onFailure mode (throw on the execution path, scrub for logs)
and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values
- Granular flag gate now rejects only NEW off→on granular enablement, so orgs that
already configured granular stages can still save retention settings when the flag is off
* feat(secrets): ingest env secrets at container runtime instead of fanning into ECS taskdef
The app/socket ECS taskdefs were ~42KB, ~93% of which was the secrets[] array:
268 pointer entries each restating the full ~78-char secret ARN, marching toward
the 64KB taskdef limit and growing ~150 bytes per hosted key added. The secret
blob itself is only ~18KB/268 keys.
Move secret delivery to container boot: new @sim/runtime-secrets loadRuntimeSecrets()
reads SIM_ENV_SECRET_ID, fetches the combined secret once, and hydrates process.env
(no-clobber, no-op when unset, fail-fast). Bootstrap entrypoints for app + realtime
await it before importing the real server (env-flags reads env at module load). The
app bootstrap is bun-bundled in the Dockerfile builder stage since it runs outside
the Next standalone bundle; realtime keeps full node_modules and runs the TS entry.
Backward-compatible: with the current fan-out taskdef the loader no-ops and the app
reads the injected env vars unchanged. The matching infra change (empty secrets[] +
SIM_ENV_SECRET_ID) ships separately, after this image is live.
* fix(runtime-secrets): address review feedback
- Move the binary-secret guard outside the retry loop (sendWithRetry) so a
missing SecretString throws immediately instead of burning 3 attempts + backoff.
- Bound each Secrets Manager request with AbortSignal.timeout(5s) so a stalled
response can't hang boot indefinitely.
- Drop the redundant @aws-sdk/client-secrets-manager pin from apps/realtime; it
resolves transitively via @sim/runtime-secrets.
- Add a test for the non-retriable binary-secret path.
* fix(logs): run PII redaction over HTTP and fix Presidio provisioning
- resolve the guardrails venv via candidate paths and fail fast instead of
silently falling back to system python3 (the misleading "Presidio not
installed" that broke redaction and the guardrails block in deployed runtimes)
- install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile
- route log redaction through an internal /api/guardrails/mask-batch endpoint
so Presidio always runs in the app container, including async executions that
persist inside the trigger.dev runtime
* fix(guardrails): chunk + time-bound internal PII mask requests
- chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large
executions split across requests and never hit the contract's 100k cap
- add AbortSignal.timeout(45s) per request so a slow/unreachable app container
aborts and the caller scrubs, instead of hanging the trigger.dev job
- catch maskPIIBatch failures in the route: log and return a structured 500
(broken venv fails loudly server-side; caller still scrubs, no leak)
- add mask-client tests (order across chunks, count split, non-2xx, empty)
* fix(guardrails): mint internal token per mask request
A single token (5min TTL) could expire mid-batch when a large execution
fans out into many sequential chunk requests; mint one per request instead.
* feat(guardrails): run PII via Presidio sidecars + TS recognizer registry
- replace the per-call python3 subprocess (cold spaCy load every call) with
two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP;
the app image no longer carries Python/Presidio/venv
- add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL
- move VIN out of Python into a TS recognizer (check-digit validated) behind a
CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is
handled uniformly by the anonymizer
- drive the guardrails block's PII type picker from the shared pii-entities
catalog (adds VIN, fixes drift) so block + Data Retention never diverge
- delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step
* fix(guardrails): bound-parallelize mask batch; refresh stale comments
- maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via
mapWithConcurrency, so a chunk of many small leaves finishes within the 45s
request timeout instead of aborting and scrubbing; order + fail-on-error kept
- drop stale comments referencing the deleted Python venv / 30s subprocess timeout
* refactor(guardrails): single Presidio image, native VIN, per-rule redaction language
- collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image
serves /analyze + /anonymize)
- remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native +
multi-language in the image; validate_pii is a thin analyze→anonymize client
- trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image)
- add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum,
the Data Retention rule modal, and the guardrails block dropdown; resolver +
logger thread it through to maskPIIBatch (default en), so non-English entity
rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en
* fix(guardrails): correct sidecar port (5001) + README for combined image
The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize
on a single port 5001 with native VIN + multi-language recognizers. Fix the
PRESIDIO_URL default (was 5002) and rewrite the README, which still described two
stock containers and a TS VIN recognizer.
* fix(guardrails): coerce stored redaction language in the resolver
The persist-path resolver accepted any stored language string, so a stale/invalid
code (e.g. a dropped locale) would reach Presidio and scrub the log even though the
admin UI shows English. Coerce against the supported set via a shared
coercePiiLanguage helper (now reused by the data-retention route too), falling back
to en for unknown values.
* fix(guardrails): rename PRESIDIO_URL env var to PII_URL
Match the infra taskdef, which sets PII_URL on the app container for the
combined Presidio sidecar.
* fix(pii): bind a configurable $PORT to avoid app :3000 collision
The pii image hardcoded uvicorn --port 3000 and ignored env. In the app ECS
task (awsvpc) all containers share one network namespace, and the app owns
3000 — so the sidecar must listen elsewhere (the stock presidio images honored
PORT and ran on 5002/5001). Bind ${PORT} (shell-form CMD), default 5001, and
update EXPOSE/HEALTHCHECK accordingly so the taskdef can set PORT=5001.
Verified: default binds 5001; PORT=5002 override binds 5002; /analyze works on
the overridden port.
* fix(pii): hardcode port 5001 (drop $PORT indirection)
EXPOSE can't be parameterized, so the configurable-PORT approach left EXPOSE
showing 5001 regardless (Greptile P2). We own both the image and the taskdef
and only ever need 5001, so hardcode it: exec-form CMD on 5001, EXPOSE 5001,
healthcheck on 5001. Runtime cmdline is identical to the verified ${PORT}
default (uvicorn ... --port 5001).
* feat(presidio): build & own combined analyzer+anonymizer image
Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single
image we build and push to ECR/GHCR. A thin FastAPI service constructs one
AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000
(/health, /supportedentities, /analyze, /anonymize) so the app needs one
PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0.
Bakes in the native check-digit VIN recognizer and registers 12 English
recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*,
SG_*), taking the supported English set from 19 to 32.
* feat(presidio): add multi-language support (es/it/pl/fi)
Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and
explicitly register the national-id recognizers Presidio ships but does not
load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/
IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine +
explicit-registration path detects in-language (Finnish id, score 1.0).
* improvement(presidio): address review feedback
- Register VIN under all served languages, not just en (Bugbot: VIN missed for
non-English language routing).
- Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot).
- Drop --no-cache-dir so the pip cache mount actually works (Greptile).
- Pydantic request models for /analyze + /anonymize so missing 'text' returns
422 not 500; default operator 'type' to 'replace' instead of KeyError->500
(Greptile).
* refactor(pii): rename presidio image artifacts to pii
Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the
service does PII detection + anonymization (and backs the guardrails block's
block/mask), not just redaction, and 'pii' matches existing pii-* naming.
docker/presidio.Dockerfile -> docker/pii.Dockerfile
docker/presidio/ -> docker/pii/
ghcr.io/simstudioai/presidio -> .../pii
ECR_PRESIDIO secret -> ECR_PII (infra side already renamed)
No behavior change — paths/identifiers only.
* refactor(pii): move service to apps/pii, make image ECR-only
- Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs
under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal
@sim/pii package.json so the apps/* bun workspace glob accepts the Python service.
- Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user
presidio -> pii.
- Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published.
Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64
tag step to skip GHCR when no ghcr_image is set.
* fix(docker): restore NEXT_PUBLIC_APP_URL build arg with dummy fallback
getBaseUrl() in lib/core/utils/urls is evaluated at module load during
next build's page-data collection and throws if NEXT_PUBLIC_APP_URL is
unset. PR #4658 removed the build arg, breaking the Docker build at the
"/_not-found" page-data collection step.
Restore the dummy localhost fallback (mirroring DATABASE_URL). The CORS
fix from #4658 is preserved: next.config.ts no longer reads
NEXT_PUBLIC_APP_URL at build time, and no module-level expression
captures getBaseUrl() — every caller invokes it at request time, where
getEnv() reads the deployed container env. The dummy localhost value
cannot leak into runtime CORS response headers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(docker): trim verbose comment on build-time env args
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(security): remove localhost CORS origin, consolidate CORS in proxy
Move all /api/* CORS handling from next.config.ts to proxy.ts so the
runtime can resolve allowed origin per-request instead of baking it at
build time (which produced "Access-Control-Allow-Origin: http://localhost:3000"
with credentials:true in production).
- proxy.ts: per-route CORS policy table covering auth, MCP, form, and
workflow execute endpoints; OPTIONS preflight short-circuit; Vary:
Origin when origin is not '*'; form routes defer to route handler's
addCorsHeaders to avoid double-setting
- next.config.ts: drop all /api/* Access-Control-Allow-* headers; keep
COEP/COOP/CSP
- deployment.ts: addCorsHeaders sets Vary: Origin alongside reflected
Allow-Origin
- Dockerfile: drop NEXT_PUBLIC_APP_URL build placeholder (Zod has
skipValidation:true; build path doesn't read it)
- Remove 8 dead OPTIONS handlers and their preflight tests now that the
proxy handles preflight uniformly
* refactor(cors): consolidate API CORS into proxy as single source of truth
Move CORS for /api/chat/* and /api/form/* into the proxy policy table with
reflected-origin + credentials:false, and delete the per-route addCorsHeaders
helper. Routes no longer set CORS headers — the proxy is the only writer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(cors): convert proxy CORS policy chain to a rule table + add tests
Replace the if/else chain in resolveApiCorsPolicy with a CORS_RULES table
so each route's policy lives in one place and is trivially scannable.
Add proxy.test.ts covering each rule and the wildcard-with-credentials
invariant.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cors): scope embed CORS rule to /api/{chat,form}/[identifier] only
The embed policy (reflected origin, credentials:false) was matching
workspace-internal session-authed routes — /api/chat, /api/chat/manage/*,
/api/chat/validate, and the form equivalents — which need the default
credentialed policy. Tighten the matcher to the embed paths only and add
tests covering the exclusion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(cors): replace embed-path regex with explicit segment check
The regex form `^/api/(chat|form)/(?!manage|validate)[^/]+(/(otp|sso))?$`
was opaque on review and would silently exclude any future identifier
subroute outside the hard-coded (otp|sso) group from the embed policy.
Replace it with an imperative segment check and a named
EMBED_RESERVED_SEGMENTS Set, so the policy boundary is visible at the
top of the function and adding a reserved subpath is a one-line diff.
Add a test asserting that future identifier subroutes also get the
embed policy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(cors): allow PUT in embed CORS policy for OTP verification
Both /api/chat/[identifier]/otp and /api/form/[identifier]/otp export
PUT for OTP code verification. The embed policy advertised only
GET/POST/OPTIONS, so cross-origin embed clients failed preflight on
verify. Add PUT and assert it in the embed policy test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(auth): resolve CORS errors for self-hosted deployments behind reverse proxies
- auth client now uses browser origin first, falling back to NEXT_PUBLIC_APP_URL
- socket client falls back to page origin when served from non-localhost (assumes /socket.io is proxied)
- add TRUSTED_ORIGINS env var to extend Better Auth trustedOrigins (apex+www, alias hostnames)
- warn at startup when NEXT_PUBLIC_APP_URL is localhost in production
- preprocess empty NEXT_PUBLIC_SOCKET_URL so docker-compose ${VAR:-} works
- migrate remaining uuid/nanoid/randomUUID usages to @sim/utils generateId/generateShortId
- extend generateShortId with optional alphabet param (rejection sampling)
- document TRUSTED_ORIGINS in .env.example, docker-compose.prod.yml, and helm values.yaml
Fixessimstudioai/sim#1243
* fix(auth): address PR review comments
* chore(env): drop unnecessary NEXT_PUBLIC_SOCKET_URL preprocess (skipValidation is true)
* fix(docker): include @sim/utils in migrations image
Migration scripts now import generateId from @sim/utils/id; without copying packages/utils into the image, bun install fails to resolve the workspace dep at build time and the import fails at runtime.
* fix(helm): remove unused NEXT_PUBLIC_SOCKET_URL from realtime sections
The realtime service never reads NEXT_PUBLIC_SOCKET_URL — its env schema
only includes BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, ALLOWED_ORIGINS,
BETTER_AUTH_SECRET, INTERNAL_API_SECRET, DATABASE_URL, and REDIS_URL.
Remove the dead config from all helm values files and the values schema.
* fix(helm): allow empty NEXT_PUBLIC_SOCKET_URL in values schema
The default in values.yaml is now "" (empty string), which falls back to
the page origin at runtime. The schema previously required a valid URI,
which would reject the default. Mirror the INTERNAL_API_BASE_URL pattern
using anyOf with const "". Also add TRUSTED_ORIGINS to the schema.
* docs(self-hosting): mark NEXT_PUBLIC_SOCKET_URL as optional
The page-origin fallback in getSocketUrl() means self-hosters no longer
need to set NEXT_PUBLIC_SOCKET_URL when realtime is on the same origin
as the app. Update docs to reflect this:
- Remove NEXT_PUBLIC_SOCKET_URL from .env scaffolding examples in
docker.mdx, platforms.mdx, environment-variables.mdx
- Mark the variable as Optional in the env vars table with the new
default behavior described
- Update troubleshooting to point at reverse-proxy /socket.io routing
rather than the env var
- Flip dev docker-compose defaults (local, ollama, devcontainer) from
http://localhost:3002 to empty for consistency with prod.yml; the
in-code localhost fallback handles the dev case identically
Applied across all 6 documentation languages (en/fr/de/ja/es/zh).
* chore: untrack and ignore .claude/scheduled_tasks.lock
* fix(docker): use full bun.lock and copy it into builder
The staging build for app.Dockerfile (commit dc2022995, PR #4322) is
failing in two ways after switching to turbo prune:
1. turbo 2.9.6's pruned bun.lock is malformed for bun 1.3.x:
error: Failed to resolve prod dependency 'wrap-ansi' for package
'log-update' at bun.lock:2688:5
Bun ignores it and falls back to a fresh resolve (~7m install).
2. Next.js 16.1.6's Turbopack production build can't infer the workspace
root because /app/bun.lock doesn't exist in the builder stage:
Error: We couldn't find the Next.js package (next/package.json)
from the project directory: /app/apps/sim
This blocks the build entirely.
Fix:
- deps stage: use the full bun.lock from /app/bun.lock (the original
lockfile after `COPY . .` in pruner) instead of the broken
/app/out/bun.lock that turbo prune emits.
- builder stage: also copy the full bun.lock to /app/bun.lock so
Turbopack and turborepo can detect the workspace root.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(ci): bump deprecated Node.js 20 actions to Node.js 24 versions
GitHub Actions runners emit deprecation warnings for actions still on the
Node.js 20 runtime. Node.js 20 will be force-upgraded by GitHub on
2026-06-02 and removed on 2026-09-16.
Bumps to the latest stable major versions, all of which use Node.js 24:
- actions/cache: v4 -> v5
- actions/setup-node: v4 -> v6
- aws-actions/configure-aws-credentials: v4 -> v6
- docker/login-action: v3 -> v4
All require GHA runner v2.327.1+ which Blacksmith already runs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* perf(docker): refactor app.Dockerfile to use turbo prune
Replaces manual workspace package.json copies with `turbo prune sim --docker`,
matching the canonical Vercel/Turborepo monorepo pattern (and the existing
realtime.Dockerfile).
- New `pruner` stage emits `out/json` (manifests + lockfile) and `out/full`
(sources) for only the packages sim actually depends on.
- `deps` stage installs from the pruned manifest, so cache invalidates only
when package.json/bun.lock change — not on source edits.
- Drops 24 lines of brittle manual COPYs (one per workspace package).
- Single full install in deps (no --omit=dev) so build-time devDeps like
tailwindcss/postcss are available — replaces the earlier hotfix that did
a second install in the builder stage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(docker): pin turbo to 2.9.6 in pruner stage
Match the version locked in package.json so pruner output is reproducible
across builds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* improvement(repo): restructuring to make realtime image narrower scoped
* improvements
* chore(repo): rebase fixes and quality improvements for realtime split
Addresses merge-time issues and gaps from the realtime app split:
- Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks
- Restore README branding, fix AGENTS.md script reference
- Restore TSDoc on workflow-persistence subblocks helpers
- Use toError() from @sim/utils/errors in save.ts
- Add vitest config + local mocks so @sim/audit tests run standalone
- Move socket.io-client to devDependencies in apps/realtime
- Add missing package COPY steps to docker/app.Dockerfile
- Add check:boundaries/check:realtime-prune scripts and wire into CI
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(security): consolidate crypto primitives into @sim/security
Move general-purpose crypto primitives out of apps/sim into the
@sim/security package so both apps/sim and apps/realtime can share them.
@sim/security exports (all pure, dependency-free):
./compare safeCompare (constant-time HMAC-wrapped equality)
./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format)
./hash sha256Hex
./tokens generateSecureToken (base64url)
Migrate apps/sim call sites to use these + @sim/utils helpers:
crypto.randomUUID() -> generateId() from @sim/utils/id
createHash('sha256').digest -> sha256Hex
timingSafeEqual on hashed hex -> safeCompare
new Promise(setTimeout) -> sleep from @sim/utils/helpers
No behavior change: encryption format, digest output, and token
length are preserved exactly.
* refactor(copilot): use toError in remaining otel/finalize sites
Replace the last two `error instanceof Error ? error : new Error(String(error))`
patterns with toError from @sim/utils/errors. Completes the sweep of clean
candidates — no behavior change.
* refactor(security): consolidate HMAC-SHA256 primitives into @sim/security
Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates
15 webhook providers plus 5 other hot paths (deployment token signing,
outbound webhook requests, workspace notification delivery, notification
test route, Shopify OAuth callback) off bare `createHmac` calls. Secret
parameter accepts `string | Buffer` to cover base64-decoded Svix-style
secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and
Textract tools intentionally retains direct `createHmac` usage — its
multi-step key derivation chain doesn't fit a generic helper.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(packages): post-audit test + packaging polish
- Add safeCompare unit tests (identity, length mismatch, hex-nibble diff).
- Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract.
- Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports.
- Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained.
- Document the data-field invariant in load.ts for loop/parallel subflow patching.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(realtime): address PR review feedback
- Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage
(env.PORT already defaults to 3002 via zod schema).
- Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over
the schema default for PORT; keeps SOCKET_PORT functional as an
override instead of dead code.
- Add dedicated type-check CI step for @sim/realtime so TS errors
surface pre-deploy (the Dockerfile runs source TS via Bun and has
no implicit build-time type check).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(realtime): remove unused SOCKET_PORT env var
SOCKET_PORT has lived in the socket server since the June 2025 refactor
but was never actually set in any deploy config — docker-compose.prod,
helm values/templates, .env.example, and docs all use PORT or the 3002
default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so
removing it is safe.
Simplifies realtime port resolution to `env.PORT` (zod-validated with a
3002 default) and drops the orphaned sim-side schema entry.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(execution): run pptx/docx/pdf generation inside isolated-vm sandbox
Retires the legacy doc-worker.cjs / pptx-worker.cjs pipeline that ran user
DSL via node:vm + full require() in the same UID/PID namespace as the main
Next.js process. User code now runs inside the existing isolated-vm pool
(V8 isolate, no process / require / fs, no /proc/1/environ reachability).
Introduces a first-class SandboxTask abstraction under apps/sim/sandbox-tasks/
that mirrors apps/sim/background/ — one file per task, central typed
registry, kebab-case ids. Adding a new thing that runs in the isolate is
one file plus one registry entry.
Runtime additions in lib/execution/:
- task-mode execution in isolated-vm-worker.cjs: load pre-built library
bundles, run task bootstrap, run user code, run finalize, transfer
Uint8Array result as base64 via IPC
- named broker IPC bridge (generalizes the existing fetch bridge) with
args size, result size, and per-execution call caps
- cooperative AbortSignal support: cancel IPC disposes the isolate, pool
slot is freed, pending broker-call timers are swept
- compiled scripts + references explicitly released per execution
- isolate.isDisposed used for cancellation detection (no error-string
substring matching)
Library bundles (pptxgenjs, docx, pdf-lib) are built into isolate-safe
IIFE bundles by apps/sim/lib/execution/sandbox/bundles/build.ts and
committed; next.config.ts / trigger.config.ts / Dockerfile updated to
ship them instead of the deleted dist/*-worker.cjs artifacts.
Call sites migrated:
- app/api/workspaces/[id]/pptx/preview/route.ts
- app/api/files/serve/[...path]/route.ts (+ test mock)
- lib/copilot/tools/server/files/{workspace-file,edit-content}.ts
All pass owner key user:<userId> for per-user pool fairness + distributed
lease accounting.
Made-with: Cursor
* improvement(sandbox): delegate timers to Node, add phase timings + saturation logs
Follow-ups on top of the isolated-vm migration (da14027b2):
Timer delegation (laverdet/isolated-vm#136 recommended pattern):
- setTimeout / setInterval / clearTimeout / clearImmediate delegate to
Node's real timer heap via ivm.Reference. Real delays are honored;
clearTimeout actually cancels; ms is clamped to the script timeout
so callbacks can't fire after the isolate is disposed.
- Per-execution timer tracking + dispose-sweep in finally. Zero stale
callbacks post-dispose.
- unwrapPrimitive helper normalizes ivm.Reference-wrapped primitives
(arguments: { reference: true } applies uniformly to all args).
- _polyfills.ts shrinks from ~130 lines to the global->globalThis alias.
Timers / TextEncoder / TextDecoder / console all install per-execution
from the worker via ivm bridges.
AbortSignal race fix (pre-existing bug surfaced by the timer smoke):
- Listener is registered after await tryAcquireDistributedLease. If the
signal aborted during that ~200ms window (Redis down), AbortSignal
doesn't fire listeners registered after the fact — the abort was
silently missed. Now re-checks signal.aborted synchronously after
addEventListener.
Observability:
- executeTask returns IsolatedVMTaskTimings (setup, runtimeBootstrap,
bundles, brokerInstall, taskBootstrap, harden, userCode, finalize,
total) in every success + error path. run-task.ts logs these with
workspaceId + queueMs so 'which tenant is slow' is queryable.
- Pool saturation events now emit structured logger.warn with reason
codes: queue_full_global, queue_full_owner, queue_wait_timeout,
distributed_lease_limit. Matches the existing broker reject pattern.
Security policy:
- New .cursor/rules/sim-sandbox.mdc codifies the hard rules for the
worker process: no app credentials, all credentialed work goes
through host-side brokers, every broker scopes by workspaceId.
Pre-merge checklist for future changes to isolated-vm-worker.cjs.
Measured phase breakdown (local smoke, Redis down): pptx wall=~310ms
with bundles=~16ms, finalize=~83ms; docx ~290ms / 17ms / 70ms; pdf
~235ms / 17ms / 5ms. Bundle compilation is not the bottleneck —
library finalize is.
Made-with: Cursor
* fix(sandbox): thread AbortSignal into runSandboxTask at every call site
Three remaining callers of runSandboxTask were not threading a
cancellation signal, so a client disconnect mid-compile left the pool
slot occupied for the full 60s task timeout. Matching the pattern the
pptx-preview route already uses.
- apps/sim/app/api/files/serve/[...path]/route.ts — GET forwards
`request.signal` into handleLocalFile / handleCloudProxy, which
forward into compileDocumentIfNeeded, which forwards into
runSandboxTask.
- apps/sim/lib/copilot/tools/server/files/workspace-file.ts — passes
`context.abortSignal` (transport/user stop) into runSandboxTask.
- apps/sim/lib/copilot/tools/server/files/edit-content.ts — same.
Smoke: simulated client disconnect at t=1000ms during a task that would
otherwise have waited 10s. The pool slot unwinds at t=1002ms with
AbortError; previously would have sat 60s until the task-level timeout.
Made-with: Cursor
* chore(build): raise node heap to 8GB for next build type-check
Next.js's type-check worker OOMs at the default 4GB heap on Node 23 for
this project's type graph size. Bumps the heap to 8GB only for the
`next build` invocation inside `bun run build`.
Docker builds are unaffected — `next.config.ts` sets
`typescript.ignoreBuildErrors: true` when DOCKER_BUILD=1, which skips
the type-check pass entirely. This only fixes local `bun run build`.
No functional code changes.
Made-with: Cursor
* fix lint
* refactor(copilot): dedup getDocumentFormatInfo across copilot file tools
The same extension -> { formatName, sourceMime, taskId } mapping was
duplicated in workspace-file.ts and edit-content.ts. Any future format
or task-id change had to happen in two places.
Exports getDocumentFormatInfo + DocumentFormatInfo from workspace-file.ts
(which already owned the PPTX/DOCX/PDF source MIME constants) and
imports it in edit-content.ts. Same source-of-truth pattern the file
already uses for inferContentType.
Made-with: Cursor
* fix(sandbox): propagate empty-message broker/fetch errors
Both bridges in the isolate used truthiness to detect host-side errors:
if (response.error) throw new Error(response.error); // broker
if (result.error) throw new Error(result.error); // fetch
If a host handler ever threw `new Error('')`, err.message would be ''
(falsy), so { error: '' } was silently swallowed and the isolate saw
a successful null result. Existing call sites don't throw empty-message
errors, but the pattern was structurally unsafe.
Switch both to typeof check === 'string' and fall back to a default
message if the string is empty, so all host-reported errors propagate
into the isolate regardless of message content.
Made-with: Cursor
* improvement(ci): parallelize Docker builds with tests and remove duplicate turbo install
* fix(test): use SecureFetchResponse shape in mock instead of standard Response
* feat(tools): added speech to text with openai whisper, elevenlabs, and deepgram
* added new file icons, implemented ffmpeg
* updated docs
* revert environment
* added blacksmith optimizations to workflows and dockerfiles to enhance performance. please review before pushing to production
* remove cache from and cache to directives from docker based actions, per blacksmith docs
---------
Co-authored-by: Connor Mulholland <connormul@Connors-MacBook-Pro.local>
Theodore Li