mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
Theodore Li
8f312d299b
* fix(logs): run PII redaction over HTTP and fix Presidio provisioning - resolve the guardrails venv via candidate paths and fail fast instead of silently falling back to system python3 (the misleading "Presidio not installed" that broke redaction and the guardrails block in deployed runtimes) - install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile - route log redaction through an internal /api/guardrails/mask-batch endpoint so Presidio always runs in the app container, including async executions that persist inside the trigger.dev runtime * fix(guardrails): chunk + time-bound internal PII mask requests - chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large executions split across requests and never hit the contract's 100k cap - add AbortSignal.timeout(45s) per request so a slow/unreachable app container aborts and the caller scrubs, instead of hanging the trigger.dev job - catch maskPIIBatch failures in the route: log and return a structured 500 (broken venv fails loudly server-side; caller still scrubs, no leak) - add mask-client tests (order across chunks, count split, non-2xx, empty) * fix(guardrails): mint internal token per mask request A single token (5min TTL) could expire mid-batch when a large execution fans out into many sequential chunk requests; mint one per request instead. * feat(guardrails): run PII via Presidio sidecars + TS recognizer registry - replace the per-call python3 subprocess (cold spaCy load every call) with two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP; the app image no longer carries Python/Presidio/venv - add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL - move VIN out of Python into a TS recognizer (check-digit validated) behind a CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is handled uniformly by the anonymizer - drive the guardrails block's PII type picker from the shared pii-entities catalog (adds VIN, fixes drift) so block + Data Retention never diverge - delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step * fix(guardrails): bound-parallelize mask batch; refresh stale comments - maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via mapWithConcurrency, so a chunk of many small leaves finishes within the 45s request timeout instead of aborting and scrubbing; order + fail-on-error kept - drop stale comments referencing the deleted Python venv / 30s subprocess timeout * refactor(guardrails): single Presidio image, native VIN, per-rule redaction language - collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image serves /analyze + /anonymize) - remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native + multi-language in the image; validate_pii is a thin analyze→anonymize client - trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image) - add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum, the Data Retention rule modal, and the guardrails block dropdown; resolver + logger thread it through to maskPIIBatch (default en), so non-English entity rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en * fix(guardrails): correct sidecar port (5001) + README for combined image The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize on a single port 5001 with native VIN + multi-language recognizers. Fix the PRESIDIO_URL default (was 5002) and rewrite the README, which still described two stock containers and a TS VIN recognizer. * fix(guardrails): coerce stored redaction language in the resolver The persist-path resolver accepted any stored language string, so a stale/invalid code (e.g. a dropped locale) would reach Presidio and scrub the log even though the admin UI shows English. Coerce against the supported set via a shared coercePiiLanguage helper (now reused by the data-retention route too), falling back to en for unknown values. * fix(guardrails): rename PRESIDIO_URL env var to PII_URL Match the infra taskdef, which sets PII_URL on the app container for the combined Presidio sidecar.
132 lines
5.2 KiB
Docker
132 lines
5.2 KiB
Docker
# ========================================
|
|
# Base Stage: Debian-based Bun with Node.js 22
|
|
# ========================================
|
|
FROM oven/bun:1.3.13-slim AS base
|
|
|
|
# Install Node.js 22 and common dependencies once in base stage
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
apt-get update && apt-get install -y --no-install-recommends \
|
|
python3 python3-pip python3-venv make g++ curl ca-certificates bash ffmpeg \
|
|
&& curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
|
|
&& apt-get install -y nodejs
|
|
|
|
# ========================================
|
|
# Pruner Stage: Emit a minimal monorepo subset that sim depends on
|
|
# ========================================
|
|
FROM base AS pruner
|
|
WORKDIR /app
|
|
|
|
RUN bun install -g turbo@2.9.6
|
|
|
|
COPY . .
|
|
|
|
RUN turbo prune sim --docker
|
|
|
|
# ========================================
|
|
# Dependencies Stage: Install Dependencies
|
|
# ========================================
|
|
FROM base AS deps
|
|
WORKDIR /app
|
|
|
|
# Pruned manifests from the pruner stage. This layer only invalidates when
|
|
# package.json/bun.lock content changes — not on source edits.
|
|
COPY --from=pruner /app/out/json/ ./
|
|
# Use the full bun.lock (not the pruned out/bun.lock). turbo prune emits a
|
|
# bun.lock that bun 1.3.x rejects with "Failed to resolve prod dependency",
|
|
# forcing a slow fresh resolve. The full lockfile parses cleanly and bun
|
|
# only installs what the pruned package.jsons reference.
|
|
COPY --from=pruner /app/bun.lock ./bun.lock
|
|
|
|
# Install all dependencies (including devDependencies — tailwindcss/postcss are
|
|
# devDeps but required at build time). Then rebuild isolated-vm against Node.js.
|
|
# JOBS=4 caps node-gyp parallelism — higher values OOM isolated-vm (laverdet/isolated-vm#428).
|
|
RUN --mount=type=cache,id=bun-cache,target=/root/.bun/install/cache \
|
|
--mount=type=cache,id=npm-cache,target=/root/.npm \
|
|
HUSKY=0 bun install --ignore-scripts --linker=hoisted && \
|
|
cd node_modules/isolated-vm && JOBS=4 npx node-gyp rebuild --release
|
|
|
|
# ========================================
|
|
# Builder Stage: Build the Application
|
|
# ========================================
|
|
FROM base AS builder
|
|
ARG TARGETPLATFORM
|
|
WORKDIR /app
|
|
|
|
# Copy node_modules from deps stage (cached if dependencies don't change)
|
|
COPY --from=deps /app/node_modules ./node_modules
|
|
|
|
# Copy pruned source tree (apps/sim + workspace packages it depends on)
|
|
COPY --from=pruner /app/out/full/ ./
|
|
|
|
# Next.js 16 / Turbopack workspace-root detection looks for a lockfile next to
|
|
# the workspace package.json. Without it, `next build` fails with
|
|
# "couldn't find next/package.json from /app/apps/sim". turbo also warns
|
|
# "Lockfile not found at /app/bun.lock" without it.
|
|
COPY --from=pruner /app/bun.lock ./bun.lock
|
|
|
|
ENV NEXT_TELEMETRY_DISABLED=1 \
|
|
VERCEL_TELEMETRY_DISABLED=1 \
|
|
DOCKER_BUILD=1
|
|
|
|
# Dummy values so next build can evaluate modules. Override at runtime.
|
|
ARG DATABASE_URL="postgresql://user:pass@localhost:5432/dummy"
|
|
ENV DATABASE_URL=${DATABASE_URL}
|
|
|
|
ARG NEXT_PUBLIC_APP_URL="http://localhost:3000"
|
|
ENV NEXT_PUBLIC_APP_URL=${NEXT_PUBLIC_APP_URL}
|
|
|
|
# Per-platform cache id keeps arm64/amd64 SWC artifacts isolated.
|
|
RUN --mount=type=cache,id=next-cache-${TARGETPLATFORM},target=/app/apps/sim/.next/cache \
|
|
--mount=type=cache,id=turbo-cache-${TARGETPLATFORM},target=/app/.turbo \
|
|
bun run build
|
|
|
|
# ========================================
|
|
# Runner Stage: Run the actual app
|
|
# ========================================
|
|
|
|
FROM base AS runner
|
|
WORKDIR /app
|
|
|
|
# Node.js 22, Python, ffmpeg, etc. are already installed in base stage
|
|
ENV NODE_ENV=production
|
|
|
|
# Create non-root user and group
|
|
RUN groupadd -g 1001 nodejs && \
|
|
useradd -u 1001 -g nodejs nextjs
|
|
|
|
# Copy application artifacts from builder
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/public ./apps/sim/public
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/.next/standalone ./
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/.next/static ./apps/sim/.next/static
|
|
|
|
# Copy blog/author content for runtime filesystem reads (not part of the JS bundle)
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/content ./apps/sim/content
|
|
|
|
# Copy isolated-vm native module (compiled for Node.js in deps stage)
|
|
COPY --from=deps --chown=nextjs:nodejs /app/node_modules/isolated-vm ./node_modules/isolated-vm
|
|
|
|
# Copy the isolated-vm worker script
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/lib/execution/isolated-vm-worker.cjs ./apps/sim/lib/execution/isolated-vm-worker.cjs
|
|
|
|
# Copy the pre-built sandbox library bundles (pptxgenjs, docx, pdf-lib) that
|
|
# run inside the V8 isolate. Committed into the repo; see
|
|
# apps/sim/lib/execution/sandbox/bundles/build.ts to regenerate.
|
|
COPY --from=builder --chown=nextjs:nodejs /app/apps/sim/lib/execution/sandbox/bundles ./apps/sim/lib/execution/sandbox/bundles
|
|
|
|
# Guardrails PII runs in dedicated Presidio sidecar containers (analyzer +
|
|
# anonymizer), reached over localhost — no Python/Presidio in this image.
|
|
|
|
# Create .next/cache directory with correct ownership
|
|
RUN mkdir -p apps/sim/.next/cache && \
|
|
chown -R nextjs:nodejs apps/sim/.next/cache
|
|
|
|
# Switch to non-root user
|
|
USER nextjs
|
|
|
|
EXPOSE 3000
|
|
ENV PORT=3000 \
|
|
HOSTNAME="0.0.0.0"
|
|
|
|
CMD ["bun", "apps/sim/server.js"]
|