feat(pii): build & own combined PII (analyzer + anonymizer) image (#5176)

* feat(presidio): build & own combined analyzer+anonymizer image

Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single
image we build and push to ECR/GHCR. A thin FastAPI service constructs one
AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000
(/health, /supportedentities, /analyze, /anonymize) so the app needs one
PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0.

Bakes in the native check-digit VIN recognizer and registers 12 English
recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*,
SG_*), taking the supported English set from 19 to 32.

* feat(presidio): add multi-language support (es/it/pl/fi)

Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and
explicitly register the national-id recognizers Presidio ships but does not
load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/
IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine +
explicit-registration path detects in-language (Finnish id, score 1.0).

* improvement(presidio): address review feedback

- Register VIN under all served languages, not just en (Bugbot: VIN missed for
  non-English language routing).
- Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot).
- Drop --no-cache-dir so the pip cache mount actually works (Greptile).
- Pydantic request models for /analyze + /anonymize so missing 'text' returns
  422 not 500; default operator 'type' to 'replace' instead of KeyError->500
  (Greptile).

* refactor(pii): rename presidio image artifacts to pii

Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the
service does PII detection + anonymization (and backs the guardrails block's
block/mask), not just redaction, and 'pii' matches existing pii-* naming.

docker/presidio.Dockerfile -> docker/pii.Dockerfile
docker/presidio/ -> docker/pii/
ghcr.io/simstudioai/presidio -> .../pii
ECR_PRESIDIO secret -> ECR_PII (infra side already renamed)
No behavior change — paths/identifiers only.

* refactor(pii): move service to apps/pii, make image ECR-only

- Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs
  under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal
  @sim/pii package.json so the apps/* bun workspace glob accepts the Python service.
- Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user
  presidio -> pii.
- Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published.
  Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64
  tag step to skip GHCR when no ghcr_image is set.
This commit is contained in:
Theodore Li
2026-06-23 03:41:35 -04:00
committed by GitHub
parent ccc6954257
commit 0191a614b6
7 changed files with 296 additions and 5 deletions
+9 -3
View File
@@ -88,6 +88,8 @@ jobs:
ecr_repo_secret: ECR_MIGRATIONS
- dockerfile: ./docker/realtime.Dockerfile
ecr_repo_secret: ECR_REALTIME
- dockerfile: ./docker/pii.Dockerfile
ecr_repo_secret: ECR_PII
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@@ -115,7 +117,7 @@ jobs:
id: ecr-repo
run: echo "name=$ECR_REPO" >> $GITHUB_OUTPUT
env:
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || '' }}
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || matrix.ecr_repo_secret == 'ECR_PII' && secrets.ECR_PII || '' }}
- name: Build and push
uses: useblacksmith/build-push-action@fb9e3e6a9299c78462bfadd0d93352c316adc9b8 # v2
@@ -153,6 +155,10 @@ jobs:
- dockerfile: ./docker/realtime.Dockerfile
ghcr_image: ghcr.io/simstudioai/realtime
ecr_repo_secret: ECR_REALTIME
# pii is ECR-only (private ECS sidecar) — no ghcr_image, so the tag
# step below skips GHCR for it.
- dockerfile: ./docker/pii.Dockerfile
ecr_repo_secret: ECR_PII
steps:
- name: Checkout code
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
@@ -188,7 +194,7 @@ jobs:
id: ecr-repo
run: echo "name=$ECR_REPO" >> $GITHUB_OUTPUT
env:
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || '' }}
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || matrix.ecr_repo_secret == 'ECR_PII' && secrets.ECR_PII || '' }}
- name: Generate tags
id: meta
@@ -206,7 +212,7 @@ jobs:
TAGS="${ECR_IMAGE}"
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
if [ "${{ github.ref }}" = "refs/heads/main" ] && [ -n "$GHCR_IMAGE" ]; then
GHCR_AMD64="${GHCR_IMAGE}:latest-amd64"
GHCR_SHA="${GHCR_IMAGE}:${{ github.sha }}-amd64"
TAGS="${TAGS},$GHCR_AMD64,$GHCR_SHA"
+5 -2
View File
@@ -26,6 +26,9 @@ jobs:
- dockerfile: ./docker/realtime.Dockerfile
ghcr_image: ghcr.io/simstudioai/realtime
ecr_repo_secret: ECR_REALTIME
# pii is ECR-only (private ECS sidecar) — no ghcr_image.
- dockerfile: ./docker/pii.Dockerfile
ecr_repo_secret: ECR_PII
outputs:
registry: ${{ steps.login-ecr.outputs.registry }}
@@ -80,8 +83,8 @@ jobs:
# Build tags list
TAGS="${ECR_IMAGE}"
# Add GHCR tags only for main branch
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
# Add GHCR tags only for main branch (and only for images with a GHCR target)
if [ "${{ github.ref }}" = "refs/heads/main" ] && [ -n "$GHCR_IMAGE" ]; then
GHCR_AMD64="${GHCR_IMAGE}:latest-amd64"
GHCR_SHA="${GHCR_IMAGE}:${{ github.sha }}-amd64"
TAGS="${TAGS},$GHCR_AMD64,$GHCR_SHA"