mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
feat(pii): build & own combined PII (analyzer + anonymizer) image (#5176)
* feat(presidio): build & own combined analyzer+anonymizer image Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single image we build and push to ECR/GHCR. A thin FastAPI service constructs one AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000 (/health, /supportedentities, /analyze, /anonymize) so the app needs one PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0. Bakes in the native check-digit VIN recognizer and registers 12 English recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*, SG_*), taking the supported English set from 19 to 32. * feat(presidio): add multi-language support (es/it/pl/fi) Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and explicitly register the national-id recognizers Presidio ships but does not load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/ IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine + explicit-registration path detects in-language (Finnish id, score 1.0). * improvement(presidio): address review feedback - Register VIN under all served languages, not just en (Bugbot: VIN missed for non-English language routing). - Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot). - Drop --no-cache-dir so the pip cache mount actually works (Greptile). - Pydantic request models for /analyze + /anonymize so missing 'text' returns 422 not 500; default operator 'type' to 'replace' instead of KeyError->500 (Greptile). * refactor(pii): rename presidio image artifacts to pii Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the service does PII detection + anonymization (and backs the guardrails block's block/mask), not just redaction, and 'pii' matches existing pii-* naming. docker/presidio.Dockerfile -> docker/pii.Dockerfile docker/presidio/ -> docker/pii/ ghcr.io/simstudioai/presidio -> .../pii ECR_PRESIDIO secret -> ECR_PII (infra side already renamed) No behavior change — paths/identifiers only. * refactor(pii): move service to apps/pii, make image ECR-only - Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal @sim/pii package.json so the apps/* bun workspace glob accepts the Python service. - Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user presidio -> pii. - Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published. Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64 tag step to skip GHCR when no ghcr_image is set.
This commit is contained in:
@@ -88,6 +88,8 @@ jobs:
|
||||
ecr_repo_secret: ECR_MIGRATIONS
|
||||
- dockerfile: ./docker/realtime.Dockerfile
|
||||
ecr_repo_secret: ECR_REALTIME
|
||||
- dockerfile: ./docker/pii.Dockerfile
|
||||
ecr_repo_secret: ECR_PII
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
@@ -115,7 +117,7 @@ jobs:
|
||||
id: ecr-repo
|
||||
run: echo "name=$ECR_REPO" >> $GITHUB_OUTPUT
|
||||
env:
|
||||
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || '' }}
|
||||
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || matrix.ecr_repo_secret == 'ECR_PII' && secrets.ECR_PII || '' }}
|
||||
|
||||
- name: Build and push
|
||||
uses: useblacksmith/build-push-action@fb9e3e6a9299c78462bfadd0d93352c316adc9b8 # v2
|
||||
@@ -153,6 +155,10 @@ jobs:
|
||||
- dockerfile: ./docker/realtime.Dockerfile
|
||||
ghcr_image: ghcr.io/simstudioai/realtime
|
||||
ecr_repo_secret: ECR_REALTIME
|
||||
# pii is ECR-only (private ECS sidecar) — no ghcr_image, so the tag
|
||||
# step below skips GHCR for it.
|
||||
- dockerfile: ./docker/pii.Dockerfile
|
||||
ecr_repo_secret: ECR_PII
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
||||
@@ -188,7 +194,7 @@ jobs:
|
||||
id: ecr-repo
|
||||
run: echo "name=$ECR_REPO" >> $GITHUB_OUTPUT
|
||||
env:
|
||||
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || '' }}
|
||||
ECR_REPO: ${{ matrix.ecr_repo_secret == 'ECR_APP' && secrets.ECR_APP || matrix.ecr_repo_secret == 'ECR_MIGRATIONS' && secrets.ECR_MIGRATIONS || matrix.ecr_repo_secret == 'ECR_REALTIME' && secrets.ECR_REALTIME || matrix.ecr_repo_secret == 'ECR_PII' && secrets.ECR_PII || '' }}
|
||||
|
||||
- name: Generate tags
|
||||
id: meta
|
||||
@@ -206,7 +212,7 @@ jobs:
|
||||
|
||||
TAGS="${ECR_IMAGE}"
|
||||
|
||||
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
|
||||
if [ "${{ github.ref }}" = "refs/heads/main" ] && [ -n "$GHCR_IMAGE" ]; then
|
||||
GHCR_AMD64="${GHCR_IMAGE}:latest-amd64"
|
||||
GHCR_SHA="${GHCR_IMAGE}:${{ github.sha }}-amd64"
|
||||
TAGS="${TAGS},$GHCR_AMD64,$GHCR_SHA"
|
||||
|
||||
@@ -26,6 +26,9 @@ jobs:
|
||||
- dockerfile: ./docker/realtime.Dockerfile
|
||||
ghcr_image: ghcr.io/simstudioai/realtime
|
||||
ecr_repo_secret: ECR_REALTIME
|
||||
# pii is ECR-only (private ECS sidecar) — no ghcr_image.
|
||||
- dockerfile: ./docker/pii.Dockerfile
|
||||
ecr_repo_secret: ECR_PII
|
||||
outputs:
|
||||
registry: ${{ steps.login-ecr.outputs.registry }}
|
||||
|
||||
@@ -80,8 +83,8 @@ jobs:
|
||||
# Build tags list
|
||||
TAGS="${ECR_IMAGE}"
|
||||
|
||||
# Add GHCR tags only for main branch
|
||||
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
|
||||
# Add GHCR tags only for main branch (and only for images with a GHCR target)
|
||||
if [ "${{ github.ref }}" = "refs/heads/main" ] && [ -n "$GHCR_IMAGE" ]; then
|
||||
GHCR_AMD64="${GHCR_IMAGE}:latest-amd64"
|
||||
GHCR_SHA="${GHCR_IMAGE}:${{ github.sha }}-amd64"
|
||||
TAGS="${TAGS},$GHCR_AMD64,$GHCR_SHA"
|
||||
|
||||
Reference in New Issue
Block a user