mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
Theodore Li
4e6594dc54
* feat(pii): add opt-in GLiNER NER engine (PII_ENGINE), device-agnostic Swap the 4 NER entity types (PERSON/LOCATION/NRP/DATE_TIME) to a single multilingual GLiNER zero-shot model when PII_ENGINE=gliner; spaCy stays the default and all ~36 regex/checksum recognizers are identical on both engines. Device-agnostic via PII_DEVICE / cuda auto-detect — same code on Fargate CPU now and EC2-GPU later. - engines.py: side-effect-free builders; SharedModelGLiNERRecognizer loads ONE model shared across the 5 per-language instances and restricts labels to the entities it owns; small spaCy models keep tokenization/lemmas for the regex recognizers; fail-fast on the lean image - pii.Dockerfile: multi-stage — default target unchanged (lean spaCy); --target gliner is a superset (torch CPU + gliner + baked model) where both engines work; gliner-gpu scaffold for the GPU fleet - CI publishes the gliner variant (:staging-gliner/:latest-gliner, amd64) - Helm: pii.engine / pii.device values wired to PII_ENGINE/PII_DEVICE - scripts/bench_engines.py: throughput + NER-parity diff harness - tests: unit (mocked GLiNER) + in-image integration for both engines Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up3F97mjCH9HCj1pX4J8VJ * refactor(pii): ship both engines in one image — engine is a pure env flip Collapse the gliner build target into the single pii image: spaCy lg models, torch (CPU), gliner, and the baked GLiNER weights all ship in it, so PII_ENGINE switches engines with no image swap and no tag matrix. CI reverts to the single pii build (no -gliner tags). The GPU variant becomes the same Dockerfile built with --build-arg TORCH_INDEX_URL=.../cu128. Image grows ~6.1GB -> ~9.6GB. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up3F97mjCH9HCj1pX4J8VJ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>