mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
v0.8.14
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Theodore Li
|
bcad4e308c |
fix(pii): mask offloaded large payloads chunk-by-chunk instead of aborting at 16MB (#5810)
* fix(pii): mask offloaded large payloads chunk-by-chunk and retry transient mask failures A block output past the 16MB inline materialization ceiling aborted the run before masking even started: the redaction path hydrated the whole offloaded value at once, and the pre-flight size assert fired on the manifest's total byteSize. Large-array manifests now page one stored chunk at a time (materialize -> mask -> re-store, rebuilt via the manifest writer with preview derived from masked items), so peak heap stays ~one chunk regardless of payload size. Single refs up to the 64MB durable cap hydrate with a raised budget and run serially outside the concurrency pool. Mask-batch chunk requests now retry transient failures (network errors, 408/429/5xx, honoring Retry-After) with jittered backoff, so a single ALB blip or Presidio pod restart no longer fails a whole payload's redaction. Nested-ref masking now runs the string pass before ref substitution, fixing a latent double-mask when a masked nested value shrinks back inline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): retry runtime timeouts and socket closes in mask-batch chunks Verified end-to-end against a 26MB / 40k-record offloaded output: the chunk-wise path masks it in ~54s on a single local Presidio worker where the old path aborted at the 16MB ceiling. The exercise surfaced two more transient error shapes the retry classifier missed — runtime-level request timeouts (undici's default 300s headers timeout, Bun's TimeoutError) and mid-flight socket closes — both of which previously failed the whole payload's redaction on the first occurrence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): gate the spaCy fast path on entities the loaded models can produce The registry's SpacyRecognizers claim every entity in Presidio's default NER-model mapping — including PHONE_NUMBER/AGE/ID/EMAIL, which exist for transformer de-identification backends and which no spaCy model can emit. The NER_ENTITIES derivation trusted that claim, so any request naming PHONE_NUMBER (present in nearly every redaction rule) silently forced the full spaCy pass and the regex-only fast path never fired. Intersect the claimed set with the entities the loaded models' actual NER labels map onto; the hard floor of core NER entities is unchanged, and a future backend that genuinely emits phone labels would re-gate automatically. Verified live: PHONE_NUMBER-only requests take nlp=skip with span parity against the full path, PERSON still forces NER, and a 26MB/40k-record block-output redaction with the realistic entity set runs entirely on the fast path (~3.2min vs ~15min projected full-NER on one worker). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): hydrate oversized-chunk manifests serially, not just single refs Review finding: a manifest whose packer emitted a chunk past the inline ceiling (one item larger than the chunk target) hydrates that chunk with the raised 64MB budget inside the REF_CONCURRENCY pool, so several such manifests could hydrate oversized blobs concurrently — the exact heap scenario the serial path exists to prevent. The serial gate now covers any ref whose hydration can exceed the inline ceiling: oversized single refs and manifests containing an oversized chunk. Normally-chunked manifests stay pooled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): serialize oversized hydrations globally across nested redaction passes Review finding: the serial gate was per-resolveReplacements invocation, so nested oversized refs discovered inside different pooled parents each got their own pool and could hydrate oversized blobs concurrently. A shared promise-chain gate now threads through the options from the entry points, and a reentrancy flag lets a gated ref's own nested oversized work run directly instead of deadlocking on the hold. Covered by a cross-parent max-in-flight assertion and a nested-oversized deadlock regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): fail fast on a null mask-batch body; use sleep() in gate test A 200 response with a null JSON body threw TypeError on the data.masked read, which the retry classifier treats as transient — burning the full retry budget on a deterministic shape failure. Null-guard the body so it throws the non-retryable shape error immediately. Also swap the gate test's inline setTimeout promise for sleep() to satisfy check:utils, which failed CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Theodore Li
|
61de0b5808 |
feat(pii): custom user-supplied regex patterns for redaction (#5732)
* feat(pii): custom user-supplied regex patterns for redaction * fix(pii): enforce custom-regex syntax + safety at the boundary schema * improvement(pii): always wrap custom-pattern redaction token in angle brackets * chore(pii): register guardrails_validate in the dev minimal tool registry * fix(pii): coerce empty guardrails entity-type checkbox (null) so the contract accepts it * fix(pii): keep detect-all when a custom pattern is added; custom patterns win overlaps |
||
|
Theodore Li
|
4d6301c900 |
feat(pii): regex-only block-output redaction + drop GLiNER/GPU image (#5697)
* chore(pii): remove GLiNER/GPU image + add spaCy-skip fast path to CPU server * feat(pii): restrict block-output redaction to regex-only entities * fix(pii): derive spaCy-NER set from registry + skip fast path when score_threshold set * fix(pii): include ORGANIZATION in app-side NER set (align with server) |
||
|
Waleed
|
f9a5d8b113 |
fix(security): close code-scanning and dependabot alerts (#5557)
* fix(security): close code-scanning and dependabot alerts - markdown-paste.ts: strip <style>/<script> in a loop, not a single pass, so nested/overlapping tags can't leave a surviving <script> behind (incomplete multi-character sanitization) - block-identity.ts: annotate the two SHA-1 uses as intentional (UUIDv5 per RFC 4122, deterministic id derivation only, not a security use of the hash) rather than swap algorithms, which would change every derived fork block id - apps/pii: bump transformers 4.56.2 -> 5.3.0 (CVE-2026-4372 RCE via crafted config.json, CVE-2026-1839 RCE via Trainer torch.load), huggingface_hub 0.35.3 -> 1.3.0 (transformers 5.3.0's floor), and pytest 8.4.1 -> 9.0.3 (CVE-2025-71176 tmpdir handling); verified pip resolves cleanly and the unit test suite passes on 9.0.3 * fix(files): make markdown-paste sanitizer O(n) instead of O(n*depth) Greptile flagged the repeated-replace loop from the prior commit: it strips <style>/<script> correctly but rescans the whole string once per nesting level, so deeply nested clipboard HTML can freeze the tab. Replace it with a single linear pass that tracks nesting depth of the open tag via a tag-token scan, dropping the element in one pass no matter how deeply nested. * style: fold inline comments into TSDoc per repo comment convention Repo convention is TSDoc-only documentation, no non-TSDoc explanatory comments. Moved the uuidV5 SHA-1 rationale and the stray-close-tag note into the existing TSDoc blocks above each function. Left the two lgtm[...] annotations as trailing comments since those are functional CodeQL suppression directives (must sit on the flagged line), not documentation. * test(files): lock in nested-tag stripping regression for markdown paste Covers the case Greptile flagged: nested and 50-deep <script> tags must strip in one pass without leaking a dangling tag. * fix(files): drop unterminated <script>/<style> instead of leaking it Cursor Bugbot caught two related bugs in the single-pass rewrite: if pasted HTML ends while a script/style element is still open (truncated or malformed clipboard HTML), cursor never advanced past the open tag, so the final flush re-appended the untouched tag/content (leaking an unstripped <script>) and duplicated the prefix already copied into result. Fix: advance cursor the moment a tag opens, not when it closes, and only do the final flush when we end at depth 0. An element that never closes has cursor already past its open tag, so it and everything after it is dropped instead of reappearing. |
||
|
Theodore Li
|
4e6594dc54 |
feat(pii): add opt-in GLiNER NER engine (PII_ENGINE), device-agnostic (#5495)
* feat(pii): add opt-in GLiNER NER engine (PII_ENGINE), device-agnostic Swap the 4 NER entity types (PERSON/LOCATION/NRP/DATE_TIME) to a single multilingual GLiNER zero-shot model when PII_ENGINE=gliner; spaCy stays the default and all ~36 regex/checksum recognizers are identical on both engines. Device-agnostic via PII_DEVICE / cuda auto-detect — same code on Fargate CPU now and EC2-GPU later. - engines.py: side-effect-free builders; SharedModelGLiNERRecognizer loads ONE model shared across the 5 per-language instances and restricts labels to the entities it owns; small spaCy models keep tokenization/lemmas for the regex recognizers; fail-fast on the lean image - pii.Dockerfile: multi-stage — default target unchanged (lean spaCy); --target gliner is a superset (torch CPU + gliner + baked model) where both engines work; gliner-gpu scaffold for the GPU fleet - CI publishes the gliner variant (:staging-gliner/:latest-gliner, amd64) - Helm: pii.engine / pii.device values wired to PII_ENGINE/PII_DEVICE - scripts/bench_engines.py: throughput + NER-parity diff harness - tests: unit (mocked GLiNER) + in-image integration for both engines Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up3F97mjCH9HCj1pX4J8VJ * refactor(pii): ship both engines in one image — engine is a pure env flip Collapse the gliner build target into the single pii image: spaCy lg models, torch (CPU), gliner, and the baked GLiNER weights all ship in it, so PII_ENGINE switches engines with no image swap and no tag matrix. CI reverts to the single pii build (no -gliner tags). The GPU variant becomes the same Dockerfile built with --build-arg TORCH_INDEX_URL=.../cu128. Image grows ~6.1GB -> ~9.6GB. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Up3F97mjCH9HCj1pX4J8VJ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Theodore Li
|
34b2abadcb |
perf(pii): mask offloaded refs + parallelize + raise redaction concurrency (#5436)
* perf(pii): mask offloaded refs in block outputs/input + parallelize + raise concurrency - Block-output and input stages now hydrate → mask → re-store large-value refs (function/tool outputs are offloaded to refs before reaching the redactor, which treats refs as opaque) — fixes big block outputs coming back unredacted - Parallelize large-value ref hydrate/mask/re-store: collect refs across the whole payload and process them with bounded concurrency instead of one sequential pass per key (PII_REF_CONCURRENCY, default 4) - Raise mask-batch chunk concurrency default 4 -> 64 to saturate the load-balanced Presidio fleet behind the internal ALB (PII_MASK_CHUNK_CONCURRENCY, env-tunable) * fix(pii): keep resolveReplacements mapper total (respect mapWithConcurrency contract) - Catch per-ref errors inside the mapWithConcurrency mapper and rethrow the first after the pool drains, instead of letting a throwing mapper reject the pool (the util documents fn MUST NOT reject). Preserves fail-fast abort in throw mode. - Add multi-ref throw-mode test: one of several refs failing aborts the redaction * feat(pii): make route->Presidio chunk concurrency env-tunable (PII_SERVICE_CHUNK_CONCURRENCY) Was hardcoded to 4; now env-tunable (default 4) so the inner route->Presidio fan-out can scale with the fleet alongside the outer PII_MASK_CHUNK_CONCURRENCY. * feat(pii): combined /redact + /redact_batch endpoint (one round-trip) - Presidio: add /redact and /redact_batch (analyze + anonymize server-side, feeding analyzer results straight to the anonymizer, no dict round-trip). All existing endpoints kept, so old clients keep working during rollout. - App: maskPIIBatch now uses /redact_batch (halves app<->service round-trips, no shipping text back up for anonymize). Falls back to the legacy analyze_batch + anonymize_batch pair on a 404 (older Presidio image), so the app is safe to deploy before or after the service in either order. Same length-check fail-closed guarantee. |
||
|
Theodore Li
|
69b81a679b |
feat(data-retention): granular PII redaction stages (input + block outputs) (#5272)
* feat(data-retention): granular PII redaction stages (input + block outputs) * fix(data-retention): propagate block-output redaction into child workflows * fix(data-retention): close block-output redaction gaps on streaming + resume * fix(data-retention): drain+mask streamed output, resolve PII policy unconditionally (no fail-open) * test(testing): support leftJoin().where().limit() in shared db mock * fix(data-retention): mask agent/Pi memory writes under block-output redaction * fix(data-retention): guard partial PII stages in GET normalize * fix(data-retention): mask seeded memory messages under block-output redaction * fix(guardrails): fail closed on misaligned Presidio batch responses * fix(data-retention): enabled stage with no entity types redacts all (no fail-open) * fix(data-retention): reject enabled stage with no entity types; empty = off everywhere * docs(data-retention): note resume remask covers inline values only * fix(data-retention): scrub offloaded large-value refs from logs when block-output redaction is off * fix(data-retention): hydrate, mask, and re-store large-value refs in logs (preserve redacted content) * fix(data-retention): always apply logs policy to large-value refs when logs stage is on * perf(data-retention): drop redaction byte ceiling, parallelize chunks (env-tunable), remove request timeouts, sync large-value walk * feat(data-retention): gate granular PII stages behind pii-granular-redaction flag - New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION), layered on pii-redaction, gating the execution-altering input + block-output stages - Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off - UI shows only the Logs stage tab unless the flag is on; clamps active stage - Drop the per-search Select all toggle; add a Deselect all action to the PII section header * docs(pii): describe Presidio as a standalone service, not a sidecar Presidio now runs as its own ECS service (and, in Helm, its own Deployment + Service) reached over the network via PII_URL — not a sidecar in the app task. Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to match, and note the deploy requirement that PII_URL must be reachable. * fix(data-retention): re-mask offloaded large-value refs on resume + don't lock out granular saves - Resume/run-from-block restore now hydrates → masks → re-stores large-value refs in restored blockStates (not just inline strings), so a value offloaded before the block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast. - pii-large-values: add onFailure mode (throw on the execution path, scrub for logs) and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values - Granular flag gate now rejects only NEW off→on granular enablement, so orgs that already configured granular stages can still save retention settings when the flag is off |
||
|
Theodore Li
|
23ec96b02a |
feat(pii): add redaction timing metrics across sidecar and persist path (#5264)
- Log per-request duration in the Presidio sidecar (/analyze, /anonymize) - Add durationMs to the mask-batch endpoint log line - Emit per-execution PII redaction timing (stringCount, totalBytes, durationMs, scrubbed) |
||
|
Theodore Li
|
0191a614b6 |
feat(pii): build & own combined PII (analyzer + anonymizer) image (#5176)
* feat(presidio): build & own combined analyzer+anonymizer image Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single image we build and push to ECR/GHCR. A thin FastAPI service constructs one AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000 (/health, /supportedentities, /analyze, /anonymize) so the app needs one PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0. Bakes in the native check-digit VIN recognizer and registers 12 English recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*, SG_*), taking the supported English set from 19 to 32. * feat(presidio): add multi-language support (es/it/pl/fi) Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and explicitly register the national-id recognizers Presidio ships but does not load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/ IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine + explicit-registration path detects in-language (Finnish id, score 1.0). * improvement(presidio): address review feedback - Register VIN under all served languages, not just en (Bugbot: VIN missed for non-English language routing). - Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot). - Drop --no-cache-dir so the pip cache mount actually works (Greptile). - Pydantic request models for /analyze + /anonymize so missing 'text' returns 422 not 500; default operator 'type' to 'replace' instead of KeyError->500 (Greptile). * refactor(pii): rename presidio image artifacts to pii Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the service does PII detection + anonymization (and backs the guardrails block's block/mask), not just redaction, and 'pii' matches existing pii-* naming. docker/presidio.Dockerfile -> docker/pii.Dockerfile docker/presidio/ -> docker/pii/ ghcr.io/simstudioai/presidio -> .../pii ECR_PRESIDIO secret -> ECR_PII (infra side already renamed) No behavior change — paths/identifiers only. * refactor(pii): move service to apps/pii, make image ECR-only - Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal @sim/pii package.json so the apps/* bun workspace glob accepts the Python service. - Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user presidio -> pii. - Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published. Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64 tag step to skip GHCR when no ghcr_image is set. |