mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
Theodore Li
34b2abadcb
* perf(pii): mask offloaded refs in block outputs/input + parallelize + raise concurrency - Block-output and input stages now hydrate → mask → re-store large-value refs (function/tool outputs are offloaded to refs before reaching the redactor, which treats refs as opaque) — fixes big block outputs coming back unredacted - Parallelize large-value ref hydrate/mask/re-store: collect refs across the whole payload and process them with bounded concurrency instead of one sequential pass per key (PII_REF_CONCURRENCY, default 4) - Raise mask-batch chunk concurrency default 4 -> 64 to saturate the load-balanced Presidio fleet behind the internal ALB (PII_MASK_CHUNK_CONCURRENCY, env-tunable) * fix(pii): keep resolveReplacements mapper total (respect mapWithConcurrency contract) - Catch per-ref errors inside the mapWithConcurrency mapper and rethrow the first after the pool drains, instead of letting a throwing mapper reject the pool (the util documents fn MUST NOT reject). Preserves fail-fast abort in throw mode. - Add multi-ref throw-mode test: one of several refs failing aborts the redaction * feat(pii): make route->Presidio chunk concurrency env-tunable (PII_SERVICE_CHUNK_CONCURRENCY) Was hardcoded to 4; now env-tunable (default 4) so the inner route->Presidio fan-out can scale with the fleet alongside the outer PII_MASK_CHUNK_CONCURRENCY. * feat(pii): combined /redact + /redact_batch endpoint (one round-trip) - Presidio: add /redact and /redact_batch (analyze + anonymize server-side, feeding analyzer results straight to the anonymizer, no dict round-trip). All existing endpoints kept, so old clients keep working during rollout. - App: maskPIIBatch now uses /redact_batch (halves app<->service round-trips, no shipping text back up for anonymize). Falls back to the legacy analyze_batch + anonymize_batch pair on a 404 (older Presidio image), so the app is safe to deploy before or after the service in either order. Same length-check fail-closed guarantee.