mirror of
https://github.com/simstudioai/sim.git
synced 2026-08-31 01:11:53 +08:00
Theodore Li
bcad4e308c
* fix(pii): mask offloaded large payloads chunk-by-chunk and retry transient mask failures A block output past the 16MB inline materialization ceiling aborted the run before masking even started: the redaction path hydrated the whole offloaded value at once, and the pre-flight size assert fired on the manifest's total byteSize. Large-array manifests now page one stored chunk at a time (materialize -> mask -> re-store, rebuilt via the manifest writer with preview derived from masked items), so peak heap stays ~one chunk regardless of payload size. Single refs up to the 64MB durable cap hydrate with a raised budget and run serially outside the concurrency pool. Mask-batch chunk requests now retry transient failures (network errors, 408/429/5xx, honoring Retry-After) with jittered backoff, so a single ALB blip or Presidio pod restart no longer fails a whole payload's redaction. Nested-ref masking now runs the string pass before ref substitution, fixing a latent double-mask when a masked nested value shrinks back inline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): retry runtime timeouts and socket closes in mask-batch chunks Verified end-to-end against a 26MB / 40k-record offloaded output: the chunk-wise path masks it in ~54s on a single local Presidio worker where the old path aborted at the 16MB ceiling. The exercise surfaced two more transient error shapes the retry classifier missed — runtime-level request timeouts (undici's default 300s headers timeout, Bun's TimeoutError) and mid-flight socket closes — both of which previously failed the whole payload's redaction on the first occurrence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): gate the spaCy fast path on entities the loaded models can produce The registry's SpacyRecognizers claim every entity in Presidio's default NER-model mapping — including PHONE_NUMBER/AGE/ID/EMAIL, which exist for transformer de-identification backends and which no spaCy model can emit. The NER_ENTITIES derivation trusted that claim, so any request naming PHONE_NUMBER (present in nearly every redaction rule) silently forced the full spaCy pass and the regex-only fast path never fired. Intersect the claimed set with the entities the loaded models' actual NER labels map onto; the hard floor of core NER entities is unchanged, and a future backend that genuinely emits phone labels would re-gate automatically. Verified live: PHONE_NUMBER-only requests take nlp=skip with span parity against the full path, PERSON still forces NER, and a 26MB/40k-record block-output redaction with the realistic entity set runs entirely on the fast path (~3.2min vs ~15min projected full-NER on one worker). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): hydrate oversized-chunk manifests serially, not just single refs Review finding: a manifest whose packer emitted a chunk past the inline ceiling (one item larger than the chunk target) hydrates that chunk with the raised 64MB budget inside the REF_CONCURRENCY pool, so several such manifests could hydrate oversized blobs concurrently — the exact heap scenario the serial path exists to prevent. The serial gate now covers any ref whose hydration can exceed the inline ceiling: oversized single refs and manifests containing an oversized chunk. Normally-chunked manifests stay pooled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): serialize oversized hydrations globally across nested redaction passes Review finding: the serial gate was per-resolveReplacements invocation, so nested oversized refs discovered inside different pooled parents each got their own pool and could hydrate oversized blobs concurrently. A shared promise-chain gate now threads through the options from the entry points, and a reentrancy flag lets a gated ref's own nested oversized work run directly instead of deadlocking on the hold. Covered by a cross-parent max-in-flight assertion and a nested-oversized deadlock regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ * fix(pii): fail fast on a null mask-batch body; use sleep() in gate test A 200 response with a null JSON body threw TypeError on the data.masked read, which the retry classifier treats as transient — burning the full retry budget on a deterministic shape failure. Null-guard the body so it throws the non-retryable shape error immediately. Also swap the gate test's inline setTimeout promise for sleep() to satisfy check:utils, which failed CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A1JYstmLHk9qMGyBDqYRcJ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>