* feat(data-retention): granular PII redaction stages (input + block outputs)
* fix(data-retention): propagate block-output redaction into child workflows
* fix(data-retention): close block-output redaction gaps on streaming + resume
* fix(data-retention): drain+mask streamed output, resolve PII policy unconditionally (no fail-open)
* test(testing): support leftJoin().where().limit() in shared db mock
* fix(data-retention): mask agent/Pi memory writes under block-output redaction
* fix(data-retention): guard partial PII stages in GET normalize
* fix(data-retention): mask seeded memory messages under block-output redaction
* fix(guardrails): fail closed on misaligned Presidio batch responses
* fix(data-retention): enabled stage with no entity types redacts all (no fail-open)
* fix(data-retention): reject enabled stage with no entity types; empty = off everywhere
* docs(data-retention): note resume remask covers inline values only
* fix(data-retention): scrub offloaded large-value refs from logs when block-output redaction is off
* fix(data-retention): hydrate, mask, and re-store large-value refs in logs (preserve redacted content)
* fix(data-retention): always apply logs policy to large-value refs when logs stage is on
* perf(data-retention): drop redaction byte ceiling, parallelize chunks (env-tunable), remove request timeouts, sync large-value walk
* feat(data-retention): gate granular PII stages behind pii-granular-redaction flag
- New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION),
layered on pii-redaction, gating the execution-altering input + block-output stages
- Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off
- UI shows only the Logs stage tab unless the flag is on; clamps active stage
- Drop the per-search Select all toggle; add a Deselect all action to the PII section header
* docs(pii): describe Presidio as a standalone service, not a sidecar
Presidio now runs as its own ECS service (and, in Helm, its own Deployment +
Service) reached over the network via PII_URL — not a sidecar in the app task.
Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to
match, and note the deploy requirement that PII_URL must be reachable.
* fix(data-retention): re-mask offloaded large-value refs on resume + don't lock out granular saves
- Resume/run-from-block restore now hydrates → masks → re-stores large-value refs
in restored blockStates (not just inline strings), so a value offloaded before the
block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast.
- pii-large-values: add onFailure mode (throw on the execution path, scrub for logs)
and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values
- Granular flag gate now rejects only NEW off→on granular enablement, so orgs that
already configured granular stages can still save retention settings when the flag is off
Theodore Li
69b81a679b