domain-privacy-patterns

Files (2)

• SKILL.md
• SKILL.md.lock

SKILL.md (first lines)

<!-- bodyRef: skills/sources/domain-privacy-patterns.md -->

<!-- Generated by `harness skills bootstrap`. Edit to tune; re-run to regenerate. -->
<!-- Static skills point at live docs by design — do not hardwire version-pinned API here. -->

# Privacy domain patterns

Domain patterns + compliance for building privacy-respecting features in this release-to-content engine (TypeScript/Next.js + Python/LangGraph + Aurora/pgvector + Bedrock). The governing rule: **personal data must be minimized and redacted at the boundary, before it is persisted, embedded, logged, or shown to a client.** Generation, vector search, and checkpoints all operate on *redacted* evidence; raw PII never flows downstream by default.

## Pull current docs first

Ground every change in the live docs — APIs and entity lists change:

- Bedrock Guardrails — sensitive information / PII filters: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html
- Bedrock `ApplyGuardrail` (standalone, for non-LLM text) & Converse integration: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html
- Microsoft Presidio (deterministic Python PII detect/anonymize): https://microsoft.github.io/presidio/ (analyzer + anonymizer + structured)
- LangGraph persistence (what checkpoints store): https://docs.langchain.com/oss/python/langgraph/persistence
- React taint APIs (keep PII server-side): https://react.dev/reference/react/experimental_taintObjectReference
- GDPR Article 5 principles: https://gdpr-info.eu/art-5-gdpr/
- CCPA/CPRA (CA AG): https://oag.ca.gov/privacy/ccpa

## Core concepts

**Redact-before-persist pipeline.** In the release-intelligence graph, `redact_evidence` must run *before* `persist_evidence` / embedding. Raw bundles (diffs, PR bodies, issues) routinely contain emails, names, tokens, secrets. Treat the redaction node as the trust boundary: everything after it (Aurora rows, pgvector embeddings, S3 derived artifacts, generated content) is assumed PII-free.

**Two-layer detection.** Use deterministic detection first, ML second — never one alone:
- *Deterministic* (Python): Presidio `AnalyzerEngine` + `AnonymizerEngine`. Operators: `replace`, `redact`, `mask`, `hash`, `encrypt`. Add custom recognizers/regex for project-specific identifiers (feature-flag owners, internal usernames). Catches secrets/keys reliably.
- *Probabilistic* (Bedrock Guardrails): context-aware PII model with `BLOCK` / `ANONYMIZE` (mask → `{EMAIL}`) / `NONE` actions, plus custom `regexesConfig`. Good for free-text the regex misses; weak on short/contextless strings.

**Guardrails on every model boundary.** Attach a guardrail to Converse calls (extraction, clustering, generation). For redacting *non-LLM* text (raw evidence before it reaches a model or DB), call the standalone `ApplyGuardrail` API — don't wait for an inference call.

**State & checkpoints are durable storage.** A LangGraph checkpointer writes a `StateSnapshot` (`channel_values`) to the backing store (here Aurora/Postgres) at *every super-step*, keyed by `thread_id`, and retains the full thread history. PII placed in graph state is therefore persisted and replayable — a storage-limitation/minimization problem. Keep large raw evidence in S3 and carry only **references + redacted summaries** in state. The same applies to `interrupt()` payloads surfaced to the dashboard at the approval gates.

**Embeddings are personal data.** A pgvector embedding derived from text containing PII is itself personal data and is reversible-ish. Only embed redacted text. Erasure requests must reach embeddings too.

**Keep PII server-side in Next.js.** Never send raw personal data to client components. Select explicit fields server-side; optionally fence whole objects with `experimental_taintObjectReference` and secrets/tokens with `experimental_taintUniqueValue` (defense-in-depth, not a guarantee). Never put PII-bearing or secret values in `NEXT_PUBLIC_*` env vars — they are inlined into the client bundle.

**Compliance posture (GDPR + CCPA/CPRA).** GDPR Art. 5 principles to enforce in code: lawfulness/transparency, purpose limitation, data minimisation, accuracy, storage limitation, integrity/confidentiality, accountability. GDPR is **opt-in** (need consent/legal basis before processing); CCPA/CPRA is **opt-out** (right to opt out of sale/sharing) and adds the right to correct and to limit use of sensitive PI. Build for the stricter (opt-in) where audiences overlap. Support data-subject rights operationally: access, erasure/deletion, correction, portability — and ensure deletion fans out to Aurora rows, checkpoints, pgvector embeddings, S3 artifacts, and any caches.

## Checklist

- [ ] `redact_evidence` runs before any persist/embed/generate step; downstream nodes consume only its output.
- [ ] Deterministic pass (Presidio + custom regex for secrets/internal IDs) runs before the probabilistic pass.
- [ ] A Bedrock Guardrail is attached to every Converse call; `ApplyGuardrail` used for raw-text redaction outside inference.
- [ ] Graph state / `channel_values` carries references + redacted text only — raw PII bundles live in S3, not in checkpoints or interrupt payloads.
- [ ] Only redacted text is embedded into pgvector.
- [ ] No PII in logs (app logs, LangGraph traces, CloudWatch model-invocation logs, guardrail `trace`).
- [ ] Client components receive selected, non-PII fields; no PII/secrets in `NEXT_PUBLIC_*`.
- [ ] Data retention/storage-limitation: TTL or purge path exists for evidence, checkpoints, and media.
- [ ] Erasure path deletes across Aurora + checkpoints + embeddings + S3 + caches.
- [ ] Purpose-limitation documented: evidence collected for release-content generation isn't repurposed silently.
- [ ] Consent/lawful-basis model chosen (opt-in vs opt-out) and matches the most restrictive applicable jurisdiction.

## Gotchas

- **Guardrails Mask mode still returns the original PII in the trace.** The `match` field of `GuardrailPiiEntityFilter` in the Converse `trace` (and in `ApplyGuardrail` output) contains the *unmasked* value by design. If you log or persist the trace, you've re-leaked the PII you just masked. Strip or never store `trace.match`.
- **CloudWatch model-invocation logs store the original, unmodified request regardless of the guardrail.** Masking does not touch them. If invocation logging is on, you must separately apply CloudWatch Logs data-protection policies — the guardrail won't.
- **The Bedrock PII filter only scans text — it ignores `tool_use`/function-call output parameters.** Structured tool outputs (e.g. extractor JSON) can carry PII straight past the guardrail. Redact tool outputs deterministically.
- **Bedrock PII detection is probabilistic and context-dependent.** Single words/short fragments are detected poorly. Don't rely on it as the sole redactor — that's why the deterministic Presidio layer runs first.
- **Custom Guardrails regex does not support lookaround.** Patterns relying on `(?=...)`/`(?<=...)` silently won't match. Rewrite without lookaround or do that match in Presidio.
- **LangGraph checkpoints persist full state at every super-step and keep history per `thread_id`.** Putting raw evidence into state means it's written to Aurora repeatedly and survives in thread history even after a node 'consumes' it — an erasure/minimization landmine. Reference S3, don't inline.
- **pgvector embeddings of PII are personal data and outlive the source row.** Deleting the Aurora evidence row without deleting its embedding leaves recoverable personal data and breaks erasure requests.
- **React taint APIs are experimental and trivially defeated.** Destructuring, cloning, or re-deriving a tainted object yields an untainted copy (`{secret: tainted.secret}` is not protected). Treat as a tripwire, not a control — select fields server-side regardless.
- **`NEXT_PUBLIC_` inlines values into the client bundle at build time.** A PII-bearing or secret value there ships to every browser permanently, even after you remove it (cached bundles).
- **Presidio guarantees nothing.** Automated detection misses entities; combine deterministic + ML + the human approval gate, and never market 'PII-free' as a guarantee.
- **GDPR opt-in vs CCPA opt-out are opposite defaults.** Assuming opt-out (collect until told to stop) for EU data subjects is a lawful-basis violation. Pick the consent model per audience, defaulting to opt-in when jurisdictions overlap.

<!-- sources fetched at generation: https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html, https://microsoft.github.io/presidio/, https://gdpr-info.eu/art-5-gdpr/, https://react.dev/reference/react/experimental_taintObjectReference, https://docs.langchain.com/oss/python/langgraph/persistence, https://oag.ca.gov/privacy/ccpa -->

From ichiorca/shipsignal, Apache-2.0.

More from ichiorca/shipsignal