data-identity-resolution

Files (3)

• SKILL.md
• attach-workflow.md · references/
• validate_skill.py · scripts/

SKILL.md (first lines)

# Data identity resolution — attach, don't guess

`data-semantic-quality` owns the **rules**: trust ladder, confidence as a gate,
cohort fences. This skill owns the **procedure** for linking a messy incoming
record to a canonical entity key without inventing a second identity system.

No product match thresholds, no catalog taxonomies. Those stay in the product repo.

## When to invoke

- Parsing free text / third-party ids onto a canonical entity key.
- NULL or unresolved identity keys growing day over day.
- Remapping / restamping historical links after a resolver change.
- Strong structured evidence disagrees with a weak title/heuristic.

## Don't use

| Task | Go here instead |
|---|---|
| Quality flags, outliers, golden *entity* packs | **data-semantic-quality** |
| Scoring published estimates vs later facts | **data-product-eval** |
| Schema, watermarks, publish/OCC | **data-apache-lakehouse** / **data** hub |
| Serving honors stored keys; does not re-resolve | **data-api** |
| “Should this table exist?” | **data-table-lifecycle** |
| Concrete matcher constants / synonym lists | **product-repo skill** |

## Non-negotiables

1. **No domain product nouns or concrete match thresholds** in advice from this skill.
2. **Ladder rules live in `data-semantic-quality`.** Do not fork a second ladder here.
3. **Unresolved is a first-class state**, not a blank to be filled by the next weak hint.
4. **Collision / twin is a first-class state**, not another matcher branch.
5. **Remap cannot undo a stronger fence.** Restamp is ordered and is not the incremental timer.
6. **Low confidence never enters identity-assuming aggregates.**

## Principles

Each has a falsifiable **Test:**. Ladder mechanics live in **data-semantic-quality**
(trust ladder + confidence gates). Workflow detail:
[`references/attach-workflow.md`](references/attach-workflow.md).

### 1. Evidence before assignment

Collect ordered evidence (strong structured → catalog default → weak text) *then*
decide assign / quarantine / leave unresolved. Do not assign inside the first
matching `if`.

**Test:** invent a row where weak text and a strong structured field disagree. Does the weak path still win if you reorder branches?

### 2. Fail closed

Below the product’s confidence threshold the key stays null/unresolved and the
row is excluded from identity-assuming rollups. “Average it in at a discount”
is not fail-closed.

**Test:** can a below-threshold row appear in the primary aggregate without an explicit override flag?

### 3. Conflict quarantines

Strong-vs-weak mismatch is a quarantine (or a conflict label), not a coin flip.
Record which evidence class produced the decision.

**Test:** is there a durable conflict/quarantine state a later job can count, or does the row silently pick a side?

### 4. Collision is a state, not a second pass of identify()

Two assigned keys for one physical event (or one key stamped on two events that must stay
distinct) is **collision**. Persist it. Count it. An ordered restamp may pick a survivor; the
incremental matcher must not "fold" the twin away to make the rollup look unique.

**Test:** inject a duplicate physical event with two structured ids. Does a collision row (or pair)
survive publish, or does the next identify() pick a winner with no audit?

### 5. Unresolved accumulation is a metric

NULL / unresolved keys are measured debt: count per source per day, with a
rate (unresolved / landed). A resolver that “works” while the null pile grows
is losing.

**Test:** can you plot unresolved-landed per source for the last N days from published tables, without reading code? If the rate is up and no one is paging, the metric is missing.

### 6. Remap is ordered and one-way through stronger fences

A restamp / re-resolve pass applies weaker evidence only where stronger evidence
is absent. It must not clear a strong assignment or a quarantine to please a
new heuristic.

**Test:** run restamp on a fixture that has a strong key *and* a tempting weak hint the other way. Does the strong key survive?

Restamp is a **separate resumable lane** from incremental attach (principle 7). It does not ride the
hourly identify timer. Residual rows after a restamp are a keyset for the next catchup start, not a
reason to widen the incremental window to lifetime.

### 7. Backfill is a bounded, resumable lane

Unresolved backfill is not the incremental attach path. It has its own watermark
or keyset, chunk size, and idempotent write. Hub principle 2 (watermark, not
full recompute) applies — a crash must not restart the whole historical pile.

**Test:** kill a backfill at 40%. Does retry resume, or re-scan from record one and rewrite already-good keys?

### 8. One writer for the link

The attach job (or documented reconcile) is the only writer of the canonical
key + confidence + evidence class. APIs, UIs, and rollups consume. A serving
path that “fixes” identity at request time is a second resolver.

**Test:** change the attach threshold in one place. Do all consumers move on the next publish, or does one surface still re-parse text?

### 9. Coverage audit names identity rate

When someone asks “how much data do we have,” unresolved rate is part of the
answer — row count alone lies. The **data** hub coverage-audit playbook includes
this slice; this skill defines what “resolved” means (key present *and* above
gate).

**Test:** a source with 1M rows and 85% null keys — does the audit report “1M rows” or “150k resolved”?

## Workflow: attach or remap

From Evan-Kim2028/agent-skills, MIT.

More from Evan-Kim2028/agent-skills