data-product-eval

Files (3)

• SKILL.md
• eval-loop.md · references/
• validate_skill.py · scripts/

SKILL.md (first lines)

# Data product eval — estimates vs later truth

`data-semantic-quality` asks “is this *row* the right entity with the right flags?”
This skill asks “is this *published number* honest against what later happened?”
Code tests (`quality-check` / `tdd`) do not answer that.

No product thresholds, no marketplace rule books. Those stay in the product repo.

## When to invoke

- Scoring a published estimate / mark / forecast / fill against later realized facts.
- Preparing a data-product surface for release (coverage vs error, honesty labels).
- Adding a canary, shadow pack, or truth panel that must share policy with prod.
- A stored-vs-recomputed reconciliation that is RED, or an issue closed but the live error/coverage metric did not move.

## Don't use

| Task | Go here instead |
|---|---|
| Quality flags, trust ladders, entity golden packs | **data-semantic-quality** |
| Attaching messy records to entity keys | **data-identity-resolution** |
| Schema, watermarks, Iceberg publish | **data-apache-lakehouse** / **data** hub |
| Serving honors attributes; keyset/sidecars | **data-api** |
| Code TDD, browser proof, merge checklist | **quality-check** |
| Domain constants (bands, floors, venue weights) | **product-repo skill** |

## Non-negotiables

1. **No domain product nouns or concrete business thresholds** in advice from this skill.
2. **Eval and production run the same policy function.** A second scorer is a second truth.
3. **Snapshots are frozen.** Scoring a moving publish against a moving fact table is not an eval.
4. **Coverage and error are different gates.** Raising coverage by filling junk is a fail.
5. **Observe-only is allowed; calling it a release gate is not.**

## Principles

Each has a falsifiable **Test:**.

### 1. One policy path

The function that decides “eligible / excluded / estimate / thin” in production is the
function the eval imports. Shadow copies drift within a week.

**Test:** change one eligibility rule. Does tonight’s eval and tonight’s publish move together, or only one?

### 2. Freeze both sides of the comparison

Pin the estimate snapshot *and* the realized-fact snapshot (or a temporal as-of). Do not
score “whatever is live now” against “whatever sold since.”

**Test:** re-run yesterday’s eval job. Are the inputs byte-identical to yesterday’s freeze? If numbers wiggle with no code change, the freeze is missing.

### 3. Separate fit from forecast

- **Fit:** estimate published at T vs facts already known at T (did we describe the past).
- **Forecast:** estimate at T vs facts in (T, T+k] (did we predict).

Mixing them hides stale-but-pretty marks.

**Test:** can you name the fit window and the forecast window, and are they computed as two series?

### 4. Stratify before you quote a single error

Global MAPE/median-abs-log is a vanity number. Slice by the dimensions that change
the distribution (status, cohort, volume band, age, venue, language — whatever the
product actually serves). Details: [`references/eval-loop.md`](references/eval-loop.md).

**Test:** can a thin, stale, or wide slice look “fine” inside the global number? If yes, the report is not stratified.

### 5. Coverage floor ≠ accuracy gate

A coverage gate asks “what fraction of served slots have a number.” An accuracy gate
asks “of slots that have a number, how wrong is it.” Filling empty slots with
unlabeled guesses raises coverage and poisons accuracy.

**Test:** if you force 100% coverage with a naive fill, does the accuracy gate go red? If both stay green, you have one gate pretending to be two.

### 6. Dual budgets on *values*, not only flags

Entity packs (`data-semantic-quality`) guard known-good / known-bad *rows*.
Estimate eval also needs known-good / known-bad *values* (this slot should stay
near X; this slot must not print a number). Dual budgets: false-negative on
known-bad, false-positive on known-good.

**Test:** delete the known-bad value cases. Does the release path still go green?

### 7. Honesty labels travel with the number

A served estimate that can be thin, stale, wide, or ambiguous must carry that
status into eval. Scoring only the happy `ok` slice hides the product.

**Test:** turn off status filtering in the report. Do thin/stale/wide slots still appear, labeled, or did they vanish from the denominator?

### 8. Observe-only vs release gate is explicit

Nightly accuracy can be report-only. A release / promote / “ready for customers”
path names the gate (coverage floor, pack budgets, calibration band) and fails
closed. Do not say “we eval” when you only write a parquet nobody reads.

**Test:** break a core-tier pack case. Does anything block promote/release, or only a log line?

### 9. Merge is not the metric

Closing a ticket or merging a policy PR is not evidence the live error or
coverage number moved. **quality-check** owns the closeout ritual; this skill
owns *which* number to measure.

**Test:** after “done,” can you show before/after on the named eval metric from the frozen path? If you only have CI green, you did not eval.

### 10. Stored-vs-recomputed recon is the same policy, windowed

A job that diffs the published table against a recompute must **import the production policy
module** (principle 1) and bound the recompute with the same lookback the publisher uses. An
unbounded recon is hub leak 4 on a timer, then a RED that does not tell you whether the mark is
wrong or the recon scanned 2016. Catchup recon is a catchup unit (**data-pipeline-operations**).

**Test:** change one eligibility rule. Do publish, eval freeze, *and* stored-vs-recomputed recon
move together on the lookback window — or only one of the three, on lifetime rows?

From Evan-Kim2028/agent-skills, MIT.

More from Evan-Kim2028/agent-skills