eval-suite-audit

Files (1)

• SKILL.md

SKILL.md (first lines)

# Eval Suite Audit

Use this skill when the task is to review an eval suite for an AI system — the harness that measures whether the model, prompts, and tools actually work. A bad eval suite is worse than no eval suite: it gives confidence where there should be doubt.

## Use when

- the team is about to use eval results to gate a release
- eval scores are suspiciously high or suspiciously stable
- a prompt or model change went green on evals but red in production
- adopting a new evaluation framework and the team wants a sanity check

## Workflow

1. **Inventory the suite.** Find the eval runner, the suites, the datasets, the metrics, the report sink. For DAX specifically: `script/eval.ts`, suites under `evals/`, the `eval:smoke` and `eval:proof` scripts.
2. **Classify metrics.** For each metric, record:
   - What it measures (correctness, latency, cost, determinism, safety)
   - How it scores (exact match, LLM-as-judge, programmatic check, human review)
   - The score range and the pass threshold
3. **Coverage.**
   - Each major user journey or capability has at least one case
   - Edge cases and known-bad inputs included, not only happy paths
   - Negative cases: what the agent should *refuse* or *escalate*
4. **Dataset hygiene.**
   - Training data, system prompts, and few-shot examples are disjoint from eval data
   - Eval data is not pasted into prompts as examples
   - Synthetic eval data is regenerated when prompts change (so the data is not implicitly tuned to a specific prompt)
5. **Determinism vs stochasticity.**
   - Metrics that depend on the model are stochastic — they should be run multiple times and reported with variance, not a single number
   - Metrics that should be deterministic (replay, structured output validity) are run with the same seed and any flake is a finding
6. **Ladder structure.**
   - Smoke / fast: runs on every change, low cost, catches regressions
   - Proof / full: runs before release, comprehensive
   - Each rung has a clear gate (block release if X)
   - Gates not bypassed by a hand-edited "known failures" list without an expiry
7. **LLM-as-judge correctness.**
   - The judge model is different from the model being evaluated (or at least the audit notes the cross-contamination risk)
   - The judge has a rubric, not just "rate 1-10"
   - The judge's outputs are spot-checked by humans periodically
8. **Cost and runtime.**
   - Total cost per run within budget
   - Total runtime within CI budget
   - Cost reported per suite so expensive cases can be paid attention to
9. **Stability and signal.**
   - The same suite + same model + same prompt → same score (within stochastic variance)
   - Score variance reported as part of the result
   - Recent regressions in stable metrics flagged
10. **Reporting and storage.**
    - Results stored historically, not just printed
    - Diffs between runs visible (which cases moved, in which direction)
    - Results tied to the commit / model / prompt-set under test

## Checks

- "It passes" on a single run is not evidence — flag suites that do not report variance for stochastic metrics
- A suite that always passes is suspicious — flag suites with no failures in the last 30 days
- LLM-as-judge alone is not enough for safety-critical metrics — pair with programmatic checks
- Distinguish "coverage gap" (Medium) from "leakage" (High) from "metric does not measure what its name says" (Critical)

## Output contract

Findings follow [docs/skills/OUTPUT_CONTRACT.md](../../docs/skills/OUTPUT_CONTRACT.md).

Return:

1. **Verdict** — one line + a small table:
   - `Suites`, `Total cases`, `Models evaluated`, `Ladder rungs`, `Used as release gate?`
2. **Counts** — Critical/High/Medium/Low/Info totals
3. **Findings** — one table per category, ordered by impact:
   - Metric validity (does each metric measure what its name implies?)
   - Coverage
   - Dataset hygiene and leakage
   - Determinism handling
   - Ladder structure and gating
   - LLM-as-judge correctness
   - Stability and variance reporting
   - Reporting and historical storage
4. **Open questions / assumptions** — intended ladder semantics, intended gate policy
5. **Residual risk** — anything the audit cannot verify without running the suite
6. **Next actions** — concrete fixes ordered by impact

## Evidence to collect

- Eval runner path, suite definitions, dataset paths
- Metric names, score functions, thresholds
- Most recent run results (pass/fail counts, variance if reported)
- Any "known failures" or skip list with its expiry policy
- Whether the suite is wired to a release gate (`release:verify` or equivalent)

From ShaileshRawat1403/dax, MIT.

More from ShaileshRawat1403/dax