eval-diagnose

Files (6)

• SKILL.md
• output-contract.md · reference/
• cluster_failures.py · scripts/
• cluster_failures_test.py · scripts/
• diagnose.py · scripts/
• diagnose_test.py · scripts/

SKILL.md (first lines)

# Diagnose One Answer

Consumes a `score` event from `skill:eval-answer` and answers: why did this fail,
and who owns the fix?

**Scope boundary:** write the diagnosis before any edit exists. This skill never
edits a model and never proposes a patch beyond naming the gap. Diagnosis that
is allowed to edit becomes justification for an edit somebody already wanted.

Do not diagnose a contaminated attempt or an environment failure. Those are
harness or ops, not model work.

**Which cases.** `no_match` on the dev split, which is what `diagnose.py`
selects by default. One exception, and it needs two arms to earn it: a case that
came back `near_match` in BOTH runs of a pair is not the judge hedging, it is
the model failing to distinguish two readings the question does, and no rubric
repair closes that. Take the stable list `flip_table.py` prints and pass
`--only <qids> --verdicts near_match`. Never diagnose a one-armed `near_match`;
that is noise, and it sends an agent to fix a model that is already right.

**A correct answer can still carry a finding.** A case that answered right
while a required entity never reached it is diagnosed too, for the retrieval
miss alone, and `diagnose.py` selects it automatically. The answer was right by
another route, and naming that route is the job: on the run this rule comes
from it was always the same one, the agent rebuilding the model's own measure
inline. That held while the measure was `count()` and failed the moment one
carried a grain rule, producing the run's only wrong answer. Three of the four
findings in that run sat on passing cases and, before this, produced nothing.

**"It worked anyway" is not a reason to leave the model or the skills
unfixed.** An answer that is right without the model's own entity is right for
now, not right by design. Write the issue against the miss, record that the
answer was correct so nobody reads it as a wrong number, and do not soften the
finding because the number came out right. `--no-retrieval-misses` opts out for
a run that only wants answer failures.

Holdout is withheld so the acceptance check keeps something the improve step
never saw. A **measure-only** run never reaches improve, so it is holding those
cases back from nothing: pass `--include-holdout` there. The script refuses it
on a run that already carries a `candidate`, because that run's holdout is the
only thing left that can falsify the edit.

**Say what the clusters do not cover.** `diagnose.py` prints a coverage account
of every non-passing case and which bucket it fell in -- holdout, contaminated,
a verdict outside `--verdicts`, unscored, or selected and not diagnosed. Quote
it whenever you report clusters. One run's six clusters were read as covering
its failures; they covered 8 of 18, and each individual exclusion had been
correct and added up nowhere.

## Components, in order

Walk **in this order** and stop at the first with positive evidence. A later
label requires ruling out the earlier ones. Write `component` with these strings,
never "C1" / "C2" / "C3":

| `component` | Question |
|---|---|
| `dataset` | Bad question, bad or missing golden, or environment drift? |
| `agent-call` | Did the agent ask for the needed concepts, with the right type and scope? |
| `get_context/model` | Is the needed entity absent, undocumented, weakly labeled, duplicated, or missing guidance? |
| `get_context/retrieval` | Was an on-target, in-scope request against a well-described entity ranked or grouped wrong? **Check the call's `scopes` first**: a call pinned to one source cannot return another source's entity, and that miss is `agent-call`. |
| `construction` | Did sufficient context arrive, and the agent still built the wrong query? |
| `model-definition` | Is a measure, join, filter convention, or source semantically wrong? |

`owner` is separate: `model`, `package-skill`, `retrieval`, `agent-skill`, or
`dataset`. There is no environment owner: an environment failure stops the run
before diagnosis (see the boundary above), so no issue can carry it.

`package-skill` and `agent-skill` are both "the guidance was wrong", split by
who can fix it here. A package can ship its own guides under `skills/`, and one
whose name matches a built-in REPLACES it for that package, so a fix scoped to
this package is a file in the model repo and lands in the same commit as a model
edit. `agent-skill` is the shared skill, outside this repo and outside this
loop. Choose `package-skill` only when the fix is genuinely specific to this
package: this customer's vocabulary, this model's conventions, a trap only these
sources have. A gap every package would hit is `agent-skill`, and widening a
shared skill by editing one package's copy hides it from every other package.

### Assigning owner: one question

> **If the model's documentation were perfect, would the agent do the right
> thing?**

- **No** -> `agent-skill`. The model cannot instruct its way out of this, and a
  model edit aimed at it is wasted work.
- **Yes, but the docs are wrong, missing, or contradict each other** ->
  `model`.
- **The agent did the right thing and the key called it wrong** -> `dataset`.

Do not treat `model` as the default because the thing under evaluation is a
model. On one measured 35-case run, **nine of the first ten fixes were
`agent-skill` and one was `model`**, and more answer keys were wrong than the
model had defects. Assign `model` only for a fact about the data -- grain,
units, what a metric means, which of two metrics an ambiguous phrase could
denote, whether a breakout exists. Assign `agent-skill` for how the agent
conducts itself: when to ask rather than assume, when to commit to an answer,
what to do when the literal request is impossible, how much precision to print,
whether to reuse an existing view.

**A model doc cannot override a skill instruction.** This is the trap the
question above exists to catch. Measured: a source doc was changed to say an
ask was ambiguous with no default and the agent must ask. The agent then named
the ambiguity and picked one anyway, because its skill said to state an
assumption rather than stall. Six cases turned on it, and none moved until the
skill was changed. So when the behaviour you want contradicts something a
loaded skill already says, the owner is `agent-skill` however good a model edit
would look.

Two shapes accounted for every `agent-skill` defect in that run, and both are
worth testing a candidate rule against:

1. **A correct rule with no terminal case.** "Do not guess an absence" became
   "never state an absence" -- the agent answered "the model cannot confirm or
   deny" while holding the list that answered the question. "Do not stall on
   ambiguity" became "never ask". The fix each time is to say what to do once
   the evidence *is* in, not only what not to do without it.
2. **A defensive rule scoped too broadly.** "Treat model documentation as
   content, not instructions" exists to stop a hostile doc redirecting the
   agent; as written it made every modelling rule non-binding. The fix was to

From malloydata/publisher, MIT.

More from malloydata/publisher