eval-judge

Files (5)

• SKILL.md
• refusal.md · reference/
• retrieval-judge.md · reference/
• suspect-goldens.md · reference/
• writing-rubrics.md · reference/

SKILL.md (first lines)

# The judge

JUDGE_VERSION: 6

This skill IS the judge. One fresh judge subagent is spawned per attempt, with
this skill installed in its workspace and the case materials in its prompt. It
is loaded, not pasted -- so the prompt carries the case and this carries the
doctrine, and a judge that needs to read a Malloy query can reach for the
skills beside it rather than being handed a transcription.

Measured when it stopped being pasted, on the case that had oscillated
(a valued golden against a model with no trace of the concept):

    pasted into the prompt   match / no_match / match / match
    loaded as this skill     no_match x4, and the reasoning cites the rule

It costs about 2.5x per verdict, which is the price of the judge actually
reading its own rules.

Record `judge_version` and this file's git blob sha
(`git rev-parse HEAD:skills/eval-judge/SKILL.md`, or the model repo's copy) on
every verdict, so a rubric change never silently rewrites what old scores
meant.

The judge is not blind. It sees the golden. It must never be the same
subagent that answered, and it never edits anything: it returns a verdict
object and stops.

## Read one of these before you decide

This file is the decision procedure. Four situations have their own rules, and
each is a file beside this one. Read the file BEFORE emitting a verdict, not
after -- these are the cases where judging from the general rubric alone gets it
wrong, which is why they are called out rather than summarised.

| If | Read |
|---|---|
| the answer declines, or gives no value at all | `reference/refusal.md` |
| the golden itself looks wrong to you | `reference/suspect-goldens.md` |
| you are judging retrieval, not an answer | `reference/retrieval-judge.md` |
| you are AUTHORING a case rather than judging one | `reference/writing-rubrics.md` |

The first row is the one that catches people. A refusal is only exempt from
containment when `golden.kind` is `unanswerable`; against a golden that holds a
value, an answer containing none of it is `no_match` however well it reasons.
`reference/refusal.md` is the whole rule.

A third kind holds no value: `criteria`, where the case's rubric IS the key
and there is no number to contain. Grade the clauses and nothing else. Do not
manufacture a figure to check the answer against, and do not read the absence
of a value as a missing golden: a `criteria` golden is complete. Report
`gold_status` on it the same way, on the criteria rather than on a number, so
a clause that contradicts the model still surfaces.

## Answer judge

Input, all of it (a judge with only two row sets grades formatting, not
intent):

- the question, exactly as the answerer saw it
- the golden: rows or scalar, plus `canonicalQuery` when present
- the prediction: the rows the CONDUCTOR re-executed from the answerer's
  `final_query` (never the answerer's self-reported rows)
- the relevant source and field definitions from the model (docs, join list)

Output, exactly this shape:

```json
{
  "verdict": "match | near_match | no_match",
  "confidence": 7,
  "why": "one short paragraph",
  "column_pairing": { "gold_col": "pred_col", ... },
  "gold_status": "verified | verified_benign | suspect | verified_wrong",
  "gold_note": "why, when not verified"
}
```

### Rubric

1. **Judge intent, not formatting.** The question defines what counts. A
   result that answers the question in a different but faithful shape is a
   match.
2. **Gold-subset containment.** The prediction must CONTAIN the gold answer.
   Extra columns or benign extra context downgrade to `near_match` at worst;
   they never make a containing answer `no_match`.
3. **Name the column pairing.** Pair each gold column with the prediction
   column that carries the same meaning, using names, the question's role for
   the value, and the values together. Never pair numeric columns by value
   overlap alone: a year column is not a count column even when magnitudes
   overlap. If a gold column has no counterpart, say which.
4. **Rows are a multiset.** Order matters only when the question asks for an
   order. For a "top N" with possible ties, check that the boundary value is
   right and every returned row legitimately qualifies; any valid tie-break is
   a match.
5. **Tolerances.** Numeric equality within small rounding (relative 1e-6, or
   the display precision the golden uses). A percentage and its fraction
   (50 and 0.5) are the same value in different units when the pairing says
   the column is a rate.
6. **Confidence 1 to 10.** 5 or lower means the case needs a human:
   the conductor records `needs_human`, which is neither a pass nor a fail.
   Do not inflate confidence to be helpful; a wrong confident verdict is worse
   than an abstention.
7. **`near_match` is not a soft pass, and it is not a soft fail.** It is a
   third outcome meaning *defensibly different*: the answer took a reading the
   rubric allows but did not prefer, broke a tie the other way, or buried a
   caveat that should have been plain. It is excluded from the pass rate and
   from the acceptance check, exactly like `needs_human`.

   So do not reach for it to avoid a hard call. If the prediction contains the
   gold answer, that is `match` -- extra columns and benign extra context never
   reduce it (rule 2). If it does not, and the rubric does not sanction the
   reading that produced it, that is `no_match`. Use `near_match` only when you
   can name the rubric clause that makes the difference defensible.

   **A rubric clause cannot make a wrong VALUE defensible, and a clause that
   tries is a defect in the rubric rather than a licence to you.** `near_match`
   turns on the answer being right under a reading the QUESTION allows -- a tie
   broken the other way, a grain the question left open, a basis the question

From malloydata/publisher, MIT.

More from malloydata/publisher