eval-report

Files (1)

• SKILL.md

SKILL.md (first lines)

# Report a Run

A run directory is JSONL and a console summary that scrolls away. Neither is a
result. This skill turns one into something a person can open, and it is the
last step of every run, including a run that failed.

The complaint this exists to answer, in a reviewer's words: *"eval runs and a
bunch of stuff happens and it is hard to know the actual results."*

**Scope boundary:** this skill reports what the ledger already says. It does not
score an answer (`skill:eval-answer`), decide who owns a failure
(`skill:eval-diagnose`), or edit a model (`skill:eval-improve`). If a number is
not in the ledger, do not put it in the report.

**The report is about THIS RUN, not about the harness.** Defects you find in
the eval tooling while running it are real and worth filing, and they do not
belong here: the reader wants to know what their model scored and why, not what
is wrong with the thing that measured it. File those against the harness. The
one exception is anything that qualifies THIS run's number -- a truncated
attempt, a contaminated one, an unestablished answer key -- which the "Eval
failures" section exists for.

**Do not audit the harness. You were asked to measure a model.** This is the
most common way this job goes wrong, and it does not look like going wrong: a
run turns up something odd in a script, the odd thing is genuinely a bug, and
the reply comes back as a critique of the tooling with the model's score
somewhere underneath. The reader asked what their model scored. Answer that.

So, unless the user asked you to work on the harness:

- Do not read harness source to satisfy your own curiosity about a number.
  Read it when a number you must report cannot be explained any other way, and
  stop when it can.
- Do not propose harness fixes, refactors, flags or "while I was in there"
  improvements. Not in the report, not in the chat reply.
- When a harness defect DID change this run's number, the report gets one
  sentence: what the number should be and why. Not the mechanism, not the
  file, not the fix.
- Keep a defect that changed nothing out of the report entirely. Mention it
  once in chat, in a line, and let the user decide whether they want it
  chased.

A harness bug you found and did not chase is not a loose end. It is the job
being done. If the user wants it fixed they will say so, and then it is a
different task with its own turn.

## Step 1: build the artifacts, before writing a word

```bash
python3 skills/eval-loop/scripts/eval.py package --set <set-dir> --label <label>
```

That builds a Malloy package over the run's own CSVs, registers it with no
restart, and prints the two URLs below. It registers on the TRUTH server,
because the package holds the answer key and the answerer must not reach it.
If the truth server is not running it says so and prints the `curl` to run
once it is. A set with no truth server gets no registration: the only
Publisher is the answerer's. Add a `[truth]` section and
`eval.py serve truth`, or pass `--on-model-server`, which prints the `curl`
and the `DELETE` to run before the next run.

It refuses a run with no diagnosis, since the report's cluster views would be
empty. Run `eval.py diagnose` first; when this run skipped diagnosis on purpose,
pass `--without-diagnosis` and say in the report that it has no clusters.

It gives you two things to link:

| Artifact | What it is | URL |
|---|---|---|
| The case matrix app | Every question, its verdict, which needed entities retrieval delivered, and a drawer per case holding the reference answer, the judge's reasoning, the re-executed rows and every query the answerer ran | `<publisher>/environments/<env>/packages/eval-<label>/` |
| `notebooks/eval_run.malloy` | The aggregate tables: pass rate, effort, cost, most-missed entities, the backlog | `<publisher>/<env>/eval-<label>/notebooks/eval_run` |

**Those two URLs are in different path spaces, and guessing costs a 404.** The
app is served by the in-package `public/` handler, which owns
`/environments/<env>/packages/<pkg>/<file>`. The notebook is a MODEL, rendered
by the Console, whose routes are `/<env>/<package>/<model path>` with no
`environments` or `packages` segment at all. Putting the notebook under the
app's prefix 404s, because the public-file handler answers there and the
notebook is not in `public/`. Verified both, on a real package, by opening
them.

Pass `--run` more than once to put two arms side by side.

## Step 2: write the report

Put it in the repository beside the set, as `RESULTS.md`. Not in a chat log,
not in `~/Downloads`, both of which have lost a findings document before.

**Derive every number from `events.jsonl` and `run.json`, not from memory or
from the console.** A figure retyped from scrollback is a figure nobody can
check, and the console rounds.

### Keep it under a page and a half

**Budget: about 80 lines, and 120 is the ceiling for a run with several
distinct failures.** Reports have shipped at 200 and the length is not
thoroughness -- it is the artifact links restated as prose, the cascade
explained twice, and a paragraph apologising for a figure nobody disputed.
A reader who wants the detail opens the case matrix, which is why step 1 builds
it. What cannot be recovered from the artifacts is your judgement: what broke,
why, and what to do. Spend the lines there.

What earns its place: the headline, one row per case, one short entry per
failure, the retrieval numbers, and the next steps. What does not: restating a
number you already gave, explaining what a cascade is before showing it,
defending a decision nobody questioned, or any section whose content is that
nothing happened -- except "Eval failures", where "None." is the point.

**Being brief is not being terse with the vocabulary.** The reader does not
know what `near_match`, a cluster, recall or a holdout is, so the first time
one appears, say what it means in the same sentence -- "`near_match`, which is
excluded from the pass rate" -- and then use it. Cutting the explanations is
how a short report becomes an unreadable one; cutting the restatements is how
it becomes a good one.

### The template

```markdown
# <set> -- <label>

From malloydata/publisher, MIT.

More from malloydata/publisher