eval-loop

Files (32)

• SKILL.md
• acceptance-check.md · reference/
• auditing-an-answer-key.md · reference/
• checking-the-judge.md · reference/
• golden-side-door.md · reference/
• measurement.md · reference/
• running-a-run.md · reference/
• setting-up-a-set.md · reference/
• agent_harness.py · scripts/
• agent_harness_test.py · scripts/
• build_run_package.py · scripts/
• build_run_package_test.py · scripts/
• check_judge.py · scripts/
• check_judge_test.py · scripts/
• check_set.py · scripts/
• check_set_test.py · scripts/
• eval.py · scripts/
• eval_test.py · scripts/
• flip_table.py · scripts/
• flip_table_test.py · scripts/
• package_skills_test.py · scripts/
• platform_target_test.py · scripts/
• run_baseline.py · scripts/
• run_baseline_test.py · scripts/
• serve.py · scripts/
• serve_test.py · scripts/
• README.md · templates/eval-run-package/
• eval_run.malloy · templates/eval-run-package/
• eval_run.malloy · templates/eval-run-package/notebooks/
• app.js · templates/eval-run-package/public/
• index.html · templates/eval-run-package/public/
• publisher.json · templates/eval-run-package/

SKILL.md (first lines)

# The Evaluation Loop

You conduct this loop. There is no batch orchestrator to start, no eval API,
and no eval MCP tools. The ledger is plain files: the set in the model
package's git repository, and each run in the set's workdir
(`reference/ledger-schema.md` in `skill:eval-answer` defines every file and
event). `scripts/eval.py` runs each step from the set's `eval.toml`;
`reference/running-a-run.md` has the commands. Scoring is an LLM judge you spawn per case. There is no
scripted scorer, and there will not be one: a script that can pass a wrong
answer is worse than none. The scripts under `scripts/` run the loop -- they
answer, re-execute, spawn the judge, compare runs, and write the ledger -- but
none of them decides whether an answer was right.

```
scrape/run  ->  eval  ->  diagnose  ->  improve  ->  checkpoint
```

**This skill conducts; it does not restate.** Scoring lives in
`skill:eval-answer`. Components and owners live in `skill:eval-diagnose`.
Edit rules live in `skill:eval-improve`.

Do not merge **eval** into **diagnose**. A conductor who scores while
explaining writes the explanation into the score. Do not skip the **acceptance
check** inside improve. The acceptance check decides whether *this* edit
stays. **Checkpoint** decides whether a *sequence* of accepted edits can be
undone.

## Where the rest of this lives

This file is the procedure. The things it used to carry inline are files beside
it now, because each is needed at one moment rather than every run, and loading
all of them for every run is how a skill stops being read.

| When | Read |
|---|---|
| the set does not exist yet, or has never run | `reference/setting-up-a-set.md` |
| about to run one | `reference/running-a-run.md` |
| a golden is wrong, doubted, or out of step with the model | `reference/golden-side-door.md` |
| auditing a key you doubt, or a set you did not author | `reference/auditing-an-answer-key.md` |
| deciding whether an edit stays | `reference/acceptance-check.md` |
| about to quote a number, set the band, or read a set's flips | `reference/measurement.md` |
| the run finished and someone has to read it | `skill:eval-report` |
| you changed judge doctrine or its inputs | `reference/checking-the-judge.md` |

Read the file, do not work from the summary here. The acceptance-check rules and
the golden side door are both places where acting on a half-memory of the rule
produces a confident wrong answer rather than an error.

## The five steps

| Step | Job | Writes |
|---|---|---|
| **a. scrape / run** | Put cases in the ledger; spawn a blind answerer | cases; `attempt`, `tool_call` |
| **b. eval** | Judge the answer; score which required entities retrieval delivered | `score` |
| **c. diagnose** | Why it failed, who owns it | `issue` / `issue_status`. Stop. Do not edit. |
| **d. improve** | One smallest model edit, then the acceptance check | improve writes `candidate`; you write `acceptance_check`. Revert on reject. |
| **e. checkpoint** | Git commit after an accepted acceptance check | `checkpoint` event, then the commit |

Every run then **reports** (below). A run whose result exists only as JSONL and
a scrolled-away console summary has not been delivered.

**scrape** and **run** share a letter but are not the same job. Scrape writes
cases. Run writes attempts. Do not invent questions and score
them in one breath.

**Step d is `skill:eval-improve`, not a hand edit plus another arm.** The
failure mode is specific and it has happened: the edits were made by hand,
published, and validated by re-running the whole arm -- with the answer key
repaired in the same window. No `candidate`, `acceptance_check` or `checkpoint`
event exists in that run's ledger, and the resulting move cannot be attributed
to the model, the key, or the answerer. Re-running a whole arm after changing
two things is the one thing the acceptance check exists to replace: it scores
the edit against dev AND holdout, and it is cheaper than the arm.

### Scrape, minimally

Importing an existing corpus IS the scrape step: copy the set from its home
(for example a benchmarks checkout) into `evals/<set>/` and convert to the
ledger shapes. **`skill:eval-import` is that job in full** -- how to classify
what arrived with each question, why nothing imports as a verified value, and
the seal that makes a later edit to a question detectable. Read it whenever the
questions came from outside, which is most of the time. While importing:

- Freeze each case's `split`: `dev` or `holdout`. Diagnose and improve read
  dev cases only; the acceptance check runs both. A set that is all dev cannot defend an
  accept.
- Later, each diagnosed-and-fixed failure becomes a new frozen dev case, so a
  fixed bug cannot silently return.

Scraping from production logs (chat transcripts, retrieval traces) is the
other supported source, and usually the better one: real traffic asks what
people actually ask. Where your logs physically live is a host concern; look
for a host-specific log-fetching skill. `skill:eval-import` takes over once
you have the text, and its `reference/case-format.md` covers what a log pull
needs that a question list does not.

Prefer variety over volume when you sample, from either source. Cases that
differ in grain, source, filter shape, and phrasing are what move a
measurement; a second sample of the same case is nearly free of new
information.

### Mode aliases

Older mode names still work as aliases for how far one run walks:

| Alias | Steps |
|---|---|
| `measure` | scrape/run + eval (`eval.py package` then needs `--without-diagnosis`) |
| `triage` | plus diagnose |
| `improve` | plus improve + acceptance check + checkpoint on accept |

Say which alias (or which steps) you are running before the first question.
Record it in `run.json`. Do not mix steps in a way that lets the answerer see
gold, issues, or the model file.

Most runs should stop after eval. Diagnose when you need a histogram of
components and owners. Improve only for diagnosed *model* gaps, one batch at
a time. Checkpoint only after the acceptance check **accepts**.

From malloydata/publisher, MIT.

More from malloydata/publisher