eval-hillclimb

Files (3)

• SKILL.md
• grader.example.mjs · templates/
• runner.mjs · templates/

SKILL.md (first lines)

# eval-hillclimb

Provider-neutral eval design and hillclimbing. Two modes, chosen by the argument:

- `build-eval` builds `evals/<name>/` in the target repo and runs a validated baseline.
- `hillclimb` improves a chosen surface against an existing `evals/<name>/`.

If no argument is given, ask which mode. If `hillclimb` is requested and no eval exists, offer `build-eval` first.

## Ground rules (both modes)

- Every model call costs money and time. State the run size (cases x repeats x configs, rough duration) and get approval before any run.
- Pause for the user at the checkpoints marked **STOP**. Do not continue past one on your own.
- Keep secrets, customer data, and personal data out of committed fixtures. Ask about retention and sensitivity before reading production transcripts.
- The judge model must differ from the model under test. Record exact model IDs, thinking levels, and skill or prompt versions in every results file.
- Evals must exercise real pi behavior. The runner shells out to pi non-interactively, for example:
  `pi -p --no-session --model <provider/id[:thinking]> --skill <path> -- "<input>"`
  Useful isolation flags: `--no-skills` (then add only `--skill` paths under test), `--no-context-files`, `--no-extensions`, `--tools <allowlist>`, `--no-tools` for judges, `--session-dir <tmp>` for multi-turn cases.
- Run each trial in a fresh temp copy of the fixture workspace. Leftover files, git history, or sessions from earlier trials must not leak answers.

## Artifacts in `evals/<name>/`

```
cases.jsonl        one case per line: id, input, setup (optional), expect or claims, why_hard
split.json         {"train": [...ids], "test": [...ids], "seed": N}   (hillclimb only)
grader.mjs         programmatic grader, or judge prompt builder plus parser (start from templates/grader.example.mjs)
judge.md           rubric of yes/no claims (LLM judge only)
run.mjs            runner, copy from templates/runner.mjs and adapt
results/<run-id>.jsonl      one line per trial: case, repeat, config, output, grade, tokens, ms, error
results/<run-id>/<case>-<r>.txt   full transcript per trial
report.md / report.html     score with 95% interval, per-case table linking transcripts
```

Report pages are static files that open locally and load nothing from the network.

---

## Mode: build-eval

### 1. Interview

Ask briefly:
- What is under test (app path, prompt, skill, tool description) and how users hit it in real use.
- What a good output looks like, and what a bad one looks like.
- Which models or configs matter (tested model, candidate cheaper models, judge model).
- What sources of real examples exist.

### 2. Source cases, in this order

1. Real transcripts or production traffic (after the retention and sensitivity check).
2. Bug reports, tickets, and complaints.
3. 5 to 10 cases the user writes by hand.
4. Cases synthesized from the codebase, anchored on the real ones above.

Rules:
- Cases must mirror the real task distribution, not what is easy to generate or grade. User traffic can skew easy, so add deliberately hard cases.
- Do not pick cases because today's model fails them. That measures one model's weak spots. For every hard case, write `why_hard` in plain words; if you cannot say why it is hard, drop it.
- Everything the grader checks must be stated or implied by the case input. Two domain experts should reach the same verdict.
- Aim for 20 to 60 cases to start. Fewer makes the noise too large to see anything.

Write a simple review page (`evals/<name>/inputs.md` or `.html`) listing every input with its source and `why_hard`.
**STOP**: the user confirms, edits, or removes cases before you go on.

### 3. Pick the cheapest grader that fits

- **Programmatic first** when outputs are constrained: exact match, label from a fixed set, JSON schema, tests passing, file exists, command ran.
- **LLM judge** only for open-ended output. The rubric is a list of yes/no claims that can each be checked against the output, never a 1-to-5 scale. Score = fraction of claims met, and name any claim that is a hard fail. The judge returns JSON: `{"claims":[{"id":..., "met":true|false, "evidence":"..."}]}`.
- **Pairwise blind** when comparing to a baseline: the judge sees both outputs in random order, labeled A and B, is not told which is the baseline, and picks the better one or a tie. Record the position to check for position bias.
- The judge runs with no tools, no skills, no context files, and a model different from the tested one.

### 4. Validate the grader

- Grade 5 to 10 real outputs, show each verdict with its evidence, and ask: "Would you have scored any of these differently?" **STOP** until the user agrees. Fix the rubric or the cases, not the verdicts.
- Grade the same output twice for several cases. If a verdict flips, tighten the claim wording or make it programmatic.
- Include at least one known-good and one known-bad output. The good one must pass and the bad one must fail.

### 5. Baseline run

State the size (cases x repeats x configs, rough time), get approval, run with at least 3 repeats. Then check:

- **Plumbing**: count timeouts, API errors, empty or cut-off outputs, and nonzero exits. Report them separately; never let infra failures count as model failures. Rerun those trials.
- **Noise**: report the score with a 95% interval (bootstrap over cases, repeats averaged per case) and the run-to-run spread. Estimate the smallest difference the eval can detect (about 2 x the standard error of a difference). If that is larger than improvements the user would care about, say so and suggest more cases or repeats.
- **Always-fail cases**: a case failing every repeat is a sign of ambiguity or a broken grader. Read its transcripts and flag it.
- **Headroom**: if the baseline is about 95% or higher, warn that quality has no room to move. Suggest aiming the hillclimb at cost, latency, or output length at equal quality, or adding harder cases.
- Optional sanity check: a stronger model or higher thinking level should score higher. If it does not, suspect the cases or grader.

Write `report.md` (and `report.html` if asked): score, interval, noise estimate, plumbing counts, flagged cases, per-case table linking transcripts.

---

## Mode: hillclimb

### 1. Setup

Ask:
- **Goal**: better quality, or lower cost, latency, or length while quality holds. Also the smallest gain worth acting on.
- **Editable surface**: which of system prompt, skill files, instruction files (AGENTS.md etc.), tool descriptions, model and thinking level, harness code. Everything else is read-only.
- **Budget**: maximum rounds and spend.

Then:
- Split cases at random into train (about 2/3) and test (about 1/3) with a recorded seed, stratified by source if possible. Save `split.json`.
- Confirm the eval noise is smaller than the smallest gain the user cares about. If not, stop and recommend more repeats or cases first.
- Run or reuse the baseline on both splits under the current config. Commit or stash so every round can be reverted cleanly (one git commit per kept change).

### 2. Isolation rules

- **The test set is never read.** Do not open test cases, test outputs, or test transcripts. Only aggregate test scores are seen.
- **Never paste failure content into the prompt or skill.** No copied case inputs, expected answers, or transcript snippets. Fix the general behavior instead.
- **Keep answers out of reach.** The model under test must not be able to read `evals/`, expected outputs, or grader code during a trial. Run from a temp workspace that does not contain them, and restrict tools if needed.

### 3. One round

1. Read the previous round's **train** transcripts and grades only.
2. Find the most common failure and its root cause.
3. Propose **one** change as a patch that fixes the cause (rewrite the section that causes it, add the missing rule, change the parameter). Avoid cosmetic rewording whose effect cannot rise above noise.
4. Show the patch and apply it.
5. Run train and test with the same repeats as the baseline.
6. Decide:
   - Train and test both improve beyond noise: **keep** (commit).

From yasuhito/skills, MIT.

More from yasuhito/skills