eval

Files (1)

• SKILL.md

SKILL.md (first lines)

# eval

**You own the experiment design. Plan, blind, run, synthesize.** Evals test how a
change affects agent behavior before promoting it: a new skill variant, a structural
change, a prompt tweak. This is the measurement half of
[encode-lessons-in-structure](../encode-lessons-in-structure/SKILL.md): a skill edit is
a hypothesis until an eval (or real-work observation) confirms it changed behavior.

The failure mode is the observer effect: an agent that knows it's being evaluated
behaves differently, so candidates must run blind.

## Non-negotiables for blinding

- No `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
  `candidate`, or `arena` in any directory, file, or prompt the candidate sees.
- The candidate prompt looks like an organic user request. State the goal, not the
  meta: "build me a small todo cli", never "show how you follow the skill".
- No chain-eliciting cues. Asking the candidate to list which skills or principles it
  applied inflates citation behavior; ask for design notes generally and grade
  skill-following from the shape of the work, not self-report.
- Sanitize directory and slug names: project-shaped names a user might pick, never
  `candidate-1` or `agent-a`. Don't tell a candidate other candidates exist.
- The judge can know it's judging but sees outputs by sanitized label only, never by
  model or variant name. Comparing two variants: one judge scores both sets in a
  single pass on one scale, blind to which set each came from — two judge runs with
  different prompts don't compare, the calibration drifts.

## Steps

1. **Frame.** State the variant under test and what behavior counts as success. Write
   the rubric (3–6 concrete criteria) for the judge only; hold it back from candidates.
2. **Set up sanitized environments.** Per-candidate working dir with the variant in
   place, plus the context an organic task would have: a project skeleton, the skills
   the candidate would naturally read.
3. **Author one organic prompt.** What a user would type; no leakage of what's
   measured.
4. **Run N candidates in parallel**, one per sanitized dir, same prompt — on different
   models where the harness offers them, otherwise independent fresh sessions.
5. **Run one blinded judge**, on a different model where possible, seeing outputs by
   sanitized label plus the rubric.
6. **Verify behavior from evidence, not self-report.** Where the harness exposes
   candidate transcripts or tool logs, read which files each candidate actually
   opened — citing a principle is not reading it, and reading is not applying. Where
   it doesn't, grade from the artifacts alone: the code shape either shows the
   behavior or it doesn't.
7. **Read every candidate output yourself** end to end and compare with the judge's
   verdict. Disagreement means a biased model or an ambiguous rubric — resolve before
   concluding. Synthesize.

**Reply:** variant under test, rubric, per-candidate notes, judge's verdict, your
synthesis, and a promote / don't-promote recommendation.

From baranbingol1/bbstack, MIT.

More from baranbingol1/bbstack