eval-guide

Files (1)

• SKILL.md

SKILL.md (first lines)

# Eval Guide

Use this guide when you are trying to answer **"did this skill or prompt change actually help?"** for `agent-tty` evals.

The short version: **do not trust a single run**. This eval stack now supports multi-trial sampling, parallel execution, trial aggregation, and paired baseline comparison because the underlying model behavior is noisy enough that one pass/fail result is not decision-grade.

## 1. What we learned about eval non-determinism

- Identical serial reruns showed a **~15-17% pass/fail flip rate** in practice, across both Codex and Claude runs.
- Scores moved even more often than hard pass/fail: **~30-39% of identical reruns changed score**.
- The movement was directionally balanced, which is the important point: this looked like **noise**, not a systematic drift up or down.
- Cross-provider checks reinforced that conclusion: in the parallel safety analysis, **Codex and Claude shared zero common regressions**, which is strong evidence that parallelism itself was not introducing consistent failures.
- Treat these findings as the baseline noise floor. If your "improvement" is smaller than that noise, it is not persuasive.

## 2. Run evals with enough statistical power

Always set `--trials` for real prompt or skill experiments.

Recommended trial counts:

- **Prompt lane:** `--trials 5` to `--trials 10`
- **Execution lane:** `--trials 3`
- **Dogfood lane:** `--trials 2` to `--trials 3`

Use concurrency to keep those sample sizes affordable:

- Start with **`--concurrency 4`** for real-provider runs.
- Recent measurements showed about a **3.4x wall-clock speedup** at that setting.
- Leave `--concurrency 1` only when you explicitly want fully serial behavior.

When `--trials` is greater than `1`, reports automatically include **Trial Aggregation** in `report.md` and `report.json`, including per-case:

- pass rate,
- pass-rate confidence interval,
- mean score,
- score confidence interval,
- standard deviation,
- and min/max score.

## 3. Compare before vs. after a skill change

Use a paired baseline comparison whenever you want to know whether a change helped. The comparison report uses **paired bootstrap confidence intervals** and paired win/loss/tie counts, so it is much more reliable than eyeballing two single runs.

1. Run a **baseline** on the exact lane, cases, provider, model, and trial count you care about.
2. Save the baseline `report.json` path.
3. Make the skill or prompt change.
4. Run the **candidate** with `--compare-baseline <baseline-report-path>`.
5. Read the comparison verdicts: `improved`, `regressed`, or `inconclusive`.

Practical reading rules:

- Do **not** call something better just because the mean moved in the right direction.
- Treat a win as meaningful only when the paired CI excludes `0` **and** the effect is practically large enough to matter.
- In the current implementation, the practical cutoffs are **`0.05` score delta** and, for overall pass rate, **`0.05` absolute pass-rate delta**.
- Tiny but statistically significant deltas are still not worth celebrating.

A reliable prompt-lane A/B loop looks like this:

```bash
BASELINE_JSON=$(npx tsx evals/run.ts \
  --provider codex \
  --model gpt-5.4 \
  --lane prompt \
  --condition self-load \
  --trials 5 \
  --concurrency 4 \
  --output evals/reports/prompt-baseline \
  --json | jq -r '.jsonReportPath')

# edit the skill or prompt

npx tsx evals/run.ts \
  --provider codex \
  --model gpt-5.4 \
  --lane prompt \
  --condition self-load \
  --trials 5 \
  --concurrency 4 \
  --output evals/reports/prompt-candidate \
  --compare-baseline "$BASELINE_JSON" \
  --json
```

## 4. Interpret results correctly

- **`inconclusive` is the default, healthy outcome when nothing meaningful changed.** In our same-skill sanity check, **23 of 24 cases were inconclusive** and the paired win/loss/tie total was **14W / 15L / 43T**.
- A few false positives are normal. At **95% confidence**, a **~1 in 24** spurious finding in a 24-case sweep is not surprising even when nothing changed.
- Read **wins / losses / ties** alongside the CI. They give an intuitive sense of whether the candidate is consistently helping or just bouncing around.
- If a comparison is mostly ties with a wide CI, the result is noise-dominated; add more trials before making a claim.
- If you only have two standalone reports and no paired baseline comparison, diff them only after stripping timing- and path-specific fields. That can catch large shifts, but it is much less trustworthy than `--trials` plus `--compare-baseline`.

## 5. Concurrency is safe and should be used

- `--concurrency 1` remains the default and preserves serial behavior.
- **Parallelism did not introduce regressions** in the safety checks; observed parallel flip rates were at or below the normal serial noise floor.
- `--concurrency 4-20` is a reasonable operating range when provider limits and budget allow.
- The current runner can execute **all three lanes concurrently** when concurrency is above `1`.
- Execution and dogfood work items run in **isolated temp homes/output directories** and clean up in `finally` blocks, so parallel runs do not share session state.

Use serial mode only when debugging; use parallel mode when sampling.

## 6. Quick reference commands

### Prompt-lane experiment

```bash
npx tsx evals/run.ts \
  --provider claude \
  --model claude-opus-4-6 \
  --lane prompt \
  --condition self-load \
  --trials 5 \
  --concurrency 4 \
  --output evals/reports/prompt-self-load
```

### Execution-lane check after workflow changes

```bash

From coder/agent-tty, Apache-2.0.

More from coder/agent-tty