model-bench

Files (12)

• .gitignore
• SKILL.md
• evals.json · evals/
• campaign.yaml · evals/fixtures/
• compare.yaml · evals/fixtures/
• usage.md · references/
• ruff.toml
• quality-gate.sh · scripts/
• run.py · scripts/
• validate_skill.py · scripts/
• conftest.py · tests/
• test_run.py · tests/

SKILL.md (first lines)

# model-bench

Benchmark several models on the same dataset and report which is best — end to end —
by reusing the eval-harness multi-model comparison (F-024) and A/B eval campaign (F-025)
features. This skill **orchestrates, it does not re-implement**: its runner forwards to the
already-tested `eval-harness compare` and `eval-harness campaign` CLI subcommands, which in
turn reuse `run_comparison`, `record_run`, and `analyze`. With the real model-backed target
(F-027) wired into your config, the same commands benchmark live models; the bundled fixtures
use deterministic `echo` targets so the skill's own evals run offline.

Two modes:

* **compare** — run one dataset/scorers against ≥2 named models and rank them, with per-metric
  deltas vs a baseline and a self-contained comparative report. One-shot.
* **campaign** — a persistent A/B test of two arms whose per-arm pass/total counts accumulate
  across runs in an append-only store; `analyze` decides via Wilson intervals and never claims
  significance below its power floor.

## 1. Preconditions (input contract)

Confirm before executing. If any fails, stop and report the exact missing requirement.

* `langfuse-eval-harness` is importable (`pip install -e .` from the repo root; add the
  provider extra — `[openai]`, `[bedrock]`, `[anthropic]` — only when benchmarking real models).
* An eval config YAML exists with the relevant block:
  * **compare**: a `comparison` block with ≥2 uniquely-named models (each a `{type, params}`
    target) and a shared `dataset` + `scorers`.
  * **campaign**: an `ab_campaign` block with two distinctly-named arms, the `score` to test,
    and a `min_sample` power floor.
* For real models, the provider credentials are present in the environment (never in the
  config — use `${VAR}` interpolation). The bundled fixtures need no credentials.

## 2. Procedure

### compare

```
python scripts/run.py compare --config <eval.yaml> [--offline] [--html OUT.html] [--json OUT.json]
```

Prints the ranking (`ranked by <score> (<metric>): A > B > C`). `--offline` uses an in-memory
Langfuse client; `--html`/`--json` write the comparative report.

### campaign

```
# Record one run (repeat over time to accumulate counts):
python scripts/run.py campaign --config <eval.yaml> --store <store.jsonl> --mode record [--offline]

# Analyze the accumulated store:
python scripts/run.py campaign --config <eval.yaml> --store <store.jsonl> --mode analyze [--offline] [--json OUT.json]
```

`record` appends per-arm pass/total counts to the JSONL store; `analyze` accumulates them and
prints a decision (`a_better` / `b_better` / `no_difference` / `cant_tell` below power).

## 3. Output contract

* `compare` exits 0 and prints the ranking; with `--json`/`--html` it writes a deterministic,
  self-contained report (no external assets).
* `campaign --mode record` exits 0 and appends one record per arm; `--mode analyze` exits 0 and
  prints the decision and delta.
* A config missing the required block exits 2 with a clear message; a missing harness install
  exits 1.

## 4. No hard-coded values

Models, endpoints, datasets, the ranking score/metric, and campaign power floor all come from
the config; credentials come from the environment via `${VAR}` interpolation. Nothing about a
specific model or endpoint is baked into this skill.

## 5. Validation

```
python scripts/validate_skill.py --skill . --tier structural,behavioral
```

The behavioral evals drive `compare` and a `record` → `analyze` campaign over the bundled
`echo`-target fixtures — fully offline and deterministic.

## 6. References

* `references/usage.md` — config block shapes and worked examples.

From ianshank/Agents, Apache-2.0.

More from ianshank/Agents