skill-evaluator

Files (14)

• CHANGELOG.md
• SKILL.md
• generate_review.py · eval-viewer/
• evals.json · evals/
• adapters.md · references/
• grader.md · references/
• methodology.md · references/
• schemas.md · references/
• evaluation_isolation.py · scripts/_support/
• agent_adapters.py · scripts/
• aggregate_benchmark.py · scripts/
• run_quality_eval.py · scripts/
• run_script_checks.py · scripts/
• run_trigger_eval.py · scripts/

SKILL.md (first lines)

# Skill Evaluator

Test whether a skill is **correct, discoverable, and valuable** — not just whether
its unit tests pass. The harness is agent-agnostic: it drives whichever coding-agent
CLI the user uses, because Agent Skills are portable across all of them.

## When to use which layer

Three layers, increasing cost and fidelity (full rationale in
`references/methodology.md`):

| Layer | Question | Script | Needs a CLI? |
|------|----------|--------|--------------|
| 1. Deterministic | Do the scripts emit the **documented** numbers? | `run_script_checks.py` | No |
| 2. Trigger | Does the **description** activate on the right prompts? | `run_trigger_eval.py` | Yes |
| 3. Quality | Does following the SKILL.md **beat no skill**? | `run_quality_eval.py` → grade → `aggregate_benchmark.py` | Yes |

Always run Layer 1 (it's free). Add Layers 2–3 when you can run a coding-agent CLI.

## Step 0 — pick the agent CLI

Ask the user which coding agent they use, then map it to an adapter id. Supported:
`claude-code`, `openai-codex`, `antigravity` (the `agy` CLI that replaced Gemini
CLI on 2026-06-18), `cursor-cli`, `github-copilot-cli`, `amp`, `opencode`,
`grok-cli`. See the full matrix and auth in `references/adapters.md`, or run:

```bash
python scripts/agent_adapters.py list
```

Confirm the binary is installed and the auth env var is set (the matrix lists it).
**Before any real run, dry-run it** to see the exact command:

```bash
python scripts/agent_adapters.py build <agent> --prompt "test" --workdir /tmp/wd
```

## Step 1 — deterministic script checks (always)

```bash
python scripts/run_script_checks.py --skill <path-to-skill> --json
```

Runs the `script_checks` in the skill's `evals/evals.json`, executing each script
and grading its `--json` output against machine-checkable assertions. Exit non-zero
on any failure — safe for CI. If the skill has few/no `script_checks`, add them for
every eval whose answer is computable (schema in `references/schemas.md`); this is
the cheapest, most durable guard against doc↔code drift.

## Step 2 — trigger / discovery eval

Does the description fire on the right prompts and stay quiet on near-misses?

```bash
# Dry-run first (prints the per-CLI commands, runs nothing):
python scripts/run_trigger_eval.py --skill <path> --agent <agent> --dry-run

# Real run with a labelled query set (~20: half should-trigger, half near-miss):
python scripts/run_trigger_eval.py --skill <path> --agent <agent> \
  --queries queries.json --runs-per-query 3 --json
```

Design the query set per `references/methodology.md` (positives + tricky
negatives). Without `--queries`, the skill's eval prompts are used as
should-trigger cases — add negatives for a real discrimination test.

## Step 3 — output-quality eval (the with/without delta)

The headline measure: does an agent following the SKILL.md beat no skill?

```bash
# 1. Dry-run the plan (no tokens spent):
python scripts/run_quality_eval.py --skill <path> --agent <agent> \
  --workspace <skill>-workspace --dry-run

# 2. Real run: with-skill AND no-skill baseline, isolated clean dirs each:
python scripts/run_quality_eval.py --skill <path> --agent <agent> \
  --workspace <skill>-workspace --iteration 1 --json
```

This installs the skill into a temp project skills dir for the with-skill run,
runs a clean baseline without it, and captures `outputs/`, `response.txt`, and
`timing.json` per run.

**Then grade each run** against its `assertions` and write `grading.json`
(`references/grader.md` — re-derive numbers, require concrete evidence, no partial
credit, critique weak assertions). For mechanically checkable assertions, reuse
Layer 1 rather than eyeballing.

**Then aggregate** into the benchmark with the delta:

```bash
python scripts/aggregate_benchmark.py <skill>-workspace/iteration-1 \
  --skill-name <name> --agent <agent> --json
```

`run_summary.delta.pass_rate` is the value of the skill. Surface patterns the
averages hide (`references/methodology.md`): non-discriminating assertions,
high-variance evals, time/token tradeoffs.

**Then generate the review and put it in front of the user *before* you self-grade**
(a standalone HTML page — no server needed):

```bash
python eval-viewer/generate_review.py <skill>-workspace/iteration-1/benchmark.json -o review.html
```

It renders the with/without delta, per-configuration stats, and an expandable
per-eval breakdown of each graded assertion (text, pass/fail, evidence).

## Step 4 — iterate

Improve the skill from the signals (failed assertions, weak-assertion feedback,
transcripts, human review), generalizing rather than overfitting, keeping it lean,
explaining the *why*, and bundling repeated work into scripts. Rerun into
`iteration-<N+1>/` and compare. Stop when results satisfy the user, feedback is
empty, or gains plateau. For "is the new version actually better?", use the blind
comparison described in `references/methodology.md`.

From HeshamFS/materials-simulation-skills, Apache-2.0.

More from HeshamFS/materials-simulation-skills