evaluate-skill-with-copilot-sdk

Files (19)

• SKILL.md
• quality_evalset.example.json · assets/
• quality_review_template.html · assets/
• trigger_dashboard_template.html · assets/
• trigger_queryset.example.json · assets/
• copilot-sdk.md · references/
• models.md · references/
• quality-evals.md · references/
• schemas.md · references/
• triggering-evals.md · references/
• eval_harness.py · scripts/
• generate_review.py · scripts/
• generate_trigger_dashboard.py · scripts/
• grade_quality.py · scripts/
• list_models.py · scripts/
• optimize_description.py · scripts/
• run_quality_eval.py · scripts/
• run_trigger_batch.py · scripts/
• run_trigger_eval.py · scripts/

SKILL.md (first lines)

# Evaluate a skill with the Copilot SDK

This skill measures whether another skill actually works by running it in
isolated GitHub Copilot SDK sessions. It follows the evaluation methodology
documented at
[agentskills.io](https://agentskills.io/skill-creation/evaluating-skills), and
runs anywhere the Copilot CLI is installed.

There are two independent questions, and you can answer either or both:

1. **Triggering** — does the skill's `description` cause the agent to invoke it
   for relevant prompts, and stay quiet for irrelevant ones? This is decided
   entirely by the `name` + `description` the agent sees at startup.
2. **Output quality** — when the skill *is* used, does it produce meaningfully
   better results than the same agent with no skill? This is decided by the
   skill's body, bundled scripts, and references.

Keep them separate: a skill can trigger perfectly but give mediocre output, or
give great output but never trigger. Different fixes, different evals.

## The core loop

**Run → review → improve → repeat**, with a human in the loop. Don't act on the
numbers alone — the benchmark tells you *what* changed; the actual outputs and
transcripts tell you *why*. The scripts only ever read the skill and write
results elsewhere: you apply every description and body edit yourself, so a bad
automated suggestion can't corrupt a real skill.

```mermaid
flowchart LR
  A[Pick skill + goal] --> B{Triggering<br/>or Quality?}
  B -->|Triggering| T1[Write queryset] --> T2{Single skill<br/>or plugin?}
  T2 -->|Single| T3[run_trigger_eval] --> T4[generate_trigger_dashboard] --> T5[optimize_description] --> R
  T2 -->|Plugin| T6[run_trigger_batch] --> T7[generate_trigger_dashboard] --> R
  B -->|Quality| Q1[Write evalset + assertions] --> Q2[run_quality_eval] --> Q3[grade_quality] --> Q4[generate_review] --> R
  R[Human review + improve skill] --> B
```

## Prerequisites (check once)

The scripts need the Copilot SDK and a logged-in Copilot CLI. Verify before the
first run:

```bash
python -c "import copilot; print('SDK', copilot.__version__ if hasattr(copilot,'__version__') else 'ok')"
```

If that errors, install with `pip install "github-copilot-sdk>=1.0.0"` and then
run `python -m copilot download-runtime` to cache the CLI binary. Make sure you
are signed in to GitHub Copilot. The scripts authenticate as the logged-in user
automatically — no token wrangling.

All scripts live in the `scripts/` folder next to this file. In the commands
below, `$SCRIPTS` stands for that folder's path. Running a script by its full
path is enough — each one adds its own directory to the import path, so the
shared `eval_harness.py` resolves without any install step.

**Choosing a model.** Runs default to `claude-haiku-4.5` — fast and cheap, which
matters because evals fan out into many sessions. If the user names a model,
resolve it to an id first — `python $SCRIPTS/list_models.py --resolve "opus 4.8"`
prints the id, or, when the name is ambiguous or unknown, the models available on
this account for the user to pick from. Pass the chosen id to any eval script via
`--model` (or set `COPILOT_EVAL_MODEL` to change the default globally).
Availability is plan-dependent, so the live list is the truth;
[references/models.md](references/models.md) is a snapshot. For *triggering* evals,
match the model your users actually run. For *quality grading and the
description-improvement judge*, a stronger model (`claude-sonnet-4.6`,
`claude-opus-4.8`) gives sharper verdicts — pass it via `--model`.

**Where results go.** Every script writes under a results root: whatever you pass
as `--out DIR`, or by default `<skill-name>_eval_results/` at the repo root, keyed
to the skill under test so evaluating several skills never collides. Artifacts are
filed by eval type — `triggering/` (trigger results, description optimization) and
`quality/` (the with/without workspace of `iteration-N/...`, benchmark, and review
page). For the quality sequence, pass the same skill (or `--out`) through
`run_quality_eval` → `grade_quality` → `generate_review` so they share one location.

---

## Path A — Triggering evaluation

Use this when the question is "does it fire at the right times?" Full details on
queryset design and the optimization loop are in
[references/triggering-evals.md](references/triggering-evals.md) — **read it
before writing queries**, because query quality is what makes or breaks this eval.

### A.1 — Write a queryset

Realistic prompts labeled with whether the skill should trigger. The canonical
format is an object with a `skill_name` and an `evals` array:

```json
{
  "skill_name": "my-skill",
  "evals": [
    {"id": 1, "query": "...", "should_trigger": true, "rationale": "..."},
    {"id": 2, "query": "...", "should_trigger": false, "rationale": "..."}
  ]
}
```

A flat JSON list of `{"query", "should_trigger"}` objects is also accepted as
shorthand. Aim for ~20 queries: 8-10 should-trigger, 8-10 should-not, weighted
toward *near-misses* (queries that share keywords but need something else). See
the schema in [references/schemas.md](references/schemas.md) and the worked
example in [assets/trigger_queryset.example.json](assets/trigger_queryset.example.json).

### A.2 — Single-skill measurement

Use this when evaluating one skill in isolation or against its siblings.

```bash
python $SCRIPTS/run_trigger_eval.py \
  --skill-path PATH/TO/skill \
  --queryset queryset.json \
  --runs 3
```

Each query runs 3× (triggering is nondeterministic) and passes if its trigger

From tezansahu/ai-garage, GPL-3.0.

More from tezansahu/ai-garage