develop-skill-evals

Files (16)

• CITATION.cff
• SKILL.md
• cspell-words.txt
• SUMMARY.md · references/
• README.md · references/agent-skills-eval/
• schema.md · references/agent-skills-eval/
• agent-skills-native-evals.md · references/
• evaluating-skills.md · references/agentskills/
• promptfoo-agent-skill-testing.md · references/
• SUMMARY.md · references/promptfoo/
• configuration-expected-outputs-deterministic.md · references/promptfoo/
• configuration-expected-outputs-model-graded-llm-rubric.md · references/promptfoo/
• configuration-expected-outputs-model-graded-search-rubric.md · references/promptfoo/
• guides-test-agent-skills.md · references/promptfoo/
• skill-judge-lessons.md · references/
• README.md · references/skill-judge/

SKILL.md (first lines)

# Develop Skill Evals

Turn an already-approved skill's lean evaluation plan
into concrete Promptfoo artifacts:
`promptfooconfig.yaml`, test cases, and assertions.
Treat the plan as settled —
this skill does not decide what should be evaluated,
and it does not run evals.

Do not use this skill to invent an evaluation plan from scratch,
edit the target skill's behavior, or execute Promptfoo.

## Scope

Trigger for: authoring or updating Promptfoo test artifacts for a skill whose
`specs/<skill-name>.md` already contains a completed `### Evaluation plan`
section.

Redirect instead of triggering when:

- No spec or lean evaluation plan exists yet for the target skill →
  hand off to `grill-this-skill` first; do not improvise scenarios.
- The request is about generic Promptfoo work unrelated to an agent skill
  (a bare prompt, a non-skill LLM app), or about running/debugging an
  already-authored suite's regressions → hand off to `promptfoo-evals`.

A second eval methodology exists — the agentskills.io project's own native
`evals/evals.json` format (`references/agent-skills-native-evals.md`) —
but it is not a target format for this skill and never will be authored
here; it is read only to mine transferable eval-design patterns (assertion
wording, baseline comparison) already folded into
`references/promptfoo-agent-skill-testing.md`. If a request explicitly
asks for an `evals/evals.json` file, decline and redirect to
`grill-this-skill` rather than treating it as this skill's job.

## Workflow

### 1. Gather grounding

Validate the target skill name against the skill-name rules in
`specs/references/agent-skills/skill-format.md` before using it to build any
file path or command. Treat it as data, not literal path syntax.

Read, in this order:

- the target skill's `specs/<skill-name>.md`, specifically its
  `### Evaluation plan` section (behavior to distinguish, scenario types,
  evidence source, judging approach);
- the target skill's `SKILL.md` body — the ground truth for expected
  behavior; scenarios must trace back to it, not to assumptions;
- any existing `skills/<skill-name>/evals/` content, if updating rather
  than creating.

If the spec has no `### Evaluation plan` section, or it is missing a
scenario type, evidence source, or judging approach, stop and redirect to
`grill-this-skill`. Do not fabricate the missing piece.

While reading the target's `SKILL.md`, note which design pattern it
follows (Tool: precise operations on a format; Navigation: routes between
distinct scenarios; Process: multi-step; Mindset/Philosophy: creative,
high-freedom output) — it predicts which assertion style and scenario type
will actually exercise the skill in step 4. See
`references/skill-judge-lessons.md` for why. Scoring the skill's adherence
to the pattern is a different skill's job; identifying it here is just
input to this skill's own assertion choices.

### 2. Choose how to invoke a multi-turn or agentic skill

Most skills in this repo are multi-turn or tool-using (interview flows,
multi-step agent workflows) rather than a bare prompt. Promptfoo now has
first-class support for this: the `anthropic:claude-agent-sdk` provider
(also `openai:codex-sdk`, `opencode:sdk`) can invoke a skill installed
under `.claude/skills/` or `.agents/skills/` directly, and the `skill-used`
and `trajectory:*` assertions check which skill/tool ran and in what order.

Read `references/promptfoo-agent-skill-testing.md` — it is the default
starting point for evaluating a skill in this repo, and also covers the
hand-rolled Python/JS/HTTP provider fallback for when the target skill
runs under something native support doesn't cover. Say so explicitly
rather than silently choosing the fallback.

Either way, a test case built this way still only proves what the provider
and its assertions actually observe (skill routing, traced tool steps, or a
judged final output) — label anything short of that plainly as a partial
check, not a full behavioral test of the workflow.

### 3. Scaffold the suite

Use the canonical layout under `skills/<skill-name>/evals/`:

```text
evals/
  promptfooconfig.yaml
  prompts/
  tests/
```

Run `scripts/init-skill-eval skills/<skill-name>` to create this structure
with a schema-aware `promptfooconfig.yaml` and a minimal prompt template.
The command is safe to run even when `evals/` already exists as an empty
directory left by skill scaffolding (`init-skill`); it only refuses when
`promptfooconfig.yaml` already exists, meaning a suite was already
initialized.

### 4. Map scenarios to test cases

For each scenario type in the spec's evaluation plan (normal case, near
miss, scope-boundary near miss, and so on), write one Promptfoo test case:

- Word the prompt/input from the scenario's own description in the spec;
  do not invent new scenarios.
- Set a `description:` on the test case stating the scenario and expected
  behavior, not just the prompt — this is what lets a future reader (or
  future you) recover *why* the test exists once the target `SKILL.md`
  has moved on from its current wording.
- If the spec's scenario types have no scope-boundary/routing case at all,
  flag that gap before writing test cases — a skill that never fires in
  practice fails regardless of how its content assertions would have
  scored (`references/skill-judge-lessons.md`).

From ibuchanan/agent-skills-toolkit, Apache-2.0.

More from ibuchanan/agent-skills-toolkit