eval

Files (1)

• SKILL.md

SKILL.md (first lines)

# /eval: Skill Evaluation & Self-Improvement

Evaluate skills across three tiers, track quality over time, and suggest improvements
based on failure analysis. The goal: skills that get better with every iteration.

## Parse Arguments

```
/eval [skill-name]            → Run all tiers on the named skill
/eval --static                → Tier 1 only, all skills (fast, free)
/eval --judge [skill-name]    → Tier 2: LLM quality scoring
/eval --e2e [skill-name]      → Tier 3: E2E test via claude -p
/eval --improve [skill-name]  → Analyze failures, suggest edits
/eval --compare A B           → Blind A/B comparison
/eval --history [skill-name]  → Show score history
/eval --regression            → Compare latest scores against baseline
```

## Output Directory

```bash
mkdir -p .ai-team/evals
```

All eval results go to `.ai-team/evals/`:
```
.ai-team/evals/
├── history.jsonl                    # Append-only score log
├── baseline.json                    # Baseline scores for regression detection
├── browse/
│   ├── eval-2026-03-17.json         # Full eval result
│   ├── eval-2026-03-17-improve.md   # Improvement suggestions
│   └── test-cases.md                # Eval definitions
├── qa-browser/
│   └── ...
└── review/
    └── ...
```

---

## Tier 1: Static Validation (free, <5s)

Structural checks — no LLM calls. **Token-efficient: use Bash to parse files
instead of reading each SKILL.md into context.**

Determine the skills directory to operate on:
```bash
SOURCE_REPO=$(cat ~/.claude/.ai-team-source 2>/dev/null)
if [ -n "$SOURCE_REPO" ] && [ -d "$SOURCE_REPO/skills" ]; then
  SKILLS_DIR="$SOURCE_REPO/skills"    # Edit source repo directly
else
  SKILLS_DIR="$HOME/.claude/skills"   # Fallback to installed copies
fi
```

### Run static checks via script (DO NOT read each SKILL.md individually)

Run the static validator script which checks all skills in a single Bash call
and returns a compact report. This avoids reading 30+ SKILL.md files into context
(~42K tokens). Only read individual SKILL.md files if a skill FAILS and you need
to investigate or fix it.

```bash
bash ~/.claude/scripts/eval-static.sh "$SKILLS_DIR"
```

The script checks each `$SKILLS_DIR/*/SKILL.md`:

### 1.1 Frontmatter Check
- Has `name` field
- Has `description` field (non-empty, >20 chars)
- `allowed-tools` is a list (if present)
- `name` matches directory name

### 1.2 Reference Integrity
- All file references in the SKILL.md actually exist
- No broken relative paths

### 1.3 Command Consistency
- If skill references `playwright-cli`, it has `Bash(playwright-cli:*)` in allowed-tools
- If skill references `git` commands, it has `Bash(git:*)` in allowed-tools
- Tool references in the body match allowed-tools declarations

### 1.4 Template Freshness (if .tmpl exists)
```bash
bash ~/.claude/scripts/gen-skill-docs.sh --check
```

### 1.5 Description Trigger Quality
- Description is specific enough to distinguish this skill from others
- No overly generic phrases ("use for anything", "general purpose" without specifics)
- Contains concrete trigger keywords/phrases

### Static Score
Each check: PASS (1) or FAIL (0). Score = passes / total checks * 100.

---

## Tier 2: LLM-Judge Quality Scoring (uses Claude Code subscription)

Use `claude -p` to score the skill document on five dimensions. This runs through
your Claude Code subscription — no API key or extra charges needed.

```bash
cat $SKILLS_DIR/{skill}/SKILL.md | claude -p "You are evaluating the quality of an AI agent skill document.
Score each dimension 1-5 (1=poor, 5=excellent). Be critical — 3 is average.

Dimensions:
1. Clarity — Is the skill easy to understand? Are instructions unambiguous?
2. Completeness — Does it cover all cases? Edge cases? Error handling?
3. Actionability — Can an agent follow this without guessing? Are examples concrete?
4. Efficiency — Is it concise? No redundancy? Good information density?
5. Trigger precision — Will the description correctly match intended use cases
   without false positives or false negatives?

Output JSON only:
{\"clarity\": N, \"clarity_note\": \"...\",
 \"completeness\": N, \"completeness_note\": \"...\",

From aitechnerd/ai-dev-team, MIT.

More from aitechnerd/ai-dev-team