pi-skill-creator

Files (17)

• README.md
• SKILL.md
• analyzer.md · agents/
• comparator.md · agents/
• grader.md · agents/
• generate_review.py · eval-viewer/
• evals.json · evals/
• evals_array.json · evals/
• trigger_evals.json · evals/
• verify_code_change_evals.json · evals/
• schemas.md · references/
• __init__.py · scripts/
• aggregate_benchmark.py · scripts/
• improve_description.py · scripts/
• literal_run_eval.py · scripts/
• literal_run_loop.py · scripts/
• package_skill.py · scripts/

SKILL.md (first lines)

# Pi Skill Creator

Create and iteratively improve skills for coding agent harnesses like **pi**, **opencode**, and **codex**.

The core loop: draft a skill → run test prompts → review outputs → improve → repeat.

## Workflow Overview

1. **Capture intent** — what should the skill do, when should it trigger
2. **Write SKILL.md** — name, description, instructions, examples
3. **Create test prompts** — 2-3 realistic user queries per skill
4. **Run evals** — spawn subagents with/without skill, compare outputs
5. **Review** — HTML viewer shows outputs + quantitative benchmark
6. **Improve** — rewrite based on feedback, repeat until satisfied
7. **Package** — bundle as `.skill` file for distribution

---

## Creating a Skill

### Step 1: Capture Intent

Ask the user:
1. What should this skill enable the agent to do?
2. When should it trigger? (what phrases/contexts)
3. Expected output format?
4. Test cases needed? (skills with verifiable outputs benefit most)

### Step 2: Write SKILL.md

```
skill-name/
├── SKILL.md          ← required
├── scripts/          ← optional helper scripts
├── references/       ← optional docs loaded on demand
└── assets/           ← optional templates, icons
```

**SKILL.md anatomy:**

```yaml
---
name: skill-name
description: When to trigger + what it does. Be "pushy" — overtrigger rather than undertrigger.
---

# Skill Name

## What this skill does

## When to use it

## How to use it

## Examples
```

**Description writing tips:**
- Use imperative: "Use this skill for..." not "This skill does..."
- Focus on user intent, not implementation
- Be distinctive — the description competes with other skills for the agent's attention
- Keep under 200 words / 1024 chars

### Step 3: Create Test Prompts

Save to `evals/evals.json`:

```json
{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "Realistic user query",
      "expected_output": "What the skill should produce",
      "files": []
    }
  ]
}
```

## Running and Evaluating

### Run all evals

```bash
python -m scripts.run_eval \
  --agent pi \
  --skill-path /path/to/skill \
  --workspace /tmp/skill-workspace \
  --evals evals/evals.json
```

This spawns subagents for each eval in parallel — one **with skill**, one **without** (baseline).

### Spawn subagents directly

Use `delegate_task` (if using the `pi-subagents` skill) or the pi CLI in each subagent:

**With-skill:**
```bash
cd /project && pi -p "task..." --skill my-skill
```

**Baseline (no skill):**
```bash
cd /project && pi -p "task..."
```

### Grading

```bash
python -m scripts.aggregate_benchmark /workspace/iteration-1 \
  --agent pi \
  --skill-name my-skill
```

### Review in browser

From kwanpham2195/agent-stuff, MIT.

More from kwanpham2195/agent-stuff