skill-creator

Files (22)

• README-en.md
• README-es.md
• README-ptbr.md
• README.md
• SKILL.md
• analyzer.md · agents/
• comparator.md · agents/
• grader.md · agents/
• eval_review.html · assets/
• generate_review.py · eval-viewer/
• viewer.html · eval-viewer/
• schemas.md · references/
• __init__.py · scripts/
• aggregate_benchmark.py · scripts/
• generate_report.py · scripts/
• improve_description.py · scripts/
• package_skill.py · scripts/
• quick_validate.py · scripts/
• run_eval.py · scripts/
• run_loop.py · scripts/
• runtime.py · scripts/
• utils.py · scripts/

SKILL.md (first lines)

# Skill Creator

A skill for creating new Kiro skills and iteratively improving them.

## High-level workflow

1. **Capture intent.** What should the skill do? When should it trigger? What test cases?
2. **Draft `SKILL.md`.** Frontmatter (`name`, `description`) + imperative-form body.
3. **Author bundled resources** (only what helps): `agents/`, `scripts/`, `references/`, `assets/`.
4. **Run trigger evals.** `scripts/run_eval.py` — tests whether the description triggers correctly via `runtime.py`.
5. **Review with the user.** Use `eval-viewer/generate_review.py` to show results in-browser.
6. **Iterate.** `scripts/run_loop.py` auto-improves the description across iterations with train/test split.
7. **Package.** `scripts/package_skill.py` creates a distributable `.skill` zip.

Your job: figure out which stage the user is at and jump in from there.

## Runtime backends

Eval scripts invoke a model to test triggering. They route through `scripts/runtime.py`:

| Backend | Selector | Invocation |
|---|---|---|
| `kiro_cli` | `KIRO_RUNTIME=kiro_cli` or `which kiro` | `kiro -p "<query>" --output-format stream-json` |
| `kiro_ide_terminal` | `KIRO_RUNTIME=kiro_ide_terminal` | shells to `$KIRO_IDE_BIN` with IDE session env |
| `kiro_ide_prompt` | `KIRO_RUNTIME=kiro_ide_prompt` | writes prompt to `$KIRO_IDE_PROMPT_FIFO` |
| `mock` | `KIRO_RUNTIME=mock` | deterministic, no network |

**Auto-detection order:** `kiro_cli` → `mock`.

**Env vars** (all optional):
- `KIRO_RUNTIME` — force a backend
- `KIRO_BIN` — path to Kiro CLI binary (default: `kiro`)
- `KIRO_PROMPT_FLAG` — prompt flag (default: `-p`)
- `KIRO_OUTPUT_FORMAT_FLAG` — stream flag (default: `--output-format stream-json`)
- `KIRO_IDE_PROMPT_FIFO` — for `kiro_ide_prompt` backend

Trampoline skill files are written to `.kiro/skills/` so Kiro can discover them during trigger testing.

## Authoring a SKILL.md

Frontmatter is YAML between `---` markers.

```yaml
---
name: my-skill        # kebab-case, ≤64 chars
description: One declarative sentence that names the user intent and lists trigger contexts.
---
```

**Description tips:**
- Imperative voice: "Use this skill for…" not "This skill does…"
- Focus on user intent, not implementation details
- ≤1024 characters (hard limit — descriptions over this are truncated)
- The description competes with other skills; make it distinctive

## Eval workflow details

### Writing eval sets

Create `evals/trigger-evals.json`:

```json
[
  {"query": "create a skill for summarizing PDFs", "should_trigger": true},
  {"query": "fix the bug in my Python script",     "should_trigger": false}
]
```

### Running evals

```bash
# From the skill-creator directory:
python -m scripts.run_eval \
  --eval-set path/to/trigger-evals.json \
  --skill-path path/to/your-skill \
  --verbose
```

The backend is auto-selected. Override with `KIRO_RUNTIME=kiro_cli python -m scripts.run_eval ...`.

### Optimization loop

```bash
python -m scripts.run_loop \
  --eval-set path/to/trigger-evals.json \
  --skill-path path/to/your-skill \
  --model auto \
  --max-iterations 5 \
  --verbose
```

This runs eval → improve → eval in a loop, opens a live HTML report, and prints the best description at the end.

## Agents

The `agents/` directory has three sub-agents used in the full benchmark workflow:

| Agent | Role |
|---|---|
| `agents/analyzer.md` | Post-hoc analysis: why did the winner win? |
| `agents/comparator.md` | Blind comparison of two outputs |
| `agents/grader.md` | Grades expectations against a transcript |

See `references/schemas.md` for the JSON schemas each agent produces.

## Scripts reference

| Script | Purpose |
|---|---|
| `scripts/runtime.py` | Backend adapter (kiro/mock) |
| `scripts/utils.py` | `parse_skill_md()` helper |
| `scripts/run_eval.py` | Single eval run — trigger accuracy |
| `scripts/improve_description.py` | Rewrite description based on eval failures |
| `scripts/run_loop.py` | Eval + improve loop |
| `scripts/aggregate_benchmark.py` | Aggregate multiple benchmark runs |
| `scripts/generate_report.py` | Generate HTML report from loop results |
| `scripts/quick_validate.py` | Validate SKILL.md structure |
| `scripts/package_skill.py` | Package skill into `.skill` zip |
| `eval-viewer/generate_review.py` | Serve interactive eval review page |

From salomao-santos/kiro-bridge-tools, Apache-2.0.

More from salomao-santos/kiro-bridge-tools