better-skill-creator

Files (200)

• LICENSE.txt
• NOTICE
• SKILL.md
• analyzer.md · agents/
• comparator.md · agents/
• grader.md · agents/
• audit.md · agents/panel/
• characterizer.md · agents/panel/
• composer.md · agents/panel/
• seat-frame.md · agents/panel/
• eval_review.html · assets/
• generate_review.py · eval-viewer/
• viewer.html · eval-viewer/
• description-optimization.md · references/
• distribution.md · references/
• environments.md · references/
• frontmatter.md · references/
• grading-panel.md · references/
• how-skills-load.md · references/
• judge-registries.md · references/
• schemas.md · references/
• requirements.txt
• __init__.py · scripts/
• aggregate_benchmark.py · scripts/
• gate_panel.py · scripts/
• generate_report.py · scripts/
• improve_description.py · scripts/
• package_skill.py · scripts/
• preflight.py · scripts/
• quick_validate.py · scripts/
• render_seats.py · scripts/
• run_eval.py · scripts/
• run_loop.py · scripts/
• utils.py · scripts/
• validate_grading.py · scripts/
• __init__.py · tests/
• benchmark.json · tests/fixtures/abstain/iteration-1/
• eval_metadata.json · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/
• grading.json · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/with_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/with_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/with_skill/run-1/
• grading.json · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/without_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/without_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-0-every-check-abstained/without_skill/run-1/
• eval_metadata.json · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/
• grading.json · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/with_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/with_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/with_skill/run-1/
• grading.json · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/without_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/without_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-1-thin-versus-complete/without_skill/run-1/
• eval_metadata.json · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/
• grading.json · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/with_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/with_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/with_skill/run-1/
• grading.json · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/without_skill/run-1/
• out.txt · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/without_skill/run-1/outputs/
• timing.json · tests/fixtures/abstain/iteration-1/eval-2-both-reasons/without_skill/run-1/
• SKILL.md · tests/fixtures/hostile-skill/
• keeper.md · tests/fixtures/hostile-skill/assets/nested/evals/
• …and 140 more

SKILL.md (first lines)

# Better Skill Creator

Build a skill, find out whether it actually helps, and improve it on the evidence.

**A SKILL.md is prompt bytes.** Before drafting one, invoke `prompt-engineering`
and `hypershot-protocol` via the Skill tool — both, before the first authored
byte — and draft against them. If either is not installed, say so once and
continue without it.

The loop:

1. Work out what the skill should do, and whether it should be a skill at all
2. Draft it, and validate the draft immediately
3. Run it against real test prompts — with the skill and without — so you have a comparison
4. Put the results in front of the person and get their read
5. Improve it on what you both learned
6. Repeat until it stops getting better
7. Tune the description so it actually triggers, then package it

Your job is to work out where in that loop this person already is, and start there. Someone saying
"I want a skill for X" needs step 1. Someone arriving with a draft needs step 3. Someone saying "it
never fires" needs step 7 and nothing else. And if they say "skip the evals, just work with me on
it" — do that. The loop is a default, not a gate.

## Read this before you trust a number

The costliest failure here is not a crash but a measurement that looks fine and means nothing — a
benchmark computed from zero discovered runs, a trigger score from probes that never executed, a
token column derived from character counts.

The tooling refuses instead of guessing: zero runs exits non-zero, unmeasured values render `—` and
never `0`, an errored probe is recorded as an error and not as a clean negative. So **if a number
looks surprising, check it came from data before acting on it**, and when you report results say what
was measured and what wasn't. "The benchmark shows no difference" and "the benchmark found no runs"
are different sentences, and only one is about the skill.

## Talking to the person you're helping

The people who use this span a wide range — some write software for a living, some opened a terminal
for the first time last week. Read the cues in how they write and match them. Rough calibration:
"evaluation" and "benchmark" are usually fine; "JSON", "assertion", and "held-out set" want either
evidence they know the term or a half-sentence of explanation the first time. Explaining briefly
costs almost nothing; a term that lands as noise costs you their attention.

---

## Creating a skill

### Work out what it's for

The conversation you're already in often contains the answer — if they said "turn this into a
skill," the workflow is right there in the history. Pull out the steps, the tools used, the
corrections along the way, the shape of the input and output, then fill the gaps with them rather
than guessing.

What you need before drafting:

1. What should this let Claude do that it doesn't reliably do now?
2. When should it fire? What would someone actually be saying or doing?
3. What does good output look like?
4. Is this even a skill? Some things are better as a slash command, a sub-agent, a CLAUDE.md
   convention, or a script. A skill earns its place when the work is multi-step and Claude needs
   *judgment* plus *procedure*, not when it needs a single fixed action.

Ask about edge cases, formats, example files, and what "done" means, before writing test prompts.

### Draft it

Write the SKILL.md. `references/frontmatter.md` has the field-by-field detail and the YAML hazards;
the two that carry the weight are:

- **`name`** — kebab-case, and it should **match the directory name**. Claude Code takes the
  invocation name from the directory and treats this field as display-only; uploading and packaging
  both require them to agree.
- **`description`** — the entire mechanism by which Claude decides to consult the skill. Nothing
  else in the file participates in that decision.

Quote the description. An unquoted colon-space makes YAML parse it as a nested mapping, and the body
then loads with empty metadata — so typing `/your-skill` works while auto-triggering silently never
fires, and you go tune a description that was never read at all.

### Validate before you spend anything

```bash
python -m scripts.quick_validate <path-to-skill>
```

Run it the moment a draft exists, and again after any frontmatter edit. It reports every problem at
once rather than the first, and names which target each finding came from.

That last part matters, because there is no single frontmatter spec — Claude Code recognizes 31 keys
and ignores unknown ones, the portable agentskills.io set is 6 keys and rejects the rest, and
claude.ai caps descriptions at 200 characters. A skill can be valid for where it's going and still
fail a checker aimed somewhere else, so use `--target` and read each finding as "wrong *for this
target*" rather than "wrong." `references/frontmatter.md` has the per-target table.

### What makes a skill work

```
skill-name/
├── SKILL.md          required — frontmatter + instructions
├── scripts/          code that runs; doesn't consume context unless read
├── references/       detail loaded on demand
└── assets/           files used in output
```

**Only a prefix of SKILL.md survives.** When the conversation compacts, Claude Code re-attaches
roughly the **first 19,900 characters** of each invoked skill. It's a character slice: it cuts
mid-sentence, with no marker, and what falls off is gone at exactly the moment a long session needs
it.

So character count is the check that matters, and **order is a design decision**:

```bash
python -c "print(len(open('SKILL.md',encoding='utf-8').read()))"
```

Count characters, not bytes — `wc -c` reports bytes, and substantial non-ASCII content makes the two
diverge enough to send you cutting material that was never at risk.

From lilCnid/dovetail, Apache-2.0.

More from lilCnid/dovetail