skill-creator

Files (38)

• LICENSE.txt
• SKILL.md
• analyzer.md · agents/
• comparator.md · agents/
• grader.md · agents/
• eval_review.html · assets/
• generate_review.py · eval-viewer/
• viewer.html · eval-viewer/
• evals.json · evals/
• SKILL.md · evals/files/incident-summarizer/
• SKILL.md · evals/files/log-triage/
• competition-evals.json · evals/files/log-triage/
• reviewed-trigger-evals.json · evals/files/log-triage/
• vendor-invoices.csv · evals/files/
• trigger-evals.json · evals/
• anthropic-best-practices.md · references/
• description-optimization.md · references/
• environment-adaptations.md · references/
• evaluation-workflow.md · references/
• graphviz-conventions.dot · references/
• persuasion-principles.md · references/
• schemas.md · references/
• skill-quality-standard.md · references/
• testing-pressure-scenarios.md · references/
• __init__.py · scripts/
• aggregate_benchmark.py · scripts/
• generate_report.py · scripts/
• improve_description.py · scripts/
• package_skill.py · scripts/
• quick_validate.py · scripts/
• run_competing_eval.py · scripts/
• run_eval.py · scripts/
• run_loop.py · scripts/
• utils.py · scripts/
• test_generate_review.py · tests/
• test_improve_description.py · tests/
• test_run_eval.py · tests/
• test_run_loop.py · tests/

SKILL.md (first lines)

# Skill Creator

Create and refine skills through evaluation-driven development: observe the unassisted
failure, add the smallest useful guidance, and verify the behavioral change.

This skill owns the authoring and validation workflow. The `writing-great-skills` skill
owns prose craft: invocation load, leading words, information hierarchy, context pointers,
and pruning. Apply both when editing skill text.

## Critical rules

1. **Gate creation first.** Confirm that a skill is the right artifact before creating one.
2. **Establish RED before drafting.** For a new skill, run representative prompts without
   it. For an existing skill, snapshot the old version before editing and use that snapshot
   as the baseline.
3. **Grade behavior, not vibes.** Write observable assertions whose truth can be supported
   by files, command output, or quoted evidence. Keep subjective output for human review.
4. **Preserve comparison integrity.** Run baseline and candidate against the same prompt,
   input files, model, and environment.
5. **Let the user judge qualitative quality.** Show outputs and benchmark evidence before
   claiming that one version is better.
6. **Apply the local standard.** In this repository, every touched skill must pass
   [`references/skill-quality-standard.md`](references/skill-quality-standard.md).

## Quick start

1. **Decide** whether the need belongs in a skill and classify its type.
2. **Capture** 2–3 representative evals and run the baseline.
3. **Draft** only enough guidance to address observed failures.
4. **Compare** candidate and baseline, grade objective assertions, and show both to the user.
5. **Iterate** on observed failures, then validate description triggering and deployment.

The skill is ready only when its applicable quality checks pass, the candidate has evidence
against the baseline, and the user has reviewed qualitative outputs.

## Should this be a skill?

Create or retain a skill when the behavior is recurring, specialized, and benefits from
instructions that an agent must discover or a human must deliberately invoke.

Choose a different artifact when the need is primarily:

| Need | Better artifact |
|---|---|
| One conversation or one-off transformation | Prompt or prompt template |
| Fully deterministic operation | Script or tool |
| Repository-wide contributor rule | `AGENTS.md` or project documentation |
| Architectural choice and rationale | ADR |
| Large factual corpus with no agent workflow or discovery need | External reference |
| Existing skill already owns the same ground | Improve or route to that skill |

Before creating anything, search the available skills for overlap. Split a proposed skill only
when a branch needs independent invocation or when a sequence boundary must hide later steps;
do not split merely to shorten a file.

## Full workflow

### 1. Capture intent and classify

Extract answers already present in the conversation before asking the user. Resolve:

- What behavior should change?
- Which user requests or project conditions should invoke it?
- What output or action proves success?
- Which existing skill is nearest, and why is extending it insufficient?
- Is the output objectively gradable, qualitatively reviewable, or both?

Classify the skill because the type determines its evals:

| Type | Distinguishing question | Default evaluation |
|---|---|---|
| **Discipline** | Does it enforce a costly rule agents may rationalize away? | Functional evals plus pressure scenarios |
| **Technique** | Does it prescribe concrete steps for performing a task? | Functional evals |
| **Pattern** | Does it teach recognition and a reusable decision model? | Functional plus recognition evals |
| **Reference** | Does it supply facts, APIs, or syntax needed on demand? | Functional plus retrieval evals |

A technique says **how to act**; a pattern says **what shape to recognize and reason with**.
If one skill contains independent types with different invocation conditions, reconsider its
boundary.

**Completion criterion:** the intended behavior, invocation mode, nearest existing skill,
type, success evidence, and expected output are explicit.

### 2. Establish the baseline (RED)

Create 2–3 prompts that resemble real requests, including input files when the task depends on
them. Write verifiable assertions before seeing candidate output so the target cannot move to
fit the result.

- **New skill:** run the prompts without the skill.
- **Existing skill:** copy the original skill to the evaluation workspace before editing, then
  run prompts with that snapshot.
- **Discipline skill:** use scenarios combining at least three pressures and record the exact
  rationalizations. Read
  [`references/testing-pressure-scenarios.md`](references/testing-pressure-scenarios.md)
  before writing those scenarios.
- **Technique skill:** include successful execution plus a realistic error or recovery case.
- **Pattern skill:** include recognition cases and vocabulary-sharing near-misses.
- **Reference skill:** include retrieval cases and cases where the reference does not apply.

Save the eval definitions in `evals/evals.json`. Use the schema in
[`references/schemas.md`](references/schemas.md).

**Completion criterion:** every eval has a baseline output and at least one documented failure
or gap that the proposed skill change will address. If the baseline already succeeds, sharpen
the scenario or reject unnecessary skill content.

### 3. Draft the candidate (GREEN)

Write the smallest change that addresses the baseline failures.

For frontmatter:

- Preserve an existing skill's `name`.
- Choose invocation mode deliberately.
- For model-invoked skills, start the description with “Use when” or “Use whenever,” include
  only triggering conditions, use third person, stay within three sentences and about 75 words,
  and keep any trigger list to eight user-shaped items.
- For user-invoked skills, set `disable-model-invocation: true` and use one plain human-facing

From dekoza/oh-my-slop, Unlicense.

More from dekoza/oh-my-slop