skill-maker

Files (59)

• LICENSE.txt
• SKILL.md
• eval_review.html · assets/
• README.md · evals/
• __init__.py · evals/
• evals.json · evals/
• failure-modes.md · evals/
• SKILL.md · evals/files/broken-skill/
• SKILL.md · evals/files/eval-suite-valid/
• evals.json · evals/files/eval-suite-valid/evals/
• failure-modes.md · evals/files/eval-suite-valid/evals/
• trigger_queries.json · evals/files/eval-suite-valid/evals/
• SKILL.md · evals/files/skip-evals-skill/
• SKILL.md · evals/files/standup-summary/
• grade_artifacts.py · evals/
• judge_labels.json · evals/
• generic-procedure-detection.md · evals/judges/
• safety-least-surprise.md · evals/judges/
• self-improvement-overfitting.md · evals/judges/
• self-improvement-scope.md · evals/judges/
• skip-evals-scope.md · evals/judges/
• eval-1-create-from-workflow-grading.json · evals/runs/2026-09-30/
• eval-2-make-spec-compliant-grading.json · evals/runs/2026-09-30/
• eval-4-improve-installed-skill-grading.json · evals/runs/2026-09-30/
• eval-5-safety-refusal-grading.json · evals/runs/2026-09-30/
• eval-6-skip-evals-grading.json · evals/runs/2026-09-30/
• judge-dry-run.md · evals/runs/2026-09-30/
• summary.md · evals/runs/2026-09-30/
• trigger_queries.json · evals/
• authoring-guide.md · references/
• description-optimization.md · references/
• environment-adaptations.md · references/
• evaluation.md · references/
• schemas.md · references/
• scripts.md · references/
• spec-provenance.md · references/
• spec-reference.md · references/
• requirements.txt
• __init__.py · scripts/
• eval_adapter.py · scripts/
• eval_models.py · scripts/
• eval_store.py · scripts/
• install_skill.py · scripts/
• package_skill.py · scripts/
• quick_validate.py · scripts/
• skill_eval.py · scripts/
• utils.py · scripts/
• __init__.py · tests/
• __init__.py · tests/fixtures/
• fake_eval_adapter.py · tests/fixtures/
• test_eval_adapter.py · tests/
• test_eval_models.py · tests/
• test_evals_data.py · tests/
• test_grade_artifacts.py · tests/
• test_install_skill.py · tests/
• test_package_skill.py · tests/
• test_quick_validate.py · tests/
• test_skill_eval.py · tests/
• test_utils.py · tests/

SKILL.md (first lines)

# Skill Maker

Create and improve portable Agent Skills. A skill is a directory containing
`SKILL.md` plus optional `scripts/`, `references/`, and `assets/`. Keep the
portable artifact self-contained. Resolve bundled paths relative to this skill's
own root. Do not search another checkout for missing resources.

## Core loop

The core loop branches by task. Each path starts with the same foundation:

- **Decide** whether the content is a skill (discrete task-class capability) or an
  always-on instruction (ambient discipline). Read `references/authoring-guide.md`.
- **Inspect** an existing skill with import mode before changing it:
  ```bash
  python -m scripts.quick_validate <skill> --mode import --json
  ```
  Import mode reports client extensions and name mismatches as warnings so you
  can examine an existing skill without rejecting it. Always **deliver** with
  strict validation:
  ```bash
  python -m scripts.quick_validate <skill>
  ```
  Strict mode rejects non-portable fields and is required by package/install.
- **Capture** expertise, triggers, output format, dependencies, and boundaries
  from real artifacts (runbooks, schemas, API docs, failure reports).
- **Draft** the portable core: `name` (kebab-case, matching directory),
  `description` (action + scope + trigger signal), and concise body with
  progressive disclosure. Keep the body under 500 lines / ~5000 tokens.
- **Add direct conditional resource pointers**: name each `scripts/` and
  `references/` file in the body with a one-line purpose and when to read it.
  Bundled scripts not named in `SKILL.md` trigger audit warnings.
  Flatten references to one level under `references/`.
  Read `references/scripts.md` when bundling executable code.
  Read `references/spec-provenance.md` when tracing the authority tier of
  a spec, validator, or hardening rule.
  Use `assets/eval_review.html` to present trigger-review results when a
  browser is available (fill `__EVAL_DATA_PLACEHOLDER__`,
  `__SKILL_NAME_PLACEHOLDER__`, and `__SKILL_DESCRIPTION_PLACEHOLDER__`
  with real data before opening).
- **Establish or preserve** task and trigger baselines before edits. Read
  `references/evaluation.md` and `references/description-optimization.md`. Read
  `references/skill-optimization.md` when diagnosing task quality, context cost, or tool
  efficiency, or when choosing among optimization experiments.
- **Make the smallest justified change** to a writable copy; never modify source
  fixtures or an installed read-only directory.
- **Run paired evidence** (with-skill vs baseline), then select by held-out
  results.
- **Strictly validate**, then package or install:
  ```bash
  python -m scripts.package_skill <skill> <dist>
  python -m scripts.install_skill <skill> [--target <dir>] [--force]
  ```

### Create a new skill

Follow the core loop foundation. Draft the portable core with only the six
recognized frontmatter fields. Omit `allowed-tools` by default; pre-approval is
client-specific and requires explicit trust review. Agent-agnostic output uses
only the portable fields. Client-specific fields, install paths, precedence,
plugins, hosted IDs, and marketplace metadata remain opt-in target adapters
outside the canonical artifact; do not invent an adapter when no target was
requested.

### Update an existing skill

1. Preserve the skill name and unrelated behavior.
2. Inspect with import mode to surface client extensions without silently deleting them.
3. Copy to a writable workspace. Make the smallest justified change.
4. Revalidate strictly on the copy.
5. Require an explicit portable-conversion decision before stripping or relocating
   client extensions.

### Improve a skill (sandboxed candidate loop)

```bash
python -m scripts.skill_eval improve <skill> --workspace <workspace> \
  --max-iterations 3 --adapter-arg <executable> --adapter-arg <argument>
```

Revision adapters receive a temporary staging directory outside the workspace.
Workspace markers (grading.json, history.json, evals/, tests/) are unreachable
by walking ancestors from the provided paths. Held-out prompts, labels, and
scores are never sent to revision requests.

Read `references/evaluation.md` for the full adapter contract, workspace
artifacts, exit codes, benchmark diagnostics, and manual fallback.

## Authoring rules

Read `references/spec-reference.md` before changing frontmatter and
`references/authoring-guide.md` before drafting the body. Use only the six
portable frontmatter fields by default. `name` must be kebab-case and match the
directory. `description` must say what the skill does and when to use it, with
a first-sentence routing signal. Write imperative, specific instructions and
explain why important steps exist. Avoid hidden access, data exfiltration,
misleading descriptions, and client-specific assumptions.
Use a comma, period, colon, or parentheses instead of U+2014.

Format validation is not a safety review. The bundled validator checks format
constraints, not whether the skill's behavior matches its description.

Always-on instructions remain separate from skills. If the honest trigger
description would be "use this on every task", recommend the always-on layer
instead.

## Evaluation and improvement

The shipped evaluator uses the `skill-eval/v1` command adapter. It runs
deterministic expectations locally, sends only `judge` expectations to the
adapter, and records paired with-skill and baseline results. Command
expectations are denied unless `--allow-command-checks` is explicit.
Import `scripts/utils.py` for shared helpers (`parse_frontmatter`,
`should_exclude`, `walk_skill`) when writing custom evaluation or
packaging tools.

The offline audit (`python -m scripts.skill_eval audit`) now includes
progressive-disclosure and portable-resource warnings:
- `skill.allowed-tools`: experimental pre-approval field requires trust review.

From klarrimore/skill-maker, Apache-2.0.