skillz

Files (48)

• SKILL.md
• openai.yaml · agents/
• records.py · assets/
• wedge.toml · assets/
• autoimprove.json · evals/
• evals.json · evals/
• cheese-skill.md · evals/files/
• fixture.md · evals/files/
• improve-offload-skill.md · evals/files/
• notes-skill.md · evals/files/
• report-skill.md · evals/files/
• wedge-supported-skill.md · evals/files/
• journey.py · evals/gates/
• journeys.json · evals/
• agent-orchestration.md · references/
• analytics-ceremony.md · references/
• anti-patterns.md · references/
• autoimprove-run.md · references/
• calibration.md · references/
• canonical-schema.md · references/
• criteria-drafting.md · references/
• decision-frameworks.md · references/
• description-optimization.md · references/
• drift-regression.md · references/
• eval-formats.md · references/
• experiment-harness.md · references/
• experiments.md · references/
• fromargs.md · references/
• harness-coverage.md · references/
• harness-layout.md · references/
• harness-recipes.md · references/
• hooks-catalog.md · references/
• offload.md · references/
• progressive-disclosure.md · references/
• query-conventions.md · references/
• raw-log-fallback.md · references/
• autoimprove.schema.json · references/schemas/
• draft-case.schema.json · references/schemas/
• evals.schema.json · references/schemas/
• journeys.schema.json · references/schemas/
• skill-usage.md · references/
• wedge-brief.md · references/
• wedge-packaging.md · references/
• agent-session-analytics.pyz · scripts/
• inspect-skill.pyz · scripts/
• skillz-experiment.pyz · scripts/
• wedge.pyz · scripts/
• wedge.toml

SKILL.md (first lines)

# skillz

Add, improve, audit, wedge, and autoimprove skills.
Improve and audit also accept agent definitions.
The product is a **predictable** definition: the same process on every run and on every harness.
Every lens asks one question of each line: *does this make the run more predictable, or is it sediment?*

The mode is the first word after the skill name.
The Aliases column lists the names that select the same mode.
Require a target for each mode.
Ask for the mode when it is missing.
Ask for a target when the mode requires one.

## Modes

| Mode | Aliases | Target | Analytics | Writes | Product |
|---|---|---|---|---|---|
| `add <name>` | none | a new skill name | no | creates `skills/<name>/` | a registered skill that passes the rubric |
| `improve <path>` | `optimize`, `tighten` | a `SKILL.md` or agent file | yes, unless it reuses a current audit | the analytics cache; the target after approval | approved fixes + residual findings |
| `audit <path>` | none | a `SKILL.md` or agent file | yes | the analytics cache only; never the target | calibrated report |
| `wedge <path>` | none | a `SKILL.md` | no | the target skill: each selected CLI, its delivery files, its call site, and any dependency write | an offload brief, then one CLI per selected candidate, each in its own directory when wedged, or with a stated reason |
| `autoimprove <path>` | `experiment` | a skill directory; no contract or manifest | cases that the user approves; agent-drafted session cases when analytics exists | isolated run directory | a holdout gate verdict and a private candidate patch |

`audit` never modifies the target. The Usage ceremony of `audit` and `improve` runs `scripts/agent-session-analytics.pyz ingest`, which creates or refreshes the analytics cache.
`references/analytics-ceremony.md` has the details. Skip Usage and the raw-log fallback write no cache.

For `autoimprove`, read `references/experiments.md`. Follow its workflow, not the shared audit protocol.
Before you write or check `evals/evals.json`, `evals/journeys.json`, or `evals/autoimprove.json`, read `references/eval-formats.md`.
Read `references/autoimprove-run.md` before you run `autoimprove`; it has the run, question, and report steps.

## Shared protocol

### 1. Read and classify

Read the target. Read a linked reference only when its stated trigger matches the run.
A full-package audit may explicitly read every file in the target package.
For a skill directory, run `python3 <this-skill-directory>/scripts/skillz-experiment.pyz audit-facts <skill-directory>` before applying the rubric.
Also run `python3 -I <this-skill-directory>/scripts/inspect-skill.pyz <target>` for the prose facts.
Use both JSON outputs as objective package facts, not a fitness score. An agent file has no package, so skip `audit-facts`.
Read `references/experiments.md § Audit facts contract` for the check ids and statuses.
Report a helper error without treating it as a successful audit.
Classify it as **agent** (`tools:` / `disallowedTools:` or an `agents/registry.yaml` entry) or **skill** (`name:` + `description:`).
For a skill, run `o=$(git ls-files -z -co --exclude-standard <skill-dir> | LC_ALL=C sort -z | xargs -0 sh -c 'for f; do b=$(git hash-object -- "$f") || exit 1; printf "%s %s\n" "$b" "$f"; done' sh) && [ -n "$o" ] && printf '%s\n' "$o" | git hash-object --stdin`.
Its output is the content id. It covers every package file, so a sidecar or reference edit changes it.
For an agent file, run `git hash-object <target>`. Its output is the content id.
When a command fails, the content id is `unavailable`.
Report the body size as `~N tok` against the 5k budget.
For a skill, copy it from the `body.token-estimate` check. For an agent file, count bytes/4 after the frontmatter block.
List the deploy targets the definition reaches (`references/harness-layout.md § Layout`).

### 2. Rubric

Score each lens.
Each row names the principle, the failure mode it catches, and a checkable test.
Read `references/anti-patterns.md` when a finding needs the expanded form.

| Lens | Principle → catches | Check |
|---|---|---|
| **Predictability** | Fixed protocol with checkable completion → «premature completion» | Every step ends on a done-condition an agent can verify. |
| **Invocation** | `description` = trigger conditions, front-loaded, third person → «workflow summary», «summary description» | Trigger phrases + "Do NOT use for"; no internal workflow; skills take length from `description.length`, agent files stay at ≤1024 chars by hand; first sentence carries the trigger. `references/description-optimization.md`. |
| **Portability** | Spec-core frontmatter plus additive Claude fields; user-only policy on every host → «Claude-only assumption», «sidecar missing», «`$ARGUMENTS` dependence» | Cite failed `audit-facts` checks for keys, sidecar pairing, and argument, skill-directory, or file-mention syntax. Judge by hand: args are parsed from the text after the skill name; skills are cross-referenced by `/name`; dispatch and GitHub ops name the contract before host syntax. Full matrix: `references/harness-layout.md`. |
| **Information hierarchy** | Disclose only what some runs skip; body ≤5k tok; references one level deep, each with a read trigger → «sprawl», «untriggered split», «`@file` force-load» | Cite failed `audit-facts` checks for the token budget, nested references, orphans, and missing read triggers. Judge by hand: relocation counts only when runs branch on the block and the `## References` entry names the trigger. `references/progressive-disclosure.md`. |
| **Prose (ASD-STE100)** | Active voice, present tense, one instruction per sentence, short sentences → «passive voice», «multi-instruction sentence», «long sentence» | Cite `inspect-skill.pyz` `long_sentences` facts (sentences over 25 words). Cite `advisory_sentences` (21 to 25 words) only for procedural steps. Report passive voice and multi-instruction sentences as findings. |
| **Leading words** | One pretrained word beats a restated triad → «duplication», «no-op weak word» | Collapse restatements; strengthen weak words (`be thorough` → `relentless`). |
| **Pruning** | Single source of truth; delete no-ops → «sediment» | No meaning in two places; no line the model obeys by default. Delete whole sentences. |
| **Deterministic offload** | Fixed computation runs as a bundled command, not regenerated prose → «inline script» | No step makes the model write or re-derive the same parse, count, filter, sort, or projection on every run; `scripts.invocation-line` covers the invocation line; `wedge.own-targets` and `wedge.foreign-binary` cover wedge state; judge whether each bundled script has an output contract. Fix with `references/offload.md`. |
| **Tool scoping** | Read-only / write-scoped / focused; use host enforcement when available → «prose-only constraint» | Claude skills use `disallowed-tools` to remove tools for the current turn. Their `allowed-tools` grants permission without prompts; it is not a deny list. Claude agents use `disallowedTools`. Report actual enforcement per mode and mark prose-only limits as degraded. Do not disable writes for `improve`. |
| **Context & fork** | Fork when output > ~500 lines or only a digest is needed → «monolithic output» | Fork matches size; a wrap-up signal exists; the `model-policy.*` checks cover `model:` + `effort:`. |
| **Prompt quality** | Positive framing, why-over-what, one strong example, "What this never does" → «negation-heavy», «rules without reasons» | Judgment tasks use a scaffold, not always/never. `references/decision-frameworks.md`. |
| **Calibration** | Judgment agents tag confidence × severity → «judgment without calibration» | `<certain>` / `<speculative>` / `<don't know>`; don't-know never surfaces. |
| **Output format** | Summary first, tables for findings, clean-vs-issues signal → «no output format» | Format defined; summary and detail split. |
| **Usage** *(audit, improve)* | Declared matches actual → «declared-vs-actual», «decay» | Declared tools are the used tools; error rate near baseline; usage not declining. |

### 3. Calibrate

Tag each finding with severity × confidence.
The kernel is `references/calibration.md`; the defaults:

- **Severity by lens** — Predictability, Invocation, Portability, Tool scoping, Calibration → `high`; Information hierarchy, Context & fork, Pruning → `high`/`medium`; Leading words, Prose, Deterministic offload, Output format, Usage → `medium`.
- **Confidence** — a cited line plus a concrete failure, a reference that does it right, or analytics data → `<certain>`; a checkable but unverified observation → `<speculative>`; a misread → `<don't know>`, dropped.
- **Re-derive** — re-read the target and re-derive each `<speculative>` finding once without the first pass. Drop it when it does not reproduce.
- Order by severity, then `<certain>` first.

## Prose check

`add` step 6 and `improve` step 5 run this check.

1. Run `python3 -I <this-skill-directory>/scripts/inspect-skill.pyz <target file>` on the `SKILL.md` or agent file.
2. Rewrite every sentence in `long_sentences` (over 25 words).
   Rewrite each `advisory_sentences` entry (21 to 25 words) that is a procedural step.
3. For a changed reference file, apply both limits by hand: 25 words, or 20 words for a procedural step.

The helper rejects a file that has no frontmatter, so it cannot check a reference file.

## Mode: add

1. Confirm the name: kebab-case, ≤64 chars, directory name equals `name:`, and no collision in `skills/`, `~/.claude/skills`, `~/.agents/skills`.
2. Collect from the argument, or ask for: a one-sentence purpose, ≥5 trigger phrases, anti-triggers, invocation policy, and the harness set.
   The invocation policy is model-invoked or user-only.
3. Write the skill from `references/harness-layout.md § Template`.
   Keep the body ≤2k tok at birth.
   Create a reference only for a block that some runs skip.
   Add `agents/openai.yaml` when the skill is user-only.
   Set `model` + `effort` only when the skill is model-invoked.
4. Write `evals/evals.json` from `references/harness-layout.md § Evals`.
   Add one trigger case for each trigger phrase and one near-miss case for each anti-trigger.
   Draft at least three task cases from the purpose and the Flow.
   Give each task case `assertions` that a grader can check in the output.
5. Register the skill in its repo's index (`references/harness-layout.md § Registration`).
   A repo-local skill stays out of every global list; place it per `references/harness-layout.md § Repo-local skills`.
6. Run the prose check on `SKILL.md` and on each changed reference before you report the mode as done.
7. Run `improve` on the new file once. Skip Usage, because a new skill has no sessions.
   Then run the repo's deploy step when one exists.
   Example deploy steps are a dotfiles sync, `npx skills add`, or a copy into the harness skills directory.

Done means: the file exists and the repo index names it, or every repo-local host path resolves.
`evals/evals.json` holds a case for each trigger phrase and anti-trigger, plus at least three task cases.
The deploy step exits 0 when one exists. The Invocation lens passes.
The inspector reports an empty `long_sentences` list.

From paulnsorensen/skillz-that-grillz, MIT.

More from paulnsorensen/skillz-that-grillz